NIWA speeds environmental modeling and hazard forecasting

The National Institute of Water & Atmospheric Research Ltd. (NIWA) is a New Zealand Crown Research Institute, with a global reputation for its expertise in water and atmospheric research. Its mission is to conduct leading environmental science to enable the sustainable management of natural resources for New Zealand and the rest of the planet. NIWA is a separate organization from the Meteorological Service of New Zealand Limited (MetService), which provides the country's daily national forecasts.

NIWA wanted to help create a new national capability for research on grand-challenge scientific problems—these are fundamental problems in science and engineering that potentially have broad social, political, economic and scientific impact. Equally, the organization wanted to improve its own modeling capabilities in order to create more accurate and more specific hazard forecasts.

Dr. Michael Uddstrom, Principal Scientist: Environmental Forecasting, Meteorology & Remote Sensing, and HPCF Manager at NIWA, says: “We are always striving to create better forecast models that more closely reflect New Zealand's complex geography, use more data, and deliver faster and more reliable forecasts of hazards such as river flooding and coastal inundation. In addition, we wanted to support research into other grand-challenge problems, to enable New Zealand scientists to carry out internationally significant work. Our existing supercomputer from Cray was effectively five years past its ‘use-by’ date.”

Intensive selection

NIWA required a supercomputer offering both extreme compute power and very high-speed networking. The organization compiled a request for proposals (RFP) spanning 288 specific requirements and specifying eight science- and two industry-benchmark codes.

“We were setting out to create a major strategic asset for NIWA and for New Zealand, and making a major investment, so it was vital to find the best possible solution for our requirements,” says Michael Uddstrom.

After an exhaustive analysis of the vendor responses, the decision-makers at NIWA unanimously chose IBM to supply the new supercomputer, named FitzRoy in honor of the pioneering meteorologist. “We had never worked with IBM before, but it soon became clear that their proposed solution was the superior option and would be backed by genuine expertise in high-performance computing,” says Michael Uddstrom.

Powering up

The FitzRoy supercomputer at NIWA consists of 56 IBM Power® 575 servers, each with 32 IBM POWER6 processing cores running AIX® 5.3L, for a total of 1,792 cores. Each of the cores runs at 4.7 GHz, and the system has 5.38 TB of memory; a planned upgrade will see FitzRoy grow to 3,456 processor cores and 8.7 TB of memory. Management tools for the supercomputer include IBM Tivoli® Workload Scheduler LoadLeveler®, which enables NIWA to optimize the utilization of compute resources at all times.

FitzRoy uses ultra-fast InfiniBand networking to connect to a data storage cluster powered by IBM General Parallel File System (GPFS™) running on IBM Power Systems™ servers. The data itself is stored on two high-performance IBM System Storage® DCS9900 arrays, currently with 600 TB capacity and due to be upgraded to 2 PB. IBM Tivoli Storage Manager software enables hierarchical storage management, whereby data that is not required for high-speed access can be moved to lower-cost media. Such data is first moved from the SAS tier to the slower SATA tier in the DCS9900, then ultimately to an IBM System Storage TS3500 Tape Library. A second IBM TS3500 backs up all data for disaster recovery purposes.

“Before implementing the supercomputer, we first needed to build the infrastructure to house it, including a water cooling system,” says Michael Uddstrom. “During that time, IBM hosted the solution in one of its data centers, later handling the move to our new site. That meant two separate sets of complex acceptance and verification tests; we were very impressed with the quality of the IBM people and processes—they did an excellent job.”

FitzRoy is powered and cooled by a new plant facility that incorporates a 1 MW transformer, two 370 kW air-cooled water chillers, and a cooling circuit that can hold 5,500 liters of chilled water. Water cooling accounts for 70 percent of requirements, enabling NIWA to minimize its air-conditioning capacity and consequently to keep its electricity usage down.

The IBM supercomputer at NIWA runs a large number of different simulations, including operational models for weather and wave prediction and research models for weather, climate, oceans and environmental chemistry. It is also available for use by the broader scientific community in New Zealand, who can access it via KAREN [the Kiwi Advanced Research & Education Network].

Speeding through work

FitzRoy is already a major strategic asset for NIWA and New Zealand, and is supporting the New Zealand scientific community in conducting internationally significant research. Relative to the Cray T3E supercomputer that it replaced, FitzRoy in its pre-upgrade state offers 3.3 times the processor count and 7.8 times the processor frequency, 45 times the memory and 500 times the disk capacity, yet requires only 3.1 times the electrical power. Its peak performance of 34 trillion floating-point operations per second (teraFlops) is 50 to 80 times higher. When upgraded, the system will offer 65 teraFlops.

“With higher performance and reliability, the new IBM supercomputer is enabling our users to accomplish more work in less time,” says Michael Uddstrom. “For example, one user reported that a 20-year climate model simulation that used to take one month to complete is now done in 24 hours. A weather model that took 80 minutes to complete on 40 percent of the previous system will now run in eight minutes on about eight percent of the new system—and that's with a model twice as detailed as before. This is a key point: we now have the capacity to significantly increase the complexity (and thereby the quality) of simulations.”

John Morgan, NIWA Chief Executive, says: “We have no doubt that FitzRoy will provide benefits for all New Zealanders, as accurate environmental forecasting is essential for the future growth of New Zealand's most important industries.”