Published on 17-Jun-2012
Validated on 02 Dec 2013
"We needed a new supercomputer that was much more powerful than the one we used to have. And from the different possible vendors, IBM turned out to be, by far, the best solution for us." - Arndt Bode, chairman of the board, Leibniz Supercomputing Centre
Leibniz Supercomputing Centre (LRZ)
Technical Computing, Connectivity - High availability, Connectivity - High volume, Data Serving, Energy Efficiency, General Parallel File System (GPFS), High Availability
Leibniz Supercomputing Centre (LRZ) in Munich required a new high-performance computing solution to replace a slower, power-hungry system. The new solution had to deliver superb performance while meeting strict performance and energy requirements.
Leibniz Supercomputing Centre in Munich needed a new high-performance computing solution that could deliver superb performance while conserving as much energy as possible to save costs.
The solution is an IBM Intelligent ClusterTM with IBM System x® iDataPlex® Direct Water Cooled dx360 M4 servers featuring the Intel® Xeon® processor E5 2600 family
The IBM solution saves $1.2 million annually in energy costs and delivers 50 times more computing power than the previous solution, while only using three times more electricity.
To read the German version of the case study click here.
Located on the campus of the Technical University and Ludwig Maximilian University, the Leibniz Supercomputing Centre (LRZ) delivers high-performance computing to researchers across Europe.
“This is really the German hub for science and technology,” says Arndt Bode, chairman of the board of the Leibniz Supercomputing Centre, which is part of the Bavarian Academy of Sciences. “We bring together universities, research centers and research from the university. All of these institutions need access to supercomputers, and that’s why we are out here.”
Delivering more power with less energy
Over time, the existing 10,000-core supercomputer at LRZ was showing signs of age. When Bode and his team decided to upgrade the supercomputing capacity, a set of 25 benchmarks were created for a replacement solution, ranging from performance to energy efficiency.
Herbert Huber, head of the high performance systems computing division at Leibniz Supercomputing Centre, describes their challenge this way: “The old solution used too much electricity for the computing power it delivered. We needed something that had 50 times more compute power and did not need 50 times more electrical power.”
Continues Bode, “We needed a new supercomputer that was much more powerful than the one we used to have. Our users expected something that would deliver several petaflops. But it also had to be affordable. In the German context, affordable means that the system does not consume too much electricity. And from the different possible vendors, IBM turned out to be, by far, the best solution for us.”
Called SuperMUC (“MUC” is the airport code for Munich), the new solution includes an IBM Intelligent Cluster with IBM System x iDataPlex Direct Water Cooled dx360 M4 half-depth, two-socket servers featuring the intelligent Intel Xeon processor E5-2600 family. The 126-rack solution contains about 150,000 cores, 324 terabytes of addressable memory and is designed to deliver up to 3 petaflops of performance. One petaflop equals a thousand trillion floating-point operations per second.
The solution uses IBM General Parallel Files System (GPFS) as the file system for 2 petabytes of NetApp NAS storage and 10 petabytes of DDN disk storage with high-speed InfiniBand interconnects. NetApp WAFL is used as the file structure in tandem with the xCAT open source scalable distributed computing management and provisioning tool. Bode states. “GPFS is important for us, as we have users that have large amounts of data, but also users that have many files—and GPFS can deal with applications that have very large numbers of files.”
Tivoli Storage Manager handles backups and mirroring to a secondary storage facility at a neighboring supercomputing center. “Tivoli is really easy to use, it’s reliable, and we can use it in conjunction with other users and supercomputing centers with which we are cooperating,” says Bode.
Pioneering cooling advances to save costs
While performance was essential, Huber said the total cost of ownership is perhaps the most important benefit of the new solution. IBM scientists from Zurich, Switzerland and Boeblingen, Germany developed an innovative water-cooling solution which Huber claims is 300 times more efficient than traditional air-cooled systems. This unique design means SuperMUC is one-tenth as large as current, state-of-the-art, air-cooled systems.
A complex system of tubes to deliver cooling to the chips that need it most, Huber explains, and excess heat is pumped away to help warm other buildings on the LRZ campus. Powered entirely by renewable energy sources, the cooling system alone saves $1.2 million in energy costs annually, adds Huber.
“It’s a completely different cooling technology which allows us to operate our supercomputer the whole year without any chiller, without any compressor,” says Huber. “When we started procurement, no large company could offer us this direct liquid cooling system. There were prototypes in the labs but no product, so we had to take some risks. But the result is very good and it’s very exciting.”
Balancing raw performance with efficiency is further aided by Tivoli Workload Scheduler LoadLeveler, which provides different applications running on SuperMUC with optimal amounts of processing power. “LoadLeveler is a crucial part of the energy efficiency concept of SuperMUC,” says Huber.
Enabling research in Germany, and beyond
SuperMUC is now considered the fastest supercomputer in Europe and the fourth fastest in the world according to the TOP500 ranking of HPC systems. Researchers are already tapping this power of SUPERMUC to simulate the blood flow behind an artificial heart valve, to devise quieter airplanes and to unearth new insights in geophysics. Peter Bunge, professor of geophysics at Munich University, says SuperMUC has become a key resource for earth scientists. The power of the solution means he can model an entire planet instead of just small parts of a planet, which is essential to helping unravel the mysteries of what causes earthquakes, and whether they can be predicted more accurately.
“Earthquakes are difficult to reproduce because they happen on time scales that are very difficult to reproduce,” Bunge explains. “What we can ultimately do on the computer, that we could never do in the lab, is to find out if our theories for describing these very complicated events hold for the time scales and the length scales that are so large, so long and so difficult to reproduce in the lab. What SuperMUC will allow us is an order of magnitude of better understanding.”
Bode says the ability of SuperMUC to deliver 50 times more computing power while consuming only three times more electricity is a wonderful bit of math for the LRZ. “We are very well supported by IBM in this process. IBM Global Technology Services will provide hardware and software maintenance as well as system and user support. We have a relationship that is dominated by good cooperation, as well as cooperation on research and development,” says Bode. “This solution is really a considerable advantage to the old system and our scientists are happy.”
For more information
For more information about LRZ, visit: www.lrz.de/english
Products and services used
IBM products and services that were used in this case study.
Intelligent Cluster, System x: iDataPlex dx360 M4
© Copyright IBM Corporation 2012 IBM Corporation Systems and Technology Group Route 100 Somers, New York 10589 Produced in the United States of America June 2012 IBM, the IBM logo, ibm.com, GPFS, iDataPlex, Intelligent Cluster, LoadLeveler, System x, and Tivoli are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright and trademark information” at ibm.com/legal/copytrade.shtml Intel, the Intel logo, Xeon and Xeon Inside are trademarks of Intel Corporation in the U.S. and other countries. This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates. The client examples cited are presented for illustrative purposes only. Actual performance results may vary depending on specific configurations and operating conditions. THE INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements under which they are provided. Actual available storage capacity may be reported for both uncompressed and compressed data and will vary and may be less than stated