Leibniz Supercomputing Centre (LRZ) taps IBM

To read the German version of the case study click here.

Located on the campus of the Technical University and Ludwig Maximilian University, the Leibniz Supercomputing Centre (LRZ) delivers high-performance computing to researchers across Europe.

“This is really the German hub for science and technology,” says Arndt Bode, chairman of the board of the Leibniz Supercomputing Centre, which is part of the Bavarian Academy of Sciences. “We bring together universities, research centers and research from the university. All of these institutions need access to supercomputers, and that’s why we are out here.”

Delivering more power with less energy

Over time, the existing 10,000-core supercomputer at LRZ was showing signs of age. When Bode and his team decided to upgrade the supercomputing capacity, a set of 25 benchmarks were created for a replacement solution, ranging from performance to energy efficiency.

Herbert Huber, head of the high performance systems computing division at Leibniz Supercomputing Centre, describes their challenge this way: “The old solution used too much electricity for the computing power it delivered. We needed something that had 50 times more compute power and did not need 50 times more electrical power.”

Continues Bode, “We needed a new supercomputer that was much more powerful than the one we used to have. Our users expected something that would deliver several petaflops. But it also had to be affordable. In the German context, affordable means that the system does not consume too much electricity. And from the different possible vendors, IBM turned out to be, by far, the best solution for us.”

Called SuperMUC (“MUC” is the airport code for Munich), the new solution includes an IBM Intelligent Cluster with IBM System x iDataPlex Direct Water Cooled dx360 M4 half-depth, two-socket servers featuring the intelligent Intel Xeon processor E5-2600 family. The 126-rack solution contains about 150,000 cores, 324 terabytes of addressable memory and is designed to deliver up to 3 petaflops of performance. One petaflop equals a thousand trillion floating-point operations per second.

The solution uses IBM General Parallel Files System (GPFS) as the file system for 2 petabytes of NetApp NAS storage and 10 petabytes of DDN disk storage with high-speed InfiniBand interconnects. NetApp WAFL is used as the file structure in tandem with the xCAT open source scalable distributed computing management and provisioning tool. Bode states. “GPFS is important for us, as we have users that have large amounts of data, but also users that have many files—and GPFS can deal with applications that have very large numbers of files.”

Tivoli Storage Manager handles backups and mirroring to a secondary storage facility at a neighboring supercomputing center. “Tivoli is really easy to use, it’s reliable, and we can use it in conjunction with other users and supercomputing centers with which we are cooperating,” says Bode.

Pioneering cooling advances to save costs

While performance was essential, Huber said the total cost of ownership is perhaps the most important benefit of the new solution. IBM scientists from Zurich, Switzerland and Boeblingen, Germany developed an innovative water-cooling solution which Huber claims is 300 times more efficient than traditional air-cooled systems. This unique design means SuperMUC is one-tenth as large as current, state-of-the-art, air-cooled systems.

A complex system of tubes to deliver cooling to the chips that need it most, Huber explains, and excess heat is pumped away to help warm other buildings on the LRZ campus. Powered entirely by renewable energy sources, the cooling system alone saves $1.2 million in energy costs annually, adds Huber.

“It’s a completely different cooling technology which allows us to operate our supercomputer the whole year without any chiller, without any compressor,” says Huber. “When we started procurement, no large company could offer us this direct liquid cooling system. There were prototypes in the labs but no product, so we had to take some risks. But the result is very good and it’s very exciting.”

Balancing raw performance with efficiency is further aided by Tivoli Workload Scheduler LoadLeveler, which provides different applications running on SuperMUC with optimal amounts of processing power. “LoadLeveler is a crucial part of the energy efficiency concept of SuperMUC,” says Huber.

Enabling research in Germany, and beyond

SuperMUC is now considered the fastest supercomputer in Europe and the fourth fastest in the world according to the TOP500 ranking of HPC systems. Researchers are already tapping this power of SUPERMUC to simulate the blood flow behind an artificial heart valve, to devise quieter airplanes and to unearth new insights in geophysics. Peter Bunge, professor of geophysics at Munich University, says SuperMUC has become a key resource for earth scientists. The power of the solution means he can model an entire planet instead of just small parts of a planet, which is essential to helping unravel the mysteries of what causes earthquakes, and whether they can be predicted more accurately.

“Earthquakes are difficult to reproduce because they happen on time scales that are very difficult to reproduce,” Bunge explains. “What we can ultimately do on the computer, that we could never do in the lab, is to find out if our theories for describing these very complicated events hold for the time scales and the length scales that are so large, so long and so difficult to reproduce in the lab. What SuperMUC will allow us is an order of magnitude of better understanding.”

Bode says the ability of SuperMUC to deliver 50 times more computing power while consuming only three times more electricity is a wonderful bit of math for the LRZ. “We are very well supported by IBM in this process. IBM Global Technology Services will provide hardware and software maintenance as well as system and user support. We have a relationship that is dominated by good cooperation, as well as cooperation on research and development,” says Bode. “This solution is really a considerable advantage to the old system and our scientists are happy.”

For more information

Please contact your IBM sales representative or IBM Business Partner, or visit us at: ibm.com/systems/x

For more information about LRZ, visit: www.lrz.de/english