Published on 06-May-2011
Validated on 01 Nov 2012
"Our new server and storage cluster gives us the ability to experiment with vast, sophisticated models of the universe and answer fundamental questions about our cosmic environment." - Professor Carlos Frenk, Director of the Institute for Computational Cosmology, Durham University
Deep Computing, Deep Computing: (GPFS) General Parallel File System, Dynamic Infrastructure, Energy Efficiency, General Parallel File System (GPFS), Optimizing IT, Optimizing IT, Service Management
IBM Business Partner:
Durham University is the third oldest university in England after Oxford and Cambridge, and has been a leading European centre of learning for over 1,000 years. With more than 14,000 students and 3,800 staff, the University’s 16 colleges provide degree courses in a wide range of subjects across three faculties: Arts and Humanities, Social Science and Health, and Science. Established in 2002, the University’s Institute for Computational Cosmology has become a leading international centre for research into the origin and evolution of the universe.
To maintain its position at the forefront of international research, the Institute for Computational Cosmology at Durham University wanted to develop a new high-performance computing cluster that would enable even more sophisticated simulations of the universe. With constraints on data centre space, power and cooling, the Institute looked to base the new cluster on compact, energy-efficient technologies.
Working with OCF and IBM, the Institute designed and deployed COSMA4, a 25 teraflop cluster based on water-cooled IBM System x iDataPlex servers with 2,640 Intel Xeon cores. IBM System Storage DS3500 hardware provides 610 TB of storage capacity, which is managed by the IBM General Parallel File System and IBM Tivoli Storage Manager.
Enables simulations of universe models that provide unprecedented levels of detail. Performs seven times faster than COSMA3, and 50 times faster than COSMA2, which has now been decommissioned. Uses the same amount of electricity as COSMA2, and does not require air cooling – reducing overall data centre energy consumption by 60 kW. Achieves a Linpack benchmark of 91 percent efficiency, delivering more than 400 megaflops per Watt. It is currently the UK’s greenest supercomputer and would make 19th place on the November 2010 worldwide Green500 list.
Durham University is the third oldest university in England after Oxford and Cambridge, and has been a leading European centre of learning for over 1,000 years. With more than 14,000 students and 3,800 staff, the University’s 16 colleges provide degree courses in a wide range of subjects across three faculties: Arts and Humanities, Social Science and Health, and Science.
Established in 2002, the University’s Institute for Computational Cosmology has become a leading international centre for research into the origin and evolution of the universe, using high-performance computing (HPC) clusters to simulate cosmological events and answer some of the most fundamental questions in science: What were the first objects in the Universe? How do galaxies form? What is the nature of dark matter and dark energy? Where does the large-scale structure of the universe come from? What is the fate of the Universe?
“Along with engaging the general public and helping people to understand their place in the cosmos, our research helps raise the profile of science in general,” says Professor Carlos Frenk, Director of the Institute for Computational Cosmology. “It serves as an important factor in motivating young people to become scientists. There are also technological spin-offs of our work, in the form of better numerical algorithms and novel solutions to problems which have applications in many areas other than cosmology.”
Since its foundation, the Institute has run its simulations on a series of HPC clusters known as COSMA. COSMA2 and COSMA3 were both based on Sun servers and storage and the Solaris operating system, but as the field of computational cosmology developed, the need for a larger-scale solution became evident.
“The level of detail in the models we use is continually increasing, so there’s a perpetual need for more processor power and data storage,” says Dr Lydia Heck, Senior Computer Manager. “In addition to running our own projects, we are also part of an international cosmology consortium called Virgo, which unites HPC resources from around the world to run simulations based on hundreds of terabytes of data. To meet these needs, we decided to build COSMA4 – a new cluster that would provide at least 500 TB of storage and 25 teraflops of processing power.”
Choosing the right partners
The Institute received proposals from eight leading vendors, including Dell, HP and Viglen, as well as a joint bid from IBM and OCF, a specialist provider of HPC server and storage clusters.
“What we liked about the IBM offer was that it could deliver all the infrastructure – servers, storage and software – from a single vendor,” comments Dr Heck. “We had allowed vendors to bid for the different elements separately, but the advantages of a coherent single-vendor strategy were clear. We had also worked with OCF before, and we had been impressed with the level of service they offered.”
Rapid, successful delivery
The OCF team worked with the Institute’s in-house HPC specialists to deliver, install and test the COSMA4 cluster. OCF used the PRINCE2 methodology to manage the project, which helped to accelerate the testing phase and prepare COSMA4 for go-live in record time. OCF also provided comprehensive training on the new IBM technologies, and will continue to support the infrastructure in accordance with a detailed service level agreement.
“The hardware and software integration from OCF was very smooth,” says Dr Heck. “The OCF team is very knowledgeable, and provided four days of invaluable intensive training on the system.”
Exploring the infrastructure
From the server perspective, COSMA4 is built on 220 IBM System x iDataPlex dx360 M3 servers, each of which contains two Intel® Xeon® X5650 processors running at 2.67 GHz. Each processor has six cores, which add up to a total of 2,640 cores across the whole cluster. The servers are supported by a storage infrastructure based around eight IBM System Storage DS3500 disk systems, which provide 620 TB of storage capacity. The IBM General Parallel File System (GPFS) is used to provide rapid parallel access for multiple users, while IBM Tivoli Storage Manager handles backups, and may also be used for policy-based archiving for long-term storage in the near future.
“The IBM infrastructure has two vital advantages,” comments Dr Heck. “First, the iDataPlex architecture provides a very dense compute capacity in a small physical footprint, so it gives us a huge increase in performance compared to our previous clusters without requiring a move to a larger data centre. Second, it is water-cooled, which significantly reduces the need for air conditioning, reducing energy costs and improving our green computing profile.”
Space and energy
The iDataPlex and DS3500 racks are fitted with IBM Rear Door Heat eXchanger technology, which passes the hot air generated by the disks and processors over a series of sealed coils filled with chilled water. As a result, the air expelled from the back of the rack is 3 °C colder than the data centre’s ambient temperature, enabling the Institute to retire three of its seven air conditioning units and use the remaining units less intensively.
“Our data centre has an upper limit on energy consumption so it was essential we procured a powerful but well balanced and energy-efficient machine,” says Dr Heck. “Although COSMA4 is 50 times faster than the COSMA2 cluster which it replaced, it only draws about the same amount of power. It also doesn’t require any air conditioning, so we save about 60 kW in total data centre energy consumption.”
According to Linpack benchmarks, COSMA4 is currently running at 91 per cent efficiency, and has an energy efficiency of over 400 megaflops per Watt of energy consumed. This level of energy efficiency makes it currently the greenest supercomputer in the UK, and would put it in 19th place on the November 2010 worldwide Green500 list.
Most important of all, COSMA4 is seven times faster than its predecessor, COSMA3, which means that researchers at the Institute and their international collaborators can run more complex and sophisticated simulations of the cosmos.
Professor Frenk comments: “It is very difficult to test theories about the universe using traditional methods, as the scale is so large and the algorithms are so complex. Our new server and storage cluster gives us the ability to experiment with vast, sophisticated models of the universe and answer fundamental questions about our cosmic environment – how does gravity operate, for example, and how does the Universe expand?”
Making an international contribution
COSMA4 already has 100 registered users and 240 TB of data, and is preparing for a number of major international projects with other supercomputing centres both within the UK and internationally.
Dr Heck concludes: “The launch of COSMA4 helps Durham University maintain its status as one of the world’s leading cosmology research centres, and enables us to play a major role in national and international HPC initiatives such as the Virgo consortium. The projects we run on COSMA4, whether they are internal or in partnership with other academic institutions, are helping us to reveal some of the universe’s oldest mysteries.”
Products and services used
© Copyright IBM Corporation 2011 IBM United Kingdom Limited, PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU. Produced in the United Kingdom. May 2011. All Rights Reserved. IBM, the IBM logo, ibm.com, General Parallel File System, iDataPlex, System Storage, System x and Tivoli are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. A current list of other IBM trademarks is available on the Web at “Copyright and trademark information” at: ibm.com/legal/copytrade.shtml. Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Other company, product or service names may be trademarks, or service marks of others. IBM and OCF are separate companies and each is responsible for its own products. Neither IBM nor OCF makes any warranties, express or implied, concerning the other’s products. References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program or service is not intended to imply that only IBM’s product, program or service may be used. Any functionally equivalent product, program or service may be used instead. All customer examples cited represent how some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. IBM hardware products are manufactured from new parts, or new and used parts. In some cases, the hardware product may not be new and may have been previously installed. Regardless, IBM warranty terms apply. This publication is for general guidance only. Photographs may show design models.