University of Bristol powers up new IBM supercomputer

Founded in 1876, the University of Bristol is a leading UK academic institution, with almost 12,000 undergraduate and 6,500 postgraduate students across six schools.

From the late 1990s onwards, individual University departments had taken advantage of low-cost clustering technology to build their own high-performance computing environments. As the servers became heavier, hotter and hungrier for electricity, this distributed model became less and less viable, and the University looked to create a central supercomputing resource.

The University sees supercomputing as basic infrastructure for the whole institution, rather than just a specialist tool for traditional areas such as mathematics and engineering. Arts and humanities researchers are beginning to use computationally demanding research methods and more than 40 individual groups across four schools were involved in the business case for a new supercomputer.

Dr Ian Stewart, Director of Advanced Research Computing at the University of Bristol, comments: “Our plans for a new large scale cluster required a new data center with appropriate floorspace and modularly expandable power and cooling to allow for future growth in both cpu and storage. IBM won the contract for both elements, giving us a seamless single-vendor solution that combines performance and scalability with cost-efficiency.”


Winning on price-performance

The University of Bristol specified the basic architecture for the new supercomputer, then invited vendors to propose their best solution for the available budget. Says Stewart, “The IBM System x platform offered excellent build-quality and price-performance, but what really won us over was access to IBM expertise and programs such as the IBM Academic Initiative. IBM clearly had the strongest heritage and interest in high-performance computing, and could add significant value beyond simply providing the best hardware.”

IBM has run a number of training workshops for the University, and the two organizations are collaborating on projects, including porting one of the University’s flagship codes—for modeling helicopter rotors—to the IBM Blue Gene platform. The University also benefits from access to IBM Cell processor technology, which it is considering for use in molecular simulations.


High speed, low power

In addition to supplying the main 3360-core cluster, IBM built two clusters for the University’s Earth Systems modeling group, BRIDGE —one with 16 IBM eServer® 326 m systems and the other, a NERC funded system, with 96 IBM System x3455 systems, all running ClusterVisionOS™, based on Linux®. IBM Business Partner ClusterVision installed, configured and tested the clusters, which are already operational.

The main cluster, called BlueCrystal, expands the existing 2007 phase one installation of 96 IBM System x3455 systems with dual-socket, dual-core AMD Opteron processors, Qlogic InfiniBand® networking and a 15 TB IBM General Parallel File System™ (GPFS™) file store with a phase two installation of 424 IBM System x3450 quad core Intel® servers to running ClusterVisionOS, with high-speed Qlogic InfiniPath® networking and a 200 TB DCS9550 Storage System that also uses GPFS.

The University has also deployed four IBM System x3755 systems, each with 32 GB of memory and a ClearSpeed floating-point accelerator card, to support applications that require more than 2 GB per process. The University has increased its investment in ClearSpeed technology by recently adding two ClearSpeed CATS™ (ClearSpeed Accelerated Terascale System) systems.

“The ClearSpeed cards provide around 2 TFlop/s while consuming very little power,” says Stewart. “Our efforts to reduce energy consumption can also be seen in the new data center, which uses targeted water-based cooling in addition to air conditioning.”


State-of-the-art data center

IBM Global Technology Services worked with the University and a project management consultancy to design and build the new data center—housed in a disused water tank on the roof of the University’s Physics Department. Says Stewart, “The unique location presented a number of difficulties, and IBM has been very helpful in overcoming those—including building a Faraday cage to prevent interference with a research radio telescope which is also on top of the building.”

The modular data center has up to 2 MW capacity, and space for 48 APC InfraStruXure water-cooled racks. The main cluster uses 18 racks, and a further 12 racks will be used for a planned petascale data storage environment. The University plans to keep at least 25 percent of the data center empty to enable a future cluster to be brought into service before the existing one is decommissioned.

“We’re very pleased with what IBM has delivered: the System x platform offers excellent performance, and GPFS is proving to be very robust,” says Stewart. “IBM is a great company to work with, and we look forward to taking advantage of future opportunities to collaborate on research.”