MASSIVE sheds new light on complex research

The city of Melbourne in the Australian state of Victoria has developed a reputation for innovations in the field of High Performance Computing (HPC). Monash University (MU), CSIRO and the Australian Synchrotron (AS) are research institutions at the forefront of this scientific endeavor, and are home to a number of powerful HPC platforms. MU, CSIRO and AS work with the Victorian Partnership for Advanced Computing (VPAC) to run the Multi-modal Australian ScienceS Imaging and Visualization Environment, known as MASSIVE.

The Victorian Government supported the establishment of MASSIVE and the National Computational Infrastructure (NCI) supports MASSIVE as a specialized facility for researchers across Australia.

Growing data demands

Australian scientists have access to a range of high-resolution imaging instruments, which includes the Imaging and Medical Beamline at Australian Synchrotron. The MASSIVE partnership was formed to help scientists extract the most from these instruments produce by providing a powerful, massively parallel HPC system optimized to process imaging data.

“The Australian scientific community is fortunate to have access to a range of amazing instruments, including the Australian Synchrotron,” explains Wojtek James Goscinski, Coordinator of the MASSIVE project. Over the past few years, there’s been a huge increase in the availability of imaging equipment, such as new MRI and CT facilities and new generation instruments such as the Imaging and Medical Beamline at AS.

Researchers use these facilities to perform high-resolution 3D scans of research samples, which can be anything from live organs to rock fragments. The 3-dimensional images that are produced—called a “data volume”—are enormous.

“In the past, getting meaningful results from a complex series of scans could take weeks or even months to achieve,” says Goscinski. “Cutting down the time it takes to process such crucial data can have a real impact on delivering new insights ahead of other research groups.”

The MASSIVE partners wanted to help researchers get the most out of increases in the prevalence and performance of imaging modalities. In order to achieve this, the partners had to develop a HPC platform tailored to the specific demands of processing high-resolution data volumes.

High performance, high visibility

MASSIVE produced a detailed schematic of the HPC solution it required, and put it out to tender. “We needed a massively parallelized cluster, built on a hybrid GPU/CPU infrastructure,” says Goscinski. “As publicly funded institutions, we had a duty to ensure that the IT solution meets strict green IT requirements.

IBM offered a high floating-point computational performance to power ratio for our IT spend—which impressed us during tender.”

The full solution comprises two linked clusters—MASSIVE1 and MASSIVE2—managed as a single system.

Each MASSIVE cluster has 42 IBM System x iDataPlex dx360 servers. Each of these iDataPlex nodes has two six-core Intel Xeon 5600 Series processors running at 2.66 GHz (for a total of 504 cores per cluster) and two NVIDIA M2070 GPUs (for a total of 84 GPUs per cluster). The Intel Xeon processors provide industry-leading performance combined with extreme energy-efficiency. iDataPlex is also a highly efficient solution, offering a unique half-depth form factor that maximizes the effect of cooling, enabling more processors to be packed reliably into a smaller space.

The iDataPlex nodes can run Microsoft Windows Server or Linux depending on the individual requirements of a simulation, with volume reconstruction running in a Windows HPC environment and the core services on Linux. IBM General Parallel File System (GPFS™) provides high-performance parallelized access to data for both operating systems.

“The CT scan reconstruction algorithms that are used by imaging scientists to create 3D volumes are well parallelized on a GPU,” explains Goscinski. “The reconstruction algorithms actually run so quickly that the challenge is getting data into the GPUs fast enough. We configured MASSIVE with an optimal ratio of GPUs to file-system performance, providing the best possible price-performance ratio.”

He adds: “We also use the GPUs in more conventional ways to perform real-time or offline rendering for visualization projects. At the lower end of the workload spectrum, we also offer an interactive desktop environment running on a individual nodes, with a whole range of tools that researchers can use to process their data.”

One of the most important functions of the MASSIVE system is its ability to provide a real-time preview of scan data. “One of the major inefficiencies in high-resolution imaging experiment was that it was difficult to know if you’d captured your data correctly,” says Goscinski. “With the visualization capabilities of our IBM iDataPlex solution, we’ve given researchers the chance to check that they’re collecting all the data they want, allowing them to get maximum value from their allotted scanning slots.”

Throughout the implementation, MASSIVE worked closely with the IBM team. “We found our local IBM representative to be highly motivated and professional,” says Goscinski. “The whole process went smoothly, and we’re very satisfied with the results.”

Clear benefit

MASSIVE’s resources are shared between Australian Synchrotron, CSIRO, Monash University and the VPAC. In addition, a portion of the total resources was also purchased by the National Computational Infrastructure – one of the three peak Australian supercomputing facilities—for leading researchers across Australia. Computing time is allocated to projects based on their scientific merit.

While researchers can use hundreds of thousands of computing hours, some require only a few hours using a desktop interface, making MASSIVE a highly flexible HPC solution.

“We support a lot of neuroimaging research, as well as engineering and materials science. Scientists who use microscopy and electron microscopy also use the system,” continues Goscinski. “We’re also getting increasing demand from researchers across Australia in the fields of molecular dynamics and astrophysics, who use MASSIVE’s parallelized GPUs to run complex simulations.”

With MASSIVE’s support from the Victorian Government and the National Computational Infrastructure, Australian researchers benefit from the exchange of knowledge and ideas on the use and application of massively parallel HPC.

“MASSIVE is a key specialized HPC resource in Australia,” says Goscinski. “Data generated from imaging modalities across the country can now be processed far more effectively, generating more significant insights, faster.”

Future resolutions

Inspired by the success of the new clusters, MASSIVE is now looking to extend their capacity. “We’re planning to scale up in a year’s time by doubling the size of MASSIVE2,” concludes Goscinski. “MASSIVE has established itself an invaluable tool for accelerating data analysis and visualization. By continuing our relationship with IBM, we’re confident the system will meet the processing demands of the future.”