DEISA builds a continental supercomputing network

The Distributed European Infrastructure for Supercomputing Applications (DEISA) is a consortium of leading national supercomputing centers that aims to foster pan-European scientific research. The 15 partners are located in 10 countries and separated by great distances—from Scotland, Sweden and Finland in the north, to Italy and Spain in the south. DEISA’s task, funded by the European Commission’s Seventh Framework Program, is to design, develop and enhance an infrastructure that will help these partners work together more effectively.

Supporting a heterogeneous environment
“Several of the members of DEISA run supercomputing clusters that are based on IBM Power Systems™, PowerPC® or Blue Gene®/P hardware, while the others use a variety of architectures including Cray XT5/6, SGI-Altix and NEC SX9,” comments Dr. Stefan Heinzel, Director of the Rechenzentrum Garching at the Max Planck Society, which is one of the principal partners of DEISA. “Creating an infrastructure that would allow all these different supercomputing architectures to share data effectively across distances of several thousand kilometers was a significant challenge—and something that very few organizations had ever attempted before.”

The sites were connected via the pan-European GÉANT communications infrastructure—but the creation of the physical network was not the only challenge DEISA faced. To develop an effective distributed supercomputing infrastructure, one of the main requirements was to build a shared file system that would allow each center to access data from the other locations.

Making the most efficient use of resources
“Different research projects have different computational needs, and are suited to the characteristics of different supercomputing clusters,” explains Dr. Hermann Lederer, Deputy Director of the Rechenzentrum Garching. “So, for example, it might be more efficient for a team of German scientists to run some parts of a specific project on one of the French supercomputing clusters, or vice versa. We wanted to make it possible to route the jobs from one center to another for processing, and then bring the results home again, without involving the end-users in the complexity of managing the infrastructure.”

Mr. Heinzel adds: “To achieve this, we needed a way to share large files across the wide-area network, and the best option was IBM General Parallel File System, or GPFS. Many of the DEISA partners had already been using GPFS within their own clusters, and we also had confidence in IBM’s expertise in deep computing solutions. In fact, it was DEISA’s close relationship with IBM that stimulated the development of many of the GPFS features that we are using today.”

Boosting reliability
The first major challenge was to ensure that the infrastructure was reliable enough to support international projects without having any negative impact on local research.

“We had to ensure that the individual supercomputing clusters were autonomous, so that a failure on one cluster wouldn’t affect the others,” explains Dr. Lederer. “IBM helped us by considerably enhancing the multiclustering capabilities of GPFS and the client-side reliability. It was essential to reengineer the communication patterns so that compute nodes in one cluster wouldn’t need to communicate with compute nodes in another cluster, which helped to improve reliability and performance.”

Solving network communication challenges
A second issue was the solution’s reliance on the network infrastructure. If there was a minor problem with the wide-area network, it could cause I/O errors that disrupted long computing jobs. To resolve this, DEISA decided to store all job-related data on a local file system for the duration of the job, and then copy it globally via GPFS at the end of the job.

“In the future, we are interested in testing a new IBM technology called Panache, which serves as a clustered file system cache for parallel data-intensive applications that require wide-area file access,” says Mr. Heinzel. “If it works as intended, it should help to further combat WAN latencies and outages by storing all updates of data and metadata in the cache, and then asynchronously replicating them to the remote GPFS cluster.”

Advances in science and technology
By working closely with the IBM Deep Computing team, DEISA has succeeded in creating a distributed supercomputing infrastructure that supports scientists and researchers at companies and institutions throughout Europe. Each year, the DEISA Extreme Computing Initiative (DECI) calls for proposals for “grand challenge” projects that use the DEISA infrastructure to enable groundbreaking research in all areas of science and technology. The most promising projects are selected by the DEISA Executive Committee and granted access to DEISA computing resources.

DECI projects have already enabled great advances in a wide range of scientific fields. One notable example is nuclear fusion research.

“The fusion community has benefited from over 10 million hours of computer resources under DEISA in the last couple of years, via the different organizations and projects,” says Dr. Duarte Borba, a researcher at the European Fusion Development Agreement (EFDA). “This has allowed significant progress to be made in the use of large-scale computer resources for fusion applications. DEISA facilitates access to a diverse set of computer architectures, which has created new opportunities for the fusion community.”

DEISA has also started working on the development of an intercontinental distributed supercomputing architecture. By working with TeraGrid and leveraging the robustness of GPFS, DEISA has been able to demonstrate the viability of collaboration between sites in Europe and the USA, linking supercomputing resources across enormous physical distances—from Bologna, Italy to San Diego, California.

Building a strong relationship
“We have come a long way since the first launch of DEISA,” comments Dr. Lederer. “Thanks to dedicated support from IBM and collaboration with our colleagues across Europe, we have created a reliable, high-performance infrastructure that has contributed to many important scientific projects.”

Mr. Heinzel concludes: “Our work with IBM GPFS demonstrates the viability and usefulness of a global file system for collaboration and data sharing between supercomputing centers—even when the individual supercomputing clusters are based on very different technical architectures. The flexibility of GPFS and its ability to support all the different DEISA supercomputers is highly impressive.”