Bank of Montreal enables rapid recovery with high availability technology from IBM

One of the five largest banks in Canada, Bank of Montreal (BMO) is a highly diversified financial services organization with assets that totaled over US$366 billion in 2007. BMO utilizes a wide array of technologies to support the processing of financial data, including a significant investment in IBM systems and software. Bank of Montreal serves the U.S. market through Chicago-based Harris N.A., which has the third largest number of branches in the Chicago area and over $42 billion in assets.

For financial institutions like BMO, the ability to recover from major system failures is critical. When the events of September 11, 2001, highlighted significant deficiencies in the disaster preparedness of many companies, the U.S. government issued a series of new regulations and industry-specific guidelines designed to address this problem. The “Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System” and other guidelines provide numerous recommendations for improving the recovery capabilities of financial institutions, including locating the disaster recovery center beyond the immediate region of a primary facility, and delivering zero percent data loss up to the point of disaster.

“That white paper definitely had a strong influence on our plans,” explains Malcolm Sanderson, a senior technical architect at BMO. BMO responded to the challenge by implementing an ambitious program of improvements to its disaster recovery strategy, beginning with the development of a new data center location.

BMO’s Operational Resiliency Program
The original backup and disaster recovery (DR) site was located less than 10 km from the primary data center, an inadequate distance to be considered “out-of-region.” To solve this problem, BMO set up a new site—one that was 100 km away. This effort, part of BMO’s Operational Resiliency Program (ORP), had several important goals: to increase the distance between the primary and backup sites, to achieve a recovery point objective of zero, to establish two control points for operations, and to enable system and workload restoration within a two-hour timeframe.

IBM helped supply BMO with the technologies necessary to achieve these resiliency goals and supported the project throughout the implementation. Says Sanderson, “We have a very strong IBM team locally that's been involved in the project. We’ve had them engaged all the way through.”

Early adopters of STP
Critical to the success of the new data center was the ability to synchronize transactions between the Bank’s primary and backup servers—10 IBM System z™ mainframes in all. Peer-to-peer remote copy (PPRC) was used to achieve data mirroring between sites, effectively synchronizing transactional data between the servers. Geographically Dispersed Parallel Sysplex (GDPS), a multi-site end-to-end application availability solution, was used to automate the recovery process and manage the PPRC environment.

But the primary challenge of the BMO implementation was the increased distance between the two sites. GDPS has typically relied on Sysplex Timer® technology, which could only support distances of up to 40 km without intermediate sites. The Bank’s two sites are linked through nearly 100 km of fiber optic cable, a length that far exceeds that limit. To meet this challenge, BMO became an early adopter of a newly developed IBM timing technology known as Server Time Protocol (STP).

STP, which keeps multiple processor clocks at different locations in sync with each other, was designed to enhance the capabilities of GDPS/PPRC. STP supports a multi-site Coordinated Timing Network (CTN) that allows clock synchronization between System z mainframe servers up to 100 km apart. BMO was the first IBM customer to implement GDPS/PPRC via STP at this distance.

In order to implement the solution, BMO had to expand capacity of their storage subsystem, which consists of an IBM System Storage™ DS8300 at the primary data center and an IBM Enterprise Storage Server® 2105 at the backup site. They also had to upgrade their IBM z/OS® version on an accelerated timeline.

As an early adopter through the IBM Implementation Assistance Program (IAP), BMO gained advance access to cutting-edge STP technology, and they had the opportunity to give specific feedback about their needs early in the development process, ensuring a more timely and personalized response.

Measurable successes
The IBM solution has fulfilled BMO’s requirement for longer-distance recovery, and it has enabled a recovery point objective of zero. “Prior to this implementation, if we had a disaster we would lose an entire day’s transactions for some applications,” says Sanderson. “We wanted to get to the point where we could recover all transactions up to the point of disaster.”

Reaching that goal was a team effort, and one that required aggressive testing before, during and after the implementation. “We did an enormous amount of testing on all of this technology prior to implementation,” says Sanderson, “And IBM was at the table throughout this process.” Even before testing began at the new site, a variety of tests were conducted in a lab environment, including tests on a 100-km spool of fiber cabling so GDPS and STP could be tested thoroughly at the full distance.

Now that the solution is in production, BMO continues to conduct recovery tests several times a year, a process that also measures how long the recovery process takes. “Our goal is to recover services within two hours from the time we initiate the recovery process,” says Sanderson. A recent BMO test showed a recovery time of just one hour and 54 minutes, surpassing the goal by six minutes.

“What GDPS brings to the table is the automation necessary to allow us to recover within a two-hour timeframe,” Sanderson explains. “It automates a lot of manual processes, and with automation you get speed. It also detects conditions that might cause the recovery to fail. With our new GDPS system, once we make the decision to do a site takeover, we simply tell the system, and the system completes the recovery automatically.”

Reducing operational risk
BMO and Harris N.A. have benefited from this implementation in other ways as well. By bringing their disaster recovery strategy into compliance with federal guidance, they have significantly reduced their risk of data loss when recovering from a system failure. And although this improvement in operational resiliency is largely invisible to the Bank’s customers, it helps to ensure that institutional trust will be maintained in the event of a catastrophe. As Sanderson says, “The real benefit to the business is a reduction in risk. We have a better insurance policy now than we had before.”

“Data protection and reputational risk are our critical success factors,” says Lee Dunn, vice president of BMO’s Operational Resiliency Program. “Businesses have evolved from paper-based transactions to electronic transactions through a number of channels and are heavily reliant on technology. The implementation of PPRC, GDPS and STP technologies has enabled BMO to provide the right level of resiliency to our internal lines of business.”

“Part of the team”
Sanderson says that the IBM GDPS High Availability Team provided excellent service and support to the BMO staff throughout the project. “IBM has been an integral part of the team,” reports Sanderson. “They’ve been living side-by-side with our system programmers—attending meetings, helping with planning, coming in at midnight when we’re testing.”

“They’ve also done a lot of the integration prep work,” he continues. “For example, IBM has a partnership with Nortel, so they were able to test the GDPS solution with Nortel to make sure everything was going to work properly.”

Extending the solution
With the BMO implementation, GDPS has been proven to operate at greater distances than previously possible, helping BMO achieve its disaster recovery objectives for Harris N.A. The Bank’s Operational Resiliency Program has since moved into its next phase, which includes applying these same technologies to applications for Bank of Montreal.

For more information
Contact your IBM sales representative or IBM Business Partner. Visit us at: ibm.com/systems/z

For more information about Bank of Montreal and Harris N.A. of Chicago, visit: www.bmo.com