Nationwide Uses Linux and High-Power Virtualization for Web Presence

Nationwide is a leading provider of insurance and financial services and ranks No. 104 on the Fortune 500 list. It has more than 36,000 associates, with 6,000 working in IT. The industry is highly regulated and information-centric. Nationwide faced multiple challenges as it grew its application load.

• Growing server sprawl that included more than 5,000 distributed servers, with more than 75% of them with peak utilization less than 50%.
• Long server provisioning times that stretched from a few weeks to months.
• Continuing high TCO if it pursued a business-as-usual case, with software license fees, cabling and network administration among the major concerns.
• For its Tier 4 data center, growing data center power and cooling requirements that would require a new facility, with a projected cost of more than $10 million, if nothing was done.
• Difficulty in dynamically matching processor power available to the variable needs of its application workloads and performance test systems, administration among the major concerns.
• As with any early adopter of a new technology or approach, Nationwide faced the challenge that the early implementation would be a learning process for itself and the vendor.

In addition to the above points, Nationwide’s IT infrastructure was required to:

• Support all new Java/Linux server growth and refresh because all new custom workloads were to be developed using Java, WebSphere and Linux.
• Provide a high-availability environment augmented with a solid disaster recovery plan.
• Adhere to a three-tiered security architecture.

Nationwide’s goal was to use virtualization to address the above challenges. The company was determined to draw on its centralized and distributed experience to create the best server optimization possible, while addressing as many challenges as possible. Without solving the server sprawl problems, it felt it would be slowly losing control of its budget and agility to handle dynamically changing workloads. The various applications had been running in Linux on x86, Solaris, HP-UX or OS/400 environments.

Approach
As already stated, Nationwide decided to use the Java Platform, Enterprise Edition (Java EE, formerly known as J2EE) environment for new applications and to make virtualization the standard operating environment. After looking at its options, Nationwide made the following environment standard for deployment of new applications:
1. Java EE with Linux using IBM zSeries/Virtual Machine (z/VM)
2. If not possible, then Java EE on x86 using VMware ESX server
3. With extreme justification, a physical (nonvirtualized) environment

Nationwide had used z/VM in its mainframe environment and was also a big user of VMware on its x86 distributed servers. Some of the major reasons for choosing the System z virtualization with z/VM over x86 virtualization with VMware as the first choice virtual environment were that it has:

• Mature and proven virtualization with low overhead for most workloads
• Smaller footprint/power/cooling
• Simpler disaster recovery solution
• High availability of hardware
• Attractive middleware pricing
• Sophisticated capacity on demand
• Certified application isolation

Nationwide chose a solution based on the IBM System z9 running multiple Novell SUSE Linux Enterprise Server 9 (SLES 9) systems implemented under IBM’s mainframe virtualization software, z/VM 5.2. The middleware selected was IBM WebSphere. To meet its requirements, Nationwide populated two data centers: the production data center and the test/development/disaster recovery data center.

In the production data center, it initially installed (on 1 September 2005) a Linux-only zSeries z990 with three Integrated Facility for Linux (IFL) processors with 40GB of central storage. This has grown to a System z9 with eight IFLs and 144GB of central storage. It moved from z/VM 5.1 to now run z/VM 5.2 across four
logical partitions (LPARs) to support its virtual Linux systems.

In the test/development/disaster recovery data center, it initially installed a Linux-only zSeries z990 with three IFL processors with 40GB of central storage. This has grown to a System z9 with 13 IFLs and 160GB of central storage. It now runs z/VM 5.2 across five LPARs to support its virtual Linux systems.
3
Nationwide deployed 14 mission-critical applications on its production system, including its Web site (nationwide.com), agency portal and absence management ASP. More than 100,000 active users access these major applications every day. The total number of Linux systems initially deployed in December 2005 (150 virtual Linux servers) has grown steadily to more than 450 virtual Linux servers across the two data centers as of June 2007. (Note: This number is only a subset of its previously mentioned more than
5,000 servers.)

Nationwide has a significant mainframe installation, although this particular consolidation solution was set up as a separate and distinct Linux-only mainframe complex because there was no interaction between the z/OS systems and the new Linux systems. The company had some in-house z/VM skills that it leveraged in
creating this second mainframe silo. It also had in-house Unix skills that were easily transferable to Linux.

Results
When comparing the results to the challenges it faced, Nationwide feels the project is a resounding success. The fast deployment (four months) of the solution was far quicker than the time it would normally have taken to upgrade 250 individual servers.
It has addressed the growing server sprawl of low-utilized servers by deploying more than 450 Linux systems on two System z9 systems running with an average utilization of 70%. It has already been able to increase the number of Linux virtual systems substantially within the same physical footprint and has the potential to increase capacity in the same footprint by nearly 3.5 times. This will enable Nationwide to easily support new Java/Linux workload growth.

• Long server provisioning times have been reduced from months to days and, in many cases, to hours.
• TCO was reduced in several areas, with an estimated savings of $15 million over three years:
• Fifty percent reduction in Web hosting monthly costs.
• Fifty percent reduction in hardware and operating system support cost. Nationwide has been working at a ratio of 100 servers per system administrator. By using Linux on System z, it has avoided adding system administrators to address growth, a cost it would have incurred with its previous strategy.
• Significant savings on middleware costs (WebSphere, DB2, Oracle). Software licensing on a per CPU basis is quite favorable with IBM’s IFL implementation.
• Reduction in floor space and power consumption.
• Nationwide is seeing an 80% reduction in floor space requirements and power consumption. Therefore, it has avoided expanding a Tier 4 data center and the substantial costs associated with that requirement, which was projected to cost more than $10 million if nothing was done.
• The System z solution provides dynamic capabilities to increase or reduce computer capacity available as demanded through capacity-on-demand features and through the ability of the LPAR support and z/VM to shift capacity from lower-priority work to higher-priority work. The large capacity of the System z to run many virtual systems provides a large selection of workload to be balanced. One example was Nationwide’s ability
• to temporarily scale up capacity to support an intensive advertisement/marketing campaign in conjunction with the 2006 Super Bowl (in the U.S.). Nationwide used the capacityon- demand capabilities of the System z9 to add the equivalent of an additional processor’s worth of capacity for two weeks. It was able to test a processor capacity load 22 times the anticipated load from the Super Bowl.
• Using the System z hardware’s high-availability functions and z/VM availability capabilities, the solution provides a simple, but robust, high-availability environment.
• It has been easier to do business continuity/disaster recovery planning with two mainframes than trying to manage a plan across more than 400 distributed servers. Nationwide has kept its z/OS mainframe separate from the Linux-only System z environment to facilitate disaster recovery plan management.
• The solution enabled it to adhere to its three-tier security architecture.
• The new data center environment has enabled Nationwide to leverage the more than 40 years of resource-sharing experience of the centralized mainframe with its extensive experience in the distributed world.

Critical Success Factors
Some of the critical success factors for this project were:

• Setting clear goals for the project.
• Gaining executive management’s full support by explaining the advantages of the project.
• To get broad commitment, Nationwide put together a dedicated cross-disciplinary team that worked on this project for four months. Representatives were from the Unix, Linux and z/VM operating environments, networking, firewall and application development.
• Establishing an effective vendor partnership. Nationwide worked closely with Novell and IBM to make the transition as seamless as possible.
• Overcommunicating at the front end of the project to eliminate the chance for teams to misunderstand requirements and to keep everyone aware of the project’s status.
• Offering economic incentives to the application teams to achieve significant adoption. This was important because the application development teams had grown up with their own machines and had to be convinced to adopt different technology.
• Providing good enforcement and technology governance. Within the organization, there was some resistance to moving to a virtualized environment because some people were concerned about not having their own physical servers. Education and commitment were needed to address these concerns. An architectural review board was put in place, with the full support of executive management.

Nationwide felt it was allowed to be more aggressive in its implementation from the start because the solution was built on the mature, high-performance virtualization capabilities of the System z and z/VM.
4
Lessons Learned
Skills needed were easily transferred from the current distributed environment:

• Unix skills were easy to turn into Linux skills.
• No changes were needed in WAS and DB2 expertise.
• You need to look at changing the billing model to one that has fixed and consumption-based components because cost is no longer tied to the costs of an isolated server environment. Nationwide started this project with a simple fixed model, similar to what had been in place for its distributed environment. However, it realized that a model where an amount was charged per month for basic minimal services plus a variable cost, depending on the consumption of MIPS and memory, proved to be more equitable. The model was also designed to motivate the proper scheduling for running batch workloads, with financial incentives for workloads that could be run outside the peak window.
• You need to use a complete TCO (and not just acquisition cost) view to show the full benefits. Using only a hardware and/or Linux operating system view does not show the total savings. For example, software savings were significant because of licensing algorithms, administration was simplified with less
• staffing needs, provisioning new applications was faster and easier, and the facilities impact/cost avoidance was substantial.
• The System z mainframe Linux support offers a viable solution when viewed from a total solution perspective.
• Do not underestimate the mental shift needed by the application teams as they move to a virtual environment on a mainframe. For example, teams are now on a shared environment, and paging is not a bad thing.
• You need to include operational people on the project team