gkd-el boosts SAP system throughput by 270 per cent and cuts costs by 30 per cent by migrating SAP solutions to IBM System z10 Enterprise Class

Click here to read a German version of this technical paper.

Background, starting point and objectives
Gelsenkirchen is a city in northwest Germany with almost 300,000 inhabitants. Once the most important coal-mining city in Europe, Gelsenkirchen has invested heavily in new infrastructure to support its move towards a service-based economy and towards a focus on alternative energies, including the production of solar cells.

With 84 employees, gkd-el (www.gkd-el.de) manages the entire IT infrastructure for the Gelsenkirchen city government. Additionally, it provides some services to the city of Herne and several other customers, including Gesellschaft für Energie und Wirtschaft (GEW), WiN Emscher-Lippe GmbH, and Kreishandwerkerschaft Emscher-Lippe West.

Since 7th October 2004 gkd-el has been a certified SAP Customer Competence Center, offering hosting services for SAP systems, consulting services (for the modules HCM, FI/CO, PSCD, MM, RE-FX, PM), implementation services for SAP solutions (including project management), first-level support for SAP production environments, and training classes for SAP users and administrators.

The SAP on IBM System z landscape at gkd-el is accessed by about 2,000 users and contains the complete data for all administrative procedures used by the city of Gelsenkirchen, in addition to the SAP applications used by gkd-el’s other customers. Out of the 84 gkd-el employees, 8 are responsible for managing the complete SAP on System z landscape, including the administration of SAP, Linux on System z & z/VM, z/OS, DB2 for z/OS and the associated networks.


The SAP landscape at gkd-el
gkd-el runs a broad range of SAP modules, based on SAP ERP 6.0 and SAP BW 7.0, to accommodate the requirements of its different customers. The specific modules are:

• SAP HCM for payroll accounting. This is the largest use case for the SAP system, because all employees of the city of Gelsenkirchen, GEW, Musiktheater im Revier, Revierpark Nienhausen, Gelsenkirchener Kindertagesstätten, WiN Emscher-Lippe GmbH, and Kreishandwerkerschaft Emscher-Lippe West have their wage and salary calculation handled by gkd-el. Taking into account all the customers, the SAP HR specialists at gkd-el are responsible for 11,000 accounting cases per month from a technical point of view.
• SAP OM, another HR module of SAP, with municipal enhancements written in-house, is not only the central component for all of the other HR modules, but also the foundation for the SAP access control concept and is tightly integrated into the FI/CO and NKF modules. Additionally, it is used for staff appointments, advertisement of vacancies, travel expense reimbursements, and pension management.
• SAP FI/CO, with specific adjustments for Neues Kommunales Finanzmanagement (NKF). The NKF legislation, introduced by the Federal State of North Rhine-Westphalia, required public sector organizations to move to double-entry book-keeping. This change, and the limited amount of time allowed for the transition, was the main driver for gkd-el to move from proprietary applications to the standard ERP application offered by SAP.
• SAP PSCD for the municipal cash office
• SAP RE-FX for real estate management
• SAP MM/PM
• SAP BW 7.0
• SAP Solution Manager 7.0 as an administration tool for SAP Service.
• SAP NetWeaver Portal
• SAP extensions for municipal administrative procedures by GES Systemhaus GmbH, Wiesbaden. These include: EWO-MW & EWO-AW for the residents and foreigners registration office; KA Finanzen for duties and taxes; KFZ-FZ & KFZ-FS for the Driver & Vehicle Licensing Agency; and KFZ-BV Fines and Summary Offenses, which automatically handles the whole process associated with traffic offenses, from recording the offense with mobile and non-mobile devices through to the court hearing and collection of fines.

The IT foundation
gkd-el uses the IBM System z mainframe platform to host its SAP applications, and has recently consolidated multiple servers to a single IBM System z10 EC. The result is a simplified technology infrastructure, with fewer physical systems to manage, and the possibility of reduced licensing fees.

The z10 EC runs the SAP database server for each application on IBM DB2 for z/OS. The SAP application servers run on multiple virtual SUSE Linux Enterprise Server (SLES) 10 instances under IBM z/VM 5.3. Disk storage is handled using an IBM System Storage DS8100, and data is backed up to an IBM TotalStorage 3494 Tape Library.
“We felt that the IBM System z platform offered the most reliable technology at an attractive price,” says Dieter Schiffer. “The ability to upgrade processing capacity on demand was also important. We currently use two engines on the z10 EC, and we can activate more in a non-disruptive manner if we require them.


At a glance: IT Infrastructure

• System z10 EC model 2097-E12 with 1 CP, 1 zIIP, 3 IFL, 64 GB RAM
• z/OS 1.8 (migration to 1.10 planned in April 2009)
• DB2 for z/OS V8 (migration to V9 planned in October 2009)
• DB2 Connect server 8.1 (migration to a database thin client, based on the IBM DB2 Driver for ODBC and CLI, is planned for October 2009)
• Backup/Recovery/Maintenance: DB2 utilities, FlashCopy, InfoMat and InfoHcopy (InfoDesign)
• z/VM 5.3 (migration to 5.4 planned in February 2009)
• Linux on System z SLES 9 (migration to 10 ongoing)
• Job scheduling: Control-M (BMC)
• IBM System Storage DS8100 with 38.4TB
• IBM TotalStorage 3494 Tape Library
• Kodak System Nexpress DigiMaster 110
• Kodak System Nexpress DigiMaster 150
• Kern K686 enveloping system
• Network monitoring: WhatsUpGold (IPSWITCH)
• Firewall: GeNUGate 800 (GeNUA)

Linux advantages
The Linux operating system combines very low total cost of ownership with enterprise-class support for gkd-el’s SAP software. Moreover, with the SAP applications running on Linux virtual servers, gkd-el has enormous flexibility and can respond rapidly to changing business requirements.

The number of servers operating on Linux on System z at gkd-el is always growing. At present, all SAP application servers, the central print server for the mass printing facilities, and a server for GIS (land register information system) have been implemented as virtualized servers.

The ongoing process of server consolidation to Linux on System z replaces more and more physical servers, reducing administration effort and hardware costs. It also reduces server- based software licensing costs.


Highly productive workforce
Operations play an important role in the manageability and serviceability of the whole landscape. Backup, recovery and database maintenance are managed using DB2 utilities, automated with InfoMat from InfoDesign and Control-M from BMC. Some highlights of the InfoDesign software solutions are the possibility to recover – automatically – to any point in time and to create SAP systems for education and test purposes from homogeneous system copies. The migration of backups from disk to tape is automated by the HSM component of z/OS.

“Despite the ever-increasing workload and the addition of new SAP functionality in response to customer requests, we continue to require just 8 of our 84 employees to run the SAP landscape on IBM System z,” say Dieter Schiffer and Karl Große Vogelsang.

“This is only possible with the ease of administration that comes with using a centralized server on Linux for System z and DB2 for z/OS. We run a broad range of systems, from Intel-processor based servers through midrange UNIX systems right up to the System z mainframe, enabling us to draw clear conclusions about the relative efficiency of management for these systems.

“Based on this experience, we estimate that a skilled administrator in the non-mainframe environment can handle 10 systems effectively, while an administrator for Linux on System z can easily maintain some 50 systems.”


Coping with growth
Successful organizations like gkd-el are organizations that grow profitably. That should come as no surprise, because vigorous, sustained growth is what business executives in these organizations aim for in their strategies.

There are different ways in which companies can increase profit and strengthen their market position. One strategy is to expand activities into new geographical regions or to new customer segments. Mergers and acquisitions offer additional opportunities to penetrate new markets, enlarge the company portfolio and gain a better market foothold.

gkd-el took the most difficult – and potentially most beneficial – approach by introducing innovative products and services. The challenge was to implement the changes demanded by the NKF legislation introduced by the Federal State of North Rhine-Westphalia without letting costs explode. gkd-el not only overcame the challenge, it even turned it into a profit opportunity by offering SAP-related services to other city governments facing the same legislative challenge.

IBM System z has provided an IT infrastructure that is able to scale with the growth of gkd-el’s business, and without a corresponding increase in IT complexity. Greater complexity tends to add administrative IT costs; the simplification enabled by the IBM System z solution helps gkd-el to maintain downward pressure on its operational costs.

“Each month, I see municipal data centers being liquidated and outsourced. It’s no wonder to me when I look at the number of employees laid off at the same time, which is usually way over 150 employees. gkd-el has stayed flat for years at 84 employees. We were only able to master the new challenges due to our highly skilled workforce that worked together with the board to keep the costs down with the help of automation and consolidation,” says Dieter Schiffer.


Need for scalability
Several years of sustained growth necessitated an increase in server capacity to ensure sufficient performance and disk capacity. The gkd-el Service Level Agreements guarantee to its customers sub-second response times, and with increasing workload there was a risk of failing to meet this guarantee during peak load times. When gkd-el reviewed the possibilities with IBM, the conclusion reached was that a new System z10 EC would be 30 per cent less costly than upgrading the existing z990 to a similar capacity. In addition, the z10 EC was able to provide a wide range of new features like MIDAW, contributing significantly to the performance improvements. Since migrating to the z10 EC, gkd-el has seen average SAP dialog response times fall to just 190 ms, from 570 ms previously.

It came as a surprise to the gkd-el board and the city government of Gelsenkirchen that the upgrade from z990 to z10 EC could actually produce a reduction in monthly costs. This reduction can largely be attributed to reduced charges for z/OS and other server-based software licenses through the use of the IBM System z Integrated Information Processor (zIIP) specialty engine. The zIIP execution environment accepts eligible work from z/OS, which manages and directs the work between the general purpose processor and the zIIP. IBM does not impose software charges on zIIP capacity. In SAP environments, major parts of the workload of the DB2 for z/OS database are directed to the zIIP, reducing the costs for DB2 and all other related software considerably.

“By migrating to z10 EC and conducting associated streamlining work, we were able to reduce our monthly operational costs for the ‘Mainframe’ environment by 30 per cent. In this way, previously fixed and urgently needed capital became available for other necessary investments,“ explains Willi Lohmann, CEO of gkd-el.


Massive benefits of processor power
The Linux on System z application servers have experienced massive benefits from the increased clock speed and the doubling of effective RAM. Additionally, several other features of the z10 EC hardware contributed to a measured increase of approximately 270 per cent in SAP system throughput.

The Modified Indirect Data Address Words (MIDAW) feature allows gkd-el to transfer the same amount of data with fewer signals up and down the channel. The legendary I/O capabilities of the System z platform are thus further enhanced, and the performance increase is particularly noticeable for Extended Format VSAM data sets. In the SAP environment, tables or partitions bigger than 4 GB are Extended Format VSAM data sets. Additionally, MIDAW can help to increase overall DB2 for z/OS log performance, because it effectively allows log data sets to be striped, which requires Extended Format VSAM data sets.


HiperSockets enhancements
HiperSockets Layer 2 support: HiperSockets Layer 2 support is exclusive to System z10, and is supported by Linux on System z, and by z/VM for guest exploitation. While this feature is not exploited by SAP, its benefits for network management are considerable.

HiperSockets Multiple Write Facility: HiperSockets performance has been enhanced to allow for the streaming of bulk data over a HiperSockets link between logical partitions (LPARs). The receiving LPAR – containing, for instance, the SAP database on DB2 for z/OS – can now process a much larger amount of data per I/O interrupt. This enhancement is transparent to the operating system in the receiving LPAR. HiperSockets Multiple Write Facility, with fewer I/O interrupts, is designed to reduce CPU utilization of the sending and receiving LPAR and is supported in the z/OS environment.


Connectivity enhancements
6Bps InfiniBand^® host buses for I/O speed up communication considerably by having a speed of 6GBps within the z10 EC to the I/O cages.

Additionally, the use of a 4-channel FICON 4 adapter to connect the z10 EC and the DS8100 (which are 10 meters apart) has improved auto-negotiated link date rates from 1Gbps to 4Gbps. FICON Express4 provides significant improvements in start I/Os and data throughput. The z10 EC is connected to the rest of the IT infrastructure with OSA-Express3 10 Gigabit Ethernet.

A new intermediary level 1.5 cache (L1.5) of 3MB per PU between Layer 1 (which is very small) and Layer 2 (which takes more time to access, because it is located on the book) has been introduced. The speed is near to L1 cache, because the L1.5 cache is also located on the PU chip.


Exploiting IBM System z10 EC
gkd-el also plans to exploit several other performance-related features of the new IBM System z10 EC.

HiperDispatch greatly enhances the processor affinity support to further improve the usable capacity of the server. Now PR/SM, WLM and z/OS cooperate on affinity assignments, which are done at the task level rather than just at the partition level. Despite the RAM being distributed through the books, and books having individual Level 2 caches, all processors have access to all the Level 2 caches and memory. Thus, the server is managed as a coherent symmetric multi-processor (SMP). Nevertheless, it enhances performance considerably, if data and instructions are processed within a book.

In order to minimize latency, z10 EC aims to dispatch work to a group of physical CPUs that share the same Level 2 (L2) cache. PR/SM manages the utilization of physical processors by logical partitions by means of assigning the logical processors to the physical processors. But PR/SM is not aware of which workloads are being assigned by the operating system to which logical processors. The Workload Manager (WLM) component of z/OS has the information at the task level, but is unaware of physical processors.

This disconnect is solved by new enhancements on z10 EC that allow PR/SM and WLM to work more closely together. They can cooperate on creating an affinity between task and physical processor rather than between logical partition and physical processor.

HiperDispatch is available with z/OS release 1.10 on IBM System z10 EC servers.


Additional HiperSockets enhancements
zIIP-Assisted HiperSockets for large messages: in z/OS release 1.10, HiperSockets has been enhanced for zIIP exploitation. Specifically, the z/OS Communications Server allows the HiperSockets Multiple Write Facility processing for outbound large messages originating from z/OS to be performed on a zIIP, reducing software monthly license charges.
zIIP Assisted HiperSockets for large messages is available with z/OS V1.10 (plus PTF UK37306) on IBM System z10 EC servers.

Large page support for 1MB pages is considered to be especially useful for SAP Application Servers on Linux on System z. The size of pages and page frames has been 4KB for a long time. While maintaining the ability to use the current 4KB pages, the z10 EC server introduces the capability of having large pages 1 MB in size. Both page frame sizes can be simultaneously used.

Large pages enable the translation lookaside buffer (TLB) to better represent the working set and suffer fewer misses by allowing a single TLB entry to cover more address translations. Exploiters of large pages are better represented in the TLB and are expected to perform better. This support is primarily of benefit for long-running applications that are memory access intensive – precisely matching the profile of SAP application servers.

Large page support for 1 MB pages is available with z/OS release 1.10 on IBM System z10 EC servers.


IBM Systems Storage
In order to meet the Service Level Agreements and to provide round-the-clock services to its customers, gkd-el also refreshed its data storage infrastructure, upgrading from an IBM TotalStorage Enterprise Storage Server 800 (ESS800) to the latest IBM System Storage DS8100 disk array. The DS8000 series is designed to support the most demanding business applications, offering exceptional performance, superior data throughput and world-class resiliency features.

“During the first few weeks after the migration to z10 we continued to use the ESS800. After the upgrade to the DS8100 we noticed a performance improvement of 35 per cent for DB2 utilities, which are backup and batch jobs. These improvements are due to the move to FICON 4 and to the improved architecture of the DS8100”, explains Dieter Schiffer.


Making use of DS8100
gkd-el makes use of the following features of the DS8100:

Parallel Access Volume (PAV)
Within a z/OS system, each z/OS device number has a dedicated Unit Control Block (UCB) which holds information about an I/O device, such as features of the device and state information. An alias UCB is a control block used to keep the status of a z/OS I/O operation during pending, connect and disconnect time. PAV allows multiple concurrent I/Os to the same volume at the same time from applications running on the same z/OS system image. This allows applications to share the same logical volumes, and improves performance. When an I/O request is made to a base address, z/OS uses the base UCB or one of its alias UCBs, depending on availability, to initiate the request. gkd-el has licensed PAV for up to 50TB capacity.

IBM FlashCopy
IBM FlashCopy on the DS8100 allows entire volumes to be instantaneously ‘copied’ to another volume. It is also possible to create a consistent point-in-time copy across multiple volumes, and even across multiple storage subsystems using ‘consistency groups.’ gkd-el is using FlashCopy to backup its SAP database on a daily basis.

FlashCopy SE (Space Efficient)
The FlashCopy SE feature allocates storage space on an as-needed basis by using space on a target volume only when it actually copies tracks from the source volume to the target volume. This dramatically reduces the time needed for backups, enabling several backups to be created each day. This in turn reduces the time needed for the roll-forward processing during a recovery, because there are fewer changes in the database. gkd-el has licensed this feature but is currently not using it, since the current backup window is sufficient to backup the required data. The utilization of this feature is planned for the future.

Disaster Recovery
gkd-el plans to build a separate location for disaster recovery purposes. The DS8100 Metro Mirror feature will be used to keep the stored data in both the production and the disaster recovery locations synchronized. Since the locations will be relatively close together there will be no need to use z/OS Global Mirror.

Virtual Disk Capacity
Currently gkd-el is using virtual disk sizes of 9GB (disk type 3390-9). In the near future, the company will migrate to larger disk sizes of 27GB (disk type 3390-9). This will lower the maintenance effort and also have a positive impact on performance.


Conclusion
The transition from IBM z990 to IBM System z10 EC has improved overall SAP system throughput at gkd-el by 270 per cent while reducing the total cost of ownership by 30 per cent through the use of zIIP and IFL specialty engines.

“The ability to utilize the breakthrough technology of z10 EC while simultaneously decreasing our costs by 30 per cent was an offer we couldn’t refuse. The transition to the z10 EC was smooth and without any problems thanks to excellent support from IBM. The advantages of IBM System z – such as excellent stability, advanced virtualization technologies, high security and easy management – make us confident that we are prepared for any future challenge,” say Dieter Schiffer and Karl Große Vogelsang.