Kommunale Datenzentrale Mainz selects IBM Power Systems and Linux as the platform for its clients’ SAP applications

About this Paper

This brief describes the planning and implementation of the DZ-Kommunalmaster^® double-entry bookkeeping solution, which is based on SAP ERP 6.0 and IBM Power Systems technologies, for the Mainz Municipal Administration. Kommunale Datenzentrale Mainz (KDZ), a municipal facility, operates this system. The infrastructure also serves as a consolidation platform for other SAP and non-SAP applications, enabling KDZ to offer its services to local government authorities in the Rhineland-Palatinate region on a stable, powerful platform. This brief shows how the project workflow and technical implementation enabled KDZ to leverage a virtualized server and storage platform and the open operating system, Linux.

Customer objectives

• Launch double-entry bookkeeping in accordance with the latest statutory requirements
• Set up a future-safe, flexible IT platform for hosting SAP applications
• Establish Linux as a strategic operating system platform
• Ensure failsafe operation of the SAP production systems that KDZ hosts
• Consolidate an existing SAP system for facilities management onto the new platform
• Integrate the new landscape into the existing systems management environment, especially for backup/restore and monitoring purposes

Customer benefits

• Enjoy the benefits of the easy integration of IBM Power Systems and an open Linux operating system (SUSE Linux Enterprise Server) in terms of performance, hardware stability, and scalability, as well as the flexibility delivered by PowerVM virtualization
• Achieve synergies by running SAP and non-SAP applications on the same hardware: reduce energy requirements by retiring old physical servers; improve overall capacity utilization and dynamic load-sharing by means of workload consolidation and PowerVM resource sharing; and save on IT peripherals through I/O virtualization in the server and hard disk area
• Enable a swifter response to changing customer requirements, make it easier to adapt to user and application growth, and enable rapid activation of new SAP instances

IBM solution

• Launch a fully virtualized server and storage landscape: IBM Power Systems and PowerVM for server and resource virtualization and IBM System Storage SAN Volume Controller (SVC) for storage virtualization
• Set up a high-availability cluster for critical customer applications: IBM Tivoli System Automation for Multiplatforms for SAP instances; Oracle DataGuard for databases used by the Residents’ Registration Office
• Use Nagios to provide open source availability and performance monitoring; monitor IBM hardware components (storage, server, SAN, etc.) and link Nagios to SAP CCMS and Oracle for integrated application monitoring
• Work with IBM Global Technology Services, providing Dynamic Infrastructure Tools for Open Source Environments: introduce standardized and semi-automated methods to install and run the overall system, relying on the extensive experience and skill set of IBM Global Technology Services for Linux and SAP Basis services
• Work with IBM Global Business Services to provide advice and project management throughout the launch of the SAP double-entry bookkeeping solution
• Upgrade the Oracle database from the outmoded Release 8 to the newer version, Oracle 10g
• Implement CommVault Galaxy Data Protection as backup agent to ensure reliable, rapid recovery in the event of a disaster

Introduction and Overview

About Kommunale Datenzentrale Mainz (KDZ)
Kommunale Datenzentrale Mainz is an enterprise owned and operated by the city of Mainz, which provides information processing services for the city and other public bodies and for corporations under public law and other associated institutions. The information processing tasks performed by KDZ are comprehensive, ranging from analysis through to production.

Many other local government authorities and public institutions in the Rhineland-Palatinate area make use of the know-how of the KDZ Mainz and its 50 or so employees. For resident registration, for example, KDZ Mainz handles payroll services and hosting for 170 local government authorities in the region.

The main focus of this expertise is on specialized public service applications. Some are based on standard software such as SAP ERP and others on separate legacy applications that must all be run in a standardized computing center facility.
An application in the former category is DZ-Kommunalmaster^®, the double-entry bookkeeping solution that KDZ wanted to implement. This solution is a version of SAP ERP 6.0, specially customized by Datenzentrale Baden-Württemberg to fulfill local government requirements.

In accordance with Rhineland-Palatinate state law, the city of Mainz, and with it KDZ as the city’s IT service provider, needed to introduce double-entry bookkeeping within three years, making a rapid implementation critical.

For the past seven years, KDZ has also hosted the EWOISneu Resident Registration system for around 170 Rhineland-Palatinate local government authorities on distributed x86 systems. EWOISneu consists mainly of the MESO and DIGANT applications. MESO is an integrated solution for Residents’ Registration Office clerks. DIGANT software is used to make digital passport and identity card applications to the Bundesdruckerei, or Federal Printing House, in Berlin. As a part of the replacement and consolidation of existing database systems, the MESO database instances were merged in Linux partitions on IBM Power Systems.

This database consolidation and virtualization has achieved the maximum possible system capacity utilization. Thanks to Capacity on Demand (CoD) and virtualization in the SAN area, a flexible response to additional resource requirements is possible without hardware expansion. As the application servers continue to be run on upstream Citrix terminal servers using IBM BladeCenter servers running Microsoft Windows, there has been no change to the user experience.

KDZ has extensive knowledge and experience with Linux. Linux was chosen as the back-end operating system seven years ago because it provided greater stability and better automation capabilities than a Microsoft Windows platform. With well over 400 database instances to be managed, Linux’s ability to run scripts was decisive. The option of using another Unix-based operating system was ruled out because of this.

IBM IT Consulting and Project Implementation
Four years ago, IBM Global Business Services began assisting the city of Mainz with the introduction of double-entry bookkeeping by implementing standard SAP ERP software along with Baden-Württemberg Datazentrale’s preconfigured SAP template, DZ-Kommunalmaster^®. As the general contractor, IBM was in charge of project management and project launch. Datenzentrale Baden-Württemberg (DZ BW) was involved in implementing the double-entry bookkeeping solution as a partner and subcontractor.

IBM Global Technology Services (GTS) provided the design and implementation of the infrastructure, SAP Basis and associated middleware. The IBM Business Partner SVA System Vertrieb Alexander GmbH supplied a flexible and scalable consolidation platform for the application landscape that included two IBM Power System p570 servers and the IBM System Storage SAN Volume Controller.


The application landscape at KDZ and resulting requirements

Double-entry bookkeeping with DZ-Kommunalmaster^®
Double-entry bookkeeping is replacing Kameralistik, the system of public sector fiscal accounting that was previously widespread in Germany. The aim is to manage and control the economic use of public funds better. The old system, which merely compared planned income and expenditure with actual income and expenditure, did not permit an analysis of resource consumption or a profitability appraisal. The adoption of double-entry bookkeeping in local government is intended to provide administration staff and decision makers with better commercial management options (cost accounting, budgeting, reporting, etc.) and to highlight any potential future budget savings.

Using SAP Business Suite, a team from Datenzentrale Baden-Württemberg (DZ BW) worked with SAP to develop the DZ-Kommunalmaster^® reference solution. DZ-Kommunalmaster^® is a preconfigured, customized SAP solution that is already executable and includes many functions that a local government authority or public sector administration requires. The most significant include variable accounts and product structure, cost accounting, product management, investment management and built-in asset accounting and asset assessment. SAP’s new Public Sector Collection and Disbursement (PSCD) component, also known as ‘Bürgerkonto’ or ‘citizen’s account’, is also fully integrated.

As an SAP Special Expertise Partner, DZ BW uses the SAP reference model as its basis and complements it with experience gained from implementations on behalf of more than 350 clients. The advantages of the DZ-Kommunalmaster^® module, which is customized to fulfill local government requirements, are self-evident:

• A swift and therefore cost-effective launch
• Few customized settings are required
• Fewer bugs and high quality due to standardization across a large number of installations
• Centralized product maintenance/updates/development by DZ BW
• Change of release as a part of general SAP upgrades with timely use of new functionalities.

Along with the existing SAP ERP 6.0 landscape, the DZ-Kommunalmaster^® application is a critical core application for KDZ. For this reason, the entire SAP environment was designed to deliver uninterruptible service and data availability. Where the hardware is concerned, appropriate redundancies were incorporated to create a high-availability cluster and enable data replication for the Oracle database.

Facilities Management with PROMOS.CITY at Gebäudewirtschaft Mainz
PROMOS.CITY, a solution from PROMOS consult GmbH & Co. KG, is an add-on to the SAP ERP 6.0 RE-FX module. Industry-specific adaptations and expansions combined with preset processes in areas that include real estate and management of local authority buildings and property make it easier for Gebäudewirtschaft Mainz (GWM) to keep stock of real estate under public ownership totaling around EUR 1.2 billion.

Four years ago, nearly 20 municipal departments in Mainz consolidated a variety of legacy systems to PROMOS.CITY. For a year, the solution was run on two x86 systems under Linux. At this point, a heterogeneous system copy was used to migrate these SAP systems to the new SAP operating platform at the KDZ. Distributed among two IBM System p570 servers and several logical partitions (LPARs), the solution is now hosted by KDZ and is highly available.

Residents’ Registration software: MESO and DIGANT for the Rhineland-Palatinate
MESO is an application for Residents’ Registration Offices. It is a client-server application that is independent of SAP ERP. It supports the administrative processes that typically occur at a local authority Residents’ Registration Office. The data it holds is based on a standard German federal registration data set.

DIGANT software enables Residents’ Registration Offices to apply online to the Bundesdruckerei in Berlin for passports and identity cards.

Due to their close dovetailing with Office applications, the client components do not run on Linux; they require a Microsoft Windows operating system. The server component consists mainly of Oracle database instances, and can therefore run either on Microsoft Windows servers or under UNIX/Linux.

KDZ Mainz hosts and runs the MESO and DIGANT applications for the EWOISneu residents’ registration process for around 170 of the 212 local authority Residents’ Registration Offices in the Rhineland-Palatinate. The largest database instance is for the city of Mainz. Just over a year ago, the x86-based hardware on which the MESO applications and back-end databases had been hosted for several years had to be replaced. At the same time the Oracle database was due for an upgrade from the outmoded Release 8 to the current version, Oracle 10g.

Due to good experiences with Linux and Oracle 10g for the SAP systems on IBM Power Systems p5 570 servers, consolidating the database instances for MESO on this scalable and flexible platform was an obvious choice. The client components are delivered by a Citrix terminal server farm, distributed among several IBM BladeCenters.

Many of the same standardized processes created for the SAP environment were used to implement and operate the new MESO environment. The SAN storage environment and the database server systems process the MESO workload as well as the SAP workload, so the overall environment is characterized by a very high degree of consolidation and efficiency.

Design and implementation of the new infrastructure
The essential requirements for the new IT environment, supporting the double-entry bookkeeping and facilities management systems, were a high degree of automation, low administrative costs and a system landscape that could be managed as simply as possible.

The customer’s desire for a high degree of consolidation and, accordingly, the highest possible level of virtualization of the new environment led to additional requirements, especially with respect to the availability of systems and components.

Possible implementation scenarios were discussed at several architecture workshops. As Linux was explicitly required as the operating system platform, suitable reference architectures from other IBM customer projects served as a basis. Availability requirements were taken into account in choosing both the servers and the subsequent system design.

The Power Architecture platform was chosen because it fulfilled all of the KDZ’s requirements:

• The possibility of a high degree of consolidation and virtualization of individual systems
• Virtualization of non-production and production Oracle database instances
• The certification of the virtualization technology by SAP
• Support for Linux as an operating system
• Flexibility and uninterruptible expansion options
• A substantial degree of scalability to accommodate future workloads.

Design and implementation of the solution were materially influenced by the following conditions:

• Distribution of systems between two fire defense sections around 150 meters apart
• Availability of applications for end users between 6:30 a.m. and 6:00 p.m.
• Availability of applications for batch runs between 6:00 p.m. and 6:30 a.m. including backup and restore.

The basic SAP infrastructure
Three years ago, two IBM Business Partners, Computacenter AG & Co. oHG Kerpen and SVA System Vertrieb Alexander GmbH (SVA), were awarded contracts after a public tender by KDZ Mainz to supply the IBM storage components and IBM server systems required for the double-entry bookkeeping project’s infrastructure.

The system capacity parameters (SAPS) for the new servers were deduced from configurations similar in size used by the cities of Karlsruhe (an SAP-based DZ-Kommunalmaster^® solution) and Münster (an SAP-based PROMOS.CITY solution). Further growth forecasts, specific entry codes used by the Mainz municipal administration and an additional loading for the use of Unicode were taken into consideration.

Once the hardware was supplied, IBM Global Technology Services and the IBM Enterprise Linux Services (ELS) team assisted KDZ with the finer details of the complex implementation and assumed partial responsibility for implementing the basic technical system.

A modern SAN infrastructure was established as the first step. The next involved installing the SAP systems and high-availability clusters, with Linux as the operating system.

The requirements for the new system environment were first specified in architecture workshops. The system design was then undertaken and the LPAR layout was defined.

In the process, a number of availability requirements for the environment led to changes in the distribution of logical systems or virtual machines (LPARs) from the originally plans. Due to the very high flexibility of the PowerVM virtualization and the I/O subsystem used, this did not have a negative influence on the original system design and hardware configuration.

MESO and SAN virtualization
Two years ago, the IBM Power Systems environment used for SAP workload was expanded considerably. Increased capacity was required because of the migration project to replace the residents’ registration databases hosted at the KDZ in Mainz for around 170 local government authorities in the Rhineland-Palatinate. At the same time, the existing SAN infrastructure was virtualized.

In the course of the MESO project, the two IBM System p570 servers were expanded to 16 cores each. The systems continue to have Capacity on Demand (CoD) reserves in order to respond swiftly and flexibly to new requirements. These reserves exist to cover spikes in demand for individual applications, while also providing capacity for the creation and rapid activation of additional logical partitions.

By using IBM SAN Volume Controller (SVC), KDZ was able to standardize its SAN further, and thereby gain significantly more flexibility, which is indispensable for a modern IT service provider. For example, all servers now use a uniform device driver to access the SAN storage unit via SVC. The provision of new, virtual SAN disks was simplified. In this way, an option to optimize existing storage and expand it flexibly, or even swap entire disk subsystems, was created. This can be done transparently for connected services and can largely be carried out while the system is up and running.

Synergies between SAP and MESO
Mixed operation of SAP and non-SAP applications made it possible to achieve substantial consolidation, which is partly reflected by a higher overall utilization of system capacity and significantly lower power consumption.

The IT infrastructure currently used by SAP and MESO is as follows:

• Two IBM Power System p570 servers with a total of 47 virtual systems (LPARs), each with
  • 16 cores, including 4 cores as CoD reserve
  • 256 GB RAM, including 32 GB as CoD reserve
  • Two virtual I/O servers for the network and SAN connection for SAP
  • Two virtual I/O servers for the network and SAN connection for MESO
• Two IBM System Storage SAN Volume Controllers (SVCs)
• Two IBM System Storage DS4800 storage subsystems for SAP and MESO data
• Backup/restore infrastructure comprising
  • One IBM System Storage DS4700 storage subsystem
  • One IBM Total Storage TS3310 tape swapping unit
• Four IBM BladeCenters with IBM HS21 blades for MESO, DIGANT and further uses.

For redundancy reasons, two virtual I/O servers per system are in use for SAP and two for MESO applications to make it possible to separate the administration of the SAP environment from the MESO environment.

A number of very small new ERP environments are planned for the future. As the planning stands, this will require no hardware extensions to the IBM Power Systems p570 servers. The computing and main storage capacity required can be delivered by activating the current Capacity on Demand (CoD) reserves permanently.


Solutions for computing center operation

Dynamic Infrastructure Tools for Open Source Environments
In its Infrastructure Tools for Open Source Environments, IBM GTS’s Enterprise Linux Services Team has created and established a tool that makes it possible to achieve a high level of automation and thus accelerate the individual job steps in installing, configuring, and running virtualized and highly available environments under Linux.

By using this library, an environment can be created in a very short time. New Linux systems for further SAP landscapes or database instances can be provided, and regularly recurring operating tasks can be optimized and automated. The benefits for the customer include cost savings for services and faster infrastructure availability for installing the actual application. The customer was provided from the outset with detailed documentation of the system environment and matching administration tools.

IBM Tivoli System Automation for Multiplatforms
To achieve the high SAP application availability required, the IBM Tivoli System Automation for Multiplatforms (TSA) cluster solution was implemented at KDZ. The product is partly characterized by a very powerful, rule-based configuration mechanism that permits simple and flexible mapping of dependencies and groupings of cluster resources. This makes it easier to automate complex application landscapes such as the SAP environment at KDZ.

A total of three independent TSA clusters have been implemented at KDZ. Each is controlled by a configurable rulebook or automation policy.

The rulebook is based on standard policies that are provided by IBM free of charge for standard applications such as SAP Business Suite and Oracle. In combination with scripts from the above-mentioned Dynamic Infrastructure Tools for Open Source Environments, a customized rulebook has been written. It links the standard policies for SAP components, the Oracle database, and central services with the rules and dependencies for critical file systems and other applications such as interface processes.
In addition to their high availability functions, the TSA sequences generated are used during regular offline backups. This ensures that SAP processes are stopped and restarted in the right sequence in all cluster nodes.

As a cluster solution, TSA constantly monitors critical SAP processes. If one of them – or an LPAR or even the entire p570 server – fails, TSA automates the critical process transfer to the backup computer. In particular, policy contents include managing the failover of the database and enqueue server. To accelerate the transfer the enqueue server was implemented as a replicated enqueue. Thus the backup system always has a current version of the central locking table that no longer needs to be recreated laboriously when the SAP instance is started. So downtime during data transfer to the backup server is drastically reduced.

Oracle DataGuard and Oracle Enterprise Manager Grid Control
Due to substantially higher availability requirements and the larger data volume (c. 1 TB per LPAR) of database instances, a more far-reaching concept than the above-mentioned cluster solution had to be devised for MESO to protect against disaster.
This led to the launch of Oracle DataGuard and Oracle Enterprise Manager Grid Control as management tools at KDZ during the MESO project. This introduced logical database replication between two spatially separated fire defense sections. Compared to a cluster solution with data mirroring, this has the advantage that in the event of an error, an immediate switchover to an intact copy of the database can be undertaken. A protracted restore of the database from the backup medium followed by a consistency check is no longer required. This presented one of the main reasons for choosing this solution.

With this important move, KDZ has achieved an even higher service level, and outage times in the event of a disaster have been minimized further. As the virtualized basic systems or the LPARs of the SAP systems and those of the new MESO database systems differ only minimally, in the future Oracle DataGuard can be used as an additional option in the SAP environment at KDZ.

IT Monitoring with the Nagios
The KDZ uses an open source solution, Nagios, to monitor the entire IT infrastructure. As a part of the SAP and MESO project, Nagios plug-ins for the IBM hardware (DS4700, DS4800, SVC, HMC, IBM Power Systems p570) and software components (Virtual I/O Server, Linux, SAP, Oracle) were developed, installed and configured, thereby enabling end-to-end monitoring of the environment’s availability and performance.

CommVault Galaxy^® Data Protection backup solution
A fundamental criterion in the choice of hardware platform was – in addition to the availability of Linux as an operating system and of Oracle and SAP as applications – the availability of a backup agent. By porting the basic backup agent for the CommVault Galaxy^® Data Protection solution used at KDZ, the manufacturer CommVault contributed toward the successful completion of the project. A fully-fledged CommVault Galaxy client that can also be integrated in SAP is planned and is already under development.


Summary and Outlook
One of the first extensive double-entry bookkeeping installations under Linux has been implemented at KDZ. The use of Linux as the operating system has proven that it has the necessary flexibility and stability as a consolidation platform for heterogeneous mission-critical applications. The combination of virtualization by means of IBM PowerVM, IBM SAN Volume Controller and an open source operating system has proven ideal both for successful project realization and in ongoing operation.

The KDZ can now respond swiftly to customers wishing to host further SAP applications. The present environment can be expanded very quickly through the addition of two new server systems. The preconditions for this were created mainly by the modifications to the SAN and network infrastructure and by the design of the present storage and server environment.