andagon delivers deep insights into application performance with SAP and IBM

Published on 25-Jan-2013

"The Rational Performance Tester support team is excellent, providing solutions quickly and actively helping us to execute successful performance test projects with our customers." - Stefan Gogoll, CEO, andagon

Customer:
andagon

Industry:
Computer Services

Deployment country:
Germany

Solution:
Business Performance Transformation, DevOps, Enterprise Resource Planning, High Availability , Virtualization

IBM Business Partner:
SAP

Overview

IBM Business Partner andagon GmbH (www.andagon.com) is a renowned consulting company in the field of quality assurance, specializing in software testing, process and test automation. Headquartered in Cologne, Germany, andagon employs more than 50 quality experts.

Business need:
andagon wanted to provide a cost-efficient software testing service to end-customers deploying mission-critical SAP applications. The company also aimed to help its customers to avoid costly application failures that interrupt business processes, as well as reduce the risk of system downtime and cut the business risk when implementing new applications.

Solution:
andagon leveraged IBM Rational® Performance Tester and the IBM Rational Performance Tester Extension for SAP Solutions to evaluate SAP application performance and support customers with detailed reports in optimizing SAP applications to avoid business disruptions.

Benefits:
Improved application throughput by 100 percent, from 6,000 units per day to 12,000 units per day. Ensured that a certain operation was completed in less than one second for 90 percent of all users. Identified and eliminated issues that would have caused severe business disruption by analyzing application behavior in extreme situations and examine how fast normal operations can continue after overloading occurred.

Case Study

Background, starting point and objectives

About andagon

IBM Business Partner andagon GmbH (www.andagon.com) is a renowned consulting company in the field of quality assurance, specializing in software testing, process and test automation. Headquartered in Cologne, Germany, andagon employs more than 50 quality experts.

The company has developed aqua (aqua.andagon.com), a comprehensive software solution designed to improve application lifecycle management by integrating quality management into IT project management. aqua revolutionizes the software business processes in IT projects by enrolling all parties in the process: business users, project leaders, analysts, developers and testers, and creates more transparency through improved control and communication – including internationally-distributed teams.

The company’s consultants help customers across multiple sectors to implement test automation and quality assurance processes including functional testing and performance testing. For those customers developing and using SAP applications, andagon offers a comprehensive range of tailored testing services designed to optimize performance and reliability.

Initial IT environment

When customers first approach andagon, often the application to be examined is almost completed. Frequently, customers have identified quality issues or performance bottlenecks during the implementation phase, and turn to andagon to analyze the difficulties and develop a customized test plan to efficiently address the performance issues.

In typical commercial environments software testing is often delayed until the very end of a project. As the new solution is about to be rolled out, functional acceptance and verification tests are conducted. When the software has passed these tests, performance tests are considered as a last, quick check before the solution is approved for production operation.

Guy-Dominic Gorek, IT Consultant for Quality Management and Test Automation at andagon explains, “In most projects, performance testing is the last stage. Usually customers schedule only between four and six weeks’ time for final performance tests, which when compared with the project duration is a tiny fraction of the total.”

The late involvement of test specialists increases the pressure on the testing team and process. To assess the performance of a complex application quickly and effectively, efficient tools are required to create and execute the relevant test cases.

Business challenges and project objectives

Thousands of companies rely on SAP software to run their core business processes, and need to make sure that application errors do not disrupt business operations. Particularly for manufacturing companies, system downtime can cause significant financial damage as a stopped production line can result in lost profits and material loss.

Software is a key enabler for companies that wish to grow their business, manage product complexity efficiently and meet corporate targets. If software quality and performance issues are spotted at a late stage in a development project, the costs and delays incurred by corrective work damages companies’ abilities to reach their goals.

To reduce costs and avoid complex reengineering works, companies need to find efficient ways to test new software thoroughly and only release mature code into production systems.

As compliance with international standards becomes more important, companies reach out to establish recognized quality management measures in IT, as they before did in manufacturing by implementing the ISO 9001 standard for quality management systems. At first, organizations focused on standardizing and improving general IT services by implementing the international ITIL v3 standard. The ISO 25000 series of standards addresses the demand for improved software quality by defining a range of criteria, allowing companies to assess software quality within a defined framework. Relying on approved quality measures increases the success of software development projects and allows for independent assessment of software quality.

Technical solution

Software testing is a complex task that often does not receive the attention required to make software implementation projects a success. andagon advises its customers to integrate testing into general development processes to monitor software quality throughout the project and avoid discovering functional or performance issues at the end of an implementation. As an IBM Business Partner, andagon is a strong advocate of the comprehensive IBM toolset for application development and testing.

Software testing can be split up into different tasks. First, functional testing is required to ensure that the system processes data as expected and delivers the results required to drive mission-critical business processes. Once the system passes its functional tests, the next step is to ensure that the application processes the data within the required timeframe.

“Performance testing is crucial throughout the software development process,” explains Guy-Dominic Gorek. “Our experience shows that the earlier performance tests are conducted, the easier it is to identify and eliminate performance problems in software projects.”

IBM Rational Performance Tester provides a broad range of features to enable efficient performance testing, and helps to identify application architecture and infrastructure bottlenecks. A simple graphical user interface allows for the creation of test cases without the need to write code, and integrated test data management enables flexible and dynamic customization of test cases. IBM Rational Performance Tester automatically executes tests with dynamically generated test data, and tracks the dynamic data throughout the tests to ensure that correct, useful test results are generated.

IBM Rational Performance Tester uses flexible IBM Rational Performance Tester Agents to generate the performance test load. The load test agents emulate user behavior to generate a workload that is as similar as possible to real users.

Efficient performance testing

andagon has a track record of successful performance testing projects. For best test results, andagon develops a tailored test plan for each scenario, based on a standard method of performance testing. Working with its customers, andagon divides the processes to be tested into steps of a project, as they occur in real world operations. The performance of each step in the process is measured independently.

Once the test is completed, the data created during the test run is deleted using database tools to restore the initial testing data and the test environment is reset back to its initial status to ensure that the results of different test runs are not skewed by side effects such as increasing data volume. By starting every test run with exactly the same initial data, test results can be reproduced, helping to identify bottlenecks and improve application performance.

“By breaking business processes down into the actual tasks that occur in everyday work, and testing these tasks independently using a set of test cases comprising one or more separate test steps, we obtain realistic results,” says Guy-Dominic Gorek. “If you test a process from start to finish, you can test the application performance as well, but it is more difficult to interpret the results in terms of real user load on the systems.”

In general, there are two approaches to performance testing. The first approach is to treat all process steps independently and generate separate test data for each single process step. But in many business processes, subsequent process steps require input data from previous process steps and cannot reliably be tested independently.

For example, to dispatch a delivery, you need a complete purchase order including all required delivery details. This is where the second approach shows its benefits, when testing a process from start to finish, only test data for the first step needs to be provided, test data for all subsequent tests are generated in the testing process. But the downside of this method is simply that if there are performance problems in one step, subsequent tests cannot be executed, limiting the scope of the test, reducing the overall insight provided by the performance test and rendering the test results largely useless.

Testing multiple tasks independently requires additional preparation, as the test data for each step in the complete business process needs to be provided in advance. Separating steps of a process and testing the performance of each step individually, provides much greater insight into the performance of an application, even if one or two steps of a process do not pass the performance test. If this happens, test cases later in the workflow can still be tested accurately. This complete view leaves the customer with a much better dataset to analyze the performance issues and optimize application performance.

To automate performance testing and generate the required workload, andagon relies on IBM Rational Performance Tester. The software provides testers with automated performance testing capabilities to validate the scalability of business applications. IBM Rational Performance Tester supports load testing against a broad base of applications, including SAP software.

IBM Rational Performance Tester helps companies to identify the presence and cause of system performance bottlenecks. Tests can be created quickly without developing code, and no programming knowledge is necessary. The tool provides advanced modeling functionality to create diverse user populations, making it easy to improve the value of performance tests by adapting the workload to real user behavior. A comprehensive test editor provides high-level overviews and deep insights into tests and test results.

Comprehensive reporting with real-time capability enables testers to identify performance problems immediately – even while the test is running. Since the tool is configured to collect server resource data, the user has a full view of a system’s performance in real-time. The test editor control includes the ability to build tests in a tree structure rather than as script code, which facilitates debugging and test maintenance. Finding errors in extensive tests can be very hard; the tree view enables test developers to debug scripts that would almost be impossible to debug without such an intuitive user interface.

“Especially for complex test setups, the tree view is a unique asset to gain a quick overview,” says Guy-Dominic Gorek. “This visual interface really helps explaining to users what we are going to test and how this performance testing works.”

IBM Rational Performance Tester with the IBM Rational Performance Tester Extension for SAP also provides the capability to dynamically change tests. The tool can display screenshots of the SAP user interface, and test engineers can adjust and extend tests easily by clicking on any UI object shown in the screenshot, triggering the test software to perform the requested action and gathering performance measures for this interaction.

In addition, the capability to extend tests with custom Java code supported by a fully integrated development environment allowed quick implementation of complicated tests with a widely used programming language. This flexibility is an important feature, essential to implement advanced test scenarios. For more advanced performance test scenarios, tests can be augmented flexibly with custom Java code, taking advantage of the full capability of the integrated Eclipse Java development environment.

Using Rational Performance Tester for SAP application testing

andagon recently supported a wholesale logistics company with its implementation of a new and highly customized SAP solution. The implementation was considered a failure, principally owing to performance issues.

The company engaged andagon to investigate the performance bottlenecks and verify the application performance with a comprehensive test setup before the solution is rolled out.

andagon started by analyzing the performance requirements and worked with the customer to create a test plan. End users were asked about their usage patterns to identify the relevant test cases. The goal was to develop a comprehensive test plan that would simulate the peak performance throughout one busy working day. In this example, the wholesaler is highly affected by seasonal peaks, and wanted to make sure it would not hit trouble on its busiest days of the year.

The customer wanted to combine a performance test with a user acceptance test, involving staff to take part in the test activities on a Sunday to avoid disruptions to business operations. Alongside ordinary workers, automated tests would simulate a second and third location.

To avoid concurrency issues between real users and virtual users, the users at each location operated on a different set of master data. Transactions from the additional locations would include SAP ERP financial and controlling activity, and the interaction with the central SAP ERP system across multiple locations and departments would mirror the workload of a real working day.

“At first, we talked to the business users and captured their transactions without interrupting their workflows. We then analyzed these processes and transactions together with the users,” explained Guy-Dominic Gorek. “Based on this analysis, we created the test cases in IBM Rational Performance Tester with the IBM Rational Performance Tester Extension for SAP Solutions and the SAP GUI Scripting API, an interface SAP implemented to enable automated testing of SAP software.”

Following the creation of the basic test cases, andagon parameterized the test cases with generated test data. To efficiently test customized SAP functionality, andagon developed custom code to analyze and automate user interaction in great detail.

“Although most tests can be developed easily in the visual editor, it is a big advantage of IBM Rational Performance Tester that custom code can be developed in Java,” says Guy-Dominic Gorek. “This enables very flexible tests beyond SAP standard functionality. It also allows us to tweak important test cases to make sure that the tests will provide the insights defined in the test requirements.”

andagon implemented automated tests for up to 80 different SAP ERP transactions, making sure that the workload of a peak working day can be simulated accurately.

“Most performance tests include significantly fewer test cases and are not carried out to accurately simulate real workloads,” explains Guy-Dominic Gorek.

“But after a performance issue delayed the implementation of the new solution, the customer required a very realistic test environment. The performance test needed to prove the viability of the new and enhanced solution including a number of performance-critical customizations. To support these specific requirements, an extended test setup was chosen for this customer.”

The test configuration varied the number of virtual users throughout the day to mimic real business operations. Using IBM Rational Performance Tester, the team simulated up to 500 users. Some of the virtual users were creating a significantly higher workload than a real business user to increase the peak workload. During the test, a number of IBM Rational Performance Tester instances generated loads at different locations to mimic real network traffic and transaction volumes.

“We ran this performance test twice. In the first run, a label printer could not handle the workload. The application developers then optimized the system and when we did the final test, the solution delivered the required performance,” says Guy-Dominic Gorek.

“This complex SAP ERP testing project was a major challenge, but with IBM Rational Performance Tester we implemented and executed the test, proving that the customized solution can be safely deployed.”

Project achievements

At the wholesale logistics company, andagon was able to identify slow transactions that by themselves appeared to be insignificant yet could potentially cause the business process to freeze. Thanks to the comprehensive performance test, the company was able to identify and investigate the performance limitations of the equipment used in its shipping department and successfully eliminate the issues that would have caused severe business disruptions in production operations.

With detailed performance reports, the project team was able to provide the application developers with accurate feedback about bottlenecks and performance issues, which led to optimizations in customized SAP ERP functionality.

Initially, the system did not process the required workload in a timely manner to ensure smooth operations. Following the performance tests, application throughput was improved by 100 percent from 6,000 units per day to 12,000 units per day. As a result, the company was able to avoid the delay and expense of application re-engineering, and avoid the cost of new processor and memory resources, in the certainty that the new solution was able to handle peak workloads, comfortably.

Choosing andagon to improve software quality

Compared with classical test script programming, andagon leverages IBM Rational Performance Tester to implement test cases quickly and reduce the script development cycle by 70 percent. The efficient load test architecture of IBM Rational Performance Tester allows andagon to generate more system load from a smaller systems footprint. The flexible agents also provide cross-platform support for comprehensive software testing on different operating systems with a single solution.

Stefan Gogoll, CEO at andagon concludes: “The close partnership with IBM allows us exclusive, speedy and efficient access to the IBM Rational Performance Tester support team.

“The Rational Performance Tester support team is excellent, providing solutions quickly and actively helping us to execute successful performance test projects with our customers. With IBM, andagon provides detailed performance analysis and is able to improve application performance very efficiently.”

Products and services used

IBM products and services that were used in this case study.

Software:
Rational Performance Tester, Rational Performance Tester Extension for SAP Solutions

Service:
IBM-SAP Alliance

Legal Information

© Copyright IBM Corp. 2013 All Rights Reserved. IBM Deutschland GmbH D-71137 Ehningen ibm.com Produced in Germany IBM, the IBM logo, ibm.com, i5/OS, DB2, Domino, FlashCopy, Lotus, Notes, POWER, POWER4, POWER5, POWER6, System i, System x, and Tivoli are trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of other IBM trademarks is available on the Web at: http://www.ibm.com/legal/copytrade.shtml UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product or service names may be trademarks, or service marks of others. This brochure illustrates how IBM customers may be using IBM and/or IBM Business Partner technologies/services. Many factors have contributed to the results and benefits described. IBM does not guarantee comparable results. All information contained herein was provided by the featured customer/s and/or IBM Business Partner/s. IBM does not attest to its accuracy. All customer examples cited represent how some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication is for general guidance only. Photographs may show design models.