Verizon Business Improves Software Development Governance, Quality, and Customer Satisfaction with the IBM Rational Software Development Platform

As one of the three operating units of Verizon Communications, Inc., Verizon Business is a leading provider of advanced communications and information technology (IT) solutions to large business and government customers worldwide. The unit’s Application Solutions group offers next generation e-government solutions, leveraging one of the world's largest Internet backbones and more than 100 high-speed fiber-optic networks. Application Solutions has developed a number of successful and specialized e-government application solutions for law enforcement, vehicle inspection and hunting and fishing licenses. The group’s Web-enabled Wildlife Data Systems have been implemented by several state governments and account for more than four million transactions per month on average.

These successes are due, in part, to a transformation of the group’s development process which enabled the company to address several challenges that hampered the team’s ability to consistently deliver quality solutions that met customer needs. “In the past, customer satisfaction was an issue because the quality of our deliveries did not always meet customer expectations or the standards which we regularly meet today. We had some late deliveries which affected our ability to win contract extensions. And, any successes that we did have were not easily repeatable, because we lacked an established process,” recalls Douglas Cook, Senior Quality Assurance Specialist at Verizon Business.

In addition to lacking a repeatable process, there were several other technical challenges that Verizon Business wanted to address. Cook continues, “At the time there were a number of different cultures with in our software development team. Everyone was working differently, using different methods and different tools – some were home grown utilities, some were freeware. Another problem was geographic separation; at the time our team was in multiple locations. There was no common development platform and collaboration was difficult. Defects were tracked in spreadsheets, and nobody was sure where the requirements were. It was, at times, a chaotic environment.”

Recognizing the need for more effective governance of its software and systems development, Verizon Business implemented the IBM Rational Software Development Platform, including integrated solutions for requirements management, visual modeling, software asset management, defect and change tracking, and automated testing. “We needed repeatable process and a unifying platform that we could use to develop applications, version source code, write test plans, manage defects -- the whole lifecycle – and IBM Rational provided everything we needed,” notes Cook.

Starting with Requirements

The Verizon Business team started by addressing one of their most pressing needs first. “One of our main challenges in being able to consistently deliver quality applications on-time was that we did not have a mechanism that enabled us to test against customer requirements. That is where we started,” says Cook. “In the past, we did not have an established way to elicit and manage customer requirements, so our development team would build what they thought the customer wanted.”

Today, when a customer proposal is accepted, the team creates a high level customer vision document that clearly defines what they will do contractually. Requirements analysts then use IBM Rational RequisitePro to elicit requirements and define business rules and use cases for the system the team will build. The vision document and requirements are all versioned with IBM Rational ClearCase, which the team uses throughout development for management and control of software development assets.

Once the requirements are established, the project manager meets with the project’s lead developer, tester, and requirements analyst to define the scope of the first build. Cook continues, “From that point, the designers access the requirements in Rational RequisitePro and begin creating a design document. In parallel, our testing team uses Rational RequisitePro to begin building test cases based upon those requirements. After the release and tests have been developed, we run the tests and then either return for another build or send that build out for user acceptance testing.”

To complement the textual description of use cases in Rational RequisitePro, the requirements team models use cases with IBM Rational Rose Developer for Java. The design team also uses Rational Rose Developer to develop an architecture of the system in the Unified Modeling Language (UML). Developers then leverage these models to code applications. “In the past, our designers would sometimes get too involved in the development. Now, we have a process; by separating design from coding the entire team is able collaborate more effectively. For example, as a tester if I only get business rules or use cases, my test cases cannot be as deep because the requirements are not as detailed as I need them. But if I have a design document and models, it tells me exactly how they are going to build it, and I can use that to develop better test cases,” says Cook.

He adds, “Rational Rose has provided our business with a number of advantages. It helps communicate the needs of the business to our technical team, and it enables the design, development and quality assurance teams to communicate using a common language. Rational Rose provides a common tool platform for all of our designers.”

The team links requirements in Rational RequisitePro to test cases in IBM Rational TestManager. Tight integration between the two products enables the Verizon Business development team to trace requirements throughout testing. “As we continue to grow as a company, traceability is becoming more important because, in essence, the purpose of a test case is to validate a requirement. With the ability to write test cases to specifically validate requirements, we can bring the whole picture together,” Cook notes.

All Assets in One Place

Prior to managing their artifacts in Rational ClearCase, the Digital Government Enterprises team used a disparate set of tools to manage and version their source code. As the team began to implement other IBM Rational solutions, it became clear that Rational ClearCase, as part of an end-to-end development solution, offered advantages over the piecemeal approach. “When we were using different tools, we never knew where everything was. With Rational ClearCase, when the configuration management team needs to build a product they only have to go to one place instead of three. That has really helped them and has really helped stabilize our builds,” says Cook. “We selected Rational ClearCase because one of our integration experts was very enthusiastic about it and we were implementing several other Rational solutions. It is excellent and it fits right in line with the other products. It has added a level of calm to our parallel development, because everyone has a better understanding of what is going on.”

Integration between Rational ClearCase and other solutions from the IBM Rational Software Development Platform provides a range of benefits to the Verizon Business team. “A key factor In choosing IBM Rational ClearCase was the ability to seamlessly integrate our version control system with our design, testing and change management tools. The out-of-the-box Unified Change Management (UCM) capabilities allowed us to quickly leverage the power of Rational ClearCase and get up and running with a unified change and configuration management system,” says Cook. “The integration between Rational ClearCase and Rational ClearQuest is enabling our development leads a continuous flow of information and an increased level of communication between functional teams for Verizon Business. Seamless integration also allows the function groups to maintain a higher level of visibility with respect to changes in our applications.”


Improved Visibility and Coordination

Effective software development governance depends on the ability to view and understand how each project is progressing. Verizon Business is using IBM Rational ClearQuest to improve visibility and coordination of projects. As a comprehensive defect tracking and change management system, Rational ClearQuest benefits the entire development team, including its on-site developers, and additional developers at two remote sites. “Visibility into projects has been a real benefit, because we have one place where a project manager can go and see the state of the project, including all the defects and their status. Our off-shore developers access it through a Web interface and they can see exactly what defects they are responsible for. As a result, when we have a conference call we can all speak on a common level because we all have access to the same information. Rational ClearQuest has brought our team closer together, because now everyone is on the same page on how we are going to manage defects.”

Previously the team managed defects manually using a spreadsheet, but as the team grew and projects became more complex, this approach was no longer workable. Rational ClearQuest meets the team’s immediate requirements, and provides the flexibility to adapt as the team’s needs change. “Everyone on the team is using Rational ClearQuest – the requirements team, developers, testers, and project managers. Everyone can add notes, and everyone can see the status of defects. We run a substantial number of reports using Rational ClearQuest to examine a range of metrics. We recently set up a new schema, and we’re having great success with it. Down the road, I see Rational ClearQuest moving even closer into the core project group, with less of QA focus. It will be our ‘team tool.’”, Cook explains.

Test Automation Enables Cost and Time Savings

When a software application reaches the SQA team for testing, Cook and his colleagues use IBM Rational Robot to create automated functional and regression tests. The group is working on developing basic regression tests scripts for each project. “We want to get some horizontal coverage first, and then create more advanced scripts as we go,” says Cook.

On one project the team has already created a comprehensive test suite and demonstrated the benefits of test automation. The application under test was a government fish and game Web site that sold more than 200 different variations of hunting and fishing licenses. “Because, each license had its own rules regarding residency and age limitations, thoroughly testing each item and all of its conditions was very labor intensive. There was no way we could manually test each of them for every build,” says Cook.

Cook and another tester built an automated test suite with Rational Robot. Factoring in labor costs, Cook estimates the effort’s investment cost at $6,400. Over time, the development team released 24 builds of the application to SQA; on each build test automation saved an average of $3,200. Cook’s final estimate for cost savings from automation on this one project was $76,800. “Manual testing this system would have been extremely labor intensive. With Rational Robot, for each build we would run our automated tests. They would test 98% of the items that the site sold, verifying for a range of conditions that the site offered the correct items for sale. It was simple, but exceptionally powerful.”

Successful User Acceptance Tests Increased

Cook notes that delivering quality applications has always been a priority at Verizon Business. However, when the team lacked a repeatable process, a unified development platform, and an effective governance approach they struggled to consistently meet the organization’s quality standards. He explains, “When SQA completes testing, we promote the code for user acceptance testing. In some cases, the user acceptance tests found new defects and other defects that had been reported as fixed. At one time, 65% of what we sent out was returned as failed in user acceptance testing. It was a huge problem.”

Since adopting the IBM Rational Software Development Platform that number has dropped. “We use Rational TestManager to gather our test cases and designs and sent them to the user acceptance testing team. We use Rational Robot scripts to provide vertical as well as horizontal testing. We use Rational ClearQuest to communicate our findings – and steps needed to reproduce the defect – to developers. And we use Rational ClearCase to stabilize our build process. As a result, we increased the quality of our builds, and on one recent project 98% of our user acceptance tests passed.”

CMMi, RUP, and New Business Opportunities

Although the Digital Government Enterprises group currently works mainly with state and local governments, the group’s target market is expanding to include the federal government and other agencies, many of which require their contractors to attain Capability Maturity Model Integration® (CMMI®) Level 3. The group is relying on the IBM Rational Software Development Platform to support its efforts around this maturity framework. “We just had a Standard CMMI Appraisal Method for Process Improvement (SCAMPI) assessment to determine our CMMi level. There are many things we must show for compliance, including artifacts and processes. As we move forward we will rely on Rational ClearCase and the Rational platform to help manage all of that,” notes Cook.

A key element of the group’s effort in this area is the adoption of IBM Rational Unified Process, or RUP. Verizon Business has customized RUP to meet the specific needs of the organization. The tailored methodology provides Verizon Business with a repeatable process that continues to improve over time. “Before RUP, when we were successful on a project it was often due to personal heroics. The process wasn’t trackable, it wasn’t measurable, and it definitely wasn’t repeatable. Even after we were all began using common tools, some people were still doing things their own way. Going from that kind of shop to a process shop requires a change in mindset. But gradually we got everyone on board with RUP, and now we are thinking about the future and how we will progress as a shop in the company,” says Cook. “In the past, we didn’t have a consistent methodology; now we do, and we are continuing to improve it.”


Connecting Business with Development

As a provider of communications and IT solutions, Verizon Business is always looking to create new revenue generating products and services. One challenge in this effort is connecting the program managers, line of business management, and customers that have ideas for new products with the development teams that will build them. Cook reports that this connection has been significantly strengthened through the use of RUP and the IBM Rational Software Development Platform, and that is reflected in improved customer satisfaction. “Customers are much more satisfied with our work. They see we are using a different process, and the chaos is gone. We are no longer shifting resource around, struggling to meet deadlines. In fact, we recently won a five-year contract extension, as a result of our customer’s improved opinion of us,” says Cook.

There have been other advantages as well – less apparent from outside the company, but vital to the group’s continued success. “Our teams are seeing that if you follow the process you need fewer iterations; and people see that when things go smoothly everything goes faster. Although we are just starting to capture numbers on productivity, I believe we have seen a substantial increase. By providing our development teams with a common platform from which to analyze, design, develop, and test, IBM Rational solutions have created opportunities to improve the quality of our software as well as our organization,” Cook adds.