IBM Support

MustGather: Performance, Hang, or High CPU issues with CLM applications

Troubleshooting


Problem

This document will assist you in collecting the data necessary to diagnose and resolve performance, hang, or high CPU issues with the IBM Rational Collaborative Lifecycle Management (CLM) products (includes Rational Team Concert (RTC), IBM Rational Quality Manager (RQM) and IBM Rational DOORS Next Generation (DNG) formerly Rational Requirements Composer (RRC)).

Symptom

Performance, hang, or high CPU issues with the CLM products may occur when installed in a distributed environment. Each issue may contribute to a variety of symptoms and behavioral deficiencies

Cause


This MustGather will assist you in collecting the data necessary to help you diagnose and resolve the issue. If you are unable to determine the root cause using the information collected, you should open a case with IBM Support for further investigation providing the data collected.

Resolving The Problem


You can use the IBM Support Assistant Lite (ISA Lite) Data Collector tool to quickly collect diagnostic files, such as log files, configuration files or to run traces. This tool is bundled with CLM 4.x and above. ISA Lite collects information about your Jazz Team Server environment and stores the information in a .zip archive file. If you have a need to open a case with IBM Support for further assistance, you can send the archive file with the data collection so that they can help diagnose and fix problems.


The information below should be gathered in addition to the normal information and log gathering done by ISA Lite.

Note: Further information to help troubleshoot performance issues with your CLM products is also available in the Performance troubleshooting section of the Deployment wiki on jazz.net.

 


 

 

Business Impact

Includes:

  • What affect is this having on the business
  • Is this a production or test environment
  • How many users are affected
     

 

Unexpected Behavior

Details include:

  • Problem description
  • Steps to recreate
  • Were there any recent changes made to the environment
  • Were new users able to log into any CLM applications?
  • What behavior did the logged-in users encounter?
  • Any errors or exceptions logged at the time the incident occurred

 

Topology

Description of the Topology includes:

  • Is this a stand alone or distributed environment?
  • The version of the CLM applications including all applied iFixes and hot fixes
  • Are the CLM applications deployed on Apache Tomcat or IBM WebSphere

    If WebSphere, provide the output of:

    <WebSphere_Install_Root>\bin\versionInfo -maintenancePackages

    and indicate if WebSphere is installed as 64bit or 32 bit

     
  • The Operating System the CLM applications are installed on
    • Specify 64- or 32- bit
    • Include the number of CPU's
    • How much memory is available
    • Available disk space
    • Size of indices on disk, for example, conf\<app>\indices
  • The Database vendor and version being used.

    If multiple database instances are being used, provide details on which CLM application is using which database instance and indicate if the database is installed as 32bit or 64bit
  • JVM params provided at server startup (heap sizes and config cache tuning) .  If ISA tool is used, this information is already available.
Javacore Log Files and Application Logs


Due to the distributed and interconnected nature of the CLM applications we will need at minimum data from the JTS server and the affected CLM application which are collected at the time the incident occurs. If possible, providing data from all JVMs would be preferred

  • Enable verbose gc and javacore creation
    • Websphere instructions are located here
    • Liberty Profile javacore collection instructions are located here
    • Tomcat instructions:
      • Add the following two lines to the server.startup script or to the Java Options in the Tomcat Control Panel if running as a Windows Service (ensure that the - character is present at the beginning)
        -verbose:gc
        -Xdump:java:events=user
      • Restart the Application Server for the changes to take effect.
  • Include all the CLM logs including ETL logs (located in

    WebSphere location:
    <WebSphere_Install_Root>\profiles\(profile_name)\logs directory
    Tomcat location: <CLM_Install_Root>\server\logs directory)[[
     
  • If CLM is running on WebSphere, provide the WebSphere Performance, hang, or high CPU mustgathers for the operating system CLM is installed on (in addition to those listed above)

 

Additional Information
  • Gather a screenshot of the following page from the web UI and save the content as Full HTML. Ensure you gather a screenshot for each application (example: jts, ccm, jazz, and others). For DNG, you may have to add the JVM parameter,  -Dcom.ibm.team.repository.service.internal.vvc.enableCounters=true  and restart the server prior to navigating to this page.  https://<hostname>/<context root>/service/com.ibm.team.repository.service.internal.counters.ICounterContentService
  • Enable queryStats (Use Chrome!):
    - Open /rm/admin#action=com.ibm.team.repository.admin.configureAdvanced
    - Find the entry "Enable repodebug service" and set the value to "true"
    - Open /rm/repodebug/database/queryStats and select start
    - Perform one of the operations that is slow
    - Refresh /rm/repodebug/database/queryStats, save the entire page, and submit in the PMR
    - Disable "Enable repodebug service" (set to false)
  • For DOORS Next Generation, it is also useful for support to know the following:  
    • # of projects
    • Configuration Management enabled for projects?
    • Estimated # of artifacts per project (does not need to be precise!)
    • Value for "JDBC Connection Pool Size" (under ConnectionPoolService heading) and "Mediator Pool Size" in RM Advanced Properties
    • Estimated # of concurrent users
    • If Windows, check the value of -x[no]compressedrefs in server startup
    • If Windows, check value for Jena File Access Mode (and Default) in RM Advanced Properties

Leverage the Jazz Community

The Jazz based products have an active community that can provide you with additional resources. Browse and contribute to the User forums, contribute to the Team Blog and review the Team wiki.

Review technote 1319600: Leveraging the Jazz Community on Jazz.net for details.

 

 


Cross reference information
Product Component Platform Version Edition
Rational Quality Manager Linux, Windows All Versions
Rational Team Concert AIX, Linux, Windows All Versions

Document information

More support for: Rational DOORS Next Generation

Software version: All Versions

Operating system(s): Linux, Windows

Reference #: 1607533

Modified date: 13 November 2018