IBM Support

MustGather: ClearQuest Web 7.1.x Server

Question & Answer


Question

What steps can you follow to gather additional information for IBM Rational ClearQuest Support in resolving issues with the ClearQuest Web 7.1.x?

Answer

Table of Contents:



Diagnostic Automation with IBM Support Assistant



You can automate these data collection steps for Windows and UNIX/Linux for ClearQuest V7.1.x.
To do so, install IBM Support Assistant Lite for ClearQuest 7.1.x. You can find download and usage information about this tool in technote 1407905.

The tool provides you with these collector types for ClearQuest Web and CM Server diagnostics:
  • General Change Management Server Collector: Gathers information for general issues relating to Change Management (CM) Server. This includes CM Server configuration, IBM Installation Manager, as well as information related to embedded IBM WebSphere Application Server.
  • Server Configuration Collector: Gathers only CM Server configuration information.
  • Server Processes Trace Collector: Enables process tracing on the CM Server. You will be prompted to recreate the server issue found, and when completed, to continue with the data collection tool to disable trace settings and gather information.

It is recommended that you run these collectors with administrator privileges.



Default Installation Locations

These are the default installation directories for Rational home and Rational common components.

If you are using a non-default installation path, keep that in mind when reviewing these instructions.


RATIONAL_HOME refers to the location where Rational products are installed.

32-bit Windows: C:\Program Files\IBM\RationalSDLC

64-bit Windows: C:\Program Files (x86)\IBM\RationalSDLC

Linux/UNIX: /opt/IBM/RationalSDLC

RATIONAL_COMMON refers to the location where common components are installed.

32-bit Windows:  C:\Program Files\IBM\RationalSDLC\common

64-bit Windows:  C:\Program Files (x86)\IBM\RationalSDLC\common

Linux/UNIX: /opt/IBM/RationalSDLC/common

Note: On some UNIX operating systems, the path might have a lowercase "ibm" directory. You can also access the Rational home and common directories using these symbolic paths:

/opt/rational

/opt/rational/common




Basic Verification Information

Note: With steps 2 through 5, if you do not receive the expected verification that the software is running correctly, take a screenshot or save the message to file for IBM Support.
  1. Check that the system requirements are met. For more information, see technote 1294762.

  2. Verify that IBM HTTP Server is running by accessing this URL:
    http://<server_name>

    If the service is running correctly, you will see this welcome message:
    Rational Web Platform: If you see this page, it means that Rational Web Platform was installed and is running on this system.


  3. Verify that CM Server is running by accessing this URL:
    http://<server_name>/TeamWeb/services/Team

    If the service is running correctly, you will see this welcome message:
    Hi there, this is a Web service!

  4. Verify that the ClearQuest Web application is running by accessing this URL:
    http://<server_name>/cqweb

    If running, you will see the ClearQuest Web interface and login screen appear.

  5. Verify that the RPC services are running. Refer to the Checking RPC servers topic in the ClearQuest Information Center for details.



Gathering Diagnostic Logs for ClearQuest Web on Microsoft Windows

  1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.

  2. If requested by a ClearQuest Support Engineer, refer to the CM Server Diagnostic Logging and Information section for instructions on increasing the CM Server logging levels.

  3. Stop the CM Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting CM Server topic of the ClearQuest Information Center.

    Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.

    If installed, also disable the service for full-text searching.
    • IBM WebSphere Application Server <version> - cqsearchprofile

  4. Enable diagnostic tracing. Download the file core_trace_for_web.zip from this technote. Extract the core_trace_for_web_enable.reg file onto your Web server. Double-click on the file to update the Windows registry. Note: See the end of this section for an alternative way to enable tracing.

  5. Restart all the web services except for full-text search.

  6. Reproduce the problem behavior with ClearQuest Web.

  7. Stop all web services again.

  8. Gather all the logs in these directories:

    <RATIONAL_COMMON>\CM\profiles\cmprofile\logs

    The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.

    <RATIONAL_COMMON>\eWAS\logs
    <
    RATIONAL_COMMON>\IHS\logs

    Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later.

    <RATIONAL_COMMON>\CM
    <
    RATIONAL_COMMON>\CM\bin
    <
    RATIONAL_COMMON>\CM\profiles\cmprofile

    If instructed by a ClearQuest Support Engineer, gather the CM Server MBean configuration data. You can find instructions in the CM Server Diagnostic Logging and Information section.

    Also collect the trace file, which should be located in the root C: drive directory.

  9. Disable diagnostic tracing. Extract the core_trace_for_web_disable.reg file onto your Web server. Double-click on the file to update the Windows registry.

  10. Restart all necessary web services.

  11. If you adjusted the logging levels for CM Server, you must restore the original settings. See the CM Server Diagnostic Logging and Information section for instructions.

  12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.

  13. You may now resume normal ClearQuest Web operation.


Another way to enable tracing: This is another way that you can enable ClearQuest Core tracing without having to restart the ClearQuest Web service.

You can enable, disable, and change ClearQuest Core trace setting dynamically without having to stop ClearQuest Web. Adding the Reset register key to the ClearQuest Core diagnostic keys. Here is an example of how you set it in the trace file:

[HKEY_USERS\.DEFAULT\Software\Rational Software\ClearQuest\Diagnostic]


"Trace"="CQRPC;THREAD;SQL;SESSION;HOOKS;THROW;EDIT;RESULTSET;API;PERL;DB_CONNECT;DBDESC;LICENSE"
"Report"="DIAG_FLAGS=-1;MESSAGE_INFO=0xF0B;OUTPUT_LIMIT=1000000"
"Output"="C:\\cq_core_trace_web.log"
"Reset"="True"


NOTE: This key is only applicable when WebSphere Application Server (WAS) service is run as the "local system" account. If the WAS profile service is not running as "Local System" account, then the user "must" log onto the machine "as" the user that is running the web service, and add the key to the following location:
[HKEY_CURRENT_USER\Software\Rational Software\ClearQuest\Diagnostic]

To disable tracing, again without restarting ClearQuest Web, change the values of Trace,
Report, and Output to empty strings.

To stop dynamic changes, change the value of Reset to "False" or an empty string. Doing so is the same as if "Reset" does not exist.



Gathering Diagnostic Logs for ClearQuest Web on Linux or UNIX

  1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.

  2. If instructed by a ClearQuest Support Engineer, refer to the CM Server Diagnostic Logging and Information section for instructions on increasing the CM Server logging levels.

  3. Stop the CM Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting CM Server topic of the ClearQuest Information Center.

    Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.

  4. Enable diagnostic tracing.

    Modify the file cqrpc.sh, which is found in this directory:

    <RATIONAL_COMMON>/CM/bin

    If you are on ClearQuest 7.1.1 and later versions, you will find these trace details in the file:
    #CQ_DIAG_TRACE="CQRPC;THREAD;SQL;SESSION;HOOKS;THROW;EDIT;RESULTSET;API;PERL;DB_CONNECT;DBDESC;LICENSE"
    #CQ_DIAG_REPORT="
    DIAG_FLAGS=-1;MESSAGE_INFO=0xF0B;OUTPUT_LIMIT=1000000"
    #CQ_DIAG_OUTPUT="/opt/rational/common/CM/profiles/cmprofile/logs/server1/cq_trace.log"
    #export CQ_DIAG_TRACE
    #export CQ_DIAG_REPORT
    #export CQ_DIAG_OUTPUT


    Remove the comment prefixes (#) for these lines.

    If you are on a 7.1.0.x version, these lines might not exist in the file. If this is the case, locate these lines:
    echo Execing CQ RPC server with this command line:
    echo cqrpc $*
    exec cqrpc $*


    Directly before these lines, add the above trace details.

  5. Restart all the web services except for full-text search.

  6. Reproduce the problem behavior with ClearQuest Web.

  7. Stop all web services again.

  8. Gather all the logs in these directories:
    • <RATIONAL_COMMON>/CM/profiles/cmprofile/logs

      The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.
    • <RATIONAL_COMMON>/eWAS/logs
    • <RATIONAL_COMMON>/IHS/logs

      Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later:
    • <RATIONAL_COMMON>/CM
    • <RATIONAL_COMMON>/CM/bin
    • <RATIONAL_COMMON>/CM/profiles/cmprofile

      If instructed by a ClearQuest Support Engineer, gather the CM Server MBean configuration data. You can find instructions in the CM Server Diagnostic Logging and Information section.

      If you defined a directory other than the cmprofile logs directory for the diagnostic trace output, collect the logs from that directory.

  9. Disable diagnostic tracing by commenting out (#) all six diagnostic lines in the cqrpc.sh file. Even if you added these lines to the file, make them comments rather than removing them from the file.

  10. Restart all necessary web services.

  11. If you adjusted the logging levels for CM Server, you must restore the original settings. See the CM Server Diagnostic Logging and Information section for instructions.

  12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.

  13. You may now resume normal ClearQuest Web operation.



Change Management (CM) Server Diagnostic Logging and Information
In some instances, it is necessary to gather diagnostic information on CM Server as well as ClearQuest Web. These sections instruct you on how to prepare this diagnostic information for gathering along with the ClearQuest Web information. You should only follow these instructions at the request and guidance of a ClearQuest Support Engineer.



Collecting MBean attribute information

You can configure CM Server using a variety of different variables called MBean attributes. When experiencing problems with ClearQuest Web or CM Server, it is worth verifying that these attributes are correctly established. Incorrect attribute values can cause problems with the starting or stability of the server processes.

You can send IBM Rational Support a summary of your custom MBean configuration settings with these instructions:

  1. Locate these directories:
    • Windows 7.1.0.x:
      <RATIONAL_COMMON>\CM\profiles\cmprofile\config\cells\<host>CMProfileNode01Cell\nodes\<host>CMProfileNode01Cell\servers\server1\stp
    • Windows 7.1.1.x and 7.1.2.x:
      <RATIONAL_COMMON>\CM\profiles\cmprofile\config\cells\dfltCell\nodes\dfltNode\servers\server1\stp
    • Linux/UNIX 7.1.x:
      <RATIONAL_COMMON>/CM/profiles/cmprofile/config/cells/dfltCell/nodes/dfltNode/servers/server1/stp

      Note: If the "stp" directory does not exist, you have not established any custom MBean attributes. You do not have to continue with gathering this particular data.

  2. These files store the current MBean attributes configured for CM Server. Collect all of the files in this directory.

More information on proper MBean configuration

You can find more information about this in these documents:



Increasing CM Server logging levels

These are the steps for increasing the logging level detail on CM Server. By changing the logging level, you are increasing the trace output and diagnostic information in the CM Server logs. With these instructions, you will also verify that file count and file size of these logs is sufficient for reproducing a problem during a trace.

Increasing the diagnostic detail can affect system performance. This should only be done for short periods of time. Be sure to restore logging to original settings when such detail is not needed.


  1. Browse to the CM Server administration console URL:
    http://<server_name>:<port>/ibm/console

    Note: The default port is 12060 for out-of-the-box CM Server. For CM Server hosted on your own IBM WebSphere Application Server, the default port is 9060.

  2. Login to the administration console.

  3. If you are using ClearQuest 7.1.1.x and later versions, continue onto step 4. If you are using ClearQuest 7.1.0.x, you will need to follow steps a through j to assure proper logging.
    1. Access the menu option Troubleshooting > Logs and Trace.

    2. Click on your server. The default for out-of-the-box CM Server is server1.

    3. Click on Diagnostic Trace.

    4. Under the Runtime tab, change the Maximum File Size and Maximum Number of Historical Log Files values to 20. Click OK at the end of the page.

    5. Click the Save link in the Messages area at the top of the page that opens.

    6. Click on JVM Logs.

    7. Under the Configuration tab, change the Maximum Size and Maximum Number of Historical Log Files values to 20. Do this for both error log files, System.out and System.err. Click OK at the end of the page.

    8. Click the Save link in the Messages area at the top of the page that opens.

    9. You will need to restart the CM Server process for these changes to take affect.

    10. Once restarted, return to the CM Server administration console.

      Note: You do not have to reset the file size and number settings when you have completed your diagnostics. These are the suggested values for appropriate CM Server logging. These are now the default values starting in version 7.1.1.0.

  4. Access the menu option Troubleshooting > Logs and Trace.

  5. Click on your server. The default for out-of-the-box CM Server is server1.

  6. Click on Change Log Detail Levels.

  7. Under the Runtime tab, expand the com.ibm.rational.* group.

  8. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > fine. Click OK at the end of the page.

  9. Click the Save link in the Messages area at the top of the page that opens.


Restoring standard CM Server logging levels

These are the steps for restoring the logging levels of CM Server to their original state.


  1. Browse to this URL:
    http://<server_name>:<port>/ibm/console

    Note: The default port is 12060 for out-of-the-box CM Server. For CM Server hosted on your own IBM WebSphere Application Server, the default port is 9060.

  2. Login to the administration console.

  3. Access the menu option Troubleshooting > Logs and Trace.

  4. Click on your server. The default for out-of-the-box CM Server is server1.

  5. Click on Change Log Detail Levels.

  6. Under the Runtime tab, expand the com.ibm.rational.* group.

  7. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > info. Click OK at the end of the page.

  8. Click the Save link in the Messages area at the top of the page that opens.

Disclaimer

All source code and/or binaries attached to this document are referred to here as "the Program". IBM is not providing program services of any kind for the Program. IBM is providing the Program on an "AS IS" basis without warranty of any kind. IBM WILL NOT BE LIABLE FOR ANY ACTUAL, DIRECT, SPECIAL, INCIDENTAL, OR INDIRECT DAMAGES OR FOR ANY ECONOMIC CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS OR SAVINGS), EVEN IF IBM, OR ITS RESELLER, HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.


core_trace_for_web.zipcore_trace_for_web.zip

[{"Product":{"code":"SSSH5A","label":"Rational ClearQuest"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"CM Server","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"7.1;7.1.1;7.1.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSSH5A","label":"Rational ClearQuest"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"CM Server","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
16 June 2018

UID

swg21366509