IBM Support

MustGather: ClearQuest Web 8.0.x Server

Technote (FAQ)


Question

What steps can you follow to gather additional information for IBM Rational ClearQuest Support in resolving issues with the ClearQuest Web 8.0.x?

Answer




Default Installation Locations
These are the default installation directories for Rational home and Rational common components.

If you are using a non-default installation path, keep that in mind when reviewing these instructions.


RATIONAL_HOME refers to the location where Rational products are installed.

32-bit Windows: C:\Program Files\IBM\RationalSDLC

64-bit Windows: C:\Program Files (x86)\IBM\RationalSDLC

Linux/UNIX: /opt/IBM/RationalSDLC

RATIONAL_COMMON refers to the location where common components are installed.

32-bit Windows:  C:\Program Files\IBM\RationalSDLC\common

64-bit Windows:  C:\Program Files (x86)\IBM\RationalSDLC\common

Linux/UNIX: /opt/IBM/RationalSDLC/common

Note: On some UNIX operating systems, the path might have a lowercase "ibm" directory. You can also access the Rational home and common directories using these symbolic paths:

/opt/rational

/opt/rational/common





Basic Verification Information

Note: With steps 2 through 5, if you do not receive the expected verification that the software is running correctly, take a screenshot or save the message to file for IBM Support.

  1. Check that the system requirements are met. For more information, see technote 7023170.

  2. Verify that IBM HTTP Server is running by accessing this URL:
    http://<server_name>

    If the service is running correctly, you will see the welcome page for IBM HTTP Server Version 8.0.
  3. Verify that the ClearQuest Web Server is running by accessing this URL:
    http://<server_name>/TeamWeb/services/Team

    If the service is running correctly, you will see this welcome message:
    Hi there, this is a Web service!

  4. Verify that the ClearQuest Web application is running by accessing this URL:
    http://<server_name>:12080/cqweb

    If running, you will see the ClearQuest Web interface and login screen appear.

  5. Verify that the RPC services are running. Refer to the Monitoring the ClearQuest RPC servers topic in the ClearQuest product documentation for details.



Gathering Diagnostic Logs for ClearQuest Web on Microsoft Windows

  1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.

  2. If requested by a ClearQuest Support Engineer, refer to the ClearQuest Web Server Diagnostic Logging and Information section for instructions on increasing the ClearQuest Web Server logging levels.

  3. Stop the ClearQuest Web Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting the ClearQuest Web profile topic of the ClearQuest product documentation.

    Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.

  4. Enable diagnostic tracing. Download the file core_trace_for_web.zip from this technote. Extract the core_trace_for_web_enable.reg file onto your Web server. Double-click on the file to update the Windows registry.

    Note: See the end of this section for an alternative way to enable tracing.

  5. Restart all the web services except for full-text search.

  6. Reproduce the problem behavior with ClearQuest Web.

  7. Stop all web services again.

  8. Gather all the logs in these directories:
    • <RATIONAL_HOME>\ClearQuest\cqweb\cqwebprofile**\logs

      The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.
    • <WebSphere Install Directory>*\AppServer\logs
    • <HTTPServer Install Directory>*\logs

      Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later:
    • <RATIONAL_HOME>\cqweb
    • <RATIONAL_HOME>\cqweb\bin
    • <RATIONAL_HOME>\cqweb\cqwebprofile**

      If instructed by a ClearQuest Support Engineer, gather the ClearQuest Web Server MBean configuration data. You can find instructions in the ClearQuest Web Server Diagnostic Logging and Information section.

      Also collect the trace file, which should be located in the root C: drive directory.

      * With ClearQuest 8.0 and later versions, customers have to install WebSphere and the HTTPServer separately, so they have to be aware of these two directories to collect proper data for analysis.

      ** Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.

  9. Disable diagnostic tracing. Extract the core_trace_for_web_disable.reg file onto your Web server. Double-click on the file to update the Windows registry.

  10. Restart all necessary web services.

  11. If you adjusted the logging levels for the ClearQuest Web Server, you must restore the original settings. See the ClearQuest Web Server Diagnostic Logging and Information section for instructions.

  12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.

  13. You may now resume normal ClearQuest Web operation.


NOTE: If you are on ClearQuest 8.0.0.5 and later versions, you can enable, disable, and change ClearQuest Core trace setting dynamically without having to stop and restart the ClearQuest Web service. This is done by adding the Reset register key to the ClearQuest Core diagnostic keys. Here is an example of how to set it in the trace file:

[HKEY_USERS\.DEFAULT\Software\Rational Software\ClearQuest\Diagnostic]
"Trace"="CQRPC;THREAD;SQL;SESSION;HOOKS;THROW;EDIT;RESULTSET;API;PERL;DB_CONNECT;DBDESC;LICENSE"
"Report"="DIAG_FLAGS=-1;MESSAGE_INFO=0xF0B;OUTPUT_LIMIT=1000000"
"Output"="C:\\cq_core_trace_web.log"
"Reset"="True"

NOTE: This key location is only applicable when WebSphere Application Server (WAS) service is run as the "local system" account. If the WAS profile service is not running as "Local System" account, then the user must log onto the machine as the user that is running the web service, and add the key to the following location:
[HKEY_CURRENT_USER\Software\Rational Software\ClearQuest\Diagnostic]

To disable tracing (without restarting ClearQuest Web), change the value of at least Output to an empty string. The other values can remain or also be cleared; they will have no effect if there is no specification of where to send the trace output.

To stop dynamic changes, change the value of Reset to an empty string. Doing so is the same as if "Reset" does not exist.



Gathering Diagnostic Logs for ClearQuest Web on Linux or UNIX

  1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.

  2. If instructed by a ClearQuest Support Engineer, refer to the ClearQuest Web Server Diagnostic Logging and Information section for instructions on increasing the ClearQuest Web Server logging levels.

  3. Stop the ClearQuest Web Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting the ClearQuest Web profile topic of the ClearQuest product documentation.

    Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.

  4. Enable diagnostic tracing.

    Modify the file cqrpc.sh, which is found in this directory:

    <RATIONAL_HOME>/cqweb/bin

    You will find these trace details in the file:
    #CQ_DIAG_TRACE="CQRPC;THREAD;SQL;SESSION;HOOKS;THROW;EDIT;RESULTSET;API;PERL;DB_CONNECT;DBDESC;LICENSE"
    #CQ_DIAG_REPORT="DIAG_FLAGS=-1;MESSAGE_INFO=0xF0B;OUTPUT_LIMIT=1000000"
    #CQ_DIAG_OUTPUT="/opt/rational/common/CM/profiles/cmprofile/logs/server1/cq_trace.log"
    #export CQ_DIAG_TRACE
    #export CQ_DIAG_REPORT
    #export CQ_DIAG_OUTPUT


    Remove the comment prefixes (#) for these lines.
    Modify CQ_DIAG_OUTPUT to a directory and file where the trace file will be stored.

  5. Restart all the web services except for full-text search.

  6. Reproduce the problem behavior with ClearQuest Web.

  7. Stop all web services again.

  8. Gather all the logs in these directories:
    • <RATIONAL_HOME>/ClearQuest/cqweb/cqwebprofile**/logs

      The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.
    • <WebSphere Install Directory>*/AppServer/logs
    • <HTTPServer Install Directory>*/logs

      Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later:
    • <RATIONAL_HOME>/cqweb
    • <RATIONAL_HOME>/cqweb/bin
    • <RATIONAL_HOME>/cqweb/cqwebprofile**

      If instructed by a ClearQuest Support Engineer, gather the ClearQuest Web Server MBean configuration data. You can find instructions in the ClearQuest Web Server Diagnostic Logging and Information section.

      If you defined a directory other than the cqwebprofile logs directory for the diagnostic trace output, collect the logs from that directory.

      * With ClearQuest 8.0 and later versions, customers have to install WebSphere and the HTTPServer separately, so they have to be aware of these two directories to collect proper data for analysis.

      ** Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.

  9. Disable diagnostic tracing by commenting out (#) all six diagnostic lines in the cqrpc.sh file. Even if you added these lines to the file, make them comments rather than removing them from the file.

  10. Restart all necessary web services.

  11. If you adjusted the logging levels for the ClearQuest Web Server, you must restore the original settings. See the ClearQuest Web Server Diagnostic Logging and Information section for instructions.

  12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.
  13. You may now resume normal ClearQuest Web operation.



ClearQuest Web Server Diagnostic Logging and Information
These sections instruct you on how to prepare the diagnostic information for gathering along with the ClearQuest Web Server information. You should only follow these instructions at the request and guidance of a ClearQuest Support Engineer.


Collecting MBean attribute information

You can configure the ClearQuest Web Server using a variety of different variables called MBean attributes. When experiencing problems with the ClearQuest Web Server, it is worth verifying that these attributes are correctly established. Incorrect attribute values can cause problems with the starting or stability of the server processes.

You can send IBM Rational Support a summary of your custom MBean configuration settings with these instructions:

  1. Locate these directories:
    • Windows:
      <RATIONAL_HOME>\cqweb\cqwebprofile*\config\cells\dfltCell\nodes\dfltNode\servers\server1\stp
    • Linux/UNIX:
      <RATIONAL_HOME>/cqweb/cqwebprofile*/config/cells/dfltCell/nodes/dfltNode/servers/server1/stp

      Note: If the "stp" directory does not exist, you have not established any custom MBean attributes. You do not have to continue with gathering this particular data.

      * Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.

  2. These files store the current MBean attributes configured for the ClearQuest Web Server. Collect all of the files in this directory.

More information on proper MBean configuration

You can find more information about this in these documents:



Increasing ClearQuest Web Server logging levels

These are the steps for increasing the logging level detail on the ClearQuest Web Server. By changing the logging level, you are increasing the trace output and diagnostic information in the ClearQuest Web Server logs. With these instructions, you will also verify that file count and file size of these logs is sufficient for reproducing a problem during a trace.

Increasing the diagnostic detail can affect system performance. This should only be done for short periods of time. Be sure to restore logging to original settings when such detail is not needed.

  1. Browse to the ClearQuest Web Server administration console URL:
    http://<server_name>:<port>/ibm/console

    Note: The default port is 12060 for out-of-the-box ClearQuest Web Server. For a ClearQuest Web Server hosted on your own IBM WebSphere Application Server, the default port is 9060.

  2. Login to the administration console.

  3. Access the menu option Troubleshooting > Logs and Trace.

  4. Click on your server. The default for out-of-the-box ClearQuest Web Server is server1.

  5. Click on Change Log Detail Levels.

  6. Under the Runtime tab, expand the com.ibm.rational.* group.

  7. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > fine. Click OK at the end of the page.

  8. Click the Save link in the Messages area at the top of the page that opens.


Restoring standard ClearQuest Web Server logging levels

These are the steps for restoring the logging levels of the ClearQuest Web Server to their original state.

  1. Browse to this URL:
    http://<server_name>:<port>/ibm/console

    Note: The default port is 12060 for out-of-the-box ClearQuest Web Server. For a ClearQuest Web Server hosted on your own IBM WebSphere Application Server, the default port is 9060.

  2. Login to the administration console.

  3. Access the menu option Troubleshooting > Logs and Trace.

  4. Click on your server. The default for out-of-the-box ClearQuest Web Server is server1.

  5. Click on Change Log Detail Levels.

  6. Under the Runtime tab, expand the com.ibm.rational.* group.

  7. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > info. Click OK at the end of the page.

  8. Click the Save link in the Messages area at the top of the page that opens.

Document information

More support for: Rational ClearQuest
CM Server

Software version: 8.0, 8.0.0.1, 8.0.0.2, 8.0.0.3, 8.0.0.4, 8.0.0.5, 8.0.0.6, 8.0.0.7, 8.0.0.8, 8.0.0.9, 8.0.1, 8.0.1.1, 8.0.1.2

Operating system(s): AIX, Linux, Solaris, Windows

Reference #: 1638434

Modified date: 06 April 2015