MustGather: ClearQuest Web 7.0.x, 8.0.x and 9.0.x Server

If you are using a non-default installation path, keep that in mind when reviewing these instructions.

32-bit Windows: C:\Program Files\IBM\RationalSDLC

64-bit Windows: C:\Program Files (x86)\IBM\RationalSDLC

Linux/UNIX: /opt/IBM/RationalSDLC

RATIONAL_COMMON refers to the location where common components are installed.

32-bit Windows:  C:\Program Files\IBM\RationalSDLC\common

64-bit Windows:  C:\Program Files (x86)\IBM\RationalSDLC\common

Linux/UNIX: /opt/IBM/RationalSDLC/common

Note: On some UNIX operating systems, the path might have a lowercase "ibm" directory. You can also access the Rational home and common directories using these symbolic paths:

/opt/rational

/opt/rational/common

1. Check that the system requirements are met. For more information, see technote 7023170.
    
2. Verify that IBM HTTP Server is running by accessing this URL:
   http://<server_name>
   
   If the service is running correctly, you will see the welcome page for IBM HTTP Server Version 8.0.
3. Verify that the ClearQuest Web Server is running by accessing this URL:
   http://<server_name>/TeamWeb/services/Team
   
   If the service is running correctly, you will see this welcome message:
   Hi there, this is a Web service!
    
4. Verify that the ClearQuest Web application is running by accessing this URL:
   http://<server_name>:12080/cqweb
   
   If running, you will see the ClearQuest Web interface and login screen appear.
    
5. Verify that the RPC services are running. Refer to the Monitoring the ClearQuest RPC servers topic in the ClearQuest product documentation for details.

1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.
    
2. If requested by a ClearQuest Support Engineer, refer to the ClearQuest Web Server Diagnostic Logging and Information section for instructions on increasing the ClearQuest Web Server logging levels.
    
3. Stop the ClearQuest Web Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting the ClearQuest Web profile topic of the ClearQuest product documentation.
   
   Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.
    
4. Enable diagnostic tracing. Download the file core_trace_for_web.zip from this technote. Extract the core_trace_for_web_enable.reg file onto your Web server. Double-click on the file to update the Windows registry.
   
   Note: See the end of this section for an alternative way to enable tracing.
    
5. Restart all the web services except for full-text search.
    
6. Reproduce the problem behavior with ClearQuest Web.
    
7. Stop all web services again.
    
8. Gather all the logs in these directories:
   • <RATIONAL_HOME>\ClearQuest\cqweb\cqwebprofile**\logs
     
     The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.
   • <WebSphere Install Directory>*\AppServer\logs
   • <HTTPServer Install Directory>*\logs
     
     Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later:
   • <RATIONAL_HOME>\cqweb
   • <RATIONAL_HOME>\cqweb\bin
   • <RATIONAL_HOME>\cqweb\cqwebprofile**
     
     If instructed by a ClearQuest Support Engineer, gather the ClearQuest Web Server MBean configuration data. You can find instructions in the ClearQuest Web Server Diagnostic Logging and Information section.
     
     Also collect the trace file, which should be located in the root C: drive directory.
     
     * With ClearQuest 8.0 and later versions, customers have to install WebSphere and the HTTPServer separately, so they have to be aware of these two directories to collect proper data for analysis.
     
     ** Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.
      
9. Disable diagnostic tracing. Extract the core_trace_for_web_disable.reg file onto your Web server. Double-click on the file to update the Windows registry.
    
10. Restart all necessary web services.
     
11. If you adjusted the logging levels for the ClearQuest Web Server, you must restore the original settings. See the ClearQuest Web Server Diagnostic Logging and Information section for instructions.
     
12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.
     
13. You may now resume normal ClearQuest Web operation.

NOTE: If you are on ClearQuest 8.0.0.5 and later versions, you can enable, disable, and change ClearQuest Core trace setting dynamically without having to stop and restart the ClearQuest Web service. This is done by adding the Reset register key to the ClearQuest Core diagnostic keys. Here is an example of how to set it in the trace file:

[HKEY_USERS\.DEFAULT\Software\Rational Software\ClearQuest\Diagnostic]

1. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to full.
    
2. If instructed by a ClearQuest Support Engineer, refer to the ClearQuest Web Server Diagnostic Logging and Information section for instructions on increasing the ClearQuest Web Server logging levels.
    
3. Stop the ClearQuest Web Server and ClearQuest Web services. Refer to the Starting, stopping, and restarting the ClearQuest Web profile topic of the ClearQuest product documentation.
   
   Note: This will also stop the ClearCase Remote Client (CCRC) server and CM API operations if hosted on the same machine.
    
4. Enable diagnostic tracing.
   
   Modify the file cqrpc.sh, which is found in this directory:
   
   <RATIONAL_HOME>/cqweb/bin
   
   You will find these trace details in the file:
   #CQ_DIAG_TRACE="CQRPC;THREAD;SQL;SESSION;HOOKS;THROW;EDIT;RESULTSET;API;PERL;DB_CONNECT;DBDESC;LICENSE"
#CQ_DIAG_REPORT="DIAG_FLAGS=-1;MESSAGE_INFO=0xF0B;OUTPUT_LIMIT=1000000"
#CQ_DIAG_OUTPUT="/opt/rational/common/CM/profiles/cmprofile/logs/server1/cq_trace.log"
#export CQ_DIAG_TRACE
#export CQ_DIAG_REPORT
#export CQ_DIAG_OUTPUT
   
   Remove the comment prefixes (#) for these lines.
   Modify CQ_DIAG_OUTPUT to a directory and file where the trace file will be stored.
    
5. Restart all the web services except for full-text search.
    
6. Reproduce the problem behavior with ClearQuest Web.
    
7. Stop all web services again.
    
8. Gather all the logs in these directories:
   • <RATIONAL_HOME>/ClearQuest/cqweb/cqwebprofile**/logs
     
     The contents of this directory must include the ffdc directory. This directory contains the First Failure Data Capture logs.
   • <WebSphere Install Directory>*/AppServer/logs
   • <HTTPServer Install Directory>*/logs
     
     Inform the ClearQuest Support Engineer if core dump files (javacore*.txt) or heap files (*.phd) exist in these directories, as these might need to be gathered later:
   • <RATIONAL_HOME>/cqweb
   • <RATIONAL_HOME>/cqweb/bin
   • <RATIONAL_HOME>/cqweb/cqwebprofile**
     
     If instructed by a ClearQuest Support Engineer, gather the ClearQuest Web Server MBean configuration data. You can find instructions in the ClearQuest Web Server Diagnostic Logging and Information section.
     
     If you defined a directory other than the cqwebprofile logs directory for the diagnostic trace output, collect the logs from that directory.
     
     * With ClearQuest 8.0 and later versions, customers have to install WebSphere and the HTTPServer separately, so they have to be aware of these two directories to collect proper data for analysis.
     
     ** Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.
      
9. Disable diagnostic tracing by commenting out (#) all six diagnostic lines in the cqrpc.sh file. Even if you added these lines to the file, make them comments rather than removing them from the file.
    
10. Restart all necessary web services.
     
11. If you adjusted the logging levels for the ClearQuest Web Server, you must restore the original settings. See the ClearQuest Web Server Diagnostic Logging and Information section for instructions.
     
12. Login to ClearQuest Web as a Super User. Access the Site Administration > Site Configuration screen. Under the Application Options tab, set the Enable Diagnostics value to default.
13. You may now resume normal ClearQuest Web operation.

Collecting MBean attribute information

You can configure the ClearQuest Web Server using a variety of different variables called MBean attributes. When experiencing problems with the ClearQuest Web Server, it is worth verifying that these attributes are correctly established. Incorrect attribute values can cause problems with the starting or stability of the server processes.

You can send IBM Rational Support a summary of your custom MBean configuration settings with these instructions:

1. Locate these directories:
   • Windows:
     <RATIONAL_HOME>\cqweb\cqwebprofile*\config\cells\dfltCell\nodes\dfltNode\servers\server1\stp
   • Linux/UNIX:
     <RATIONAL_HOME>/cqweb/cqwebprofile*/config/cells/dfltCell/nodes/dfltNode/servers/server1/stp
     
     Note: If the "stp" directory does not exist, you have not established any custom MBean attributes. You do not have to continue with gathering this particular data.
     
     * Customers have the option of installing the ClearQuest web application into a new or deploy into an existing WebSphere Application Server profile. For the latter option the directory of cqwebprofile is going to be different from what is listed above. Refer to the Using WebSphere Application Server profiles section for more information.
      
2. These files store the current MBean attributes configured for the ClearQuest Web Server. Collect all of the files in this directory.

More information on proper MBean configuration

You can find more information about this in these documents:

• The Modifying available MBean attributes topic in the ClearQuest product documentation.

Increasing ClearQuest Web Server logging levels

These are the steps for increasing the logging level detail on the ClearQuest Web Server. By changing the logging level, you are increasing the trace output and diagnostic information in the ClearQuest Web Server logs. With these instructions, you will also verify that file count and file size of these logs is sufficient for reproducing a problem during a trace.

Increasing the diagnostic detail can affect system performance. This should only be done for short periods of time. Be sure to restore logging to original settings when such detail is not needed.

1. Browse to the ClearQuest Web Server administration console URL:
   http://<server_name>:<port>/ibm/console
   
   Note: The default port is 12060 for out-of-the-box ClearQuest Web Server. For a ClearQuest Web Server hosted on your own IBM WebSphere Application Server, the default port is 9060.
    
2. Login to the administration console.
    
3. Access the menu option Troubleshooting > Logs and Trace.
    
4. Click on your server. The default for out-of-the-box ClearQuest Web Server is server1.
    
5. Click on Change Log Detail Levels.
    
6. Under the Runtime tab, expand the com.ibm.rational.* group.
    
7. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > fine. Click OK at the end of the page.
    
8. Click the Save link in the Messages area at the top of the page that opens.

Restoring standard ClearQuest Web Server logging levels

These are the steps for restoring the logging levels of the ClearQuest Web Server to their original state.

1. Browse to this URL:
   http://<server_name>:<port>/ibm/console
   
   Note: The default port is 12060 for out-of-the-box ClearQuest Web Server. For a ClearQuest Web Server hosted on your own IBM WebSphere Application Server, the default port is 9060.
    
2. Login to the administration console.
    
3. Access the menu option Troubleshooting > Logs and Trace.
    
4. Click on your server. The default for out-of-the-box ClearQuest Web Server is server1.
    
5. Click on Change Log Detail Levels.
    
6. Under the Runtime tab, expand the com.ibm.rational.* group.
    
7. Click on the com.ibm.rational.stp.* sub-group. Select the context menu item Message and Trace Levels > info. Click OK at the end of the page.
    
8. Click the Save link in the Messages area at the top of the page that opens.