500 Internal Server Errors with Rational Team Concert Enterprise Extensions, Websphere Application Server, and IHS reverse proxy

Technote (troubleshooting)


Problem(Abstract)

When the IHS reverse proxy with IBM WebSphere Application Server 7 is used with Rational Team Concert (RTC) Enterprise Extensions' long running functions such as builds, scans, queries, or searches, 500 Internal Server Errors can occur if the serverIOTimeout value in the IHS plugin is set to the default of 60 seconds.

Symptom

Similar errors will show in the dialog windows and in the logs such as:

2013-11-14 12:40:41,199 [WebContainer : 12]  INFO com.ibm.team.jfs.app.http.client.JazzHttpClient     - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request: The target server failed to respond

or

CRJAZ0098I The "com.ibm.team.build.internal.common.ITeamBuildRequestService(/com/service/com.ibm.team.build.internal.common.ITeamBuildRequestService)" service fails.  The server returned the HTTP error 500 with error text "Internal Server Error".


Cause

Due to increased network traffic or long running functions, the WebSphere Application Server running the Rational Team Concert CCM application may not respond to the build agent, z/OS, or the client within the time required or vice versa. The IHS server being the intermediate handler will not process the request from the application server within this time and an ambiguous 500 Internal Server Error is returned.

Timeout values can be set higher for the build agent and the client to overcome individual problems, but if IHS is running as the reverse proxy, the 500 errors may still randomly occur.

There have been some defects resolved that do affect specific timeout issues. If after applying these, the 500 Internal Server Errors still occur, you should investigate the IHS plugin settings.


Environment

Enterprise Extension dependency builds, promotions, scans, context aware search, or any function that is long running through the IHS reverse proxy with WebSphere Application Server 7 with a default value of 60 as ServerIOTimeout can cause these errors.

Diagnosing the problem

The best method of diagnosing is to enable logging in the IHS plugin and monitor the access.log, error.log, and httpd_plugin.log for the response times and any 500 errors over a specific time frame.


To enable logging in the IHS http and plugin:

For specifics, see http://www-01.ibm.com/support/docview.wss?uid=swg21570830
  • Enable the http plugin trace
See http://www-01.ibm.com/support/docview.wss?uid=swg21174894 for more information

In the httpd.conf, this example shows mod_mpmstats and mod_status enabled:
    # Allow server status reports generated by mod_status,
    # with the URL of http://servername/server-status
    #
    <IfModule mod_status.c>
    <Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from specific_company.com
    </Location>
    </IfModule>

    # mod_mpmstats logs statistics about server activity to the
    # main
    # error log. No records are written while the server is idle.
    LoadModule mpmstats_module modules/debug/mod_mpmstats.so
    <IfModule mod_mpmstats.c>
    ReportInterval 60
    # Include details of active module in the statistics.
    TrackModules On
    </IfModule>


Also, in the httpd.conf, edit LogFormat to include %T to determine which requests consistently take the longest amount of time

LogFormat "%h %l %u %t TIME: %T  \"%r\" %>s %b" common

This edit does require a restart of the IHS server.


Lastly, for the IHS plugin trace, change the log level to trace. Setting the plugin trace in plugin-cfg.xml. does not need a server reboot to apply.

As an example,

    <Log LogLevel="Trace" Name="/opt/IBM/WebSphere/Plugins/logs/http_plugin.log"/>


Monitor the access.log and error.log and httpd_plugin.log for errors and the times.

Once the longest response time is determined, then the value for serverIOTimeout should be set to 2-3 times higher than the longest response time.

Resolving the problem

If 500 Internal Server errors from the application server are seen during long builds or functions when using Rational Team Concert Enterprise Extensions and the IHS reverse proxy, check the serverIOTimeout setting for the IHS plugin, plugin-cfg.xml.

In the plugin-cfg.xml, the value is listed as ServerIOTimeout=60 as below.


    <Server ConnectTimeout="5" ExtendedHandshake="false" MaxConnections="-1" Name="Server1" ServerIOTimeout="60" WaitForContinue="false">
    <Transport Hostname="Emeric" Port="9081" Protocol="http"/>
    <Transport Hostname="Arlin" Port="9444" Protocol="https">
    <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.kdb"/>
    <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/etc/plugin-key.sth"/>
    </Transport>


In WebSphere Application Server 8, the ServerIOTimeout default is 900 seconds.

Note: Setting to 0 indicates no timeout and could cause the IHS server to crash as incoming threads are not released until they are responded to. A valid value should be determined and set.

Related information

Recommended Values for WAS/IHS web server plugin config
IHS mod_stats
IHS mod_mpmstats
IHS Perforamnce Measurements

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

Rational Team Concert
Enterprise Extensions

Software version:

4.0, 4.0.5

Operating system(s):

Linux, z/OS

Reference #:

1663293

Modified date:

2014-01-29

Translate my page

Machine Translation

Content navigation