HTTP 400 Bad Request: server to server communication fails when system clocks are not synchronized
This technote provides information to help troubleshoot an issue that can occur when you execute an action which invokes communication (OAuth calls) between IBM Rational Team Concert and other IBM Rational Jazz based servers which results in an error, HTTP 400 Bad Request.
The problem can manifest itself in different ways depending on the use case.
- When adding a link to a project area on a different server, the action fails with a message:
Error fetching remote data
- When opening the 'users' page in IBM Rational Change and Configuration Management (CCM), IBM Rational Quality Manager (RQM) or IBM Rational Jazz Team Server (JTS), it displays:
The same error displays when viewing a user's profile.
- When navigating to the 'source control' > 'stream' page of any projects in the CCM application, it shows a message indicating that the user does not have a required license to run the operation.
- When running the repotools-<app> resumeindex command, it fails. The following error is found in the repotools-<app>_resumeindex.log:
com.ibm.team.repository.common.PermissionDeniedException: CRJAZ0097I The request for the URL "/ccm/indexing?action=resume" was denied. The status is "Forbidden".
System clocks on the server machines or on a proxy (such as IBM HTTP Server (IHS)) are not synchronized.
IBM Rational Jazz based products are running on different physical machines or a proxy will route requests between the application and the Jazz Team Server.
Diagnosing the problem
- Add the following entry in log4j.properties of the CCM application:
- Reload log settings by navigating to the following page and click on the "Reload Log Settings" button:
- Reproduce the problem.
- In the ccm.log (or jazz.log) file, the following exception is present:
DEBUG org.apache.http.wire - << "HTTP/1.1 400 Bad Request[EOL]"
DEBUG org.apache.http.wire - << "Date: Sun, 24 Feb 2013 09:06:34 GMT[EOL]"
DEBUG org.apache.http.wire - << "Server: IBM_HTTP_Server[EOL]"
DEBUG org.apache.http.wire - << "WWW-Authenticate: OAuth realm="Jazz%20Team%20Server", oauth_acceptable_timestamps="1361696516049-1361697116049", oauth_problem="timestamp_refused"[EOL]"
DEBUG org.apache.http.wire - << "Content-Length: 87[EOL]"
DEBUG org.apache.http.wire - << "Connection: close[EOL]"
DEBUG org.apache.http.wire - << "Content-Type: application/x-www-form-urlencoded;charset=UTF-8[EOL]"
DEBUG org.apache.http.wire - << "Content-Language: en-US[EOL]"
DEBUG org.apache.http.wire - << "[EOL]"
DEBUG org.apache.http.wire - << "oauth_acceptable_timestamps=1361696516049-1361697116049&oauth_problem=timestamp_refused"
Resolving the problem
- Adjust the time and time zone on all the Jazz-based server machines to be in sync. The time difference between the two servers must be less than 5 seconds.
- There is a property on the Advanced Properties page of the JTS server that controls how much request timestamps can differ from the time of the system processing the request. The property is "OAuth nonce tracking period (in seconds) ". The default is 5 seconds.
- To ensure that the problem will not reoccur, use a time synchronization daemon such as Network Time Protocol (NTP) server. This is a good way to ensure that system clocks on these machines stay in sync.
In IBM Rational Collaboration LifeCycle Management (CLM) version 4.0.x and above, "System Clock" diagnostic task is added to verify if an NTP server is configured.
CRJAZ2108W An NTP server is not configured to use for system clock verification. If unsynchronized clocks are on different application servers, problems might occur. To configure the NTP server to be used to verify the accuracy of the system clock, go to the Advanced Properties page, and for the NTP Server Address property, enter the address of an NTP server. To disable this diagnostic, click Disable.
Jazz and Rational Team Concert have an active community that can provide you with additional resources. Browse and contribute to the User forums, contribute to the Team Blog and review the Team wiki.
More support for:
Rational Team Concert
Software version: 2.0, 188.8.131.52, 184.108.40.206, 3.0, 3.0.1, 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199, 188.8.131.52, 4.0, 184.108.40.206, 220.127.116.11, 4.0.1
Operating system(s): AIX, Linux, Solaris, Windows
Reference #: 1391597
Modified date: 28 August 2013