How do you diagnose IBM Rational Change errors?
This paper is intended to help troubleshoot problems with the IBM Rational Change tool. It is divided up into 2 sections: First there is a general introduction to the required processes and what happens when the IBM Rational Change User or Admin session starts. Then there is a list of commonly seen login/startup problems.
This document assumes a working knowledge and installation of IBM Rational Synergy on top of which IBM Rational Change will be running. This bulletin is also based on the corresponding “Diagnosing ccm start problems ”, and should be referenced for any particular IBM Rational Synergy session issues.
In order to login to a IBM Rational Change installation either as a User or an Admin, several different IBM Rational Synergy and IBM Rational Change processes need to be running properly. In order to clarify this, let’s first introduce the main processes involved in logging into IBM Rational Change:
- The Jetty Server- This is the main IBM Rational Change server process
- The IBM Rational Directory Server (RDS) - The default user authentication and user preference server. Also known as Telelogic Directory Server(TDS) or LDAP server.
- IBM Rational License Server/ Telelogic License Server (RLS/TLS) - The license server as used by Telelogic Change 4.4 and newer. Also known as the FlexLM Server
- The IBM Rational Synergy server and related processes- The backend database server that IBM Rational Change is built upon
The IBM Rational Change product is built upon and requires the IBM Rational Synergy to be up and running properly in order for IBM Rational Change to function as expected. IBM Rational Change is a web-based front end to the IBM Rational Synergy database which allows for controlled access to Change Requests and Task information.
In order for IBM Rational Change to access the database(s) appropriately, the Jetty server must be started, as this is the server process which allows access to the tool. On Windows, this will be noted as a "IBM Rational Change <port>" line-item in the services dialogue which can be stopped and started with a mouse-click. While on a Unix installations the stop/start routine will require the <cs_home>/cs_app/csctl.sh script to be run with the appropriate arguments.
Once the Jetty server has been started, the login page should be accessible. When a login attempt is made, the Rational Directory Server is contacted for user authentication. After an initial install or machine reboot, the Rational Directory Server will need to be started to allow the Admin user to login and begin configuring the tool via the Admin dialogues. This is the default setting, Though OS authentication can be configured which would then bypass the Rational Directory Server. The Rational License Server should also be started, though no license calls will be made when initially logging in as an Admin.
General Failure Troubleshooting Tips:
Most errors encountered with IBM Rational Change are due to failed backend sessions. The backend sessions are extremely vital to the operation of IBM Rational Change. Without the backend sessions there is absolutely no communication to the databases from the IBM Rational Change server. Without being able to communicate with the databases, no process packages can be seen/installed, no user information can be seen/imported/modified, and most importantly no users can login at the User role level to submit/review/ or otherwise work with CR data.
The following are some general items to check to help get the backend session(s) running properly:
- Ensure that the backend session user (most likely csuser as documented in the installation guide) has been added with the role "no_privs" to each database you have added to IBM Rational Change.
- Ensure the backend session user is a valid OS level user on the host machine or host domain. The backend session user must exist in the Domain or OS level in order to allow it to start valid sessions.
- Ensure that the backend user is defined with the correct os level password within IBM Rational Change. This is done via the Edit CM Session User button from the Server tab in the Admin GUI.
- Ensure that the backend user can start a normal CM session from the IBM Rational Change server to each of the databases IBM Rational Change is installed against. ie. ccm start –d <db_path> -nogui
- Ensure that both the hosts defined and the databases added to this IBM Rational Change installation via the Server tab are selected as "Enabled". This is done from the Server tab in the Admin GUI. Note that this is an item to check only once the first four items are validated. If any backend session failures occur, they can and will cause the database and/or host to become disabled.
If the above 5 steps do not help alleviate the issue, the next step is to check the appropriate log files. Specifically, the event.log in "<cs_home>/cs_app/webapps/synergy/logs/" (or as accessible from the Debug/Logging Tab in the Admin login), Prior to release Change 4.7 the event.log was called pt.log and available in "<cs_home>/cs_app/webapps/WEB-INF/wsconfig/tmpdir/".
There is also the backend session user’s ccm_ui.log and ccm_eng.log files can be located in a few possible paths:
- The backend session user’s $HOME directory on Unix servers,
- In $CCM_HOME\log\web_sessions\<backend_userid>\ on Windows Servers and Unix servers utilizing ESD.
The combination of data found in the the pt.log file, ccm_ui.log and ccm_eng.log will nearly always point to the particular issue causing the backend session failures. A thorough review of these files will be necessary if failures continue. Some of the more frequent messages found in these log files are briefly explained in the sections to follow.
Errors seen by users on login:
• Unable to authenticate user/password pair: Most often this is simply due to a mis-match of the username and password indicating either an incorrect username or password was entered. This can, however, also indicate an inability to authenticate if OS authentication has been configured. Note that in IBM Rational Change 5.2 and Telelogic Change 5.1, 5.0 the admin user is called "Admin" and the password is set at installtion time. For Synergy Change 4.7 to 4.4 the admin user is called "ChangeAdmin", with password= ‘password‘ is pre defined in the Telelogic Directory Server and should be used when first logging in.
• Synergy sessions are not available. Contact your administrator.: This can indicate that Rational Change cannot start the Rational Synergy back end sessions. Check the event.log for to conform if back end sessions can be started. Confirm that the CM session user is defined correctly and the password entered is the correct password for this user.
• Failed to Load User Details: This message is typically encountered when the backend session is unable to connect to the database and/or directory server. To resolve this issue, a close review of the event.log and backend session user’s ccm_ui.log and ccm_eng.log will point to the problem. Often times this will reveal an issue which can be resolved by using “Diagnosing ccm start problems ”.
• Database down for maintenance: Often this message indicates that the database the user is attempting to log into is currently protected or disabled. Typically this issue can be resolved by unprotecting the database via IBM Rational Synergy and then re-enabling the database in the Admin ? Server tab in IBM Rational Change. Unprotecting a database is documented in the IBM Rational Synergy Administration guide.
• Invalid Login Role: This is most commonly seen when OS Authentication is in use but has not been completely configured. Refer to the OS Authentication Settings section in the document below for further details on the changes required. This can also indicate the user does not have sufficient database privileges to allow for a Login to IBM Rational Change using the role noted in the error, Check the IBM Rational Synergy users attribute in the database to verify the user id is set with the appropriate permission.
• Blank Screen: A pop-up blocker may have interfered with the proper loading of the main IBM Rational Change page. Try logging in again, but holding the ‘Shift’ key when clicking the login button to temporarily disable pop-up blocking. If this succeeds, then you will need to add the URL of the IBM Rational Change server to your Trusted sites list within your browser.
• No login page seen (404 Not Found/ 503 Service unavailable): This can be due to a number of potential culprits; specifically failed backend sessions, a simple timing issue (ie jetty taking a while to actually start), or the Jetty server not starting properly. Backend sessions are discussed above. Restarting the jetty server and the Directory server should help. Further troubleshooting would require reviewing the jetty log files as found in <cs_home>\cs_app\logs
Errors encountered by users after login:
• Failed to Load the template: Commonly due to failed backend sessions, this can also be seen at times when a conflict between the lifecycle process package and a custom control package is encountered or if no process package is installed. Removing any custom packages and testing with just the process package in place should indicate if a custom package is the culprit. Otherwise, begin backend session debugging as discussed above.
• License not available: This indicates the user can not obtain a license as requested by the tool to the license server. Ensure you are running at the latest patch levels, as some license issues which generate this error have been resolved in patch releases. Otherwise, review your license file in conjunction with the license.log for any indications of failure or if you are simply running low on available licenses and need to increase the number. Most license issues will require Technical Support’s Assistance to resolve.
• No users or packages showing in Admin dialogs: This is most commonly due to failed backend sessions. Use the steps and guidelines at the beginning of the document to locate and resolve any backend session issues. Once resolved, the packages list will display properly and user data will again be visible. If backend sessions are indeed up and running, this behavior can also indicate that the logged in user does not have the appropriate ccm_admin or pt_admin database privileges to administer the users or install packages. Removing a newly added database can often determine if this is indeed the issue.
• Various users lost shared data (queries and reports): This is indicative of corrupt Directory Server data. In order to restore the shared reports etc then you need to unpack the zip files in the $VDE_HOME/vde/backup directory and unpack this into the /vde/data directory, overwriting the current contents. It is recommended that you stop the VDE process via VDE_HOME/vde/vde.sh stop before attempting this and then bring the service/process. . More information on restoring Directory Server backup data can be found in the Synergy Support FAQs.
Log file entries and their common meanings/solutions:
• Failed to create CommandInterfaceESD: This indicates that the Engine Startup Daemon(ESD) is not configured properly. Refer to the ESD Settings section as well as Setting up ESD in CM Synergy.
• WARN: ESD communication error: Response command started with illegal '-1. Expected '!'. : This error can indicate that the backend session user does not have write permissions to the $CCM_HOME/log/web_sessions/ directory and as such can not create or write to the ccm_ui.log file in this directory. Further examination of the ccm_esd_<hostname>.log file will confirm if this is indeed a permission's issue. Setting the $CCM_HOME/log/web_sessions/ directory to unix 777 permission's will typically resolve this issue.
• Could not start a new session: Session capacity reached: When seen alone (ie. with no other session errors prior in the log files) this indicates that the maximum number of database sessions has been reached and the IBM Rational Change tool can not create another backend session to support the user attempting to login. To alleviate this issue, you will need to adjust the Host/database session settings in the Admin login > Server tab. You can reference the Session Settings Chart in this document below as a guideline for adjustment.
• Serious: Rational Change license not available: As before, this indicates the user can not obtain a license as requested by the tool to the license server. Ensure you are running at the latest patch levels, as some license issues which generate this error have been resolved in patch releases. Otherwise, review your license file in conjunction with the license.log for any indications of failure or if you are simply running low on available licenses and need to increase the number. Most license issues will require Technical Support’s Assistance to resolve.
• Invalid login role: This indicates the user does not have sufficient database privileges to allow for a Login to IBM Rational Change using the role noted in the error. This can also indicate that OS Authentication is in use but has not been completely configured and will require you to edit the users.cfg and/or <db_name>_users.cfg files to add the appropriate mapping for the user in question as explained in the OS Authentication section below.
• Stack trace: java.lang.NullPointerException
at com.continuus.websynergy.cmapi.CcmSessionInfo.Start (CcmSessionInfo.java:222):
ccmAPIException: Could not start ccm: null:
Failed to add a session on database:
Failed to add a session to the pool:
These errors are often seen together and are all very dependant on other data points to resolve. They specifically indicate that IBM Rational Change can not start a valid backend session. Begin troubleshooting this as noted in the general tips above, paying special attention to the errors encountered in the ccm_ui.log file. This error can be due to simple password problems with the backend session user, Engine Startup Daemon mis-configurations, patch mis-matches, or if the database has been left in a protected state, amongst other causes. The backend session user’s log files will nearly always point to the particular cause.
• Failed to look up ccm users attribute: This error, when seen in the log file, is often indicative of user administration errors being seen in the Admin interface. When encountered alone, this means that the user does not have the appropriate pt_admin or ccm_admin database privileges as required by the tool for administrative tasks. When seen after other session errors, this is simply a side effect of not being able to start a backend session.
• Stack trace: EOFException(java.net.SocketException: Broken pipe): This is an error which can typically be ignored. Most often this is seen in the log file when a user has navigated away from a page in IBM Rational Change before it had completely loaded. This can. However, also be encountered if your network has become unstable. If you are not experiencing other network issues outside of IBM Rational Change, then you can safely ignore this message.
• netscape.ldap.LDAPException: error result 32: This usually indicates some sort of corruption in one of the data files from the Directory server’s \vde\data directory. The solution is to revert to a backup of the Directory Derver data file as found in the \vde\backup\ directory. Depending on how long the error has been presenting itself, you may need to restore from a backup a few days old. More information on restoring Directory Server backup data can be found in the IBM Rational Synergy technical papers.
• rexecException: Socket Create Error: Connection refused: This error typically indicates a problem with rexec. Often times this is simply due to rexec not enabled on the host machine. This may very well be a legitimate setup if you intend to use Engine Startup Daemon(ESD). As such, either ensure rexec is indeed running, or continue on below to ensure Engine Startup Daemon(ESD) is setup and configured properly.
• Failed to look up ccm users attribute: Session Error: Failed to read the CM users and roles. Ensure the 'raw_users' database parameter is configured in the 'ptcli.cfg' file.: This error is seen when the $CCM_HOME/etc/ptcli.cfg file does not contain the ‘raw_users’ config entries, or if the ptcli.cfg file is not readable by the backend session user as defined in Change. To resolve this, ensure the following line is located within the ptcli.cfg file on a single line and open permisisons to the file so the backend session user is able to read it:
Session Settings Guidelines:
The following table gives some general recommendations for configuring IBM Rational Change backend sessions on a per database level. For the best results, start with these figures and refine them through experimentation. These settings would be changed from the Admin login --> Server tab.
Active Users Min/Max Sessions Session Ratio
1-15 2-5 4
15-50 4-10 5
50-100 6-13 8
100-200 8-20 10
200+ 10-? 10-20
As an example, this chart indicates for 1-15 Active users, your minimum sessions settings for a single database should be 2, your maximum 5, and the session ratio be set at 4 (ie 4 users per session). Your host settings should always be equal to or greater than the maximum number of total database sessions.
Engine Startup Daemon (ESD) Settings:
By default the ENGINE_DAEMON is not set to true in the pt.cfg file, which means IBM Rational Change is using r* protocols to start the backend CM sessions on both systems local and remote after an initial installation. If the r* protocols are disabled/commented (with a # character), and it is your intention to use Engine Startup Daemon(ESD), then a process needs to be running on the IBM Rational Synergy host machines and IBM Rational Change must also be configured to use Engine Startup Daemon(ESD). Amongst other things, you will need to:
- Change the ENGINE_DAEMON flag in the <cs_home>/cs_app/webapps/synergy/WEB-INF/wsconfig/pt.cfg to TRUE.
- Edit $CCM_HOME/etc/esd.adr and add a line with the host name: port information for the CM host.
- Add the entry engine_daemon=TRUE to the [Options] section of the backend user's $HOME/.ccm.ini file
Recall that Engine Startup Daemon(ESD) is used for situations where you require a fixed port for IBM Rational Synergy sessions to use in starting engines (as in firewall situations). Further information on specific Engine Startup Daemon(ESD) setup and settings can be found in Setting up ESD in CM Synergy and Configuring Telelogic Change to run with ESD or in a firewall environment with ESD.
OS Authentication Settings:
After the installation of IBM Rational Change, there are a couple of additional steps that need to be taken to configure IBM Rational Change to use OS Authentication.
IBM Rational Change and the Rational Directory Server are predefined to use Admin user as the initial admin user. ( fro earlier versions ChangeAdmin was used). You must configure a different user when using OS authentication. Rational recommends using ccm_root as the initial admin user. OS authentication is done on the machine where the IBM Rational Change server is installed. Note See SYNERGY/CM Administration Guide about setting up PAM (described in the section about setting up ESD).
1. On the IBM Rational Change server machine, set user to ccm_root.
$ su ccm_root
2. Change directory to <cs_home>/cs_app/webapps/synergy/WEB-INF/wsconfig/
3. Replace Adnin(ChangeAdmin) user with the OS user in users.cfg.
a. Open the users.cfg file.
b. Find the entry for Admin(ChangeAdmin), for example: Admin = 101
c. Change user (assuming the OS user is ccm_root), for example: ccm_root = 101
d. Save and close the users.cfg file.
4. Add the OS user with ccm_admin role to IBM Rational Synergy database. For an example, see “Define SYNERGY session user in the database” in the IBM Rational Change Installation Guide.
5. Set ccmauth executable setuid root.
$ su root
$ cd <cs_home>/cs_app/webapps/synergy/WEB-INF/bin
$ chown root ccmauth
$ chmod 4755 ccmauth
This document is not intended to provide an exhaustive list of all potential errors within IBM Rational Change, but rather to cover the majority of start up problems a user or administrator may encounter, such as why a session is unable to start. If you cannot locate or resolve your particular issue using this document, please contact Technical Support and provide them with the relevant sections of the pt.log and backend session user’s ccm_ui.log and ccm_eng.log files for further review. Obtaining the appropriate log files will allow Technical Support to more easily diagnose and resolve the problem in a more timely fashion.