Mustgather: Read first for IBM UrbanCode Deploy
What kind of Information does IBM need in order to help diagnose my UrbanCode Deploy issue and identify my root cause? And in what formats should I provide this information to IBM?
Resolving The Problem
Any information provided should use the following suggested formats for faster resolution:
||PDF or Portable Document Format is the most usable format (almost all documents can be exported to a PDF file).|
|Server/agent logs are in plain text format. Do not copy/paste them into another file, but provide them in their entirety.|
ZIP, TAR, TAR.GZ, TAR.GZ.BZ2
|RAR and 7Z are proprietary compression types.|
|Most image file formats are portable.|
Types of Issues and files required to diagnose them:
You will need to gather different diagnostic information based on where you experience your issue. The below table and categories help to identify what should be gathered.
NOTE: The Server section should always be gathered for any issue type.
Click on each "Issue Type" to jump to the relevant data collection section.
|Errors specific to the server. Errors in the deployserver.out file.
HTTP Requests which return with the response code 500 Internal Server Error.
|Errors related to a specific Agent Relay.|
|Errors related to a specific Agent.|
|Issues with the agent not showing Online in the UI|
|Attempts to upgrade the Agent remotely fails and leaves the agent offline.|
|The process execution gives unexpected results and the normal server logs do not produce enough information.|
|Server errors which appear to be specific to databases.
For example: Error 'ORA0001 Constraint Violation'
|Any error with having to do with a specific plugin; loading, usage.|
|Excessive page load times, processes taking longer than normal, and other time related problems.|
|Attempts to connect to IBM SmartCloud Orchestrator or IBM PureApplication System or IBM WorkLoad Deployer and to provision an image result in errors, or no image is created, or the Agent cannot be contacted, when using IBM UrbanCode Deploy.|
|Attempts to create new versions fail because the version imports processes remain in Waiting State and do not complete.|
|Attempts to send notifications fail, emails are not received as expected.|
|Attempts to integrate with LDAP for user Authentication and Authorization produce unexpected results.|
|A process uses the Acquire Lock / Release Lock steps. One of these steps hangs and the process remains in Executing state indefinitely.|
|You configured Artifact Cleanup in Settings > System in IBM UrbanCode Deploy, but the Component Versions are not getting archived or deleted as expected.|
|Resource Synchronization||You need to investigate the behavior of Resource synchronization which is a background process that propagates changes from Resource Templates to the Base Resources mapped to Environments.|
|SSL on server, relay, and agent||You experience SSL issues from the server, relay, and agent|
|SSL connecting from Step to Remote Server||You experience SSL issues when connecting from the Web Utilities plugin or other steps to a remote server|
|Impersonation with Unix, Linux and z/OS USS||You experience impersonation issues with Unix, Linux, and z/OS USS systems|
|Contact IBM Support for assistance.|
- Operating System vendor and version.
- Java type (JDK/JRE), version and update number. You can obtain this information as follows:
installed.properties, as the value of the property:
<install.java.home>/java -version (Unix)
<install.java.home>\java -version (Windows)
<install.java.home> needs to be replaced by its value.
NOTE: Running the command with the absolute path of the executable is necessary to avoid picking up the system JVM instead of the one configured to run UCD.
3. Installation Details:
- Single server installation or High Availability (HA) cluster? For HA configurations, how many servers and which HA configuration?
- How many agent relays?
- How many agents? If using agent relays, please describe the agent/agent relay configuration. For ex: how many agents connected to each relay.
- What are the geographies of the servers, agent relays, agents, and database.
4. Diagnostic Files
Diagnostic Files are gathered differently depending upon which version of UCD you are running:
For version 188.8.131.52 or higher, the data collection process is automated.
In the UI, click on the "Download" link from Settings > System > Diagnostics (Download) to download the Diagnostics Bundle.
It will include:
- All files in the server's
bindirectory (for debugging configuration)
- All files in the server's
confdirectory (for debugging configuration)
- All files in the server's
- All files in the server's
diagnosticsdirectory, which contains:
latest5000Requests.json: Timing/details about the last 5000 REST calls made to the server
longestRequestsLast30Days.json: Timing/details about the longest requests over the last 30 days
recordCounts.json: Counts of various types of objects present on the server
For version 184.108.40.206 or earlier you will need to collect the required diagnostic files manually:
i. Collect all the following files (The single most important file is the deployserver.out file):
5. Patch details
The list of patches and the patched Java files can be viewed in the Patches page of the UCD Server. Please include a screen shot of the page: Settings > System > Patches with all patches expanded (if any).
1. Add the following trace strings to the agent
2. Add the following trace string to the server in the Web UI under System Settings > Logging
3. Stop and restart the agent.
4. Provide the following to support:
a. From the Agent:
b. From the Server:
<server>/var/log/deployserver.outfrom the server or servers if running a high availability (HA) configuration. NOTE: The deployserver.out file is unique to each server. In an HA configuration the deployserver.out file will need to be collected from each server
5. Change the previously added trace strings to INFO in the Web UI under System Settings > Logging
6. Remove the above added trace strings from System Settings > Logging
7. Remove the below added logging from the agent
<agent>/conf/agent/log4j.properties file and restart the agent
- Check the agent's bin folder for the file upgrade.out. If present, send this file to support.
- Check if there is a folder called agent-upgrade inside agent's bin folder and if there are any files inside it (send a directory listing to support).
- Check if there is a file called upgrade.jar inside the agent's bin folder. Provide the size of that file to support.
<agent>\var\logs to support.
- Send all the files from <agent>\conf\agent to support.
- Check if there are any java processes running for the agent:
Linux: ps -ef |grep java
Windows: Use Task Manager
Verify what is the complete command line of those processes, to make sure they belong to the agent.
The Agent should normally have two running processes; the monitor and the worker.
a. The error message or issue being observed
b. Screen shots of the process flow and details regarding how the process is configured to be run (Ex: It runs against multiple resources).
c. The step or steps involved and screen shots of the step(s) properties.
d. Whether this is a new process or an existing process that has started to fail.
2. The process log(s)
For versions 220.127.116.11 and later, gather the logs by opening the process request and clicking "Download All Logs."
For versions prior to 18.104.22.168, navigate to the specific step that failed in the process request, and from the icons located to the right of the Step name in the row:
a. Download the Output log
b. Provide the Full Details from the Error log
c. Provide screen shots of the Input/Output Properties
3. If requested by support, the Trace of the execution process.
For issues that appear to be specific to the database, please include the database type and version. Additionally, please ensure that your Database Administrator is available to perform diagnostic on the database server, as requested by IBM Support.
If the issue is related to a user's application or component process, screen shots of the process and the step properties of steps that are failing are very useful.
- Provide the full Plugin version including build number.
- Collect a copy of the plugin if it is a custom plugin or modified plugin.
- Collect the following from an the execution of a plugin step, if the issue is relating to the execution of a plugin step :
- input properties
- output properties
- output log
- error log
- Collect any custom post-processing scripts.
- Provide also screen shots, but do not crop off the complete UrbanCode Deploy page.
- Specific times to load pages (for example if the Applications Tab is loading slowly).
Use something like the Google Chrome Developer Console to load the page and check the Response time for the HTTP requests and responses.
If you are using version 22.214.171.124 or higher, you can view this information directly from the UrbanCode Deploy server as follows:
- Log into the UrbanCode Deploy Server
- Open the Settings > System > Diagnostics page. This page logs how long different HTTP requests take, so you can determine which pages are being slow for which users.
- If the server appears to be operating slowly, Collect Stack Traces and Heap Dumps.
- Provide the browser used and its specific version.
- If the browser has the ability to show in a console what are the console errors, provide a screen shot.
- In general it is useful to identify the REST call that is causing the problem using a tool to gather times for the HTTP responses and the return code (e.g. 503 Bad Gateway). Some browsers have this capability built in.
How to submit diagnostic data to IBM Support
After you have collected the preceding information, you can begin analyzing the data or simply submit the diagnostic data to IBM support. After a Case is opened, you can update the Case from the IBM Support Site with the diagnostic data. Also, update the case to indicate that data has been sent.
For a listing of all technotes, downloads, and educational materials search the IBM UrbanCode Deploy product support site.
Note: Do not send any confidential information from your company.
Related informationIBM Support Guide
More support for:
IBM UrbanCode Deploy
Component: General Information
Software version: 6.0, 6.0.1, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.2, 6.2.1
Operating system(s): AIX, HP-UX, Linux, OS X, Solaris, Windows, Windows Mobile, z/OS
Reference #: 1673000
Modified date: 09 April 2019