IBM Support

Mustgather: Read first for IBM UrbanCode Deploy

Troubleshooting


Problem

What kind of Information does IBM need in order to help diagnose my UrbanCode Deploy issue and identify my root cause?  And in what formats should I provide this information to IBM?

Resolving The Problem

File Formats:
 

Any information provided should use the following suggested formats for faster resolution:

Data Types
File Types
Explanation
Documents
PDF
PDF or Portable Document Format is the most usable format (almost all documents can be exported to a PDF file).
Logs
plain text
Server/agent logs are in plain text format. Do not copy/paste them into another file, but provide them in their entirety.
Compressed
ZIP, TAR, TAR.GZ, TAR.GZ.BZ2
RAR and 7Z are proprietary compression types.
Images
PNG, JPG
Most image file formats are portable.



Types of Issues and files required to diagnose them:

You will need to gather different diagnostic information based on where you experience your issue.  The below table and categories help to identify what should be gathered.  

NOTE: The Server section should always be gathered for any issue type.

Click on each "Issue Type" to jump to the relevant data collection section.

Issue Type
Explanation
Errors specific to the server. Errors in the deployserver.out file.
HTTP Requests which return with the response code 500 Internal Server Error.
Errors related to a specific Agent Relay.
Errors related to a specific Agent.
Issues with the agent not showing Online in the UI
Attempts to upgrade the Agent remotely fails and leaves the agent offline.
The process execution gives unexpected results and the normal server logs do not produce enough information.
Server errors which appear to be specific to databases.
For example: Error 'ORA0001 Constraint Violation'
Any error with having to do with a specific plugin; loading, usage.
Excessive page load times, processes taking longer than normal, and other time related problems.
JavaScript errors and other problems which cause the UI to not render properly. Note that UI issues caused by HTTP Response code 500 Internal Server Error are both UI and server issues.
Attempts to connect to IBM SmartCloud Orchestrator or IBM PureApplication System or IBM WorkLoad Deployer and to provision an image result in errors, or no image is created, or the Agent cannot be contacted, when using IBM UrbanCode Deploy.
Attempts to create new versions fail because the version imports processes remain in Waiting State and do not complete.
Attempts to send notifications fail, emails are not received as expected.
Attempts to integrate with LDAP for user Authentication and Authorization produce unexpected results.
A process uses the Acquire Lock / Release Lock steps. One of these steps hangs and the process remains in Executing state indefinitely.
You configured Artifact Cleanup in Settings > System in IBM UrbanCode Deploy, but the Component Versions are not getting archived or deleted as expected.
Resource Synchronization You need to investigate the behavior of Resource synchronization which is a background process that propagates changes from Resource Templates to the Base Resources mapped to Environments.
SSL on server, relay, and agent You experience SSL issues from the server, relay, and agent
SSL connecting from Step to Remote Server You experience SSL issues when connecting from the Web Utilities plugin or other steps to a remote server
Impersonation with Unix, Linux and z/OS USS You experience impersonation issues with Unix, Linux, and z/OS USS systems
Everything Else
Contact IBM Support for assistance.


Server Issues:

Include the following:

1. A brief description of what happened and what you were doing when the error occurred.

2. Environment Details:

  • Operating System vendor and version. 
  • Java type (JDK/JRE), version and update number. You can obtain this information as follows:
Locate the JVM installation directory that is listed in the file installed.properties, as the value of the property:  install.java.home
Execute the command:

<install.java.home>/java -version (Unix)

<install.java.home>\java -version (Windows)

where <install.java.home> needs to be replaced by its value.

 NOTE: Running the command with the absolute path of the executable is necessary to avoid picking up the system JVM instead of the one configured to run UCD.

3. Installation Details:

  • Single server installation or High Availability (HA) cluster? For HA configurations, how many servers and which HA configuration? 
  • How many agent relays?  
  • How many agents?  If using agent relays, please describe the agent/agent relay configuration.  For ex:  how many agents connected to each relay. 
  • What are the geographies of the servers, agent relays, agents, and database.

4. Diagnostic Files

Diagnostic Files are gathered differently depending upon which version of UCD you are running:

  For version 6.1.0.2 or higher, the data collection process is automated.

In the UI, click on the "Download" link from Settings > System > Diagnostics (Download)  to download the Diagnostics Bundle.
It will include:

  • All files in the server's bin directory (for debugging configuration)
  • All files in the server's conf directory (for debugging configuration)
  • All files in the server's patches directory
  • All files in the server's var/log directory
  • The diagnostics directory, which contains:

- latest5000Requests.json: Timing/details about the last 5000 REST calls made to the server
- longestRequestsLast30Days.json: Timing/details about the longest requests over the last 30 days
- recordCounts.json: Counts of various types of objects present on the server

For version 6.1.0.1 or earlier you will need to collect the required diagnostic files manually:

    i. Collect all the following files (The single most important file is the deployserver.out file):

  • <ucd>/bin/set_env
  • <ucd>/conf/installed.version
  • <ucd>/conf/log4j.properties
  • <ucd>/conf/server/installed.properties
  • <ucd>/var/log/deployserver.out
  • <ucd>/var/log/*
  • <ucd>/patches/*

5. Patch details

 The list of patches and the patched Java files can be viewed in the Patches page of the UCD Server.  Please include a screen shot of the page: Settings > System > Patches with all patches expanded (if any).


Back to top

Agent Relay Issues:

For issues that appear to be specific to an Agent Relay, collect the following:

  • <agent>/conf/*
  • <agent>/var/logs/*

Back to top

Agent Issues:

For issues that appear to be specific to an Agent, collect the following:

  • <agent>/bin/agent(.cmd)
  • <agent>/conf/*
  • <agent>/var/log/*

Back to top

Agent Connectivity:

For issues where the Agent is not able to successfully connect to the server:

1. Add the following trace strings to the agent <agent>/conf/agent/log4j.properties file:

  • log4j.logger.com.urbancode.air.agent=DEBUG
  • log4j.logger.com.urbancode.air.devilfish=DEBUG


2. Add the following trace string to the server in the Web UI under System Settings > Logging

  • log4j.logger.com.urbancode.ds.subsys.deploy.agent=DEBUG
  • log4j.logger.com.urbancode.air.devilfish=DEBUG


3. Stop and restart the agent.

4. Provide the following to support:

   a. From the Agent: <agent>/var/log/*

   b. From the Server:

  • The <server>/var/log/deployserver.out from the server or servers if running a high availability (HA) configuration.  NOTE:  The deployserver.out file is unique to each server.  In an HA configuration the deployserver.out file will need to be collected from each server
  • <server>/logs/connection/<agent-name>/* Or <appdata>/logs/connection/<agent-name>/*

5. Change the previously added trace strings to INFO in the Web UI under System Settings > Logging

  • log4j.logger.com.urbancode.ds.subsys.deploy.agent=INFO
  • log4j.logger.com.urbancode.air.devilfish=INFO

6.  Remove the above added trace strings from System Settings > Logging

7. Remove the below added logging from the agent <agent>/conf/agent/log4j.properties file and restart the agent

  • log4j.logger.com.urbancode.air.agent=DEBUG
  • log4j.logger.com.urbancode.air.devilfish=DEBUG

Back to top


Remote Agent Upgrade Issues:

For problems where upgrading the Agent remotely from the server fails and leaves the agent offline (as seen from the UCD server), collect the following.
 

  1. Check the agent's bin folder for the file upgrade.out. If present, send this file to support.
     
  2. Check if there is a folder called agent-upgrade inside agent's bin folder and if there are any files inside it (send a directory listing to support).
     
  3. Check if there is a file called upgrade.jar inside the agent's bin folder. Provide the size of that file to support.
     
  4. <agent>\var\logs to support.
     
  5. Send all the files from <agent>\conf\agent to support.
     
  6. Check if there are any java processes running for the agent:
    Linux: ps -ef |grep java
    Windows: Use Task Manager

    Verify what is the complete command line of those processes, to make sure they belong to the agent.
    The Agent should normally have two running processes; the monitor and the worker.

  a. The error message or issue being observed

  b. Screen shots of the process flow and details regarding how the process is configured to be run (Ex: It runs against multiple resources).

  c. The step or steps involved and screen shots of the step(s) properties.

  d. Whether this is a new process or an existing process that has started to fail.

2. The process log(s)

For versions 6.1.1.1 and later, gather the logs by opening the process request and clicking "Download All Logs."

For versions prior to 6.1.1.1, navigate to the specific step that failed in the process request, and from the icons located to the right of the Step name in the row:

  a. Download the Output log

  b. Provide the Full Details from the Error log

  c. Provide screen shots of the Input/Output Properties

3. If requested by support, the Trace of the execution process.



Database Issues:

For issues that appear to be specific to the database, please include the database type and version. Additionally, please ensure that your Database Administrator is available to perform diagnostic on the database server, as requested by IBM Support.

If the issue is related to a user's application or component process, screen shots of the process and the step properties of steps that are failing are very useful.


Back to top


Plugin Issues:

If the issue is related to the loading of the plugin, collect server information. Otherwise collect the following.

  1. Provide the full Plugin version including build number.
     
  2. Collect a copy of the plugin if it is a custom plugin or modified plugin.
     
  3. Collect the following from an the execution of a plugin step, if the issue is relating to the execution of a plugin step :
    • input properties
    • output properties
    • output log
    • error log
       
  4. Collect any custom post-processing scripts.
     
  5. Provide also screen shots, but do not crop off the complete UrbanCode Deploy page.

6. If the issue is specifically related to the WebSphere Deploy and Configure plugins, provide additional information as documented  here


Back to top


Performance Issues:

  • Specific times to load pages (for example if the Applications Tab is loading slowly).

    Use something like the Google Chrome Developer Console to load the page and check the Response time for the HTTP requests and responses.

    If you are using version 6.1.0.2 or higher, you can view this information directly from the UrbanCode Deploy server as follows:
    - Log into the UrbanCode Deploy Server
    - Open the Settings > System > Diagnostics page. This page logs how long different HTTP requests take, so you can determine which pages are being slow for which users.
     
  • If the server appears to be operating slowly, Collect Stack Traces and Heap Dumps.


UI Issues:

For all UI related issues there can never be to many screen shots; though do be careful of taking partial screen shots as they can lack context.

  • Provide the browser used and its specific version.
     
  • If the browser has the ability to show in a console what are the console errors, provide a screen shot.
     
  • In general it is useful to identify the REST call that is causing the problem using a tool to gather times for the HTTP responses and the return code (e.g. 503 Bad Gateway). Some browsers have this capability built in.


Back to top


How to submit diagnostic data to IBM Support

After you have collected the preceding information, you can begin analyzing the data or simply submit the diagnostic data to IBM support. After a Case is opened, you can update the Case from the IBM Support Site with the diagnostic data. Also, update the case to indicate that data has been sent.

For a listing of all technotes, downloads, and educational materials search the IBM UrbanCode Deploy product support site.

Note: Do not send any confidential information from your company.

Related information

IBM Support Guide
Fix Central
My notifications

Document information

More support for: IBM UrbanCode Deploy

Component: General Information

Software version: 6.0, 6.0.1, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.2, 6.2.1

Operating system(s): AIX, HP-UX, Linux, OS X, Solaris, Windows, Windows Mobile, z/OS

Reference #: 1673000

Modified date: 09 April 2019