Gathering documentation for a hanging WebSphere MQ queue manager on UNIX and Linux

Technote (troubleshooting)


Problem(Abstract)

IBM WebSphere MQ support teams need a variety of diagnostic data to examine hanging queue managers, some of which may be difficult to gather.

Resolving the problem

The mqhangdoc script automates the collection of diagnostic data for hanging WebSphere MQ queue managers on AIX, HP-UX, Linux and Solaris systems.



Using mqhangdoc

The mqhangdoc script relies on two other scripts, stackit and sigdump. In order to use mqhangdoc, you must download all three scripts to your system and make them executable, for example by running: chmod a+x mqhangdoc stackit sigdump



syntax: mqhangdoc -?

        mqhangdoc -p PMR -m Qmgr [-d Directory] [-o Options] [-t Time]


Required options:

-p PMR
    The mqhangdoc script needs your IBM Problem Management Record (PMR) number, for example 12345,67R,890. If you do not have a PMR number, you should open one through IBM Service Request before gathering hang documentation.

-m Qmgr
    You must also give the name of the queue manager which is experiencing the problem. The mqhangdoc script can only gather data for one queue manager at a time.



Additional options:

-d Directory
    By default, mqhangdoc saves its output in a unique directory under /var/tmp. Use this option to choose a different output directory, for example on a filesystem with more free space.

-o Options
    Any options provided here are forwarded to the stackit script, which mqhangdoc uses to analyze queue manager processes. IBM support will advise you if any special options are required.

-t Time
    This option controls the time (in seconds) that mqhangdoc will trace the queue manager. You may trace anywhere from 1 to 300 seconds, though the default is 30 seconds of trace. If you specify a value of zero, mqhangdoc will not trace the queue manager.



Examples


  • Example 1: To see detailed for mqhangdoc:
      sh> mqhangdoc  - ?

  • Example 2: To gather hang documentation from queue manager TEST.QMGR for IBM problem record number 12345,67R,890 and run trace for ten seconds:
      sh> mqhangdoc -p 12345,67R,890 -m TEST.QMGR -t 10

  • Example 3: To gather hang documentation from queue manager PROD.QMGR for IBM problem record number 12345,67R,890 and put the output under the /var/data directory:
      sh> mqhangdoc -p 12345,67R,890 -m PROD.QMGR -d /var/data

  • Example 4: To gather hang documentation from queue manager DEV.QMGR for IBM problem record number 12345,67R,890 without gathering a trace:
      sh> mqhangdoc -p 12345,67R,890 -m DEV.QMGR -t 0



Security

The mqhangdoc script relies on two other scripts to collect data, and these must be in the same directory as mqhangdoc, in your current directory, or elsewhere in your PATH. You must also be sure to run mqhangdoc from an account with sufficient authority, or these scripts will fail:



If you started your queue manager from the mqm account, either by logging in as mqm or using a command like "su - mqm" to switch to the mqm user, then you may be able to run mqhangdoc from the mqm account. However, if you started the queue manager as another user in the mqm group, you must run mqhangdoc as root in order for it to succeed.


Watch the output from the mqhangdoc script carefully to make sure it has sufficient authority to gather data from the queue manager. If you see an error messages like the one below, you must run mqhangdoc as a different user:

    mqhangdoc: You do not have authority to gather debugging information from
               queue manager DEV.QMGR.  Run this script as root instead.



Notes

The mqhangdoc script gathers three snapshots of queue manager activity when it runs:

    1. The first snapshot collects stackit output only.
    2. The second snapshot collects stackit again, while tracing the queue manager (unless you disabled tracing with the '-t 0' option).
    3. The third snapshot collects stackit again after tracing is over, and uses the sigdump script to generate a set of diagnostic FDC files for the queue manager


The mqhangdoc script is mistrustful of WebSphere MQ commands, which may hang given the state of the queue manager. The mqhangdoc script will wait a few seconds for WebSphere MQ commands to complete before issuing a message and carrying on. The mqhangdoc script may take longer to complete when commands are unresponsive, but it should always finish its work.

When mqhangdoc is finished, it will print the number of errors it encountered and display the name of the file or directory containing the hang documentation. Even if mqhangdoc reports some errors, please send the collected data to IBM for review.

The mqhangdoc script does not remove any files from your system. Therefore you may wish to clean up trace and FDC files after IBM has confirmed receipt of the mqhangdoc information. Please be sure to clean up files before running mqhangdoc again or using the WebSphere MQ runmqras tool.



Sample Output

This sample output shows mqhangdoc in operation. Please be sure to send the output file generated by mqhangdoc, not just the on-screen status information. In this example, the output file is /var/tmp/12345,67R,890-mqhangdoc-2012-08-20-18570.tar.gz:

    sh> ./mqhangdoc -p 12345,67R,890 -m PROD.QMGR
    mqhangdoc: V3.3 running on AIX 7.1 (powerpc) with arguments: -p 12345,678
               -m PROD.QMGR
               
    mqhangdoc: Starting collection on 20 Augus t  2012 at 20:04:03 GMT.
    mqhangdoc: Collecting hang documentation for queue manager PROD.QMGR.
    mqhangdoc: Collecting WebSphere MQ configuration information.
    mqhangdoc: Collecting first debug snapshot from PROD.QMGR.
    stackit: Analyzing process 3866650 (amqzmuc0).
    stackit: Analyzing process 15204464 (amqrrmfa).
    stackit: Analyzing process 15794288 (amqfcxba).
    stackit: Analyzing process 19726572 (runmqchi).
    stackit: Analyzing process 20447366 (amqzmgr0).
    stackit: Analyzing process 20643978 (amqzlaa0_nd).
    stackit: Analyzing process 21037164 (amqzmuf0).
    stackit: Analyzing process 21233820 (amqpcsea).
    stackit: Analyzing process 21561532 (amqzmur0).
    stackit: Analyzing process 21692652 (amqzfuma).
    stackit: Analyzing process 22085872 (amqzdmaa).
    stackit: Analyzing process 22216878 (amqzxma0_nd).
    stackit: Analyzing process 23265336 (amqfqpub).
    stackit: Success rate: 100%
    mqhangdoc: Starting a trace of runmqsc on PROD.QMGR.
    mqhangdoc: Trace will run for 30 seconds.
    mqhangdoc: Trace will run for 25 seconds.
    mqhangdoc: Trace will run for 20 seconds.
    mqhangdoc: Collecting second debug snapshot from PROD.QMGR.
    stackit: Analyzing process 3866650 (amqzmuc0).
    stackit: Analyzing process 15204464 (amqrrmfa).
    stackit: Analyzing process 15794288 (amqfcxba).
    stackit: Analyzing process 19726572 (runmqchi).
    stackit: Analyzing process 20447366 (amqzmgr0).
    stackit: Analyzing process 20643978 (amqzlaa0_nd).
    stackit: Analyzing process 21037164 (amqzmuf0).
    stackit: Analyzing process 21233820 (amqpcsea).
    stackit: Analyzing process 21561532 (amqzmur0).
    stackit: Analyzing process 21692652 (amqzfuma).
    stackit: Analyzing process 22085872 (amqzdmaa).
    stackit: Analyzing process 22216878 (amqzxma0_nd).
    stackit: Analyzing process 23265336 (amqfqpub).
    stackit: Success rate: 100%
    mqhangdoc: Trace will run for 15 seconds.
    mqhangdoc: Trace will run for 10 seconds.
    mqhangdoc: Trace will run for 5 seconds.
    mqhangdoc: Trace will run for 3 seconds.
    mqhangdoc: Trace will run for 2 seconds.
    mqhangdoc: Trace will run for 1 second.
    mqhangdoc: Trace of runmqsc completed successfully.
    mqhangdoc: Collecting third debug snapshot from PROD.QMGR.
    stackit: Analyzing process 3866650 (amqzmuc0).
    stackit: Analyzing process 15204464 (amqrrmfa).
    stackit: Analyzing process 15794288 (amqfcxba).
    stackit: Analyzing process 19726572 (runmqchi).
    stackit: Analyzing process 20447366 (amqzmgr0).
    stackit: Analyzing process 20643978 (amqzlaa0_nd).
    stackit: Analyzing process 21037164 (amqzmuf0).
    stackit: Analyzing process 21233820 (amqpcsea).
    stackit: Analyzing process 21561532 (amqzmur0).
    stackit: Analyzing process 21692652 (amqzfuma).
    stackit: Analyzing process 22085872 (amqzdmaa).
    stackit: Analyzing process 22216878 (amqzxma0_nd).
    stackit: Analyzing process 23265336 (amqfqpub).
    stackit: Success rate: 100%
    sigdump: V3.1 running on AIX 7.1 (powerpc) with arguments: -m PROD.QMGR
    sigdump: Creating diagnostic FDCs for queue manager PROD.QMGR:
    sigdump:   AMQ3866650.0.FDC - amqzmuc0: Critical utilities
    sigdump:   AMQ15204464.0.FDC - amqrrmfa: Cluster repository manager
    sigdump:   AMQ15794288.0.FDC - amqfcxba: Publish/subscribe worker
    sigdump:   AMQ19726572.0.FDC - runmqchi: Channel initiator
    sigdump:   AMQ20447366.0.FDC - amqzmgr0: Process controller
    sigdump:   AMQ20643978.0.FDC - amqzlaa0: Standard application agent
    sigdump:   AMQ21037164.0.FDC - amqzmuf0: Publish/subscribe utilities
    sigdump:   AMQ21233820.0.FDC - amqpcsea: Command server
    sigdump:   AMQ21561532.0.FDC - amqzmur0: Recoverable utilities
    sigdump:   AMQ21692652.0.FDC - amqzfuma: Object authority manager
    sigdump:   AMQ22085872.0.FDC - amqzdmaa: Deferred message processor
    sigdump:   AMQ22216878.0.FDC - amqzxma0: Execution controller
    sigdump:   AMQ23265336.0.FDC - amqfqpub: Publish/subscribe controller
    sigdump: Created 13 diagnostic FDC files in the /var/mqm/errors
             directory.  Please send these files to IBM for review.
    mqhangdoc: Copying /var/mqm/mqs.ini (1 file).
    mqhangdoc: Copying /etc/opt/mqm/mqinst.ini (1 file).
    mqhangdoc: Copying /var/mqm/qmgrs/PROD!QMGR/qm.ini (1 file).
    mqhangdoc: Copying /var/mqm/qmgrs/PROD!QMGR/qmstatus.ini (1 file).
    mqhangdoc: Copying /var/mqm/qmgrs/PROD!QMGR/errors/AMQERR*.LOG (3 files).
    mqhangdoc: Copying /var/mqm/errors/AMQ*.FDC (15 files).
    mqhangdoc: Copying /var/mqm/errors/AMQERR*.LOG (3 files).
    mqhangdoc: Formatting trace files from PROD.QMGR (14 files).
    mqhangdoc: Hang data collection completed at 20:05:53 GMT with no errors.
    mqhangdoc: Creating an archive with the collected files.
    mqhangdoc: Compressing archive for transmission to IBM.
    mqhangdoc: The diagnostic data for queue manager PROD.QMGR is in
               /var/tmp/12345,67R,890-mqhangdoc-2012-08-20-18570.tar.gz.
    mqhangdoc: Remove any unwanted traces from /var/mqm/trace.
    mqhangdoc: Please send the diagnostic output to IBM for review.

DISCLAIMER: All source code and/or binaries attached to this document are referred to here as "the Program". IBM is not providing program services of any kind for the Program. IBM is providing the Program on an "AS IS" basis without warranty of any kind. IBM WILL NOT BE LIABLE FOR ANY ACTUAL, DIRECT, SPECIAL, INCIDENTAL, OR INDIRECT DAMAGES OR FOR ANY ECONOMIC CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS OR SAVINGS), EVEN IF IBM, OR ITS RESELLER, HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Product Alias/Synonym

WebSphere MQ WMQ

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

WebSphere MQ
Problem Determination

Software version:

7.0, 7.1, 7.5

Operating system(s):

AIX, HP-UX, Linux, Solaris

Software edition:

All Editions

Reference #:

1608628

Modified date:

2013-06-12

Translate my page

Machine Translation

Content navigation