Technote (troubleshooting)
Problem(Abstract)
Your WebSphere MQ system is having problems with queue manager clusters, and you need to collect data to find a solution. This document explains how to generate diagnostic data when the problem is happening, how to collect the available data from the system, and how to analyze that data and share it with IBM support.
Environment
These instructions are for WebSphere MQ on AIX, HP-UX, IBM i, Linux, Solaris, and Windows systems. Return to the WebSphere MQ MustGather index for other systems.
Resolving the problem
Step 1: Generate Data
If the problem has already happened and you cannot reproduce it reliably, proceed to Step 2 to collect the information already logged by WebSphere MQ. Otherwise, please try to generate the following diagnostic data while the problem is happening:
- WebSphere MQ trace on the queue manager with the cluster problem.
- WebSphere MQ trace on the full repository queue managers for the cluster, taken at the same time as the first trace.
Try to start and stop trace as quickly as possible in order to limit both the overhead of tracing and the size of the trace files. On systems other than IBM i and Windows, format the trace files after stopping the trace.
Step 2: Collect Data
Collect the following data for your cluster problem. Use the runmqras tool if your WebSphere MQ installation is V7.0.1.8 or later, V7.1.0.1 or later, or V7.5, otherwise follow the manual instructions:
- WebSphere MQ data from the queue manager with the cluster problem.
- WebSphere MQ data from the full repository queue managers for the cluster.
WebSphere MQ includes an automated data collection tool called runmqras which you can use if you are running WebSphere MQ V7.0.1.8 or later, V7.1.0.1 or later, or V7.5.
Follow the WebSphere MQ runmqras usage instructions to gather data for your cluster problem:
- Always gather object definitions using the section defs
- Always gather cluster information using the section cluster
- If you generated trace files in Step 1, collect them by including the section trace
Examples
- To collect basic information for a cluster problem on queue manager QMA:
runmqras -section defs,cluster -qmlist QMA
- If you have traces from QMA and want to send the data directly to IBM by standard FTP:
runmqras -section defs,cluster,trace -qmlist QMA -pmrno 12345,67R,890 -ftp IBM
- If QMA belongs to a cluster with two full repository queue managers called REPOS1 and REPOS2, you should gather the same information from both full repositories too:
runmqras -section defs,cluster,trace -qmlist REPOS1
runmqras -section defs,cluster,trace -qmlist REPOS2
- If your system has multiple WebSphere MQ installations, issue the setmqenv command to choose the right installation. On Linux and UNIX, source setmqenv using a dot. For example:
Linux & UNIX
sh> . /path/to/mqm/bin/setmqenv -n Installation0
Windows
C:\> C:\Program Files (x86)\IBM\WebSphere MQ\bin\setmqenv -s
- Record the operating system version and maintenance level on both systems.
- Record the WebSphere MQ version and maintenance level on both systems.
- Collect the WebSphere MQ error logs as well as any WebSphere MQ FFST files. On IBM i, follow these instructions to save the necessary files and job logs.
- Use runmqsc to record your queue manager, queues, and namelist information:
DISPLAY QMGR ALL
DISPLAY QLOCAL(*) ALL
DISPLAY QREMOTE(*) ALL
DISPLAY NAMELIST(*) ALL
- Use runmqsc to record your cluster queue managers, cluster queues, and cluster topics:
DISPLAY CLUSQMGR(*) ALL
DISPLAY QCLUSTER(*) ALL
DISPLAY TCLUSTER(*) ALL
- Use runmqsc to record your channels, listeners, and their status:
DISPLAY CHANNEL(*) ALL
DISPLAY CHSTATUS(*) ALL
DISPLAY LISTENER(*) ALL
DISPLAY LSSTATUS(*) ALL
- Dump the queue manager cluster repository cache using the amqrfdm utility. Be sure to provide your queue manager name and to use the correct input file (attached to the end of this TechNote) for the platform. For example, to dump queue managers QMA, QMB and QMC on different platforms:
IBM i Q shell
===> /QSYS.LIB/QMQM.LIB/AMQRFDM.PGM -m QMA < cluster-IBMi.txt > QMA.cluster.txt
Linux & UNIX
sh> amqrfdm -m QMB < cluster-unix.txt > QMB.cluster.txt
Windows
C:>\ amqrfdm -m QMC < cluster-win.txt > %TMP%\QMC.cluster.txt
- On Linux and UNIX systems, please send the output from the mqconfig script from the failing queue manager and the full repository queue managers in addition to the other WebSphere MQ data.
Step 3: Analyze Data
You can now analyze WebSphere MQ data with IBM Support Assistant tools such as IBM Support Assistant Team Server. Using a web browser, you can see an overview of your system based on the data you collected, review problem symptoms found in the data, and search IBM's Global Knowledge Base for relevant Fix Packs, APARs and Technotes. Follow these steps until you find a solution for your clustering problem:
- Review the IBM Support Assistant findings.
- Follow the cluster troubleshooting guide.
- Use IBM Service Request to open a problem record with IBM.
- Send your data to IBM for further analysis.
ATTENTION: A good problem description is one of the most important tools IBM needs to analyze your data! When sending data to IBM, be sure to update your PMR or send a note with the following information:
- Tell us what errors you saw, where you saw them, and what time they happened
- Let us know if you made any changes to WebSphere MQ or the system before the problem
- Share any other observations which you think will help us to better understand the problem
Please see the IBM Software Support Handbook for more information on working with IBM support.
Rate this page:
Copyright and trademark information
IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.