Technote (troubleshooting)
Problem(Abstract)
MustGather documents aid in problem determination and save time resolving Problem Management Records (PMRs).
Collecting MustGather data early, even before opening the PMR, helps IBM® Support personnel to quickly determine if:
1. the symptoms match known problems.
2. there is a non-defect problem that can be identified and resolved.
3. there is a defect that identifies a workaround to reduce severity.
4. a root cause can be identified to expedite development of a code fix.
Resolving the problem
Gathering general information
You must manually gather the following general information for each host machine:
Basic host machine and product information
- LLM product version
- Hostname & IP address information (to correlate with log/event data)
- Operating system name and version
- Operating system patches or maintenance level
- Number of CPUs on the machine
- Amount of memory (RAM) on the machine
Linux system information
On Linux systems, run the following commands, redirecting output to a file:
- cat <llm-root-dir>/version.properties
- uname -a
- netstat -rn
- ifconfig -a
- cat /etc/redhat-release
- cat /proc/cpuinfo
- cat /proc/meminfo
- sysctl -a
Solaris system information
On Solaris systems, run the following commands, redirecting output to a file:
- ndd /dev/udp udp_xmit_hiwat
- ndd /dev/udp udp_recv_hiwat
- ndd /dev/udp udp_max_buf
- ndd /dev/rawip icmp_xmit_hiwat
- ndd /dev/rawip icmp_recv_hiwat
- ndd /dev/rawip icmp_max_buf
Gathering application files
You must manually gather your application information on each host machine. Provide the relevant files listed here:
- transmitter/receiver LLM logging output
- rmmTbasic.cfg**
- rmmTadvanced.cfg**
- rmmRbasic.cfg**
- rmmRadvanced.cfg**
- rumAadvanced.cfg**
** NOTE: These configuration files may or may not be present depending on whether your LMM application sets its configuration characteristics programmatically or using configuration files.
Gathering LLM logging output
By default, LMM does not generate logging information. There are two ways to enable logging:
- Registering an application log event handler
- Setting the log level using the basic/advanced configuration files
Each approach is discussed in the following sections.
Your application can control the filesystem logging of RMM/RUM log events. You must identify any log files your application creates, and collect them. Your application should set the RMM/RUM logging to a sufficiently high level so that detailed problem determination information is generated.
Here is information to help identify how/where your application handles RMM/RUM log events, and how to set the logging level.
- RMM logging
All RMM log messages are delivered to your application's on_log_event callback routine. This routine is registered by the rmmTxInit or rmmRxInit calls during application initialization. Examine your code's on_log_event callback handler to determine what your application does with RMM log events that are generated.
-
The log level is set by the
rmmTxInit or
rmmRxInit calls as well. It is passed in by the field
LogLevel in the
rmmTConfig structure. The log level should be set to
RMM_LOGLEV_INFO or higher (
RMM_LOGLEV_XTRACE is the most verbose log level).
- RUM logging
All RUM log messages are delivered to your application's on_log_event callback routine. This routine is registered by the rumInit call during application initialization. Examine your code's on_log_event callback handler to determine what your application does with RUM log events that are generated.
-
The log level is set by the
rumInit call as well. It is passed in by the field
LogLevel in the
rumConfig structure. The log level should be set to
RUM_LOGLEV_INFO or higher (
RUM_LOGLEV_XTRACE is the most verbose log level)..
You can enable logging by adding the following line to either your basic or advanced configuration file:
-
rmmTbasic.cfg:
LogLevel = 9
LogFile = /tmp/rmmsender.log
rmmRbasic.cfg:
LogLevel = 9
LogFile = /tmp/rmmreceiver.log
rumBasic.cfg:
LogLevel = 9
LogFile = /tmp/rum.log
Generating RMM/RUM snapshot logging
Snapshot logging is an effective problem determination tool for LLM performance issues. To enable snapshots, create an advanced configuration file (as indicated below) in your application's current directory and add the appropriate transmitter/receiver attributes :
-
rmmTadvanced.cfg:
SnapShotCycleMilli = 5000
PrintStreamInfo_tx = 1
DefaultLogLevel = 9
LogFile = /tmp/rmmsender.log
rmmRadvanced.cfg:
SnapShotCycleMilli = 5000
PrintStreamInfo_rx = 1
DefaultLogLevel = 9
LogFile = /tmp/rmmreceiver.log
rumAdvanced.cfg:
SnapShotCycleMilli_tx = 5000
PrintStreamInfo_tx = 1
DefaultLogLevel = 9
LogFile=/tmp/rumsender.log
rumAdvanced.cfg:
SnapShotCycleMilli_rx = 5000
PrintStreamInfo_rx = 1
DefaultLogLevel = 9
LogFile=/tmp/rumreceiver.log
NOTE: When enabling RUM snapshots, the LogFile attribute can not have any whitespace before or after the "=" character (see above). This is a known issue that will be addressed in a future release.
Restart your application. Using the above attributes, snapshot data will be written to the file indicated by the LogFile attribute every 5 seconds (5000 milliseconds). Since the DefaultLogLevel is set to 9 (which is a verbose level), there will be other, additional logging messages in the logfile as well.
Submitting information to IBM Support
After opening a PMR, submit your diagnostic MustGather data to IBM. If you are using ESR, update the PMR to indicate that data has been sent.
For a listing of all technotes, downloads, and educational materials specific to this product, search the WebSphere MQ Low Latency Messaging support page.
Rate this page:
Copyright and trademark information
IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.