MustGather: Read first for all WebSphere MQ Low Latency Messaging products

Technote (troubleshooting)


Problem(Abstract)


MustGather documents aid in problem determination and save time resolving Problem Management Records (PMRs).

Collecting MustGather data early, even before opening the PMR, helps IBM® Support personnel to quickly determine if:

1. the symptoms match known problems.
2. there is a non-defect problem that can be identified and resolved.
3. there is a defect that identifies a workaround to reduce severity.
4. a root cause can be identified to expedite development of a code fix.

Resolving the problem

Gathering general information

You must manually gather the following general information for each host machine:

Basic host machine and product information

  • LLM product version
  • Hostname & IP address information (to correlate with log/event data)
  • Operating system name and version
  • Operating system patches or maintenance level
  • Number of CPUs on the machine
  • Amount of memory (RAM) on the machine

Linux system information
On Linux systems, run the following commands, redirecting output to a file:
  • cat <llm-root-dir>/version.properties
  • uname -a
  • netstat -rn
  • ifconfig -a
  • cat /etc/redhat-release
  • cat /proc/cpuinfo
  • cat /proc/meminfo
  • sysctl -a

Solaris system information
On Solaris systems, run the following commands, redirecting output to a file:
  • ndd /dev/udp udp_xmit_hiwat
  • ndd /dev/udp udp_recv_hiwat
  • ndd /dev/udp udp_max_buf
  • ndd /dev/rawip icmp_xmit_hiwat
  • ndd /dev/rawip icmp_recv_hiwat
  • ndd /dev/rawip icmp_max_buf

Gathering application files
You must manually gather your application information on each host machine. Provide the relevant files listed here:

  • transmitter/receiver LLM logging output
  • rmmTbasic.cfg**
  • rmmTadvanced.cfg**
  • rmmRbasic.cfg**
  • rmmRadvanced.cfg**
  • rumAadvanced.cfg**

** NOTE: These configuration files may or may not be present depending on whether your LMM application sets its configuration characteristics programmatically or using configuration files.


Gathering LLM logging output
By default, LMM does not generate logging information. There are two ways to enable logging:
  1. Registering an application log event handler
  2. Setting the log level using the basic/advanced configuration files

Each approach is discussed in the following sections.
  • Registering an application log event handler

    Your application can control the filesystem logging of RMM/RUM log events. You must identify any log files your application creates, and collect them. Your application should set the RMM/RUM logging to a sufficiently high level so that detailed problem determination information is generated.

    Here is information to help identify how/where your application handles RMM/RUM log events, and how to set the logging level.
    • RMM logging

        All RMM log messages are delivered to your application's on_log_event callback routine. This routine is registered by the rmmTxInit or rmmRxInit calls during application initialization. Examine your code's on_log_event callback handler to determine what your application does with RMM log events that are generated.
        The log level is set by the rmmTxInit or rmmRxInit calls as well. It is passed in by the field LogLevel in the rmmTConfig structure. The log level should be set to RMM_LOGLEV_INFO or higher ( RMM_LOGLEV_XTRACE is the most verbose log level).

    • RUM logging

        All RUM log messages are delivered to your application's on_log_event callback routine. This routine is registered by the rumInit call during application initialization. Examine your code's on_log_event callback handler to determine what your application does with RUM log events that are generated.
        The log level is set by the rumInit call as well. It is passed in by the field LogLevel in the rumConfig structure. The log level should be set to RUM_LOGLEV_INFO or higher ( RUM_LOGLEV_XTRACE is the most verbose log level)..

  • Setting the log level using the basic/advanced configuration files

    You can enable logging by adding the following line to either your basic or advanced configuration file:
      rmmTbasic.cfg:
      LogLevel                  = 9
      LogFile                   = /tmp/rmmsender.log

      rmmRbasic.cfg:
      LogLevel                  = 9
      LogFile                   = /tmp/rmmreceiver.log

      rumBasic.cfg:
      LogLevel                  = 9
      LogFile                   = /tmp/rum.log


    Generating RMM/RUM snapshot logging
    Snapshot logging is an effective problem determination tool for LLM performance issues. To enable snapshots, create an advanced configuration file (as indicated below) in your application's current directory and add the appropriate transmitter/receiver attributes :
      rmmTadvanced.cfg:
      SnapShotCycleMilli        = 5000
      PrintStreamInfo_tx        = 1
      DefaultLogLevel           = 9
      LogFile                   = /tmp/rmmsender.log

      rmmRadvanced.cfg:
      SnapShotCycleMilli        = 5000
      PrintStreamInfo_rx        = 1
      DefaultLogLevel           = 9
      LogFile                   = /tmp/rmmreceiver.log

      rumAdvanced.cfg:
      SnapShotCycleMilli_tx     = 5000
      PrintStreamInfo_tx        = 1
      DefaultLogLevel           = 9
      LogFile=/tmp/rumsender.log

      rumAdvanced.cfg:
      SnapShotCycleMilli_rx     = 5000
      PrintStreamInfo_rx        = 1
      DefaultLogLevel           = 9
      LogFile=/tmp/rumreceiver.log

    NOTE: When enabling RUM snapshots, the LogFile attribute can not have any whitespace before or after the "=" character (see above). This is a known issue that will be addressed in a future release.

    Restart your application. Using the above attributes, snapshot data will be written to the file indicated by the LogFile attribute every 5 seconds (5000 milliseconds). Since the DefaultLogLevel is set to 9 (which is a verbose level), there will be other, additional logging messages in the logfile as well.


    Submitting information to IBM Support

    After opening a PMR, submit your diagnostic MustGather data to IBM. If you are using ESR, update the PMR to indicate that data has been sent.

    For a listing of all technotes, downloads, and educational materials specific to this product, search the WebSphere MQ Low Latency Messaging support page.

  • Rate this page:

    (0 users)Average rating

    Document information


    More support for:

    WebSphere MQ Low Latency Messaging

    Software version:

    2.4, 2.4.0.1, 2.5, 2.6, 2.6.0.1, 2.6.0.2

    Operating system(s):

    Linux Red Hat - iSeries, Solaris, Windows

    Software edition:

    All Editions

    Reference #:

    1286342

    Modified date:

    2008-11-04

    Translate my page

    Machine Translation

    Content navigation