IBM Support

WebSphere Application Server Log Watcher: Using TrapIt.ear to watch for WebSphere Application Server events

Troubleshooting


Problem

While investigating a problem with WebSphere Application Server, you may need to watch for events such as messages to the SystemOut.log and take action when they occur.

Resolving The Problem

The TrapIt.ear provides an easy way to perform actions based on events(message ids) in the WebSphere Application Server or based on time. If you need to monitor files (for example SystemOut.log, ffdcs, application or operating system logs, and the like), use the trapit shell script for UNIX and Linux systems.





About TrapIt.ear

TrapIt.ear uses the WebSphere Application Server JMX interface to watch for all JVM log events and perform an action when a matching message appears. TrapIt.ear can also trigger actions based on one or more timers or delays. TrapIt.ear can trigger several types of actions:



  • Dumping the contents of the in-memory trace ring buffer
  • Changing the runtime trace specification
  • Running an operating system command
  • Generating one or more javacores
  • Generating a heap dump


For example, you could use TrapIt.ear to watch for WSVR0605W messages and generate a javacore when they occur. You could also use TrapIt.ear to enable tracing in the middle of the night, generate a javacore five minutes later, then stop tracing after ten minutes.

TrapIt.ear is a client EAR (Enterprise Archive) which you should run using the launchClient.sh (or launchClient.bat) command located in the profile of the server you wish to watch. To use TrapIt.ear, first download it to your system and then run it with no command-line arguments to read its usage help.




    Displaying the TrapIt.ear help

    sh> $WAS_HOME/profiles/AppSrv01/bin/launchClient.sh /full/path/to/TrapIt.ear



    C:\> %WAS_HOME%\profiles\AppSrv02\bin\launchClient C:\full\path\to\TrapIt.ear

Back to top



Syntax

launchClient[.sh] /full/path/to/TrapIt.ear Hostname:Port TRIGGER1 ACTION1 [TRIGGER2 ACTION2] [TRIGGER3 ACTION3] ...



    Hostname:Port

    TrapIt.ear needs to connect to the address and boostrap port of a running WebSphere Application Server instance in order to run. You can find the port under the WebSphere Application Server profile by looking up the server's BOOTSTRAP_ADDRESS in the serverindex.xml file. Alternatively, you can navigate in the WebSphere Application Server administrative console to "Servers > Server Types > WebSphere Application Servers > $ServerName" and review the list of ports:




    If you point TrapIt.ear at the bootstrap port of a nodeagent or a deployment manager, it will print a list of the servers known to them and let you watch any or all of those servers at the same time:

      Example of choosing servers

      TrapIt: Connecting on "aix1:9809"


      TrapIt: 0: Server "dmgr" PID=16318472
      TrapIt: 1: Server "Server1" PID=29819056
      TrapIt: 2: Server "Server2" PID=12903372
      TrapIt: 3: Server "nodeagent" PID=24314062
      Type a comma separated list of servers to trigger from, or type 'all': 1,2


    When watching for messages on systems with a lot of log traffic, it is much more efficient to point a copy of TrapIt.ear directly at each server. Otherwise, this feature can make it easy to use time-based triggers to gather traces and other data from multiple servers (for example, all servers in a cluster, or a selection of servers in a cell).

    If security is enabled, TrapIt.ear will interactively prompt for a userid and password. The userid must have sufficient privilege to invoke the MBeans that TrapIt.ear uses, for example the userid which you use to log in to the administrative console and make configuration changes. More precise authority may be given declaratively; see MBean security policy for more details. The userid is masqueraded at the server on TrapIt.ear's behalf.

    If you use a tool such as nohup to run TrapIt.ear, set the following properties in the properties/sas.client.props file under the same WebSphere Application Server profile from which you run launchClient.sh or launchClient.bat. You can encode your password in the file so that it won't be visible in clear text:

      sas.client.props

      com.ibm.CORBA.loginSource=properties



      # RMI/IIOP user identity
      com.ibm.CORBA.loginUserid=YourUserid
      com.ibm.CORBA.loginPassword=YourPassword


    TRIGGER options

    Each TRIGGER on the command line can follow one of these two forms:




    EVENT=limit,STRING1[,STRING2[,STRING3]]
      Triggers when one of the strings appears in SystemOut.log. The limit counter allows you to control how many times TrapIt.ear will trigger. If you use a limit of 0, TrapIt.ear will trigger indefinitely.


    TIME=YYYY,MM,DD,hh,mm,ss
      Trigger at a given time, specified by year, month, day, hour, minute and second. Use a 24-hour clock to trigger after noon.


    DELAY=[+]seconds
      Trigger after a delay, given in seconds. A value of zero will trigger immediately.

      Normally the delay is relative to the time when TrapIt.ear first started. If you put a plus sign in front of the seconds value, the delay is relative to the time when the previous trigger action completes.


    ACTION options

    Each ACTION on the command line can follow one of these five forms:




    TRACE=*=info
      Changes the runtime trace detail level to the value given. If you set the trace to a value with an asterisk, be sure to use quotation marks as in the second and third examples below.


    JAVACORE[=count,delay]
      Generate one or more javacores, with a delay (in seconds) between each one.


    HEAPDUMP
      Generates a heap dump.

      Attention: Full heap dumps are costly and could pause your server for many seconds, so be aware of the impact when using this action on production systems.


    DUMPBUFFER=/path/to/filename.trc
      Dumps the in-memory trace rung buffer to a file.


    EXEC=/path/to/myscript.sh[ parameters][,/path/to/stdout.log[,/path/to/stderr.log]]
      Run an operating system command, with output going to the named file(s). If the command requires parameters, be sure to use quotation marks as in the fourth example below.

Back to top



Examples

     

    Example 1

    Watch for message WSVR0605W, then generate three javacores at 60 second intervals:



    sh> ./launchClient.sh /full/path/to/TrapIt.ear localhost:2809 EVENT=1,WSVR0605W JAVACORE=3,60

     

    Example 2

    Turn on tracing for a server and watch the log for the message JMSWMQ1107 or the string MQRC_CONNECTION_BROKEN. When either of these appears, create a heap dump, then wait ten more seconds and turn off tracing:



    sh> ./launchClient.sh /scratch/TrapIt.ear localhost:2810 DELAY=0 TRACE="*=info:jmsApi=all:Messaging=all:com.ibm.mq.*=all" EVENT=1,JMSWMQ1107,MQRC_CONNECTION_BROKEN DELAY=+0 HEAPDUMP DELAY=+10 "TRACE=*=info"

     

    Example 3

    Enable trace at a specific date and time, stop tracing ten minutes later, than take a javacore five minutes after that:



    C:\> launchClient %TEMP%\TrapIt.ear 192.168.1.2:2810 TIME=2013,07,17,03,00,00 TRACE=WAS.messaging=all DELAY=+600 "TRACE=*=info" DELAY=+300 JAVACORE


    Example 4

    Connect to a cell deployment manager and choose a list of target servers, then turn on tracing immediately. Wait one minute and generate a series of three javacores, each a minute apart, and finally turn off tracing after another minute:



    sh> ./launchClient.sh /full/path/to/TrapIt.ear dmgr.example.com:2811 DELAY=0 TRACE=Transactions=all:WAS.j2c=all +DELAY=60 JAVACORE=3,60 DELAY=+60 "TRACE=*=info"


    Example 5

    Watch for message HMGR0152W, then run solperf.sh against the application server's process identifier. Stop watching for the message after triggering three times.



    sh> ./launchClient.sh /full/path/to/TrapIt.ear sun1.example.com:2811 EVENT=3,HMGR0152W "EXEC=/full/path/to/solperf.sh 230937,/tmp/solperf.out,/tmp/solperf.err"

    Back to top




Sample Output

The following output shows the TrapIt.ear watching for message "WMSG0019E" and triggering a javacore when the message appears. As the timestamps in the TrapIt.ear output show, after printing the message "Setting Trigger for log events [WMSG0019E]. Action: JAVACORE" the TrapIt.ear command waited for nearly six minutes before finding a match


     

    TrapIt.ear Output

    sh>  ./launchClient.sh /scratch/TrapIt.ear 192.168.1.2:2809 EVENT=1,WMSG0019E JAVACORE


    IBM WebSphere Application Server, Release 7.0
    Java™ EE Application Client Tool
    Copyright IBM Corp., 1997-2008
    WSCL0012I: Processing command line arguments.
    WSCL0013I: Initializing the Java EE Application Client Environment.
    WSCL0035I: Initialization of the Java EE Application Client Environment has completed.
    WSCL0014I: Invoking the Application Client class com.ibm.ws.support.trapit.TrapIt
    [20130731 17:37:33] TrapIt: WebSphere Application Server Log Watcher
    [20130731 17:37:33] TrapIt: © Copyright IBM Corp. 2006, 2013
    [20130731 17:37:33] TrapIt: Version: 1.0.1.20130731
    [20130731 17:37:33] TrapIt: Connecting on "192.168.1.2:2809"
    [20130731 17:37:33] TrapIt: Connected to "Server1" PID=29819056
    [20130731 17:37:33] TrapIt: "Server1": Setting Trigger for log events [WMSG0019E]. Action: JAVACORE
    [20130725 17:43:22] TrapIt: Server1: MSG MATCH: WMSG0019E: Unable to start MDB Listener SampleMDB, JMSDestination jms/SampleMDBTopic : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'TEST.QMGR' with connection mode 'Client' and host name '192.168.1.3(1422)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
            at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:496)
            at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
            at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:451)
            at com.ibm.msg.client.wmq.internal.WMQXAConnection.<init>(WMQXAConnection.java:70)
            at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:190)
            at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6579)
            at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createProviderXAConnection(WMQXAConnectionFactory.java:102)
            at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createXAConnectionInternal(JmsConnectionFactoryImpl.java:388)
            at com.ibm.mq.jms.MQXAConnectionFactory.createXAConnection(MQXAConnectionFactory.java:103)
            at com.ibm.ejs.jms.JMSManagedConnection.createConnection(JMSManagedConnection.java:1558)
            at com.ibm.ejs.jms.JMSManagedConnection.<init>(JMSManagedConnection.java:352)
            ...
    [20130731 17:43:22] TrapIt: Invoking "dumpThreads" on "JVM" for "Server1". Params=[]
    [20130731 17:43:24] TrapIt: Disabled event listener on "Server1" for action JAVACORE
    [20130731 17:43:24] TrapIt: Exiting

Back to top



Source Code

The source code for TrapIt.ear is included, in case you wish to extend the existing functionality of the tool. For example, to add a new action that fires once and completes, refer to:




    com.ibm.ws.support.trapit.actions.HeapdumpAction



To add a new action that fires multiple times at a regular interval, use the following class as a start point:


    com.ibm.ws.support.trapit.actions.JavacoreAction



To add a new trigger, start with the following class:


    com.ibm.ws.support.trapit.triggers.TimeTrigger


Back to top




Feedback

A discussion thread for the TrapIt.ear tool is available on the IBM developerWorks site. If you have any comments or suggestions for TrapIt.ear, please submit them at:



Back to top

DISCLAIMER: All source code and/or binaries attached to this document are referred to here as "the Program". IBM is not providing program services of any kind for the Program. IBM is providing the Program on an "AS IS" basis without warranty of any kind. IBM WILL NOT BE LIABLE FOR ANY ACTUAL, DIRECT, SPECIAL, INCIDENTAL, OR INDIRECT DAMAGES OR FOR ANY ECONOMIC CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS OR SAVINGS), EVEN IF IBM, OR ITS RESELLER, HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"PD tools (for example Log Analyzer)","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"},{"code":"PF035","label":"z\/OS"}],"Version":"9.0;8.5.5;8.0;7.0","Edition":"Base;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21644180

Page Feedback