IBM Support

PI85490: DEADLOCK CAUSED BY WSLOGMANAGER AND SIB TRACE CODE

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • When setting trace at runtime, a deadlock might happen with the
    following thread stack in the java dump file, indicating a lock
    contention when WsLogManager tries to set trace at runtime
    while SIB debug code is locking the code.
    1LKDEADLOCK    Deadlock detected !!!
    NULL           ---------------------
    NULL
    2LKDEADLOCKTHR  Thread "Default : 8" (0x0000000052C04700)
    3LKDEADLOCKWTR    is waiting for:
    4LKDEADLOCKMON      sys_mon_t:0x00000100198EA4F8 infl_mon_t:
    0x00000100198EA540:
    4LKDEADLOCKOBJ
    com/ibm/ws/wmqcsi/trace/TraceImpl@0x00000000414F1090
    3LKDEADLOCKOWN    which is owned by:
    2LKDEADLOCKTHR  Thread "Thread-161" (0x0000000052C7E000)
    3LKDEADLOCKWTR    which is waiting for:
    4LKDEADLOCKMON      sys_mon_t:0x000001001963C3D8 infl_mon_t:
    0x000001001963C420:
    4LKDEADLOCKOBJ
    com/ibm/ws/bootstrap/WsLogManager@0x0000000040DDDB50
    3LKDEADLOCKOWN    which is owned by:
    2LKDEADLOCKTHR  Thread "Default : 8" (0x0000000052C04700)
    
    3XMTHREADINFO      "Default : 8" J9VMThread:0x0000000052C04700,
    j9thread_t:0x000001001830FAA0,
    java/lang/Thread:0x00000000438BA838,
    state:B, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x9E,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x5470091, native
    priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread
    flags:0x00000201)
    3XMCPUTIME               CPU usage total: 1.024461000 secs,
    user:
    0.638929000 secs, system: 0.385532000 secs
    3XMTHREADBLOCK     Blocked on:
    com/ibm/ws/wmqcsi/trace/TraceImpl@0x00000000414F1090 Owned by:
    "Thread-161" (J9VMThread:0x0000000052C7E000,
    java/lang/Thread:0x00000000438906E0)
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0
    (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/msg/client/commonservices/trace/Trace.setOn(Trace.java:5
    15)
    5XESTACKTRACE                   (entered lock:
    com/ibm/msg/client/commonservices/trace/Trace@0x0000000040D58A20
    , entry
    count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/wmqcsi/trace/TraceImpl$1.traceStateChanged(TraceImpl.
    java:232
    )
    4XESTACKTRACE                at
    com/ibm/ejs/ras/TraceComponent.applyTraceEnableEvent(TraceCompon
    ent.java
    :163(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/Group.applyTraceEnableEvent(Group.java:112(Compi
    led
    Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/ComponentManager.fireTraceEnableEvent(ComponentM
    anager.j
    ava:786)
    4XESTACKTRACE                at
    com/ibm/ejs/ras/ComponentManager.processLoggingSpecString(Compon
    entManag
    er.java:1125)
    4XESTACKTRACE                at
    com/ibm/ejs/ras/ComponentManager.processTraceString(ComponentMan
    ager.jav
    a:380)
    
    3XMTHREADINFO      "Thread-161" J9VMThread:0x0000000052C7E000,
    j9thread_t:0x000001001556CEA0,
    java/lang/Thread:0x00000000438906E0,
    state:B, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x10D,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x6EE00A1, native
    priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread
    flags:0x00000201)
    3XMCPUTIME               CPU usage total: 24.078727000 secs,
    user:
    19.058190000 secs, system: 5.020537000 secs
    3XMTHREADBLOCK     Blocked on:
    com/ibm/ws/bootstrap/WsLogManager@0x0000000040DDDB50 Owned by:
    "Default
    : 8" (J9VMThread:0x0000000052C04700,
    java/lang/Thread:0x00000000438BA838)
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0
    (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    java/util/logging/LogManager.initializeGlobalHandlers(LogManager
    .java:14
    07(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/bootstrap/WsLogManager@0x0000000040DDDB50, entry
    count: 1)
    4XESTACKTRACE                at
    java/util/logging/LogManager.access$1500(LogManager.java:160(Com
    piled
    Code))
    4XESTACKTRACE                at
    java/util/logging/LogManager$RootLogger.accessCheckedHandlers(Lo
    gManager
    .java:1506(Compiled Code))
    4XESTACKTRACE                at
    java/util/logging/Logger.getHandlers(Logger.java:1362(Compiled
    Code))
    4XESTACKTRACE                at
    java/util/logging/Logger.log(Logger.java:624(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/Tr.logToJSR47Logger(Tr.java:1720(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/Tr.fireEvent(Tr.java:1675(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/Tr.fireTraceEvent(Tr.java:1594(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ejs/ras/Tr.debug(Tr.java:651(Compiled Code))
    
    See Liberty issue:
    https://github.ibm.com/was-liberty/WS-CD-Open/issues/7279
    Se Liberty pull request:
    https://github.ibm.com/was-liberty/WS-CD-Open/pull/7280
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server who enable MQ Resource adapter       *
    *                  tracing.                                    *
    *                  Also fixed in Liberty 17.0.0.3              *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock between WslogManager and Sib   *
    *                      MQ                                      *
    *                      client trace.                           *
    ****************************************************************
    * RECOMMENDATION:  This is hard to reproduce and only rarely   *
    *                  occurs,                                     *
    *                  if necessary apply the fix.                 *
    ****************************************************************
    A dump will show a server thread locked at
    com/ibm/msg/client/commonservices/trace/Trace.setOn(Trace.java
    :5
    15)
    

Problem conclusion

Temporary fix

Comments

APAR Information

  • APAR number

    PI85490

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-08-03

  • Closed date

    2017-09-26

  • Last modified date

    2017-10-10

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP

  • R850 PSY

       UP

  • R900 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 850

Reference #: PI85490

Modified date: 10 October 2017