IBM Support

PI88622: ON DEMAND CONFIGURATION (ODC) THREAD CAUSES HANG IN DEPLOYMENT MANAGER (DMGR)

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • WAS ND 8.5.5.12 on Linux
    
    Thread Name Thread-62
    State Waiting on condition
    
    Monitor
    Owns Monitor Lock on
    com/ibm/ws/odc/nd/ODCTreeImpl@0x0000000080BC35F8
    Java Stack
    at
    com/ibm/apc/xd/demand/predictors/APCXDDemandPredictorShim$1.
    cell
    Added(APCXDDemandPredictorShim.java:59)
    at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCHelper_Notific
    atio
    n.notifyListeners_Cell(XdCsODCHelper_Notification.java:309)
    at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCSensor$1.handl
    eCha
    nge_CellAddRemove(XdCsODCSensor.java:566)
    at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCDeMuxImpl.demu
    xHan
    dler_ODCEventNodeChange(XdCsODCDeMuxImpl.java:246(Compiled
    Code))
    at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCDeMuxImpl.rout
    eOdc
    ConfigChangeEvent(XdCsODCDeMuxImpl.java:113(Compiled Code))
    at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCSensor$2.handl
    eEve
    nt(XdCsODCSensor.java:680(Compiled Code))
    at
    com/ibm/ws/odc/nd/ODCTreeImpl.commitTransaction(ODCTreeImpl.
    java
    :1182(Compiled Code))
    at
    com/ibm/ws/odc/nd/ODCTreeImpl.commitTransaction(ODCTreeImpl.
    java
    :959)
    at
    com/ibm/ws/odc/nd/ODCTreeImpl.performEvents(ODCTreeImpl.java
    :275
    )
    at
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.performEvents(ODCGroup.ja
    va:1
    637)
    at
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.handleGetTreeResponse(ODC
    Grou
    p.java:1385(Compiled Code))
    at
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.consumerOnMessage(ODCGrou
    p.ja
    va:550(Compiled Code))
    at
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.processMessage(ODCGroup.j
    ava:
    725(Compiled Code))
    at
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.onMessage(ODCGroup.java:6
    82(C
    ompiledCode))
    at
    com/ibm/ws/wsgroup/WsGroup.deliverMessage(WsGroup.java:219(C
    ompi
    ledCode))
    at
    com/ibm/ws/wsgroup/bb/CombinedGroup.onP2PMessage(CombinedGro
    up.j
    ava:294(Compiled Code))
    at
    com/ibm/ws/wsgroup/bb/CombinedGroup$MyP2PGroupListener.onMes
    sage
    (CombinedGroup.java:449(Compiled Code))
    at
    com/ibm/ws/wsgroup/WsGroup.deliverMessage(WsGroup.java:219(C
    ompi
    ledCode))
    at
    com/ibm/ws/wsgroup/WsGroup.receiveMessage(WsGroup.java:212(C
    ompi
    ledCode))
    at
    com/ibm/ws/wsgroup/WsGroup.receiveMessages(WsGroup.java:194(
    Comp
    iledCode))
    at
    com/ibm/ws/wsgroup/p2p/P2PGroup.calledWhenMessageArrives(P2P
    Grou
    p.java:209(Compiled Code))
    at
    com/ibm/ws/wsgroup/p2p/SonP2PShimImpl$Worker.oneStep(SonP2PS
    himI
    mpl.java:1347(Compiled Code))
    at
    com/ibm/ws/wsgroup/p2p/SonP2PShimImpl$Worker.run(SonP2PShimI
    mpl.
    java:1213)
    at
    java/lang/Thread.run(Thread.java:785)
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server WAS ND edition- Virtual              *
    *                  Enterprise/IM Component                     *
    ****************************************************************
    * PROBLEM DESCRIPTION: Hung thread attempting to initialize    *
    *                      com/ibm/apc/xd/demand/predictors/APCXDD *
    *                      emandPredictorShim.<clinit>.            *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    In dmgr or nodeagent processes a thread is hung attempting to
    initialize
    com/ibm/apc/xd/demand/predictors/APCXDDemandPredictorShim.<clini
    t>(APCXDDemandPredictorShim.java:69) and is blocked waiting on
    com/ibm/ws/odc/nd/ODCTreeImpl. The signature of the deadlock
    will reflect the following two thread snippets:
    
    3XMTHREADINFO      "Thread-62" J9VMThread:0x0000000002CE3F00,
    omrthread_t:0x00007FC45441BBB8,
    java/lang/Thread:0x00000000883CF120, state:B, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x76,
    isDaemon:false)
    3XMTHREADINFO1            (native thread ID:0x331B, native
    priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread
    flags:0x00000201)
    3XMTHREADINFO2            (native stack address range
    from:0x00007FC3EDE89000, to:0x00007FC3EDEC9000, size:0x40000)
    3XMCPUTIME               CPU usage total: 0.020904504 secs,
    current category="Application"
    3XMTHREADBLOCK     Blocked on:
    com/ibm/ws/odc/nd/ODCTreeImpl@0x00000000877B9E58 Owned by:
    "P=674747:O=0:CT" (J9VMThread:0x0000000000BC4700,
    java/lang/Thread:0x0000000085DB3CA0)
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCSensor.start(XdCsO
    DCSensor.java:804)
    4XESTACKTRACE                at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCSensor.getInstance
    (XdCsODCSensor.java:772)
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCSensor@0x000000008
    7541798, entry count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/xd/container/state/odc/impl/XdCsSubscriptionManagerFa
    ctory.resolve(XdCsSubscriptionManagerFactory.java:49)
    4XESTACKTRACE                at
    com/ibm/ws/xd/container/state/odc/impl/XdCsODCHelper_Subscriptio
    n.subscribe(XdCsODCHelper_Subscription.java:69)
    5XESTACKTRACE                   (entered lock:
    com/ibm/apc/xd/demand/predictors/APCXDDemandPredictorShim$1@0x00
    000000883D87D8, entry count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/xd/container/state/odc/XdCsListener_Cell.subscribe(Xd
    CsListener_Cell.java:46)
    4XESTACKTRACE                at
    com/ibm/apc/xd/demand/predictors/APCXDDemandPredictorShim.<clini
    t>(APCXDDemandPredictorShim.java:69)
    
    3XMTHREADINFO      "P=674747:O=0:CT"
    J9VMThread:0x0000000000BC4700, omrthread_t:0x00007FC474007FD0,
    java/lang/Thread:0x0000000085DB3CA0, state:CW, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x1,
    isDaemon:false)
    3XMTHREADINFO1            (native thread ID:0x3259, native
    priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
    flags:0x00000101)
    3XMTHREADINFO2            (native stack address range
    from:0x00007FC47B885000, to:0x00007FC47C085000, size:0x800000)
    3XMCPUTIME               CPU usage total: 8.928669194 secs,
    current category="Application"
    3XMTHREADBLOCK     Waiting on:
    java/lang/J9VMInternals$ClassInitializationLock@0x000000008826EC
    68 Owned by: <unowned>
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    sun/misc/Unsafe.ensureClassInitialized(Native Method)
    4XESTACKTRACE                at
    java/lang/J9VMInternals.initialize(J9VMInternals.java:87(Compile
    d Code))
    

Problem conclusion

  • The Application Placement Controller (APC) code was altered such
    that it no longer requires obtaining locks within global static
    blocks of
    com.ibm.apc.xd.demand.predictors.APCXDDemandPredictorShim which
    execute during class initialization, avoiding the possibility
    for deadlock in this area.
    
    The fix for this APAR is currently targeted for inclusion in fix
    packs 8.5.5.15 and 9.0.0.10.  Please refer to the Recommended
    Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI88622

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-10-10

  • Closed date

    2018-09-10

  • Last modified date

    2018-09-10

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 April 2022