IBM Support

PM63622: WLM THREAD DEADLOCK IN NODEAGENT ON STARTUP

Fixes are available

7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.5.0.1: WebSphere Application Server V8.5 Fix Pack 1
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Description:
    
    
    During the startup of the nodeagent processes, and if there are
    currently EJB requests being sent to that nodeagent, there is a
    timing
    window between the startup code paths and the request handling
    code
    paths which can cause a thread deadlock to occur.  The two
    threads would
    show the following stack traces:
    
    3XMTHREADINFO      "P=611144:O=0:CT" (TID:0x000000000F54E200,
    sys_thread_t:0x000000000F5354A0, state:B, native
    ID:0x0000000000004075)
    prio=5
    4XESTACKTRACE          at
    com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
    r.setBul
    letinBoardScopes(BBDescriptionManager.java:173)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
    r.publis
    h(BBDescriptionManager.java:242)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/topography/DescriptionManagerA.publish(Descri
    ptionMan
    agerA.java:346)
    4XESTACKTRACE          at
    com/ibm/websphere/cluster/topography/DescriptionA.updateMemento(
    Descript
    ionA.java:159)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/service/ClusterMemberServiceImpl.start(Cluste
    rMemberS
    erviceImpl.java:302)
    4XESTACKTRACE          at
    com/ibm/ws/runtime/component/ContainerImpl.startComponents(Conta
    inerImpl
    .java:977)
    4XESTACKTRACE          at
    com/ibm/ws/runtime/component/ContainerImpl.start(ContainerImpl.j
    ava:673)
    
    3XMTHREADINFO      "ORB.thread.pool : 0"
    (TID:0x0000000010C69700,
    sys_thread_t:0x0000000010BE5D80, state:B, native
    ID:0x00000000000040D8)
    prio=5
    4XESTACKTRACE          at
    com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
    r.subscr
    ibe(BBDescriptionManager.java:644)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
    r.getDes
    cription(BBDescriptionManager.java:206)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/topography/ClusterDescriptionImpl.<clinit>(Cl
    usterDes
    criptionImpl.java:170)
    4XESTACKTRACE          at
    java/lang/J9VMInternals.initializeImpl(Native
    Method)
    4XESTACKTRACE          at
    java/lang/J9VMInternals.initialize(J9VMInternals.java:196(Compil
    ed
    Code))
    4XESTACKTRACE          at
    sun/reflect/NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    4XESTACKTRACE          at
    sun/reflect/NativeConstructorAccessorImpl.newInstance(NativeCons
    tructorA
    ccessorImpl.java:67)
    4XESTACKTRACE          at
    sun/reflect/DelegatingConstructorAccessorImpl.newInstance(Delega
    tingCons
    tructorAccessorImpl.java:45)
    4XESTACKTRACE          at
    java/lang/reflect/Constructor.newInstance(Constructor.java:522)
    4XESTACKTRACE          at
    com/ibm/websphere/cluster/topography/DescriptionFactory.createDe
    scriptio
    n(DescriptionFactory.java:196)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/topography/DescriptionManagerA.getDescription
    (Descrip
    tionManagerA.java:144)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
    r.getDes
    cription(BBDescriptionManager.java:205)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/selection/SelectionCriteriaImpl.<init>(Select
    ionCrite
    riaImpl.java:143)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/selection/SelectionServiceImpl.getCriteria(Se
    lectionS
    erviceImpl.java:266)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/router/selection/WLMLSDRouter.getCriteria(WLM
    LSDRoute
    r.java:384)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/router/selection/WLMLSDRouter.select(WLMLSDRo
    uter.jav
    a:250)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/router/selection/WLMLSDRouter.getNextTarget(W
    LMLSDRou
    ter.java:216)
    4XESTACKTRACE          at
    com/ibm/ws/cluster/router/selection/WLMClientForCommonRouterImpl
    .getNext
    Target(WLMClientForCommonRouterImpl.java:306)
    4XESTACKTRACE          at
    com/ibm/ws/wlm/client/WLMClient.getNextTarget(WLMClient.java:202
    )
    4XESTACKTRACE          at
    com/ibm/ws/orbimpl/services/lsd/LocationServiceDaemon.getDirectI
    OR(Locat
    ionServiceDaemon.java:197)
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  Users of IBM WebSphere Application Server   *
    *                  Network Deployment edition                  *
    ****************************************************************
    * PROBLEM DESCRIPTION: Thread deadlock on startup of           *
    *                      nodeagent                               *
    *                      process                                 *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    If requests are being sent to a nodeagent while it is in the
    process of starting up, the code which handles the incoming
    requests may run before the startup is complete, resulting in
    locks being held which are required for the startup process,
    ultimately causing a thread deadlock.
    

Problem conclusion

  • Additional synchronization logic was adding to prevent the
    thread deadlock during the startup of the nodeagent in the path
    where requests are also being handled.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 6.1.0.45, 7.0.0.25, 8.0.0.5, and 8.5.0.1,  Please
    refer to the Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

  • If possible, avoid sending requests to a nodeagent process as it
    starting up and the deadlock cannot be hit
    

Comments

APAR Information

  • APAR number

    PM63622

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    61I

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-04-30

  • Closed date

    2012-06-25

  • Last modified date

    2012-07-17

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R61A PSY

       UP

  • R61H PSY

       UP

  • R61I PSY

       UP

  • R61P PSY

       UP

  • R61S PSY

       UP

  • R61W PSY

       UP

  • R700 PSY

       UP

  • R800 PSY

       UP

  • R850 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 6.1

Reference #: PM63622

Modified date: 17 July 2012