PM63622: WLM THREAD DEADLOCK IN NODEAGENT ON STARTUP

Fixes are available

APAR status

Closed as program error.

Error description

Description:


During the startup of the nodeagent processes, and if there are
currently EJB requests being sent to that nodeagent, there is a
timing
window between the startup code paths and the request handling
code
paths which can cause a thread deadlock to occur.  The two
threads would
show the following stack traces:

3XMTHREADINFO      "P=611144:O=0:CT" (TID:0x000000000F54E200,
sys_thread_t:0x000000000F5354A0, state:B, native
ID:0x0000000000004075)
prio=5
4XESTACKTRACE          at
com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
r.setBul
letinBoardScopes(BBDescriptionManager.java:173)
4XESTACKTRACE          at
com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
r.publis
h(BBDescriptionManager.java:242)
4XESTACKTRACE          at
com/ibm/ws/cluster/topography/DescriptionManagerA.publish(Descri
ptionMan
agerA.java:346)
4XESTACKTRACE          at
com/ibm/websphere/cluster/topography/DescriptionA.updateMemento(
Descript
ionA.java:159)
4XESTACKTRACE          at
com/ibm/ws/cluster/service/ClusterMemberServiceImpl.start(Cluste
rMemberS
erviceImpl.java:302)
4XESTACKTRACE          at
com/ibm/ws/runtime/component/ContainerImpl.startComponents(Conta
inerImpl
.java:977)
4XESTACKTRACE          at
com/ibm/ws/runtime/component/ContainerImpl.start(ContainerImpl.j
ava:673)

3XMTHREADINFO      "ORB.thread.pool : 0"
(TID:0x0000000010C69700,
sys_thread_t:0x0000000010BE5D80, state:B, native
ID:0x00000000000040D8)
prio=5
4XESTACKTRACE          at
com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
r.subscr
ibe(BBDescriptionManager.java:644)
4XESTACKTRACE          at
com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
r.getDes
cription(BBDescriptionManager.java:206)
4XESTACKTRACE          at
com/ibm/ws/cluster/topography/ClusterDescriptionImpl.<clinit>(Cl
usterDes
criptionImpl.java:170)
4XESTACKTRACE          at
java/lang/J9VMInternals.initializeImpl(Native
Method)
4XESTACKTRACE          at
java/lang/J9VMInternals.initialize(J9VMInternals.java:196(Compil
ed
Code))
4XESTACKTRACE          at
sun/reflect/NativeConstructorAccessorImpl.newInstance0(Native
Method)
4XESTACKTRACE          at
sun/reflect/NativeConstructorAccessorImpl.newInstance(NativeCons
tructorA
ccessorImpl.java:67)
4XESTACKTRACE          at
sun/reflect/DelegatingConstructorAccessorImpl.newInstance(Delega
tingCons
tructorAccessorImpl.java:45)
4XESTACKTRACE          at
java/lang/reflect/Constructor.newInstance(Constructor.java:522)
4XESTACKTRACE          at
com/ibm/websphere/cluster/topography/DescriptionFactory.createDe
scriptio
n(DescriptionFactory.java:196)
4XESTACKTRACE          at
com/ibm/ws/cluster/topography/DescriptionManagerA.getDescription
(Descrip
tionManagerA.java:144)
4XESTACKTRACE          at
com/ibm/ws/cluster/propagation/bulletinboard/BBDescriptionManage
r.getDes
cription(BBDescriptionManager.java:205)
4XESTACKTRACE          at
com/ibm/ws/cluster/selection/SelectionCriteriaImpl.<init>(Select
ionCrite
riaImpl.java:143)
4XESTACKTRACE          at
com/ibm/ws/cluster/selection/SelectionServiceImpl.getCriteria(Se
lectionS
erviceImpl.java:266)
4XESTACKTRACE          at
com/ibm/ws/cluster/router/selection/WLMLSDRouter.getCriteria(WLM
LSDRoute
r.java:384)
4XESTACKTRACE          at
com/ibm/ws/cluster/router/selection/WLMLSDRouter.select(WLMLSDRo
uter.jav
a:250)
4XESTACKTRACE          at
com/ibm/ws/cluster/router/selection/WLMLSDRouter.getNextTarget(W
LMLSDRou
ter.java:216)
4XESTACKTRACE          at
com/ibm/ws/cluster/router/selection/WLMClientForCommonRouterImpl
.getNext
Target(WLMClientForCommonRouterImpl.java:306)
4XESTACKTRACE          at
com/ibm/ws/wlm/client/WLMClient.getNextTarget(WLMClient.java:202
)
4XESTACKTRACE          at
com/ibm/ws/orbimpl/services/lsd/LocationServiceDaemon.getDirectI
OR(Locat
ionServiceDaemon.java:197)

Local fix

```
N/A
```

Problem summary

****************************************************************
* USERS AFFECTED:  Users of IBM WebSphere Application Server   *
*                  Network Deployment edition                  *
****************************************************************
* PROBLEM DESCRIPTION: Thread deadlock on startup of           *
*                      nodeagent                               *
*                      process                                 *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
If requests are being sent to a nodeagent while it is in the
process of starting up, the code which handles the incoming
requests may run before the startup is complete, resulting in
locks being held which are required for the startup process,
ultimately causing a thread deadlock.

Problem conclusion

Additional synchronization logic was adding to prevent the
thread deadlock during the startup of the nodeagent in the path
where requests are also being handled.

The fix for this APAR is currently targeted for inclusion in
fix packs 6.1.0.45, 7.0.0.25, 8.0.0.5, and 8.5.0.1,  Please
refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

If possible, avoid sending requests to a nodeagent process as it
starting up and the deadlock cannot be hit

Comments

APAR Information

APAR number
PM63622
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
61I
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2012-04-30
Closed date
2012-06-25
Last modified date
2012-07-17

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800

Applicable component levels

R61A PSY
UP
R61H PSY
UP
R61I PSY
UP
R61P PSY
UP
R61S PSY
UP
R61W PSY
UP
R700 PSY
UP
R800 PSY
UP
R850 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 October 2021

Tips

PM63622: WLM THREAD DEADLOCK IN NODEAGENT ON STARTUP

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R61A PSY

R61H PSY

R61I PSY

R61P PSY

R61S PSY

R61W PSY

R700 PSY

R800 PSY

R850 PSY

Document Information

Share your feedback

Need support?