PM61079: EXCESSIVE THREADS ON SYNC OPERATION WAITING FOR ONE BLOCKED THREAD FROM ACTIVE SYNC WITH SOCKETINPUTSTREAM.SOCKETREAD0.

Fixes are available

APAR status

Closed as program error.

Error description

The sync requests should not be allowed after one blocked thread
on active sync awaiting for a response from DMgr. In this case,
there was a total of 1299 threads of the same stack below in the
NodeAgent address space that caused OOM and brought down
Nodeagent.
.
"SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:199)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
  pareFolder(NodeSync.java:1200)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
  java:1133)
  at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
    ...
.
The same java stack was also displayed on the hung threads in
the NA log with  WSVR0605W messages:
.
&#65517;12/20/11 16:57:58:926 EST&#65529; 0000491b ThreadMonitor W
BBOO0221W: WSVR0605W: Thread "SoapConnectorThreadPool : 13172"
(00004906) has been active for 715000 milliseconds and may be
hung.  There is/are 1033 thread(s) in total in the server that
may be hung.
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:199)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
  pareFolder(NodeSync.java:1200)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
  java:1133)
  at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
   ...
.
All threads above were waiting for one blocked thread on active
sync below doing SocketInputStream.socketRead0 and awaiting a
response from DMgr.
"NodeSync" (TID:0x3EEF3800, sys_thread_t:0x3EEF1808, state:R,
native ID:0x21AF9000) prio=5
  at java/net/SocketInputStream.socketRead0(Native Method)
  at
java/net/SocketInputStream.read(SocketInputStream.java:155(Compi
led Code))
  at com/ibm/jsse2/a.a(a.java:188(Compiled Code))
  at com/ibm/jsse2/a.a(a.java:232(Compiled Code))
  at com/ibm/jsse2/pc.a(pc.java:211(Compiled Code))
  at com/ibm/jsse2/pc.g(pc.java:376(Compiled Code))
  at com/ibm/jsse2/pc.a(pc.java:573(Compiled Code))
  at com/ibm/jsse2/pc.startHandshake(pc.java:37(Compiled Code))
  at
org/apache/soap/util/net/SSLUtils.buildSSLSocket(SSLUtils.java:2
85(Compiled Code))
  at sun/reflect/GeneratedMethodAccessor9.invoke(Bytecode
PC:220(Compiled Code))
  at
sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
AccessorImpl.java:43(Compiled Code))
  at java/lang/reflect/Method.invoke(Method.java:618(Compiled
Code))
  at
org/apache/soap/util/net/HTTPUtils.buildSocket(HTTPUtils.java:26
1(Compiled Code))
  at
org/apache/soap/util/net/HTTPUtils.post(HTTPUtils.java:474(Compi
led Code))
  at
org/apache/soap/transport/http/SOAPHTTPConnection.send(SOAPHTTPC
onnection.java:404(Compiled Code))
  at org/apache/soap/rpc/Call.WASinvoke(Call.java:433(Compiled
Code))
  at
com/ibm/ws/management/connector/soap/SOAPConnectorClient$8.run(S
OAPConnectorClient.java:720(Compiled Code))

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V6.1, V7.0, V8.0, and V8.5           *
****************************************************************
* PROBLEM DESCRIPTION: In the presence of another issue that   *
*                      caused SSL handshakes to hang           *
*                      indefinitely, repeated                  *
*                      synchronization requests continued to   *
*                      be dispatched to new threads, all of    *
*                      which were waiting on the hung          *
*                      thread. Eventually, node agent          *
*                      ran out of memory.                      *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
In the presence of another issue that caused SSL handshakes to
hang indefinitely, repeated synchronization requests continued
to be dispatched to new threads, all of which were waiting on
the hung thread.  The node agent eventually ran out of memory.
A thread dump showed many SOAP connector threads all waiting
at:
"SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:199)
at
com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
pareFolder(NodeSync.java:1200)
at
com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
java:1133)
at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)

Problem conclusion

The code was modified to better manage existing and new
synchronization operations and avoid out of memory errors from
an excessive number of waiting threads.

APAR PM61079 is currently targeted for inclusion WebSphere
Application Server Fix Packs 6.1.0.45, 7.0.0.25, 8.0.0.5,
and V8.5.0.1.

The fix will also be delivered as sysrouted APAR PM61376 in
WebSphere Application Server V7.0 Fix Pack 7.0.0.25 for z/OS.

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM61079
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2012-03-23
Closed date
2012-07-10
Last modified date
2013-10-21

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM61376

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK81646
UP12/09/23 P F209

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 October 2021

Tips

PM61079: EXCESSIVE THREADS ON SYNC OPERATION WAITING FOR ONE BLOCKED THREAD FROM ACTIVE SYNC WITH SOCKETINPUTSTREAM.SOCKETREAD0.

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK81646

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?