PM61376: EXCESSIVE THREADS ON SYNC OPERATION WAITING FOR ONE BLOCKED THREAD FROM ACTIVE SYNC WITH SOCKETINPUTSTREAM.SOCKETREAD0.

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• The sync requests should not be allowed after one blocked thread
  on active sync awaiting for a response from DMgr. In this case,
  there was a total of 1299 threads of the same stack below in the
  NodeAgent address space that caused OOM and brought down
  Nodeagent.
  .
  "SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
  sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:199)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
    pareFolder(NodeSync.java:1200)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
    java:1133)
    at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
      ...
  .
  The same java stack was also displayed on the hung threads in
  the NA log with  WSVR0605W messages:
  .
  ￭12/20/11 16:57:58:926 EST 0000491b ThreadMonitor W
  BBOO0221W: WSVR0605W: Thread "SoapConnectorThreadPool : 13172"
  (00004906) has been active for 715000 milliseconds and may be
  hung.  There is/are 1033 thread(s) in total in the server that
  may be hung.
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:199)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
    pareFolder(NodeSync.java:1200)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
    java:1133)
    at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
     ...
  .
  All threads above were waiting for one blocked thread on active
  sync below doing SocketInputStream.socketRead0 and awaiting a
  response from DMgr.
  "NodeSync" (TID:0x3EEF3800, sys_thread_t:0x3EEF1808, state:R,
  native ID:0x21AF9000) prio=5
    at java/net/SocketInputStream.socketRead0(Native Method)
    at
  java/net/SocketInputStream.read(SocketInputStream.java:155(Compi
  led Code))
    at com/ibm/jsse2/a.a(a.java:188(Compiled Code))
    at com/ibm/jsse2/a.a(a.java:232(Compiled Code))
    at com/ibm/jsse2/pc.a(pc.java:211(Compiled Code))
    at com/ibm/jsse2/pc.g(pc.java:376(Compiled Code))
    at com/ibm/jsse2/pc.a(pc.java:573(Compiled Code))
    at com/ibm/jsse2/pc.startHandshake(pc.java:37(Compiled Code))
    at
  org/apache/soap/util/net/SSLUtils.buildSSLSocket(SSLUtils.java:2
  85(Compiled Code))
    at sun/reflect/GeneratedMethodAccessor9.invoke(Bytecode
  PC:220(Compiled Code))
    at
  sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:43(Compiled Code))
    at java/lang/reflect/Method.invoke(Method.java:618(Compiled
  Code))
    at
  org/apache/soap/util/net/HTTPUtils.buildSocket(HTTPUtils.java:26
  1(Compiled Code))
    at
  org/apache/soap/util/net/HTTPUtils.post(HTTPUtils.java:474(Compi
  led Code))
    at
  org/apache/soap/transport/http/SOAPHTTPConnection.send(SOAPHTTPC
  onnection.java:404(Compiled Code))
    at org/apache/soap/rpc/Call.WASinvoke(Call.java:433(Compiled
  Code))
    at
  com/ibm/ws/management/connector/soap/SOAPConnectorClient$8.run(S
  OAPConnectorClient.java:720(Compiled Code))
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server                                      *
  *                  V7.0                                        *
  ****************************************************************
  * PROBLEM DESCRIPTION: In the presence of another issue that   *
  *                      caused SSL handshakes to hang           *
  *                      indefinitely, repeated                  *
  *                      synchronization requests continued to   *
  *                      be dispatched to new threads, all of    *
  *                      which were waiting on the hung          *
  *                      thread. Eventually, node agent          *
  *                      ran out of memory.                      *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  In the presence of another issue that caused SSL handshakes to
  hang indefinitely, repeated synchronization requests continued
  to be dispatched to new threads, all of which were waiting on
  the hung thread.  The node agent eventually ran out of memory.
  A thread dump showed many SOAP connector threads all waiting
  at:
  "SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
  sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:199)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
  pareFolder(NodeSync.java:1200)
  at
  com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
  java:1133)
  at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
  

Problem conclusion

• The code was modified to better manage existing and new
  synchronization operations and avoid out of memory errors from
  an excessive number of waiting threads.
  
  APAR PM61379 is currently targeted for inclusion WebSphere
  Application Server V7.0 Fix Packs 7.0.0.25.
  
  Please refer to URL:
  //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
  for Fix Pack availability.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PM61079

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE FOR Z

• Fixed component ID

  5655I3500

Applicable component levels

• R700 PSY UK81738

     UP12/10/04 P F210

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.