IBM Support

PM61079: EXCESSIVE THREADS ON SYNC OPERATION WAITING FOR ONE BLOCKED THREAD FROM ACTIVE SYNC WITH SOCKETINPUTSTREAM.SOCKETREAD0.

Fixes are available

7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.5.0.1: WebSphere Application Server V8.5 Fix Pack 1
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • The sync requests should not be allowed after one blocked thread
    on active sync awaiting for a response from DMgr. In this case,
    there was a total of 1299 threads of the same stack below in the
    NodeAgent address space that caused OOM and brought down
    Nodeagent.
    .
    "SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
    sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:199)
      at
      com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
      pareFolder(NodeSync.java:1200)
      at
      com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
      java:1133)
      at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
        ...
    .
    The same java stack was also displayed on the hung threads in
    the NA log with  WSVR0605W messages:
    .
    ■12/20/11 16:57:58:926 EST 0000491b ThreadMonitor W
    BBOO0221W: WSVR0605W: Thread "SoapConnectorThreadPool : 13172"
    (00004906) has been active for 715000 milliseconds and may be
    hung.  There is/are 1033 thread(s) in total in the server that
    may be hung.
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:199)
      at
      com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
      pareFolder(NodeSync.java:1200)
      at
      com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
      java:1133)
      at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
       ...
    .
    All threads above were waiting for one blocked thread on active
    sync below doing SocketInputStream.socketRead0 and awaiting a
    response from DMgr.
    "NodeSync" (TID:0x3EEF3800, sys_thread_t:0x3EEF1808, state:R,
    native ID:0x21AF9000) prio=5
      at java/net/SocketInputStream.socketRead0(Native Method)
      at
    java/net/SocketInputStream.read(SocketInputStream.java:155(Compi
    led Code))
      at com/ibm/jsse2/a.a(a.java:188(Compiled Code))
      at com/ibm/jsse2/a.a(a.java:232(Compiled Code))
      at com/ibm/jsse2/pc.a(pc.java:211(Compiled Code))
      at com/ibm/jsse2/pc.g(pc.java:376(Compiled Code))
      at com/ibm/jsse2/pc.a(pc.java:573(Compiled Code))
      at com/ibm/jsse2/pc.startHandshake(pc.java:37(Compiled Code))
      at
    org/apache/soap/util/net/SSLUtils.buildSSLSocket(SSLUtils.java:2
    85(Compiled Code))
      at sun/reflect/GeneratedMethodAccessor9.invoke(Bytecode
    PC:220(Compiled Code))
      at
    sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:43(Compiled Code))
      at java/lang/reflect/Method.invoke(Method.java:618(Compiled
    Code))
      at
    org/apache/soap/util/net/HTTPUtils.buildSocket(HTTPUtils.java:26
    1(Compiled Code))
      at
    org/apache/soap/util/net/HTTPUtils.post(HTTPUtils.java:474(Compi
    led Code))
      at
    org/apache/soap/transport/http/SOAPHTTPConnection.send(SOAPHTTPC
    onnection.java:404(Compiled Code))
      at org/apache/soap/rpc/Call.WASinvoke(Call.java:433(Compiled
    Code))
      at
    com/ibm/ws/management/connector/soap/SOAPConnectorClient$8.run(S
    OAPConnectorClient.java:720(Compiled Code))
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1, V7.0, V8.0, and V8.5           *
    ****************************************************************
    * PROBLEM DESCRIPTION: In the presence of another issue that   *
    *                      caused SSL handshakes to hang           *
    *                      indefinitely, repeated                  *
    *                      synchronization requests continued to   *
    *                      be dispatched to new threads, all of    *
    *                      which were waiting on the hung          *
    *                      thread. Eventually, node agent          *
    *                      ran out of memory.                      *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    In the presence of another issue that caused SSL handshakes to
    hang indefinitely, repeated synchronization requests continued
    to be dispatched to new threads, all of which were waiting on
    the hung thread.  The node agent eventually ran out of memory.
    A thread dump showed many SOAP connector threads all waiting
    at:
    "SoapConnectorThreadPool : 0" (TID:0x3ECEE500,
    sys_thread_t:0x3EBE75C8, state:CW, native ID:0x221B0200) prio=5
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:199)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.waitForNextCom
    pareFolder(NodeSync.java:1200)
    at
    com.ibm.ws.management.sync.NodeSync$SyncHandler.sync(NodeSync.
    java:1133)
    at com.ibm.ws.management.sync.NodeSync.sync(NodeSync.java:352)
    

Problem conclusion

  • The code was modified to better manage existing and new
    synchronization operations and avoid out of memory errors from
    an excessive number of waiting threads.
    
    APAR PM61079 is currently targeted for inclusion WebSphere
    Application Server Fix Packs 6.1.0.45, 7.0.0.25, 8.0.0.5,
    and V8.5.0.1.
    
    The fix will also be delivered as sysrouted APAR PM61376 in
    WebSphere Application Server V7.0 Fix Pack 7.0.0.25 for z/OS.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM61079

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-03-23

  • Closed date

    2012-07-10

  • Last modified date

    2013-10-21

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM61376

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK81646

       UP12/09/23 P F209

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM61079

Modified date: 21 October 2013