IBM Support

PM19834: WAS SERVICE INTEGRATION BUS: A SERIOUS BOTTLENECK OCCURS WHEN PROCESSING MESSAGES DURING A HIGH LOAD TEST. APPEARS HUNG

Fixes are available

7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • A javacore showed a large number (about 45)
    "SIBFAPInboundThreadPool" threads blocked waiting for a lock on
    the following object:
    com/ibm/ws/sib/jfapchannel/impl/rldispatcher/DispatchToAllNonEmp
    tyDispatchableImpl$DispatchToAllDispatchQueue
    The thread that owned that lock was executing the following
    method:
    com/ibm/ws/sib/jfapchannel/impl/rldispatcher/
    DispatchToAllNonEmptyDispatchableImpl$
    
    The system appeared hung, as no messages appeared to be
    consumed from queues once the system entered this state.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  Users of the default messaging provider for *
    *                  IBM WebSphere Application Server Version 7.0*
    ****************************************************************
    * PROBLEM DESCRIPTION: Close operations being processed        *
    *                      serially by a messaging engine might    *
    *                      cause transaction timeouts under load   *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    The problem occurs when applications are connecting over a
    network to the queue manager, and a large number of close
    operations are happening concurrently.
    
    A javacore taken from a messaging engine experiencing this
    problem will show a large number of "SIBFAPInboundThreadPool"
    threads blocked waiting on a
    DispatchToAllNonEmptyDispatchableImpl$DispatchToAllDispatchQueue
    object. Only a single thread is likely to be performing any
    actual work.
    
    Internally two inter-related issues contribute to the serial
    mode of processing performed by the messaging engine, when a
    large number of close operations come into the messaging engine
    concurrently:
    * Incorrect serialization caused threads to wait for the
      completion of each active work item being processed by the
      messaging engine, before processing the close operation.
    * The close operation then might be process in-line, on the
      thread pool used to accept data arriving over the network,
      and only one close operation is able to process concurrently
      in this scenario. Whereas other operations being performed by
      the messaging engine are dispatched to the separate pool of
      worker threads, allowing concurrent processing.
    
    When enough close operations arrive at the messaging engine,
    the messaging engine becomes basically serial in its
    operation, as the network receiving threads all become
    blocked and new work of any type cannot be processed until the
    close operations have been handled, which occurs serially.
    

Problem conclusion

  • The fix for this APAR corrects the serialization, preventing
    the thread handling the close operation from needing to wait
    for work being actively handled to complete.
    
    The fix also introduces a new sib.properties tuning parameter
    with the following name:
    
    com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_CLOSE
    
    When set to a value of 'true', this tuning property ensures
    that all close operations are dispatched to the messaging
    engine's worker threads, rather than sometimes being processed
    serially on the network receive threads.
    
    See the "Setting tuning properties by editing the
    sib.properties file" topic in the WebSphere Application Server
    for details of setting properties in the sib.properties file.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.15.  Please refer to the Recommended Updates
    page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM19834

  • Reported component name

    PLAT MSG COM

  • Reported component ID

    620800101

  • Reported release

    300

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2010-08-04

  • Closed date

    2010-09-21

  • Last modified date

    2010-09-21

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    PLAT MSG COM

  • Fixed component ID

    620800101

Applicable component levels

  • R300 PSY

       UP



Document information

More support for: WebSphere Application Server
Service Integration Technology

Software version: 7.0

Reference #: PM19834

Modified date: 21 September 2010