IBM Support

PM48958: A SERVICE INTEGRATION BUS MESSAGING ENGINE HANGS UNDER HIGH LOAD

Fixes are available

8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Under high load (usually a combination of high message
    throughput and large messages) clients connected remotely to
    a Messaging Engine find that the Messaging Engine becomes
    unresponsive.
    
    Javacores collected from the JVM hosting the messaging engine
    show large number of SIBFAPInboundThreadPool threads blocked
    waiting on PriorityQueue$ActivatableMonitor.
    
    The waiting threads have the following stack:
    at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive
    ListenerDispatcher.queueInvocationCommon(ReceiveListener
    Dispatcher.java:571(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive
    ListenerDispatcher.queueDataReceivedInvocation(Receive
    ListenerDispatcher.java:667(CompiledCode))
    at com/ibm/ws/sib/jfapchannel/impl/InboundTransmissionParser.
    dispatchToConversationListenerMethod(InboundTransmission
    Parser.java:1157(CompiledCode))
    
    The blocking thread has the following stack:
    
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:167(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue$Activatable
    Monitor.waitOn(PriorityQueue.java:119(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue.queueInternal
    (PriorityQueue.java:347(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/PriorityQueue.queue
    (PriorityQueue.java:337(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/Connection.send(Connection.
    java:677(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.send
    Internal(ConversationImpl.java:1157(Compiled Code))
    ..........................
    at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/DispatchTo
    AllNonEmptyDispatchableImpl$DispatchToAllDispatchQueue.
    enqueue(DispatchToAllNonEmptyDispatchableImpl.java:308
    (Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/rldispatcher/Receive
    ListenerDispatcher.queueInvocationCommon(ReceiveListener
    Dispatcher.java:571(Compiled Code))
    
    The rest of some of the stacks will indicate that either a
    stop or unlockall operation is being attempted.
    
    Once the hang occurs the application server must be restarted.
    
    A poorly performing network can also cause the hang.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  Users of the default messaging provider     *
    *                  for IBM WebSphere Application Server        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Under high load remote clients find     *
    *                      the Messaging Engine hangs.             *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    The Messaging Engine processes client requests either
    synchronously or asyncronsously depending on the request type.
    Synchronous requests, including stop and unlockall, must wait
    for the Channel Framework to complete the sending of any in
    flight asynchronous responses before being able to send the
    synchronous request's response.
    If the in flight asynchronous response takes a long time (for
    example when sending a large message to the client) then it is
    possible that the Channel Framework will run out of threads to
    call the Service Integration Bus.  All the threads from the
    thread pool allocated to the Channel Framework (in the
    SIBFAPInboundThreadPool) will become used by the synchronous
    requests that are all blocked waiting for the asynchronous
    response to complete.
    Once all the threads are used then the Channel Framework
    may be unable to finish sending the asynchronous response
    (because there are no more threads to continue the send).  The
    hang ensues.
    

Problem conclusion

  • This APAR introduces two new custom properties:
    
    com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_STOP (WSAS 7.0.x
    default of false, WSAS 8.0.x and above default of true)
    
    com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_UNLOCKALL (WSAS
    7.0.x default of false, WSAS 8.0.x and above default of true)
    
    When these properties are set to true, stop and unlockall
    requests are performed asynchronously.  This prevents the hang.
    
    These two new properties complement the existing
    com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_CLOSE property
    (introduced in PM19834).  When all three properties are set to
    true together, all client requests are processed
    asynchronously.
    
    If it is found that one of these properties needs to be set it
    is recommended that all are set.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.23 and 8.0.0.3.  Please refer to the Recommended
    Updates page for delivery information:
    
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

  • Increase the size of the SIBFAPInboundThreadPool thread pool.
    This does not make the hang impossible but delays the
    occurrence of the hang potentially long enough for the
    asynchronous response to complete.
    

Comments

APAR Information

  • APAR number

    PM48958

  • Reported component name

    WAS SIB & SIBWS

  • Reported component ID

    620800101

  • Reported release

    300

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-09-29

  • Closed date

    2011-12-08

  • Last modified date

    2015-08-24

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM60682

Fix information

  • Fixed component name

    WAS SIB & SIBWS

  • Fixed component ID

    620800101

Applicable component levels

  • R300 PSY

       UP

  • R800 PSY

       UP



Document information

More support for: WebSphere Application Server
Service Integration Technology

Software version: 7.0

Reference #: PM48958

Modified date: 24 August 2015