PM19834: WAS SERVICE INTEGRATION BUS: A SERIOUS BOTTLENECK OCCURS WHEN PROCESSING MESSAGES DURING A HIGH LOAD TEST. APPEARS HUNG

Fixes are available

APAR status

Closed as program error.

Error description

A javacore showed a large number (about 45)
"SIBFAPInboundThreadPool" threads blocked waiting for a lock on
the following object:
com/ibm/ws/sib/jfapchannel/impl/rldispatcher/DispatchToAllNonEmp
tyDispatchableImpl$DispatchToAllDispatchQueue
The thread that owned that lock was executing the following
method:
com/ibm/ws/sib/jfapchannel/impl/rldispatcher/
DispatchToAllNonEmptyDispatchableImpl$

The system appeared hung, as no messages appeared to be
consumed from queues once the system entered this state.

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider for *
*                  IBM WebSphere Application Server Version 7.0*
****************************************************************
* PROBLEM DESCRIPTION: Close operations being processed        *
*                      serially by a messaging engine might    *
*                      cause transaction timeouts under load   *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
The problem occurs when applications are connecting over a
network to the queue manager, and a large number of close
operations are happening concurrently.

A javacore taken from a messaging engine experiencing this
problem will show a large number of "SIBFAPInboundThreadPool"
threads blocked waiting on a
DispatchToAllNonEmptyDispatchableImpl$DispatchToAllDispatchQueue
object. Only a single thread is likely to be performing any
actual work.

Internally two inter-related issues contribute to the serial
mode of processing performed by the messaging engine, when a
large number of close operations come into the messaging engine
concurrently:
* Incorrect serialization caused threads to wait for the
  completion of each active work item being processed by the
  messaging engine, before processing the close operation.
* The close operation then might be process in-line, on the
  thread pool used to accept data arriving over the network,
  and only one close operation is able to process concurrently
  in this scenario. Whereas other operations being performed by
  the messaging engine are dispatched to the separate pool of
  worker threads, allowing concurrent processing.

When enough close operations arrive at the messaging engine,
the messaging engine becomes basically serial in its
operation, as the network receiving threads all become
blocked and new work of any type cannot be processed until the
close operations have been handled, which occurs serially.

Problem conclusion

The fix for this APAR corrects the serialization, preventing
the thread handling the close operation from needing to wait
for work being actively handled to complete.

The fix also introduces a new sib.properties tuning parameter
with the following name:

com.ibm.ws.sib.jfapchannel.ALWAYS_DISPATCH_CLOSE

When set to a value of 'true', this tuning property ensures
that all close operations are dispatched to the messaging
engine's worker threads, rather than sometimes being processed
serially on the network receive threads.

See the "Setting tuning properties by editing the
sib.properties file" topic in the WebSphere Application Server
for details of setting properties in the sib.properties file.

The fix for this APAR is currently targeted for inclusion in
fix pack 7.0.0.15.  Please refer to the Recommended Updates
page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PM19834
Reported component name
PLAT MSG COM
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-08-04
Closed date
2010-09-21
Last modified date
2010-09-21

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
PLAT MSG COM
Fixed component ID
620800101

Applicable component levels

R300 PSY
UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 October 2021

Tips

PM19834: WAS SERVICE INTEGRATION BUS: A SERIOUS BOTTLENECK OCCURS WHEN PROCESSING MESSAGES DURING A HIGH LOAD TEST. APPEARS HUNG

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R300 PSY

Document Information

Share your feedback

Need support?