Fixes are available
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for IBM i
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for AIX
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Windows
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for HP-UX
7.0.0.7: Java SDK 1.6 SR6 Cumulative Fix for WebSphere Application Server
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Solaris
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Linux
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for IBM i
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Windows
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for AIX
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for HP-UX
7.0.0.9: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Solaris
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Linux
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for IBM i
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Windows
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for HP-UX
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for AIX
7.0.0.11: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Solaris
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Linux
PM14844; 7.0.0.9: There is a security exposure related to JAX-WS web services.
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
APAR status
Closed as program error.
Error description
While running a JAX-WS Web Service in a high load environment we noticed a significant drop in requests being serviced per second. We are running the Web Service with several clients and a 2K SOAP request message on a multi-core/CPU standalone server. We ran some lock analysis and noticed the following large lock contention Java (Inflated) Monitors %MISS GETS NONREC SLOW REC TIER2 TIER3 %UTIL AVER_HTM MON-NAME 0 9434171 9434171 39643 0 -1393992494 85984604 22 6217 [5B339AD4] java/lang/String@01E2EB00 (Object) Here's the object in a javacore dump : 2LKMONINUSE sys_mon_t:0x5B339AD4 infl_mon_t: 0x5B339B0C: 3LKMONOBJECT java/lang/String@01E2EB00/01E2EB0C: owner "WebContainer : 46" (0x6ABCF100), entry count 1 3LKWAITERQ Waiting to enter: 3LKWAITER "WebContainer : 3" (0x60E15200) 3LKWAITER "WebContainer : 4" (0x6B742600) 3LKWAITER "WebContainer : 14" (0x61EE2400) 3LKWAITER "WebContainer : 45" (0x69B4B200) 3LKWAITER "WebContainer : 48" (0x69B4AD00) 3LKWAITER "WebContainer : 49" (0x693FFD00) 3LKWAITER "WebContainer : 51" (0x6B081600) 3LKWAITER "WebContainer : 54" (0x6C284000) 3LKWAITER "WebContainer : 55" (0x6A1FD100) 3LKWAITER "WebContainer : 57" (0x6979F100) 3LKWAITER "WebContainer : 58" (0x69853100) 3LKWAITER "WebContainer : 60" (0x6AB80200) 3LKWAITER "WebContainer : 61" (0x691CB100) 3LKWAITER "WebContainer : 62" (0x6C5C2200) 3LKWAITER "WebContainer : 63" (0x6A1FD600) 3LKWAITER "WebContainer : 64" (0x6B3DCA00) 3LKWAITER "WebContainer : 67" (0x696A0D00) 3LKWAITER "WebContainer : 68" (0x6B081100) 3LKWAITER "WebContainer : 71" (0x68278600) 3LKWAITER "WebContainer : 73" (0x6A981600) 3LKWAITER "WebContainer : 75" (0x680CED00) 3LKWAITER "WebContainer : 76" (0x629AE300) 3LKWAITER "WebContainer : 77" (0x619FFE00) 3LKWAITER "WebContainer : 78" (0x68229300) 3LKWAITER "WebContainer : 79" (0x62E27500) Picking one of these blocked WebContainer threads we see the following stack : 3XMTHREADINFO "WebContainer : 3" TID:0x60E15200, j9thread_t:0x5B1B6D2C, state:B, prio=5 3XMTHREADINFO1 (native thread ID:0xC90, native priority:0x5, native policy:UNKNOWN) 4XESTACKTRACE at org/apache/axiom/om/impl/util/OMSeriali zerUtil.isSetPrefixBefore StartElement(OMSerializerUtil.java:192( Compiled Code)) 4XESTACKTRACE at org/apache/axiom/om/impl/serialize/Stre amingOMSerializer.seriali zeElement(StreamingOMSerializer.java:20 7(Compiled Code)) 4XESTACKTRACE at org/apache/axiom/om/impl/serialize/Stre amingOMSerializer.seriali zeNode(StreamingOMSerializer.java:128(C ompiled Code))
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All IBM WebSphere Application Server users * * of JAX-WS * **************************************************************** * PROBLEM DESCRIPTION: A large lock contention occurs on a * * JAX-WS endpoint implemented as a * * Provider<OMElement> under heavy load. * **************************************************************** * RECOMMENDATION: Install a fixpack containing this APAR * **************************************************************** The customer is running a JAX-WS Web Service in a high load environment and noticing a significant drop in requests being serviced per second. The web service is being run with several clients and a 2K SOAP request message on a multi-core/CPU standalone server. The web service endpoint is implemented as a Provider<OMElement>. Here is a sample stack for one of the blocked threads. 3XMTHREADINFO "WebContainer : 3" TID:0x60E15200, j9thread_t:0x5B1B6D2C, state:B, prio=5 3XMTHREADINFO1 (native thread ID:0xC90, native priority:0x5, native policy:UNKNOWN) 4XESTACKTRACE at org/apache/axiom/om/impl/util/OMSerializerUtil.isSetPrefixBefore StartElement(OMSerializerUtil.java:192(Compiled Code)) 4XESTACKTRACE at org/apache/axiom/om/impl/serialize/StreamingOMSerializer.seriali zeElement(StreamingOMSerializer.java:207(Compiled Code)) 4XESTACKTRACE at org/apache/axiom/om/impl/serialize/StreamingOMSerializer.seriali zeNode(StreamingOMSerializer.java:128(Compiled Code)) 4XESTACKTRACE The root of the issue is synchronization in the JAX-WS runtime class, OMSerializerUtil (indicated in the stack above).
Problem conclusion
Analysis revealed that the OMSerializerUtil.isSetPrefixBeforeStartElement method has multiple synchronized blocks on a cached variable. This synchronization is appropriate, but causes locking contention in the Provider<OMElement> scenario. The indicated code is corrected to remove the synchronized blocks and use ThreadLocal cached variable. The removal of the synchronization effectively eliminates the lock contention. The use of a ThreadLocal variable guarantees that a safe cached value. The fix for this APAR is currently targeted for inclusion in fix pack 7.0.0.7. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PK92166
Reported component name
WEBSPHERE APP S
Reported component ID
5724J0800
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-07-23
Closed date
2009-08-04
Last modified date
2009-08-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE APP S
Fixed component ID
5724J0800
Applicable component levels
R700 PSY
UP
Document Information
Modified date:
24 October 2021