IBM Support

PK92166: LOCK CONTENTION when running JAX-WS application under load

Fixes are available

7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for IBM i
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for AIX
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Windows
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for HP-UX
7.0.0.7: Java SDK 1.6 SR6 Cumulative Fix for WebSphere Application Server
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Solaris
7.0.0.7: WebSphere Application Server V7.0 Fix Pack 7 for Linux
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for IBM i
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Windows
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for AIX
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for HP-UX
7.0.0.9: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Solaris
7.0.0.9: WebSphere Application Server V7.0 Fix Pack 9 for Linux
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for IBM i
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Windows
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for HP-UX
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for AIX
7.0.0.11: Java SDK 1.6 SR7 Cumulative Fix for WebSphere Application Server
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Solaris
7.0.0.11: WebSphere Application Server V7.0 Fix Pack 11 for Linux
PM14844; 7.0.0.9: There is a security exposure related to JAX-WS web services.
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • While running a JAX-WS Web Service in a high load environment
    we noticed a significant drop in requests being serviced per
    second. We are running the Web Service with several clients and
    a 2K SOAP request message on a multi-core/CPU standalone server.
    
    We ran some lock analysis and noticed the
    following large lock contention
    
    Java (Inflated) Monitors
    
    %MISS     GETS   NONREC     SLOW
    REC    TIER2    TIER3
    %UTIL AVER_HTM  MON-NAME
    
    0  9434171  9434171    39643
    0 -1393992494
    85984604    22     6217  [5B339AD4]
    java/lang/String@01E2EB00
    (Object)
    
    Here's the object in a javacore dump :
    
    2LKMONINUSE      sys_mon_t:0x5B339AD4
    infl_mon_t: 0x5B339B0C:
    3LKMONOBJECT
    java/lang/String@01E2EB00/01E2EB0C:
    owner
    "WebContainer : 46" (0x6ABCF100),
    entry count 1
    3LKWAITERQ            Waiting to enter:
    3LKWAITER
    "WebContainer : 3" (0x60E15200)
    3LKWAITER
    "WebContainer : 4" (0x6B742600)
    3LKWAITER
    "WebContainer : 14" (0x61EE2400)
    3LKWAITER
    "WebContainer : 45" (0x69B4B200)
    3LKWAITER
    "WebContainer : 48" (0x69B4AD00)
    3LKWAITER
    "WebContainer : 49" (0x693FFD00)
    3LKWAITER
    "WebContainer : 51" (0x6B081600)
    3LKWAITER
    "WebContainer : 54" (0x6C284000)
    3LKWAITER
    "WebContainer : 55" (0x6A1FD100)
    3LKWAITER
    "WebContainer : 57" (0x6979F100)
    3LKWAITER
    "WebContainer : 58" (0x69853100)
    3LKWAITER
    "WebContainer : 60" (0x6AB80200)
    3LKWAITER
    "WebContainer : 61" (0x691CB100)
    3LKWAITER
    "WebContainer : 62" (0x6C5C2200)
    3LKWAITER
    "WebContainer : 63" (0x6A1FD600)
    3LKWAITER
    "WebContainer : 64" (0x6B3DCA00)
    3LKWAITER
    "WebContainer : 67" (0x696A0D00)
    3LKWAITER
    "WebContainer : 68" (0x6B081100)
    3LKWAITER
    "WebContainer : 71" (0x68278600)
    3LKWAITER
    "WebContainer : 73" (0x6A981600)
    3LKWAITER
    "WebContainer : 75" (0x680CED00)
    3LKWAITER
    "WebContainer : 76" (0x629AE300)
    3LKWAITER
    "WebContainer : 77" (0x619FFE00)
    3LKWAITER
    "WebContainer : 78" (0x68229300)
    3LKWAITER
    "WebContainer : 79" (0x62E27500)
    
    Picking one of these blocked WebContainer threads we see the
    following stack :
    
    3XMTHREADINFO      "WebContainer : 3"
    TID:0x60E15200,
    j9thread_t:0x5B1B6D2C, state:B, prio=5
    3XMTHREADINFO1            (native
    thread ID:0xC90, native
    priority:0x5, native policy:UNKNOWN)
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/util/OMSeriali
    zerUtil.isSetPrefixBefore
    StartElement(OMSerializerUtil.java:192(
    Compiled Code))
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/serialize/Stre
    amingOMSerializer.seriali
    zeElement(StreamingOMSerializer.java:20
    7(Compiled Code))
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/serialize/Stre
    amingOMSerializer.seriali
    zeNode(StreamingOMSerializer.java:128(C
    ompiled Code))
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All IBM WebSphere Application Server users  *
    *                  of JAX-WS                                   *
    ****************************************************************
    * PROBLEM DESCRIPTION: A large lock contention occurs on a     *
    *                      JAX-WS endpoint implemented as a        *
    *                      Provider<OMElement> under heavy load.   *
    ****************************************************************
    * RECOMMENDATION:  Install a fixpack containing this APAR      *
    ****************************************************************
    The customer is running a JAX-WS Web Service in a high load
    environment and noticing a significant drop in requests being
    serviced per second.
    
    The web service is being run with several clients and a 2K SOAP
    request message on a multi-core/CPU standalone server.
    The web service endpoint is implemented as a
    Provider<OMElement>.
    
    Here is a sample stack for one of the blocked threads.
    
    
    3XMTHREADINFO      "WebContainer : 3" TID:0x60E15200,
    j9thread_t:0x5B1B6D2C, state:B, prio=5
    3XMTHREADINFO1            (native thread ID:0xC90, native
    priority:0x5, native policy:UNKNOWN)
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/util/OMSerializerUtil.isSetPrefixBefore
    StartElement(OMSerializerUtil.java:192(Compiled Code))
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/serialize/StreamingOMSerializer.seriali
    zeElement(StreamingOMSerializer.java:207(Compiled Code))
    4XESTACKTRACE          at
    org/apache/axiom/om/impl/serialize/StreamingOMSerializer.seriali
    zeNode(StreamingOMSerializer.java:128(Compiled Code))
    4XESTACKTRACE
    
    The root of the issue is synchronization in the JAX-WS runtime
    class, OMSerializerUtil (indicated in the stack above).
    

Problem conclusion

  • Analysis revealed that
    the OMSerializerUtil.isSetPrefixBeforeStartElement method
    has multiple synchronized blocks on a cached variable.  This
    synchronization is appropriate, but causes locking contention
    in the Provider<OMElement> scenario.
    
    The indicated code is corrected to remove the synchronized
    blocks and use ThreadLocal cached variable.  The removal of
    the synchronization effectively eliminates the lock
    contention.  The use of a ThreadLocal variable guarantees that
    a safe cached value.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.7.  Please refer to the Recommended Updates
    page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PK92166

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2009-07-23

  • Closed date

    2009-08-04

  • Last modified date

    2009-08-04

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R700 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 October 2021