IBM Support

IX90125: PERF BOTTLENECK IN UTILITY$REPIDTOSTUBCLASSCACHE . GET()

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Error Message: Customer's application running on RTC (on WXS)
    with security enabled, is making around 300 concurrent outgoing
    calls. Each of these threads has a distinct threadcontext
    classloader. The application hits a performance bottleneck on a
    orb cache.
    .
    Stack Trace: 3XMTHREADBLOCK     Blocked on:
    com/ibm/rmi/util/Utility$RepIdToStubClassCache@0x000000010078434
    0 Owned by: "WebContainer : 27" (J9VMThread:0x0000000007A1EB00,
    java/lang/Thread:0x0000000117B5D578)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    com/ibm/rmi/util/Utility.loadStubClass(Utility.java:1544(Compile
    d Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/CDRReader.loadStub(CDRReader.java:1494(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/CDRReader.newObjRef(CDRReader.java:1367(Compile
    d Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1190(Compi
    led Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1155(Compi
    led Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/IorURL.iorbytesToObjref(IorURL.java:105(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/IorURL.<init>(IorURL.java:154(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ORB.IORToObject(ORB.java:3653(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/CORBA/iiop/ORB.IORToObject(ORB.java:1918(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ISecurityLocalObjectBaseL13Impl/SecurityConnectionInterc
    eptor.getClassName(SecurityConnectionInterceptor.java:2771(Compi
    led Code))
    4XESTACKTRACE                at
    com/ibm/ISecurityLocalObjectBaseL13Impl/SecurityConnectionInterc
    eptor.getConnectionKey(SecurityConnectionInterceptor.java:1041(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/orbimpl/transport/WSTransport.getConnection(WSTranspo
    rt.java:293(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/CORBA/transport/TransportBase.getConnection(TransportBas
    e.java:187(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/TransportManager.get(TransportManager.java:93(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/GIOPImpl.getConnection(GIOPImpl.java:130(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/rmi/iiop/GIOPImpl.createRequest(GIOPImpl.java:174(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ClientDelegate._createRequest(ClientDelegate.j
    ava:2010(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
    va:1186(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
    va:1268(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/CORBA/iiop/ClientDelegate.createRequest(ClientDelegate.j
    ava:1342(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
    va:1164(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/CORBA/iiop/ClientDelegate.createRequest(ClientDelegate.j
    ava:1308(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/rmi/corba/ClientDelegate.request(ClientDelegate.java:188
    6(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/CORBA/iiop/ClientDelegate.request(ClientDelegate.java:12
    64(Compiled Code))
    4XESTACKTRACE                at
    org/omg/CORBA/portable/ObjectImpl._request(ObjectImpl.java:458(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/objectgrid/partition/_IDLShardStub.processMessage(_ID
    LShardStub.java:447(Compiled Code))
    ...
    .
    

Local fix

  • Switch to single classloader policy
    or turn off WAS security.
    

Problem summary

  • The cache on which the threads bottleneck
    (Utility$RepIdToStubClassCache) is backed by a HashMap and it
    stores  repository ids against a linked list of stub classes.
    The customer application slows down accessing this cache as each
    synchronized get() on this cache results in a new entry being
    added which involves parsing the entire (long) linked list and
    then loading the right class into a new entry.
    

Problem conclusion

  • This defect will be fixed in:
    6.0.0 SR15
    6.0.1 SR7
    7.0.0 SR6
    .
    The access to the cache backing the
    Utility$RepIdToStubClassCache has been changed to allow for
    concurrent access.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IX90125

  • Reported component name

    ORB

  • Reported component ID

    620700123

  • Reported release

    600

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-08-30

  • Closed date

    2013-09-23

  • Last modified date

    2013-09-23

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    ORB

  • Fixed component ID

    620700123

Applicable component levels

  • R600 PSY

       UP

  • R260 PSY

       UP



Document information

More support for: Runtimes for Java Technology
ORB

Software version: 6.0

Reference #: IX90125

Modified date: 23 September 2013