IX90125: PERF BOTTLENECK IN UTILITY$REPIDTOSTUBCLASSCACHE . GET()

APAR status

• Closed as program error.

Error description

• Error Message: Customer's application running on RTC (on WXS)
  with security enabled, is making around 300 concurrent outgoing
  calls. Each of these threads has a distinct threadcontext
  classloader. The application hits a performance bottleneck on a
  orb cache.
  .
  Stack Trace: 3XMTHREADBLOCK     Blocked on:
  com/ibm/rmi/util/Utility$RepIdToStubClassCache@0x000000010078434
  0 Owned by: "WebContainer : 27" (J9VMThread:0x0000000007A1EB00,
  java/lang/Thread:0x0000000117B5D578)
  3XMTHREADINFO3           Java callstack:
  4XESTACKTRACE                at
  com/ibm/rmi/util/Utility.loadStubClass(Utility.java:1544(Compile
  d Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/CDRReader.loadStub(CDRReader.java:1494(Compiled
  Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/CDRReader.newObjRef(CDRReader.java:1367(Compile
  d Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1190(Compi
  led Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/CDRReader.read_Object(CDRReader.java:1155(Compi
  led Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/IorURL.iorbytesToObjref(IorURL.java:105(Compil
  ed Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/IorURL.<init>(IorURL.java:154(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ORB.IORToObject(ORB.java:3653(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/CORBA/iiop/ORB.IORToObject(ORB.java:1918(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/ISecurityLocalObjectBaseL13Impl/SecurityConnectionInterc
  eptor.getClassName(SecurityConnectionInterceptor.java:2771(Compi
  led Code))
  4XESTACKTRACE                at
  com/ibm/ISecurityLocalObjectBaseL13Impl/SecurityConnectionInterc
  eptor.getConnectionKey(SecurityConnectionInterceptor.java:1041(C
  ompiled Code))
  4XESTACKTRACE                at
  com/ibm/ws/orbimpl/transport/WSTransport.getConnection(WSTranspo
  rt.java:293(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/CORBA/transport/TransportBase.getConnection(TransportBas
  e.java:187(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/TransportManager.get(TransportManager.java:93(C
  ompiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/GIOPImpl.getConnection(GIOPImpl.java:130(Compil
  ed Code))
  4XESTACKTRACE                at
  com/ibm/rmi/iiop/GIOPImpl.createRequest(GIOPImpl.java:174(Compil
  ed Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ClientDelegate._createRequest(ClientDelegate.j
  ava:2010(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
  va:1186(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
  va:1268(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/CORBA/iiop/ClientDelegate.createRequest(ClientDelegate.j
  ava:1342(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ClientDelegate.createRequest(ClientDelegate.ja
  va:1164(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/CORBA/iiop/ClientDelegate.createRequest(ClientDelegate.j
  ava:1308(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/rmi/corba/ClientDelegate.request(ClientDelegate.java:188
  6(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/CORBA/iiop/ClientDelegate.request(ClientDelegate.java:12
  64(Compiled Code))
  4XESTACKTRACE                at
  org/omg/CORBA/portable/ObjectImpl._request(ObjectImpl.java:458(C
  ompiled Code))
  4XESTACKTRACE                at
  com/ibm/ws/objectgrid/partition/_IDLShardStub.processMessage(_ID
  LShardStub.java:447(Compiled Code))
  ...
  .
  

Local fix

• Switch to single classloader policy
  or turn off WAS security.
  

Problem summary

• The cache on which the threads bottleneck
  (Utility$RepIdToStubClassCache) is backed by a HashMap and it
  stores  repository ids against a linked list of stub classes.
  The customer application slows down accessing this cache as each
  synchronized get() on this cache results in a new entry being
  added which involves parsing the entire (long) linked list and
  then loading the right class into a new entry.
  

Problem conclusion

• This defect will be fixed in:
  6.0.0 SR15
  6.0.1 SR7
  7.0.0 SR6
  .
  The access to the cache backing the
  Utility$RepIdToStubClassCache has been changed to allow for
  concurrent access.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  ORB

• Fixed component ID

  620700123

Applicable component levels

• R600 PSY

     UP

• R260 PSY

     UP