IV29776: CORRUPT SHARED CLASS CACHE CAUSES JVM TO CRASH ON WINDOWS

APAR status

• Closed as program error.

Error description

• Error Message: JVM generates error message indicating an attempt
  to use corrupt shared class cache:
  JVMSHRC097E Shared cache <cache name> is corrupt. No new JVMs
  will be allowed to connect to the cache.
    Existing JVMs can continue to function, but cannot update the
  cache.
  .
  Stack Trace: Two different cases of crash have been observed
  when JVM attempts to use corrupt shared class cache.
  In first case crash happened when accessing a method information
  stored in shared class cache.
  Stack trace -
  Java callstack:
   at java/lang/ClassLoader.defineClassImpl(Native Method)
   at
  java/lang/ClassLoader.defineClass(ClassLoader.java:275(Compiled
  Code))
   at
  org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.defineC
  lass(DefaultClassLoader.java:160(Compiled Code))
   at
  org/eclipse/osgi/baseadaptor/loader/ClasspathManager.defineClass
  (ClasspathManager.java:550(Compiled Code))
   at
  org/eclipse/osgi/baseadaptor/loader/ClasspathManager.findClassIm
  pl(ClasspathManager.java:520(Compiled Code))
   at
  org/eclipse/osgi/baseadaptor/loader/ClasspathManager.findLocalCl
  assImpl(ClasspathManager.java:451(Compiled Code))
   at
  org/eclipse/osgi/baseadaptor/loader/ClasspathManager.findLocalCl
  ass_LockClassName(ClasspathManager.java(Compiled Code))
   at
  org/eclipse/osgi/baseadaptor/loader/ClasspathManager.findLocalCl
  ass(ClasspathManager.java:417(Compiled Code))
   at
  org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.findLoc
  alClass(DefaultClassLoader.java:188(Compiled Code))
   at
  org/eclipse/osgi/framework/internal/core/BundleLoader.findLocalC
  lass(BundleLoader.java:331(Compiled Code))
   at
  org/eclipse/osgi/framework/internal/core/BundleLoader.findClass(
  BundleLoader.java:386(Compiled Code))
   at
  org/eclipse/osgi/framework/internal/core/BundleLoader.findClass(
  BundleLoader.java:347(Compiled Code))
   at
  org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.loadCla
  ss(DefaultClassLoader.java:83(Compiled Code))
   at
  java/lang/ClassLoader.loadClass(ClassLoader.java:619(Compiled
  Code))
   at java/lang/J9VMInternals.verifyImpl(Native Method)
   at
  java/lang/J9VMInternals.verify(J9VMInternals.java:72(Compiled
  Code))
   at
  java/lang/J9VMInternals.initialize(J9VMInternals.java:134(Compil
  ed Code))
   at
  com/ibm/ws/wim/RepositoryManager.initialize(RepositoryManager.ja
  va:470)
   at
  com/ibm/ws/wim/RepositoryManager.<init>(RepositoryManager.java:1
  17)
   at
  com/ibm/ws/wim/RepositoryManager.singleton(RepositoryManager.jav
  a:123)
   at
  com/ibm/ws/wim/ProfileManager.initialize(ProfileManager.java:324
  5)
   at
  com/ibm/ws/wim/ProfileManager.<init>(ProfileManager.java:219)
   at
  com/ibm/ws/wim/ProfileManager.singleton(ProfileManager.java:225)
   at
  com/ibm/websphere/wim/ServiceProvider.initialize(ServiceProvider
  .java:115)
   at
  com/ibm/websphere/wim/ServiceProvider.<init>(ServiceProvider.jav
  a:124)
   at
  com/ibm/websphere/wim/ServiceProvider.singleton(ServiceProvider.
  java:103)
   at
  com/ibm/ws/wim/management/UserManagementProcess.handleNotificati
  on(UserManagementProcess.java:340)
   at
  com/ibm/ws/management/event/ListenerInfo$1.run(ListenerInfo.java
  :137)
   at
  com/ibm/ws/security/auth/ContextManagerImpl.runAs(ContextManager
  Impl.java:4672)
   at
  com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(ContextM
  anagerImpl.java:4760)
   at
  com/ibm/ws/management/event/ListenerInfo.handleNotification(List
  enerInfo.java:153)
   at
  com/ibm/ws/management/event/NotificationDispatcher$DispatchANoti
  ficationToAListener.run(NotificationDispatcher.java:372)
   at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1604)
  Native callstack:
   processVTableMethod+0x91 (vtblsup.asm:326, 0x7FF2F771
  <OSB>j9vm24+0x3f771<CSB>)
  In another case crash happened when attempting to store a class
  in shared class cache.
  Native stack trace -
  avl_intern_sharedSearchCompare+0x13 (internavl.c:232, 0x7FF7F853
  <OSB>j9dyn24+0xf853<CSB>)
  findNode+0x39 (avlsup.c:428, 0x7FF87339
  <OSB>j9dyn24+0x17339<CSB>)
  avl_search+0x12 (avlsup.c:66, 0x7FF873D2
  <OSB>j9dyn24+0x173d2<CSB>)
  avl_intern_search+0x124 (internavl.c:1074, 0x7FF7E9B4
  <OSB>j9dyn24+0xe9b4<CSB>)
  replaceWithInternedString+0x6a (bcutil.c:955, 0x7FF7155A
  <OSB>j9dyn24+0x155a<CSB>)
  copyUTF8DataToROM+0x143 (bcutil.c:1077, 0x7FF71833
  <OSB>j9dyn24+0x1833<CSB>)
  j9bcutil_createRomClassEndian+0x6ed (bcutil.c:3168, 0x7FF7694D
  <OSB>j9dyn24+0x694d<CSB>)
  j9bcutil_createRomClass+0x3b (bcutil.c:233, 0x7FF773AB
  <OSB>j9dyn24+0x73ab<CSB>)
  callDynamicLoader+0x57 (defineclass.c:695, 0x7FF7B407
  <OSB>j9dyn24+0xb407<CSB>)
  internalLoadROMClass+0x5ba (defineclass.c:569, 0x7FF7B9DA
  <OSB>j9dyn24+0xb9da<CSB>)
  createROMClassFromClassFile+0x4d (defineclass.c:720, 0x7FF7BC5D
  <OSB>j9dyn24+0xbc5d<CSB>)
  internalDefineClass+0x10b (defineclass.c:147, 0x7FF7C2EB
  <OSB>j9dyn24+0xc2eb<CSB>)
  defineClassCommon+0x1b2 (jcldefine.c:136, 0x7FC595B2
  <OSB>jclscar_24+0x95b2<CSB>)
  Java_java_lang_ClassLoader_defineClassImpl+0x77 (clsldr.c:52,
  0x7FC564F7 <OSB>jclscar_24+0x64f7<CSB>)
  VMprJavaSendNative+0x44c (jnisend.asm:443, 0x7FF0B34C
  <OSB>j9vm24+0x1b34c<CSB>)
  callLoadClass+0xb9 (classsupport.c:885, 0x7FEF6019
  <OSB>j9vm24+0x6019<CSB>)
  arbitratedLoadClass+0x199 (classsupport.c:1108, 0x7FEF6BE9
  <OSB>j9vm24+0x6be9<CSB>)
  loadNonArrayClass+0x186 (classsupport.c:761, 0x7FEF6E76
  <OSB>j9vm24+0x6e76<CSB>)
  internalFindClassUTF8+0x9a (classsupport.c:806, 0x7FEF6F4A
  <OSB>j9vm24+0x6f4a<CSB>)
  _findClass@8+0x11c (jnisup.asm:919, 0x7FF0BE1C
  <OSB>j9vm24+0x1be1c<CSB>)
  gpProtectedFindClass+0x10 (jnicsup.c:324, 0x7FF074A0
  <OSB>j9vm24+0x174a0<CSB>)
  signalProtectAndRunGlue+0xa (jnicsup.c:1840, 0x7FF07F2A
  <OSB>j9vm24+0x17f2a<CSB>)
  j9sig_protect+0x41 (j9signal.c:144, 0x7FECC161
  <OSB>J9PRT24+0xc161<CSB>)
  gpProtectAndRun+0x38 (jnicsup.c:410, 0x7FF08788
  <OSB>j9vm24+0x18788<CSB>)
  gpCheckFindClass+0x3e (jnicsup.c:348, 0x7FF0906E
  <OSB>j9vm24+0x1906e<CSB>)
  JNU_CallStaticMethodByName+0xb2 (jni_util.c:241, 0x00403274
  <OSB>java+0x3274<CSB>)
  newString646_US+0x2 (jni_util.c:444, 0x00403662
  <OSB>java+0x3662<CSB>)
  .
  

Local fix

• The issue can be worked around by disabling the use of shared
  class cache. This can be achieved either by removing
  -Xshareclasses option from the command or by adding the
  -Xshareclasses:none option after all existing -Xshareclasses
  options in the command used to launch JVM.
  
  Please note that disabling the use of shared class cache
  will have negative impact on the performance and is not
  recommended.
  

Problem summary

• This problem is highly intermittent and has been observed only
  on Windows platform.
  Root cause of shared class cache corruption is not known yet.
  Likely cause of corruption is overwriting of data in shared
  class cache when the cache is close to full, possibly
  following a restart of the machine when WAS is running as a
  service.
  

Problem conclusion

• This defect will be fixed in:
  6.0.0 SR12
  6.0.1 SR4
  7.0.0 SR3
  .
  Following changes have been made in an attempt to avoid shared
  class cache corruption:
  1) The JVM has been updated such that the last 1K bytes in the
  shared class cache are not used on Windows platform.
  2) Protect the class data added by other JVMs by marking the
  area as read-only.
  3) When the available bytes in the shared class cache goes below
  a threshold, JVM adds data to make the cache full.
  This gives consistency with respect to multiple JVMs discovering
  and marking a cache as full, as each JVM marks the cache full
  privately.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  J9 COMMON CODE

• Fixed component ID

  620700127

Applicable component levels

• R600 PSY

     UP