IBM Support

PM91326: SPORADIC DEADLOCK ON JVM STARTUP

Fixes are available

8.5.5.1: WebSphere Application Server V8.5.5 Fix Pack 1
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
8.5.5.2: WebSphere Application Server V8.5.5 Fix Pack 2
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
8.5.5.3: WebSphere Application Server V8.5.5 Fix Pack 3
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.5.5.4: WebSphere Application Server V8.5.5 Fix Pack 4
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.5.5.5: WebSphere Application Server V8.5.5 Fix Pack 5
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Sporadic deadlock on JVM startup.
    
    Javacore shows deadlock while loading either Java Security
    provider or application classes.
    
    Found one Java-level deadlock:
    =============================
    "Worker-0":
    waiting to lock monitor 0x0010a9c0 (object 0xc3c242a0, a
    sun.misc.Launcher$ExtClassLoader),
    which is held by "main"
    "main":
    waiting to lock monitor 0x00109f30 (object 0xc3c37898, a
    sun.misc.Launcher$AppClassLoader),
    which is held by "Worker-0"
    
    Java stack information for the threads listed above:
    ===================================================
    "Worker-0":
    at java.lang.ClassLoader.loadClass(ClassLoader.java:291)
    - waiting to lock <0xc3c242a0> (a
    sun.misc.Launcher$ExtClassLoader)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
    - locked <0xc3c37898> (a sun.misc.Launcher$AppClassLoader)
    at
    sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    - locked <0xc3c37898> (a sun.misc.Launcher$AppClassLoader)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at
    org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(
    BundleLoader.java:363)
    at
    org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(
    BundleLoader.java:347)
    at
    org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadCla
    ss(DefaultClassLoader.java:83)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at
    org.eclipse.core.internal.jobs.JobManager.createMonitor(JobManag
    er.java:369)
    at
    org.eclipse.core.internal.jobs.JobManager.startJob(JobManager.ja
    va:1115)
    - locked <0xc87c3ab0> (a java.lang.Object)
    at
    org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.ja
    va:219)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:51)
    
    "main":
    at
    sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:
    188)
    - waiting to lock <0xc3c37898> (a
    sun.misc.Launcher$AppClassLoader)
    at
    sun.security.jca.ProviderList.getProvider(ProviderList.java:215)
    at
    sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.ja
    va:416)
    at
    sun.security.jca.ProviderList$ServiceList.access$200(ProviderLis
    t.java:358)
    at
    sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList
    .java:468)
    at java.security.Signature.getInstance(Signature.java:192)
    at sun.security.pkcs.SignerInfo.verify(SignerInfo.java:327)
    at sun.security.pkcs.PKCS7.verify(PKCS7.java:494)
    at sun.security.pkcs.PKCS7.verify(PKCS7.java:511)
    at
    sun.security.util.SignatureFileVerifier.processImpl(SignatureFil
    eVerifier.java:199)
    at
    sun.security.util.SignatureFileVerifier.process(SignatureFileVer
    ifier.java:176)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
    at java.util.jar.JarVerifier.update(JarVerifier.java:199)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
    at java.util.jar.JarFile.getInputStream(JarFile.java:388)
    - locked <0xcf68d060> (a java.util.jar.JarFile)
    at
    sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.ja
    va:692)
    at sun.misc.Resource.cachedInputStream(Resource.java:61)
    - locked <0xcf68db18> (a sun.misc.URLClassPath$JarLoader$2)
    at sun.misc.Resource.getByteBuffer(Resource.java:144)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at
    sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    - locked <0xc3c242a0> (a sun.misc.Launcher$ExtClassLoader)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
    java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:60
    3)
    at
    java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.jav
    a:1574)
    at
    java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1
    495)
    at
    java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.j
    ava:1731)
    at
    java.io.ObjectInputStream.readObject0(ObjectInputStream.java:132
    8)
    at
    java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
    at com.ibm.crypto.provider.JceKeyStore.engineLoad(Unknown
    Source)
    - locked <0xcf6779c0> (a java.util.Hashtable)
    at java.security.KeyStore.load(KeyStore.java:1185)
    at
    com.hkex.was.security.util.KeyManager.init(KeyManager.java:52)
    at
    com.hkex.was.security.util.KeyManager.<init>(KeyManager.java:35)
    at
    com.hkex.was.security.util.KeyManager.getInstance(KeyManager.jav
    a:64)
    - locked <0x7d340a50> (a java.lang.Class for
    com.hkex.was.security.util.KeyManager)
    at
    com.hkex.was.security.util.AESEncryptor.<init>(AESEncryptor.java
    :19)
    at
    com.hkex.was.security.util.AESEncryptor.getInstance(AESEncryptor
    .java:29)
    - locked <0x7d33f2b8> (a java.lang.Class for
    com.hkex.was.security.util.AESEncryptor)
    at
    com.hkex.was.security.PasswordEncryptor.getEncryptor(PasswordEnc
    ryptor.java:45)
    - locked <0xcf47a750> (a
    com.hkex.was.security.PasswordEncryptor)
    at
    com.hkex.was.security.PasswordEncryptor.decrypt(PasswordEncrypto
    r.java:20)
    at
    com.ibm.ISecurityUtilityImpl.PasswordCipherUtil.decipher(Passwor
    dCipherUtil.java:282)
    at ... more lines in stack
    

Local fix

  • None available
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server who configured custom                *
    *                  password encryption                         *
    ****************************************************************
    * PROBLEM DESCRIPTION: WebSphere Application Server process    *
    *                      occasionally hangs during startup       *
    *                      without any detailed log messages.      *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    When the Application server loads application classes,
    there may be another classloading for Java Security Providers
    when custom password encryption is configured.  This
    occasionally triggers a race condition and server startup will
    hang due to classloading deadlock.
    

Problem conclusion

  • Code is updated to provide an option to load Security provider
    classes before application classes are loaded.
    
    To enable the option:
    
    From adminconsole,
    Application Servers > (server name) > Process Definition > Java
    Virtual Machine > custom properties
    
    Property name:
    com.ibm.websphere.security.load.provider
    Value: true (default value is false)
    
    Once this property is set to true, security component will
    call java.security.Security.getProviders()
    before system management component loads application classes.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.31, 8.0.0.8 and 8.5.5.1.  Please refer to
    the Recommended Updates
    page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM91326

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-06-18

  • Closed date

    2013-07-16

  • Last modified date

    2013-07-16

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R700 PSY

       UP

  • R800 PSY

       UP

  • R850 PSY

       UP



Document information

More support for: WebSphere Application Server
General

Software version: 7.0

Reference #: PM91326

Modified date: 16 July 2013