IBM Support

PM35764: WLM INFINITE LOOP IN HASMAP

Fixes are available

7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
8.0.0.1: WebSphere Application Server V8.0 Fix Pack 1
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
8.0.0.2: WebSphere Application Server V8.0 Fix Pack 2
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Customer reports an ABENDSEC3 RSN04130004 in the Adjunct region.
    Dump shows a TCB ran a request in the Adjunct region for 60
    minutes. The call stack for this TCB (native and java) is:
    
    Entry      Offset     Function
    -----      ------     --------
    1c5aaed0   000020c4   CEEOPCT
    1c88fed8   000000c0   pthread_cond_timedwait
    1e983bb0   0000044e   monitor_wait_original
    1e987700   00000084   j9thread_monitor_wait_abortable
    1e961560   00000010   callMonitorWaitAbortable
    1e8e9c08   00000418   Z_OBJECTMONITORENTERBLOCKING
    1e94adf0   00000014   objectMonitorEnterBlocking
    1e8db838   fffdcac4   RUNCALLINMETHOD
    1e918058   0000003e   gpProtectedRunCallInMethod
    1e917e30   0000001c   signalProtectAndRunGlue
    1e9f1218   0000071a   j9sig_protect
    1e91ae50   000000a4   gpCheckCallin
    1e9135a8   0000005e   callVirtualObjectMethodA
    1cbbc5f8   00001230   ORBEJSBridge::invoke(void*)
    1cd40988   00000ea8   threadDispatch(BOSS_Object_Key*,
                          Internal_CORBA_Request&,ORB_Request*)
    1cd55060   00001ba2   SR_ExecutionThread::RemoveAndProcessWork(
                          ThreadCleanUp*,TCB*)
    1cd57ff8   000001b8   SR_ExecutionRoutine
    1c5a6df0   00000f50   CELQPCMM
    
    Java:
    
    com/ibm/ws/sib/utils/ThreadLocalStack.push
        ThreadLocalStack.java:87
    com/ibm/ws/sib/utils/ras/SibTr.push
        SibTr.java:448
    com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.membershi
        pChanged
        HAManagerMessagingEngineImpl.java:669
    com/ibm/ws/hamanager/proxy/DispatchHAGroupCallbackImpl.membershi
        pChanged
        DispatchHAGroupCallbackImpl.java:166
    com/ibm/ws/hamanager/proxy/_DispatchHAGroupCallbackImplBase._inv
        oke
        _DispatchHAGroupCallbackImplBase.java:140
    com/ibm/ws390/orb/CommonBridge.invoke
        CommonBridge.java:1190
    com/ibm/ws390/orb/ORBEJSBridge.invoke
        ORBEJSBridge.java:270
    sun/reflect/GeneratedMethodAccessor33.invoke
    sun/reflect/DelegatingMethodAccessorImpl.invoke
        DelegatingMethodAccessorImpl.java:37
    java/lang/reflect/Method.invoke
        Method.java:600
    com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
        keParmSetter
        ORBEJSBridgeInvoke.java:186
    
    The extracted javacore shows TCB 8B64F8 is waiting for a monitor
    owned by the SIB Activation Thread on TCB 8b5a48 (again, in the
    adjunct)
    
    2LKMONINUSE      sys_mon_t:0x0000000A352CC700 infl_mon_t:
                        0x0000000A352CC770:
    3LKMONOBJECT
    com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl
                          @0x000000081CBB92E0/0x000000081CBB92F8:
                          Flat locked by "SIB Activation Thread"
                          (0x000000080943ED00), entry count 1
    3LKWAITERQ            Waiting to enter:
    3LKWAITER                "WebSphere t=008b64f8"
    (0x0000000809428600)
    
    TCB 8B5A48 has gone outbound with a request:
    
    TCB 8b5a48 caa c13dfecf0 pthread_id 1d4579000000007a
               SIB Activation Thread J9VMThread 80943ed00 (daemon)
    
    Dsa         Entry      Offset     Function
    ---         -----      ------     --------
    c13bfd1e0   1cf065d0   00000000
    bboosout(BBOOSOUT_Functions,ORB_Request_SharedMemberData*,long
    long,int)
    c13bfd4e0   1cd7b480   00000594
    ORB_Request::comm_outbound_request()
    c13bfd720   1cdce160   00000c0e
    Java_com_ibm_ws390_orb_ClientDelegate_invokeRequestCFW
    c13bfe420   1e8dbdf0   e1724210   RUNJAVATHREAD
    c13bfe860   1e953ae8   000000e0   javaProtectedThreadProc
    c13bfe960   1e9f1218   0000071a   j9sig_protect
    c13bff160   1e9565e8   00000072   javaThreadProc
    c13bff260   1e9855e0   0000051c   thread_wrapper
    c13bff360   1c5a6df0   00000f50   CELQPCMM
    
    Java stack:
    -----------
    com/ibm/ws390/orb/ClientDelegate.invokeRequestCFW
        ClientDelegate.java
    com/ibm/ws390/orb/ClientDelegate.commonInvoke
        ClientDelegate.java:1018
    com/ibm/ws390/orb/ClientDelegate.invoke
        ClientDelegate.java:865
    org/omg/CORBA/portable/ObjectImpl._invoke
        ObjectImpl.java:484
    com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Stub.publis
        h
        _ControllerClusterManagerImpl_Stub.java:544
    com/ibm/ws/cluster/zos/SRDescriptionManager.publish
        SRDescriptionManager.java:287
    com/ibm/ws/cluster/zos/SRDescriptionManager.publish
        SRDescriptionManager.java:364
    com/ibm/ws/cluster/zos/ZDescriptionManager.publish
        ZDescriptionManager.java:196
    com/ibm/websphere/cluster/topography/DescriptionA.updateMemento
        DescriptionA.java:160
    com/ibm/ws/cluster/topography/ClusterDescriptionImpl.addMember
        ClusterDescriptionImpl.java:387
    com/ibm/ws/cluster/service/ClusterManagementImpl.joinCluster
        ClusterManagementImpl.java:283
    com/ibm/ws/cluster/zos/ZosClusterManagementImpl.joinCluster
        ZosClusterManagementImpl.java:108
    com/ibm/ws/cluster/service/ClusterMemberServiceImpl.joinCluster
        ClusterMemberServiceImpl.java:503
    com/ibm/ws/sib/trm/wlm/server/Join.join
        Join.java:211
    com/ibm/ws/sib/trm/wlm/server/Join.toShortToLongName
        Join.java:136
    com/ibm/ws/sib/trm/wlm/server/Manager.start
        Manager.java:144
    com/ibm/ws/sib/trm/TrmMeMainImpl.start
        TrmMeMainImpl.java:266
    com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl.start
        JsMessagingEngineImpl.java:609
    com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.activate
        HAManagerMessagingEngineImpl.java:995
    com/ibm/ws/sib/admin/impl/JsActivationThread.run
        JsActivationThread.java:92
    
    THIS IS THE LOOPING TCB in the Controller region, processing the
    request on
    TCB=8b5a48 TCB 8b6e00 caa a4f7fecf0 pthread_id 1d952000000006d
    WebSphere t=008b6e00
    J9VMThread 8098df000 (daemon)
    
    Dsa         Entry      Offset     Function
    ---         -----      ------     --------
    b2fafa4e0   1fe29838   fd722688   RUNCALLINMETHOD
    b2fafa920   1fe66058   0000003e   gpProtectedRunCallInMethod
    b2fafaa20   1fe65e30   0000001c   signalProtectAndRunGlue
    b2fafab20   1ff3f218   0000071a   j9sig_protect
    b2fafb320   1fe68e50   000000a4   gpCheckCallin
    b2fafb460   1fe615a8   0000005e   callVirtualObjectMethodA
    b2fafb560   1cbc15f8   00001230   ORBEJSBridge::invoke(void*)
    b2fafc560   1cd45988   00000ea8 threadDispatch(BOSS_Object_Key*,
                Internal_CORBA_Request&, ORB_Request*)
    b2fafd1a0   1cd47ba8   00000a56   ACR_ExecutionThread::
                ProcessInboundRequest(acrwObj*,
                ThreadCleanUp*,BOSS_Object_Key&,  Interna...
    b2fafe3a0   1cd577e8   0000086e   ACR_ExecutionThread::
                RemoveAndProcessWork(ThreadCleanUp*)
    b2faff220   1cd599c8   0000022e   ACR_ExecutionRoutine
    b2faff360   1c5a9df0   00000f50   CELQPCMM
    
    Java stack:
    -----------
    java/util/HashMap.findNonNullKeyEntry
        HashMap.java:526
    java/util/HashMap.putImpl
        HashMap.java:622
    java/util/HashMap.put
        HashMap.java:605
    java/util/HashSet.add
        HashSet.java:95
    com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish
        CRDescriptionManagerImpl.java:206
    com/ibm/ws/cluster/zos/ControllerClusterManagerImpl.publish
        ControllerClusterManagerImpl.java:582
    com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie.publish
        _ControllerClusterManagerImpl_Tie.java:255
    com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie._invoke
        _ControllerClusterManagerImpl_Tie.java:84
    sun/reflect/GeneratedMethodAccessor15.invoke
    sun/reflect/DelegatingMethodAccessorImpl.invoke
        DelegatingMethodAccessorImpl.java:37
    java/lang/reflect/Method.invoke
        Method.java:600
    com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke
        RIRProtectedProxyHandler.java:186
    $Proxy0._invoke
    com/ibm/ws390/orb/CommonBridge.invoke
        CommonBridge.java:1190
    com/ibm/ws390/orb/ORBEJSBridge.invoke
        ORBEJSBridge.java:270
    sun/reflect/GeneratedMethodAccessor11.invoke
    sun/reflect/DelegatingMethodAccessorImpl.invoke
        DelegatingMethodAccessorImpl.java:37
    java/lang/reflect/Method.invoke
        Method.java:600
    com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
        keParmSetter
        ORBEJSBridgeInvoke.java:186
    
    Evidence suggest the loop on TCB=8b5a48 ran for 1 hour.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server z/OS on versions 7.0 & 8.0           *
    ****************************************************************
    * PROBLEM DESCRIPTION: Infinite loop in HashMap                *
    ****************************************************************
                                                                   *
    ****************************************************************
    Unprotected concurrent access to a hashmap in the WLM class
    com/ibm/ws/cluster/zos/CRDescriptionManagerImpl can cause an
    infinite loop in the underlying HashMap inplementation.  A
    customer hitting this problem would see symptoms as described
    above, any stack should have the following piece present:
    com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish
    CRDescriptionManagerImpl.java:206
    

Problem conclusion

  • The WLM code in question had synchronization added around the
    usage of the hashmap to prevent the infinite loop from
    occurring.
    
    APAR PM35764 is currently targeted for inclusion in
    Service Level (Fix Pack) 7.0.0.19, 8.0.0.1 of WebSphere
    Application Server
    
    Please refer to the Recommended Updates page for delivery
    information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM35764

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-03-28

  • Closed date

    2011-07-08

  • Last modified date

    2011-10-04

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R700 PSY UK71280

       UP11/09/10 P F109

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 7.0

Reference #: PM35764

Modified date: 04 October 2011