IBM Support

PM44028: AN ATTEMPT TO RE-ENABLE A DISABLED MESSAGING ENGINE IN CAUSES THE JVM TO PANIC

Fixes are available

7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
8.0.0.2: WebSphere Application Server V8.0 Fix Pack 2
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • In a two JVM cluster bus member, WebSphere Application Server
    (WAS) v7.0.x Service Integration Bus messaging engine is
    started on member1. Due to a database problem, member1 is
    stopped. Messaging engine failover fails and the messaging
    engine gets disabled on member2. An attempt to re-enable the
    disabled messaging engine through WAS adminconsole causes
    the JVM to terminate abnormally and following errors are
    written to the SystemOut.log:
    
    CWSID0046E: Messaging engine xxx detected an error and cannot
    continue to run in this server.
    HMGR0130I: The local member of group IBM_hc=ABC,WSAF_SIB_
    BUS=XYZBUS,WSAF_SIB_MESSAGING_ENGINE=XXX,type=WSAF_SIB has
    indicated that is it not alive. The JVM will be terminated.
    SystemOut      Panic:component requested panic from isAlive
    SystemOut      java.lang.RuntimeException: emergencyShutdown
    called:
    SystemOut       at com.ibm.ws.runtime.component.ServerImpl.
    emergencyShutdown(ServerImpl.java:633)
    SystemOut       at com.ibm.ws.hamanager.runtime.Runtime
    ProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
    SystemOut       at com.ibm.ws.hamanager.coordinator.impl.
    JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
    SystemOut       at com.ibm.ws.hamanager.impl.HAGroupImpl.
    doIsAlive(HAGroupImpl.java:866)
    SystemOut       at com.ibm.ws.hamanager.impl.HAGroupImpl$
    HAGroupUserCallback.doCallback(HAGroupImpl.java:1364)
    SystemOut       at com.ibm.ws.hamanager.impl.Worker.
    run(Worker.java:64)
    SystemOut       at com.ibm.ws.util.ThreadPool$Worker.run
    (ThreadPool.java:1563)
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  Users of the default messaging provider     *
    *                  for                                         *
    *                  IBM WebSphere Application Server            *
    ****************************************************************
    * PROBLEM DESCRIPTION: When a Service Integration Bus          *
    *                      Messaging Engine which is part of a     *
    *                      cluster that is configured for high     *
    *                      availability, loses a connection to     *
    *                      the database, then the Messaging        *
    *                      Engine continuously attempts to         *
    *                      obtain a lock on the database. After    *
    *                      exceeding the number of attempts to     *
    *                      obtain a lock on the database, the      *
    *                      Messaging Engine gets disabled. If      *
    *                      now the Messaging Engine is enabled     *
    *                      through the admin console via the       *
    *                      coregroup settings without restarting   *
    *                      the server, the JVM panics and          *
    *                      terminates.                             *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    In a cluster as a bus member configured for high availability,
    having two servers on different Nodes (say Node1 and Node2)
    with one Service Integration Bus Messaging Engine, Messaging
    Engine on Node1 successfully obtains a lock on the database.
    Later the Messaging Engine on Node1 loses connection with the
    database. The Messaging Engine on Node1 attempts to obtain a
    lock on the database but then fails as the Database is still
    not aware of the broken connection and does not release the
    lock.
    A stop request is being issued for the server on Node1 and the
    server on Node1 is stopped and hence, the Messaging Engine
    failsover to the server on Node2.
    Messaging Engine on Node2 attempts to get started and
    tries to obtain a lock on the Database, but then fails as
    the Database is still not aware of the broken connection.
    Messaging Engine on Node2 gets disabled after exceeding the
    number of attempts to obtain a lock. The datasource gets
    disabled and then Messaging Engine gets disabled. A local
    error is reported. Database realizes of the connection being
    broken and releases the lock that it was holding. When
    Messaging Engine on Node2 is re-enabled via the core group
    settings through the admin console, the Messaging Engine
    attempts to get started and starts aquiring the lock and
    continues as expected. During its periodical check, the
    HAManager checks for the health of the Messaging Engine and
    terminates the JVM due to the local error that was reported
    earlier and still being present.
    0000007b HAGroupImpl   I   HMGR0129I: The local member of
    group
    IBM_hc=GRP,WSAF_SIB_BUS=MYBUS,WSAF_SIB_MESSAGING_ENGINE=
    MYENGINE
    MYBUS,type=WSAF_SIB has been disabled. The reason is
    disable
    called internally, the reason is < Messaging Engine
    <MessagingEngine_Name> could not be activated: Refer to
    earlier messages>.
    Messages after the Messaging Engine was re-enabled through the
    admin console,
    000000be SibMessage    I   [<Bus_Name>:<MessagingEngine_Name>]
    CWSIS1537I: The messaging engine,
    ME_UUID=XXXXXXXXXXXXXX, INC_UUID=XXXXXXXXXXXXXX, has
    acquired an exclusive lock on the data store.
    00000012 SibMessage    E   [<Bus_Name>:<MessagingEngine_Name>]
    CWSID0046E: Messaging engine <MessagingEngine_Name> detected
    an error and cannot continue to run in
    this server.
    00000012 HAGroupImpl   I   HMGR0130I: The local member of
    group
    IBM_hc=GRP,WSAF_SIB_BUS=<Bus_Name>,WSAF_SIB_MESSAGING_ENGINE=<Me
    ssagingEngine_Name>,type=WSAF_SIB has indicated that is it not
    alive. The JVM will be terminated.
    00000012 SystemOut     O Panic:component
    requested panic from isAlive
    

Problem conclusion

  • The code has been modified to reset the state of the Messaging
    Engine at the start of the Message Store so that the local
    error reported earlier is overwritten.
    
    The fix for this APAR is currently targeted for inclusion in
    fix pack 7.0.0.21 and 8.0.0.2.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM44028

  • Reported component name

    WAS SIB & SIBWS

  • Reported component ID

    620800101

  • Reported release

    300

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-07-20

  • Closed date

    2011-09-21

  • Last modified date

    2011-11-15

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WAS SIB & SIBWS

  • Fixed component ID

    620800101

Applicable component levels

  • R300 PSY

       UP

  • R800 PSY

       UP



Document information

More support for: WebSphere Application Server
Service Integration Technology

Software version: 7.0

Reference #: PM44028

Modified date: 15 November 2011