PM44028: AN ATTEMPT TO RE-ENABLE A DISABLED MESSAGING ENGINE IN CAUSES THE JVM TO PANIC

Fixes are available

APAR status

Closed as program error.

Error description

In a two JVM cluster bus member, WebSphere Application Server
(WAS) v7.0.x Service Integration Bus messaging engine is
started on member1. Due to a database problem, member1 is
stopped. Messaging engine failover fails and the messaging
engine gets disabled on member2. An attempt to re-enable the
disabled messaging engine through WAS adminconsole causes
the JVM to terminate abnormally and following errors are
written to the SystemOut.log:

CWSID0046E: Messaging engine xxx detected an error and cannot
continue to run in this server.
HMGR0130I: The local member of group IBM_hc=ABC,WSAF_SIB_
BUS=XYZBUS,WSAF_SIB_MESSAGING_ENGINE=XXX,type=WSAF_SIB has
indicated that is it not alive. The JVM will be terminated.
SystemOut      Panic:component requested panic from isAlive
SystemOut      java.lang.RuntimeException: emergencyShutdown
called:
SystemOut       at com.ibm.ws.runtime.component.ServerImpl.
emergencyShutdown(ServerImpl.java:633)
SystemOut       at com.ibm.ws.hamanager.runtime.Runtime
ProviderImpl.panicJVM(RuntimeProviderImpl.java:92)
SystemOut       at com.ibm.ws.hamanager.coordinator.impl.
JVMControllerImpl.panicJVM(JVMControllerImpl.java:56)
SystemOut       at com.ibm.ws.hamanager.impl.HAGroupImpl.
doIsAlive(HAGroupImpl.java:866)
SystemOut       at com.ibm.ws.hamanager.impl.HAGroupImpl$
HAGroupUserCallback.doCallback(HAGroupImpl.java:1364)
SystemOut       at com.ibm.ws.hamanager.impl.Worker.
run(Worker.java:64)
SystemOut       at com.ibm.ws.util.ThreadPool$Worker.run
(ThreadPool.java:1563)

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider     *
*                  for                                         *
*                  IBM WebSphere Application Server            *
****************************************************************
* PROBLEM DESCRIPTION: When a Service Integration Bus          *
*                      Messaging Engine which is part of a     *
*                      cluster that is configured for high     *
*                      availability, loses a connection to     *
*                      the database, then the Messaging        *
*                      Engine continuously attempts to         *
*                      obtain a lock on the database. After    *
*                      exceeding the number of attempts to     *
*                      obtain a lock on the database, the      *
*                      Messaging Engine gets disabled. If      *
*                      now the Messaging Engine is enabled     *
*                      through the admin console via the       *
*                      coregroup settings without restarting   *
*                      the server, the JVM panics and          *
*                      terminates.                             *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
In a cluster as a bus member configured for high availability,
having two servers on different Nodes (say Node1 and Node2)
with one Service Integration Bus Messaging Engine, Messaging
Engine on Node1 successfully obtains a lock on the database.
Later the Messaging Engine on Node1 loses connection with the
database. The Messaging Engine on Node1 attempts to obtain a
lock on the database but then fails as the Database is still
not aware of the broken connection and does not release the
lock.
A stop request is being issued for the server on Node1 and the
server on Node1 is stopped and hence, the Messaging Engine
failsover to the server on Node2.
Messaging Engine on Node2 attempts to get started and
tries to obtain a lock on the Database, but then fails as
the Database is still not aware of the broken connection.
Messaging Engine on Node2 gets disabled after exceeding the
number of attempts to obtain a lock. The datasource gets
disabled and then Messaging Engine gets disabled. A local
error is reported. Database realizes of the connection being
broken and releases the lock that it was holding. When
Messaging Engine on Node2 is re-enabled via the core group
settings through the admin console, the Messaging Engine
attempts to get started and starts aquiring the lock and
continues as expected. During its periodical check, the
HAManager checks for the health of the Messaging Engine and
terminates the JVM due to the local error that was reported
earlier and still being present.
0000007b HAGroupImpl   I   HMGR0129I: The local member of
group
IBM_hc=GRP,WSAF_SIB_BUS=MYBUS,WSAF_SIB_MESSAGING_ENGINE=
MYENGINE
MYBUS,type=WSAF_SIB has been disabled. The reason is
disable
called internally, the reason is < Messaging Engine
<MessagingEngine_Name> could not be activated: Refer to
earlier messages>.
Messages after the Messaging Engine was re-enabled through the
admin console,
000000be SibMessage    I   [<Bus_Name>:<MessagingEngine_Name>]
CWSIS1537I: The messaging engine,
ME_UUID=XXXXXXXXXXXXXX, INC_UUID=XXXXXXXXXXXXXX, has
acquired an exclusive lock on the data store.
00000012 SibMessage    E   [<Bus_Name>:<MessagingEngine_Name>]
CWSID0046E: Messaging engine <MessagingEngine_Name> detected
an error and cannot continue to run in
this server.
00000012 HAGroupImpl   I   HMGR0130I: The local member of
group
IBM_hc=GRP,WSAF_SIB_BUS=<Bus_Name>,WSAF_SIB_MESSAGING_ENGINE=<Me
ssagingEngine_Name>,type=WSAF_SIB has indicated that is it not
alive. The JVM will be terminated.
00000012 SystemOut     O Panic:component
requested panic from isAlive

Problem conclusion

The code has been modified to reset the state of the Messaging
Engine at the start of the Message Store so that the local
error reported earlier is overwritten.

The fix for this APAR is currently targeted for inclusion in
fix pack 7.0.0.21 and 8.0.0.2.  Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PM44028
Reported component name
WAS SIB & SIBWS
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-07-20
Closed date
2011-09-21
Last modified date
2011-11-15

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WAS SIB & SIBWS
Fixed component ID
620800101

Applicable component levels

R300 PSY
UP
R800 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
27 October 2021

Tips

PM44028: AN ATTEMPT TO RE-ENABLE A DISABLED MESSAGING ENGINE IN CAUSES THE JVM TO PANIC

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R300 PSY

R800 PSY

Document Information

Share your feedback

Need support?