Fixes are available
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
APAR status
Closed as program error.
Error description
An application server JVM can hang if a thread logging an FFDC hangs. The problem can be detected by observing the WSVR0605W hung thread notifications in the SystemOut.log. One thread will be running the FFDCFilter.processException() which can be invoked from any WebSphere Application Server component. The thread will get stuck doing its work. In the following example, the thread which invoked FFDCFilter.processException() is stuck waiting on a response to the OracleConnectionWrapper.getMetaData() method call: [05/24/12 12:06:35:286 EST] 00000016 ThreadMonitor W WSVR0605W: Thread "ORB.thread.pool : 32" (000000d6) has been active for 304721 milliseconds and may be hung. There is/are 16 thread(s) in total in the server that may be hung. at oracle.jdbc.OracleConnectionWrapper.getMetaData(OracleConnection Wrapper.java:150) at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.introspectSe lf(WSRdbManagedConnectionImpl.java:1480) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor Impl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at com.ibm.ws.ffdc.impl.Formatters$IntrospectSelfFormatter$1.run(Fo rmatters.java:99) at com.ibm.ws.ffdc.impl.Formatters$IntrospectSelfFormatter$1.run(Fo rmatters.java:96) ... at com.ibm.ffdc.util.provider.IncidentEntry.log(IncidentEntry.java: 96) at com.ibm.ffdc.util.provider.Ffdc.log(Ffdc.java:90) at com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:114) at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java :1671) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2496) Looking later in the SystemOut.log, there will be more hung thread notifications which show that the FFDCFilter.processException() method is stuck waiting on an Ffdc.<init> call to complete. The stack trace of such a thread will look like this: [05/24/12 12:42:35:914 EST] 00000003 ThreadMonitor W WSVR0605W: Thread "ORB.thread.pool : 29" (000007f4) has been active for 224151 milliseconds and may be hung. There is/are 23 thread(s) in total in the server that may be hung. at com.ibm.ffdc.util.provider.Ffdc.<init>(Ffdc.java:64) at com.ibm.ffdc.util.provider.FfdcProvider.getFfdc(FfdcProvider.jav a:289) at com.ibm.ffdc.impl.Ffdc.getFfdc(Ffdc.java:188) at com.ibm.ws.ffdc.FFDCFilter.processException(FFDCFilter.java:112) at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpda te(WSJdbcPreparedStatement.java:1179) at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate( WSJdbcPreparedStatement.java:816) It is possible that the first FFDC thread which got hung (in this example, "ORB.thread.pool : 32") will eventually complete. If it does complete, the threads will start moving and the hang will clear up. However, if it never completes, then it is the same as if a true thread deadlock had occurred. This same type of analysis can be performed using a javacore/thread dump, however, javacores are not always available.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V7.0 and V8.0 * **************************************************************** * PROBLEM DESCRIPTION: Syncrhonization between FFDC * * incidents can cause a server hang * * when an unrecoverable problem * * occurs in a given incident. * **************************************************************** * RECOMMENDATION: * **************************************************************** Syncrhonization between FFDC incidents can cause a server hang when an unrecoverable problem occurs in a given incident.
Problem conclusion
FFDC has been modified to ensure that incidents can continue to be logged from other components when a given component's incident-handling hangs a thread. The impact to this is that the FFDC summary log may indicate that the incident file of a bad thread was created, while the thread may not have returned so the incident file will not exist. Therefore, a missing incident entry that might have appeared in a summary file is an indication that the component logging to FFDC did not return control. The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.23 and 8.0.0.3. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PM45011
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-08-03
Closed date
2011-12-15
Last modified date
2012-06-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R700 PSY
UP
R800 PSY
UP
Document Information
Modified date:
27 October 2021