IBM Support

PM81463: WEBSPHERE DAEMON PROCESS HUNG IN DUMP PROCESSING

Fixes are available

7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Customer reported that the daemon process hung.
    

Local fix

  • n/a
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1, V8.0, and V8.5 for z/OS        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Daemon does not come all the way down   *
    *                      after RRS becomes unavailable.          *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    The daemon is most of the way down but did not come down
    completely because the main thread is waiting in LE for all
    threads to come down. Thread 008ACC68 is waiting in GRS for
    an ENQ held by the main thread so it can not come down. This
    deadlock prevents the daemon from coming down.
    TCB 8ff110 caa 2eb1ebd0 pthread_id 2f0f9c0000000000
    Dsa        Entry      Offset     Function
    ---        -----      ------     --------
    2ed22cb0   00000000   00ff562a   (unknown)
    35a6c028   099247e8   d51b7707   CEEOPEF
    2ed22520   09955bd0   000011be   CEEOSIGJ
    2ed22360   098c0d68   00000096   CEEHRNUH
    33dcae20   2dd0f4f0   d22f0b10   getpid
    33dcaee0   343fb598   000002f0   RasLog::addMsgToLog
    (char*,char*,int,void*,int)
    33dcc160   343fb530   0000002a   RasLog::addMsgToLog
    (char*,char*,int)
    33dcc1e0   34316a10   00000040   BossMsg::logError(char*,int)
    33dcc2a0   34021e18   0000449c   BBO_BOA::impl_is_ready
    (CORBA::ImplementationDef*)
    33dcca80   2eb008a8   00004410   main
    33dcd720   099adce0   00001200   CEEVROND
    2ed22e48   0995af48   00000f96   CEEOTERM
    2ed22da0   099dcf78   0000033a   CEEZRET
    2ed22c80   09876660   f67942d2   CEEBBEXT
    TCB 8acc68 caa 35a1e4e8 pthread_id 2f1fbf000000001c
    Dsa        Entry      Offset     Function
    ---        -----      ------     --------
    35fbfbc0   34316a10   cbce95f0   BossMsg::logError(char*,int)
    35fbfc80   34406740   000006fe   RasAtProcessExit
    35fc06a0   2ddd1b80   00000048   __atterm
    35fc0720   099adce0   00001200   CEEVROND
    35a1fe30   099cb448   0000009c   CEEZATEX
    35a1fc08   0995af48   000008ec   CEEOTERM
    35a1fb60   099dcf78   0000033a   CEEZRET
    35a1fa40   0000cd88   ffffdbaa   CEEOPCMM
    Analyze resource reports the following contention
    RESOURCE #0002: MAJOR=SYSBBOLG MINOR=W6DMNC.ASID0011.CERR
    RESOURCE #0002 IS HELD BY:     JOBNAME=W6DMNC    ASID=0011
    TCB=008FF110 - in CEEOTERM+x'57A'
    RESOURCE #0002 IS REQUIRED BY: JOBNAME=W6DMNC    ASID=0011
    TCB=008ACC68 - in RasLog::addMsgToLog()
    Also seen is an ABEND0C4/ABENDS0C4 in bbodenfl at offset 230.
    The 0c4 abend is in bbodenfl's CLEANUP_RTN. CLEANUP_RTN  was
    called by BBORFRR that was processing a different abend.
    BBORFRR's FRR handles the 0c4 abend and retries back to the
    first FRR which takes a dump and percolates.
    

Problem conclusion

  • Changed the code in RasAtProcessExit to log the error in a way
    that does not get the ENQ. Changed bbodenfl to not register
    the unnecessary cleanup routine.
    
    APAR PM81463 is currently targeted for inclusion WebSphere
    Application Server Fix Packs 6.1.0.47, 8.0.0.7, and 8.5.5.
    
    The fix will also be delivered as sysrouted APAR PM84856 in
    WebSphere Application Server V7.0 Fix Pack 7.0.0.29 for z/OS.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM81463

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-01-25

  • Closed date

    2013-03-14

  • Last modified date

    2013-10-04

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM84856

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK97031

       UP13/09/07 P F309

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM81463

Modified date: 04 October 2013