IBM Support

PM17170: S0C4 IN METHODS SUCH AS ORB_REQUEST::GETACRWORKELEMENTPTR() DUE TO DOUBLE FREE OF BBOOORBR

Fixes are available

6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • High volume load tests are performed, with over 500 RMI/IIOP
    requests per second being generated in serverX to another
    server serverY.
    
    If serverY is stopped or abended then the serverX controller
    region will ABEND with various symptoms, mainly S0C4, due to
    what appears to be a double free of a BBOOORBR control block.
    
    Some example call stacks that were encountered are:
    
    Traceback:
      DSA      Entry       E  Offset  Statement   Load Mod
    (jvm "take dump" methods removed)
      20       __zerro     +00000EE4              CELQLIB
      21       __zerros    +0000032E              CELQLIB
      22       CEEHDSP     +00003F64              CELQLIB
    CEEHDSP
      23       CEEOSIGJ    +0000096E              CELQLIB
    CEEOSIGJ
      24       CELQHROD    +00000248              CELQLIB
    CELQHROD
      25       CEEOSIGG    +00000000              CELQLIB
    CEEOSIGG
      26       CELQHROD    +00000248              CELQLIB
    CELQHROD
      27       ORB_Request::getACRWorkElementPtr()     <-- exception
                           +0000000C              BBGBOA
      28
    Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                           +00000F9C              BBGBOA
      29       RUNCALLINMETHOD
    
    Traceback:
    
      DSA      Entry       E  Offset  Statement   Load Mod
    ( ... "take dump" methods removed ... )
      12       __zerro     +00000EE4              CELQLIB
      13       __zerros    +0000032E              CELQLIB
      14       CEEHDSP     +00003F64              CELQLIB
    CEEHDSP
      15       CEEOSIGJ    +0000096E              CELQLIB
    CEEOSIGJ
      16       CELQHROD    +00000248              CELQLIB
    CELQHROD
      17       CEEOSIGG    +00000000              CELQLIB
    CEEOSIGG
      18       CELQHROD    +00000248              CELQLIB
    CELQHROD
      19
    Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                           +00000FCC              BBGBOA <-exception
    
    Traceback:
      DSA      Entry       E  Offset  Statement   Load Mod
    
     1        ORB_Request::~ORB_Request()     <-exception
                          +00000122              BBGBOA
     2        ORB_Request::~ORB_Request()
                          +00000D88              BBGBOA
     3        BBO_BOA::return_ORB_request(ORB_Request*)
                          +00000074              BBGBOA
     4
    Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                          +0000131E              BBGBOA
     5        RUNCALLINMETHOD
                          +00000000              *PATHNAM
     6        gpProtectedRunCallInMethod
                          +0000003E              *PATHNAM
     7        signalProtectAndRunGlue
                          +0000001C              *PATHNAM
     8        j9sig_protect
                          +00000356              *PATHNAM
     9        gpCheckCallin
                           +000000A4              *PATHNAM
    10       callStaticIntMethodA
                           +00000072              *PATHNAM
    11
     ZAioTCPChannelCppUtilities::readComplete(int,long,int,xrcd*)
                           +00000602              BBGCF
    12
     CF_TCP_Request::readComplete(int,CF_Connection*,int,xrcd*)
                           +0000003A              BBGCF
    13
     CF_TCP_Connection_Object_Manager::processAsyncReadComplete(a
                           +000010B6              BBGCF
    14
     ACR_ExecutionThread::processTcpAsyncReadComplete(acrwObj*)
                          +00000024              BBGBOA
     15
    ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*)
                          +00002036              BBGBOA
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1.0                               *
    ****************************************************************
    * PROBLEM DESCRIPTION: ABEND0C4/ABENDS0C4 in WebSphere         *
    *                      Application Server for z/OS Controller  *
    *                      during handling of outbound IIOP        *
    *                      requests under load.                    *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Storage corruption may occur in a WebSphere Application Server
    for z/OS Controller while handling a large number of outbound
    IIOP requests.  The server in question is acting as a client by
    sending an IIOP request to some other server.  If the backend
    server handling the IIOP request goes down for some reason, the
    frontend client server becomes exposed to a timing window that
    could result in storage corruption.
    
    Symptoms of storage corruption can vary, but the most common
    pattern is that the Controller will experience an ABEND0C4
    and a stack trace of the failing thread will show that the
    abend occurred in a function of an ORB_Request object, for
    example:
    
    ORB_Request::getACRWorkElementPtr()
    

Problem conclusion

  • In some cases, a timing window allowed for a double delete of
    an ORB_Request object, causing corruption in the heap pool that
    contains those objects.  This timing window was fixed so that
    the double delete can no longer occur.
    
    APAR PM17170 is currently targeted for inclusion in
    Service Level (Fix Pack) 6.1.0.35 of WebSphere
    Application Server V6.1
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM17170

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2010-06-24

  • Closed date

    2010-08-30

  • Last modified date

    2011-01-03

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM17172

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK62676

       UP10/12/17 P F012

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM17170

Modified date: 03 January 2011