IBM Support

PM13797: ABEND S0C4-4 IN BBOBOA+171A8C

Fixes are available

6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for AIX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for HP-UX
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for IBM i
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Linux
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Solaris
7.0.0.13: WebSphere Application Server V7.0 Fix Pack 13 for Windows
7.0.0.13: Java SDK 1.6 SR8FP1 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for AIX
7.0.0.15: Java SDK 1.6 SR9 Cumulative Fix for WebSphere Application Server
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for HP-UX
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for IBM i
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Linux
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Solaris
7.0.0.15: WebSphere Application Server V7.0 Fix Pack 15 for Windows
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
7.0.0.17: WebSphere Application Server V7.0 Fix Pack 17
7.0.0.17: Java SDK 1.6 SR9 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.19: WebSphere Application Server V7.0 Fix Pack 19
7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.19: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Customer runs an EJB invocation stress test on a server.
    The server has one servant region.
    
    For some reason, the servant region experiences a timeout.
    
    BBOO0327I IIOP REQUEST TIMEOUT: (022D):(FFFFEEEE):(0000081C):
    (007BD8B0):(STC01545):(2010/05/04 10:21:37.074040):
    (2010/05/04 10:21:37.075264):(2010/05/04 10:21:37.075315):
    (com.abc.def.ejb.EJSRemoteStateless_00000000):
    (evaluateXXX):(hostname=host.name.com port=33333):():():()
    
    BBOO0232W A request for Class Name
    'com.abc.def.ejb.EJSRemoteStateless_00000000' and
    Method Name 'evaluateXXX', from
    hostname=host.name.com port=33333, has timed out.  The servant
    process associated with the request will be terminated. Request
    Id(FFFFEEEE)
    
    Since there is only one servant region, the server issues
    BBOO0299I SERVER XXXPLEX/YYYY/SERVERA HAS NO SERVANTS. WORK IS
    BEING REJECTED.
    
    2 seconds later the controller region ABENDs with S0C4-4.
    
    The S0C4 psw instruction counter points into method
    ORB_Request::setRTN_RSN_SevereError(ORB_Request::commRSN_Codes)
    
    The call stack looks like this:
    
    ORB_Request::construct_and_send_message(SessionHandle*,GIOP_
                +00087E3C              BBOBOA
    ORB_Request::comm_outbound_response()
                +00001194              BBOBOA
    ORB_Request_Registry::drainRegistry(unsigned int)
                +000009A2              BBOBOA
    cleanupForLastSR()
                +000008DA              BBOBOA
    ACR_ExecutionThread::ProcessSrCleanup(acrwObj*)
                +00000132              BBOBOA
    ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*)
                +0000132E              BBOBOA
    ACR_ExecutionRoutine
                +0000053E              BBOBOA
    CEEVROND    +000011F4              CEEPLPKA
    CEEOPCMM    +000009A2                                   CEEOPCMM
    
    The S0C4 occurs because the ORBR being used by method
    comm_outbound_response() has been freed.
    
    DSA for comm_outbound_response + 0x840 -> ORBR control block.
    
    The ORBR eyecatcher in the first 8 bytes say RBROOOBB. It
    should be BBOOORBR while the ORBR is
    allocated. The letters are reversed just before the storage
    is released.  The Heap Pool Prefix for this ORBR  indicates
    the storage has been freed because the free chain pointer is
    non-zero.
               hp 3     hp 3    reversed
              element  free      |  eyecatcher
               |       chain     |
               |       ptr       |
               |        |        |
               V        V        V
    5C7B7FD8. 00000003 5C7B96F8 D9C2D9D6 D6D6C2C2 |        RBROOOBB|
    5C7B7FE0. 00000000 00000000 00000000 00000000 |................|
    5C7B7FF0. 00000000 00000000                   |........        |
    
    The eyecatcher should be BBOOORBR.
    
    This ORBR control block is associated with a new request coming
    into the controller. The controller places the ORBR into the
    ORBR Registry. Since there are no servant regions, the
    controller returns it to the requestor and removes it from the
    ORBR Registry . Because the servant region has recently
    ended due to a timeout, another thread in the controller is
    running ORB_Request_Registry::drainRegistry() to remove every
    ORBR in the ORBR Registry. There is a small window where
    drainRegistry() can obtain an ORBR that is being freed
    concurrently on another thread. This causes the S0C4.
    

Local fix

  • Run with multiple servant regions to minimize the chance that
    all servants will be unavailable at the same time.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.0.1                               *
    ****************************************************************
    * PROBLEM DESCRIPTION: ABEND0C4/ABENDS0C4 in WebSphere         *
    *                      Application Server for z/OS             *
    *                      Controller following a BBOO0299I        *
    *                      message.                                *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Message BBOO0299I is issued when a server has lost its last
    servant and autopause is enabled (control_region_dreg_on_no_srs
    set to 1).  The ORB_Request registry is drained of work at this
    time (ORB_Request_Registry::drainRegistry). There exists a
    timing window where drainRegistry can process a newly arrived
    request at the same time as the request is being forwarded or
    rejected because of the server's paused state.  Both of these
    threads will attempt to cleanup the ORB_Request.
    
    An additional symptom is a dump issued with ERRNO=C9C212F7.
    This is a dump taken after detecting an incorrect ORB_Request
    object.
    

Problem conclusion

  • Code has been modifed in the inbound request paths to not
    register inbound requests unless we are accepting work for
    servants.
    
    APAR PM13797 is currently targeted for inclusion in Service
    Level (Fix Pack) 6.0.2.43 of WebSphere Application Server
    V6.0.1.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM13797

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    601

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2010-05-05

  • Closed date

    2010-07-13

  • Last modified date

    2010-10-02

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM13866 PM13871

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R601 PSY UK60277

       UP10/09/23 P F009

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.0.1

Reference #: PM13797

Modified date: 02 October 2010