PM17170: S0C4 IN METHODS SUCH AS ORB_REQUEST::GETACRWORKELEMENTPTR() DUE TO DOUBLE FREE OF BBOOORBR

Fixes are available

APAR status

Closed as program error.

Error description

High volume load tests are performed, with over 500 RMI/IIOP
requests per second being generated in serverX to another
server serverY.

If serverY is stopped or abended then the serverX controller
region will ABEND with various symptoms, mainly S0C4, due to
what appears to be a double free of a BBOOORBR control block.

Some example call stacks that were encountered are:

Traceback:
  DSA      Entry       E  Offset  Statement   Load Mod
(jvm "take dump" methods removed)
  20       __zerro     +00000EE4              CELQLIB
  21       __zerros    +0000032E              CELQLIB
  22       CEEHDSP     +00003F64              CELQLIB
CEEHDSP
  23       CEEOSIGJ    +0000096E              CELQLIB
CEEOSIGJ
  24       CELQHROD    +00000248              CELQLIB
CELQHROD
  25       CEEOSIGG    +00000000              CELQLIB
CEEOSIGG
  26       CELQHROD    +00000248              CELQLIB
CELQHROD
  27       ORB_Request::getACRWorkElementPtr()     <-- exception
                       +0000000C              BBGBOA
  28
Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                       +00000F9C              BBGBOA
  29       RUNCALLINMETHOD

Traceback:

  DSA      Entry       E  Offset  Statement   Load Mod
( ... "take dump" methods removed ... )
  12       __zerro     +00000EE4              CELQLIB
  13       __zerros    +0000032E              CELQLIB
  14       CEEHDSP     +00003F64              CELQLIB
CEEHDSP
  15       CEEOSIGJ    +0000096E              CELQLIB
CEEOSIGJ
  16       CELQHROD    +00000248              CELQLIB
CELQHROD
  17       CEEOSIGG    +00000000              CELQLIB
CEEOSIGG
  18       CELQHROD    +00000248              CELQLIB
CELQHROD
  19
Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                       +00000FCC              BBGBOA <-exception

Traceback:
  DSA      Entry       E  Offset  Statement   Load Mod

 1        ORB_Request::~ORB_Request()     <-exception
                      +00000122              BBGBOA
 2        ORB_Request::~ORB_Request()
                      +00000D88              BBGBOA
 3        BBO_BOA::return_ORB_request(ORB_Request*)
                      +00000074              BBGBOA
 4
Java_com_ibm_ws390_ziop_ZIOPCPPUtilities_process_1inbound_1r
                      +0000131E              BBGBOA
 5        RUNCALLINMETHOD
                      +00000000              *PATHNAM
 6        gpProtectedRunCallInMethod
                      +0000003E              *PATHNAM
 7        signalProtectAndRunGlue
                      +0000001C              *PATHNAM
 8        j9sig_protect
                      +00000356              *PATHNAM
 9        gpCheckCallin
                       +000000A4              *PATHNAM
10       callStaticIntMethodA
                       +00000072              *PATHNAM
11
 ZAioTCPChannelCppUtilities::readComplete(int,long,int,xrcd*)
                       +00000602              BBGCF
12
 CF_TCP_Request::readComplete(int,CF_Connection*,int,xrcd*)
                       +0000003A              BBGCF
13
 CF_TCP_Connection_Object_Manager::processAsyncReadComplete(a
                       +000010B6              BBGCF
14
 ACR_ExecutionThread::processTcpAsyncReadComplete(acrwObj*)
                      +00000024              BBGBOA
 15
ACR_ExecutionThread::RemoveAndProcessWork(ThreadCleanUp*)
                      +00002036              BBGBOA

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V6.1.0                               *
****************************************************************
* PROBLEM DESCRIPTION: ABEND0C4/ABENDS0C4 in WebSphere         *
*                      Application Server for z/OS Controller  *
*                      during handling of outbound IIOP        *
*                      requests under load.                    *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Storage corruption may occur in a WebSphere Application Server
for z/OS Controller while handling a large number of outbound
IIOP requests.  The server in question is acting as a client by
sending an IIOP request to some other server.  If the backend
server handling the IIOP request goes down for some reason, the
frontend client server becomes exposed to a timing window that
could result in storage corruption.

Symptoms of storage corruption can vary, but the most common
pattern is that the Controller will experience an ABEND0C4
and a stack trace of the failing thread will show that the
abend occurred in a function of an ORB_Request object, for
example:

ORB_Request::getACRWorkElementPtr()

Problem conclusion

In some cases, a timing window allowed for a double delete of
an ORB_Request object, causing corruption in the heap pool that
contains those objects.  This timing window was fixed so that
the double delete can no longer occur.

APAR PM17170 is currently targeted for inclusion in
Service Level (Fix Pack) 6.1.0.35 of WebSphere
Application Server V6.1

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM17170
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2010-06-24
Closed date
2010-08-30
Last modified date
2011-01-03

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

PM17172

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R610 PSY UK62676
UP10/12/17 P F012

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 October 2021

Tips

PM17170: S0C4 IN METHODS SUCH AS ORB_REQUEST::GETACRWORKELEMENTPTR() DUE TO DOUBLE FREE OF BBOOORBR

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R610 PSY UK62676

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?