IBM Support

PM40551: BBOT0004E: RRS SERVICE ATREND FAILED WITH RETURN CODE=931 ALSO, BBO0006E, REASON=C9C2120C WEBSPHERE Z/OS 6.1.0.33

Fixes are available

7.0.0.21: WebSphere Application Server V7.0 Fix Pack 21
8.0.0.2: WebSphere Application Server V8.0 Fix Pack 2
8.0.0.3: WebSphere Application Server V8.0 Fix Pack 3
7.0.0.23: WebSphere Application Server V7.0 Fix Pack 23
8.0.0.4: WebSphere Application Server V8.0 Fix Pack 4
7.0.0.25: WebSphere Application Server V7.0 Fix Pack 25
8.0.0.5: WebSphere Application Server V8.0 Fix Pack 5
7.0.0.27: WebSphere Application Server V7.0 Fix Pack 27
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.27: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
7.0.0.21: Java SDK 1.6 SR9 FP2 Cumulative Fix for WebSphere
7.0.0.23: Java SDK 1.6 SR10 FP1 Cumulative Fix for WebSphere
7.0.0.25: Java SDK 1.6 SR11 Cumulative Fix for WebSphere Application Server
7.0.0.27: Java SDK 1.6 SR12 Cumulative Fix for WebSphere Application Server
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Reported on 6.1.0.33 (AM34450)
    
    Failure would resemble this:
     Trace: 2011/04/18 06:06:01.913 01 t=nnnnnn c=UNK key=P8
    (13007002)
       ThreadId: 00000058
       FunctionName: com.ibm.ws390.tx.NativeTransactionContext
       SourceId: com.ibm.ws390.tx.NativeTransactionContext
       Category: AUDIT
       Extended Message: BBOT0004E: RRS SERVICE ATREND FAILED WITH
    RETURN CODE=931.
     Trace: 2011/04/18 06:06:05.649 01 t=nnnnnn c=UNK key=P8
    (0000000A)
       Description: Log Boss/390 Error
       from filename: ./bbooboai.cpp
       at line: 5731
       error message: BBOO0006E WEBSPHERE FOR Z/OS PROCESS servant
    ENDED ABNORMALLY, REASON=C9C2120C.
    
    Additionally, protocol_http_timeout_output_recovery=SESSION
    
    There should also be HTTP REQUEST TIMEOUTs.
    

Local fix

  • As a workaround, investigate why the HTTP requests are timing
    out and consider changing the current HTTP timeout value.
    Because the timeouts occur, having then
    protocol_http_timeout_output_recovery=SESSION causes this abend.
    
    This setting is documented in the Info Center and does note:
    "Using the session setting might result in a loss of resources
    if the dispatched HTTP request loops or becomes inactive."
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V6.1 and V8.0                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Servant region terminates after         *
    *                      receiving a 931 return code from        *
    *                      RRS' ATREND service call.               *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    The servant region terminates after having received a 931
    invalid return code from RRS' ATREND service call.
    The invalid return is triggered by the WebSphere Application
    Server transaction manager's attempt to end the Unit
    of Recovery (UR) a second time. This condition is prompted
    by the following: High HTTP workload,
    protocol_http_timeout_output_recovery setting set
    to SESSION, and an insufficient HTTP timeout value for
    high volume workloads such that numerous HTTP
    timeouts are encountered prior to the termination
    of the servant region.
    The following output can be seen in the controller region's
    sysprint log immediately prior to the time of the
    servant region failure.
    Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx
    Description: Throw CORBA system exception
    exception id: CORBA::INTERNAL
    minor code: c9c21210
    from filename: ./bbooboat.cpp
    at line: 911
    Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx
    Description: Java exception created
    from filename: ./bboocjex.cpp
    Method Name: createExceptionForJava(...)
    org.omg.CORBA.INTERNAL: Exception occured during native
    method call from JAVA  vmcid: 0xc9c21000  minor code: 210
    completed: No
    at com.ibm.ws390.tx.ControllerTransactionManagerImpl.
    contextToGtidRow(Native Method)
    at com.ibm.ws390.tx.ControllerTransactionManagerImpl.
    begin(ControllerTransactionManagerImpl.java:1588)
    at com.ibm.ws390.tx.ControllerTransactionManagerImpl.
    getControl(ControllerTransactionManagerImpl.java:1967)
    at com.ibm.ws390.tx.ControllerTransactionManagerImpl.
    enlistXaResources(ControllerTransactionManagerImpl.java:2827)
    at
    com.ibm.ws390.tx.ControllerTransactionManagerImpl.
    commit(ControllerTransactionManagerImpl.java:2316)
    The following can be seen in the servant region sysprint log
    at the time of the failure:
    Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx
    ThreadId: xxxxxxxx
    FunctionName: com.ibm.ws390.tx.NativeTransactionContext
    SourceId: com.ibm.ws390.tx.NativeTransactionContext
    Category: AUDIT
    ExtendedMessage: BBOT0004E: RRS SERVICE ATREND FAILED WITH
    RETURN CODE=931.
    

Problem conclusion

  • Code was added to prevent a second attempt to end the
    unit of recovery under the stated conditions.
    
    APAR PM40551 is currently targeted for inclusion in Service
    Level (Fix Pack) 6.1.0.43 of WebSphere Application Server V6.1.
    and Fix Pack 8.0.0.2 for WebSphere Application Server V8.0.
    
    Sysroute APAR PM43823 will be used to deliver this fix in
    WebSphere Application Server V7.0 Fix Pack 7.0.0.21.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM40551

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2011-06-01

  • Closed date

    2011-07-15

  • Last modified date

    2012-04-03

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PM43823

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK76696

       UP12/03/18 P F203

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere Application Server for z/OS
General

Software version: 6.1

Reference #: PM40551

Modified date: 03 April 2012