A fix is available
APAR status
Closed as program error.
Error description
Reported on 6.1.0.33 (AM34450) Failure would resemble this: Trace: 2011/04/18 06:06:01.913 01 t=nnnnnn c=UNK key=P8 (13007002) ThreadId: 00000058 FunctionName: com.ibm.ws390.tx.NativeTransactionContext SourceId: com.ibm.ws390.tx.NativeTransactionContext Category: AUDIT Extended Message: BBOT0004E: RRS SERVICE ATREND FAILED WITH RETURN CODE=931. Trace: 2011/04/18 06:06:05.649 01 t=nnnnnn c=UNK key=P8 (0000000A) Description: Log Boss/390 Error from filename: ./bbooboai.cpp at line: 5731 error message: BBOO0006E WEBSPHERE FOR Z/OS PROCESS servant ENDED ABNORMALLY, REASON=C9C2120C. Additionally, protocol_http_timeout_output_recovery=SESSION There should also be HTTP REQUEST TIMEOUTs.
Local fix
As a workaround, investigate why the HTTP requests are timing out and consider changing the current HTTP timeout value. Because the timeouts occur, having then protocol_http_timeout_output_recovery=SESSION causes this abend. This setting is documented in the Info Center and does note: "Using the session setting might result in a loss of resources if the dispatched HTTP request loops or becomes inactive."
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V7.0 * **************************************************************** * PROBLEM DESCRIPTION: Servant region terminates after * * receiving a 931 return code from * * RRS' ATREND service call. * **************************************************************** * RECOMMENDATION: * **************************************************************** The servant region terminates after having received a 931 invalid return code from RRS' ATREND service call. The invalid return is triggered by the WebSphere Application Server transaction manager's attempt to end the Unit of Recovery (UR) a second time. This condition is prompted by the following: High HTTP workload, protocol_http_timeout_output_recovery setting set to SESSION, and an insufficient HTTP timeout value for high volume workloads such that numerous HTTP timeouts are encountered prior to the termination of the servant region. The following output can be seen in the controller region's sysprint log immediately prior to the time of the servant region failure. Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx Description: Throw CORBA system exception exception id: CORBA::INTERNAL minor code: c9c21210 from filename: ./bbooboat.cpp at line: 911 Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx Description: Java exception created from filename: ./bboocjex.cpp Method Name: createExceptionForJava(...) org.omg.CORBA.INTERNAL: Exception occured during native method call from JAVA vmcid: 0xc9c21000 minor code: 210 completed: No at com.ibm.ws390.tx.ControllerTransactionManagerImpl. contextToGtidRow(Native Method) at com.ibm.ws390.tx.ControllerTransactionManagerImpl. begin(ControllerTransactionManagerImpl.java:1588) at com.ibm.ws390.tx.ControllerTransactionManagerImpl. getControl(ControllerTransactionManagerImpl.java:1967) at com.ibm.ws390.tx.ControllerTransactionManagerImpl. enlistXaResources(ControllerTransactionManagerImpl.java:2827) at com.ibm.ws390.tx.ControllerTransactionManagerImpl. commit(ControllerTransactionManagerImpl.java:2316) The following can be seen in the servant region sysprint log at the time of the failure: Trace: xxxx/xx/xx xx:xx:xx.xxx xx t=xxxxxx c=UNK key=xx ThreadId: xxxxxxxx FunctionName: com.ibm.ws390.tx.NativeTransactionContext SourceId: com.ibm.ws390.tx.NativeTransactionContext Category: AUDIT ExtendedMessage: BBOT0004E: RRS SERVICE ATREND FAILED WITH RETURN CODE=931.
Problem conclusion
Code was added to prevent a second attempt to end the unit of recovery under the stated conditions. APAR PM43823 is currently targeted for inclusion in Service Level (Fix Pack) 7.0.0.21 of WebSphere Application Server V7.0. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM43823
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-07-15
Closed date
2011-09-21
Last modified date
2012-02-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R700 PSY UK74996
UP12/01/18 P 1201
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
10 February 2022