A fix is available
APAR status
Closed as program error.
Error description
During periods of high-CPU utilization your CMAS network connections timeout and may fail repository synchronization possibly leading to an eventual CMAS isolation event. In this specific instance, the isolation events occurred during the evenings when CPU was highly utilized by higher priority batch service classes, however any slowdown or logic error where a CICS MAS might acquire a global lock could also surface a CMAS isolation event. The specific symptoms for this APAR show up as follows: - During high CPU period the FS (First-Speaker) CMAS begin communication with the QS (Quiet-Side) CMAS. - Each CMAS connection requires a pair of send/receive link tasks to be attached in both the FS an QS CMASes. - The FS CMAS initiated communication with the QS CMAS. The QS CMAS attached a send link task and receive link task. - The receive link task spawns a network lock task under transid LPLK in the QS CMAS. Due to the high CPU, the LPLK task enters an XSWX wait for exclusive access to the network lock. - During the time the receive link task in the QS CMAS is awaiting for exclusive control of the network lock, the send link task in the QS CMAS is timed out as indicated in the QS CMAS EYULOG by message: EYUCS0204W A timeout has occurred with CMAS xyz (where zyz is the FS CMAS). - The send link task abnormal termination due to timeout cleans up and resets the link descriptor for the connection to the FS CMAS. - During this time the initial receive link task in the QS CMAS is still waiting for exclusive access to the network lock so it can generate new DLSTs and TLSTs for the direct connection to the FS CMAS. - The FS CMAS now attempts to re-establish the connection by contacting the QS CMAS again. This results in a new receive link task being attached in the QS CMAS as it finds the link in RESET from the previous send link task timeout termination. - The timing issue occurs as the new receive link task starts as the link is in RESET and then the previous receive link task obtains exclusive access to the network lock and calls CPDG and CPAG to generate DLSTSs and TSLTs for the direct connection to the FS CMAS and then is purged by CSLT with AZI2 for MRO as it is in CICS communications. - At this point the communications control structures indicate a direct connection however there is no send link task in the QS CMAS. - The FS CMAS continues to attempt contact every 2 minutes with the QS CMAS until it finally is isolated after almost 1 hour. - The QS CMAS issues the following message every 2 minutes: EYUCL0113E Receive Link Task terminated abnormally for MRO Network connection with CMAS (FS CMAS). - The FS CMAS issues the following message every 2 minutes: EYUCS0009I Message received from CMAS (QS CMAS) : EYUCS0206W Exception trace issued: Point Id= 28 ,Debug text= ConExist . - The FS CMAS eventually issues message: EYUCP0023E CPQCM08 A communications failure has occurred while performing Repository Synchronization with CMAS (QS CMAS). That CMAS is being isolated. - The QS CMAS issues corresponding message: EYUCP0024E A communications failure has occurred while performing Repository Synchronization. This CMAS is being isolated at the request of CMAS (FS CMAS). Additional Symptom(s) Search Keyword(s): KIXREVWJB - EYUCP0205S Repository Synchronization with CMAS (QS CMAS) failed. - EYUCP0203I Repository Synchronization started with (QC CMAS). - EYUXM0505E Method = CLMU Debug = CLMUXMSM Comp = 08 SComp = L MsgID = 0000 PtID (XMSM) = 0C -
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All CICSPlex SM V5R1M0 Users * **************************************************************** * PROBLEM DESCRIPTION: - During periods of high-CPU * * utilization CMAS network connections * * may timeout and may fail repository * * synchronization, possibly leading to * * an eventual CMAS isolation event. * * * * Either CMAS partner may receive * * multiple EYUCL0113E messages * * indicating that a SEND and/or * * RECEIVE link task has terminated * * abnormally. If isolation occurs, * * one partner will issue message * * EYUCP0023E while the other partner * * will issue mesage EYUCP0024E. * * * * Additionally, one partner may issue * * multiple instances of EYULOG message * * EYUCS0009I. The text of this * * message will be similar to the * * following: * * * * EYUCS0009I Message received from * * CMAS (<partner cmasname>) : * * EYUCS0206W Exception trace issued: * * Point Id= 28 ,Debug text= * * ConExist . * * * * - If a send task for a CMAS to CMAS * * connection does not respond to a * * terminate request during CMAS * * termination and is forcepurged, an * * abend ASRA/S0C4 may occur in method * * EYU0CLMS (CLMS - MRO connection) or * * EYU0CLCS (CLCS - LU62 connection). * **************************************************************** * RECOMMENDATION: After applying the PTF that resolves this * * APAR, all CMASes must be restarted. Note * * that the restarts do not need to occur at * * the same time. * **************************************************************** - When a connection between two CMASes is timed out because one of the partners is not responding, the CMAS with the lower alphabetic SYSID will attempt to re-aquire the connection. If during the new connection attempt, the Receive task in the partner CMAS hangs waiting to acquire the CPSM communications network lock, it is possible that the new connection attempt will time out also. When that occurs, the SEND link task will be timed out prior to the RECEIVE link task. If the Receive task in the partner CMAS acuires the lock after the Send task terminates but prior to it being timed out, then method EYU0CLMU (CLMU - MRO connection) or EYU0CLCU (CLCU - LU62 connection) does not verify that the Send task is still active, and completes the Receive link side of the connection. However, since the Send task has already terminated, the connection does not fully complete, leading to the errors described above. - When a CMAS to CMAS connection send task is forcepurged, a recovery routine is given control in method CLMS or CLCS, where control block pointers are restored and link cleanup is performed. If the CMAS is in termination, not all control block pointers are restored properly, which can result in abend ASRA/S0C4 when CLMS or CLCS performs link cleanup.
Problem conclusion
- Methods CLMU and CLCU have been updated to verify if the Send task is still active after receiving the CPSM communications network lock. If the Send task is no longer active, the CLMU/CLCU will clean up the link and terminate. - Methods CLMS and CLCS have been updated to ensure that all control block pointers are restored if a send task is forcepurged during CMAS termination.
Temporary fix
********* * HIPER * ********* FIX AVAILABLE BY PTF ONLY
Comments
APAR Information
APAR number
PM77989
Reported component name
CICS TS Z/OS V5
Reported component ID
5655Y0400
Reported release
80M
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt
Submitted date
2012-11-28
Closed date
2013-01-15
Last modified date
2015-03-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UK90959
Modules/Macros
EYU0CLCS EYU0CLCT EYU0CLCU EYU0CLMS EYU0CLMT EYU0CLMU
Fix information
Fixed component name
CICS TS Z/OS V5
Fixed component ID
5655Y0400
Applicable component levels
R80M PSY UK90959
UP13/01/17 P F301
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.1","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.1","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
04 March 2015