PI11896: RECOVERING STRUCTURE IRR401I 878 ABEND DURING RACF PROCESSING LOOP LOG RBA CSQY224I

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• IRR401I 878 ABEND DURING RACF PROCESSING OF
  LOCATE REQUEST FOR ENTRY MQSTARC1.CSQ1.A0022474
  IRR401I 878 ABEND DURING RACF PROCESSING
  IRR413I RACF MANAGER REQUEST ID WAS RCK06001
  IEC614I PRLSE FAILED - RC 016, DIAGNOSTIC INFORMATION IS
  (08160878),
  CSQ1MSTR,P20119,XXXARC1.CSQ1.A0022474
  IEC216I A14-04,IFG0202E,CSQ1MSTR,CSQ1MSTR,SYS01527,2936,P20119,
  MQSTARC1.CSQ1.A0022474
  CSQJ104E xxxxx CSQJDS04 RECEIVED ERROR STATUS
  DUMP TITLE=JOB=CSQ1MSTR,CAS ESTAE-5695DF105 R210,ABEND878,
                +0000,FMID=HDZ2110 ,MAINT=NONE
  IEC342I CATALOG ABEND OCCURRED
  L3 found:
   DISPLAY QSTATUS processing has got an ILAT latch (CSQIESEL) and
  CSQM1DQQ is suspended with CSQEOCRQ waiting for CSQEWRKT to
  locate the problem structure task. However, CSQEWRKT is waiting
  for the ECTA latch for the BOOK structure, which is held by the
  BOOK structure task, STRTSK04.
  STRTSK04 is suspended in CSQEROUT waiting for structure backup/
  recovery task BKPREC04, which is waiting for BKPREC0E which is
  processing the recovery of the problem structure and is
  suspended with CSQIESEL waiting for the ILAT latch mentioned
  above (CSQIESEL is invoked by CSQERCF2). This has therefore
  resulted in a deadly-embrace, and has lead to a build-up of
  DXWB control blocks (as CSQEDSG1 is waiting for the ECTA latch
  for the BOOK structure).
  Additional Symptoms:
  QMGR restart failed with messages CSQY224I and CSQY225E
  SP229 KEY7 Subpool 229 Key 7
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of WebSphere MQ for z/OS Version 7 *
  *                 Release 1 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Structure recovery processing and       *
  *                      DISPLAY QSTATUS processing hang.        *
  *                      Additional symptoms may include the     *
  *                      queue manager going short on storage    *
  *                      and abending with S878 and              *
  *                      5C6-00E2000B.                           *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  A command to recover 2 or more structures is being issued or
  scheduled by auto-recovery.
  Recovery runs on the backup recovery task for the structure with
  the oldest backup, for instance APPL1.
  As multiple structures are being recovered, CSQERRPB schedules
  CSQERCF2 on the backup recovery tasks for the other structures,
  before calling CSQERCF2 for structure APPL1. Once completed, it
  will suspend on backup recovery task ROBs to wait for the other
  backup recovery tasks to finish CSQERCF2 processing.
  CSQERCF2 processing for structure APPL2 has not yet reached the
  point where it tries to acquire a latch on the queue synonym
  chain.
  
  As CSQERCF2 has completed for structure APPL1, all shared queues
  have been restored, and flagged as cached in the CF structure.
  CSQERCF2 has scheduled a close request to the structure task,
  which disconnects from the structure and checks if the task can
  now go dormant. As the task is no longer used, it queues a
  request to the backup recovery task to quiesce and suspends,
  waiting for the task to go away.
  
  When a DISPLAY QSTATS command comes in with a wild card,
  matching a queue hosted on structure APPL1, CSQM1DQQ is invoked
  for the queue, detects that it is cached in the CF and attempts
  to get the most recent values from the CF, causing it to
  schedule an open request to the structure task for APPL1. This
  processing is delayed, due to the structure task waiting for the
  backup recovery task to finish processing.
  The DISPLAY QSTATS command is in the middle of CSQIESEL
  processing at this point, holding the relevant queue synonym
  chain latch.
  
  CSQERCF2 for structure APPL2 now reaches the point where it
  needs to restore the shared queues, and calls CSQIESEL to
  iterate over the queues. It tries to latch the queue synonym
  chain latch that is currently held by the thread processing the
  DISPLAY QSTATS command.
  
  The dead lock occurs:
   - The structure task APPL1 is waiting for the associated
     backup recovery task to complete processing
   - The backup recovery task for APPL1 is waiting for the
     backup recovery task for APPL2 to complete processing.
   - The backup recovery task for APPL2 is waiting for the
     queue synonym chain latch held by the thread processing
     the DISPLAY QSTATUS command.
   - The command processing the DISPLAY QSTATUS command is
     waiting for the structure task for APPL1 to become
     available.
  

Problem conclusion

• The code was changed to prevent the structure task from
  attempting to quiesce the backup recovery task if recovery
  processing is still in progress.
  100Y
  CSQEROUT
  CSQERRPB
  

Temporary fix

• *********
  * HIPER *
  *********
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI20146

Modules/Macros

• CSQEROUT CSQERRPB
  

Fix information

• Fixed component name

  WMQ Z/OS V7

• Fixed component ID

  5655R3600

Applicable component levels

• R100 PSY UI20146

     UP14/08/30 P F408 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.