PH39364: MQ Z/OS: PROVIDE AN INDICATION THAT A QUEUE MANAGER IS NOT ABLE TO PROCESS QSG NOTIFICATIONS AT THE RATE THEY ARE ARRIVING

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• MQ should provide an external warning if a queue manager is not
  able to process Queue Sharing Group (QSG) notifications at the
  rate they are arriving. MQ can not prevent the notifications
  from arriving, because they are a function of:
  - configuration such as shared queue triggering
  - application design
    There will be fewer notifications if there are fewer MQOPEN
      and MQCLOSE requests for the queue, so code the triggered
      application to
      - use the MQGMO_WAIT option with a WaitInterval
      - get more than one message if business logic allows
  - workload volume
  
  Further details about the QSG notifications:
  - MQ queue managers in a QSG use XCF messaging to send
  notifications about events relating to the state of shared
  queues.
  - There are multiple reasons why notifications may be sent,
  including when the last application with the queue open for
  input closes the queue. This notification is used to evaluate
  whether further triggering is required.
  - XCF passes these messages to MQ by calling an MQ routine for
  each message that arrives.
  - The environment in which this routine is invoked by XCF is
  not suitable for performing the work that MQ needs to do in
  response to the notification. Because of this, the MQ routine
  allocates a DXWB control block to hold the notification details
  and adds the block to a chain of pending notifications.
  - A separate MQ process, with the appropriate environment to
  take action for the notifications, works through the chain of
  DXWBs.
  - For each DXWB, the process takes the necessary actions and
  releases the DXWB.
  - In some environments, the way the triggered application uses
  the shared queue causes the rate at which XCF messages are
  arriving on the queue manager to exceed the rate at which the
  MQ process can handle the notifications. The chain of DXWBs
  waiting to be processed can grow very large in private storage.
  
  Other symptoms listed in "DXWB control blocks fill Subpool 229
  Key 7 and cause ABEND878 for IBM MQ" at
  https://www.ibm.com/support/pages/node/6332733 include:
  - The amount of free storage reported in CSQY220I decreases
  suddenly, and the message becomes more frequent:
  CSQY220I Queue manager is using mm MB of local storage, nn MB
  are free
  - Eventually storage might run out:
      CSQY221I Queue manager is short of local storage
      CSQY222E Queue manager is critically short of local storage
               - take action
  - ABENDS5C6 occurs with reason codes such as 00E2000B
  - ABEND878-00000010 occurs
  
  Additional keywords:
  SP229 KEY7 K7
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 1 Modification 0 and                 *
  *                 Release 2 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Various symptoms caused by a short on   *
  *                      storage (SOS) condition occur when the  *
  *                      rate of XCF notifications sent between  *
  *                      queue managers in a QSG exceeds the     *
  *                      capacity of the queue manager to        *
  *                      process them, and leading to large      *
  *                      areas of storage containing DXWB        *
  *                      control blocks.                         *
  *                                                              *
  *                      Additional symptoms can include:        *
  *                      - abends relating to storage exhaustion *
  *                        e.g. S878                             *
  *                      - 5C6 abends relating to an inability   *
  *                        for MQ to obtain required storage     *
  *                      - local lock contention in the queue    *
  *                        manager address space                 *
  *                      - abnormal termination of the qmgr with *
  *                        abend code S6C6                       *
  ****************************************************************
  XCF messages are sent between queue managers to notify other
  members of a state change that requires other queue managers to
  take action.
  Examples of this include:
   - the last input handle for a shared queue is closed
   - the arrival of a message on a shared queue configured with
     TRIGTYPE(EVERY)
  
  When the arrival rate of such notifications exceeds the capacity
  of the queue manager to process them, the storage required to
  hold the pending requests increases, eventually leading to
  abends and abnormal queue manager termination, however there is
  no prior indication of the problem.
  

Problem conclusion

• When processing MQ notifications sent over XCF, additional
  optimisation has been added to allow increased parallelism in
  processing the notifications.
  
  When the number of outstanding notifications from a given queue
  manager exceeds 10000, new message CSQ5041E will be issued
  once a minute until the number of outstanding notifications from
  that source has dropped below 10000.
  
  The IBM MQ knowledge centre is updated as follows:
  
  The page "Db2 manager messages (CSQ5...)"
  
  https://www.ibm.com/docs/en/ibm-mq/9.2?
                             topic=zos-db2-manager-messages-csq5
  
  is updated to include new message CSQ5041E:
  
  "
  CSQ5041E
      csect-name QSG notification flood detected from qmgr-name
                                        (outstanding requests=num)
  Explanation
  
      The queue manager is receiving notifications from qmgr-name
      at a faster rate than they can be processed, and the backlog
      has reached 10000 pending requests.
  
  System action
     The message will repeat every minute until the backlog from
                                         qmgr-name is below 10000.
     If the backlog continues to grow, eventual storage exhaustion
                and abnormal queue manager termination will occur.
  
  System programmer response
  
      Determine which applications on the specified queue manager
                 are responsible for triggering the notifications.
      Consider updating the application to use designs that
                                     generate fewer notifications.
  
      Examples of application designs that typically generate high
                             numbers of QSG notifications include:
  
      - Using shared application queues that are configured with
                                                  TRIGTYPE(EVERY).
        Consider using TRIGTYPE(FIRST) instead.
  
      - Applications repeatedly closing the last input handle for
                                                   a shared queue.
        Consider keeping the handle to the shared queue open until
          it is no longer required, rather than opening/closing it
                                                       repeatedly.
  
  "
  

Temporary fix

Comments

• ×**** PE22/03/17 FIX IN ERROR. SEE APAR PH44959  FOR DESCRIPTION
  ×**** PE22/03/17 FIX IN ERROR. SEE APAR PH44959  FOR DESCRIPTION
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI78114 UI78115 UI78116 UI78117 UI78118 UI78119 UI78120 UI78121
  UI78122 UI78123 UI78124 UI78125

Modules/Macros

• CSQ5245M CSQ5ERST CSQ5LEPL CSQ5MGX0 CSQ5STRT CSQ5XWK1 CSQ5XWK2
  CSQEWCMQ CSQF5DIC CSQF5DIE CSQF5DIF CSQF5DIK CSQF5DIU CSQFMTXC
  CSQFMTXE CSQFMTXF CSQFMTXK CSQFMTXU CSQIDDA1 CSQIRECP
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R100 PSY UI78120

     UP21/12/11 P F112 ¢

• R101 PSY UI78121

     UP21/12/11 P F112 ¢

• R102 PSY UI78122

     UP21/12/11 P F112 ¢

• R103 PSY UI78123

     UP21/12/11 P F112 ¢

• R104 PSY UI78124

     UP21/12/11 P F112 ¢

• R105 PSY UI78125

     UP21/12/11 P F112 ¢

• R200 PSY UI78114

     UP21/12/11 P F112 ¢

• R201 PSY UI78115

     UP21/12/11 P F112 ¢

• R202 PSY UI78116

     UP21/12/11 P F112 ¢

• R203 PSY UI78117

     UP21/12/11 P F112 ¢

• R204 PSY UI78118

     UP21/12/11 P F112 ¢

• R205 PSY UI78119

     UP21/12/11 P F112 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.