IBM Support

PI66709: WMQ: SHUTDOWN HANGS FOR AN MQ APPLICATION AND THEN THE QUEUE MANAGER AFTER Z/OS MAINTENANCE IS APPLIED.

A fix is available

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • z/OS maintenance for PUT 1604 or RSU 1604 has been applied that
    includes PTFs
      UA81297 z/OS 2.2
      UA81298 z/OS 1.13
      UA81300 z/OS 2.1
    
    Termination of an application and/or queue manager may result
    in a hang or "no work" loop waiting for tasks to end.
    
    The problem occurs for MQ batch applications, including RRS
    batch such as WebSphere Application Server, that end without
    doing an MQDISC and which have been doing MQGETs with MQGMO_WAIT
    or MQGMO_SET_SIGNAL. Those hung applications can in turn affect
    MQ shutdown.
    
    Systrace has a timed wait / post loop.
    
    TCBs for the application address space have SVRBs that are
    waiting with PSWs in CSQMEOTC. They are waiting for the wEndECB
    to be posted by CSQBTEXT.
    
    The expectation in the MQ code is that CSQBTEXT is initially
    scheduled as an exit on a STIMERM call and then continues to
    issue 1 second STIMERM calls to reschedule itself until it is
    either notified that the queue manager is terminated or that
    the application task no longer requires the exit to be running.
    
    In the hang situation, the exit would be driven repeatedly
    until the application address space starts to end, at which
    point STIMERM is issued but the CSQBTEXT exit does not get
    re-driven.
    
    If the application is WebSphere Application Server, the shut
    down of the control region may time out with ABENDSA03 and
    ABENDSE6C and messages:
    
     BPXM020I BPXBATCH FAILED BECAUSE MVSPROCCLP (BPX1MPC) FAILED
     WITH RETURN CODE 0000009D REASON CODE 0B7000B9
    
     BPXP018I THREAD 0000000000000000, IN PROCESS nnnnnnnn, ENDED
     WITHOUT BEING UNDUBBED WITH COMPLETION CODE 80A03000 AND
     REASON CODE 00000000
    
     BPXM020I BPXBATCH FAILED BECAUSE MVSPROCCLP (BPX1MPC) FAILED
     WITH RETURN CODE 0000009D REASON CODE 0B7000B9
    
    The reason code 0B7000B9 means :
    "JRActiveProcess: Multiple processes in the address space
     prevent the termination of the job step process."
    
    For the terminating queue manager, the PSW for the SVRB for the
    STIMER (0A2F) WAIT request is in CSQ3EPX.
    
    STOP QMGR MODE(FORCE) does not end the queue manager.
    
    
    ** This fix is required even if the PTF for OA50970 is **
    ** applied to z/OS.                                    **
    
    Additional Symptom(s) Search Keyword(s):
    ABENDA03 A03 SA03 S0A03
    ABENDEC6 EC6 SEC6 S0EC6
    

Local fix

  • To prevent the problem:
     A ++APAR is available from the support center.  The queue
     manager will need to be recycled to pick up the fix.
     .
     As stated in OA50970, if the MQ relief cannot be installed
     promptly, it is advised to remove the z/OS PTF until the time
     the MQ relief can be installed.
    .
    If the problem has occurred:
    Use MVS CANCEL for the hung connected application regions.
    .
    If you need to determine the hung application, you can create a
    dump of the queue manager:
        /DUMP COMM=(dump title)
        /R xx,SDATA=(CSA,RGN,PSA,SQA,LSQA,TRT,SUM,GRSQ),CONT
        /R xx,JOBNAME=(ssidMSTR),END
    In IPCS, run the command VERBX GRSTRACE and search for ERLYOLRH,
    for instance:
     MAJOR NAME: SYSZCSQ3
       MINOR NAME: C5D9D3E8D6D3D9C816078530
       *ERLYOLRH....        *
         SCOPE: SYSTEM    SYSNAME: <id>     STATUS: *SHARED*   /OWN
           ASID: <nnnnnnnn>   TCB: <nnnnnnnn>  JOBNAME: <batch job>
           Critical ENQ Time(s):
             Request:     <date> <time>
             Grant:       <date> <time>
    Compare the ENQ time with the time the dump was created. The
    ERLYOLRH ENQ is not expected to be held for very long.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED: All users of Websphere MQ for z/OS Version 7 *
    *                 Release 1 Modification 0.                    *
    ****************************************************************
    * PROBLEM DESCRIPTION: After application of z/OS maintenance   *
    *                      APAR OA49676 (PTFs UA81297/UA81298/     *
    *                      UA81300) MQ batch                       *
    *                      application may hang and prevent        *
    *                      shutdown of the Queue Manager.          *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
     The z/OS maintenance (OA49676) affects the end of task
    processing for MQ batch applications which do not disconnect
    from MQ prior to ending. This leaves the application in a hung
    state, and prevents the queue manager from completing a
    shutdown. This requires the application to be cancelled to allow
    shutdown to complete correctly. This also prevents Websphere
    Application Server from ending correcting when running MQ
    applications.
    

Problem conclusion

  • End of task processing for MQ applications has been updated to
    correctly complete in this scenario.
    100Y
    CSQMEOTC
    

Temporary fix

  • *********
    * HIPER *
    *********
    

Comments

APAR Information

  • APAR number

    PI66709

  • Reported component name

    WMQ Z/OS V7

  • Reported component ID

    5655R3600

  • Reported release

    100

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    YesHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-07-29

  • Closed date

    2016-08-16

  • Last modified date

    2017-02-10

  • APAR is sysrouted FROM one or more of the following:

    PI66686

  • APAR is sysrouted TO one or more of the following:

    UI40127

Modules/Macros

  • CSQMEOTC
    

Fix information

  • Fixed component name

    WMQ Z/OS V7

  • Fixed component ID

    5655R3600

Applicable component levels

  • R100 PSY UI40127

       UP16/08/25 P F608 ¢

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.



Document information

More support for: WebSphere MQ
APAR / Maintenance

Software version: 7.1

Reference #: PI66709

Modified date: 10 February 2017