PM43293: DDF AVAILABILITY AND SERVICEABILITY ENHANCEMENTS

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• DB2DDF
  Allow drain of queued connections waiting for a DBAT.
  **************************************************************
  Additional Symptoms and keywords:
   This APAR also addresses the following condition:
    Short on ECSA ssnmDIST DIST storage SOS due to
    KeepDynamicRefresh support with remote client applications,
    typically SAP related.
    LCBLQREP LWEL ESCA pool - DB2 QUEUE ELEMENT POOL
    LCBLQHDP LQHD ESCA pool - DC QUEUE HEADER POOL
    SP231 SUBPOOL231 SUBPOOL SUB POOL 231 KEY 7 07 KEY7 KEY07
    Symptoms may include:
    . 04E-00E2001F, LOC=DSNSLD1.DSNSVBK+06E2
      ABEND04E AB04E S004E 04E 04E-00E2001F 00E2001F RC00E2001F
        DSNSVBK DSNSVBK+06E2 06E2
    . 878-00000004
      ABEND878 AB878 S00878 878 878-00000004 00000004 RC00000004
    . 04E-00E20022, LOC=DSNSFBK.DSNSFBK+059E
      ABEND04E AB04E S004E 04E 04E-00E20022 00E20022 RC00E20022
        DSNSFBK DSNSFBK+059E 059E
        DB2STGLK/K
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Distributed Data Facility (DDF) users.   *
  *                 Specifically where DB2 is a member of a      *
  *                 data sharing group and is configured with    *
  *                 DDF THREADS = INACTIVE specified             *
  *                 (DSN6FAC CMTSTAT INACTIVE).                  *
  *                 DB2 10 for z/OS users only.                  *
  ****************************************************************
  * PROBLEM DESCRIPTION: New function is being provided to       *
  *                      assist users in managing connections    *
  *                      relative to remote client system        *
  *                      applications.                           *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Operational and/or serviceability improvements are required in
  the following areas:
  o When MAXDBAT is reached at a member of a data sharing group,
    requests from clients (that require a DBAT to process) will
    queue up until either CONDBAT has been reached or the number
    of requests waiting for a DBAT exceeds the MAXDBAT value.
    Those requests can be a request from an inactive connection
    to execute a new transaction or a request to establish a new
    connection from a client.  When a DBAT does become available,
    DB2 (DDF) will then associate the DBAT to the next queued
    connection request so the work can be processed.
    If, for some reason, DBATs do not become available to handle
    these requests, the queue of connection requests (waiting for
    a DBAT) will continue to grow until either the maximum number
    of remote connections (CONDBAT) has been reached or the depth
    of the queue has exceeded the MAXDBAT value.
    - When CONDBAT is reached, new connection requests will be
      rejected and message DSNL030I is issued with reason code
      00D31034.
    - If the depth of queue has been exceeded before CONDBAT is
      reached, then DB2 closes the TCP/IP socket and marks the
      connection for eventual clean up, but a DBAT is still
      needed to fully complete this connection clean up.
      A serviceability issue exists in this case because no
      message is issued to indicate that the client connection
      was closed.
    Eventually, DB2 (DDF) could reach a state where no new work
    could be accepted from a client because the maximum number of
    remote connections has been reached.
    Also, since the weight of the DB2 member might not instantly
    reflect a DBAT unavailability issue with the server, clients
    may continue to send requests to establish new connections to
    the member (which would then be instantly rejected) or send
    requests to run new transactions on existing connections
    (which would then queue up waiting for a DBAT to process the
    request).  This condition would continue at the member until
    the DBAT unavailability issue is resolved.
    However, if these client requests had come from distributed
    applications using the IBM Data Server Driver or Client
    product family and/or the DB2 Connect product family
    configured to operate in sysplex workload balancing mode,
    those client requests, if given the chance, could have been
    seamlessly redirected to another member of the data sharing
    group where execution resources may be more abundant.
    Another result of the behavior by DB2 to close the client
    socket (of a connection waiting for a DBAT to process its
    next request) is that the count of the number of connections
    from a client may not be reduced to zero when all work has
    eventually been processed. This inaccurate count may be
    observed in the DISPLAY LOCATION command report.
    DB2 needs to provide users with a more effective way to
    observe and manage this connection behavior.
  o DB2 for z/OS supports a "KeepDynamic Refresh" concept.  When
    packages are bound with KEEPDYNAMIC(YES), the DBAT/connection
    association must be maintained and hence there are fewer
    DBATs available to service other work.  A refresh capability
    exists that causes the DBAT/connection combination to be
    terminated after one hour of use or 20 minutes of being idle,
    hence allowing a replacement DBAT to be created to service
    other work.  To prevent this condition from impacting the
    remote application, this termination of the DBAT/connection
    only occurs if the client identifies itself with the ability
    to seamlessly move the client connection over to a new
    connection at the same member or a different member of a data
    sharing group.
    In supporting this KeepDynamic Refresh capability, a
    serviceability condition exists because no message is issued
    at the DB2 member indicating that this DBAT/connection
    termination event has occurred.
  o When a DB2 server thread (DBAT) abnormally terminates for
    whatever reason, the information sent to, and received by,
    the client (and possibly logged for some situations) may not
    contain the necessary level of information about the DB2 DBAT
    failure so that effective problem determination can be
    conducted.
  

Problem conclusion

• The following changes are being made to DB2 10 for z/OS:
  
  Draining of queued client connection requests.
  ---------------------------------------------------------------
  Two new subsystem parameters will be supported, MAXCONQN and
  MAXCONQW, for the DSN6FAC macro.  Their defaults will be
  to have essentially no effect, i.e. the maximum number of
  possible queued connection requests will continue to be
  controlled solely by the CONDBAT subsystem parameter (which is
  current behavior).
  
  MAXCONQN will represent the maximum number of inactive or new
  connections that can be queued waiting for a DBAT to process
  the request.  When such a request is added to the connection
  request queue and MAXDBAT has also been reached, then if the
  depth of the connection request queue has also reached
  the value for MAXCONQN (unless MAXCONQN was set to OFF), DDF
  will close the client connection with the longest time queued.
  This gives a remote client system the opportunity to redirect
  the work to other members of the group that have more resources
  to process the work.  This new capability will only occur when
  the DB2 subsystem is a member of a data sharing group and DB2
  was started with DDF THREADS set to INACTIVE. This subsystem
  parameter value can be updated online.
  
  When a client connection is closed due to MAXCONQN being
  exceeded, message DSNL030I will be issued with a new reason
  code, 00D31053.
  To eliminate potential flooding of messages to the console, the
  message will only be issued if five minutes have elapsed since
  this message was last issued relative to the same client IP
  address.
  
  MAXCONQW will represent the maximum wait time that a client
  connection will remain queued for a DBAT to process the next
  transaction or new connection request.  Each queued connection
  request entry is examined to see if its time waiting in the
  queue has exceeded the MAXCONQW value.  If the time has been
  exceeded, the client connection will be closed.  If the
  parameter was set to OFF, then the client connection request
  will wait indefinitely, as traditionally done, for a DBAT.
  This new capability will only occur when the DB2 subsystem
  is a member of a data sharing group and DB2 was started with
  DDF THREADS set to INACTIVE.  This subsystem parameter value
  can be updated online.
  
  When a client connection is closed due to MAXCONQW being
  exceeded condition, message DSNL030I will be issued with a new
  reason code value of 00D31054.
  To eliminate potential flooding of messages to the console, the
  message will only be issued if five minutes have elapsed since
  this message was last issued relative to the same client IP
  address.
  
  The -DISPLAY DDF DETAIL command report will be enhanced to
  display the current configured values for the MAXCONQN and
  MAXCONQW subsystem parameters via a new message, DSNL091I.  The
  number of closed connections due to MAXCONQN and MAXCONQW will
  be displayed via a new message, DSNL094I.  These two new
  messages can only be displayed when the subsystem is a member
  of a data sharing group.  Also, regardless of the values
  specified for MAXCONQN and MAXCONQW, a value of zero (OFF) will
  be assumed if DB2 is configured with DDF THREADS ACTIVE
  (DSN6FAC CMTSTAT=ACTIVE).
  Note: The DDF THREADS subsystem parameter is not online
    changeable.
  
  Various statistics values will also be enhanced or added.
  A new counter, QDSTNCQW is added to global DDF activity
  statistics (DSNDQDST) and will report the number of client
  connections closed due to MAXCONQW being exceeded.
  The description for an existing counter, QDSTNCQC, is changed
  to indicate that it reflects the number of client connections
  closed due to MAXCONQN being exceeded.
  An existing counter, QLSTCNVT, will be now be updated at the
  server to indicate the number of connections (at this location)
  that were closed due to either MAXCONQN and/or MAXCONQW being
  exceeded.  This counter is also reported, per remote location,
  via statistics trace CLASS(7) or IFCID365 as well as an IFC
  READS call against IFCID365.
  
  Reducing member health as number of connections increases.
  ---------------------------------------------------------------
  A new message, DSNL074I, will be issued when the current number
  of client connections exceeds either 80% or 90% of CONDBAT.
  Correspondingly, the DB2 (DDF) system health will also be
  reduced to 50% or 25%, respectively, of the current calculated
  DB2 health value and reported to WLM immediately.
  A new message, DSNL075I, will be issued when the number of
  client connections no longer exceeds 80% or 90% of CONDBAT.
  Correspondingly, the DB2 (DDF) system health will be improved
  back to either 50% or 100%, respectively, of the calculated
  DB2 health value and then reported to WLM.
  The new messages and the corresponding changes to the DB2 (DDF)
  system health value will only occur when the subsystem is a
  member of a data sharing group.
  
  The health value reported to WLM can now be obtained via
  the following messages:
  - DSNV507I issued for the ACTIVE MONITOR as a result of the
    -DISPLAY THREAD(*) TYPE(SYSTEM) command.
  - DSNL094I, a new message, issued as a result of the -DISPLAY
    DDF DETAIL command.  The new message will only be issued when
    the subsystem is a member of a data sharing group.
  
  KEEPDYNAMIC Refresh messages.
  ---------------------------------------------------------------
  Two different events can occur when KEEPDYNAMIC Refresh is
  enabled on the connection.  The connection/thread has been used
  too long (greater than one hour) and the connection/thread has
  been idle too long (greater than 20 minutes).  When either of
  the events have occurred, the DSNL027I message will be issued
  with two new reason codes,  00D3003E and 00D3003F, to indicate
  which event has occurred.
  To eliminate potential flooding of messages to the console, the
  message will only be issued if five minutes have elapsed since
  this message was last issued relative to the same client IP
  address.
    Note: A connection/thread is ONLY terminated after one hour
      of use or idle for 20 minutes when the connection/thread is
      not actively processing a transaction and/or holding a
      resource (other than the DBAT and allocated KeepDynamic
      packages) past a commit.
  
  DB2 DBAT Termination processing.
  ---------------------------------------------------------------
  DB2 DBAT termination processing has been improved to more
  effectively return information describing the cause of the
  termination.
  

Temporary fix

Comments

• ž**** PE13/01/29 PTF IN ERROR. SEE APAR PM81671  FOR DESCRIPTION
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UK90325

Modules/Macros

•    DSN@XAZP DSNDQDST DSNDQWPZ DSNFLDIR DSNFMDIR
  DSNLBABR DSNLCTRC DSNLDALB DSNLEDDA DSNLEDTM DSNLIDDC DSNLIINI
  DSNLIIN2 DSNLILLM DSNLILNR DSNLQACT DSNLQCRP DSNLQCTL DSNLQDIS
  DSNLTDDF DSNLTEXC DSNLTMIN DSNTIDXA DSNTIDXB DSNTIJUZ DSNTINST
  DSNTXAZP DSNVDTM  DSNWDFDI DSNWVZPS DSNWZIFA DSN6FAC
  

Fix information

• Fixed component name

  DB2 OS/390 & Z/

• Fixed component ID

  5740XYR00

Applicable component levels

• RA10 PSY UK90325

     UP12/12/29 P F212

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.