IT16364: HANDLE LEAK IN THE V7.1.6 CLIENT INSTRUMENTATION TRACING MAY RENDER LONG RUNNING CLIENT PROCESSES INOPERATIVE

IBM Spectrum Protectâ„¢ Client 8.1.0 Downloads and READMEs
IBM Spectrum Protect client 7.1.6.x interim fix downloads

APAR status

• Closed as program error.

Error description

• The symptoms are mostly identical with APAR IT11153 because
  file handle leaks are a common denominator of both defects.
  But the origins of these leaks are different.
  
  Client Instrumentation tracing into dsminstr.log is enabled
  per default since Tivoli Storage Manager version 7.1.6
  In long running client processes, such as daemons and services,
  several threads may hold file handles for writing in parallel
  and/or successively to the dsminstr.log file.
  This APAR describes the situation where those file handles are
  not closed at thread termination.
  Instead the daemon/service process cumulates
  idle file handles pointing to dsminstr.log.
  This condition can build up until the maximum number of files
  allowed by the operating system is reached (e.g. ulimit).
  Now as the daemon/service is unable to open files, it cannot
  further pursue its normal operation.
  
  Here is a list of potentially affected daemons / services:
  
  UNIX:
     dsmc, dsmc sched, dsmcad, dsmagent, ERP prole
  Windows:
     dsmc.exe dsmcsvc.exe, dsmagent.exe, dsmcad.exe, ERP prole.exe
     (Consider some Data Protection Packages use client services
     as VSS requestor.)
  
  Applications, using the API client, e.g. DB2, Oracle and others
  can be affected too.
  
  Customer/L2 diagnostics
  -----------------------
  The symptoms were first seen with
  IBM Tivoli Storage Manager for Enterprise Resource Planning:
  Data Protection for SAP HANA on Linux x86_64
  
  This message was logged by the API client from the prole
  daemon to dsierror.log:
  
  ANS1035S Options file '...' could not be found,
           or it cannot be read.
  
  The UNIX command 'lsof -p daemon_procID' did list more than 1000
  handles pointing to dsminstr.log:
  
  prole   9862 root   8w   REG  254,9 21718541 41257885
            /hana/shared/FSP/global/hdb/opt/hdbconfig/dsminstr.log
  prole   9862 root   9w   REG  254,9 21718541 41257885
            /hana/shared/FSP/global/hdb/opt/hdbconfig/dsminstr.log
  prole   9862 root  10w   REG  254,9 21718541 41257885
            /hana/shared/FSP/global/hdb/opt/hdbconfig/dsminstr.log
  prole   9862 root  11w   REG  254,9 21718541 41257885
            /hana/shared/FSP/global/hdb/opt/hdbconfig/dsminstr.log
  prole   9862 root  12w   REG  254,9 21718541 41257885
            /hana/shared/FSP/global/hdb/opt/hdbconfig/dsminstr.log
      etc.
  
  The symptoms are not limited to the prole daemon on Linux.
  As the operations of the various daemons/services are different,
  the symptoms will be different as well, and a full description
  would exceed the scope of this APAR text.
  
  Product versions affected: Tivoli Storage Manager client 7.1.6
                             on all platforms
             Initial impact: Medium
        Additional keywords: IBM Spectrum Protect TSM
                             resource defunct
                             TDP SQL Exchange
  

Local fix

• UNIX: set 'ENABLEINSTRUMENTATION No' in dsm.sys
  Windows: set 'ENABLEINSTRUMENTATION No' in dsm.opt
  Then restart the affected daemon / service.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Tivoli Storage Manager backup-archive client versions 7.1.6  *
  * running on all platforms.                                    *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is projected *
  * to be fixed in level 7.1.6.3. Note that this is subject to   *
  * change at the discretion of IBM.                             *
  ****************************************************************
  

Problem conclusion

• After the fix instrumentation tracing works without file handle
  leak.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• DSM
  

Fix information

• Fixed component name

  TSM CLIENT

• Fixed component ID

  5698ISMCL

Applicable component levels

• R71A PSY

     UP

• R71L PSY

     UP

• R71H PSY

     UP

• R71M PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP