IC66284: TIVOLI STORAGE MANAGER CRASHES DUE TO DATA DEDUPLICATION DELETION THREAD EXCEEDING SYSTEM RESOURCES

APAR status

• Closed as program error.

Error description

• TSM server crashes with Data Deduplication storage pools
  enabled.
  
  Customer/L2 Diagnostics: Before the next crash, start show
  command collection of: txnt, threads, locks, and memory.  Also
  get query sess f=d and query proc.  Trace with flags: dedup
  dedup1 dedup2
  
  The show txnt reports several: "Creator=bfdedup.c"
  
  Trace is filled with dedup actions. Ex:
  
  19:00:46.186 [44][bfdedup.c][8296][bfDerefQueueThread]:
  Inserting chunk 33592437 for node 29 into queued chunk table
  from the deref'd chunks table with flag 1.
  19:00:46.186 [44][bfdedup.c][8296][bfDerefQueueThread]:
  Inserting chunktfile (chunk) 11333536 has already been deleted
  
  Core stack in this case was not helpful.
  
  Actlog may or may not have the following messages:
  ANR0163E dbieval.c(879): Database insufficient memory detected
  on 30:2.
  ANR0162W Supplemental database diagnostic information:
  -1:57011:-912 ([IBM][CLI Driver][DB2/AIX64]
  SQL0912N  The maximum number of lock requests has been reached
  for the database.  SQLSTATE=57011).
  ANR0106E admevent.c(1222): Unexpected error 3 fetching row in
  table "Client.Eventrules".
  ANR9999D_1910305017 smExecuteSession(smexec.c:2379) Thread<352>:
  Session xxx with client name
  (WinNT) rejected - error creating the central logging vector.
  ANR9999D Thread<352>  0x000000010035b8b0 smExecuteSession
  ANR9999D Thread<352>  0x0000000100098028 psSessionThread
  ANR9999D Thread<352>  0x0000000100009b28 StartThread
  ANR0163E dbieval.c(879): Database insufficient memory detected
  on 29:1.
  
  Initial Impact: Medium
  
  Additional Keywords:   Dedup crash dump mem
  zz61
  

Local fix

• Set the following option in the TSM server option file and
  restart TSM.  This option turns off the dedup deletion thread:
  DEDUPDELETIONTHREADS 0
  Get to fixing level
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in level 6.1.4.                              *
  *                 Note that this is subject to change at the   *
  *                 discretion of IBM.                           *
  ****************************************************************
  See ERROR DESCRIPTION.
  

Problem conclusion

• The described problem has been resolved.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP