IC67454: DATA MOVEMENT WITHIN A TIVOLI STORAGE MANAGER SERVER MAY LOOP

IBM Tivoli Storage Manager V6.1 Fix Pack 4 (6.1.4) Server Downloads

APAR status

• Closed as program error.

Error description

• Under certain - rather infrequent - conditions, data movement
  within the Tivoli Storage Manager server may get into a loop.
  
  Following data movement operations are affected:
  move data
  move nodedata
  migration
  reclamation
  
  The above data movement operations from any kind of storage pool
  could be affected by this problem, but it is more likely to
  happen when data is moved from a deduplicated storage pool into
  a non-deduplicated storage pool.
  This loop causes one file to get written over and over again to
  the output volume until this volume, or in worst case, the
  complete storage pool, becomes full.
  
  
  Customer/L2 Diagnostics:
  The ACTLOG shows:
  03/27/2010 12:56:34 ANR0984I Process YYY for MOVE DATA started
  in the FOREGROUND at 12:56:34 PM. (SESSION: XXX, PROCESS: YYY)
  03/27/2010 12:56:34 ANR1140I Move data process started for
  volume /tsm/data/00000001.BFS (process ID YYY). (SESSION: XXX,
  PROCESS: YYY)
  [...]
  03/28/2010 00:22:47 ANR8337I LTO volume 000001L3 mounted in
  drive DRIVE1 (/dev/tsmscsi/mt1). (SESSION: XXX, PROCESS: YYY)
  03/28/2010 00:22:51 ANR1340I Scratch volume 000001L3 is now
  defined in storage pool TAPEPOOL. (SESSION: XXX, PROCESS: YYY)
  [...]
  03/28/2010 12:53:01 ANR8341I End-of-volume reached for LTO
  volume 000001L3. (SESSION: XXX, PROCESS: YYY)
  03/28/2010 12:53:13 ANR8336I Verifying label of LTO volume
  000001L3 in drive DRIVE1 (/dev/tsmscsi/mt1). (SESSION: XXX,
  PROCESS: YYY)
  03/28/2010 12:53:49 ANR8337I LTO volume 000001L3 mounted in
  drive MSLDR0 (/dev/tsmscsi/mt0). (SESSION: XXX, PROCESS: YYY)
  03/28/2010 12:53:52 ANR1340I Scratch volume 000001L3 is now
  defined in storage pool TAPEPOOL. (SESSION: XXX, PROCESS: YYY)
  03/28/2010 12:54:01 ANR8468I LTO volume 000001L3 dismounted from
  drive DRIVE1 (/dev/tsmscsi/mt1) in library MSL6000. (SESSION:
  XXX, PROCESS: YYY)
  
  A trace with traceflags AF BF DEDUP DEDUP1 shows that we are
  doing over and over this sequence (here for object 22069911
  randomly chosen out of the trace):
  13:27:09.566 [14160][bfcreate.c][2891][BfReconstruct]:Dedup info
  for object 22069911: deleted: F, orig size: 0
  13:27:09.632 [14160][bfcreate.c][3188][BfReconstruct]:Object
  22069911 will be copied
  13:27:09.689 [14160][bfcreate.c][3411][BfReconstruct]:Checking
  object 22069911 in bitfile 30735997 at offset 24898692 of length
  333286 for adjacency
  13:27:09.689 [14160][bfcreate.c][3424][BfReconstruct]:Adding
  bfid 22069911 to region 1, size now 25231978
  13:27:10.623 [14160][bfcreate.c][4017][BfReconstruct]:New
  offset(24898692), length(333286) for logical file 22069911.
  13:27:17.090 [14160][bfcreate.c][2891][BfReconstruct]:Dedup info
  for object 22069911: deleted: F, orig size: 0
  13:27:17.155 [14160][bfcreate.c][3188][BfReconstruct]:Object
  22069911 will be copied
  13:27:17.212 [14160][bfcreate.c][3411][BfReconstruct]:Checking
  object 22069911 in bitfile 30735997 at offset 24898692 of length
  333286 for adjacency
  13:27:17.212 [14160][bfcreate.c][3424][BfReconstruct]:Adding
  bfid 22069911 to region 1, size now 25231978
  13:27:18.137 [14160][bfcreate.c][4017][BfReconstruct]:New
  offset(24898692), length(333286) for logical file 22069911.
  [...]
  13:37:37.001 [14160][bfcreate.c][2891][BfReconstruct]:Dedup info
  for object 22069911: deleted: F, orig size: 0
  13:37:37.066 [14160][bfcreate.c][3188][BfReconstruct]:Object
  22069911 will be copied
  13:37:37.123 [14160][bfcreate.c][3411][BfReconstruct]:Checking
  object 22069911 in bitfile 30735997 at offset 24898692 of length
  333286 for adjacency
  13:37:37.123 [14160][bfcreate.c][3424][BfReconstruct]:Adding
  bfid 22069911 to region 1, size now 25231978
  13:37:38.052 [14160][bfcreate.c][4017][BfReconstruct]:New
  offset(24898692), length(333286) for logical file 22069911.
  
  
  A QUERY CONT on the output volume being written to by the MOVE
  DATA process (here 000001L3) shows that the volume is completely
  full with one and the same file:
  NODENAME Bkup \\NODENAME\c$ 1 \DATA\FILE.TXT
  NODENAME Bkup \\NODENAME\c$ 1 \DATA\FILE.TXT
  NODENAME Bkup \\NODENAME\c$ 1 \DATA\FILE.TXT
  NODENAME Bkup \\NODENAME\c$ 1 \DATA\FILE.TXT
  NODENAME Bkup \\NODENAME\c$ 1 \DATA\FILE.TXT
  [....]
  
  
  Tivoli Storage Manager Versions Affected:
  All Tivoli Storage Managers Servers 6.1.3 and above
  
  
  Initial Impact:
  High
  
  
  Additional Keywords:
  zz61 zz62 TSM deduplicate deduplication dedup migration move
  data migrate reclaim hang looping
  
  | MDVREGR 6.1.3.0 |
  | MDVREGR 6.1.3.1 |
  | MDVREGR 6.1.3.2 |
  | MDVREGR 6.2.0.0 |
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in levels 6.1.3.4, 6.1.4 and 6.2.1.          *
  *                 Note that this is subject to change at the   *
  *                 discretion of IBM.                           *
  ****************************************************************
  See ERROR DESCRIPTION.
  

Problem conclusion

• The described problem has been resolved.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP