IBM Support

IT01366: NODE REPLICATION DEADLOCKS WITH EXPIRE INVENTORY THEN EXPIRATION HANGS

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Expire Inventory processed all nodes successfully but looks
    hung.  The Expire Inventory process will be canceled by a
    deadlock after many hours.   During the same time that Expire
    Inventory looks hung, many deadlock messages are reported for
    active Node Replication processes.
    
    Tivoli Storage Manager Versions Affected: All 6.2, 6.3 and 7.1
    Servers
    
    Customer/L2 Diagnostics:
    Example of Actlog messages:
    05:05:22 ANR0984I Process 15 for EXPIRE INVENTORY started in the
    BACKGROUND at 05:05:22. (SESSION: 22095, PROCESS: 15)
    
    06:00:24 ANR0984I Process 16 for Replicate Node started in the
    BACKGROUND at 06:00:24. (SESSION: 25296, PROCESS: 16)
    06:51:33 ANR0166I Inventory file expiration finished processing
    for node x, filespace y, copygroup z and object type GROUP BASE
    with processing statistics: examined 78012, deleted 78012,
    retrying 0, and failed 0. (SESSION: 22095, PROCESS: 15)
    
    07:03:48 ANR0159E nrmain.c(6185): Database deadlock detected on
    123:2. (SESSION: 25296, PROCESS: 16)
    07:03:48 ANR0162W Supplemental database diagnostic information:
    -1:40001:-911 ([IBM][CLI Driver][DB2/AIX64] SQL0911N  The
    current transaction has been rolled back because of a deadlock
    or timeout.  Reason code '2'.  SQLSTATE=40001). (SESSION: 25296,
    PROCESS: 16)
    
    08:11:30 ANR0408I Session 30796 started for server x (AIX)
    (Tcp/Ip)
    for replication.  (SESSION: 29886, PROCESS: 20)
    
    08:12:14 ANR0159E nrmain.c(6185): Database deadlock detected on
    232:2. (SESSION: 29886, PROCESS: 20)
    08:12:14 ANR0162W Supplemental database diagnostic information:
    -1:40001:-911 ([IBM][CLI Driver][DB2/AIX64] SQL0911N  The
    current transaction has been rolled back because of a deadlock
    or timeout.   Reason code '2'.  SQLSTATE=40001). (SESSION:
    29886, PROCESS: 20)
    
    14:55:43 ANR0159E tbrsql.c(1485): Database deadlock detected on
    64:33. (SESSION: 22095, PROCESS: 15)
    14:55:43 ANR0162W Supplemental database diagnostic information:
    -1:40001:-911 ([IBM][CLI Driver][DB2/AIX64] SQL0911N  The
    current transaction has been rolled back because of a deadlock
    or timeout.  Reason code '2'. SQLSTATE=40001). (SESSION: 22095,
    PROCESS: 15)
    
    Show Threads will has expiration working in: imReplHourlyMonitor
    Example:
    Thread 32525, Parent 32520: ExpirationProcessThread, Storage
    1243996,
    AllocCnt 231006 HighWaterAmt
      1602363
      tid=780d, ptid=6f08, det=1, zomb=0, join=0, result=0, sess=0
       Stack trace:
         0x0900000000260e10 semop
         0x0900000000b9f58c sqloSSemP
         0x0900000000b9ef64 .sqlccrecv.fdpr.clone.739
         0xffffffff89000017 *UNKNOWN*
         0x0900000000b9e82c sqljcReceive__FP10sqljCmnMgr
         0x0900000000bac440
    sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
         0x0900000000e43520
    CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
         0x0900000000eb271c
    SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
         0x0900000000ec4c64 SQLExecute
         0x00000001001a2490 tbRegExecEx
         0x00000001002518d8 NrPrune
         0x00000001006c8650 imReplHourlyMonitor      <-----
         0x0000000100a8348c ExpirationProcessThread
         0x000000010000c264 StartThread
    
    Initial Impact: Medium
    
    Additional Keywords:  repl exp inv
    

Local fix

  • Until fix is available, see work around documented at:
    http://www.ibm.com/support/docview.wss?uid=swg21661695
    Once the fix is available and there are nodes that will no
    longer be replicated any more, the  REPLRETENTION option can
    then be set back to the default of 30 or to the number of days
    that match how often the latest scheduled node replication is
    done.  After the number of days indicated by the REPLRETENTION
    value has passed, nodes that are not being replicated will have
    their information cleared from the Replication History.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Tivoli Storage Manager Users                             *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This                      *
    * problem is currently projected to be fixed                   *
    * in levels 6.3.5 and 7.1.1. Note that this is                 *
    * subject to change at the discretion of IBM.                  *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT01366

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    63A

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2014-04-28

  • Closed date

    2014-05-16

  • Last modified date

    2014-05-16

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R63A PSY

       UP

  • R63H PSY

       UP

  • R63L PSY

       UP

  • R63S PSY

       UP

  • R63W PSY

       UP

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 63A

Reference #: IT01366

Modified date: 16 May 2014