IT16210: REMOVE REPLNODE MIGHT LOCK THE DATABASE WHEN THE TARGET REPLICATION SERVER HAS VERY LARGE NUMBER OF IN-FLIGHT OBJECTS

7.1.7.000-TIV-TSMSRV-SolarisSPARC
7.1.7.000-TIV-TSMSRV-Linuxs390x
7.1.7.000-TIV-TSMSRV-Linuxppc64
7.1.7.000-TIV-TSMSRV-WIN
7.1.7.000-TIV-TSMSRV-Linuxx86_64
7.1.7.000-TIV-TSMSRV-HP-UX
7.1.7.000-TIV-TSMSRV-AIX
IBM Spectrum Protect Server V7.1 Fix Pack 7 (7.1.7.000) Downloads

APAR status

• Closed as program error.

Error description

• The REMOVE REPLNODE command might result in unexpected high
  locking activity on the source server if the target server has a
  very large number of in-flight objects, typically caused by some
  incomplete replication activity.
  
  Under the APAR condition, using the server monitoring script
  from technote swg21432937, you will see the remove replnode
  command constantly processing in
  
  0x0000000100579470  RemoveOneInFlightMember(??, ??, ??, ??, ??,
  ??) + 0x170
  0x000000010056d8b4  DelReplGroupMemberships(??, ??) + 0x3f4
  0x000000010056d3ec  imDeleteReplData(??, ??, ??, ??) + 0x10c
  0x00000001005881e4  AdmRemoveReplNode(??) + 0xe64
  0x000000010070cbe8  AdmCommandLocal(??, ??, ??, ??, ??) + 0x6e8
  0x000000010070a6fc  admCommand(??, ??, ??, ??, ??) + 0xf7c
  0x0000000100bdb9b0  SmAdminCommandThread(??) + 0x30
  
  The REMOVE REPLNODE finally fails with
  
  ANR9999D Thread<14441>  0x0000000100587be8  AdmRemoveReplNode
  ANR1632E REMOVE REPLNODE: Command failed.     Replication
   state information for the specified nodes could not be
   removed.
  ANR0171I tbrsql.c(1446): Error detected on 113:  24,
   database in evaluation mode.
  ANR0103E bfcreate.c(3375): Error 4522 updating
   row in table "BF.Aggregated.Bitfiles".
  ANR9999D_0524624341 ImDeleteBitfile(imutil.c: 10241)
   Thread<14441>: Unexpected rc=4522 from bfDestroy for
   objId 6028692774
  
  another message sequence looks like
  
  ANR0171I bfaggrut.c(1644): Error detected on 179:20,
   database in evaluation mode. (SESSION: 723495)
  ANR0157W Database operation FETCH for table
   BF.Aggregate.Attributes failed with result code 4522 and
   tracking ID: 12407bce8. (SESSION: 723495)
  ANR0158W Database operation FETCH for table
   BF.Aggregate.Attributes failed with operation code 4522
   and tracking id 12407bce8. The data for column 0 is:
   (int32)0. (SESSION: 723495)
  ANR0158W Database operation FETCH for table
   BF.Aggregate.Attributes failed with operation code 4522
   and tracking id 12407bce8. The data for column 1 is:
   (int64)7184267819. (SESSION: 723495)
  ANR0106E bfaggrut.c(1647): Unexpected error 4522 fetching
   row in table "BF.Aggregate.Attributes". (SESSION: 723495)
  ANR9999D_2112824638 bfGetSubBitfileInfoEx(bfaggrut.c:1649)
   Thread<1501198>: Unable to locate attributes for bitfile
   7184267819. (SESSION: 723495)
  ANR9999D_0524624341 ImDeleteBitfile(imutil.c:10241)
   Thread<1501198>: Unexpected rc=9979 from bfDestroy for
   objId 6443411018 (SESSION: 723495)
  ANR1632E REMOVE REPLNODE: Command failed. Replication
   state information for the specified nodes could not be
   removed. (SESSION: 723495)
  ANR9999D_0413043177 GetCommandRc(admrepl.c:7134)
   Thread<1501198>: unexpected rc=9996 (SESSION: 723495)
  
  To verify that you are exposed to the APAR, as instance user,
  from a DB2 command window, submit the following commands after
  replacing MYNODENAME with the upper case nodename that you want
  to process:
  
  db2 connect to tsmdb1
  db2 set schema tsmdb1
  db2 "select nrig.tgt_groupid, nrig.mem_count from
  tsmdb1.inflight_replgroups2 nrig where nrig.tgt_nodeid in
  (select n.nodeid from tsmdb1.nodes n where
  n.nodename='MYNODENAME') and nrig.mem_count!=(select count(*)
  from tsmdb1.group_leaders imgl where
  imgl.leaderid=nrig.tgt_groupid)" > inflight.txt
  
  If you sum up nrig.mem_count you will  get the number of
  inflight objects, in the case of the APAR this were more than 73
  million objects::
  
   awk '{sum +=$2} END {print sum}' inflight.txt
  73696203
  
  
    Tivoli Storage Manager Versions Affected: all supported V6 V7
    Initial Impact: Medium
    Additional Keywords: Commit TSM IBM Spectrum Protect
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users.                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 7.1.7. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms: AIX, HP-UX, Solaris, Linux and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels