IBM Support

IT16852: REPLICATE NODE CAN FAIL WITH ANR9999D_4160640106 DUE TO ORPHAN TOC, IF CHANGING REPLMODE FOR A NODE FROM RECEIVE TO SEND

Direct links to fixes

8.1.1.100-IBM-SPOC-WindowsX64
8.1.1.100-IBM-SPOC-Linuxx86_64
8.1.1.100-IBM-SPOC-Linuxs390x
8.1.1.100-IBM-SPOC-AIX
8.1.1.000-IBM-SPSRV-WindowsX64
8.1.1.000-IBM-SPSRV-Linuxs390x
8.1.1.000-IBM-SPSRV-AIX
8.1.1.000-IBM-SPSRV-Linuxx86_64
IBM Spectrum Protect Server V8.1 Fix Pack 1 (V8.1.1) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 8 (7.1.8.000) Downloads

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • After replication role is switched by changing the REPLMODE for
a node from RECEIVE to SEND using:

REMOVE REPLNODE
UPDATE NODE XXXX REPLSTATE=ENABLED

REPLICATE NODE on source replication server can fail with
ANR9999D_4160640106 if there exists an orphan TOC with no image
on target replication server.


ACTLOG will show for example:

ANR9999D_4160640106 AddToReplicationList(nrmain.c:20152)
Thread<xxxx>: Error 1114 getting attr obj owner for xxxxxxxxxx
<== affected object
ANR9999D Thread<xxxx>  0x00000000f23c34 OutDiagToCons
ANR9999D Thread<xxxx>  0x00000000f26a9c outDiagfExt
ANR9999D Thread<xxxx>  0x0000000094dbdb AddToReplicationList
ANR9999D Thread<xxxx>  0x00000000959b41 IterateObjectsByObjid
ANR9999D Thread<xxxx>  0x000000009637bf NrReplicateFilespace
ANR9999D Thread<xxxx>  0x00000000f54525 PcConsumerThread
ANR9999D Thread<xxxx>  0x00000000fd0824 StartThread
ANR9999D Thread<xxxx>  0x000039460079d1 *UNKNOWN*
ANR9999D Thread<xxxx>  0x000039458e88fd *UNKNOWN*
ANR1936E Replication of BACKUP data for file space xxxx (fsId x)
on node xxxxx failed with reason: Reason code 1114.

Tivoli Storage Server trace on Source Server with trace flags
REPL REPLSESS GROUP shows:

[imgroup.c][1669][imGetAttrObjOwner]:Owner object not found for
object xxxxxxxx with objType 17 in node xxx, fs x
[imgroup.c][1683][imGetAttrObjOwner]:returning main objid 0 for
attr objid xxxxxxxx, rc: 1114


This APAR will be used to address the cause of the orphan TOC so
that will be fixed as well in addition to add toleration code to
allow replication to skip the orphan TOC and continue.


Tivoli Storage Manager Server Versions Affected:
Tivoli Storage Manager Version 7.1.1 and above on all supported
platforms

Initial Impact:
Medium


Additional Keywords:
TSM, replicate node, replication, toc, IBM Spectrum Protect

Local fix

  • To avoid the problem:
Always run EXPIRE INVENTORY before running REPLICATE NODE.
Particularly before running REMOVE REPLNODE to switch roles.

To clean up from the problem after it has occurred:

1. Delete the problem filespace on target replication server
2. Run EXPIRE INVENTORY on source replication server, to clean
up
3. Run REPLICATE NODE on source replication server, to backup
data on target


Contact IBM support for an option to delete the object that
results in the ANR9999D_4160640106 message

Problem summary

  • ****************************************************************
* USERS AFFECTED:                                              *
* All Tivoli Storage Manager server users.                     *
****************************************************************
* PROBLEM DESCRIPTION:                                         *
* See error description                                        *
****************************************************************
* RECOMMENDATION:                                              *
* Apply fixing level when available. This problem is currently *
* projected to be fixed in levels 7.1.8 and 8.1.1. Note that   *
* this is subject to change at the discretion of IBM.          *
****************************************************************

Problem conclusion

  • This problem was fixed.
Affected platforms: AIX, HP-UX, Solaris, Linux, and Windows.

Temporary fix

  • 



Comments


    

  • 



APAR Information




    

  • APAR number
    IT16852
    • 

  • Reported component name
    TSM SERVER
    • 

  • Reported component ID
    5698ISMSV
    • 

  • Reported release
    71L
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt / Xsystem
    • 

  • Submitted date
    2016-09-05
    • 

  • Closed date
    2016-12-15
    • 

  • Last modified date
    2016-12-15
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    TSM SERVER
    • 

  • Fixed component ID
    5698ISMSV
    • 



Applicable component levels


    








      
              
                            

              

              [{"Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1.3"}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            13 February 2021
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback