IBM Support

IT16852: REPLICATE NODE CAN FAIL WITH ANR9999D_4160640106 DUE TO ORPHAN TOC, IF CHANGING REPLMODE FOR A NODE FROM RECEIVE TO SEND

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • After replication role is switched by changing the REPLMODE for
    a node from RECEIVE to SEND using:
    
    REMOVE REPLNODE
    UPDATE NODE XXXX REPLSTATE=ENABLED
    
    REPLICATE NODE on source replication server can fail with
    ANR9999D_4160640106 if there exists an orphan TOC with no image
    on target replication server.
    
    
    ACTLOG will show for example:
    
    ANR9999D_4160640106 AddToReplicationList(nrmain.c:20152)
    Thread<xxxx>: Error 1114 getting attr obj owner for xxxxxxxxxx
    <== affected object
    ANR9999D Thread<xxxx>  0x00000000f23c34 OutDiagToCons
    ANR9999D Thread<xxxx>  0x00000000f26a9c outDiagfExt
    ANR9999D Thread<xxxx>  0x0000000094dbdb AddToReplicationList
    ANR9999D Thread<xxxx>  0x00000000959b41 IterateObjectsByObjid
    ANR9999D Thread<xxxx>  0x000000009637bf NrReplicateFilespace
    ANR9999D Thread<xxxx>  0x00000000f54525 PcConsumerThread
    ANR9999D Thread<xxxx>  0x00000000fd0824 StartThread
    ANR9999D Thread<xxxx>  0x000039460079d1 *UNKNOWN*
    ANR9999D Thread<xxxx>  0x000039458e88fd *UNKNOWN*
    ANR1936E Replication of BACKUP data for file space xxxx (fsId x)
    on node xxxxx failed with reason: Reason code 1114.
    
    Tivoli Storage Server trace on Source Server with trace flags
    REPL REPLSESS GROUP shows:
    
    [imgroup.c][1669][imGetAttrObjOwner]:Owner object not found for
    object xxxxxxxx with objType 17 in node xxx, fs x
    [imgroup.c][1683][imGetAttrObjOwner]:returning main objid 0 for
    attr objid xxxxxxxx, rc: 1114
    
    
    This APAR will be used to address the cause of the orphan TOC so
    that will be fixed as well in addition to add toleration code to
    allow replication to skip the orphan TOC and continue.
    
    
    Tivoli Storage Manager Server Versions Affected:
    Tivoli Storage Manager Version 7.1.1 and above on all supported
    platforms
    
    Initial Impact:
    Medium
    
    
    Additional Keywords:
    TSM, replicate node, replication, toc, IBM Spectrum Protect
    

Local fix

  • To avoid the problem:
    Always run EXPIRE INVENTORY before running REPLICATE NODE.
    Particularly before running REMOVE REPLNODE to switch roles.
    
    To clean up from the problem after it has occurred:
    
    1. Delete the problem filespace on target replication server
    2. Run EXPIRE INVENTORY on source replication server, to clean
    up
    3. Run REPLICATE NODE on source replication server, to backup
    data on target
    
    
    Contact IBM support for an option to delete the object that
    results in the ANR9999D_4160640106 message
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Tivoli Storage Manager server users.                     *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This problem is currently *
    * projected to be fixed in levels 7.1.8 and 8.1.1. Note that   *
    * this is subject to change at the discretion of IBM.          *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms: AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT16852

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    71L

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-09-05

  • Closed date

    2016-12-15

  • Last modified date

    2016-12-15

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP

  • R81A PSY

       UP

  • R81L PSY

       UP

  • R81W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 7.1.3

Reference #: IT16852

Modified date: 15 December 2016