IBM Support

IT05084: TIVOLI STORAGE MANAGER SERVER REPORTS ANR9999D_0376207520 WHEN RUNNING NODE REPLICATION.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • When node replication is run between the source and target
    servers, client data will be sent but may end unexpectedly with
    the following messages seen in the target server actlog.
    
    10/06/14   17:07:29   ANR9999D_0376207520
    SmEncryptKey(smutil.c:20167) Thread<3282>: Error decrypting key.
    Magic not 0xA5A5A5A5 (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282> issued message 9999
    from: (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000ec7a43
    OutDiagToCons  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000ecaa31
    outDiagfExt  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000cffc17
    SmEncryptKey  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c64dd2
    SmParseBackInsNormE  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c6a0e8
    SmDoBackInsNormEnhanced  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000d7a8a8
    SmReplServerSession  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c4bbdf
    DoReplServer  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c4f1f0
    smExecuteSession  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000f7f067
    psSessionThread  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000f6addf
    StartThread  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x007f91c896d9d1
    *UNKNOWN*  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x007f91c545986d
    *UNKNOWN*  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D_2077889187
    SmReplServerSession(smrepl.c:1695) Thread<3282>: Replication
    protocol error, unknown verbType=0400 (SESSION: 429, PROCESS:
    12)
    10/06/14   17:07:29   ANR9999D Thread<3282> issued message 9999
    from: (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000ec7a43
    OutDiagToCons  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000ecaa31
    outDiagfExt  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000d7847a
    SmReplServerSession  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c4bbdf
    DoReplServer  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000c4f1f0
    smExecuteSession  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000f7f067
    psSessionThread  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x00000000f6addf
    StartThread  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x007f91c896d9d1
    *UNKNOWN*  (SESSION: 429, PROCESS: 12)
    10/06/14   17:07:29   ANR9999D Thread<3282>  0x007f91c545986d
    *UNKNOWN*  (SESSION: 429, PROCESS: 12)
    
    
    Customer/L2 Diagnostics:
    A trace can be run on the target server during the replication
    time using the flags addmsg and repl.
    
    The messages found in the trace file will be very similar to the
    stack in the actlog but will also show that the 9999 error was
    returned from Sm Insert Verb.
    
    17:07:30.621
    [3284][output.c][7474][PutConsoleMsg]:ANR9999D_0376207520
    SmEncryptKey(smutil.c:20167) Thread<3284>: Error decrypting key.
    Magic not 0xA5A5A5A5~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284> issued message 9999 from:~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000ec7a43 OutDiagToCons ~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000ecaa31 outDiagfExt ~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000cffc17 SmEncryptKey ~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c64dd2 SmParseBackInsNormE ~
    17:07:30.621 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c6a0e8 SmDoBackInsNormEnhanced ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000d7a8a8 SmReplServerSession ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c4bbdf DoReplServer ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c4f1f0 smExecuteSession ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000f7f067 psSessionThread ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000f6addf StartThread ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x007f91c896d9d1 *UNKNOWN* ~
    17:07:30.622 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x007f91c545986d *UNKNOWN* ~
    17:07:30.622 [3284][smrepl.c][13587][TranslateToGRC]:abort
    reason code = 1
    17:07:30.623 [3284][smrepl.c][13681][TranslateToGRC]:rc=0,
    *objRcP=9999, confirmState=NO
    17:07:30.623 [3284][smrepl.c][1620][SmReplServerSession]:Error
    9999 from Sm Insert Verb
    17:07:30.623
    [3284][smrepl.c][1098][SmReplServerSession]:Received verb
    StreamMarker (400) on session 430
    17:07:30.623
    [3284][output.c][7474][PutConsoleMsg]:ANR9999D_2077889187
    SmReplServerSession(smrepl.c:1695) Thread<3284>: Replication
    protocol error, unknown verbType=0400~
    17:07:30.623 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284> issued message 9999 from:~
    17:07:30.623 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000ec7a43 OutDiagToCons ~
    17:07:30.623 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000ecaa31 outDiagfExt ~
    17:07:30.623 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000d7847a SmReplServerSession ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c4bbdf DoReplServer ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000c4f1f0 smExecuteSession ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000f7f067 psSessionThread ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x00000000f6addf StartThread ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x007f91c896d9d1 *UNKNOWN* ~
    17:07:30.624 [3284][output.c][7474][PutConsoleMsg]:ANR9999D
    Thread<3284>  0x007f91c545986d *UNKNOWN* ~
    17:07:30.721 [3284][output.c][7474][PutConsoleMsg]:ANR0409I
    Session 430 ended for server SERVER1 (Windows).~
    
    
    Tivoli Storage Manager Versions Affected:
    7.1, 6.3
    
    
    Initial Impact: (high/med/low)
    Med
    
    Additional Keywords:
    node, replication, replicate, fail, abend, tsm
    

Local fix

  • The define server command on the target replication sever should
     be run with a password in all caps to ensure a successful
    literal match of the source server password. The source sever
    password is used in part of the server to server communications
    during node replication and is passed to the target server in
    all caps which may cause this failure to occur on the target
    server.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Tivoli Storage Manager server users.                     *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This problem is currently *
    * projected to be fixed in levels 6.3.6 and 7.1.3. Note that   *
    * this is subject to change at the discretion of IBM.          *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT05084

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    71W

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-10-23

  • Closed date

    2014-12-14

  • Last modified date

    2015-01-29

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R63A PSY

       UP

  • R63H PSY

       UP

  • R63L PSY

       UP

  • R63S PSY

       UP

  • R63W PSY

       UP

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 7.1.3

Reference #: IT05084

Modified date: 29 January 2015