IBM Support

Cannot start the Server - getting error ANR9999D_3146658462 rc 9999

Troubleshooting


Problem

After a system crash, the Tivoli Storage Manager Server could fail to start, reporting error ANR9999D_3146658462 with rc 9999

Symptom

Starting up the Tivoli Storage Manager Server in the foreground will log the following errors:


ANR7800I DSMSERV generated at 07:37:18 on Nov 23 2014.
.
Tivoli Storage Manager for Windows
Version 7, Release 1, Level 1.100

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2014.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM
Corporation.

ANR0900I Processing options file C:\tsm\tsminst1\dsmserv.opt.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0152I Database manager successfully started.
ANR0162W Supplemental database diagnostic information:
-1:HY000:-1042
([IBM][CLI Driver][DB2/NT64]
SQL1042C An unexpected system error occurred. SQLSTATE=58004 ).
ANR9999D_3146658462 tbSQLExecuteToStrings(tbnsql.c:1263) Thread<0>:
Unexpected rc 9999 from tbSQLExecuteFetch.
ANR9999D Thread<0> issued message 9999 from:
ANR9999D Thread<0> 00007FFBD9B07035 OutDiagToCons()+155
ANR9999D Thread<0> 00007FFBD9B00F6C outDiagfExt()+10c
ANR9999D Thread<0> 00007FFBD9883A1D tbSQLExecuteToStrings()+1ad
ANR9999D Thread<0> 00007FFBD98AE7AC GetDB2LevelStr()+16c
ANR9999D Thread<0> 00007FFBD98A7C77 RdbCrossCheckDb2Level()+97
ANR9999D Thread<0> 00007FFBD98065B8 dbiInit()+4b8
ANR9999D Thread<0> 00007FFBD91F090C StartServer()+2bc
ANR9999D Thread<0> 00007FFBD91EC1DF admStartServer()+ff
ANR9999D Thread<0> 00007FFBD91D23FF adsmMain()+134f
ANR9999D Thread<0> 00007FF63FCE110D main()+8d dsmserv.c:241
ANR9999D Thread<0> 00007FF63FCE271B __tmainCRTStartup()+10f
crtexe.c:536
ANR9999D Thread<0> 00007FFBF88C13D2 BaseThreadInitThunk()+22
ANR9999D Thread<0> 00007FFBF9C3E954 RtlUserThreadStart()+34
ANR9999D_3325034490 GetDB2LevelStr(rdbdb.c:8412) Thread<0>:
Rc 9999 from tbSQLExecuteToStrings for SELECT SERVICE_LEVEL
FROM SYSIBMADM.ENV_INST_INFO.
ANR9999D Thread<0> issued message 9999 from:
ANR9999D Thread<0> 00007FFBD9B07035 OutDiagToCons()+155
ANR9999D Thread<0> 00007FFBD9B00F6C outDiagfExt()+10c
ANR9999D Thread<0> 00007FFBD98AE7E7 GetDB2LevelStr()+1a7
ANR9999D Thread<0> 00007FFBD98A7C77 RdbCrossCheckDb2Level()+97
ANR9999D Thread<0> 00007FFBD98065B8 dbiInit()+4b8
ANR9999D Thread<0> 00007FFBD91F090C StartServer()+2bc
ANR9999D Thread<0> 00007FFBD91EC1DF admStartServer()+ff
ANR9999D Thread<0> 00007FFBD91D23FF adsmMain()+134f
ANR9999D Thread<0> 00007FF63FCE110D main()+8d dsmserv.c:241
ANR9999D Thread<0> 00007FF63FCE271B __tmainCRTStartup()+10f
crtexe.c:536
ANR9999D Thread<0> 00007FFBF88C13D2 BaseThreadInitThunk()+22
ANR9999D Thread<0> 00007FFBF9C3E954 RtlUserThreadStart()+34
ANR9999D_2095217645 RdbCrossCheckDb2Level(rdbdb.c:5327) Thread<0>:
Error 9999 retrieving installed DB2 level.
See previous message for problem determination.
ANR9999D Thread<0> issued message 9999 from:
ANR9999D Thread<0> 00007FFBD9B07035 OutDiagToCons()+155
ANR9999D Thread<0> 00007FFBD9B00F6C outDiagfExt()+10c
ANR9999D Thread<0> 00007FFBD98A7DD3 RdbCrossCheckDb2Level()+1f3
ANR9999D Thread<0> 00007FFBD98065B8 dbiInit()+4b8
ANR9999D Thread<0> 00007FFBD91F090C StartServer()+2bc
ANR9999D Thread<0> 00007FFBD91EC1DF admStartServer()+ff
ANR9999D Thread<0> 00007FFBD91D23FF adsmMain()+134f
ANR9999D Thread<0> 00007FF63FCE110D main()+8d dsmserv.c:241
ANR9999D Thread<0> 00007FF63FCE271B __tmainCRTStartup()+10f
crtexe.c:536
ANR9999D Thread<0> 00007FFBF88C13D2 BaseThreadInitThunk()+22
ANR9999D Thread<0> 00007FFBF9C3E954 RtlUserThreadStart()+34

Cause

IBM DB2 is failing to run the following select at startup time:

SELECT SERVICE_LEVEL FROM SYSIBMADM.ENVINST_INFO

causing the Tivoli Storage Manager startup failure as well.

The same select ran manually from a DB2 command prompt will fail with the following error:

SQL1042C An unexpected system error occurred. SQLSTATE=58004

Despite this, db2start/db2stop works fine.

Diagnosing The Problem

Look at the db2diagx.log file and search for the following warning:


2015-06-01-14.33.41.603000+120 I2203907F578 LEVEL: Warning
PID : 4020 TID : 3204 PROC : db2syscs.exe
INSTANCE: SERVER1 NODE : 000
HOSTNAME: SERVER-01
EDUID : 3204 EDUNAME: db2sysc
FUNCTION: DB2 UDB, routine_infrastructure, sqlerInitFmpTable, probe:25
MESSAGE : Warning! No segment was created for running fenced routines.
DB2_FMP_COMM_HEAPSZ is set to 0. This means no fmps (including
Health Monitor) and automatic maintenance features of DB2 will be started.
Since DB2_FMP_COMM_HEAPSZ=0, the db2sysc.exe was unable to setup
communications to db2fmp (runs non-SQL stored procedure) causing the failure of the
"SELECT SERVICE_LEVEL FROM SYSIBMADM.ENVINST_INFO"

Resolving The Problem

Run the following steps from a DB2 command prompt being logged on as the DB2 instance owner:

1)DB2_FMP_COMM_HEAPSZ=
2) db2stop

(leaving the DB2_FMP_COMM_HEAPSZ value as blank defaults this to 20 MB in size)

Startup the Tivoli Storage Manager Server again after amending the DB2_FMP_COMM_HEAPSZ value.

[{"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF033","label":"Windows"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
17 June 2018

UID

swg21959725