IBM Support

Server fails to start with error SQL0294N after upgrade

Troubleshooting


Problem

A Tivoli Storage Manager Server is upgraded from V7.1.x to V7.1.3. After the upgrade completes, the server faills to start with the following error : [] SQL0294N The container is already in use. SQLSTATE=42730[]

Symptom

Starting the server in the foreground shows the following messages :

ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.
ANR0152I Database manager successfully started.
ANR1628I The database manager is using port 51500 for server connections.
ANR9999D_3831306406 ReportSQLDiagInfo(dbieval.c:1572) Thread<1>:
Missing sqlState=42730, sqlCode=-294 from table. Returning rc = 9994.
ANR9999D Thread<1> issued message 9999 from:
ANR9999D Thread<1> 0x00000000f23c34 OutDiagToCons
ANR9999D Thread<1> 0x00000000f26a9c outDiagfExt
ANR9999D Thread<1> 0x00000000b4ce59 DbiEvalSQLOutcomeX
ANR9999D Thread<1> 0x00000000b79786 DbiExecuteOnConn
ANR9999D Thread<1> 0x00000000b79994 dbiExecuteStmt
ANR9999D Thread<1> 0x00000000bbc483 CreateDBItems
ANR9999D Thread<1> 0x00000000b47f0d dbiInit
ANR9999D Thread<1> 0x000000004baebd StartServer
ANR9999D Thread<1> 0x000000004bd5d9 admStartServer
ANR9999D Thread<1> 0x000000004a3dd0 main
ANR9999D Thread<1> 0x007ffff3e75d5d *UNKNOWN*
ANR9999D Thread<1> 0x000000004a27f9 *UNKNOWN*
ANR0162W Supplemental database diagnostic information: -1:42730:-294
([IBM][CLI Driver][DB2/LINUXX8664] SQL0294N The container is already in use. SQLSTATE=42730

Cause

Orphaned containers left from a previous upgrade failure.

Environment

Tivoli Storage Manager Server

Diagnosing The Problem

Review the db2diag.x.log of the instance. For example, the following error is logged after the server fails to start :

2015-11-23-09.13.51.091861-300 I2676981E716 LEVEL: Error
PID : 43623 TID : 140736922117888 PROC : db2sysc 0
INSTANCE: tsminst01 NODE : 000 DB : TSMDB1
APPHDL : 0-226 APPID: *LOCAL.tsmin01.151123141457
AUTHID : TSMINT01 HOSTNAME: server1
EDUID : 27 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbDMSAddContainerRequest,
probe:871
MESSAGE : ZRC=0x80020039=-2147352519=SQLB_CONTAINER_IN_USE
"Container is already being used"
DATA #1 : <preformatted>
Error acquiring container 5
(/DB1/tsminst01/NODE0000/TSMDB1/T0000020/C0000005.LRG) for tbsp 20.
Rc = 80020039
The above message shows an error for tablespace 20 (tbsp 20). The same error happens for tablespace 20 for all containers. Verify that tablespace 20 does nott exist from the "tablespace list". Logon as the instance owner and run the following commands : [
db2 connect to tsmdb1
db2 list tablespaces

The d2 list command would only show tablespaces up to tablespace 19. Tablespace 20 does not exist.

Review the list of containers of a working and non-working container. For example :

-container for tablespace 20:

ls -al /DB*/tsminst01/NODE0000/TSMDB1/T0000020/
-rw-------1 tsminst01 tsmsrvrs 2097152 Nov 18 14:44 C0000008.LRG


-container for a working tablespace :

ls -al /DB1/tsminst01/NODE0000/TSMDB1/T0000002/:
-rw-------1 tsminst01 tsmsrvrs 5593628672 Nov 23 08:41 C0000008.LRG


In the above outputs, it shows that tablespace 20 was created at a different date than the other tablespace.

Notes :
- Tivoli Storage Manager V7.1.1 only has tablespaces up to 19. The tablespace 20's container should nott exist for a V7.1.1 server.
- Tablespace T0000020 was created from a failed upgrade to V7.1.3.0 on "Nov 18 14:44". The server was rolled back to V7.1.1 after the failure. The restore was done without first removing the database (dsmserv removedb). As a result, tablespace T0000020 was left orphaned.
- Because T0000020 was created from a different db upgrade, the new upgrade fails to create this because it already exists.

Resolving The Problem

Ensure there are no orphaned container prior to do the upgrade. Follow the below steps.

  1. Keep a copy of the volhist and dsmserv.opt files.
  2. Uninstall Tivoli Storage Manager V7.1.3
  3. Install Tivoli Storage Manager V7.1.1
  4. Make sure the T0000020 tablespace does not exist or just do a "dsmserv removedb tsmdb1" to cleanup all the containers
  5. Restore the database using the V7.1.1 database backup
  6. After the restore db is complete, start the Tivoli Storage Manager server to ensure the restore database is good.
  7. Halt Tivoli Storage Manager server and redo the upgrade to V7.1.3.

[{"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
17 June 2018

UID

swg21971754