IBM Support

Database space exhausted following a restart of the server after an upgrade

Troubleshooting


Problem

Excessive database growth is witnessed following a server upgrade. The server instance may crash if all available database space is consumed.

Symptom

The activity log may show the following error message prior to the server outage:

ANR0175E tbtbl.c(9007): Space exhausted for one or more of the following: DB, LOG, or instance directory.

Cause

If the server upgrade modified any of the table indexes then it will be necessary for the indexes to be recreated. The recreation of the index occurs during the initial restart of the server after the upgrade has completed. The amount of database space required to recreate the index will vary depending on the size of the table associated with the index. If the index to be recreated is associated with a large database table, then a significant amount of database space may be required to create the index.

Diagnosing The Problem

The db2diag.log will show that the initial attempt to create the table index fails due to an out-of-space condition:

    2016-10-21-07.26.56.343318-240 I47826428A652 LEVEL: Severe (Origin)
    PID : 66388092 TID : 55258 PROC : db2sysc 0
    INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
    APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
    AUTHID : TSMSC01
    EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
    FUNCTION: DB2 UDB, buffer pool services, sqlbGrowNewStripeSet, probe:20
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
    DIA8312C Disk was full.
    DATA #1 : String, 108 bytes
    An error caused the allocation of a new stripe set of containers to an automatic storage tablespace to fail.

    2016-10-21-07.26.56.345020-240 E47827081A1122 LEVEL: Error
    PID : 66388092 TID : 55258 PROC : db2sysc 0
    INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
    APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
    AUTHID : TSMSC01
    EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
    FUNCTION: DB2 UDB, buffer pool services, sqlbAllocateExtent, probe:835
    MESSAGE : ADM6044E The DMS table space "LARGEIDXSPACE1" (ID "6") is full. If this is an autoresize or automatic storage DMS tablespace, the maximum table space size may have been reached or the existing containers or storage paths cannot grow any more. Additional space can be added to the table space by either adding new containers or extending existing ones using the ALTER TABLESPACE SQL statement. If this is an autoresize or automatic storage DMS table space, additional space can be added by adding containers to an autoresize table space or by adding new storage paths to an automatic storage database.

    2016-10-21-07.30.33.761203-240 I47840265A559 LEVEL: Severe
    PID : 66388092 TID : 55258 PROC : db2sysc 0
    INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
    APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
    AUTHID : TSMSC01
    EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
    FUNCTION: DB2 UDB, catalog services, sqlrlCreateIndexPart, probe:13803
    CALLED : DB2 UDB, data management, sqldIndexCreate
    RETCODE : ZRC=0x85020021=-2063466463=SQLB_END_OF_CONTAINER
    "DMS Container space full"

During subsequent restarts the db2diag.log file will show errors indicating that a temporary table space is full:

    2016-10-22-01.39.15.246426-240 E48053113A778 LEVEL: Error
    PID : 11075824 TID : 17131 PROC : db2sysc 0
    INSTANCE: tsmsc01 NODE : 000
    EDUID : 17131 EDUNAME: db2pclnr (TSMDB1) 0
    FUNCTION: DB2 UDB, buffer pool services, sqlbClnrAsyncWriteCompletion, probe:0
    MESSAGE : ADM6017E The table space "LGTMPTSP" (ID "7") is full. Detected on container "/home/tsmsc01/DB/db6/tsmsc01/NODE0000/TSMDB1/T0000007/C0000007.TMP" (ID "7"). The underlying file system is full or the maximum allowed space usage for the file system has been reached. It is also possible that there are user limits in place with respect to maximum file size and these limits have been reached.

The output from the SHOW THREADS command will also show an active CreateIndexesThread thread:

    Thread 208, Parent 1: CreateIndexesThread, Storage 11900, AllocCnt 14 HighWaterAmt 11900
    tid=1fd0, ptid=1, det=1, zomb=0, join=0, result=0, sess=0
    Stack trace:
    0x09000000001f1b10 semop
    0x0900000001462b0c sqloSSemP
    0x0900000001462f00 .sqlccipcrecv.fdpr.clone.784__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
    0x0900000001463904 .sqlccrecv.fdpr.clone.16
    0x0900000001463600 sqljcReceive__FP10sqljCmnMgr
    0x090000000146d040 sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
    0x09000000017c3754 CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
    0x09000000017f2d5c SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
    0x090000000180aa84 SQLExecute
    0x000000010009341c FreePrepareExecute
    0x000000010009c77c MakeIndex
    0x000000010009c174 CreateIndexesThread
    0x000000010000c264 StartThread

Resolving The Problem

If necessary, add additional database space to allow the creation of the table index(es) to complete successfully.

[{"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.3;7.1","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Product Synonym

adsm tsm

Document Information

Modified date:
17 June 2018

UID

swg21993118