Troubleshooting
Problem
Excessive database growth is witnessed following a server upgrade. The server instance may crash if all available database space is consumed.
Symptom
The activity log may show the following error message prior to the server outage:
ANR0175E tbtbl.c(9007): Space exhausted for one or more of the following: DB, LOG, or instance directory.
Cause
If the server upgrade modified any of the table indexes then it will be necessary for the indexes to be recreated. The recreation of the index occurs during the initial restart of the server after the upgrade has completed. The amount of database space required to recreate the index will vary depending on the size of the table associated with the index. If the index to be recreated is associated with a large database table, then a significant amount of database space may be required to create the index.
Diagnosing The Problem
The db2diag.log will show that the initial attempt to create the table index fails due to an out-of-space condition:
- 2016-10-21-07.26.56.343318-240 I47826428A652 LEVEL: Severe (Origin)
PID : 66388092 TID : 55258 PROC : db2sysc 0
INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
AUTHID : TSMSC01
EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbGrowNewStripeSet, probe:20
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
DATA #1 : String, 108 bytes
An error caused the allocation of a new stripe set of containers to an automatic storage tablespace to fail.
2016-10-21-07.26.56.345020-240 E47827081A1122 LEVEL: Error
PID : 66388092 TID : 55258 PROC : db2sysc 0
INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
AUTHID : TSMSC01
EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbAllocateExtent, probe:835
MESSAGE : ADM6044E The DMS table space "LARGEIDXSPACE1" (ID "6") is full. If this is an autoresize or automatic storage DMS tablespace, the maximum table space size may have been reached or the existing containers or storage paths cannot grow any more. Additional space can be added to the table space by either adding new containers or extending existing ones using the ALTER TABLESPACE SQL statement. If this is an autoresize or automatic storage DMS table space, additional space can be added by adding containers to an autoresize table space or by adding new storage paths to an automatic storage database.
2016-10-21-07.30.33.761203-240 I47840265A559 LEVEL: Severe
PID : 66388092 TID : 55258 PROC : db2sysc 0
INSTANCE: tsmsc01 NODE : 000 DB : TSMDB1
APPHDL : 0-614 APPID: *LOCAL.tsmsc01.161020162611
AUTHID : TSMSC01
EDUID : 55258 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, catalog services, sqlrlCreateIndexPart, probe:13803
CALLED : DB2 UDB, data management, sqldIndexCreate
RETCODE : ZRC=0x85020021=-2063466463=SQLB_END_OF_CONTAINER
"DMS Container space full"
During subsequent restarts the db2diag.log file will show errors indicating that a temporary table space is full:
- 2016-10-22-01.39.15.246426-240 E48053113A778 LEVEL: Error
PID : 11075824 TID : 17131 PROC : db2sysc 0
INSTANCE: tsmsc01 NODE : 000
EDUID : 17131 EDUNAME: db2pclnr (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbClnrAsyncWriteCompletion, probe:0
MESSAGE : ADM6017E The table space "LGTMPTSP" (ID "7") is full. Detected on container "/home/tsmsc01/DB/db6/tsmsc01/NODE0000/TSMDB1/T0000007/C0000007.TMP" (ID "7"). The underlying file system is full or the maximum allowed space usage for the file system has been reached. It is also possible that there are user limits in place with respect to maximum file size and these limits have been reached.
The output from the SHOW THREADS command will also show an active CreateIndexesThread thread:
- Thread 208, Parent 1: CreateIndexesThread, Storage 11900, AllocCnt 14 HighWaterAmt 11900
tid=1fd0, ptid=1, det=1, zomb=0, join=0, result=0, sess=0
Stack trace:
0x09000000001f1b10 semop
0x0900000001462b0c sqloSSemP
0x0900000001462f00 .sqlccipcrecv.fdpr.clone.784__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
0x0900000001463904 .sqlccrecv.fdpr.clone.16
0x0900000001463600 sqljcReceive__FP10sqljCmnMgr
0x090000000146d040 sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
0x09000000017c3754 CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x09000000017f2d5c SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
0x090000000180aa84 SQLExecute
0x000000010009341c FreePrepareExecute
0x000000010009c77c MakeIndex
0x000000010009c174 CreateIndexesThread
0x000000010000c264 StartThread
Resolving The Problem
If necessary, add additional database space to allow the creation of the table index(es) to complete successfully.
Product Synonym
adsm tsm
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21993118