Active log fills when a transaction runs for a long time

Abstract

The Tivoli Storage Manager V6.1 server might run out of space in the active log when one or more transactions are open for a long period of time. The server shuts down because all available active log space is exhausted.

Content

PROBLEM

The Tivoli Storage Manager V6.1 server might terminate (shut down) because the active log space is exhausted. This occurs when a transaction is running for a long period of time. A long-running transaction can occur for a number of reasons. Some examples are:

• A Tivoli Data Protection client is backing up a large database as a single transaction.
• A Tivoli Storage Manager backup-archive client is backing up or archiving a large single object such that the object is sent in a single transaction. In this case, a "large single object" is a file or object that is hundreds of gigabytes or terabytes in size.
• A Tivoli Storage Manager server operation such as the IDENTIFY process for data deduplication is processing a large single object, and requires the transaction used to be open and in-flight for a long period of time. In this case, a "large single object" is a file or object that is hundreds of gigabytes or terabytes in size.

The problem can occur when the server workload is generally light otherwise. When server workload is light, online table reorganization might occur, and because the workload on the system is light the reorganization operation can make rapid progress through the table that is selected. The rapid progress by the table reorganization drives the head of the active log forward quickly while an existing long-running transaction causes the tail of the log to seemingly be pinned, preventing the database manager from being able to truncate the active log and manage this workload. The active log can eventually span the maximum configured number of log files and reach an out-of-log space condition. When the server is out of log space, the server itself initiates a shutdown.

Another important aspect of this issue is that the active log usage can occur during times when the server is somewhat idle or only processing a light workload. This occurs because the table reorganization is opportunistic, meaning that when the system is relatively idle the reorganization is performed so as to not compete for resources such as when the system is under a heavy workload.

An alternate symptom that might be seen is that the archive log space (directory) fills up quickly even during periods of inactivity or light workload on the server. If the archive log directory fills up, the active log files are then archived to the active log directory. And then when the active log directory becomes full, the server shuts down because of the out-of-log-space condition.

This issue is documented by APAR IC62978.

RECOMMENDATION

Apply the patch or fix pack containing the fix for this APAR when it is available.

CIRCUMVENTION

There are two possible circumventions for this issue:

Circumvention #1: Increase the active log size to allow for enough log space to handle transactions that are open for a long period of time in conjunction with table reorganization activities that drive the active log use as a consequence of the reorganization operation.

Circumvention #2: An undocumented server option is available that disables the server from performing automatic table reorganization. To disable automatic table reorganization, add the following line to the server options file and then restart the server:

ALLOWREORGTABLE NO

SOLUTION and SERVER PACKAGE LEVELS CONTAINING THE FIX:

The Tivoli Storage Manager server patch 6.1.2.1 and fix pack 6.1.3.0 contain the fix for this issue. This is documented in APAR IC62978.

After the fix for APAR IC62978 has been applied, if the server option ALLOWREORGTABLE NO had been implemented, either remove it from the dsmserv.opt file or else change the setting to ALLOWREORGTABLE YES to reenable on-line table reorganization.

Related information

Recovery log overview: Active log and archive log