File Versioning, File Retention and File Expiration Explained

Resolving The Problem

This document provides a technical explanation of how the File Expiration process works on the Tivoli Storage Manager server, with relation to the backup copygroup parameters: VerExists, VerDeleted, RetExtra and RetOnly. It is a supplement to the information presented in the product documentation (ie: Tivoli Storage Manager Server Administrator Guide and Tivoli Storage Manager Administrator Reference)

Some points to note before continuing:

1. There are TWO attributes involved that control how long the Tivoli Storage Manager server stores data. They are: Versioning and Retention. Both attributes work together, thus resulting an a series of possible cases that need to be examined.

2. Modifying any Versioning related parameters in the copygroup requires a backup to be initiated to make the changes take affect. Versioning parameters are evaluated only at the time of backup.

3. Modifying any Retention related parameters in the copygroup results in the retention setting being applied to all backups using the modified management class/copygoup. Current files backed up and future backups stored on the Tivoli Storage Manager server dynamically, and immediately, inherit the new settings, no subsequent backup is required.

4. This is a summary of how the file expiration process works. When a backup object (ie: file) changes from an active to inactive state, it will have a "deactivation date" assigned at the time of the transition. The file is then tracked in a server database table entitled: Expiring.Objects (V5). The process of file expiration is one where the Tivoli Storage Manager server evaluates the deactivation dates and management class attributes of the entries in the Expiring.Objects table and purges the appropriate entries. Once purged, that particular version of the file can no longer be restored by Tivoli Storage Manager.

Case A: File Exists on the Client, Backup results in VerExists Being Exceeded

=================================================

When a client runs a backup (incremental or selective), the Tivoli Storage Manager server will evaluate whether or not the client's backup will result in the creation of an additional backup version. If this additional backup version results in the VEREXISTS setting being exceeded, the Tivoli Storage Manager server takes the oldest version of the file, inactivates it and assigns it a date of "minus-infinte". The "minus-infinite" is a special date/time stamp that is used by the Tivoli Storage Manager server to identify that the file must be purged immediately when expiration is next run. Since the evaluation of the versioning settings occurs AT THE TIME OF BACKUP, any changes made to the VEREXISTS and/or VERDELETED parameters, will only be applied on the next backup.

For example, a client is backing up to a management class with VEREXISTS=5. A file is being modified frequently and, as such, the file is backed up nightly during the incremental backups. After 5 nights of backups there will be 5 versions of this file on the Tivoli Storage Manager server; 1 active version and 4 inactive versions. On the sixth night of backups, the oldest inactive version of this file will be marked for deletion because VEREXISTS=5 will only permit the Tivoli Storage Manager server to retain a maximum total of 5 versions of this file.

If the VEREXISTS value is reduced to 2, during the next incremental backup of this file the Tivoli Storage Manager server will recognize that the VEREXISTS value has been exceeded and the 3 oldest inactive versions of this file will be marked for immediate deletion.

When the VEREXISTS is exceeded (a versioning parameter), there is no evaluation of how long to keep the deactivated version for. In other words, retention settings do not apply to versions being removed, as a result of exceeding the maximum number to be kept.

Case B: File Exists on the Client, Backup does NOT result in VerExist Being Exceeded

================================================

If a backup results in a the creation of an additional version, but VEREXISTS is NOT exceeded, the Tivoli Storage Manager server will deactivate the current version of the file for the new one. The resulting inactive version gets assigned a deactivation date and logged into the Expiring.Objects table (V5).

When file expiration runs, Tivoli Storage Manager will compare the time/date difference between the current System Date/Time and the Deactivation Date/Time of the file(s). If this period is greater than that specified by the RETEXTRA setting, the inactive file is purged. If this period is less than that specified by the RETEXTRA setting, the inactive file remains in the Expiring.Objects table for the next iteration of expiration. This cycle continues until the file is purged.

Case C: The file has been deleted from the client

================================

If a file has been deleted from the client node, AND an incremental backup is run, the Tivoli Storage Manager client will inform the server of this fact. This results in following actions on the Tivoli Storage Manager server:

a) The server deactivates the active version of the file and no more versions are inserted into the database.

b) The VERDELETED parameter is used to limit the number of versions held by the Tivoli Storage Manager server. Any number of versions in excess of the VERDELETED setting will be marked for immediate expiration and purged (via the same mechanism as described above in Case A).

c) All the inactive versions are tracked in the Expiring.Objects table (V5). Each version of the file is purged in accordance to the settings of the RETEXTRA parameter, except for the LAST version. Its deactivation date will be evaluated against the RETONLY parameter of the management class setting. This is a safety feature of Tivoli Storage Manager to allow the last version (most recent) to be recoverable for an extended period of time, should a user wish.

For example, assume the following backup copygroup settings are active:

VerExists: 5
VerDeleted: 2
RetExtra: 30
RetOnly:60

A file that is bound to this copygroup/management class, and deleted from the client workstation will result in 2 versions remaining on the Tivoli Storage Manager server. The oldest of the 2 versions will remain available for restore for 30 days from deactivation. The last version (most recent changes) will remain available for restore for 60 days from deactivation

Since retention settings don't require a backup to be run to activate them, the values can be changed and the effects are immediate. Assume a file is backed up to a management class with a RETONLY value of 30 days. This file becomes inactive on 1/1/2003 because the file was deleted on the client. Based on the RETONLY value of 30 days, this inactive file would be eligible for expiration on 1/31/2003. If, however, the RETONLY value for the management class is increased to 90 days sometime prior to 1/31/2003, the inactive file will not expire until 3/31/2003.

Case D: The Client Command: EXPIRE is used

========================================

Beginning with Tivoli Storage Manager Backup-Archive client 4.2.x, a new client command called EXPIRE was introduced. It gave users the ability to "purge" file no longer needed from the Tivoli Storage Manager server, without deleting the node's entire filespace. The effect of using the EXPIRE command is the same as deleting the file from the client. See the explanation above, in Case C for details on how the Tivoli Storage Manager server will expire the file in this case.

Case E: The Client Command: DELETE BACKUP is used

=================================================

Customers can delete their backed up files from the Tivoli Storage Manager client node workstation (if the function is enabled on the Tivoli Storage Manager node's definition). When this action is performed, the server will take all the backed up files that meet the 'filespec' and 'deltype' specified and deactivate them. It also assigns a deactivation date of 'infinite-minus' so that the files are no longer available for restore and are purged, immediately on the subsequent run of the expiration process.