Auditing and repairing a deduplicated file storage pool

Resolving The Problem

Overview

Note: The dedupRepairTool.pl script currently does not support container storage pools. If you are running server version 7.1.3 or later and have configured container storage pools, do not run the dedupRepairTool.pl script against any such storage pools.

There can be various reasons for encountering server messages that indicate damage within a deduplicated storage pool. The first step to resolving any deduplication catalog inconsistency is to run an analyzer against the storage pool to determine where the problem might be. Then you would complete the steps to resolve the error. The attached dedupAuditTool.pl script does the analyzing.

It is important to note that the dedupAuditTool.pl is not intended to be an overall health check for deduplication enabled storage pools. You should only run the tool if you have experienced one of the symptoms described in this article.

The dedupAuditTool.pl is a perl script, which performs the analysis phase. The analysis phase is designed to accept environment and symptom information from the user. With this information, the tool interrogates the various IBM Spectrum Protect tables that are associated with the given symptom that was specified. At then end of the analysis phase, an audit report is generated. Send this report to IBM support for analysis. If the analysis determines that the problems identified in the report need correcting, they will provide you another tool, dedupRepairTool.pl, to perform the recovery phase. The recovery phase is responsible for either removing the damage, restoring from a copy source, or potentially both, depending on the salvage ability for each given object.

Script Details

General Script Information:

• Perl must be installed on the system where the IBM Spectrum Protect server resides. Because the Perl scripts interrogate the DB2 database directly, they must be run from this system. To obtain code for the Perl installation, go to http://www.perl.org
• The tools must be run with a user ID that has access to the DB2 instance.
• The instance user must have FULL read/write access to the directory where the scripts are being executed and the directory where the client is installed (for example, /opt/tivoli/tsm/client/ba/bin64).
• The tools require access to both DB2 and IBM Spectrum Protect by using the dsmadmc and DB2 CLIs.
• On Windows systems, initialize the DB2 command line environment prior to executing the scripts by issuing "db2cmd" from a Windows administrative command prompt.
• You must use Perl 5.8.0 or later (A Perl interpreter can be download from - www.perl.org).
• Different versions of the dedupAuditTool scripts are available depending on which interpreter is available

- dedupAuditTool.pl (5.10 or later)
- dedupAuditTool_perl_v580 (5.8.0 - 5.10)
• The tools are interactive scripts. They do not accept command line parameters.
• The tools are multi-phase scripts (the dedupAuditTool.pl script does not include the final phase) - Environment Setup, Audit Tool Setup, Audit Analysis and Report Generation, Cleanup of Deduplicated Storage Pool.
• It may be necessary to run the dedupAuditTool.pl and dedupRepairTool.pl multiple times in order to restore the referential integrity of the database.

How Are The Scripts Executed?:
perl dedupAuditTool.pl
perl dedupRepairTool.pl

Important Usage Notes:

• An AUDIT VOLUME command should be run against all suspect volume(s) PRIOR to running the script dedupRepairTool.pl in RECOVERY mode. This will ensure that all files currently marked as damaged are validated as truly damaged before they are forcibly deleted.
• If a DELETE VOLUME DISCARDDATA=YES command has been issued against a volume that is located in a deduplicated storage pool, the dedupRepairTool.pl script should be run immediately afterward with the ANR4895E symptom to resolve all of the invalidated links. Contact IBM support for access to dedupRepairTool.pl.

• The script should be run only in a deduplicated environment for which the following requirements are all true (the script will prompt for these as well):

- A current FULL database backup is available.
- The IBM Spectrum Protect server is at 6.3.5.100 or later.
- The dedupRepairTool.pl script should only be run against primary storage pools. Do not run the script for repairs against deduplicated copy pools and active data pools. The dedupAuditTool.pl script can be run against these pools, but the CHUNKS_NOT_CATALOGED category in the report it generates may include false positives for active data pools.
- All data movement and expiration activity for the deduplicated storage pool has been quiesced; refer to the section "Preparation for running script" below for details. If the storage pool is not quiesced, the script may report false-positive results, have or cause the server to have severe performance issues, or cause deadlocks in server operations.

• It is typically recommended to run the RECOVERY mode with BOTH so that all restorable objects are recovered from a copy source and all non-restorable objects are removed. This will result in the fastest and most straightforward cleanup possible.

Audit Details
Audit Symptom Categories and Coverage Details:
- ANR4895E
An invalidated deduplication chunk link has been found.

- ANR1165E and ANR1162W
A damaged file has been found in a deduplicated storage pool.

- ANR1529I
Damaged and expired base chunks have been found in a deduplicated storage pool.

- MISSING_EXTENT_ENTRY
Category that checks for any deduplicated object that is no longer cataloged.

- ORPHANED_EXTENT
Category that checks for an invalidated chunk that only exists in the deduplication catalog table.

- INVALIDATED_LINKS
Category that checks for any invalidated links that might exist in the deduplicated storage pool.

- MISSING_EXTENT_ENTRY
Category that checks for any deduplicated object which no longer has an extent in the deduplication
catalog.

- MISSING_AF_ENTRIES
Category that checks for any deduplicated extent that no longer has a corresponding entry in the
volume tracking table.

- MISSING_VOL_ENTRIES
Category that checks for a deduplicated object that no longer has a valid volume associated to it.

- ZERO_LENGTH_CHUNKS
Category that checks for any base data chunks (not links) that no longer contain actual storage
references.

- MISSING_CHUNKS
Category that checks for an object that has been deduplicated but no longer has entries in the
database deduplication catalog.

- MISSING_CHUNKS_EXTENDED
Category that performs additional resource intensive checks beyond what the MISSING_CHUNKS
symptom covers.
- ALL
The ALL category will go through all of the above symptoms plus a few others that are not
documented above. This category can run for a long period of time, depending on the size of the database and the deduplicated storage pool.

Audit Phase Options (dedupAuditTool.pl, dedupRepairTool.pl):
* The deduplication catalog can be analyzed by either auditing a symptom or by processing a
previously generated audit file

Recovery Phase Options (dedupRepairTool.pl only) :
* The following cleanup phase options are available. The cleanup phase occurs after the analysis
is completed:
- Either an audit file can be generated, for future processing, or the script can enter a recovery
procedure. If the recovery procedure is chosen, the following modes are available:
- Restore/Delete
All invalid data is attempted for restoration. If data cannot be restored it will then be removed
from the IBM Spectrum Protect server.
- Restore Only
All invalid data is attempted for restoration from a copy pool.
- Delete Only
All invalid data will be removed from the IBM Spectrum Protect server.
- Preview
All recovery steps are taken, including restoration and deletion, but no action is performed. The
console will display each step that would be covered if the Recovery was executed without
PREVIEW.

Preparation for running script

Before proceeding with the cleanup procedure, perform the following:
• A current FULL database backup is taken
• All data movement and deletion activity in the deduplicated Storage pool has been quiesced. This would include the following processes:
• Expiration
• Reclamation
• Migration
• Move Data
• Move NodeData
• Identify Duplicates
• Backup/Archive
• To accomplish this add the following options in the server options file and restart the server:
• NOMIGRRECL
• EXPINT 0
• DISABLESCHEDS YES
• Prevent all processing in the deduplication enabled storage pool by issuing the following commands:
• DISABLE SESSIONS
• IDENTIFY DUPLICATES storage_pool_name NUMPROC=0
• IMPORTANT NOTE: Once you have resolved all issues, reset all options and allow processing to the storage pool to resume, perform the following:
• Remove the following options you added above:
• NOMIGRRECL
• EXPINT 0
• DISABLESCHEDS YES
• Restart the server and issue the following command:
• ENABLE SESSIONS

Script Examples (contains both dedupaudittool.pl and deduprepairtool.pl output)







Attached Scripts
dedupAuditTool.pl - Script that requires PERL version 5.10 or later
dedupAuditTool_v580.pl - Script that requires PERL version 5.8.0 - 5.10

NOTE: It is recommended that dedupAuditTool.pl is used if the 5.10 or later PERL
interpreter is available for the given operating environment.