Damaged or broken catalogs

Symptoms

Jobs hung in open or close processing or other hang symptoms
Unexplainable catalog errors during locate errors

How to investigate

Run the following commands in an IDCAMS batch job to identify errors:

//STEP1    EXEC  PGM=IDCAMS,REGION=0M
//SYSPRINT DD    SYSOUT=A //SYSIN    DD *
DIAGNOSE ICFCATALOG INDATASET(catalog.name
EXAMINE NAME(catalog.name) ITEST NODTEST
EXAMINE NAME(catalog.name) NOITEST DTEST
LISTCAT ENT(catalog.name) CAT(catalog.name) ALL    
LISTCAT CAT(catalog.name) ENTRIES

If DIAGNOSE or EXAMINE indicate a condition code 8 or greater, run the following DSS PRINT jobs to capture a picture of the catalog:

//STEP2    EXEC PGM=ADRDSSU,REGION=0M   
//SYSPRINT DD   SYSOUT=*   
//SYSUDUMP DD   SYSOUT=*   
//SYSIN    DD *      
   PRINT DATASET('catalog.name') - 
     INDYNAM(volser)      
   PRINT DATASET('catalog.name.CATINDEX') -  
     INDYNAM(volser)

Recovery actions

Catalog breakages can range from minor logical DIAGNOSE errors to major structural EXAMINE errors.

For only DIAGNOSE errors, follow recovery actions listed under IDC21364I in MVS System Messages, Vol 6.
If errors include EXAMINE errors, attempt the following actions:
1. Determine if a cc=0 backup can be taken by using IDCAMS EXPORT. Ensure the number of records exported resembles a number close to the number of records you expect in the catalog.
```
//STEP1    EXEC  PGM=IDCAMS,REGION=0M
//CATBKUP  DD DSNAME=CAT.BACKUP,UNIT=SYSDA,
//            DISP=(NEW,CATLG),SPACE=(CYL,(10,10))
//SYSPRINT DD    SYSOUT=A
//SYSIN    DD *
   EXPORT catalog.name OUTFILE(CATBKUP) TEMPORARY 
```
2. If EXPORT runs fine, refer to instructions listed in Chapter 6 "Backing Up and Recovering Catalogs" in DFSMS Managing Catalogs section "Recovering a BCS" to IMPORT this back-up, which will rebuild the catalog index.
3. If EXPORT fails, recovery from back-up will be necessary. With IBM products, this can be performed by locating the last good back-up taken of the catalog and then forward recovering the back-up to the current time. This will require a back-up produced by EXPORT, SMF 61,65, and 66 records, and use of the IBM supplied forward recovery utility ICFRU. Refer the chapter "Integrated Catalog Forward Recovery Utility" in DFSMS Managing Catalogs for instruction in using ICFRU.
4. If EXPORT fails and a good catalog back-up cannot be located, salvaging entries will be necessary. Refer to "Merging Catalogs" in DFSMS Managing Catalogs for direction on how to use REPRO MERGECAT to move entries from the broken catalog to a new catalog.

Actions to avoid recurrence

To help avoid breakages, ensure that catalogs being shared between systems reside on DASD generated as shared and that the catalog is defined with share options (3 4). If the catalog is accessed from outside of the sysplex, ensure sharing is performed with RESERVEs rather than through a convert RNL for SYSIGGV2.

To ensure catalog recovery can be performed in a timely manner, take regular back-ups via IDCAMS EXPORT. Ensure DIAGNOSE and EXAMINE are ran prior to each EXPORT to ensure a valid back-up is obtained. If back-ups are taken with another product like DSS, an additional step of restoring the back-up and then exporting that copy will be required in order to use the ICFRU product since that product only accepts EXPORT output for input.