Troubleshooting
Problem
When attempting to retrieve a document from IBM® Content Manager OnDemand, the following error is thrown: "The server failed while retrieving a document." What does this error mean?
Symptom
09/26/2021 18:12:54 ADMIN 57230 Error No 23 Object >1FAAA< in Application Group >Baxter Bay Credit< not found in cache, no other storage defined Srvr->od.myserver.com 9.30.154.36<-
09/26/2021 19:15:24 ADMIN 53415 Error No 24 Object >1FAAA< in Application Group >BAA< not found in node >PRI7YR< Srvr->od.myserver.com 9.30.154.36<-
Cause
This is known as an orphan index, where an index value exists in the Content Manager OnDemand database, but the associated storage object containing the document cannot be found in Content Manager OnDemand or Tivoli Storage Manager (TSM) storage.
Possible causes of an orphan index in:
Cache only application group:
- Cache Document Data for X Days is configured to less days then the Life of Data and Indexes in the application group definition.
- Content Manager OnDemand index expiration has not been run (arsmaint -d), while cache expiration has been run (arsmaint -c).
- Content Manager OnDemand cache was manually altered or is damaged.
- Configuration change in Content Manager OnDemand:
- Search Cache was changed in the application group.
- TSM client node was changed in the storage set definition.
- The Life of Data and Indexes in the OnDemand application group is configured to more days then the TSM archive copy group retention period (retver).
- Configuration change in the TSM Client API:
- Options or system options file (dsm.opt or dsm.sys) was changed to connect to a different TSM server.
- Configuration change or problem in the TSM server:
- Archive copy group retention period (retver) in TSM is configured to fewer days then the Life of Data and Indexes configuration in the Content Manager OnDemand application group.
- A change was made in the TSM server's policies (client node, policy domain, policy set, management class, storage pool, storage volume, archive copy group).
- Inconsistency between TSM database and storage pool volume. This can occur if either was restored from a point in time before the document was archived into Content Manager OnDemand or if the storage object, volume, or TSM database entry is marked as damaged or unavailable.
- The storage object, file space, or volume was manually deleted.
- Object server was changed in the Content Manager OnDemand storage node definition.
- For an Expiration Type=segment application group, the segment has not qualified for expiration, but the associated storage object(s) have expired. With an Expiration Type=segment application group, one segment (table of application group data) is expired at a time. A segment, by default, can contain a maximum of 10 million rows, which represents 10 million documents. A segment qualifies for expiration when the highest segment date in the table plus the Life of Data and Indexes is past the present day and the segment has been closed by either reaching the maximum rows setting or manually closed via the arstblsp command. If the segment has not qualified for expiration, index values might persist longer then their associated storage object's retention period. For Content Manager OnDemand cache, this would be the Cache Document Data for X Days configuration. For TSM storage, this would be the archive copy group retention (retver) configuration.
- Content Manager OnDemand database is out of sync with Content Manager OnDemand or TSM storage. This can occur if the Content Manager OnDemand database was restored, but storage was not, or vice versa. For example, if storage objects were expired by running cache or index expiration (against Expiration Type=load application groups) before a Content Manager OnDemand database was restored, this might result in orphan index values in the Content Manager OnDemand database. Since Content Manager OnDemand System Log messages are stored in the database, no record of cache expiration and index expiration would exist.
- Content Manager OnDemand database was restored to a different environment for testing or backup purposes and the storage set definition(s) from the Content Manager OnDemand database are still referencing the same TSM client node and Content Manager OnDemand object server from the original environment. All expiration and deletion of storage will be executed against the original environment (cache expiration, index expiration, and deletion of an application group).
Care should be taken when a database is restored to a different environment. Even if the object server configuration is changed in the storage set, storage operations will be performed against the original environment if the new object server still references the same TSM Client Node and TSM server as the original environment.
Note: In a TSM-enabled Content Manager OnDemand server, the TSM Client API is used to archive and retrieve data. Only the TSM Client node name and password are stored in Content Manager OnDemand. The TCP/IP address and port of the TSM server is obtained from the TSM Client API options files. This information, in addition to the unique Content Manager OnDemand application group identifier and storage object name, is used to query TSM for data. Content Manager OnDemand uses the default management class to archive objects in TSM.
Resolving The Problem
A review of each possible cause must be performed. Historical log information can be found in the Content Manager OnDemand System Log and TSM Activity Log. The Content Manager OnDemand System Log will record configuration changes to application groups and storage sets. The TSM Activity Log will record configuration changes to retention policies and when deletion processes such as audit volume, delete volume, delete file space, delete object, had occurred.
In a Content Manager OnDemand cache-only application group, it is recommended that you backup Content Manager OnDemand cache regularly. For TSM-enabled application groups, it is recommended that you copy storage pools to maintain regular backups of primary storage pools. Storage pool backup is not supported for Centera storage pools. If a Centera storage pool is used, it is recommended that you use the replication feature found in the Centera storage device instead.
If further assistance is required while investigating an orphan index, contact IBM Support.
Related Information
Was this topic helpful?
Document Information
Modified date:
14 March 2024
UID
swg21392642