Detailed description of the defect fix that avoids an index corruption due to insufficient disk space and file access violation.
The following conditions can cause the Net Search Extender text index files to become corrupt, requiring a complete rebuild of the index:
Full disk situation: The file system where the permanent and temporary Net Search Extender index files are stored can run out of disk space during a text index update or index reorganization operation.
File access violation: Anti virus scans or any other user applications can request exclusive access to files in the index or work directories during a text index update or index reorganization operation resulting in a program termination due to access violation (this condition has only been observed on Windows).
The following solutions, implemented in Net Search Extender V9.1FP8, V9.5FP6 and V9.7FP1 onwards, address these index corruption issues:
Disk full situation and file access violation fix
Disk full situation
The Net Search Extender index files can become corrupted if the file system containing the index and work directories runs out of disk space during an index update. During an index update operation additional temporary disk space is required in the index and work directories apart from the space for the actual index files. If this space is not available during an index update, the index update may be terminated causing the Net Search Extender text index files to become corrupted. The corrupted index cannot be used and a new index must be created.
To avoid an index becoming corrupted due to insufficient disk space, the current implementation tries to estimate the disk space that will be required for an index update and will stop the index update process with the following message if the estimated space is not available on the file system:
CTE0310 "Index update is not allowed due to insufficient disk space."
This check is performed by Net Search Extender before the index update operation is performed, hence no errors will be logged in the event view. The error is only logged in the diaglog.
Limitations of the current fix:
The current fix has some limitations which will be addressed in future releases.
The disk space is calculated only for an incremental update.
The index can still become corrupted if there are multiple index updates executed in parallel that share the same file system for the index directories.
Users should follow the suggested guideline and have both the index and work directories on the same file system. It is strongly recommended to not use different file systems for the index and work directories for a single index.
File access violation
Retry logic for the index file access
The index file may not be accessible during index update by Net Search Extender if another application has locked the file. This causes Net Search Extender to terminate the index update process and the index may become corrupted.
To avoid corrupting an index in such cases, a retry logic has been implemented for the Windows platform to protect files against locking violations and file sharing issues. This means that if an internal file is not accessible, that is it cannot be opened, renamed or removed, the file access operation is retried. The retry time interval for any failed file access operation is approximately 30 minutes.
If the file is not accessible due to a backup program or any other application holding it for more than 30 minutes, the index update process might still corrupt the index. Refer to the Preventive Measures Against Net Search Extender Index File Corruption Technote to handle this situation.