What is the procedure to recover when IBM Content Analytics with Enterprise Search Parser & Indexer is stuck after installing Fixpack 2?
This has been identified as a product defect under APAR IC91515
Resolving the problem
Here is the procedure for resolving when the parser appears to be stuck.
When the parser/indexer is perceived as no longer parsing & indexing documents (ESAdmin UI not showing increase in indexed documents), then perform the following procedure:
1) Login to the ICA master server with the ICA administrative userid.
2) Execute the following command a few times to see the parser/indexer detail
esadmin <cid>.indexservice getIndexBuildStatusDetail
3) If after a few executions, the parser thread seems to be stuck on a specific document for over 10 minutes, then it is stuck on that specific document. Please remember this docid.
4) Grab a javacore for the following three processes with the following commands on the master server
esadmin <cid>.indexservice getMemStatus -dumpJava
esadmin <cid>.docproc.nodex getMemStatus -dumpJava
esadmin <cid>.stellent getMemStatus -dumpJava
Note that the javacore files are stored in the $ES_NODE_ROOT/logs directory. Please send them to IBM Support after gathering them to see if anything surprising can be observed.
5) Kill the following three processes with the following commands on the master server
esadmin <cid>.indexservice destroy
esadmin <cid>.docproc.nodex destroy
esadmin <cid>.stellent destroy
Check if they are killed. If not, then use the kill command on the respective server.
Note that nodex where x is a number corresponds to the node number. For example, if node1 is the master server, then the kill must be executed on the master server for <cid>.docproc.node1. If you have additional document processing server (ie., node2) then the kill must be executed on the addition document processing server.
6) Restart these killed processes by starting the parser/indexer from the ESAdmin UI
7) Execute the command from #2 above and you should see the parser moving again.
8) In the meantime, extract that problem document from #3 above, and see if it can be placed in a test collection for reproducing the problem. If yes, then please send a copy of this document to IBM support to further the investigation.
9) Run the DumpIndex.sh command to see if that problem document is found in the index. If yes, then the parser / indexer can progress the document properly and no further action is required. This is only intermittent.
10) If problem document is not found in the dump index, then you probably should re-crawl this document again (via a full recrawl).