On Cross Commital Servers, error tuple 80,1,61 seen. Object already exists.

On Cross Commital Servers, error tuple 80,1,61 seen. This indicated object already exists.

Errors reported for 80,1,61 were seen when there were issues with duplicate write requests. The SCR 268981 changed the way these errors were reported. The object Id will now be reported to assist in correcting the issue.

The module SCR 268981containing DOCb was found to have a defect under stress testing and a new module (SCR 273002) was released to fix this defect. Please install this module using Hot Fix Pack 16.


The 80,1,61 problem itself is related to an object that was deleted from cache but the write request had been left in the system.

With the developer module currently being used, we can see these errors reported:
2007/04/24 11:16:45.138 80,1,61 <fnsw> DOCs (26166) ...
DOCb_batch_close: Write request doc_id: 4160005781 already exists, This object must have been removed from cache.

2007/04/24 11:16:58.708 80,1,61 <fnsw> DOCs (26166) ...
DOCb_batch_close: Write request doc_id: 4160005789 already exists, This object must have been removed from cache.


Symptom:
----------
Creation of Fast Batches resulted in 80,1,61

Cause:
----------
Used Delobjects in CSM tool to delete objects from cache. Some of the objects deleted were
Fast Batches. Creation of any fast batches after this resulted in a 80,1,61.

When in multi commital all docs are sent as FB and locked in bes cache till it connects to the other server which then writes these docs ( fast batch breakup) and then releases it from the other server.

checkwrt command on IMS1 didn't give us results and running from IMS2 gives us the following:

(IMS2) cltfilapap03(fnsw)/fnsw/bin> checkwrt
Object 4160005781 for fast batch 'FXPMigrationD00.1' missing from cache
Object 4160005789 for fast batch 'FXPMigrationD00.2' missing from cache

When we have errors on IMS1 : On IMS2 : MKF_tool se docs doc_id=xxx ( where xxx are the documents that have the rmt_commit problem) and also do :On IMS2 MKF_tool select write_request *

Engineering thinks they know the reason that is causing our problem :

We believe that we found the cause of the problem. The CSM logs on IMS2 show that on June 22, 2006 someone deleted a batch object on the IMS2 server. This can be seen by the following entry in the /fnsw/local/logs/csmlog directory.

2006/06/22 16:28:49.314 <fnsw> CSM_tool (92496 0x16950)

delete 1 1100200616 4160005789


2006/06/22 16:28:56.768 <fnsw> CSM_tool (92496 0x16950)

1 1100200616 4160005789 1 819200 deleted



The MKF write_request table shows that the write request for the Fast Batch object still exists.

MKF_tool > select write_request * where doc_id = 4160005798

doc_id.........4160005789 family_id..............73 num_pages..............68 req_type................1
commit_error...0x5a000018
batch_name..........."FXPMigrationD00.2"


There is also another entry in the write request table that has the same error.

MKF_tool > select write_request * where doc_id = 4160005798

doc_id.........4160005781 family_id..............73 num_pages.............211 req_type................1
commit_error...0x5a000018
batch_name..........."FXPMigrationD00.1"

The problem occurs when someone removes a Fast Batch object without removing the write request in the MKF database. This is exactly what the error tuple translates to if they run "fn_msg 80,1,61".


You will need to check the write request ( MKF_tool) for that fast batch number and also check CSM_tool (listobject) for that fast batch number.
You will probably see that there is a write request in MKF_tool but no fast batch object in CSM_tool.

You can verify that they do not have old entries in the write_request table that refer to Fast Batch objects that do not exist. This should be done at a time when there are no new committals taking place. They can verify the MKF write_request table with the Fast Batch objects cache using the CSM_tool list command and remove the old write request in MKF_tool.
-----------------
80,1,61 during remote committal


==============================

To remove the write request for these 2 fast batch objects please do the following on IMS2:

MKF_tool
allowupdates : Will ask for password : This month's password is 'antenna' Next month's password is 'installed'
delete write_request doc_id = 4160005781
delete write_request doc_id = 4160005789


/End