IBM Support

Fixup crashes Domino 851FP1, FP2 and FP3 with Invalid Pool free Chain

Technote (troubleshooting)


Problem

In an attempt to fix a possibly problematic/corrupt database a customer runs a fixup on the database using any combination of the -F, -J and -O switches and the Domino server crashes on the fixup task.

Symptom

Server will crash with Panic Invalid Pool Free Chain

Cause

Code bug and corrupt database

Diagnosing the problem

Lotus Support obtained a customers database to reproduce the issue on Domino 8.5.1FP1, FP2,and FP3 by running a fixup on the database with of the switches referenced and the server crashed.

Fatals should match or be very similar to one of these:
############################################################
### FATAL THREAD 1/4 [  nfixup:  1cb0:  0140]
### FP=0x00128844, PC=0x6000d030, SP=0x0012883c
### stkbase=00130000, total stksize=45056, used stksize=30660
### EAX=0x42bb1000, EBX=0x00040300, ECX=0x00000000, EDX=0x42bb1004
### ESI=0x42bb1004, EDI=0xfffffffc, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
@[ 1] 0x6000d030 nnotes.CscanmemNE@12+64 (42bb0ff4,10,0)
@[ 2] 0x60699201 nnotes.DumpMemoryExtended@28+321 (42b30104,fffffffc,10,ffffffff,0,0,0)
@[ 3] 0x60699552 nnotes.DumpMemoryEx@24+34 (42b30104,fffffffc,10,ffffffff,0,0)
@[ 4] 0x606b9d26 nnotes.HandleBadFreeChain@4+502 (128a3c)
@[ 5] 0x6000310d nnotes.FreeDBlock@12+429 (42b3008e,fc,128a60)
@[ 6] 0x606b9fde nnotes.OSFreeDBlockWithSize@16+78 (61044783,f0,8,1)
@[ 7] 0x606ba012 nnotes.OSFreeDBlockExt@12+34 (61044783,f0,1)
@[ 8] 0x601b9d71 nnotes.DPoolFree@8+129 (12020ae,1)
@[ 9] 0x60006c98 nnotes.Discard@4+424 (12020a0)
@[10] 0x600089f3 nnotes.OSMemoryFree@4+179 (f01028dc)
@[11] 0x609d6b6b nnotes.ValidateItem@48+8027 (12f2b0,12981c,12946c,129894,0,0,0,0,0,0,0,0)
@[12] 0x609d8c2a nnotes.ValidateNote@44+5162 (12f2b0,4a,2630568c,12b2a8,0,0,0,0,0,0,0)
@[13] 0x609db3e9 nnotes.FixupNoteCommon@36+889 (12f2b0,37cedd1c,1,2630568c,0,0,0,0,0)
@[14] 0x609dfa07 nnotes.FixupBuckets@16+7447 (12f2b0,37cedd1c,12cba0,1)
@[15] 0x609e1d03 nnotes.FixupNotes21@12+83 (12f2b0,37cedd1c,12cba0)
@[16] 0x609e1f76 nnotes.FixupNotes@12+470 (12f2b0,37cedd1c,12cba0)
@[17] 0x609e40d2 nnotes.DbFixup@4+8466 (12f2b0)
@[18] 0x609c3f23 nnotes.DoFixup@8+787 (8000,37cede03)
@[19] 0x609c4d03 nnotes.DbLoadLite@4+755 (4241f2b0)
@[20] 0x608d1690 nnotes.NSFDbOpenExtended4@48+19664 (12fc6c,800018,8000,0,0,0,0,0,0,0,0,0)
@[21] 0x60027181 nnotes.NSFDbOpenExtended3@48+65 (12fc6c,800018,8000,0,0,0,0,0,0,0,0,0)
@[22] 0x00401ef2 nfixup.AddInMain@12+1746 (46400000,1,3c8944)
@[23] 0x0040226f nfixup.NotesMain@8+47 (3,400000)
@[24] 0x004023a4 nfixup.notes_main+212 (0,0)
@[25] 0x00402296 nfixup.main+22 (3,3d2b70)
@[26] 0x0040272d nfixup.mainCRTStartup+364 (0,0)
 [27] 0x77e6f23b kernel32.ProcessIdToSessionId+521 (4025c1,0)

Another Variation but still the same functions calls after the HandleBadFreeChain function call:

############################################################
### thread 1/3: [ nfixup: 0e38: 0518] FATAL THREAD (Panic)
### FP=0x00126f14, PC=0x7c82860c, SP=0x00126ea4
### stkbase=0x00130000, total stksize=45056, used stksize=37212
### EAX=0x77e80220, EBX=0x00000000, ECX=0x000000f0, EDX=0x00000000
### ESI=0x00000178, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297
############################################################
[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (178,493e0,0,12749c)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (178,493e0,0,1276b8)
@[ 3] 0x601b5c94 nnotes.OSRunExternalScript@8+1300 (12c,1)
@[ 4] 0x601b611f nnotes.FRTerminateWindowsResources+975 (1,1010,1,0)
@[ 5] 0x601b6548 nnotes.OSFaultCleanupExt@24+984 (c46a68,1010,0,0,0,1279e0)
@[ 6] 0x601b65ca nnotes.OSFaultCleanup@12+26 (0,1010,0)
@[ 7] 0x601c1b74 nnotes.OSNTUnhandledExceptionFilter@4+276 (128a18)
@[ 8] 0x601847dd nnotes.Panic@4+589 (60d1b59a)
@[ 9] 0x606b9db0 nnotes.HandleBadFreeChain@4+640 (128a3c)
@[10] 0x6000310d nnotes.FreeDBlock@12+429 (120008e,36684,128a60)
@[11] 0x606b9fde nnotes.OSFreeDBlockWithSize@16+78 (6104477f,36678,8,1)
@[12] 0x606ba012 nnotes.OSFreeDBlockExt@12+34 (6104477f,36678,1)
@[13] 0x601b9d71 nnotes.DPoolFree@8+129 (1201fa6,1)
@[14] 0x60006c98 nnotes.Discard@4+424 (1201f98)
@[15] 0x600089f3 nnotes.OSMemoryFree@4+179 (f01028d9)
@[16] 0x609d6b6b nnotes.ValidateItem@48+8027 (12f2b0,12981c,12946c,129894,0,128ea4,f10f10,ffffffff,0,0,0,0)
@[17] 0x609d8c2a nnotes.ValidateNote@44+5162 (12f2b0,4a,4de168c,12b2a8,0,1298c8,f10f10,ffffffff,0,0,0)
@[18] 0x609db3e9 nnotes.FixupNoteCommon@36+889 (12f2b0,14c2b38,1,4de168c,0,12a06c,f10f10,ffffffff,0)
@[19] 0x609dfa07 nnotes.FixupBuckets@16+7447 (12f2b0,14c2b38,12cba0,1)
@[20] 0x609e1d03 nnotes.FixupNotes21@12+83 (12f2b0,14c2b38,12cba0)
@[21] 0x609e1f76 nnotes.FixupNotes@12+470 (12f2b0,14c2b38,12cba0)
@[22] 0x609e40d2 nnotes.DbFixup@4+8466 (12f2b0)
@[23] 0x609c3f23 nnotes.DoFixup@8+787 (8002,14c2c1f)
@[24] 0x609c4d03 nnotes.DbLoadLite@4+755 (4241f2b0)
@[25] 0x608d1690 nnotes.NSFDbOpenExtended4@48+19664 (12fc6c,800018,8002,0,0,12f348,f10f10,ffffffff,0,0,0,0)
@[26] 0x60027181 nnotes.NSFDbOpenExtended3@48+65 (12fc6c,800018,8002,0,0,12f428,f10f10,ffffffff,0,0,0,0)
@[27] 0x00401ef2 nfixup.AddInMain@12+1746 (4a400000,1,3c5454)
@[28] 0x0040226f nfixup.NotesMain@8+47 (4,400000)
@[29] 0x004023a4 nfixup.notes_main+212 (0,0,0,4)
@[30] 0x00402296 nfixup.main+22 (4,3d2b90,3d2bf8,94)
@[31] 0x0040272d nfixup.mainCRTStartup+364 (0,0,7ffd7000,0)
[32] 0x77e6f23b kernel32.ProcessIdToSessionId+521 (4025c1,0,78746341,20)

Resolving the problem

This problem has been identified in SPR # MVEO86PMLV which has been resolved in Domino 8.5.2 and above.


Document information

More support for: IBM Domino
Crash

Software version: 8.5, 8.5.1

Operating system(s): Windows

Software edition: All Editions

Reference #: 1439208

Modified date: 13 October 2010