IBM Support

BMI Causes a Memory leak on BLK_NSF_POOL

Technote (troubleshooting)


Problem

Server crashes with various fatal threads after the customer upgrades to Domino 8 from Domino 7:

############################################################
### thread 43/74: [ nHTTP: 28dc: 1804] FATAL THREAD (Panic)
### FP=0x27e0e2b0, PC=0x7c82860c, SP=0x27e0e240
### stkbase=0x27e10000, total stksize=262144, used stksize=7616
### EAX=0x00010000, EBX=0x00000000, ECX=0x00bc00a0, EDX=0x00000000
### ESI=0x00001024, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000
Flags=0x00000297
############################################################
[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (1024,493e0,0,27e0e838)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (1024,493e0,3,27e0ea54)
@[ 3] 0x601a8cf4 nnotes.OSRunExternalScript@8+1300 (12c,1)
@[ 4] 0x601a918a nnotes.FRTerminateWindowsResources+986 (1,0,1010,1)
@[ 5] 0x601a954f nnotes.OSFaultCleanupExt@24+895
(1444dd8,1010,0,0,0,27e0ed84)
@[ 6] 0x601a95da nnotes.OSFaultCleanup@12+26 (0,1010,0)
@[ 7] 0x601b4af4 nnotes.OSNTUnhandledExceptionFilter@4+276 (27e0fdbc)
@[ 8] 0x60179c08 nnotes.Panic@4+520 (60bc0ae7)
@[ 9] 0x60179c8c nnotes.Halt@4+28 (107)
@[10] 0x60103245 nnotes.AccessAllProtected@0+85 ()
@[11] 0x6004600e nnotes.AccessAll@8+46 (1,1)
@[12] 0x60047183 nnotes.ProcessGlobalEvent@4+19 (1a12ee4)
@[13] 0x600470a1 nnotes.OSProcessShouldQuit@0+33 ()
@[14] 0x600c8d69 nnotes.OSWaitEvent@8+105 (6a93c9f,bb8)
@[15] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,6a93c83,3)
@[16] 0x1002df9a nhttpstack.HTWorkerThread::ThreadMain+106
(6a93c83,0,0,0)
@[17] 0x601047ed nnotes.ThreadWrapper@4+173 (0)
[18] 0x77e6482f kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################
### FATAL THREAD 2/5 [StLogger: 128c: 2554]
### FP=0x001448c4, PC=0x00000000, SP=0x007dffbc
### stkbase=007e0000, total stksize=69632, used stksize=68
### EAX=0x00000000, EBX=0x001448b8, ECX=0x77e62457, EDX=0x5b950001
### ESI=0x00000001, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000
Flags=0x00010246
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
[ 1] 0x0044005c StLogger


Symptom

BLK_NSF_POOL is taking an excessive amount of memory:

<@@ ------ Notes Memory -> Usage Summary -> Top 10 Memory Block Usage ->
Memhandles By Size :: (Shared) (Time 05:56:56) ------ @@>

Type TotalSize Count Typename
-----------------------------------------------------------
0x82cd 889593856 226 BLK_UBMBUFFER
0x8252 527433728 503 BLK_NSF_POOL
0x82cc 16488960 226 BLK_UBMBCB
0x834a 11010048 11 BLK_GB_CACHE
0x8a05 3300000 1 BLK_NET_SESSION_TABLE
0x8311 2097152 2 BLK_NIF_POOL
0x8a03 1592758 379 BLK_NETBUFFER
0x93c5 1455680 259 BLK_NSF_MODIFIED_NOTE_LOG
0x938d 1245432 3 BLK_SHARED_UNK_HASH
0x93ba 1141524 524 BLK_VA_PNOVEC


Resolving the problem

Debug trapleaks were implemented on the server with the following debugging parameters:


debug_threadid=1
Debug_Trapleaks=0252
Debug_Trapleaks_ShowStack=1
DEBUG_SHOWLEAKS=1
DEBUG_DUMP_FULL_HANDLE_TABLE=1

Then, periodic clean shutdowns were taken after manual NSDs.

From the new console logs, we were able to capture the leaking memory blocks:

[1BA0:0002-1ED0] Shared memory blocks left allocated:
[1BA0:0002-1ED0] Leaked block: PID=0x1BA0 Handle=0xF0241C9F Type=0x0252(PKG_0x2+82) Length=1048582 Addr=110201BC
[1BA0:0002-1ED0] NOTE: This is a shared block that 'may' get freed
later by another process.
[1BA0:0002-1ED0] When the last Notes process terminates, this
block will be reported again if it is still allocated.
[1BA0:0002-1ED0] Allocated at 10/21/2009 5:05:06 by PID 0x1BA0 - thread
0x2
[1BA0:0002-1ED0] [01]: 60006F4E [load addr 0x12240000] nnotes!
OSMemoryAllocate +0x13e
[1BA0:0002-1ED0] [02]: 60637E38 [load addr 0x12240000] nnotes!
OSAllocBBlockReal +0x288
[1BA0:0002-1ED0] [03]: 60007338 [load addr 0x12240000] nnotes!
OSAllocBBlock +0x18
[1BA0:0002-1ED0] [04]: 6003CE0F [load addr 0x12240000] nnotes!
BUFAllocDBCONT +0x1f
[1BA0:0002-1ED0] [05]: 608D6E68 [load addr 0x12240000] nnotes!
BUFUseCont +0x58
[1BA0:0002-1ED0] [06]: 608C0FF7 [load addr 0x12240000] nnotes!
iNSFBufOpenStorageExt +0x357
[1BA0:0002-1ED0] [07]: 60092231 [load addr 0x12240000] nnotes!
UNIDIndexOpen +0xa1
[1BA0:0002-1ED0] [08]: 608E2F21 [load addr 0x12240000] nnotes!
OpenUNIDIndex +0x181
[1BA0:0002-1ED0] [09]: 600934E8 [load addr 0x12240000] nnotes! DbLoad
+0x1038
[1BA0:0002-1ED0] [10]: 60813D15 [load addr 0x12240000] nnotes!
NSFDbOpenExtended4 +0x49c5

[1BA0:0002-1CD4] Shared memory blocks left allocated:
[1BA0:0002-1CD4] Leaked block: PID=0x1BA0 Handle=0xF0241C9F
Type=0x0252(PKG_0x2+82) Length=1048582 Addr=110201BC
[1BA0:0002-1CD4] NOTE: This is a shared block that 'may' get freed
later by another process.
[1BA0:0002-1CD4] When the last Notes process terminates, this
block will be reported again if it is still allocated.

As you see, the leaked block has a PID of 1BA0. When I matched it to the relevant NSD, I see that it matches to the following fatal stack:

############################################################
### FATAL THREAD 1/3 [bmiimport: 1ba0: 1cd4]
### FP=0x0012ec04, PC=0x77e4bef7, SP=0x0012ebb0
### stkbase=00130000, total stksize=24576, used stksize=5200
### EAX=0x0012ebb4, EBX=0x00000000, ECX=0x00000000, EDX=0x00000000
### ESI=0x0012ec7c, EDI=0x00000000, CS=0x0012001b, SS=0x00000023
### DS=0x7c820023, ES=0x00120023, FS=0x0012003b, GS=0x00000000
Flags=0x00000202
Exception code: c06d007e ()
############################################################
[ 1] 0x77e4bef7 kernel32.RaiseException+60 (c06d007e,0,1,12ec78)
@[ 2] 0x60b90bc6 nnotes.__delayLoadHelper2@8+321 (0,12ec28)
@[ 3] 0x60b90cdf nnotes._tailMerge_nsdhelp+13 (60006f4e,12eccc)
@[ 4] 0x601a3554 nnotes.OSDumpStackTraceExt@20+164
(ffffffff,12f2b8,0,12,0)
@[ 5] 0x6018d3c1 nnotes.dump_blocktraceblock@12+1953
(12f450,2a920c8,2a90000)
@[ 6] 0x60656b2f nnotes.traverse@8+175 (12f414,2037801)
@[ 7] 0x60656bb3 nnotes.traverse@8+307 (12f414,202dc01)
@[ 8] 0x60656bb3 nnotes.traverse@8+307 (12f414,2010801)
@[ 9] 0x60656c94 nnotes.IndexTraverse@24+84
(60eba35c,290e,601900f8,0,0,0)
@[10] 0x6019fad6 nnotes.DumpLeaksInt@24+678 (2,0,0,0,0,0)
@[11] 0x601a027f nnotes.OSTermBlockTraceProcess@4+111 (0)
@[12] 0x60178fff nnotes.OSTerm@0+1439 ()
@[13] 0x632f613e nlsxbe.NotesOS::unload+110 ()
@[14] 0x63324d76 nlsxbe.ComViewEntryCollection::Release+22 ()
[15] 0x73675e26 MSVBVM60.__vbaGosubFree+8395 ()
[16] 0x004141a8 bmiimport ()
[17] 0x73579f64 MSVBVM60.EbLoadRunTime+4454 ()
[18] 0x73577a5a MSVBVM60.BASIC_CLASS_QueryInterface+3786 ()
[19] 0x73573959 MSVBVM60.ThunRTMain+989 ()
[20] 0x735736d2 MSVBVM60.ThunRTMain+342 ()
[21] 0x735735d8 MSVBVM60.ThunRTMain+92 ()
[22] 0x00403fb2 bmiimport ()

This confirms that the third party process, BMIIMPORT, is causing a memory leak on Lotus Domino.

BMI is a third party application that adds documents to domino databases. This explains why the BLK_NSF_POOL was filling up.

There is a compatibility issue between BMI and Domino 8 where BMI is causing BLK_NSF_POOL to leak memory, which causes Domino to crash.

If the customer is experiencing this problem, please have the customer contact BMI support for further support.



Document information

More support for: IBM Domino
Crash

Software version: 8.0.2

Operating system(s): Windows

Software edition: All Editions

Reference #: 1408593

Modified date: 15 April 2010