Server hang with semaphore deadlock between the between a database and collection semaphore when running 3rd party application

Technote (troubleshooting)


Problem

The Domino server hangs due to a semaphore deadlock.

Symptom

Semaphore deadlock condition


Cause

The 3rd party application's extension handle of MBEServer4051 is calling to open a collections
while being used by AddFolderOp causing the dead lock if an update thread, held collection sem, tried to update the collection will be waiting for the database sem. It should not try to open a collection
through extension manager while holding on to database semaphore


Diagnosing the problem

sq="000131E6" THREAD [0BB4:0002-0BB8] WAITING FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@087EC2E3) (E:\ProgramFiles\Lotus\Domino\data\mail\xxxx.nsf) (R=0,W=3,WRITER=0DF0:0690,1STREADER=0000:0000) FOR 30000 ms


sq="000131EA" THREAD [0DF0:001A-0690] WAITING FOR READ LOCK ON FRWSEM 0x030B Collection semaphore (@28336166) (R=0,W=1,WRITER=0BB4:0BB8,1STREADER=0000:0000) FOR 30000 ms

Owner of the database sem & waiter for the collection sem

############################################################
### thread 21/45: [ nrouter: 0df0: 0690]
### FP=0x2b85d06c, PC=0x7c82845c, SP=0x2b85cffc
### stkbase=0x2b860000, total stksize=262144, used stksize=12292
############################################################
[ 1] 0x7c82845c ntdll.KiFastSystemCallRet+0 (798,7530,0,2b85d0b0)
[ 2] 0x77e61c75 kernel32.WaitForSingleObject+18(798,7530,28336168,28336166)
@[ 3] 0x600c5413 nnotes.WaitOnNativeSemaphore@16+659 (213,7530,0,0)
@[ 4] 0x60118a06 nnotes.WaitOnNativeSemaphoreCounted@12+22(28330213,28336168,0)
@[ 5] 0x60003d03 nnotes.OSLockReadFRWSemInt@12+355 (28330213,1,0)
@[ 6] 0x60003f00 nnotes.OSLockReadFRWSem@4+16 (28336166)
@[ 7] 0x60032916 nnotes.LockCollectionRead@4+726 (2970000)
@[ 8] 0x6006db65 nnotes.OpenCollection@32+2421(2970000,2,0,0,0,2b85d768,f10f10,ffffffff)
@[ 9] 0x60c299a6 nnotes.NIFOpenCollectionExtended5@64+3014 (3cd,3cd,a0c26,2,0,2b85d964,f10f10,ffffffff,0,0,0,0,0,0,0,0)
@[10] 0x60c2a129 nnotes.NIFOpenCollectionExtended4@60+73
(547,547,a0c26,2,0,2b85d9ac,f10f10,ffffffff,0,0,0,0,0,0,0)
@[11] 0x6006d1e2 nnotes.NIFOpenCollectionExtended3@56+66
(547,547,a0c26,2,0,2b85d9f0,f10f10,ffffffff,0,0,0,0,0,0)
@[12] 0x6006d19c nnotes.NIFOpenCollectionExtended2@48+60 (547,547,a0c26,2,0,2b85da30,f10f10,ffffffff,0,0,0,0)
@[13] 0x6007b714 nnotes.NIFOpenCollection@40+52(547,547,a0c26,2,0,2b85da68,f10f10,ffffffff,0,0)
[14] 0x29d6499f MBEServer4051.EMHandlerProc+6239(29dbda98,29dbdce4,29dbdbc2,2b85daf0)
[15] 0x29d66073 MBEServer4051.EMHandlerProc+12083(2b85dd40,1,547,60230a70)
[16] 0x29d6379a MBEServer4051.EMHandlerProc+1626(29d89ff8,59e,2b85e078,c3da6)
[17] 0x29d63497 MBEServer4051.EMHandlerProc+855(2b85e140,0,6e800018,2b85e67c)
@[18] 0x60915e70 nnotes.EMDispatch@4+288 (2d5)
@[19] 0x6000b214 nnotes.EMBefore+84 (d1,547,547,59e)
@[20] 0x608d4226 nnotes.AddFolderOp@52+342 (2b85e67c,2b85e67c,6e800018,1,0,2b85e1fc,f10f10,ffffffff,0,0,0,0,0)
@[21] 0x608daa77 nnotes.DbUpdateFolder@24+2311
(1e67c,2b85e67c,59e,200005c7,0,2b85e344)
@[22] 0x60b1f3bd nnotes.FoldWrapUpdateFolder@24+541(0,2b85e67c,59e,200005c7,0,2b85e388)
@[23] 0x608ccf29 nnotes.AddNoteToFolder@16+297 (2b85e67c,a7,59e,20)
@[24] 0x60949415 nnotes.DispatchNoteUpdate@36+2149 (2b85e67c,0,100c,104,0,2b85e4b8,f10f10,ffffffff,0)
@[25] 0x60949b47 nnotes.NSFNoteUpdateExtended3@28+1431(a7,100c,104,59e,0,2b85e6e8,f10f10)
@[26] 0x60949fd2 nnotes.NSFNoteUpdateExtended2@24+34(a7,100c,104,59e,0,2b85e70c)
@[27] 0x60b08587 nnotes.xNSFNoteUpdateExtended2@24+87(547,100c,104,59e,0,2b85e764)
@[28] 0x608caf1e nnotes.DbNoteUpdateAndAddToFolders@28+302 (a7,2b85e864,2b85e830,59e,0,2b85e794,f10f10)
@[29] 0x60b1fb89 nnotes.FoldWrapNoteUpdateAndAddToFolders@28+361
(a7,0,2b85e830,59e,0,2b85e7d4,f10f10)
@[30] 0x6099c201 nnotes.iNSFNoteUpdateAndAddToFolders@28+305
(a7,2b85e864,2b85e830,59e,0,2b85e800,f10f10)
@[31] 0x6096b6e3 nnotes.NSFNoteUpdateAndAddToFolders@28+355 (a7,a0,547,59e,0,2b85e8a4,f10f10)
@[32] 0x60d36f89 nnotes.MailDeliverMessage@88+3257
(3b48f9c,a7,4a53fdb,8b88e2e,0,2b85ed60,f10f10,ffffffff,0,0,0,0,0,0,0,0,0,0,0,0,0,0)
@[33] 0x0041d565 router.AttemptMessageDelivery@40+1573(3b48f9c,0,547,0,0,2b85fcf4,f10f10,ffffffff,0,0)
@[34] 0x0041eb2f nrouter.DeliverToDestination@28+2815(47f3a0,0,4a53fdb,2b85ff50,0,2b85fec4,f10f10)
@[35] 0x0041f105 nrouter.Transfer@28+725(47f3a0,21700001,1,0,0,2b85ff20,f10f10)
@[36] 0x0041f44d nrouter.TransferThread@4+573 (2970000)
@[37] 0x601411cf nnotes.ThreadWrapper@4+175 (0)
[38] 0x77e6481f kernel32.GetModuleHandleA+223 (60141120,0,0,8000a)

Owner of the collection sem & waiter on the database sem

############################################################
### thread 1/4: [ nUpdate: 0bb4: 0bb8]
### FP=0x0013e0ec, PC=0x7c82845c, SP=0x0013e07c
### stkbase=0x00140000, total stksize=77824, used stksize=8068
############################################################
[ 1] 0x7c82845c ntdll.KiFastSystemCallRet+0 (6bc,7530,0,13e130)
[ 2] 0x77e61c75 kernel32.WaitForSingleObject+18(6bc,7530,87ec2e5,87ec2e3)
@[ 3] 0x600c5413 nnotes.WaitOnNativeSemaphore@16+659(210,7530,0,87ee70d)
@[ 4] 0x60118a06 nnotes.WaitOnNativeSemaphoreCounted@12+22(210,87ec2e5,87ee70d)
@[ 5] 0x60009e65 nnotes.OSLockWriteFRWSemInt@12+405 (0,210,87ee70d)
@[ 6] 0x600259f2 nnotes.OSLockWriteFRWSemWithInfo@8+18 (87ec2e3,87ee70d)
@[ 7] 0x6099cf9f nnotes.LockDbWriteSemCtxTimedInt@12+1231(87ec0f8,ffffffff,6118e4f0)
@[ 8] 0x6099e388 nnotes.LockDbWriteSemCtxInt@12+24 (13e488,1,6118e4f0)
@[ 9] 0x6099e7d2 nnotes.LockDbRWCondInt@16+738 (ffffffff,36,1,6118e4f0)
@[10] 0x6099f3a6 nnotes.LockDbRWInt@12+22 (13e488,36,6118e4f0)
@[11] 0x6096e5bc nnotes.NSFDbWriteObjectPExt@24+364(4eb,a0c2a,2a750884,0,0,13e4c0)
@[12] 0x60c8acdb nnotes.WriteBuffer@20+219 (0,29966148,2a750884,3aeb,0)
@[13] 0x60091cc6 nnotes.WriteCollection@4+6822 (29965148)
@[14] 0x6009fc55 nnotes.NIFUpdateCollectionNext@8+3333 (29965148,0)
@[15] 0x0040762b nUpdate.UpdateViewCollection@20+1099(4eb,13f66c,a0c26,c000003,0)
@[16] 0x00407a9a nUpdate.DesignEnumProc@28+410(13f390,4eb,a0c26,13ef58,0,13eca4,f10f10)
@[17] 0x600de671 nnotes.DesignEnum2@32+609(4eb,8,0,40,0,13efac,f10f10,ffffffff)
@[18] 0x0040822b nUpdate.UpdateCollections@44+1883(13f66c,c000003,0,0,0,13f3c8,f10f10,ffffffff,0,0,0)
@[19] 0x00405069 nUpdate.PerformRequest@20+409 (0,c000003,0,0,0)
@[20] 0x0040686c nUpdate.Update+2748 (3e6c74,505c3a45,72676f72,46206d61)
@[21] 0x00406f56 nUpdate.AddInMain@12+406 (400000,1,3e6c74)
@[22] 0x0040c8ff nUpdate.NotesMain@8+47 (1,400000)
@[23] 0x0040ca34 nUpdate.notes_main+212 (0,0,0,1)
@[24] 0x0040c926 nUpdate.main+22 (1,3d6c20,3d29d8,417000)
@[25] 0x0040d225 nUpdate.mainCRTStartup+323 (0,0,7ffdf000,0)
[26] 0x77e6f1eb kernel32.ProcessIdToSessionId+521(40d0e2,0,78746341,20
[26] 0x77e6f1eb kernel32.ProcessIdToSessionId+521(40d0e2,0,78746341,20)

Resolving the problem

The deadlock has been documented by that was raised by my colleague for the same issue. APAR LO75445 / SPR# JPMS97WKLY - Deadlock between a db sem and collection sem, which is still Open.


In this case, the third party application was a product called Spam Sentinel. Resolving the problem involved two parts:

1. Enable transaction logging on the server so that lock manager will be used to control database access rather than a database semaphore.

2. Spam Sentinel recommended a patch update. Customer will need to contact the 3rd party vendor to obtain the patch information.

http://www.maysoft.com/downloads.nsf/title/SpamSentinelServer

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

IBM Domino

Software version:

8.5.3, 9.0

Operating system(s):

Windows

Reference #:

1643114

Modified date:

2013-07-12

Translate my page

Machine Translation

Content navigation