IBM Support

Domino server went in to hang state as Deadlock detected between LkMgr locker and semaphore locker(Directory manager queue semaphore).

Technote (troubleshooting)


Problem

Domino server went in to hang state as Deadlock detected between LkMgr locker and semaphore locker(Directory manager queue semaphore)

Environment

Domino 8.5.3 FP1 running on Solaris

Resolving the problem

Deadlock detected between LkMgr locker and semaphore locker(Directory manager queue

semaphore).


06/09/2012 11:06:22 AM ZE10 sq="000740C6" THREAD [00851:01853-00041] WAITING FOR WRITE LOCK ON RWSEM 0x0294 Directory manager queue semaphore (@FAA6B3D4) (R=0,W=1,WRITER=11937:00001,OWNER=11937:00001) FOR 30000 ms


[00851:00169-00159] Lock(Mode=SIX* LockID(DB DB=/notes/data/log.nsf)) Waiters countNonIntentLocks = 1 countIntentLocks = 0, queuLength = 2
[00851:00169-00159] Req(Status=Granted Mode=SIX Class=Manual Nest=-1 Cnt=1
[00851:00169-00159] Tran=0 Func=N/A dbbktbuf.c:1666 [00851:01853-00041])
..........................................................................................................................................................
...............................................................................................................................................




[00851:00169-00159] Req(Status=Waiting Mode=S Class=Manual Nest=-1 Cnt=0
[00851:00169-00159] Tran=0 Func=N/A dbopen.c:4073 [11937:00002-00001] Delay=13min)


LkMgr Lock 00851:01853-00041(/notes/data/log.nsf) waiting on Sem Lock 11937:1(0x0294)

##################################
##### thread 40/171 :: server, pid=851, lwp=41, tid=41 ######
##################################
[1] ff24e79c lwp_cond_wait (fac84550, fac84538, f2d5cdb0, 1)
[2] ff244c1c cond_wait_kernel (fac84550, fac84538, f2d5cdb0, 0, 0, 0) + 4c
[3] ff244edc cond_wait_common (fac84550, fac84538, f2d5cdb0, 1, 0, 1) + 27c
[4] ff245088 _cond_timedwait (fac84550, fac84538, f2d5cf28, 353, ff000000, 0) + 34
[5] ff24517c cond_timedwait (fac84550, fac84538, f2d5cf28, 1c00, fab2f01c, 0) + 14
[6] ff2451bc pthread_cond_timedwait (fac84550, fac84538, f2d5cf28, 1, 1, ff2c7940) + c
[7] fc21fccc WaitOnNativeSemaphore (3b, faa6b3d8, fac84550, 1, fac84538, 0) + 62c
[8] fc21ee80 OSLockSemInt (1, 4400, faa6b3d8, d400, ff02b20c, 474c) + 700
[9] fc21e198 OSLockWriteSem (faa6b3d4, 1, fe9a43b0, fe9909cc, 139e4, 29) + 78
[10] fce50c78 DbDirUpdateCacheNoteUpdated (f2d61220, 0, faa6a51c, 9d1e0e10, 1, 26703c) + 178
[11] fcaac900 NoteUpdateImpl (f2d61220, feb50d9c, 1, 1, a51f8394, f2d5e578) + 1e060
[12] fca8e6e0 iNoteUpdate2 (f2d61220, 26703c, 105, 0, 0, 0) + 460
[13] fcb8a77c DispatchNoteUpdate (f2d61220, 26703c, 105, 4c00, 0, 0) + 8dc
[14] fcb894f0 NSFNoteUpdateExtended3 (0, 105, 0, 0, 1, feb50d9c) + ad0
[15] fcb88584 NSFNoteUpdate (3b9c, 105, f2d61466, 1c0c90, 0, 0) + 24
[16] fd415f74 LogCompleteEntry (74939c, 4178, fe9909cc, ff0cadbc, 3b9c, 0) + 1b4
[17] 00070150 DbServerTerm (f2d61621, f2d61ad4, f2d61950, a5d4da74, 54800, fc00) + 1490
[18] 0003c184 DeleteTask (f2d61b7c, 1, 54820, a448ce18, 1999, 22fb74) + 104
[19] 00099468 WorkThreadTask (a64a3f50, f3e6ab60, 1db354, 1d, a02, 26703c) + c88
[20] 00034ad0 Scheduler (26703c, 1db354, 1e82cc, 26703c, ffff, fc00) + 430
[21] fc2507fc ThreadWrapper (0, f2d62000, 0, 287c, 2800, 2d006c) + fc
[22] ff24a9c8 _lwp_start (0, 0, 0, 0, 0, 0)


Sem Lock 11937:1(0x0294) waiting on LkLock 00851:01853-00041(/notes/data/log.nsf) Delay = 13 minutes

##################################
##### thread 1/2 :: smddbs, pid=11937, lwp=1, tid=1 ######
##################################
[1] fbe4e79c lwp_cond_wait (fac8a350, fac8a338, 0, 1)
[2] fbe44c1c cond_wait_kernel (fac8a350, fac8a338, 0, 0, 0, 0) + 4c
[3] fbe44edc cond_wait_common (fac8a350, fac8a338, 0, 1, 0, 1) + 27c
[4] fbe44ff8 cond_wait (fac8a350, fac8a338, 0, 1c00, ff000000, 0) + 10
[5] fbe45034 pthread_cond_wait (fac8a350, fac8a338, 1, 1, 1, fbec7940) + 8
[6] fc225c30 WaitForThreadSem (447, fac8a350, 0, fac8a338, fe9909cc, 67c00) + 290
[7] fc21f7c4 WaitOnNativeSemaphore (f800, d4c3555a, fe9a43b8, 111c, 0, fc00) + 124
[8] fc223018 OSLockWriteFRWSemInt (1, 228130, 1, fabb0d3c, 228000, 1) + 958
[9] fc21d78c OSWaitFairEvent (d4c35554, 0, c, d4c354c4, 0, 53) + c
[10] fcf17128 __1cJnewLkLock6FpnQDBCONTEXT_STRUCT_pnMLK_ID_STRUCT_LLpnHLK_INFO_pc_H_ (4, f7bac, 20, ffbecdbc, feb50d9c, ffbecef4) + 13c8
[11] fcf12544 _LkLock (28808001, ffbed408, 4022, ffffffff, ffbed428, fef73660) + 2c4
[12] fc8eee58 LockDbReadSemCtxTimedInt (ffbf04b8, ffffffff, f7bac, 0, 0, fef41a34) + 858
[13] fcb99394 NSFDbOpenExtended6 (0, 0, 0, 5e2c00, 0, 1000) + a734
[14] fcb8ec24 NSFDbOpenExtended5 (ffbf1034, 400200, 0, 0, 0, 0) + 44
[15] fcb8e8cc NSFDbOpenExtended3 (ffbf1034, 400200, 0, 0, ffbf1136, 0) + 6c
[16] fcbb08a4 NSFDBGAccess (f8c02aee, 0, faaadf28, 500, 400200, c0000000) + 4a4
[17] fd454de8 NetInit (1ec6, f8c00c28, f8c00c28, fec9dcf0, ff0cadbc, fec4ea20) + 1408
[18] fd419b68 LogAddTextVCtx (1, c04, fefbe9a8, 0, 0, ffbf2540) + a8
[19] fd417360 LogEventTextExt (c00, ffbf2540, fefbe9a8, c04, 0, 0) + 60
[20] fce5a304 DbDirManRefreshCache (faa6a51c, 9e517230, ffbf2540, 288c, 2888, ffbf2478) + 804
[21] fce59ab8 DbDirManRefreshCheck (0, faa6a51c, faa6a51c, fe9909cc, 1c03d0, feb50d9c) + b8
[22] fcdd3344 DBGManagerInitExt (0, 2fd, 0, 0, 1c0000, fe9909cc) + 144
[23] fcdd3198 DBGManagerInit (0, faa6a51c, 1400, feb50d9c, fe9909cc, feb5d774) + 2b8
[24] fc978118 InitializeNSFExt (1c0000, 1, 7c5000, fe9909cc, 1c0000
, 1c03d0) + 1cd8
[25] fc976404 InitializeNSF (fc1fda1c, 364e8, 372a8, fc1fda00, 1, 2794468) + 4
[26] fc95c498 NSFDbSetMode (0, e, 0, 0, 139bc, e0000) + 18
[27] fc1fda1c AddInSDKInit (3, 7158c, fa3bc, 1e, fe9909cc, 2794ba8) + 1c
[28] 000372a8 NotesMain (3, 7158c, 139bc, 0, 0, 13800) + 8
[29] 0003720c notes_main (0, 0, 0, 3, ffbf40c4, 0) + 104
[30] 00036c14 _start (0, 0, 0, 0, 0, 0) + 108

This issue was reported to Quality Engineering as SPR # JPAI94HR3N & fixed in Domino 9.0.

If you suspect you are experiencing this issue, collect a NSD and contact IBM Technical Support to request hotfix containing the fix for this issue.


Document information

More support for: IBM Domino
Performance / Hang

Software version: 8.5.3.1

Operating system(s): Solaris

Reference #: 1644240

Modified date: 23 July 2013