Long held locks on Domino Directory causing performance issues leading to server hang

Technote (troubleshooting)


Problem

Long held locks on the Lotus Domino Directory is causing performance issues eventually leading to a server hang.

Symptom

Users experience slow response from the server when there are long held locks on names.nsf, usually for between 40-50 seconds every 5-10 minutes.


Cause

NAMELookupCacheCacheSize gets exhausted.

Environment

Domino 852FP1; Windows 2003

Diagnosing the problem

NSD taken at the time of the issue indicates lookup calls into NAMELookupDiskLookup. In this particular case, there were up to 53 threads waiting on names.nsf.

############################################################
### thread 12/107: [ nSERVER: 0748: 1450]
### FP=0x0d47d978, PC=0x7c8285ec, SP=0x0d47d908
### stkbase=0x0d480000, total stksize=262144, used stksize=9976
############################################################
[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (1204,7530,0,d47d9bc)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18(1204,7530,1322d25,1322d23)
@[ 3] 0x600cf5d6 nnotes.WaitOnNativeSemaphore@16+150(162,7530,0,1324a9a)
@[ 4] 0x60128eb6 nnotes.WaitOnNativeSemaphoreCounted@12+22(1320162,1322d25,1324a9a) @[ 5] 0x60003d53 nnotes.OSLockReadFRWSemInt@12+355 (1320162,1,1324a9a)
@[ 6] 0x60004c02 nnotes.OSLockReadFRWSemWithInfo@8+18(1322d23,1324a9a)
@[ 7] 0x609b2a85 nnotes.LockDbReadSemCtxTimedInt@12+1669 (0,ffffffff,60feddc0)
@[ 8] 0x609b3708 nnotes.LockDbReadSemCtxInt@12+24 (d47dedc,1,60feddc0)
@[ 9] 0x609b42cd nnotes.LockDbROCondInt@16+749 (0,6d,1,60feddc0)
@[10] 0x609b53e6 nnotes.LockDbROInt@12+22 (d47dedc,6d,60feddc0)
@[11] 0x6002f2f4 nnotes.NSFDbPurgeInfoGet@12+36 (b6,d47df40,0)
@[12] 0x60033c4e nnotes.GetDatabaseModifiedTime@8+318(3a49098,d47df78)
@[13] 0x60caa1f4 nnotes.NIFFindByKeyExtended3@44+260 (b7,d47e080,6002,1009001,0,d47df94,f10f10,ffffffff,0,0,0) @[14] 0x6006caf0 nnotes.NIFFindByKeyExtended2@36+48 (b7,d47e080,6002,1008001,0,d47dfc8,f10f10,ffffffff,0)
@[15] 0x60c8a05c nnotes.LookupNameWithRead@28+236 (17b00b7,d47e93c,0,1008001,0,d47e2a0,f10f10) @[16] 0x600c6a93 nnotes.ReturnNameInfo@16+1891 (b7,0,d47e8f4,d47e5b4)
@[17] 0x600a616c nnotes.NAMELookupDiskLookup@20+2204(a,d47e8f4,d47ed5c,d47ed60,0)
@[18] 0x600cd0de nnotes.Lookup@32+2750(8206,d470001,60f1d268,61070001,0,d47ed68,f10f10,ffffffff)
@[19] 0x60c6600f nnotes.NAMESwitchToNLThread@4+191 (d47edd0)
@[20] 0x60c82737 nnotes.ThreadedNAMELookup2@52+167(0,8206,0,0,0,d47ef1c,f10f10,ffffffff,0,0,0,0,0)
@[21] 0x600d5b97 nnotes.NAMELookup2@36+279(0,8006,1,60f1d268,0,d47ef74,f10f10,ffffffff,0)
@[22] 0x6041ceb3 nnotes.SECLookup2@40+83 (d47f316,0,6,1,0,d47efac,f10f10,ffffffff,0,0)
@[23] 0x6041cf95 nnotes.SecSearchForEntry@12+117 (d47f30c,1,2)
@[24] 0x6041d617 nnotes.SecSearchAndCheckEntry@12+23 (d47f30c,1,2)
@[25] 0x6041d6d4 nnotes.SecSearchDirectories@8+68 (d47f30c,1)
@[26] 0x6041d7cd nnotes.SecSearchNameSpaces@4+141 (1)
@[27] 0x6041d9f5 nnotes.FindAddressBookEntry@48+421(0,d470003,d47f488,d47f438,0,d47f3a8,f10f10,ffffffff,0,0,0,0)
@[28] 0x6041dc53 nnotes.SECFindAddressBookEntry@44+131(0,1,d47f488,d47f438,0,d47f3f8,f10f10,ffffffff,0,0,0)
@[29] 0x60d0d57e nnotes.LookupNameInfo@120+254 (0,3,d47f8d8,23)
@[30] 0x60d0df3b nnotes.LookupABPIandKeys@112+299 (3,d47f8d8,23,53d40884)
@[31] 0x60d0e389 nnotes.FindABPIandKeys@112+201 (d47f8d8,23,53d40884,86)
@[32] 0x60d0e4cd nnotes.CheckABPublicInfo@112+301(0,395cd,d47fa14,d47f8d8)
@[33] 0x601983c6 nnotes.LookupABPI@8+646 (fb80,0)
@[34] 0x60197dd1 nnotes.Parse_DistName@8+241 (d47fb80,1)
@[35] 0x60d24b4f nnotes.AuthServerDialog@12+2207 (d47fb80,1,54c80000)
@[36] 0x60104506 nnotes.AuthStateMachine@4+342 (d47fb80)
@[37] 0x60d00b17 nnotes.AUTHProcessNetbfr@16+199 (dea0023b,10025f50,d47fe58,d47fc10)
@[38] 0x100218ba nserverl.DbServer@8+1226 (f941063f,4cc0009d)
@[39] 0x100373d7 nserverl.WorkThreadTask@8+1655 (c661b6c,0)
@[40] 0x10001a2e nserverl.Scheduler@4+750 (0)
@[41] 0x6015586f nnotes.ThreadWrapper@4+175 (0)
[42] 0x77e6482f kernel32.GetModuleHandleA+223 (0,0,0,0)

d:\notesdata\names.nsf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x80256369:0x004c8ef2
bContQueue = NSFPool [ 00000105]
Offline = No
DeleteInProgress = No
FDGHandle = 0xf023d748, RefCnt = 348, Dirty = N
DB Sem = (FRWSEM:0x0244) state=1, waiters=53, refcnt=0, nlrdrs=1
Writer=[ : 0000]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ :
0000], n=0, wcnt=-1, Users=-1, Owner=[ : 0000]

Console log will show many long held locks on the names.nsf:

RmFlush: Error opening 'd:\notesdata\names.nsf': The caller's SemWait timeout expired. [158C:00AA-14A4] LkMgr BEGIN Long Held Lock Dump ------------------
[158C:00AA-14A4] Lock(Mode=S * LockID(DB DB=d:\notesdata\names.nsf))
Waiters countNonIntentLocks = 2 countIntentLocks = 0, queuLength = 51
[158C:00AA-14A4] Req(Status=Granted Mode=S Class=Manual Nest=0 Cnt=2
Tran=0 Func=N/A [158C:011C-1680])
[158C:00AA-14A4] Req(Status=Waiting Mode=SIX Class=Manual Nest=0Cnt=1
Tran=0 Func=N/A [179C:0002-16B4] Delay=1min)


Resolving the problem



* Increase the Namelookup cache to 128 MB using the following Notes.ini parameter:

NLCache_Size=134217728. ** Note default size in 8.5.3 64 Bit is 64 MB prior to this the default size was 16 MB.

* Make sure the ODS of the names,nsf is 51. In one case, the ODS was 43 and NLcache was set to 64MB

* Additional Info: Nlcache_size can be increased above 256 MB for 64 bit domino but should be increased with precaution and after consulting Lotus support.

* If you are concerned with this statistic you should review the Domino Server stats.

Issue the following command via the Server Console:

Show Stat database.*

Review these results.

Database.NAMELookupCacheMaxSize
Database.NAMELookupCachePool.Used

If used is near max size you may need to increase the pool accordingly.



Rate this page:

(0 users)Average rating

Document information


More support for:

IBM Domino
Performance / Hang

Software version:

8.5.2, 8.5.2.1

Operating system(s):

Windows

Software edition:

All Editions

Reference #:

1470902

Modified date:

2013-03-13

Translate my page

Machine Translation

Content navigation