Skip to main content

Software  >  Lotus  >  

Domino Server 8.5.x group cache performance issue based on workload
 Technote (troubleshooting)
 
 
Problem
You may experience performance issues (not hangs) on Domino Release 8.5.x. The performance may appear to be degrading based on workload. This issue is most likely exposed on servers that make use of user authentication for example like mail servers or web servers. The issue is related to the Domino group cache code for authentication.
 
Symptom

The NSD logs show a number of call function stacks on different Domino tasks like these below.

############################################################
### thread 24/348: [ nSERVER: 0bb0: 1418]
### FP=0x14e99eb8, PC=0x77ef0a3a, SP=0x14e99eb8
### stkbase=0x14ea0000, total stksize=4194304, used stksize=24904
############################################################
[ 1] 0x77ef0a3a ntdll.ZwWaitForSingleObject+10 (0,0,0,0)
[ 2] 0x77d704ff kernel32.WaitForSingleObjectEx+223 (a390,1b46fa8,0,329)
@[ 3] 0x0046bf88 nnotes.WaitOnNativeSemaphoreCounted+100 (9009001,1d980001,0,15dc26b)
@[ 4] 0x0046d107 nnotes.OSLockReadFRWSem+915 (e006,27009,14e9a100,0)
@[ 5] 0x01e26587 nnotes.LockCollectionRead+131 (14e9000e,27009,14e9a310,e002)
@[ 6] 0x01e7dc40 nnotes.NIFFindByKeyExtended2+468 (BB00000C30A,F687475000008F3,0,f8)
@[ 7] 0x01e4755b nnotes.LookupNameWithRead+331 (0,0,0,44fbfa)
@[ 8] 0x01e43889 nnotes.ReturnNameInfo+1269 (bc408428,2980,0,46b215)
@[ 9] 0x01e3ba68 nnotes.Lookup+10180 (3d000b9,0,824,1)
@[10] 0x01e2abb4 nnotes.NAMELookup2+1576 (14e9bdc9,0,0,ffffffff)
@[11] 0x017efa3f nnotes.AddGroupsContainingNameExtended+355 (489,19a8312,14e9d858,46b4009)
@[12] 0x017f2559 nnotes.UAFGetAccessGroupsExtend2+3629 (1,2eb,76727265,46a6e0)
@[13] 0x01deb522 nnotes.NAMESwitchToNLThread+378 (ffffffff,125763F00000001,14e9df64,1)
@[14] 0x017f84a9 nnotes.UAFGetAccessGroupsExtend+737 (35d80117,35d80117,1,185f580)
@[15] 0x100172f8 nserverl.ServerGetNameList+488 (129000ba,35d80117,14e9fb70,46b796)
@[16] 0x10041c3a nserverl.DbServer+2222 (ffd4075c,129000ba,14e9fd1c,31)
@[17] 0x100614d0 nserverl.WorkThreadTask+2344 (ffb20012,0,0,3)
@[18] 0x10001af1 nserverl.Scheduler+969 (0,0,0,0)
@[19] 0x0044e7d2 nnotes.ThreadWrapper+330 (0,0,0,14e9ffa8)
[20] 0x77d6b71a kernel32.BaseThreadStart+58 (0,0,0,0)


############################################################
### thread 40/197: [ nSERVER: 2694: 0ee4]
### FP=0x79df8658, PC=0x77ef0a3a, SP=0x79df8658
### stkbase=0x79e00000, total stksize=4194304, used stksize=31144
############################################################
[ 1] 0x77ef0a3a ntdll.ZwWaitForSingleObject+10 (0,46a6e0,0,46b215)
[ 2] 0x77d704ff kernel32.WaitForSingleObjectEx+223 (648c,4b87a4,0,2cd)
@[ 3] 0x0046ac18 nnotes.WaitOnNativeSemaphore+148 (0,4b85ae,5a16a3a0,0)
@[ 4] 0x0046b842 nnotes.OSLockSem+242 (1,5a16a3a0,11c6,5a16a3a0)
@[ 5] 0x01c06f7a nnotes.UnlockPNOVec+50 (6a5147e0,79df8cfc,5a16a3a0,1b6e296)
@[ 6] 0x01b97fbd nnotes.STMFillPinAccessDescNSF+189 (2694,5a16a3a0,0,1c06fc8)
@[ 7] 0x018a3f56 nnotes.BUFPin+322 (5a16a3a0,16c09,16c09,1)
@[ 8] 0x01c83b36 nnotes.iNSFSlotPin+286 (ffffffff,16c09,6a550000,20011c6)
@[ 9] 0x019fa46f nnotes.NSFSlotPinCtx+331 (26E000000FA,16c09,0,20011c6)
@[10] 0x01c02848 nnotes.BTGetLongData+1132 (124000000BF,6a55a6f4,1c,20011c6)
@[11] 0x01be6bf0 nnotes.BTGetCurrentEntry+1048 (6a514450,0,0,6a514450)
@[12] 0x01897e56 nnotes.BTGetDataCtx+178 (40,2b009,100BA5524,65700004307)
@[13] 0x01eaf4e8 nnotes.GetData+444 (79df9040,2b009,656,19a7ccc)
@[14] 0x01ea90c1 nnotes.CollectionNavigate+8801 (3,2c009,ffffffff,6009)
@[15] 0x01efb6c6 nnotes.FindByKey+6354 (79df000e,2b009,79df9570,e002)
@[16] 0x01e7dc72 nnotes.NIFFindByKeyExtended2+518 (26940000C30A,7eb,0,13)
@[17] 0x01e4755b nnotes.LookupNameWithRead+331 (0,44fbfa,0,44fbfa)
@[18] 0x01e43889 nnotes.ReturnNameInfo+1269 (b19d9028,4531,0,46b215)
@[19] 0x01e3ba68 nnotes.Lookup+10180 (3d000b9,0,824,1)
@[20] 0x01e2abb4 nnotes.NAMELookup2+1576 (79dfb029,0,0,ffffffff)
@[21] 0x017efa3f nnotes.AddGroupsContainingNameExtended+355 (a6baa7a0,46b8a2,79dfcab8,79dfb390)
@[22] 0x017f2559 nnotes.UAFGetAccessGroupsExtend2+3629 (0,483,0,0)
@[23] 0x01deb522 nnotes.NAMESwitchToNLThread+378 (693C00000000,1e10ea0,1670000C30A,2694)
@[24] 0x017ee629 nnotes.UAFGetAccessGroups+741 (7a0cc2,0,0,1)
@[25] 0x10048f28 nserverl.CreateNamesList+968 (7a0cc2,46a6e0,328ffaf,4b80ca)
@[26] 0x10046f95 nserverl.DoSessionMessage+661 (26180047,26180047,15e00033,2d)
@[27] 0x1004615c nserverl.ReceiveComplete+1784 (5a300172,26180047,79dffb70,46b8a2)
@[28] 0x10042577 nserverl.DbServer+4587 (cc430e4f,5a300172,79dffd1c,31)
@[29] 0x100614d0 nserverl.WorkThreadTask+2344 (cc430024,0,0,6)
@[30] 0x10001af1 nserverl.Scheduler+969 (0,0,0,0)
@[31] 0x0044e7d2 nnotes.ThreadWrapper+330 (0,0,0,79dfffa8)
[32] 0x77d6b71a kernel32.BaseThreadStart+58 (0,0,0,0)


############################################################
### thread 1/3: [ nClrepl: 1e94: 1e98]
### FP=0x00129168, PC=0x77ef0a3a, SP=0x00129168
### stkbase=0x00130000, total stksize=94208, used stksize=28312
############################################################
[ 1] 0x77ef0a3a ntdll.ZwWaitForSingleObject+10 (22000000099,1c809,1c809,8009)
[ 2] 0x77d704ff kernel32.WaitForSingleObjectEx+223 (80c,77d6e3cc,0,307)
@[ 3] 0x1004bf88 nnotes.WaitOnNativeSemaphoreCounted+100 (39f6620,3904c70,129320,79bda2c)
@[ 4] 0x1004d107 nnotes.OSLockReadFRWSem+915 (0,6289,14A00000009,0)
@[ 5] 0x11483b4e nnotes.BUFLockContainer+126 (14A00000009,1c809,1c809,11ab9c3e)
@[ 6] 0x11864275 nnotes.iNSFSlotUnPinWithHateInfo+153 (10C0000010D,444805,9AF74C30000001C,5737471D9B5DAB)
@[ 7] 0x115d9f21 nnotes.NSFSlotUnPinCtx+269 (2C000000A3,0,444809,1)
@[ 8] 0x117b8dab nnotes.BTExit+311 (4c61910,444809,129938,5)
@[ 9] 0x1147c6a1 nnotes.BTReleaseDataCtx+137 (40,1fc09,100451104,1A600004307)
@[10] 0x11a8f660 nnotes.GetData+820 (129650,27009,1a5,11587ccc)
@[11] 0x11a890c1 nnotes.CollectionNavigate+8801 (3,1fc09,ffffffff,8009)
@[12] 0x11adb6c6 nnotes.FindByKey+6354 (12000e,27009,129b80,e002)
@[13] 0x11a5dc72 nnotes.NIFFindByKeyExtended2+518 (1E940000C30A,4,0,ea)
@[14] 0x11a2755b nnotes.LookupNameWithRead+331 (0,0,0,1002fbfa)
@[15] 0x11a23889 nnotes.ReturnNameInfo+1269 (fc6ae28,2980,0,1004b215)
@[16] 0x11a1ba68 nnotes.Lookup+10180 (37c00b9,0,824,1)
@[17] 0x11a0abb4 nnotes.NAMELookup2+1576 (12b639,0,0,ffffffff)
@[18] 0x113cfa3f nnotes.AddGroupsContainingNameExtended+355 (3,0,12d0c8,0)
@[19] 0x113d2559 nnotes.UAFGetAccessGroupsExtend2+3629 (e3eef98,2,0,12d980)
@[20] 0x119cb522 nnotes.NAMESwitchToNLThread+378 (0,1004b796,12d510,12d770)
@[21] 0x113ce629 nnotes.UAFGetAccessGroups+741 (12f760,c6c81c,12f654,11bd002f)
@[22] 0x11bf103e nnotes.BuildNamesList+226 (12f557,120e56c4,e3a4aec,c125763f)
@[23] 0x11be1694 nnotes.RTRReplicateFiles+2044 (16ae,5002d2,c125763f,c125763f)
@[24] 0x004015ef nClrepl.CLREPLMainLoop+1287 (0,10118849,29cf470,0)
@[25] 0x0040109e nClrepl.AddInMain+158 (1,101189e9,0,0)
@[26] 0x00402a7a nClrepl.NotesMain+74 (2b3f910,0,0,2ad0000)
@[27] 0x00402b81 nClrepl.main+245 (2,0,0,0)
@[28] 0x00403308 nClrepl.mainCRTStartup+568 (0,0,0,0)
[29] 0x77d596ac kernel32.BaseProcessStart+44 (4030d0,0,0,0)


############################################################
### thread 1/3: [ nAMgr: 1cb8: 3044]
### FP=0x00123eb8, PC=0x77ef0a3a, SP=0x00123eb8
### stkbase=0x00130000, total stksize=94208, used stksize=49480
############################################################
[ 1] 0x77ef0a3a ntdll.ZwWaitForSingleObject+10 (6c56758,103a6e0,0,103a6e0)
[ 2] 0x77d704ff kernel32.WaitForSingleObjectEx+223 (758,103b8a2,0,299)
@[ 3] 0x0103ac18 nnotes.WaitOnNativeSemaphore+148 (f0249dfd,103a6e0,1240c8,1240c0)
@[ 4] 0x0103b842 nnotes.OSLockSem+242 (0,27d6fc8,1,5a3ea3a0)
@[ 5] 0x027d6d9e nnotes.DBCONTLockPNOVec+62 (100000B41,12452c,5a3ea3a0,273e296)
@[ 6] 0x02767fc7 nnotes.STMFillPinAccessDescNSF+199 (5a3ea4bc,5a3ea3a0,1cb8,6cb9cc8)
@[ 7] 0x02473f56 nnotes.BUFPin+322 (6c0bf68,16c09,16c09,5a3ea3a0)
@[ 8] 0x02853b36 nnotes.iNSFSlotPin+286 (ffffffff,16c09,34130000,40011b7)
@[ 9] 0x025ca46f nnotes.NSFSlotPinCtx+331 (1E4000000BB,16c09,0,40011b7)
@[10] 0x027d2b34 nnotes.BTGetLongData+1880 (124000000BF,3413cbec,1c,30011b7)
@[11] 0x027b6bf0 nnotes.BTGetCurrentEntry+1048 (3413b148,0,0,3413b148)
@[12] 0x02467e56 nnotes.BTGetDataCtx+178 (40,2b009,100A7FE44,15A00004307)
@[13] 0x02a7f4e8 nnotes.GetData+444 (1248b0,2b009,159,2577ccc)
@[14] 0x02a790c1 nnotes.CollectionNavigate+8801 (3,2c009,ffffffff,6009)
@[15] 0x02acb6c6 nnotes.FindByKey+6354 (12000e,2b009,124de0,e002)
@[16] 0x02a4dc72 nnotes.NIFFindByKeyExtended2+518 (1CB80000C30A,4,0,45)
@[17] 0x02a1755b nnotes.LookupNameWithRead+331 (0,101fbfa,0,101fbfa)
@[18] 0x02a13889 nnotes.ReturnNameInfo+1269 (6e126828,4531,0,103b215)
@[19] 0x02a0ba68 nnotes.Lookup+10180 (41d00b9,0,824,1)
@[20] 0x029fabb4 nnotes.NAMELookup2+1576 (126899,0,0,ffffffff)
@[21] 0x023bfa3f nnotes.AddGroupsContainingNameExtended+355 (40,1014fd2,128328,103a6e0)
@[22] 0x023c2559 nnotes.UAFGetAccessGroupsExtend2+3629 (6adbdf0,11,78b4f554,128a68)
@[23] 0x029bb522 nnotes.NAMESwitchToNLThread+378 (1287d8,103a6e0,5f,103b8a2)
@[24] 0x023be629 nnotes.UAFGetAccessGroups+741 (12a438,14,0,12a7a0)
@[25] 0x0209707f nnotes.CreateAgentNameList+523 (10000000852,101fbfa,0,103a6e0)
@[26] 0x02070968 nnotes.CheckAgentScriptLibrarySecurity+984 (12b5e0,25935ce,62525fa0,19c89)
@[27] 0x10008abf namgrdll.CheckScriptLibrarySecurity+187 (0,12bc20,40000002000,173a8968)
@[28] 0x1000895f namgrdll.AmgrSubProcessCallback+67 (ffffffff,0,0,0)
@[29] 0x02d26e29 nnotes.LSISession::SubProcessCallback+121 (6e1098a8,1001,7123e610,34)
@[30] 0x02d2862c nnotes.LSISession::SecurityForNewDoc+136 (0,34,6f8a987c,0)
@[31] 0x02d27e62 nnotes.LSISession::ModuleFind+2374 (0,D0000001A,0,0)
@[32] 0x02d27122 nnotes.LSISession::ProcessCallback+722 (2,40fe460,23,ff0000)
@[33] 0x02d26d57 nnotes.LSIClientProc+63 (602045c,103c5e6,c436,12c060)
@[34] 0x02d8b560 nnotes.LSsInstance::XClientCallback+1796 (0,0,0,0)
@[35] 0x02d8c2db nnotes.LSsInstance::UseModule+95 (0,2d2a4d0,0,0)
@[36] 0x02dd7d92 nnotes.LSsThread::OP_USE+226 (5fa0478,2e516bd,70fb2cd8,6f798120)
@[37] 0x02dafa5d nnotes.LSsThread::NRun+521 (0,5fa0340,5fa0378,3)
@[38] 0x02daf3d4 nnotes.LSsThread::Run+388 (6f798120,2d8c254,6e1098a8,7123e5b0)
@[39] 0x02d9c238 nnotes.LSsThread::Execute+124 (6e1098a8,7123e5b0,12c9c0,12c658)
@[40] 0x02e140a0 nnotes.LSsInstance::ModuleLoader+924 (5fa3450,2d2c594,6e126228,0)
@[41] 0x02e13cc7 nnotes.LSsInstance::LoadByName+87 (5fa3450,70f37750,24,10)
@[42] 0x02d2cbf3 nnotes.LSIModule::LoadObject+155 (371d288,371d1c0,12c650,12c650)
@[43] 0x02d25dd3 nnotes.CLSIDocument::AddModule+163 (30,34,6f7959c8,0)
@[44] 0x02d27fa8 nnotes.LSISession::ModuleFind+2700 (0,D0000001A,0,0)
@[45] 0x02d27122 nnotes.LSISession::ProcessCallback+722 (2,40fe460,23,ff0000)
@[46] 0x02d26d57 nnotes.LSIClientProc+63 (602045c,103c5e6,c436,12ca30)
@[47] 0x02d8b560 nnotes.LSsInstance::XClientCallback+1796 (0,0,0,0)
@[48] 0x02d8c2db nnotes.LSsInstance::UseModule+95 (0,2d2a4d0,0,0)
@[49] 0x02dd7d92 nnotes.LSsThread::OP_USE+226 (5fa0478,2e516bd,6f7a9438,6f798120)
@[50] 0x02dafa5d nnotes.LSsThread::NRun+521 (0,5fa0340,3c89180,6f7a842c)
@[51] 0x02daf3d4 nnotes.LSsThread::Run+388 (6f798120,2d8c254,6e1098a8,7123e570)
@[52] 0x02d9c238 nnotes.LSsThread::Execute+124 (6e1098a8,7123e570,12d390,12d028)
@[53] 0x02e140a0 nnotes.LSsInstance::ModuleLoader+924 (5fa3450,2d2c594,6e126028,0)
@[54] 0x02e13cc7 nnotes.LSsInstance::LoadByName+87 (5fa3450,70fb2cd8,28,2a)
@[55] 0x02d2cbf3 nnotes.LSIModule::LoadObject+155 (371d288,371d1c0,12d020,12d020)
@[56] 0x02d25dd3 nnotes.CLSIDocument::AddModule+163 (30,18,70fb01b8,0)
@[57] 0x02d27fa8 nnotes.LSISession::ModuleFind+2700 (0,60000000C,0,0)
@[58] 0x02d27122 nnotes.LSISession::ProcessCallback+722 (2,40fe460,23,ff0000)
@[59] 0x02d26d57 nnotes.LSIClientProc+63 (602045c,103c5e6,c436,12d400)
@[60] 0x02d8b560 nnotes.LSsInstance::XClientCallback+1796 (0,0,0,0)
@[61] 0x02d8c2db nnotes.LSsInstance::UseModule+95 (0,2d2a4d0,0,0)
@[62] 0x02dd7d92 nnotes.LSsThread::OP_USE+226 (5fa0478,2e516bd,6f7af130,6f798120)
@[63] 0x02dafa5d nnotes.LSsThread::NRun+521 (0,5fa0340,3c89180,6f7b2004)
@[64] 0x02daf3d4 nnotes.LSsThread::Run+388 (6f798120,2d8c254,6e1098a8,7123e510)
@[65] 0x02d9c238 nnotes.LSsThread::Execute+124 (6e1098a8,7123e510,12dd60,12d9f8)
@[66] 0x02e140a0 nnotes.LSsInstance::ModuleLoader+924 (5fa3450,2d2c594,6e124e28,0)
@[67] 0x02e13cc7 nnotes.LSsInstance::LoadByName+87 (5fa3450,6f7b1ff0,37,25)
@[68] 0x02d2cbf3 nnotes.LSIModule::LoadObject+155 (371d288,371d1c0,12d9f0,12d9f0)
@[69] 0x02d25dd3 nnotes.CLSIDocument::AddModule+163 (30,38,6f7b1c74,0)
@[70] 0x02d27fa8 nnotes.LSISession::ModuleFind+2700 (0,E0000001C,0,0)
@[71] 0x02d27122 nnotes.LSISession::ProcessCallback+722 (2,40fe460,23,ff0000)
@[72] 0x02d26d57 nnotes.LSIClientProc+63 (602045c,103c5e6,c436,12ddd0)
@[73] 0x02d8b560 nnotes.LSsInstance::XClientCallback+1796 (0,0,0,0)
@[74] 0x02d8c2db nnotes.LSsInstance::UseModule+95 (0,2d2a4d0,0,0)
@[75] 0x02dd7d92 nnotes.LSsThread::OP_USE+226 (5fa0478,2e516bd,6f7b0d80,6f798120)
@[76] 0x02dafa5d nnotes.LSsThread::NRun+521 (d8,f0,12df7c,2d9636a)
@[77] 0x02daf3d4 nnotes.LSsThread::Run+388 (6f798120,2d8c254,6e109c28,2efdd40)
@[78] 0x02d9c238 nnotes.LSsThread::Execute+124 (6e109c28,2efdd40,29,6adb5a8)
@[79] 0x02e140a0 nnotes.LSsInstance::ModuleLoader+924 (5fa3450,2d2c594,6e124a28,0)
@[80] 0x02e13cc7 nnotes.LSsInstance::LoadByName+87 (5fa3450,6f7b26d8,FFFFFFFFFFFFFFE,12f728)
@[81] 0x02d2cbf3 nnotes.LSIModule::LoadObject+155 (0,0,0,0)
@[82] 0x02d20a10 nnotes.CLSIDocument::RunScript+332 (12e8cc,20eeae3,1,1)
@[83] 0x02077f4a nnotes.CRawActionLotusScript::Run+894 (0,21a532a,0,12f728)
@[84] 0x0207110b nnotes.CRawAction::Execute+531 (1,1,1,6e11e828)
@[85] 0x0206be8c nnotes.CAssistant::RunAlone+16 (46534E2E34,103a6e0,ff,88c9)
@[86] 0x0206a233 nnotes.CAssistant::Run+6427 (38,1,12f628,12fa18)
@[87] 0x10006018 namgrdll.RunTask+3024 (12f8b0,77d6e3cc,0,0)
@[88] 0x100053cc namgrdll.ProcessRunMessage+104 (b4f33cf,1,6b26548,148)
@[89] 0x100051f9 namgrdll.ExecutiveMain+893 (5f1f471,6ad84dc,1,0)
@[90] 0x1000cfd1 namgrdll.AddInMain+257 (3,11089e9,0,0)
@[91] 0x0040104a nAMgr.NotesMain+74 (0,0,150000,0)
@[92] 0x00401151 nAMgr.main+245 (0,0,0,0)
@[93] 0x00401641 nAMgr.mainCRTStartup+369 (0,0,0,0)
[94] 0x77d596ac kernel32.BaseProcessStart+44 (4014d0,0,0,0)

 
Resolving the problem

This issue has been reported in SPR# AJMO7WJQ3V and is to be addressed in Domino 8.5.2. To obtain a hotfix, open a service request with IBM Support.


After applying the fix, you can confirm that it is working by issuing the following command on the Domino console:

    show stat net.*

Example:

Before Fix: (values are 0 - zero)
NET.GroupCache.Hits = 0
NET.GroupCache.Misses = 0

After Fix: (values may differ but greater than 0 - zero, if group authentication has taken place)
NET.GroupCache.Hits = 10
NET.GroupCache.Misses = 2


If you believe you are affected by this issue, you need to collect an NSD and verify if you do have multiple stacks like those shown above. The statistics will always be zero. The stats equaling zero do not indicate that a performance issue is caused by this software defect.
 
 
 

Copyright and trademark information
IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.
Rate this page
Please take a moment to complete this form to help us better serve you.
This material provides me with the information I need.




This material is clear and easy to understand.




Did the information help you to achieve your goal?
What updates, improvements, or related information would you like to see in this document?
Your response will be used to improve our document content. Requests for assistance, if applicable, should be submitted through your normal support channel as we cannot respond from this site.
Input the verification number to submit feedback:
Document information
 Product categories:
 Software
 Messaging Applications
 Advanced Messaging
 Lotus Domino
 Lotus Domino Server
 Operating system(s):
  AIX, IBM i, Linux, Solaris, Windows
 Software version:
  8.5, 8.5.1
 Software edition:
  All Editions
 Reference #:
  1406281
 IBM Group:
 Software Group
 Modified date:
 2009-10-15

Translate My Page
 
 

Rate this page

Help us improve this page. Your response will be used to improve our document content. Requests for assistance, if applicable, should be submitted through your normal support channel as we cannot respond from this site.