APAR status
Closed as returned (APAR cannot be resolved without additional info from IBM or customer.)
Error description
Customers Domino Server was running without issue when a crash was experienced. From a review of the data we can see the Domino Server has crashed with the following error message: Error Message = PANIC: LookupHandle: handle out of range referencing the following database : Database: e:\apps\notes\data\mail\abc123.nsf Based on the call stack, the crash was invoked when updating one of the following documents in e:\apps\notes\data\mail\abc123.nsf. There's cases where this type of crash can be due to a memory overwrite, when a bogus database handle is passed, or when the same database handle returned two different threads, which is an illegal state and causes the server to crash. This particular crash stack does not match a known issues in this code stream. Should you see this crash in the future to gather further information it would require you to have the additional debug parameter enabled Debug_Checkmarkers=1 **Checkmarkers is a special debug created to aide in troubleshooting memory corruption caused by a memory overwrite. Memory overwrites can be caused by Domino, or other third-party applications. In this case it appears to be on the Domino Side. If corruption is found then Domino will panic immediately. Yes, checkmarkers will cause Domino to crash sooner, but this is what we want. It gets us closer to the actual problem by dumping the faulting stack. This helps Development isolate the cause and sometimes provides information to help reproduce the problem. Moving forward, to be thorough, I would suggest to either pull a new replica of the e:\apps\notes\data\mail\abc123.nsf from a server not having an issue, and or run the following off-line maintenance for this transactional logged server (nFixup -F -J, nCompact -b, nUpdall-R) Please let me know if you have any additional questions and or comments. Customer has not made any changes on the server is a review of the data shows the following: NSD REVIEW 1) The first thing that I do when analyzing an NSD is look at the Name of the Server, Date & Time, OS Version and Notes Version. Host Name : ABCACME User Name : SYSTEM Date : Tue Nov 01 09:38:50 2011 Windows Dir : C:\WINDOWS Arguments : "e:\apps\notes\nsd.exe" -dumpandkill -termstatus 1 -panicdirect -crashpid 5532 -crashtid 5148 -runtime 600 -ini "e:\apps\notes\notes.ini" -svcreq 128 NSD Version : 8.5.22.1048 (Release 8.5.2FP2) OS Version : Windows/2003 5.2 [64-bit] (Build 3790), PlatID=2, Service Pack 2 (12 Processors) Running as 32-bit Windows application on 64-bit Windows Build time : Wed Mar 23 02:53:24 2011 Latest file mod : Fri Feb 18 11:33:30 2011 Domino Version : Release 8.5.2FP2 HF57 (32-bit server) Keyview Version : 10.8.0.0 2) Once we have verified the Build I then search for "OS Process" to see what is running on the server, when the OS was rebooted and we can see when the server crashed outlined in RED. -> 159c 15c8 18 10/08 11:29:55 ["e:\apps\notes\nSERVER.EXE" =e:\apps\notes\notes.ini: 159c] ["e:\apps\notes\nsd.exe" -svcddlg -svcreq 128 -ini "e:\apps\notes\notes.ini": 0210] 3) Now that we have reviewed the OS process we want to see what task the server crash on so we go to the top of the NSD and search for the word "FATAL" which will bring us down to the fatal thread. We then review the fatal thread to see if it a known issue and we continue our search this time searching using the following: "159c: 141c" ############################################################ ### thread 63/242: [ nSERVER: 159c: 141c] FATAL THREAD (Panic) ### FP=0x102bc168, PC=0x7d61c846, SP=0x102bc0fc ### stkbase=0x102c0000, total stksize=262144, used stksize=16132 ### EAX=0x00000000, EBX=0x102bcd38, ECX=0x00000000, EDX=0x00000000 ### ESI=0x00003af8, EDI=0x00000000, CS=0x00000023, SS=0x0000002b ### DS=0x0000002b, ES=0x0000002b, FS=0x00000053, GS=0x0000002b Flags=0x00000206 ############################################################ [ 1] 0x7d61c846 ntdll.ZwWaitForSingleObject+21 (3af8,927c0,0,102bc918) [ 2] 0x7d4d8bf1 kernel32.WaitForSingleObject+18 (3af8,927c0,102bd0ac,102bcd4e) @[ 3] 0x6020c605 nnotes.FRSendCommandToService+789 (102bc938,102bcd38,102bcd4e,0) @[ 4] 0x6020d448 nnotes.OSRunExternalScript@8+1064 (258,1) @[ 5] 0x6020d9bf nnotes.FRTerminateWindowsResources+975 (1,1010,1,0) @[ 6] 0x6020dde8 nnotes.OSFaultCleanupExt@24+984 (1e96a68,1010,0,0,0,102bd3d4) @[ 7] 0x6020de6a nnotes.OSFaultCleanup@12+26 (0,1010,0) @[ 8] 0x60219704 nnotes.OSNTUnhandledExceptionFilter@4+276 (102be40c) @[ 9] 0x601dae4d nnotes.Panic@4+589 (60eda477) @[10] 0x601df310 nnotes.LockHandleExt@16+320 (c9fc12,102be454,102be458,0) @[11] 0x601df473 nnotes.OSLockObjectExt@8+35 (1e50014,612ab080) @[12] 0x60002b8e nnotes.OSLockObject@4+14 (c9fc12) @[13] 0x60025be8 nnotes.NSFNoteClose@4+760 (5e77) @[14] 0x60a82a48 nnotes.NoteOpenExtendedImpl@52+20504 (102bf764,1de802,3000176,0,0,102bf494,f10f10,ffffffff,0,0,0,0,0) @[15] 0x60a83584 nnotes.NoteOpenExtended@28+356 (102bf764,1de802,3000176,1,0,102bf708,f10f10) @[16] 0x6089f66d nnotes.iNSFNoteOpenExtended2@28+109 (102bf764,1de802,3000176,1,0,102bf734,f10f10) @[17] 0x609882f0 nnotes.NSFNoteOpenExtended2@28+480 (94e,1de802,3000176,1,0,102bf7a8,f10f10) @[18] 0x1002eb12 nserverl.ServerNoteOpen@8+962 (5cac9918,7b40003c) @[19] 0x10021c63 nserverl.DbServer@8+2515 (7560015d,69780062) @[20] 0x10037277 nserverl.WorkThreadTask@8+1655 (68533b0,0) @[21] 0x100018ce nserverl.Scheduler@4+750 (0) @[22] 0x6015577f nnotes.ThreadWrapper@4+175 (0) [23] 0x7d4dfe21 kernel32.FlsSetValue+310 (601556d0,0,0,8000a) 4) After a couple of passes through the Fatal Thread stack frames we continue Searching on "159c: 141c" and search until we reach the MM/OS section 5) Once we reach the MM/OS section we can see when the server was started and again we see the reference to the FATAL Stack : "159c: 141c" and continue to search the NSD until we hit the Vthread Mapped to Pthread. <@@ ------ Notes Data -> OS Data -> MM/OS Structure Information (Time 09:40:34) ------ @@> Start Time = 10/08/2011 12:29:55 PM Crash Time = 11/01/2011 09:38:49 AM Console Log Enabled = 1 Console Position = 784896824 Error Message = PANIC: LookupHandle: handle out of range Console Position = 784896824 SharedDPoolSize = 4194304 FaultRecovery = 0x00010010 Cleanup Script Timeout= 600 Crash Limits = 3 crashes in 5 minutes StaticHang = Virtual Thread [ nSERVER: 159c: 016d] (Native thread [ nSERVER: 159c: 141c]) (0x159c/0x16d/0x141c) ConfigFileSem = ( SEM:#0:0x010d) n=0, wcnt=-1, Users=-1, Owner=[ : 0000] FDSem = ( RWSEM:#52:0x410f) rdcnt=-1, refcnt=0 Writer=[ : 0000], n=52, wcnt=-1, Users=0, Owner=[ : 0000] 6.) Now that we have reached the Vthread mapped to the Pthread we can see 1 database. This being mail\fk12.nsf which shows 5 documents open (class=0001) in this database at the time of the crash and this is the database I would focus on. As discussed above I would run complete offline maintenance (nFixup -F -J, nCompact -b, nUpdall -R) on this database. ** VThread [ nSERVER: 159c: 016d] .Mapped To: PThread [ nSERVER: 159c: 141c] .Description: SPR NAME/ABC/ACME .. using: Primal Thread [ nSERVER: 159c: 0044] .. SOBJ: addr=0x49f40f94, h=0xf0102d02 t=0xcf02 (BLK_FT_STATIC) .. SOBJ: addr=0x49993dbc, h=0xf0102cfd t=0xc275 (BLK_NSFT) .. SOBJ: addr=0x49cf048c, h=0xf0102d04 t=0xc130 (BLK_TLA) .. SOBJ: addr=0x4a181ff4, h=0xf0102cfb t=0xc30a (BLK_LOOKUP_THREAD) .. Database: e:\apps\notes\data\mail\abc123.nsf .... DBH: 2382, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes .... DBH: 2286, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... view: hCol=2520, cg=N noteID=1962, sessID: [134: 5103] ($Calendar)|Calendar .... DBH: 2527, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes .... DBH: 2085, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... view: hCol=1990, cg=N noteID=1482, sessID: [134: 5103] (MiniView - Notices2)|MiniView - Notices2 .... DBH: 2503, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... doc: HDB=2503, NoteID=30982, hNote=0x62c3, flags=0200, class=0001 ...... doc: HDB=2503, NoteID=4326, hNote=0x6b0c, flags=2200, class=0001 .... DBH: 2406, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... doc: HDB=2406, NoteID=231406, hNote=0x2fbf, flags=2300, class=0001 ...... doc: HDB=2406, NoteID=269118, hNote=0x4cac, flags=0200, class=0001 ...... view: hCol=6633, cg=N noteID=1962, sessID: [134: 5103] ($Calendar)|Calendar .... DBH: 7079, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... doc: HDB=7079, NoteID=2298, hNote=0x5d75, flags=2300, class=0001 .... DBH: 3870, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... view: hCol=6178, cg=N noteID=4606, sessID: [134: 5103] (MiniView - Tasks2)|MiniView - Tasks2 .... DBH: 6246, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes .... DBH: 6668, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes ...... view: hCol=1924, cg=N noteID=4578, sessID: [134: 5103] Archive Folder from Littleton\ABCTEAM 3\RUSHMORE .... DBH: 7247, By: CN=SPR NAME/OU=ABC/O=ACME, WasAccessed=Yes
Local fix
Moving forward, to be thorough, I would suggest to either pull a new replica of the e:\apps\notes\data\mail\abc123.nsf from a server not having an issue, and or run the following off-line maintenance for this transactional logged server (nFixup -F -J, nCompact -b, nUpdall-R) Please let me know if you have any additional questions and or comments. Also to gather further information on this crash it would require: Should you see this crash in the future to gather further information it would require you to have the additional debug parameter enabled Debug_Checkmarkers=1
Problem summary
Problem conclusion
Temporary fix
Comments
This APAR is associated with SPR# DADS8N7MBD. When more information in provided we can look further.
APAR Information
APAR number
LO65176
Reported component name
DOMINO SERVER
Reported component ID
5724E6200
Reported release
852
Status
CLOSED RET
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-11-01
Closed date
2011-11-11
Last modified date
2011-11-11
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Applicable component levels
[{"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Product":{"code":"SSKTMJ","label":"Lotus Domino"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5.2","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
11 November 2011