Memcheck memory analyzer

What is Memcheck and how does it work?

Memcheck is the Notes Domino memory analyzer that was originally provided to Notes Support for the Unix platforms. Each version of Notes Domino requires its own compiled version of Memcheck. The version of memcheck that is installed during an original installation should not be replaced with any newer or older gold release of memcheck currently available.

eg:
MEMCHECK Version (4.12) for Lotus Notes Release 5.0.11 (July 24, 2002)

means just 5.0.11 and should not be installed on anything less than 5.0.11 or higher than 5.0.11

Note Beginning with Domino Release 5.0.9, nsd.exe, memcheck.exe (Win32 and OS/2), and nsd.sh and memcheck (Unix) are included during installation or upgrade.

Memcheck is a binary file (executable) that is invoked by the NSD utility whenever NSD is run (usually at the time of a Notes server crash on UNIX). Memcheck runs a scan of all shared memory segments in use by Notes processes, as well as the individual stack memory allocated to each process. Memcheck determines if the memory segment format matches the expected format for the memory segment. If Memcheck does not find what it expects in the memory segment, it dumps the appropriate memory segment pool into a file with the following syntax:

memcheck.<platform>_<hostname>_mm_dd@hh_mm.dump

where "mm_dd@hh_mm" represents the month and date followed by hour and minute.
For example, on an AIX box, with a hostname of NotesSrv1, the memcheck file would be named:

memcheck.AIX_NotesSrv1_02_08@12_30.dump

This filename will be included in the Generated Info/Warnings/Errors section at the bottom of the NSD. It is important to note that a memcheck.dump file may not be created in all crash situations. A memcheck.dump file should only be created if a memory segment has been scanned and found to contain corruption.

If you are unsure whether Memcheck has found a corrupted memory pool, please consult IBM Support. Under certain conditions, Memcheck may be run manually if the NSD is not invoking memcheck properly.


Options available in Memcheck


[ -ini notesini ] -- specify notes.ini (must be first argument)\n\
[ -k c*urrent ] -- analyze shared memory using default/current environment\n\

[ -p pid ] -- analyze process 'pid' heap memory\n\
[ -pf pid ] -- analyze process 'pid' heap & shared memory\n\
[ -b path ] -- path (colon-separated) where process \n\
binaries can be found.\n\
[ -c corefile ] -- analyze core file 'corefile' memory\n\
[ -f imagedump ] -- analyze memory saved in file 'imagedump'\n\
[ -d option [ -o file] ] -- dump memory segment image(s) to file \n\
option is:\n\
a*ll: dump all segment to file.\n\
e*rr: dump only corruputed segment to file.\n\
-o outfile : dump segments to 'outfile'.\n\

[ -r shlib ] -- load additional mem check shared lib 'shlib'\n\
[ -t typevalue ] -- show/dump block/package memory only\n\
[ -u ] -- dump type usage\n\
[ -a check_level ] -- Check level [1-2] (default 2)\n\
level 1: check static dpools only\n\
level 2: and check pools, vpools, ...etc\n\
[ -v verbose_level ] -- verbose level [0-5] (default 0)\n\
level 0: No verbose\n\
level 1: show steps taken\n\
level 2: and show static dpool block only\n\

level 4: and show all pool blocks\n\
level 5: and dump pool block memory\n\

[ -x MemDumpLevel ] -- memory dump level [0-2] (default 1)\n\
level 0: No block memory dumps\n\
level 1: dump max of 200 bytes per block\n\
level 2: dump all block memory\n\
[ -l outputFileName ] -- file containing memcheck output - iSeries only\n\
[ -y pkg[,pkg]* ] -- dump selected packages only\n\
[ -s symname:module ] -- dump symbol 'symname' in module 'module'\n\
[ -s address:size ] -- dump 'size' bytes of process address 'address' \n\
[ -m PathMap] -- Path map\n\
[ -h ] -- for this help\n\
[ -i ] -- show version info only\n\

The following is abstract by running Memcheck for R4.65a for Solaris:


# ./memcheck

MEMCHECK Version (4.6) for Lotus Notes Release 4.6.5a (August 4, 1999)
Copyright ¸ 1985-1999, Lotus Development Corporation, All Rights Reserved

Usage: memcheck [options]
Options:
[ -k c*urrent ]          -- analyze shared memory using default/current environment
[ -k shmkey [-1]         -- analyze shared memory with IPC key 'shmkey'
[ -m prefix ] ]             -1 : (number 1) -- analyze one segment only.
                            -m prefix : dumps are in files <prefix><key>*.*
[ -p pid ]               -- analyze process 'pid' heap memory
[ -P pid ]               -- analyze process 'pid' heap & shared memory
[ -b path ]              -- path (colon-separated) where process
                            binaries can be found.
[ -c corefile ]          -- analyze core file 'corefile' memory
[ -f imagedump ]         -- analyze memory saved in file 'imagedump'
[ -d option [ -o file] ] -- dump memory segment image(s) to file option is:
                            a*ll: dump all segment to file.
                            e*rr: dump only corruputed segment to file.
                            -o outfile : dump segments to 'outfile'.
[ -L ]                   -- live mode (lock shared data semaphores)
[ -r shlib ]             -- load additional mem check shared lib 'shlib'
[ -t typevalue ]         -- show/dump block/package memory only
[ -u ]                   -- dump type usage
[ -a check_level ]       -- Check level [1-2] (default 2)
                            level 1: check static dpools only
                            level 2: and check pools, vpools, ...etc
[ -v verbose_level ]     -- verbose level [0-5] (default 1)
                            level 0: No verbose
                            level 1: show steps taken
                            level 2: and show static dpool block only
                            level 3: and show all pool headers
                            level 4: and show all pool blocks
                            level 5: and dump pool block memory
[ -x MemDumpLevel ]      -- memory dump level [0-2] (default 1)
                            level 0: No block memory dumps
                            level 1: dump max of 200 bytes per block
                            level 2: dump all block memory
[ -y pkg[,pkg]* ]        --  dump selected packages only
[ -s symname:module ]    --  dump symbol 'symname' in module 'module'
[ -s address:size   ]    --  dump 'size' bytes of process address 'address'

[ -Z ]                    -- scan core file for DPOOL areas
[ -h ]                    -- for this help
[ -i ]                    -- show version info only

Examples:
        memcheck -k 0xf8000000
        memcheck -p 1251 -p 1255
        memcheck -c core
        memcheck -k 0xf8000000 -p 1254 -d all -o imagedump.out
        memcheck -f imagedump.out -v2
        memcheck -k 0xf8000000 -t 0x4120   # show block type 0x4120 entries only
        memcheck -k 0xf8000000 -t 0x4100   # show package 0x4100 entries only
#



Memcheck errors running NSD:
Occasionally, when you run NSD o/p, you may see some memcheck errors, as follows:

"Could not load program /opt/lotus/notes/latest/ibmpow/memcheck: An offset in the .loader section header is too large. Examine the .loader section header with the 'dump -Hv' command."

What you should do is verify if memcheck is working in the first place before proceeding. To do so, try running memcheck with the -i option at the command line. If it is showing you the version of memcheck properly, it is in good shape; otherwise, your memcheck may be corrupted.


Supporting Information:

To confirm that a corrupted memory segment has been found, refer to the Memcheck section of the NSD. The section begins with the following syntax.

@@@@@@@@@@@@@ Notes Memory Analyzer @@@@@@@@@@@@@
MEMCHECK Version (3.32) for Lotus Notes Release 4.6a (November 3, 1997)
Copyright ¸ 1985-1997, Lotus Development Corporation, All Rights Reserved
** Open shared memory DPOOL key=0xf8000000
** Analyze shared memory DPOOL 'key=0xf8000000' size=1070752
** MMM size=1070752
** Open shared memory DPOOL key=0xf8000001
** Analyze shared memory DPOOL 'key=0xf8000001' size=76695844

Problems will be indicated with the message:

WARNING (33): Invalid POOL (0x0414) free chain (next=45055) @539352
** Dumping shared DPOOL '0xf8000001' @0x40000000 to ./memcheck.AIX_NotesSrv1_02_08@12_30.dump

or

** Analyze shared memory DPOOL 'key=0xf8000004' size=76695844
ERROR (12): Invalid DPOOL free size 1379975908 @0
WARNING (30): Invalid POOL (0x0311) block end signature 0x736f @4130906
WARNING (33): Invalid POOL (0x0311) free chain (next=30013) @4130908
ERROR (20): Invalid DPOOL block end signature 0x6a206f6e @4147236
Last Block(idx=2777, type=free): size = 132, end @4081815

followed by:

** ... Skip 5393216 bytes to end of DPOOL @76695844 (no allocted/free blocks found)
** Dumping shared DPOOL '0xf8000004' @0x70000000 to ./memcheck.AIX_NotesSrv1_02_08@12_30.dump