Monitoring slow or unresponsive servers with the Domino Diagnostic Probe

The 8.5.1 Administrator Help database that is included with release 8.5.2 does not include this information. This technote is the repository for this information until a future release when the documentation content will again be updated.

How the probe works

The probe is started from a command prompt and runs as a standalone process, not as a Domino server process. It uses the identity of the Domino server and attempts to open a session and a database as specified by the -database [-d] parameter every n seconds as specified by the -polling [-p] parameter. If the time it takes to open the database exceeds the time specified by the -threshold [-t] parameter, then the NSD program is launched to collect diagnostic data.

An NSD log file is generated in the <Domino Data Directory>\IBM_TECHNICAL_SUPPORT directory. The -nsdoptions parameter can be used to alter the behavior of the NSD program. Type nsd -help to see all available NSD parameters. The -outfile parameter can be used to enable printed output to be written to a specified log file. This option also directs printed output to the display. If an existing log file is specified on subsequent runs, the output is appended to the log file.

Note: You must use the IBM Java that comes with Lotus Domino. The probe has not been tested with SUN Java and is therefore not supported.

Note: DDP starts a Java process that loads as part of the Domino server Process ID table (PID.NBF file). Therefore, when the DDP is shut down, you must issue quit from the command prompt window in order to properly terminate the Java process. Do not use Ctrl-C or close the window abruptly without first issuing a quit, otherwise the java.exe process will terminate abnormally and you will see the following message on the Domino server console:

Process C:\Domino\jvm\bin\java.exe has terminated abnormally

If this occurs, the computer should be restarted.

Note: You should not restart the Domino server while the probe is running. The probe must be shut down before restarting the Domino server, or else certain Domino processes may hang during shutdown. (These processes would eventually be cleaned up, and an NSD created.)

New! You can also run the probe with Domino 8.5.1. To do so, open a service request with IBM Support, and request the special version of the dbopen.jar file that allows the probe to run with 8.5.1.

Options for the probe

The table below describes the options for the Domino Diagnostic Probe utility.

Parameter	Default	Purpose
-database [-d] <required>	none	Database to open; for example, mail/test.nsf
-threshold [-t]	30	Run NSD if the database open takes longer than threshold. Value in seconds.
-polling [-p]	60	The frequency the database is polled to open in seconds.
-nsdoptions	ALL	Options to pass to NSD. Must be in quotes; for example, "-nomemcheck -stacks" The default “ALL” will run all of the following commands: DUMP, MEMCHECK, PERF, HANDLES, WALKMEM, and PS. See nsd -help for details.
-outfile	none	Prints output to log file specified and to the display; for example, c:\temp\DDPlog.txt. The file name must be surrounded by quotes if the file path contains 'space' character(s). If the path is not specified, the file will be created in the Domino program directory.
-help	-	Displays help text.

To stop the probe, you must use the quit command at the command prompt from which you started the probe. Always use the quit command to stop the probe. Do not use Ctrl-C or close the window without first issuing a quit command. If you do, the java.exe process will terminate abnormally and you will see messages on the Domino server console. In this event, the computer will need to be restarted.

Note: The probe will remain active until it is stopped by the quit command. The probe will remain active even if the Domino server is brought down or issued a restart. If the Domino server becomes unresponsive for an extended period of time, the probe will execute the NSD three times. Once the Domino server is restarted, the probe will resume normal operation.

Setting up to run the probe

1. Create a new database of any type on the Domino server to be monitored; for example, mail\domprobe.nsf. Once created, open the database and go to File -> Application -> Access Control. In the Access Control List, add and then highlight the Domino server name and change the User type field to Unspecified as shown below. Save changes.



2. Copy the dbopen.jar to the Domino program directory. From a command prompt, switch to the directory where the server’s notes.ini is located (typically the data directory on UNIX and the program directory on Windows). Start the probe from a command prompt using the syntax example provided below:

Windows:

jvm\bin\java -jar dbopen.jar -d mail\domprobe.nsf -t 3 -p 30 -nsdoptions "-nomemcheck" -outfile C:\Domino\data\IBM_TECHNICAL_SUPPORT\DomPerfMon.txt

Note: If you use Windows 2008, set "Run this program as an administrator" on the "Compatibility" tab in the file properties for jvm/bin/java.exe.

UNIX (in data directory):

/opt/ibm/lotus/bin/java -jar /opt/ibm/lotus/bin/dbopen.jar -d mail/domprobe.nsf -t 3 -polling 45 -nsdoptions "-stacks" -outfile domperfmon.log

Note: UNIX and IBM i platform command examples are case sensitive. The probe will connect to the server by name.

AS400/iSeries/IBM i

The Domino Diagnostic Probe can be run in QShell (STRQSH).

Note: UNIX and IBM i platform command examples are case sensitive. The probe will connect to the server by name. On IBM i, use the command Configure TCP/IP (CFGTCP) and select Work with TCP/IP host table entries (Option 10) to display the host table. Use Add TCP/IP Host Table Entry (ADDTCPHTE) to add the server name and IP@ to the host table if needed.

1) Start QShell using RUNDOMCMD. Replace <server name> with the name of the Domino server.
RUNDOMCMD SERVER(<server name>) CMD(STRQSH) BATCH(*NO)

2) Set the JAVA_HOME environment variable
export JAVA_HOME=/QOpenSys/QIBM/ProdData/JavaVM/jdk60/32bit

3) Change to server's data directory (for example, cd /domino/data)

4) Start the probe (enter as one line). For -database, specify the name of the database being evaluated.
/QOpenSys/usr/bin/java -Djava.library.path=/QIBM/ProdData/LOTUS/domino852 -jar /QIBM/ProdData/LOTUS/domino852/dbopen.jar -database mail/domprobe.nsf


What You'll See:
The Domino server console will show sessions opened and closed by the server (one session opened by the server for each polling cycle - the previous session will be closed on the Domino server each time a new session is opened.) The output of the probe will print to the local command window and also a log file (if specified). If the database open exceeds the specified threshold, an NSD is created. Once the NSD completes, you'll see a message indicating that it is done and then logging will resume.

See sample output below (comments in italics and marked with >)

Sample Output

Domino Server
07/12/2010 04:45:48 PM Opened session for vlap2/vTest (Release 8.5.2 [Build 379])
07/12/2010 04:45:48 PM Closed session for vlap2/vTest Databases accessed: 1 Documents read: 0 Documents written: 0
07/12/2010 04:45:49 PM Opened session for vlap2/vTest (Release 8.5.2 [Build 379])
07/12/2010 04:45:49 PM Closed session for vlap2/vTest Databases accessed: 1 Documents read: 0 Documents written: 0
07/12/2010 04:45:54 PM Opened session for vlap2/vTest (Release 8.5.2 [Build 379])
07/12/2010 04:45:54 PM Closed session for vlap2/vTest Databases accessed: 1 Documents read: 0 Documents written: 0
07/12/2010 04:45:59 PM Opened session for vlap2/vTest (Release 8.5.2 [Build 379])
07/12/2010 04:45:59 PM Closed session for vlap2/vTest Databases accessed: 1 Documents read: 0 Documents written: 0
Note: Any session for the probe will be closed when the probe is shut down. Notice this time corresponds with the time of the probes 'shut down complete' message below.

DDP Prompt:
[C:\Domino] jvm\bin\java -jar dbopen.jar -d mail\domprobe.nsf -threshold 3 -polling 45 -nsdoptions "-nomemcheck" -outfile C:\Domino\data\IBM_TECHNICAL_SUPPORT\perfmon.txt
11/09/2009 5:46:10 AM Opening database mail\domprobe.nsf has taken 3.001 seconds. This exceeds the set threshold of 3.0 seconds.
> The DB open exceeded the threshold of 3 seconds so NSD was kicked off immediately.
11/09/2009 5:46:10 AM Running NSD...
> NSD being created.
11/09/2009 5:47:44 AM NSD completed with exit status 0
> NSD completed.
11/09/2009 5:47:45 AM Opening database mail\domprobe.nsf took a total of 4.001 seconds
> With the NSD done, DDP records the total time of the database open, in this case, 4.001 seconds. The NSD captured the state of the server at the time of the delayed DB open (at 3.001 seconds).
11/09/2009 5:48:33 AM Opening database mail\domprobe.nsf has taken 3.001 seconds. This exceeds the set threshold of 3.0 seconds.
11/09/2009 5:48:33 AM Running NSD...
11/09/2009 5:48:59 AM NSD completed with exit status 0
11/09/2009 5:49:00 AM Opening database mail\domprobe.nsf took a total of 5.354 seconds
> This DB open took 5.354 seconds.
...
Quit
> quit issued to stop the probe (optional). You must use 'quit' to stop the probe.
11/09/2009 5:51:42 AM Shut down in progress...
> Shut down of the probe begins.
11/09/2009 05:52:18 AM Shut down complete.
> Shut down of probe complete. It may take several minutes to complete shut down if an NSD is in progress.