IBM Support

Perl script to collect IBM Spectrum Protect server monitoring data V3.1

Troubleshooting


Problem

Servermon version 3.14.3 (August 2021) is a script used to collect Protect Server data to troubleshoot problems with server processes, client sessions and of performance nature.  

Environment

IMPORTANT NOTICE:

For Servers running on AIX 7.2 TL5:

Due to AIX APAR IJ31681, the IBM Spectrum Server can hang when Servermon issues the show threads command. More information:
IBM Spectrum Protect server crash on AIX 7.2 TL5

To avoid the issue, apply the fix for APAR IJ31681 when available, or edit "servermon.ini" and set the "showthreads" option to N:

[Run Time Options]
showthreads      = N                  # include SHOW THREADS command output
Do not forget to change it back to Y after applying the fix for APAR IJ31681.


For Servers running versions 8.1.7.000 and later, but below 8.1.10.200

Due to IT34271, upgrade the Server to 8.1.10.200 or later if using Cloud Container Pools and if Servermon appears hung on startup.

For AIX servers running 8.1.8.000 to 8.1.8.010 only:

Due to APAR IT29996, upgrade to 8.1.9.000 or later.   If it's not possible to upgrade now, disable SHOW THREADS by editing "servermon.ini" and changing the value for "showthreads" from Y to N in the [Run Time Options] stanza:

[Run Time Options]
showthreads      = N                  # include SHOW THREADS command output

For Linux x86 servers running 8.1.6.0 only:

Upgrade to 8.1.6.100 or later. If you cannot upgrade now, disable instrumentation by editing servermon.ini and changing the value of "instrumentation" to N :

Example:

instrumentation = N

Other platforms and versions do not have an issue with instrumentation and should remain enabled.

For Servers V8.1.6.100 or higher

If running 8.1.6.100 server or higher, it is recommended to make the following command replacements in commands.ini to reduce the amount of output:

1. In the [commands] stanza, change "show txntable" command to "show txntable lockdetail=no"
2. In the [more_commands] stanza, change "show locks" to "show locks onlywaiters=yes

Resolving The Problem

Collecting Data: Table of contents:

General information
Preperation (required for all methods of running)   ** NEW **
Running server monitoring in the background - always on
Running server monitoring in the foreground - one time only
Running server monitoring on a storage agent
Notes
Submitting Information to IBM Support
Online Self-Help Resources
Related Information
Downloads

General Information

servermon.pl is a Perl script used to collect data from IBM Spectrum Protect servers. It typically does so in 20 minute intervals called diagCycles. But there are also extra collections done when the script starts and ends.

The script can also be used to collect a trace and in addition to that, stop the collection when a specific server message is logged in the activity log.

The script primarily collects server instrumentation for thread level analysis, various queries and show commands used for troubleshooting. It also collects DB2 specific data for when it's necessary to do more in-depth analysis on a server problem. And finally, it also collects data from the operating system, such as CPU, memory and disk metrics.

It has been tested both internally and with customers and is not known to cause performance degradation.

With this technote you can find the most current version of the server monitoring script. This is supported at all levels of server V7.1 and higher. To set up monitoring you need the following 3 files from the downloads section:

  • servermon.pl - the script file itself
  • servermon.ini - configuration parameters
  • commands.ini - commands that are submitted by the script


Download the files to a local directory, Data collected will be written to the current directory from where the script is run.

The script can be run a few different ways:

  • In the foreground (default)
  • In the background (nohup)
  • On a storage agent (no database)


Note: The script requires a Perl interpreter to run. Some operating systems such as Windows do not ship with Perl installed. Strawberry Perl and Active Perl are two popular distributions, but there are others as well.

Preparation - Required

1.  Download the 3 files (servermon.pl, commands.ini and servermon.ini) to a local directory, data collected will be written to the current directory from where the script is run.

2. To run in the background, you need to update some of options in the servermon.ini file. The servermon.ini file contains different configuration stanzas,
The servermon.ini file assigns values to a fixed set of variables and is of the form:
variable = value # some comment

At a minimum the following options in the [TSM] stanza need to be updated:
[TSM]
server        = SERVERSTANZA     # server stanza for UNIX, TCPSERVERADDRESS for Windows
tcpport       = 1500         # only used on Windows

3.  Optional, but recommended: Create a IBM Spectrum Protect administrator ID with a name of SERVERMON and a password of your choice. Grant it SYSTEM authority, and use this administrator ID to run servermon. This way you can see in the activity log which commands are issued by servermon. It helps troubleshooting if suspecting a problem with one of the servermon commands.

4. Optional, but it's also preferable to update the CollCycleEndTime in the [Run Time Options] stanza, which  sets the time when a 24-hour cycle will end and new one will begin, to reflect the start of the client backup window. By default, it's set to 17:00.  For example, if the backup window starts at 18:00, then set it to:

[Run Time Options]
CollCycleEndTime = 18:00 # When AlwaysOn=Y, when will new collection cycle start

To find the proper value to use for the server variable, for *IX systems look at the server stanza that is either configured in the default dsm.sys file or the file DSM_CONFIG environment variable is pointing to.  On Windows systems, you can specify the IP address of the server or alternatively specify either localhost or 127.0.0.1.  The value for tcpport is the port number the server listens to.

Running servermon always on

For the script to collect a complete set of docs on a server, it must be run locally as the instance user.

NOTE: 

If running Server version 8.1.7.x, 8.1.8.x or 8.1.9.x, stop the built-in Servermon using setopt alwaysonservermon no and restart the Server instance before using the Perl version of Servermon.

If using server 8.1.10.000 or later, it's recommended to use the built-in version of Servermon unless directed otherwise by support. If the built-in version of servermon was disabled, re-enable it using setopt alwaysonservermon yes and restart the Server instance.

Windows:
When starting Servermon for the first time, there will be prompts for the IBM Spectrum Protect administrator ID and password created in Step 3 above. Those credentials will be saved to a password file.

Run the script from a DB2 command window:
Start -> Run -> db2cmd
From there, cd to the target directory where servermon.pl was saved and issue this command:
perl servermon.pl 

AIX:
When starting Servermon for the first time, there will be prompts for the IBM Spectrum Protect administrator ID and password created in Step 3 above. Those credentials will be saved to a password file.

Change directory to where servermon.pl is stored and issue:
perl64 ./servermon.pl (the perl64 interpreter prevents out of memory conditions on AIX systems)

It is often desirable to run Servermon in the background because it will not get killed if the Telnet/SSH session is interrupted. To run Servermon in the background after the credentials are saved with the initial invocation, stop Servermon using CTRL+C. It will take a few minutes for Servermon to stop because it's finishing a data collection cycle. Then start Servermon in the background using:
nohup perl64 ./servermon.pl &

Linux:
When starting Servermon for the first time, there will be prompts for the IBM Spectrum Protect administrator ID and password created in Step 3 above. Those credentials will be saved to a password file.

Change directory to where servermon.pl is stored and issue:
perl ./servermon.pl 

It is often desirable to run Servermon in the background because it will not get killed if the Telnet/SSH session is interrupted. To run Servermon in the background after the credentials are saved with the initial invocation, stop Servermon using CTRL+C. It will take a few minutes for Servermon to stop because it's finishing a data collection cycle. Then start Servermon in the background using:
nohup perl ./servermon.pl &
 

Running servermon in the foreground - one run only

For the script to collect a complete set of docs on an server, it must be run locally as the instance user.

Windows:
Run the script from a DB2 command window:
Start -> Run -> db2cmd
From there, cd to the target directory where servermon.pl was saved and issue this command:
perl servermon.pl prompt

AIX:
Change directory to where servermon.pl is stored and issue:
perl64 ./servermon.pl prompt 
(the perl64 interpreter prevents an out of memory symptom on AIX systems)

Linux:
Change directory to where servermon.pl is stored and issue:
perl ./servermon.pl prompt

3. Follow the prompts:

Press enter to accept default values shown in brackets:

Enter the servername as found in dsm.opt or dsm.sys [servermon]:
On Windows server specify the TCP/IP address of the target server, as the script is expected to be run on the same machine as the server is running usually localhost will work.
For all non-Windows environments, specify the servername as found in dsm.sys/dsm.opt.

Enter the IBM Spectrum Protect administrator login ID []:
Use the administrative ID created in step 3 or any ID with system privilege.

1 [] (no echo to the screen):
Enter the password.

Collect SHOW THREADS output? (Y/N) [Y]:
Specifies whether or not to run the "show threads" command. Default value is Y.

Collect SHOW DEDUPDELETE output? (Y/N) [Y]:
If running legacy deduplication (DEVCLASS=FILE sequential pools), specify Y for this option. If you are only running container pools enter N.

Collect SHOW for memory statistics? (Y/N) [Y]:
SHOW ALLOC/MEM/MEMTREND output is not collected by default, but the information is helpful if the symptom to investigate is memory usage related. Use as advised by IBM support

Collect SHOW REPLICATION output? (Y/N) [N]:
Show replication can sometimes take a long time to run, by default is N.  Use as advised by IBM support.

Run Procstack command? (Y/N) [N]:
Collects process call stack.  Use as advised by IBM support.

Will Servermon run continuously (Y/N) [N]:
Use the default of N (no) so that it does one collection cycle and exits.

How many hours will ServerMon run for? [24]:
Enter how many hours to run Servermon before existing, default is 24 hours and that's preferred for most performance problems. It can be set shorter when troubleshooting a problem that is quicker to re-create, but the duration should be long enough so that it captures both the beginning and the end of the activity that needs to be investigated.

Enable collectstmt instrumentation? (N/Y) [Y]:
Enabling COLLECTSTMT will result in the generation of instrumentation statistics for  SQL statements. Default value is Y.

Enable SNAPSHOT FOR DYNAMIC SQL? (N/Y) [N]:
Enabling SNAPSHOT FOR DYNAMIC SQL will result in the generation of extra DB2 statistics. As the collection could impact server performance the default is set to not collect the statistics.
Note: usually collecting snapshot for dynamic SQL requires to enable certain DB2 monitors, use as advised by IBM support.

Enter comma-separated stop events, e.g. ANR#####E,ANR9999D_########## [none]:
Here you can specify a comma-separated list of messages as logged to the activity log. If any of the messages specified is found during the runtime of the script, the script will go through some final doc collection and then stop.

Enter traceflags [none]:
Here you can specify any server traceflags to disable, for e.g. if you specified trace aggregate AF you might want to disable AFTXN subset. Specify as instructed by IBM support.

Copy FFDCLOG file? (N/Y) [Y]:
At the end of each (documentation collection cycle (see below) add a copy of the FFDCLOG file to the collected docs. Default is "Y". This is only supported for servers, not storage agents.

Running servermon on a storage agent

1. To collect monitoring data for a storage agent (STA), review the client dsm.sys or dsm.opt file for the following options:

  • Lanfreecommmethod (supported on *IX via server stanza)
  • Lanfreeshmport (supported in *IX via server stanza)
  • Lanfreetcpport
  • Lanfreetcpserveraddress


2. On *IX systems, create a server stanza in dsm.sys for storage agent that matches the communication definitions identified in Step 1.

TCPServeraddress -> Lanfreetcpserveraddress
TCPPort -> Lanfreetcpport
Commmethod -> Lanfreecommmethod
Shmport -> Lanfreeshmport


For Windows systems, you need to configure the STA to allow for TCP communication by adding "commmethod tcpip" in dsmsta.opt if it is not present for the script to work properly.

3. To start the script, do the following:
Windows:
From a Windows command prompt, change directory to where servermon.pl is stored and issue: and issue this command:
perl servermon.pl

AIX:
When starting Servermon for the first time, there will be prompts for the IBM Spectrum Protect administrator ID and password created in Step 3 above. Those credentials will be saved to a password file.

Change directory to where servermon.pl is stored and issue:
perl64 ./servermon.pl (the perl64 interpreter prevents out of memory conditions on AIX systems)

It is often desirable to run Servermon in the background because it will not get killed if the Telnet/SSH session is interrupted. To run Servermon in the background after the credentials are saved with the initial invocation, stop Servermon using CTRL+C. It will take a few minutes for Servermon to stop because it's finishing a data collection cycle. Then start Servermon in the background using:
nohup perl64 ./servermon.pl &

Linux:
When starting Servermon for the first time, there will be prompts for the IBM Spectrum Protect administrator ID and password created in Step 3 above. Those credentials will be saved to a password file.

Change directory to where servermon.pl is stored and issue:
perl ./servermon.pl (the perl64 interpreter prevents out of memory conditions on AIX systems)

It is often desirable to run Servermon in the background because it will not get killed if the Telnet/SSH session is interrupted. To run Servermon in the background after the credentials are saved with the initial invocation, stop Servermon using CTRL+C. It will take a few minutes for Servermon to stop because it's finishing a data collection cycle. Then start Servermon in the background using:
nohup perl ./servermon.pl &

Notes


1. If the script is run with no prompt with a set number of diagnostic/collection cycles, it could also be used with a task scheduler (making sure that the DB2 environment is being established correctly first). This way, the script could be run continuously on a day to day basis to gather performance data or monitoring information for a given problem. In addition, the corresponding files created can be also be archived off (and deleted) with IBM Spectrum Protect for historical purposes using the archive retention policies set up in archive copygroup.

2. You must change the [DB2] stanza if your instance name is something other than the default of tsminst1:

[DB2]
instance        = tsminst1     # DB2 instance name, if your DB2INSTANCE environment variable is set it

dbalias         = tsmdb1       # DB2 database alias for TSM database

When the script is invoked it will verify that the instance specified exists, and if not the script will provide a list of instances to select from.

Submitting Information to IBM Support


In most cases, the script will create a .zip file in the current directory, that is the file that needs to be sent to support.

In some cases if the script is interrupted before it completes the collection cycle, then it will be necessary to compress or tar the data collection directory located in the current output directory. The directory has the following naming convention: YYYYMMDD-HHMM-instance_name-swg21590928

Data collected can be submitted to IBM using either ECUREP or Blue Diamond depending on the account..

If using ESR , update the PMR to indicate that data has been sent.

Online Self-Help Resources

Related Information

Files to download

Filename

Download

Version

cksum

md5sum

servermon.pl

servermon-pl-3-14-003.zip 3.14.3 3927547571 109233 servermon.pl ccf0e44c51d6c496a93b424358e2e775 *servermon.pl

servermon.ini

servermon-ini-3-14-001.zip

3.14.001

1781179331 4824 servermon.ini

639dcf8cbab8aa262ace2e5a5eccb6de *servermon.ini

commands.ini

commands-ini-3-14-006.zip

3.14.006

2247016816 30836 commands.ini

3be6498a4c51259511176f5378dd3e2f *commands.ini

Once extracted from the .zip file, the above checksums apply.

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSEQVQ","label":"IBM Spectrum Protect"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
25 August 2021

UID

swg21432937