Must Gather - Submitting Diagnostic Information to IBM CS Linux support.

Technote (troubleshooting)


Problem(Abstract)

Gathering the necessary information before calling IBM support helps familiarize you with the troubleshooting process and saves you time by having documentation ready when you speak with a support specialist.

Resolving the problem

Contents

  1. Verify and update software
  2. Describe the problem
  3. Tracing
  4. System crashes, dumps, or hangs
  5. Submit data to IBM
  6. References


1. Update Software

Confirm that you are running the latest updates.

2. Describe the Problem

Create a new text file. Describe the problem in your own words. Include as much of the following information as possible:
  • Record any relevant messages you see; describe what was done that you believe caused them. Record any messages which appear on the remote system at the time of the problem.
  • Note the time the problem occurred. If you do not know the exact time, estimate.
  • Note the time the problem was discovered, and describe how it was discovered.
  • Note any recent changes that have been made to local configuration and to the network, even if you do not believe they are a direct cause.
  • Indicate if this is a new system or implementation, or if this has been working to date.
  • Note the release of CS Linux ('rpm -qa ibm*').
  • List any other products interfacing with CS Linux and their versions.
  • Briefly describe the network connection between CS Linux and the remote host. Include details on any intermediate routers and/or switches.

3. Tracing
  1. Introduction to tracing
  2. Line tracing
  3. API tracing
  4. Server / Client tracing

3.A. Introduction to Tracing

For most problems, especially when the problem is easy to recreate or predict, minimal tracing is necessary. When you suspect an application problem, API tracing should be included. When a problem is difficult to recreate or predict, and you need to leave tracing running for a long period of time, or if you are in production and transmitting very large volumes of data in very short times, extended tracing may be necessary. Additional tracing options are also discussed.

Full detail about trace settings is documented in the CS Linux Diagnostics Guide.

If you are unable to perform tracing or do not believe tracing will be necessary for this problem, you should still capture a test case to provide useful information to IBM. This information includes SNA configuration, software level and history, SNA status, SNA logs, system error report, and system device information. Enter the following commands to capture a test case:

  • Issue the /opt/ibm/sna/bin/snagetpd command. This will place a file named pd.tar.gz in the current directory. Save this file for later retrieval.
  • When running a client-server environment, run 'snagetpd' on both the client and server.
  • When running a Windows client, the C:\ibmcs\w32cli\snagetpd.exe tool was added with the CS Linux v6.2.1 maintenance release. The snagetpd.exe tool creates a snapd.exe file which should be sent to support for analysis.
  • When running a Windows-x64 client, the C:\ibmcs\w64cli\snagetpd.exe tool was added with the CS Linux v6.2.2 maintenance release. The snagetpd.exe tool creates a snapd.exe file which should be sent to support for analysis.

Note: you should not be in the /var/opt/ibm/sna or /var/log directories when executing 'snagetpd'. Also, you should be logged on as the 'root' userid to have snagetpd to capture all the information required for analysis.


3.B. Basic line tracing - minimal tracing and logs

Whenever possible, stop SNA and delete old SNA logs and traces. Additionally, before starting SNA, reconfigure the logs and start new traces. This process will require space in both the /tmp and /var file systems. Ensure that plenty of free space exists, more than 30MB (for the settings given below), in both file systems.

To achieve best results and provide for cleaner traces, stop SNA by following the instructions below. If SNA cannot be stopped, skip the first five commands on the following list, and begin after the 'verifysna -U' command. Be aware, however, that this could make isolating the problem more difficult.

snaadmin term_node
sna stop
rm /var/sna/*
sna start
verifysna -U
# skip to here if SNA cannot be stopped
snaadmin set_trace_file,trace_file_type=IPS,trace_file_size=10000000
snaadmin set_global_log_type,audit=YES, exception=YES
snaadmin set_global_log_type,succinct_audits=NO, succinct_errors=NO
snaadmin add_dlc_trace
snaadmin set_trace_type,trace_flags=LUA+APPC
snaadmin init_node   # if the node is not already active

At this point, reproduce the problem. Once you have recreated the problem, continue with the following steps:

# snaadmin remove_dlc_trace
# snaadmin set_trace_type,trace_flags=NONE
# rm -rf /tmp/IBM
# mkdir /tmp/IBM
# cd /tmp/IBM
# snagetpd -q /tmp/IBM/pmr##


Note: Tracing is terminated when snagetpd is executed. This tracing will also terminate when the SNA software is stopped.

Proceed to Section 5 Submit data to IBM.


3.C. API Tracing

If you suspect a problem with an application, API tracing will be necessary. CS Linux provides multiple APIs for an application to use. API tracing, if needed, should be performed in conjunction with the basic line tracing.

API tracing requires that certain environment variables be set in the environment of the application to be traced.

Typically, this is done by stopping the application, setting and exporting the variables, starting the application from the same shell in which the variables are set, and finally unsetting the variables so nothing else is traced. To stop the tracing, you must stop the application and restart it without those environment variables set.

If this application is started by another program, then determine how to set the variables. For example, in the case of WebSphere Application Server, configure them in the WAS administrative console. When you login to the administrative console, expand the 'environment' box on the left and choose 'Manage WebSphere Variables". On the main page, select the appropriate scope, then define the variables.

API tracing can also be controlled within the application using the CSV (Common Service Verbs) API. This requires changing and recompiling the application, so will typically only be usable by application developers, not end-users.

In the environment of the application, set the following environment variables:

Unlimited trace (limited only by the filesystem)


Name

Value

SNATRC

/tmp/snaapi.trc::

SNATRCRESET

NO

SNACTL

1

Round robin trace (two limited-size trace files, alternating as they fill)

Name

Value

SNATRC

/tmp/snaapi1.trc:/tmp/snaapi2.trc:

SNATRCRESET

NO

SNACTL

1

SNATRUNC

1024

SNATRACESIZE

10000000

Note that the colons in the value for SNATRC are very important. There are two colons in each case. If one is omitted, you will not capture any trace data.

It is very important that the user the application is running has permission to write to the files named by the SNATRC variable.

Space requirements are 20MB for round-robin tracing with the above settings and potentially unlimited for unlimited tracing. This is in addition to the space requirements of basic line tracing.

These trace files are not captured by snagetpd, so you will need to submit them to IBM in addition to the snagetpd output.

If these API traces prove difficult to enable, some internal trace options may provide limited API trace data. To enable internal tracing, issue the following command:
snaadmin set_trace_type,trace_flags=LUA+APPC

Note that this command is present in section 3b Basic line tracing with only a change in the parameter trace_flags. If also performing basic line tracing then just change that parameter in those instructions; do not issue the command a second time.

Once enabled, internal tracing will continue until disabled or until the SNA software (not just the node) is stopped.

To disable internal tracing, issue the following command:
snaadmin set_trace_type,trace_flags=NONE

Tracing is recorded to the same files as basic line tracing; space requirements are those for basic line tracing.

Sample Trace Instructions for Printer application:

# Stop the printer application
.
# Add snaadmin command to your path
export PATH="$PATH:/opt/ibm/sna/bin"
.
# Prepare for and Activate tracing
snaadmin term_node
sna stop
rm /var/opt/ibm/sna/*
sna start
snaadmin set_trace_file,trace_file_type=IPS,trace_file_size=10000000
snaadmin set_global_log_type,audit=YES, exception=YES
snaadmin set_global_log_type,succinct_audits=NO, succinct_errors=NO
snaadmin add_dlc_trace
snaadmin set_trace_type,trace_flags=LUA+SLIM+LLC2+GDLC+MAC
snaadmin init_node # if the node is not already active
.
# If it is not already active, start the link station with the
# printer LUs.
.
# In a terminal window, as the user that starts the
# printer application:
.
export SNATRC=/tmp/api.trc:: # the 2 colons are important
export SNATRCRESET=NO
export SNACTL=1
.
# Start the printer application from that same terminal window.
.
unset SNATRC
.
# Recreate the problem. When you believe the problem has
# occurred, continue below to capture traces.

If you capture an API or Line Trace as described in section 3B, run snagetpd after the trace captures the problem and before stopping the SNA node.
.
cd /tmp # or any other location with enough space
rm -f pd.tar.gz
snagetpd -q
tar cvf - ./pd.tar.gz ./api.trc | gzip > PMRnnnnn.tar.gz
.
# Stop the printer application. You can restart it again immediately
# if you want, but you must stop it once to stop the API trace.
.

# Ref. section 5 below for submitting diagnostic files to IBM.


3.D. Client/Server Tracing

Client/Server tracing captures the data flowing between the CS Linux Server and the Remote API Client.

Perform the following steps to capture Client/Server traces:

1. Enable tracing:
snaadmin set_trace_file,trace_file_type=CS,trace_file_size=10000000
snaadmin set_cs_trace,trace_flags=ALL

2. Attempt a client connection from the Remote API Client system.

3. Disable tracing and collect data:
snaadmin set_cs_trace,trace_flags=NONE

# Package SNA traces:
rm -f pd.tar.gz
snagetpd -q

Space requirements with the above settings are 20MB for the CS Linux trace.


# Ref. Section 5 below for submitting diagnostic files to IBM.


4. System crashes, dumps, or hangs

If you experience a crash, dump, or hang of the operating system, first pursue this with support of the appropriate operating system (i.e, VM, Linux ..etc.). They will route it to CS Linux support if they determine it was caused by CS Linux.


5. Submit data to IBM
  1. Collect the data and documentation.
  2. Submit files to IBM.

5a. Collect the Data and Documentation

If you have performed basic line tracing, TN3270 server tracing, or TN Redirector tracing, all data should be contained in the file created by snagetpd: pd.tar.gz. If you have performed API tracing or any of the additional tracing, you will probably have other files in addition to the pd.tar.gz file.

It is easier if you can combine all the files into one package. Move or copy all of the files to /tmp including the pd.tar.gz file, then use tar and compress to package them. For example:
tar cvf - file1 file2 file3 file4 | gzip > snagetpd.tar.gz

Include the text file you created in section 2 Describe the Problem as one of the files in the package.

5b. Submit files to IBM

The preferred way to submit files is through the web site http://www.ecurep.ibm.com/toibm/linux
Enter the PMR number in the format nnnnn,bbb,ccc where nnnnn is the 5 digit PMR number, bbb is the branch number, and ccc is the country number.

FTP (preferred)
  • Name the file using the format: pmrnumber.branch.ccc.filename.ext
where pmr number is your PMR number, branch is the Branch Office number,
ccc is the country code, filename is the name you give the file, and ext is the file extension
For example: 12345.999.000.snagetpd.tar.gz
  • FTP to ftp.emea.ibm.com.
  • Log on as: Anonymous
  • Password: yourE-mailAddress
  • Open (cd to) the toibm folder.
  • Open (cd to) the linux folder.
  • Enter bin at the ftp command prompt.
  • Store (put) the compressed file on the server.

For additional alternatives, such as ESR and email, see the document "Exchanging information with IBM Technical Support" at http://www.ibm.com/software/support/exchangeinfo.html



6. References

CS Linux Support

CS Linux Library

IBM Centralized Customer Data Store Service

Cross reference information
Segment Product Component Platform Version Edition
Networking Communications Server for Linux All Linux Red Hat - xSeries, Linux SUSE - xSeries, Linux xSeries, Linux Red Hat - pSeries, Linux SUSE - pSeries 6.2, 6.2.1, 6.2.2, 6.2.3, 6.2.3.1, 6.2.3.2, 6.2.3.3, 6.4, 6.4.0.1, 6.4.0.2, 6.4.0.3, 6.4.0.4, 7.0.0.0 All Editions
Networking Communications Server for Linux on zSeries All Linux Red Hat - zSeries, Linux SUSE - zSeries, Linux zSeries 6.2, 6.2.1, 6.2.2, 6.2.3, 6.2.3.1, 6.4, 6.2.3.2, 6.4.0.1, 6.4.0.2, 6.4.0.3, 6.4.0.4, 7.0.0.0 All Editions

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

Communications Server for Linux on zSeries
All

Software version:

6.2, 6.2.1, 6.2.2, 6.2.3, 6.2.3.1, 6.2.3.2, 6.4, 6.4.0.1, 6.4.0.2, 6.4.0.3, 6.4.0.4, 7.0.0.0

Operating system(s):

Linux Red Hat - zSeries, Linux SUSE - zSeries, Linux zSeries

Software edition:

All Editions

Reference #:

1169668

Modified date:

2005-06-07

Translate my page

Machine Translation

Content navigation