Submitting Diagnostic Information to IBM CS/AIX Support for Problem Determination (MustGather Doc)

Technote (FAQ)


Question

What documentation does CS/AIX support need to troubleshoot problems?

Answer

Gathering the necessary information before calling IBM support helps familiarize you with the troubleshooting process and saves you time by having documentation ready when you speak with a support specialist.

Contents

  1. Update software
  2. Describe the problem
  3. Tracing
  4. System crashes, dumps, or hangs
  5. Submit data to IBM
  6. References


1. Update Software

Confirm that you are running the latest updates. Click here to determine what version of CS/AIX you are running and if support for that version is currently available. Click here to obtain the latest updates for each version of CS/AIX.


2. Describe the Problem

Create a new text file. Describe the problem in your own words. Include as much of the following information as possible:
  • Record any relevant messages you see; describe what was done that you believe caused them. Record any messages which appear on the remote system at the time of the problem.
  • Note the time the problem occurred. If you do not know the exact time, estimate.
  • Note the time the problem was discovered, and describe how it was discovered.
  • Note any recent changes that have been made to local configuration and to the network, even if you do not believe they are a direct cause.
  • Indicate if this is a new system or implementation, or if this has been working to date.
  • Note the AIX level ('oslevel -s') and CS/AIX level ('lslpp -L sna*').
  • List any other products interfacing with CS/AIX and their versions.
  • Briefly describe the network connection between CS/AIX and the remote host. Include details on any intermediate routers, switches, and so forth.


3. Tracing
    1. Introduction to tracing
    2. Basic line tracing - minimal tracing and logs
    3. API tracing
    4. TN3270 server and TN3270 redirector tracing
    5. LU0 tracing
    6. Extended tracing
    7. Additional tracing


3a. Introduction to Tracing

For most problems, especially when the problem is easy to recreate or predict, minimal tracing is necessary. When you suspect an application problem, API tracing should be included. When a problem is difficult to recreate or predict, and you need to leave tracing running for a long period of time, or if you are in production and transmitting very large volumes of data in very short times, extended tracing may be necessary. Additional tracing options are also discussed.

Full detail about trace settings is documented in the CS/AIX Diagnostics Guide .

If you are unable to perform tracing or do not believe tracing will be necessary for this problem, you should still capture a test case to provide useful information to IBM. This information includes SNA configuration, software level and history, SNA status, SNA logs, system error report, and system device information. Enter the following commands to capture a test case:
    cd /tmp
    rm -f pd.tar.gz
    snagetpd -q

Proceed to Section 5 Submit data to IBM.

3b. Basic line tracing - minimal tracing and logs

Whenever possible, stop SNA and delete old SNA logs and traces. Additionally, before starting SNA, reconfigure the logs and start new traces. This process will require space in both the /tmp and /var file systems. Ensure that plenty of free space exists, more than 30MB (for the settings given below), in both file systems.

The Communications Server has a SNA daemon, which must be active, and an SNA node, which must be inactive. Use the following commands to prepare for tracing:

Note that these commands are only valid for SNA 5.0.2.1 or greater. Apply the latest PTFs for SNA before proceeding. To achieve best results and provide for cleaner traces, stop SNA by following the instructions below. If SNA must not be stopped, skip the first six commands on the following list, and begin after the 'verifysna -U' command. Be aware, however, that this could make isolating the problem more difficult.
    snaadmin term_node
    sna stop
    trcstop    # an error here is ok and can be ignored
    rm /var/sna/*
    sna start
    verifysna -U
    # skip to here if SNA must not be stopped
    snaadmin set_trace_file,trace_file_type=IPS, trace_file_size=20000000
    snaadmin set_log_file,log_file_type=AUDIT,file_size=10000000
    snaadmin set_log_file,log_file_type=ERROR,file_size=10000000
    snaadmin set_global_log_type,audit=YES,exception=YES
    snaadmin set_global_log_type,succinct_audits=YES, succinct_errors=YES
    snaadmin add_dlc_trace
    snaadmin set_trace_type,trace_flags=NONE
    snaadmin init_node   # if the node is not already active

At this point, reproduce the problem. Once the problem has recurred, continue with the following steps:
    cd /tmp  # or any other location with enough space
    rm -f pd.tar.Z pd.tar.gz
    snagetpd -q

This tracing is terminated when snagetpd is executed. This tracing will also terminate when the SNA software is stopped.

Proceed to Section 5 Submit data to IBM.

3c. API tracing
    1. Introduction to API tracing
    2. Determine which API to trace
    3. Back-level API tracing
    4. Up-level API tracing

3c.i Introduction to API Tracing

If you suspect a problem with an application, API tracing will be necessary. CS/AIX provides multiple APIs for an application to use. API tracing, if needed, should be performed in conjunction with the basic line tracing.

APIs that were provided by CS/AIX V4 and earlier are now considered back-level APIs; these are included to maintain compatibility with older applications, many of which are still in use. Documentation for these APIs is not included, but may still be available. Refer to the following documentation:
CS/AIX v4 Transaction Program Reference (SC31-8212)
CPI Communications Programming (GC31-8210)

APIs that are provided by CS/AIX V5 and later are considered up-level APIs. Documentation for these APIs is included with CS/AIX and is available on-line.

3c.ii Determine which API to trace

First, determine which API the application is written to use.

Applications that use back-level APIs will use libraries such as:
    /usr/lib/liblu0.a
    /usr/lib/libcpic.a
    /usr/lib/libcpic_r.a
    /usr/lib/libmsapi.a
    /usr/lib/libsna.a
    /usr/lib/libisna.a

Applications that use up-level APIs will use the libraries found in "/usr/lib/sna/".

If the application was compiled to dynamically link libraries, you may be able to use " dump -Hv" against the application binary to determine which libraries it uses.

Once you have determined which set of APIs your application uses, proceed to Back-level API Tracing or Up-level API Tracing as appropriate. If you are unable to determine which set of APIs your application uses, you can perform both sets of tracing at the same time.

3c.iii Back-level API tracing (not available in V7)

To enable back-level API tracing, issue the following command:
    snaadmin set_trace_type,trace_flags=NONE,api_flags=ALL

Note that this command is present in section 3b Basic line tracing with only a change in the parameter api_flags. If also performing basic line tracing then just change that parameter in those instructions; do not issue the command a second time.

Once enabled, back-level API tracing will continue until disabled or until the SNA software (not just the node) is stopped.

To disable back-level API tracing, issue the following command:
    snaadmin set_trace_type,trace_flags=NONE,api_flags=NONE

Tracing is recorded to the same files as basic line tracing; space requirements are those for basic line tracing.

3c.iv Up-level API tracing

Up-level API tracing requires that certain environment variables be set in the environment of the application to be traced.

Typically, this is done by stopping the application, setting and exporting the variables, starting the application from the same shell in which the variables are set, and finally unsetting the variables so nothing else is traced. To stop the tracing, you must stop the application and restart it without those environment variables set.

If this application is started by another program, then determine how to set the variables. For example, in the case of WebSphere Application Server, configure them in the WAS administrative console. When you login to the administrative console, expand the 'environment' box on the left and choose 'Manage WebSphere Variables". On the main page, select the appropriate scope, then define the variables.

Another example is SNA API tracing with DB2.

API tracing can also be controlled within the application using the CSV (Common Service Verbs) API. This requires changing and recompiling the application, so will typically only be usable by application developers, not end-users.

In the environment of the application, set the following environment variables:

Unlimited trace (limited only by the filesystem)

Name

Value

SNATRC

/tmp/snaapi.trc::

SNATRCRESET

NO

SNACTL

1

Round robin trace (two limited-size trace files, alternating as they fill)

Name

Value

SNATRC

/tmp/snaapi1.trc:/tmp/snaapi2.trc:

SNATRCRESET

NO

SNACTL

1

SNATRUNC

1024

SNATRACESIZE

10000000

Note that the colons in the value for SNATRC are very important. There are two colons in each case. If one is omitted, you will not capture any trace data.

It is very important that the user the application is running as has permission to write to the files named by the SNATRC variable.

Space requirements are 20MB for round-robin tracing with the above settings and potentially unlimited for unlimited tracing. This is in addition to the space requirements of basic line tracing.

These trace files are not captured by snagetpd, so you will need to submit them to IBM in addition to the snagetpd output.

If these API traces prove difficult to enable, some internal trace options may provide limited API trace data. To enable internal tracing, issue the following command:
    snaadmin set_trace_type,trace_flags=LUA+APPC

Note that this command is present in section 3b Basic line tracing with only a change in the parameter trace_flags. If also performing basic line tracing then just change that parameter in those instructions; do not issue the command a second time.

Once enabled, internal tracing will continue until disabled or until the SNA software (not just the node) is stopped.

To disable internal tracing, issue the following command:
    snaadmin set_trace_type,trace_flags=NONE

Tracing is recorded to the same files as basic line tracing; space requirements are those for basic line tracing.

3d. TN3270 Server and TN3270 Redirector Tracing

TN3270 server tracing captures the data flowing between the TN3270 client and the TN3270 server and also the communications between the TN3270 server and CS/AIX. TN3270 redirector tracing captures data flowing between the downstream host and TN3270 redirector and between TN3270 redirector and the upstream host. An AIX iptrace may also be helpful, especially if a socket-level problem is suspected.

Perform the following steps to capture TN server and TN redirector traces, and an AIX iptrace:

1. Enable tracing:
    snaadmin set_trace_file,trace_file_type=TN_SERVER,trace_file_size=20000000
    snaadmin set_tn_server_trace,trace_flags=ALL
    iptrace  /tmp/iptrace

2. Attempt a client connection through the tn3270 server or tn redirector.

3. Disable tracing and collect data:
    ps  -ef | grep iptrace | grep -v grep | awk '{print $2}' | xargs kill
    snaadmin  set_tn_server_trace,trace_flags=NONE
    cd /tmp # or any other location with enough space
    ipreport -ens iptrace  > iptrace.ebcdic
    ipreport -ns iptrace > iptrace.ascii
    tar cvf -  iptrac* snatn*.trc | compress > snatn.tar.Z

    # Package SNA traces:
    rm -f pd.tar.Z pd.tar.gz
    snagetpd -q

Space requirements with the above settings are 20MB for the CS/AIX trace and potentially unlimited for the AIX iptrace.

The traces produced by the set_tn_server_trace command are captured by snagetpd, but traces produced by iptrace are not.

3e. LU0 Tracing (not available in V7)

Because LU 0 is supported only as a backlevel function in CS/AIX V5 and V6, the latest documentation is from CS/AIX V4. The following items are the sections of the CS/AIX V4 manuals pertaining to LU 0 tracing:
To activation LU 0 API tracing, you must configure the LU for API tracing prior to activating the LU. Activate LU 0 server tracing at startup using the lu0 command or after startup using the lu0sndmsg command.

The traces produced by this command are captured by snagetpd.

3f. Extended Tracing

Typically, the settings provided above are sufficient for most problems. However, sometimes you require vastly larger settings in the following situations:
  • if you cannot predict when the problem will occur
  • if you can not quickly detect the problem when it occurs
  • if you have a very large volume of data flowing over a very short period of time.

In these cases, you may need to increase the size limits of the trace files. For basic line tracing, use the trace_file_size parameter of the set_trace_file command to set size limits. For up-level API tracing, set the SNATRACESIZE environment variable. For TN3270 server and TN redirector tracing, use the trace_file_size parameter of the set_trace_file command to set size limits. All of these sizes are in bytes.

If you need to increase the size limits of the trace files, be sure to consider the limitations of your filesystems. The basic line trace and TN3270 server and redirector trace each generate two files of the specified size in "/var/sna". Up-level API tracing generates its files in the location specified by the SNATRC environment variable. The snagetpd process copies all data to a temporary directory in /tmp and the runs tar on it before running compress on the tar file; this implies that you will need enough space in /tmp to contain about 2.5 times the total sum of all tracing you have active.

3g. Additional Tracing

The following trace options are available, but are rarely used or needed by CS/AIX support:
    1. GDLC
    2. iptrace
    3. external

3g.i GDLC

GDLC tracing shows the traffic going across the AIX DLC. This trace is really an AIX trace that uses the AIX trace facility, but CS/AIX provides an easier interface to use to capture just what is relevant to SNA traffic. The main things a GDLC trace will show that the regular SNA line trace will not all deal with activity on the link layer: frame sequence numbers, receiver ready (RR) frames, poll and response frames, etc. This is very seldom needed, unless link-level problems are suspected.

GDLC tracing requires the bos.sysmgt.trace fileset from the AIX install media and one of the following:
    • CS/AIX 5.0.4.2 and one of:
      • efix for APAR IY10752
      • reboot immediately after finished tracing
    • CS/AIX 5.0.4.3 or higher
    • CS/AIX 6.0.0.0 or higher

To activate GDLC tracing, issue the following command:
    snaadmin add_gdlc_trace, port_name=xxxxx

When activating the trace, a port name must be specified. The port name is for the PORT defined within the SNA configuration.

WARNING: The command does not check that the specified port name is valid. This is intended as a feature. The trace code allows you to issue an "add_gdlc_trace" for a port which does not exist, since you could, at some later stage, define the port and want to activate it with trace enabled. A side effect of this is that if you incorrectly type a port name, you will not get any error. In particular, be careful with upper and lower case!

To deactivate GDLC tracing, issue the following command:
    snaadmin remove_gdlc_trace [, port_name=xxxxx]

When deactivating the trace, a port name is optional. If a port name is not specified, all GDLC tracing is deactivated on all ports. Again, the command does not check that the specified port is valid. If a port is specified, tracing will be stopped only on the specified port; all other ports with active traces will be unaffected.

Space requirements for this trace are minimal (1.5 MB). This limit cannot be changed. This trace should be stopped quickly once the problem occurs, or critical data may be lost as new data is recorded.

The traces produced by this command are not captured by snagetpd.

3g.ii iptrace

The AIX iptrace may be useful when dealing with IP-related issues that may be affecting EE (Enterprise Extender), TN3270 server, TN redirector, or Anynet. This trace is very rarely needed.

Documentation on iptrace can be found in the AIX Commands Reference.

3g.iii external

A line sniffer or data scope provides information about the actual data flowing over the wire. This data consists of the SNA PIU (TH, RH, & RU) along with the DLC and communications media (token ring, ethernet, SDLC, and so on) transmission headers. This level of tracing is rarely needed, except where physical transmission problems are suspected or when trying to locate the network component that is the source of the problem.

4. System crashes, dumps, or hangs

If you experience a crash, dump, or hang of the operating system, first pursue this with AIX Support. AIX Support will help collect the dump and analyze it. They will route it to CS/AIX support if they determine it was caused by CS/AIX.

Managing System Dump Devices

5. Submit data to IBM
    1. Collect the data and documentation.
    2. Submit files to IBM.

5a. Collect the Data and Documentation

If you have performed basic line tracing, back-level API tracing, and TN3270 server or redirector tracing, all data should be contained in the file created by snagetpd: pd.tar.Z (CS/AIX 6.1 or earlier) or pd.tar.gz (CS/AIX 6.3 or later). If you have performed up-level API tracing or any of the additional tracing, you will probably have other files in addition to the pd.tar.Z or pd.tar.gz file.

It is easier if you can combine all the files into one package. Move or copy all of the files to /tmp including the pd.tar.Z or pd.tar.gz file, then use tar and gzip to package them. For example:
tar cvf - file1 file2 file3 file4 | gzip > csaixpd.tar.gz

Include the text file you created in section 2 Describe the Problem as one of the files in the package.

5b. Submit files to IBM

IBM provides two servers to accept customer data. One is a consolidated system intended for world wide use. The other is intended for United States use and predates the world wide server.
  • World Wide (preferred)
    • Name the file using the format:
    pmrnumber.branch.ccc.filename.ext
    where pmrnumber is your PMR number,
    branch is the Branch Office number,
    ccc is the country code, filename is the name you give the file, and ext is the file extension
    For example: 12345.999.000.csaixpd.tar.Z
    • FTP to ftp.emea.ibm.com.
    • Log on as: Anonymous
    • Password: yourE-mailAddress
    • Open (cd to) the toibm folder.
    • Open (cd to) the aix folder.
    • Enter bin at the ftp command prompt.
    • Store (put) the compressed file on the server.
    United States domestic
    • Name the file using the format:
    NNNNN.bBBB.cCCC.filename.ext
    where NNNNN is your PMR number,
    BBB is the Branch Office number,
    CCC is the country code,
    filename is the name you give the file,
    and ext is the file extension
    For example: 12345.b999.c000.csaixpd.tar.Z
    • FTP to testcase.boulder.ibm.com.
    • Log on as: Anonymous
    • Password: yourE-mailAddress
    • Open (cd to) the aix folder.
    • Open (cd to) the toibm folder.
    • Enter bin at the ftp command prompt.
    • Store (put) the compressed file on the server.

6. References

CS/AIX On-line Library
http://www-01.ibm.com/software/network/commserver/aix/library/publications.html
especially the Diagnostics Guide and the Administration Command Reference.

IBM Centralized Customer Data Store Service
http://www.ibm.com/de/support/ecurep/index.html

Sending Test cases Electronically to AIX Support
http://www.ibm.com/support/docview.wss?uid=isg3T1000214

AIX Library
http://www.ibm.com/servers/aix/library/index.html

CS/AIX V4 Library
http://publib.boulder.ibm.com/cgi-bin/bookmgr/Shelves/ASY0BK01

Cross reference information
Segment Product Component Platform Version Edition
Networking IBM Communications Server for Data Center Deployment All AIX 7.0.0.0

Product Alias/Synonym

cs/aix csaix comm communications server commserver sna

Rate this page:

(0 users)Average rating

Document information


More support for:

Communications Server for AIX
All

Software version:

5.0, 6.0, 6.1, 6.3, 6.3.1, 6.4, 6.4.0.1, 6.4.0.2, 6.4.0.3, 6.4.0.4, 7.0.0.0

Operating system(s):

AIX

Reference #:

1210597

Modified date:

2007-01-02

Translate my page

Machine Translation

Content navigation