IBM Support

How to collect Packet Traces and other TCP/IP related traces on z/OS

Technote (FAQ)


Question

How do I collect a Packet trace on z/OS? Or any other traces related to the TCP/IP stack?

Answer

Table of Contents

A. General Considerations for using an External Writer
B. Collecting Multiple Traces
C. General Considerations for using internal buffers

D. Collecting a Packet (SYSTCPDA) Trace
E. Collecting an Event (SYSTCPIP) Trace
F. Collecting an OSA Network Analysis (SYSTCPOT) Trace




A. General Considerations for using an External Writer

To have a CTRACE written via an external writer, a writer proc needs to be created in either SYS1.PROCLIB or in a library concatenated in the MASTER JCL. The name is arbitrary, but the member name and the name on the PROC statement need to match. Below is a sample writer proc:


   //CTWTR1   PROC
   //IEFPROC  EXEC PGM=ITTTRCWR  
   //TRCOUT01 DD DSNAME=USERID.TRACE.DATA,UNIT=SYSDA,
   //            SPACE=(CYL,(500,0),,CONTIG),DISP=(NEW,CATLG),DSORG=PS

Note that the DSNAME can be set to any name that can be catalogued in your environment. Rather than specifying a single data set name, it would be better to have the procedure create a new unique name each time it is used. Specify the name using system symbols (such as &SYSNAME, &LYYMMDD (date), and/or &LHHMMSS (time), or define a GDG and create a new generation each time. The space allocated should be representative of the amount of activity you plan on collecting.

Before starting the external writer proc, make sure that its dispatching priority is at least as high as the ASIDs being traced (or in the same WLM service class) to prevent lost records or missing trace records. Use of larger trace buffer sizes will reduce the instance of lost records, see the General Considerations for Internal Traces for more information. Also, make sure that the external writer data set allocation is not restricted by SMS rules or another storage manager.

For more information about the external writer and specification of the associated data set(s), see the MVS Diagnosis: Tools and Service Aids manual.

The general sequence to use is as follows. Note that the START and STOP commands cannot be used for the writer proc, you must use the TRACE WTRSTART and TRACE WTRSTOP commands. The stackname specification is generally the name of the TCPIP started proc, except as noted in the specific component sections.
  1. Start the external writer

    TRACE CT,WTRSTART=writerproc
  2. Start the CTRACE. The required reply attaches the external writer so that it can be used to write the CTRACE records in addition to any other applicable options needed.

    TRACE CT,ON,COMP=SYSTCPxx,SUB=(stackname)
    R xx,WTR=writerproc,other_options,END
  3. Some traces will require other commands to be entered here, see the following sections for the type of trace being performed.
  4. Optionally verify that the trace started successfully.

    DISPLAY TRACE,COMP=SYSTCPxx,SUB=(stackname)
  5. Recreate the problem.
  6. If a command was needed at step 3, then that will need to be disabled here.
  7. Stop the CTRACE and disconnect the writer

    TRACE CT,ON,COMP=SYSTCPxx,SUB=(stackname)
    R xx,WTR=DISCONNECT,END

  8. Stop the component trace

    TRACE CT,OFF,COMP=SYSTCPxx,SUB=(stackname)
  9. Stop the external writer

    TRACE CT,WTRSTOP=writerproc,FLUSH




B. Collecting Multiple Traces with an External Writer
Multiple traces can be collected at the same time with external writers. This includes multiple types of traces(components) or from multiple sources (subcomponents).

If a low volume of activity is anticipated, these can all be directed to a single external writer proc. Start the writer before any of the traces, and stop all of the traces before stopping the writer.

But if higher volumes of data are expected (or if you are uncertain), it would likely be best to direct each to its own writer proc. In this case, the datasets used by these writers should ideally be directed to separate DASD volumes.




C. General Considerations for using internal buffers

Before having a trace written to the internal buffers as a part of a system dump, the buffers need to have enough space allocated. The size created when TCPIP starts is set by the CTIxxxyy PARMLIB member listed in the following table.
Type SUB PARMLIB Member Specification Location
Event Trace (SYSTCPIP) TCPIP name CTIEZByy BUFSIZE(xxxx) HVCOMMON
Event trace for TN3270 TN3270 server name CTIEZBzz BUFSIZE(xxxx) TN3270 private
Packet Trace (SYSTCPDA) TCPIP name (None) Twice the SYSTCPIP BUFSIZE HVCOMMON
Intrusion detection (SYSTCPIS) TCPIP name CTIIDSyy BUFSIZE(xxxx) HVCOMMON
OSA Network (SYSTCPOT) TCPIP name CTINTAyy BUFSIZE(xxxx) HVCOMMON
Dynamic routing
(SYSTCPRT)
OMPROUTE name CTIORAyy BUFSIZE(xxxx) OMPROUTE private
Resolver (SYSTCPRE) RESOLVER name CTIRESyy BUFSIZE(xxxx) RESOLVER private
IKE trace (SYSTCPIK) IKED name CTIIKEyy BUFSIZE(xxxx) IKED private

The current setting can be verified using the following command:

DISPLAY TRACE,COMP=SYSTCPxx,SUB=(tcpipname)

The steps to collect the trace are the same as listed for using an external writer, except that:
  • Steps 1, 7, and 9 are not performed.
  • Do not specify the WTR keyword and value in step 2.
To modify the size of the trace buffer, replace the ON keyword in the TRACE command to start the CTRACE with the desired size, for example:

TRACE CT,nnnM,COMP=SYSTCPxx,SUB=(tcpipname)

where nnn is the new buffer size in megabytes. The buffer size is subject to the minimum and maximum buffer size established for each component.

The TCPIP trace buffers that are in HVCOMMON (64-bit addressing) and will be included in all system dumps that have common storage, those listed as being in private need to have the corresponding address space included in the dump.
NOTE: In z/OS 1.12 and older releases, the TCPIP trace data will be written to the TCPIPDS1 data space associated with the TCPIP address space. To have the buffers included in the dump, add DSPNAME=('tcpipname'.TCPIPDS1) in the reply to the DUMP or SLIP command.

Be aware that this method of writing trace data to internal trace buffers may result in lost trace data as the occurrence of wrapping is very possible. The dump command should be issued very soon after the problem happens or the dump should be collected using a trap or slip issued by the support center.




D. Collecting a Packet (SYSTCPDA) Trace
See the general directions for using an external writer or for internal buffers. The reply to the TRACE command for the SYSTCPDA component should not specify any OPTIONS, JOBNAME, or ASID. For step 3, you need to start the Packet Trace processing in TCPIP.

VARY TCPIP,tcpipname,PKTTRACE,ON

Note that the above command will cause all IP traffic to be traced. Add specification of one or more filters to reduce the volume of data recorded.

There are may filters that can be specified on the PKTTRACE command, the complete list can be seen in the IP System Administrator's Commands manual. The most common filters to use are:
  • An IP address (IPADDR=aa.bb.cc.dd) or a masked address (IPADDR=aa.bb.cc.0/24 or IPADDR=aa.bb.cc.0,SUBNET=255.255.255.0). This will typically be the address of the remote system being traced.
  • A port number (PORTNUM=nnn). This will typically be a server port, either local or remote.
    NOTE: For z/OS 1.8 or earlier, PORTNUM is not available. In this case an ORed combination of SRCPORT and DESTPORT will be needed (see the last example below).
  • Abbreviated packet tracing (ABBREV or ABBREV=len), which reduces the amount of data recorded per packet. This is appropriate when diagnosing network or performance issues, but should not be specified when the full set of application level data needs to be traced.
  • Discarded packets are not traced or flagged by default. Specify the DISCARD=ALL option to see both normal and discarded packets, or DISCARD=* to see only the discarded packets.

NOTE: Any changes made to PKTTRACE settings will affect the records that applications using the NETMONITOR PKTTRCSERVICE will receive. Therefore, you may want to modify the filters to allow the application to continue to work while collecting diagnostic data (or shut the application down during this period), and change the settings back to what they were once the diagnostics have completed (including doing a TRACE CT,ON,... command).

When specifying multiple filters, those provided on a single PKTTRACE command are ANDed together (they all have to be true for the packet to be traced). The filters can be ORed together by specifying each on a series of PKTTRACE commands.
  • The following example logically ANDs the two parms:

    VARY TCPIP,tcpipproc,PKTTRACE,ON,IPADDR=10.9.8.7,PORTNUM=23
  • The following example logically ORs the two parms:

    VARY TCPIP,tcpipproc,PKTTRACE,ON,PORTNUM=21
    VARY TCPIP,tcpipproc,PKTTRACE,ON,PORTNUM=20
  • The following example is specifying a port filter without using the PORTNUM keyword (do not use SRCPORT or DESTPORT unless specifically directed or if using z/OS 1.8 or earlier):

    VARY TCPIP,tcpipproc,PKTTRACE,ON,SRCPORT=53
    VARY TCPIP,tcpipproc,PKTTRACE,ON,DESTPORT=53

For step 6, use the following commands:

VARY TCPIP,tcpipproc,PKTTRACE,OFF
VARY TCPIP,tcpipproc,PKTTRACE,CLEAR





E. Collecting an Event (SYSTCPIP) Trace
See the general directions for using an external writer or for internal buffers. There is no command needed for step 3.

There are many possible options that can be specified in the reply to the TRACE command, these are listed in the IP Diagnosis Guide. In general, the set to be collected will be provided by IBM Support. An exception is the SOCKAPI option, which is useful for debugging applications that use Sockets Macro Extended API or one of its derivatives ( EZASOKET calls, CICS Sockets API (Call format or C sockets), or IMS Sockets).

The IPADDR option will filter records collected to only those associated with the remote system addresses or subnets listed. Specifying a local address will result in no records being collected (with the exception of local-to-local connections).

The PORT option will filter records collected to only those associated with the local ports listed. Specifying remote port numbers will result in no records being collected (with the exception of local-to-local connections).

The JOBNAME or ASID keywords in the TRACE command response will limit tracing to only activity related to the listed applications. Wildcards are not used, the full JOBNAME must be specified.






F. Collecting an OSA Network (SYSTCPOT) Trace
See TechNote 1232599 for the OSA models and z/OS releases that support this function, and the general directions for using an external writer or for internal buffers. The reply to the TRACE command for the SYSTCPDA component should not specify any OPTIONS, JOBNAME, or ASID.
For step 3, you need to start the OSAENTA processing in TCPIP. Repeat for each OSA port if multiple OSAs are to be traced.

VARY TCPIP,tcpipname,OSAENTA,PORTNAME=osaport,ON,CLEARFILTER,NOFILTER=ALL

Note that the above command will cause all IP traffic to be traced. Add specification of one or more filters to reduce the volume of data recorded

There are many filters that can be specified on the OSAENTA command, the complete list can be seen in the IP System Administrator's Commands manual. You should typically include the CLEARFILTER keyword to remove any previous setting when starting a new trace, otherwise any specifications are used as additional filters (ie, more specific criteria).

The default action when no filters are specified is to not record anything, with the exception of discarded packets. If you do not want any filtering to be performed, specify NOFILTER=ALL.

The most common filters to use are:
  • An IP address (IPADDR=aa.bb.cc.dd) or a masked address (IPADDR=aa.bb.cc.00/24 or IPADDR=aa.bb.cc.dd,SUBNET=255.255.255.0). This will typically be the address of the remote system being traced.
  • A port number (PORTNUM=nnn). This will typically be a server port, either local or remote.
  • Abbreviated tracing (ABBREV or ABBREV=len), which reduces the amount of data recorded per frame. Most current OSA models will limit the amount of data to 224 bytes as a maximum (even if FULL is specified).
  • By default, frames discarded by the OSA are only traced if it is an unusual condition. ARP requests for IP addresses not owned by this OSA or frames with other MAC or IP addresses which are discarded will not be traced by default. Specify the DISCARD=ALL option to see all discarded frames, or DISCARD=NONE to not see any discarded frames.
Fort step 6, use the following commands to clear out any prior filter specifications and close the OSAENTA device:

VARY TCPIP,tcpipproc,OSAENTA,PORTNAME=osaport,OFF
VARY TCPIP,tcpipproc,OSAENTA,PORTNAME=osaport,DEL




Related information

EZZ4336I error starting OSAENTA trace

Document information

More support for: z/OS Communications Server
All

Software version: 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 2.1, 2.2, 2.3

Operating system(s): z/OS

Reference #: 1292013

Modified date: 05 March 2018