NetViewGather: System Automation for z/OS Tracing

Overview
* Automation Manager Trace Facility - Component Trace
* AOCTRACE
* INGHIST
* INGAMS Diagnostic options
* Tracing for ProcOps and BCPii
* Tracing for IOOPs - Component Trace

Automation Manager Trace Facility

The MVS Component Trace that is used to trace the control flow of a process. This trace is used when requested by service. Parameters on internal Rexx execs in the manager may require component trace.

This trace requires an External Writer. Before the trace can be activated the trace data set must be allocated. The recommended data set name is: 'SYS1.HSA.sysname.CTRACE01'

An appropriate JCL is provided in the SINGSAMP SA z/OS sample library as the member HSACTWR. In addition, the JCL of the component trace's external writer must be cataloged in SYS1.PROCLIB.

To activate the trace on a z/OS console, enter the command:
TRACE CT,ON,COMP=HSAAM

When the system responds with a WTOR for the trace options, reply as follows:
xx,WTR=HSACTWR,OPTIONS=(ALL|set_of_options),END

Deactivate the trace with the command:
TRACE CT,OFF,COMP=HSAAM

This command is automatically issued when the last automation manager in a single image terminates while the trace is active.

Before the trace records can be analyzed by IPCS the trace data set or sets must be made available to IPCS. This is done with the operator command:
TRACE CT,WTRSTOP=HSACTWR

If the related message indicates that more than one data set contains trace data, you must use the IPCS command COPYTRC to combine the data sets into a new one.

You can find more information on the trace in the manuals here:
https://ibm.biz/BdqXdu

The Trace data can be analyzed using IPCS command CTRACE.

AOCTRACE
The AOCTRACE command turns the SA z/OS system operations debugging feature on or off. AOCTRACE allows you to activate:
* Debugging globally
* Debugging for specific REXX(TM) EXECs
* REXXTRACE for a specific REXX EXEC
* Tracing for a particular message

The AOCTRACE command can be issued:
With no parameters - this will pull up AOCTRACE panel AOFKAANL, and shows all current AOCTRACE activities.

From the NetView/SA stylesheet - trace may be requested for initialization problems and therefore must be started from the stylesheet being used... CNMSTYLE/AOFSTYLE.

As a linemode command - enter command text on the NetView command line AOCTRACE ON/OFF or other parms as shown below:

• Service will most often ask for a Global AOCTRACE.
  (AOCTRACE ON/OFF) to trace when a REXX exec is invoked. Message AOF700I is used to identify the trace record in the Netlog
  
  The next step after Global trace is typically DEBUG for a specific module.
  DEBUG shows more information then REXX Trace - such as the contents of a PIPE SAFE or PIPE KEEP. REXX Trace will not show this. The Change Team most often wants DEBUG trace and will specifically request REXX trace if needed. DEBUG is the first level of module level trace to run.
  AOCTRACE rexx_exec ON/OFF
  This trace is written to the Netlog.
  
  Service may also ask for a AOCTRACE for a specific REXX exec
  (AOCTRACE rexx_exec ON/OFF,rexx_options) where rexx_options may be I,R, etc. This trace is not as useful as DEBUG as it does not show the contents of a PIPE SAFE or PIPE KEEP. A REXX trace will be requested by the change team if needed. Otherwise, DEBUG is the best trace to run.
  This trace will also be written to the Netlog.
  
  The other option requested may be a message trace
  (AOCTRACE msgid ON/OFF). This trace is also written to the Netlog.

There is a video attached demonstrating how to activate AOCTRACE Global Debug and an example of what the trace looks like in the netlog.

There is a video attached that demonstrates how to activate REXX module Debug tracing and an example on what the trace looks like in the netlog.


Running AOCTRACE at initialization time
You can run GLOBAL TRACE (AOCTRACE ON) at NetView initialization time by setting 'COMMON.AOFDEBUG = Y'  in CNMSTGEN/CNMSTUSR.
This will activate the trace when NetView/SA is started up.

You can also set up a AUXINITCMD in CNMSTGEN/CNMSTUSR to issue a AOCTRACE command for program specific trace like this:
AUXINITCMD.SATRACE=AOCTRACE AOFRANT3,ON,R

There are several places for additional help and more details on the available AOCTRACE options:
* HELP AOCTRACE
* Help (PF1) is available from the AOCTRACE panel.
* Section 2.1.7 AOCTRACE in the SA for z/OS Operator's Commands manual.


INGHIST

The INGHIST command shows history information about the work item processed by the automation manager. The REQ=LOG option (INGHIST REQ=LOG) is used to display a log with automation manager messages. These are automation manager messages that are written to the system logger.

There is help available for the INGHIST command (HELP INGHIST), and from the INGHIST Panel. Also, reference APPENDIX1.3.3.4 How to Use a Log with INGHIST in the System Automation for z/OS v3r3 User's Guide.

This information is valuable to customers and service for troubleshooting resource status problems.

INGAMS Diagnostic options

The INGAMS command will pull up panel INGKYAM0. Issuing option 'D' against the PAM will invoke the Diagnostic panel (INGKYAM4).

The Diagnostic panel can be used for the following:

* Writing a snapshot of the state image queue
* Starting or stopping recording
* Displaying work item statistics

There is help available on the INGAMS diagnostic panel as well as APPENDIX1.3.2.5 How to Use the Diagnostic Option of INGAMS in the System Automation for z/OS V3R5 User's Guide:
https://ibm.biz/BdqXdL

Service will often request this information for missing workitems or status updates that cause error messages (ING008I) or resources to be in undesired status.

The recording dataset requires preallocating with RECFM FB, LRECL 80; size is highly variable and dependent on size of network and length of recording time. The recording dataset is sequential and should have sufficient size to prevent errors as there is no wrapping.

The recording can be started and stopped via the INGAMS command as follows:

• INGAMS DIAG REQ=STOPREC OUTMODE=LINE

  INGAMS DIAG REQ=STARTREC DSN=rec_file OUTMODE=LINE

The recording dataset will have to be manually stopped and restarted each day if the recording will run for several days waiting for a problem to occur. If this is the case, set up the following REXX to run once a day on a timer and invoke the INGAMS commands.

Here is a sample for the REXX:

• /* REXX to Stop and Start a INGAMS recording */
  'INGAMS DIAG REQ=STOPREC OUTMODE=LINE'
  'INGAMS DIAG REQ=STARTREC DSN=rec_file OUTMODE=LINE'
  Exit






 

Here is the command to set up a CHRON timer to run the rexx every 24 hours at 15:00. The REXX will run on Task JOHNA.

• CHRON AT=(15:00:00) EVERY=() ROUTE=JOHNA COMMAND=JAREC

This will stop and restart the Recording each day and prevent the log from wrapping until the problem scenario is captured. The same recording dataset can be reused until the problem is re-created.

The Recording will be stopped if the Agent is recycled. An AuxInitCmd can be set in CNMSTYLECNMSTGEN to issue the command to start the recording, or run the REXX on a timer if needed. Here is an example of a AuxInitCmd to set a timer to run every 24 hours at 15:30 to issue command JAREC on Task AUTO1:

• * Run a rexx every day that starts and stops INGAMS
  * recording.
  AUXINITCMD.INGAMS = CHRON AT=(15:30:00) EVERY=(),
  ROUTE=AUTO1 COMMAND=JAREC *

Here is information on INGAMS that includes Diagnostic Information from section 2.1.31 of System Automation for z/OS Operator's Commands manual:
https://ibm.biz/BdqXdC

Tracing for ProcOps and BCPii

ProcOps SNMP connections and BCPii are traced as follows.

First, global tracing of all communications related parts of ProcOps-SNMP or BCP Internal Interface used AOCTRACE as follows: AOCTRACE compname ON

Where compname is one of the following:

• INGRX810 for the INGHWCMD command list
  INGHWCOM for Communications module

To disable the global trace mode issue command: AOCTRACE compname OFF

This will trace all currently active connections to SE and HMC's, and can produce a large amount of output in the Netlog.

Next, tracing communication to a specific processor or CPC can be accomplished with these commands:

• INGHWCMD proname TRACE ON/OFF
  INGHWCMD proname UTRACE ON/OFF

where proname is the name of the target hardware or processor that is defined in the SA z/OS customization dialog, and specifying UTRACE traces only the user session of the operator that issued the command. This trace can also produce a large amount of output.

Finally, the INGHWCOM communications module supports the First Failure Data Capture debug aid, provided by NetView for high level language command processors, written in PL/I or C. In case of an INGHWCOM abnormal end, message CNM983E, CNM998E, or CNM999E is issued.