II08038: REQUESTING OPENEDITION MVS DATA TO DIAGNOSE PROBLEMS

APAR status

• INTRAN

Error description

• A loop, hang, or wait condition in a process using Unix System
  Services may require a diagnostic dump for analysis.  Shared
  file system problems also require that several types of data be
  collected before recovery actions are attempted.
  -
  For process hangs or loops:
  
  Use DISPLAY commands to display current active USS processes:
  
       D OMVS,A=ALL
  
   USER     JOBNAME  ASID        PID       PPID STATE
   OMVSKERN BPXOINIT 002A          1          0 1WI
   MVS      TCPIP    002B      65538          1 MR
   DCEKERN  DCEKERN  003A     262147          1 HK
   DCEKERN  DCEKERN  003A     262148     262147 HK
   DCEKERN  DCEKERN  003A      65541     262147 HK
   DCEKERN  DCEKERN  003A      65542     262147 HF
   DCEKERN  DCEKERN  003A          7     262147 HK
   DCEKERN  DCEKERN  003A          8     262147 HK
   TS65106  TS65106  0032          9          1 1RI
   TS65106  TS65106  0032         10          9 1CI
     LATCHWAITPID=         0 CMD=-sh
  
  The display output shows all of the active OMVS processes,
  ASIDs, process IDs, parent process IDs, and Jobnames.
  
  Display global resource serialization information to see
  possible latch contention:
  
       D GRS,C
  
  This display may show latch contention, which could be the
  cause of the problem. You should include the address space of
  the process causing the latch contention in the dump.
  
  Display jobs that may be waiting for dub processing by OMVS.
  
       D OMVS,A=DUBW
  
  If message BPXM056E is seen, consider issuing the
  'F BPXOINIT,RECOVER=LATCHES' command.
  (see the appropriate level of the z/OS MVS System Commands
  manual for details of this command).
  
  Because you are dumping multiple address spaces, multiple data
  spaces, and multiple storage data areas, you may need a much
  larger dump data set defined than is normally used for dumping a
  single address space.  You should preallocate a very large
  SYS1.DUMPnn data set.  Alternatively, using Dynamic Dump
  Allocation will automatically allocate a dump of sufficient
  size.
  
  Below is a sample dump command for dumping multiple address
  spaces using jobname and / or asid:
  -
  DUMP COMM=(Descriptive name for this OMVS dump)
  -
  R rn,JOBNAME=(OMVS,ZFS),DSPNAME=('OMVS'.SYS*,'OMVS'.BPX*),CONT
  R rn,SDATA=(ALLNUC,PSA,CSA,LPA,TRT,SQA,RGN,GRSQ,SUM),END
  
  Note that the OMVS dataspaces are required to diagnose many USS
  problems.
  
  If it is believed that NFS or DFSKERN are involved in the
  problem, please add the appropriate jobname(s) to the JOBNAME
  parameter in the DUMP command.
  
  For more information on the DUMP command, particularly on
  specifying a large number of operands, see MVS System Commands.
  
  After the dump completes, you will receive an IEA911E or IEA611E
  message indicating whether the dump was complete or partial.  If
  partial, check the SDRSN value.  If insufficient disk space is
  the reason:  delete the dump, allocate a larger dump data set,
  and request the dump again.  For more details on message IEA911E
  or message IEA611E, see MVS System Messages.
  -
  Shared file system problems:
  If a shared file system SYSPLEX problem should occur, the
  following steps should be followed to collect the necessary
  documentation.  Issue the following console commands.  The
  commands will be routed to all systems in the plex as required:
  -
  RO *ALL,D GRS,C
  -
  RO *ALL,D OMVS,W
  -
  RO *ALL,D OMVS,MF=ALL
  -
  RO *ALL,D OMVS,F
  -
  F BPXOINIT,FILESYS=DISPLAY,GLOBAL
  -
  RO *ALL,F BPXOINIT,FILESYS=DISPLAY,EXCEPTION
  -
  Capture SYSLOGs for every system in the SYSPLEX sharing HFS or
  zFS datasets.  A merged log is preferable.  The SYSLOGs need to
  contain the command responses from the above commands.
  -
  Collect a 'checkpoint' dump on all systems with the following
  command (issued from any one system in the SYSPLEX):
  -
  DUMP COMM=(DUMP OF OMVS)
  .
  R nn,JOBNAME=(OMVS,ZFS),DSPNAME=('OMVS'.SYS*,'OMVS'.BPX*),CONT
  R rn,SDATA=(CSA,LPA,TRT,RGN,SUM,SQA,ALLNUC,PSA,GRSQ,COUPLE),CONT
  R rn,REMOTE=(SYSLIST=*('OMVS','ZFS'),SDATA,DSPNAME),END
  
  If it is believed that NFS or DFSKERN are involved in the
  problem, please add the appropriate jobname(s) to the JOBNAME
  parameter and the SYSLIST parameter in the DUMP command.
  
  -
  
  To activate the OMVS ctrace, use the following command:
  
  TRACE CT,64M,COMP=SYSOMVS
  
  You may optionally use the CTIBPXxx member to control the
  SYSOMVS ctrace.
  
  You will need to reply to a WTOR to specify the desired options:
  
  
  *05 ITT006A SPECIFY OPERAND(S) FOR TRACE CT COMMAND.
     R 05,OPTIONS=(ALL),END
  
  See 'SYS1.PARMLIB(CTIBPX00)' for the possible ctrace options
  for OMVS.
  
  The ctrace buffers up to the buffer size will be contained in
  any dump containing the OMVS address space and SYSZBPX1
  dataspace, captured while the ctrace is on.
  
  Turn off the SYSOMVS ctrace with:
  
  TRACE CT,OFF,COMP=SYSOMVS
  
  To check your trace options, use:
  
  D TRACE,COMP=SYSOMVS
  -
  
  Ctraces covering a time interval too long to fit in a the
  maximum ctrace buffer size can be captured to an external
  writer.
  
  1.  Start an external writer for OMVS:
          TRACE CT,WTRSTART=XWOMVS
  2.  Turn on the OMVS ctrace:
          TRACE CT,ON,COMP=SYSOMVS
       reply:
          R XX,WTR=XWOMVS
          R XX,OPTIONS=(ALL),END
  3. Recreate the problem.
  4. Stop the OMVS CTRACE by issuing:
          TRACE CT,OFF,COMP=SYSOMVS
  5. Stop the external writer by issuing
          TRACE CT,WTRSTOP=XWOMVS
  .
  Sample external writer procedure (See MVS Tools and Service Aids
  in the  chapter on component trace for details).
  .
  Example: Cataloged Procedure for an External Writer using
  multiple trace datasets. The procedure places trace data on two
  DASD data sets. The procedure is placed in member XWOMVS of
  SYS1.PROCLIB. Please update the DSNAME and VOLSER as required
  for your installation
  
  //XWOMVS   PROC
  //IEFPROC  EXEC PGM=ITTTRCWR,REGION=32M
  //TRCOUT01 DD DSNAME=SYS1.CTRACE1,VOL=SER=TRACE1,UNIT=DASD,
  //         SPACE=(CYL,500),DISP=(NEW,KEEP),DSORG=PS
  //TRCOUT02 DD  DSNAME=SYS1.CTRACE2,VOL=SER=TRACE2,UNIT=DASD,
  //         SPACE=(CYL,500),DISP=(NEW,KEEP),DSORG=PS
  
  
  Example: Cataloged Procedure for an External Writer using
  multiple trace datasets  with &SYSNAME symbolic to facilitate
  tracing on multiple lpars. The procedure places trace data on
  two DASD data sets. The procedure is placed in member XWOMVS
  of SYS1.PROCLIB. Please update the DSNAME and VOLSER as required
  for your installation
  
  //XWOMVS   PROC
  //IEFPROC  EXEC PGM=ITTTRCWR,REGION=32M
  //TRCOUT01 DD DSNAME=SYS1.&SYSNAME..CTRACE1,VOL=SER=TRACE1,
  //         UNIT=DASD,
  //         SPACE=(CYL,500),DISP=(NEW,KEEP),DSORG=PS
  //TRCOUT02 DD DSNAME=SYS1.&SYSNAME..CTRACE2,VOL=SER=TRACE2,
  //         UNIT=DASD,
  //         SPACE=(CYL,500),DISP=(NEW,KEEP),DSORG=PS
  
  .
  ********** NOTE ***********
  The dispatching priority of the external writer should be at
  least equal to, and preferably greater than the dispatching
  priority of the component being traced.  So, the external writer
  for OMVS ctrace should have a dispatching priority at least
  equal to the OMVS task. Also, please give the external writer
  task as large a region as possible to minimize the chance of
  losing a buffer. The external writer will cut a logrec entry
  every time it fails to successfully write the buffer to the
  trace data set; please test your external writer procedure prior
  to using it in a production environment.
  .
  
  (End of documentation collection information for USS.)
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels