II14437: DIAGNOSTICS DOCUMENTATION FOR OAM OBJECT SUPPORT RELATED ISSUES. ( 5695DF180 )

APAR status

• INTRAN

Error description

• The purpose of this informational apar is to help customers
  in deciding what documentation is needed for Problem
  Determination(PD).  For OAM Object support issues there is
  specific documentation that is needed for diagnostics.  For
  your understanding, the OAM component of DFSMS is really
  more like two components ( 5695DF180 ):
  1) OAM Object Support:
     Uses DB2 to manage "Objects" on DB2 DASD, Tape, or Optical.
     (Described in the "Object PISA" manual SC35-0426-xx)
  2) OAM ATL Support - IBM Automated Tape Library Support
     Provides macros like MOUNT and DEMOUNT in addition to
     displays, enters, and ejects, etc.
     (Described in the "Tape PISA"  SC35-0427-xx)
   * Knowing the difference will help you in diagnosing the issue;
  a separate Informational apar will be created for OAM Tape
  library support customers(xxxxxxx).  There are two common
  problems in collecting documentation for  OAM Level 2 for PD,
  they are:
  1) Providing JOBLOGs instead of MVS SYSLOGs (System Logs).
     Joblogs are certainly useful, but only provide OAM related
     messaging, whereas the MVS SYSLOGs provide all systems
     messages, so L2 support will always need MVS SYSLOGs.
  2) Dumping the OAM address space instead of the invoking
     application's address space.  "OAM", as a component of
     MVS, provides several macros that are included in the
     Application's code and therefore require the Application's
     address space to be dumped.  For example, for OAM Object
     support, an application like OnDemand invokes OAM's OSREQ
     macro.  The OAM address space would be required in addition
     to OnDemand's if the Object resided on Optical or Tape.
     For ATL support, it is similar in that OAM's MOUNT macro is
     included in the invoking job's code and therefore the Job's
     ASID must be dumped.  To add confusion, it is possible
     for the invoking Job to also be OAM as in the case of an
     "Object" being accessed in an ATL.
  ----------------------------------------------------------------
  Section A.
    This section covers dumps, and how to get the correct dump for
  OAM PD.  For an OAM abend, provide an abend dump if available,
  and the MVS syslog from OAM startup to the point of failure. If
  for any reason there is no abend dump(eg. Dump suppression),
  setup the following slip:
   SLIP SET,A=SVCD,COMP=xxx,
  SDATA=(LPA,CSA,ALLNUC,NUC,GRSQ,SQA,LSQA,SWA,PSA,TRT,RGN,SUM),END
    where xxx = to the abend completion code.
       (EG. 0C4, 0C1, 878, B37... etc.)
   Set a matchlim = 2 if there are multiple dumps for that
  failure.  The initial(1st) dump is the dump we will want to look
  at if there is a chain of abend dumps available for the failure.
  Rerun the failing job or wait until the slip hits, or wait until
  another occurrence of the failure and send in the slip dump(s)
  and the MVS syslogs at the time of the failure.  Here is a
  charted breakdown of what to dump for OAM PD:
  ================================================================
             | OAM | Application | DB2 | MASTER |
  Function   |     |    job      |     |        | notes
  ---------------------------------------------------------------
  OSREQ on   |     |     yes     | yes |        |
  DB2 DASD   |     |             |     |        | see Note 1
  ---------------------------------------------------------------
  OSREQ on   |     |             |     |        |
  Tape or    | yes |     yes     | yes |        |
  Optical    |     |             |     |        | see Note 1
  ---------------------------------------------------------------
  OSMC: SG,  |     |             |     |        |
  Recovery,  | yes |             | yes |        | see Note 1
  or Movevol |     |             |     |        |
  ---------------------------------------------------------------
  ATL Mount  |     |     yes     |     |  yes   | Only need OAM
  or Demount |     |             |     |        | if Object too
  ---------------------------------------------------------------
  ATL Eject, |     |             |     |        |
  Enter, or  | yes |     yes     |     |        |
  Audit      |     |             |     |        |
  **  Note 1: DB2 has several Address Spaces:
            Master, IRLM, etc
   * Get a dump of the OAM address space using the OAM dump
     command. If you can identify the address space ID, specify
     that ASID operand in the F OAM dump command. Examples:
          F OAM,DUMP,ALL
          F OAM,DUMP,ASID,asid1,asid2,...
          F OAM DUMP,JOBN,jobname1,jobname2,...
   * If the OAM dump is not responding, please get an
    SVC dump(console dump) with the following SDATA:
     SDATA=(TRT,LPA,CSA,PSA,GRSQ,SUM,ALLNUC,SQA,LSQA,RGN,SWA)
             What ASID(s) to Dump for OAM diagnostics:
   * Example at MVS console:
    DUMP COMM=(OAM hang on MVSB)
            when prompted, reply:
    JOBNAME=(OAM,DT1BMSTR,DT1BDIST,DT1BDBM1,DB1BIRLM)
     Note. DT1BMSTR,DT1BDIST,DT1BDBM1,DB1BIRLM is just an example.
           In this case, have the customer supply the correct DB2
           parameters as needed to dump DB2's ASID's.
                  <or>
  
    ASID=(hex_OAM_ASID,hex_DB2_ASID,hex_DB2_ASID,etc.)
  
                  < and >
    SDATA=(TRT,LPA,CSA,PSA,NUC,GRSQ,SUM,ALLNUC,SQA,LSQA,RGN,SWA)
  ----------------------------------------------------------------
  Section B.
    For all OAM Object support issues, refer to the MVS syslogs
  from the last time OAM was started to when the issue has been
  identified as a hang, a loop, or an abend.  The best starting
  point is to look for any OAM messages from startup to the the
  point of the failure for any information.  OAM messages are
  prefixed with a "CBR"
      EG. CBR9103I, CBR3300I, etc.
    Other information to look for might be HW related failures,
  or for DB2 issues that may impede or stop OAM function.  Look
  for DB2 messages for any messages, and those are typically
  prefixed with a "DSNT" prefix.   Insure that DB2 is not
  degraded or that you are not running a DB2 utility to OAM
  tables when OAM is running.   This can lead to unpredictable
  results.
   EG. Search on "Degraded" or "DB2" or "DSNT" etc.
      ** Save the MVS syslogs if L2 is engaged for support.
      ** DO NOT CANCEL OAM YET or any OAM processes.
   For a hang or a loop conditions, issue all the listed
  commands in sections C. and get a set of dumps as listed in
  section A. for the issue that you may be experiencing with
  OAM, then wait 5 minutes and re-issue the same sets of
  commands and get a second set of dumps.   This tells us if
  OAM is really hung or in a loop, or if there is just a slow
  down in  processing that may appear to be a hang or loop
  condition.
  ----------------------------------------------------------------
  Section C.
    If you have not determined what the failure is at this
  point, then gather documentation for L2 support.  Here is
  a typical list of displays we will like, should this to be
  a loop or hang condition with OAM...
   *  F OAM,QUERY,ACTIVE
     (OAM active tasks in progress)
   *  F OAM,QUERY,WAITING
     (OAM tasks waiting)
   *  F OAM,QUERY,ACTIVE,DETAIL,ALL
     (OAM active tasks in progress for ALL OAM tasks with
      detailed output)
   *  F OAM,QUERY,WAITING,DETAIL,ALL
     (OAM tasks waiting for ALL OAM tasks with detailed output)
   *  DISPLAY R,L,KEY=OAM
     (Anything waiting for an operator?)
   *  DISPLAY SMS,OSMC
     (Summary of ALL OSMC tasks running if running OSMC)
   *  D SMS,STORGRP(SGNAME),DETAIL
    (if there is a specific SG that we want to focus on during
    OSMC or an OSMC task such as MOVEVOL.)
   *  D SMS,STORGRP(ALL),DETAIL
    (Summary of all OAM SGs.)
   *  DISPLAY SMS,OAM
     (Lib/drv's online?  in use?  reads waiting?)
   *  DISPLAY SMS,LIBRARY((ALL)),DETAIL
     (Detailed info on libraries)
   *  DISPLAY SMS,DRIVE((ALL)),DETAIL
     (Drive info.)
   *  DISPLAY SMS,VOL(VOLSER)
     ( If there is a failing volume involved; or the SPUFI
       output for the volume from the applicable vol table.)
  ----------------------------------------------------------------
  Section D.
   Here is a listed set of documentation and what to do next.
    * MVS syslogs showing all the display information,
    to about 10 minutes after the OAM dumps all the way up
    to when OAM was started successfully prior to the
    failure/hang/loop.
    * Dumps
      -OAM Dump, abend dump, slip dump, or console dumps
    * CBROAMxx member if needed.
    * Open a problem record with IBM and use the following FTP
      instructions to send out the documentation.  Here is the
      URL to got the FTP FAQs:
       http://techsupport.services.ibm.com/390/tcprocs.html
    * Note, if the failure is more complex than expected,
      additional  slips or tracing may be needed, if so the
      IBM support team  will provide trace/slip instructions.
  

Local fix

• After documentation has been gathered and you have determined
  that OAM is really hung, cancel OAM and restart OAM.
  

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels