II07564: MUSTGATHER: READ FIRST FOR VTAM COMMON (CSA/ECSA) STORAGE DIAGNOSIS FOR Z/OS COMMUNICATIONS SERVER

APAR status

• Closed as canceled.

Error description

• ***************************************************************
  * PROLOG:                                                     *
  ***************************************************************
  
  This is one of the APARs in a series of INFO APARs created to
  assist you in the initial diagnosis of VTAM storage problems.
  There are currently four INFO APARs in this series and others
  may be added if needed.  The current APARs are as follows:
  
  II06752  VTAM Storage Diagnosis - An Introduction/Overview
  II04548  VTAM Storage Diagnosis - Documentation Requirements
  II07563  VTAM Storage Diagnosis - Private Storage Problems
  II07564  VTAM Storage Diagnosis - Common Storage (CSA) Problems
  II09086  VTAM Storage Diagnosis - ABEND0C4 in ISTORCFB
  II09095  VTAM Storage Diagnosis - MSGIST565I During Activation
  --------------------------------------------------------------
  This APAR discusses the diagnosis of VTAM CSA/ECSA Storage
  problems.  You have reached this point as a result of being
  guided here from II06752 or by knowing from prior experience
  that you have a CSA storage problem.  You also should have
  gathered the necessary documentaion as stated in II04548.
  
  It is also assumed you have used the VTAM Storage Estimates
  Worksheet to determine the amount of CSA/ECSA required for your
  network configuration and have defined enough CSA/ECSA in the
  IEASYS00 member of PARMLIB for VTAM to co-exist with all the
  products who share this system resource.  Past experience has
  proven you can spend many hours diagnosing a problem when no
  problem exists and VTAM validly ran out of storage because
  the requirements were under estimated.
  
  
  ***************************************************************
  * OVERVIEW:                                                   *
  ***************************************************************
  
  VTAM consists of OCO code which is considered IBM Confidential.
  Therefore, the following subject matter will discuss only the
  information that is available in the customer copies of the
  VTAM manuals.  The main intent here is to help you identify
  what is filling up storage so search arguments can be created
  to find the fixing APAR/PTF in IBMLINK or RETAIN.  We can do
  that without in-depth discussion of the internal workings of
  the OCO code.
  
  We have found it is much more efficient to MVS GETMAIN a pool
  of storage, consisting of one or more pages, and then managing
  it internally rather than issuing an MVS GETMAIN for each piece
  of storage required.  These pools of storage can be expanded
  and contracted as needed.  To accomplish this, we have a set of
  VTAM Storage Management routines which are called using the
  following internal macros:
  
  GETBLK/FREEBLK to acquire and release storage in the SPANC
  pools.  These pools are internal to VTAM and the user does not
  define them in any way.  SPANC pools reside in both Private
  and CSA storage.
  
  REQSTORE/RELSTORE to acquire and release a buffer from one of
  the fixed length buffer pools that reside in CSA only.  These
  buffer pools are user defined in the VTAM START list, ATCSTRxx.
  
  VTALLOC/VTFREE to acquire and release all other variable length
  pieces of storage used internally by VTAM.  In this case only,
  each VTALLOC or VTFREE does expand into an MVS GETMAIN or MVS
  FREEMAIN and this storage resides in both Private and CSA.
  
  The invocation of each of these macros creates an entry in the
  VTAM Internal Trace (VIT) if the SMS option has been specified.
  The format of these entries can be found in the VTAM Diagnosis
  manual under REQS, RELS, GBLK, FBLK, VTAL and VTFR.  The SMS
  VIT entries can prove useful in determining if storage is being
  acquired and never being freed.  They also point to the module
  requesting or freeing the storage.
  
  We will start the diagnosis with the easiest area of storage
  first and work our way on through, covering the most difficult
  area last.  It is suggested you have a copy of the VTAM Data
  Areas, VTAM Diagnosis and IPCS Command Reference manuals handy
  for reference as needed.  For MVS/ESA/430 and above, VSMDATA
  Common Storage Tracking (DIAG=01) can be of great use.
  
  ***************************************************************
  * Important notes - Please read before continuing on.         *
  ***************************************************************
  
  1. Since CSA is shared by all ASIDs in the system, you need to
     insure the problem is being caused by VTAM.  VTAM issues a
     lot of storage shortage messages, but could just be a victim
     in the problem....not the bad guy.
  
     To be sure, check VTAM's current usage in the ATCCACSA field
     in the ATCVT.  The ATCVT is pointed to by x'408' in the dump.
     Compare the value of ATCCACSA to the normal Current CSA in
     the output of a 'D NET,BFRUSE' command.  If the value is a
     lot higher, continue on with the diagnosis.  If not, the dump
     may have been taken too late and did not capture an increase
     in VTAM CSA usage or the problem could be caused by another
     job and VTAM is just detecting it and telling you about it
     with our out of storage messages.
  
  2. If you code a CSALIM value and your network has been growing
     recently, you may have just outgrown the value you set.  It
     might be a good idea to use the Storage Estimates Worksheet,
     rather than spending a lot of time looking for a problem if
     none exists.  If you decide to use the Worksheet, compare the
     estimated value with the CSALIM value you have set and make
     any necessary adjustment.  You can spend a whole lot of time
     looking for a problem when there is none.
  
  3. If you are satisfied that everything is OK and you have
     captured a real VTAM CSA problem, continue on.
  
  
  
  ***************************************************************
  * SPANC Pool Storage - GETBLK/FREEBLK                         *
  ***************************************************************
  
  SPANC storage is allocated in both PRIVATE and CSA.  The first
  thing to do is access the dump and run the SPANC CLIST from the
  IPCS command screen.  From the output, you will be able to see
  how many pages are allocated to each SPANC.  If you see any of
  them with a high page count you will want to look into them in
  greater detail.  This page count will vary from one system to
  another based on network size and activity.  As a general rule
  of thumb, anything with a page count over 1000 for any one pool
  could indicate a problem and should be investigated.  If none
  of the SPANC pools appear to be out of line, leave the SPANC
  Pool section and go on to the next topic.
  
  Let's say, for instance, you are diagnosing a CSA problem and
  you see there are 2500 pages allocated to the RUPECOMM SPANC.
  From this, you might guess this pool contains RUPE control
  blocks and resides in CSA (Common) storage.  This would be a
  true assumption.
  
  However, some of the SPANC names are not descriptive as to what
  control blocks they contain nor where they reside in storage.
  The SPANC CLIST in VTAM VER3 does not provide information about
  which area of storage the pool resides.  VTAM VER4 and above
  does tell you where the pool resides.
  
  Take note of the pool(s) with the largest page count.  If the
  page count for any of the pools is well over 1000 pages, you
  have probably found a problem area.
  
  You now have enough information to do an initial search to see
  if you have encountered a known problem.  You will want to use
  the VTAM Release and SPANC pool name in your search.  If you
  don't see any good hits on your search with the information
  known at this point, continue on to broaden your search.
  
  The next thing to do is run a more granular version of the
  SPANC CLIST which will list out the beginning address of each
  extent that is allocated to the pool in question.  Extents for
  SPANC pools will be either x'1000' or x'10000' bytes depending
  on the pool involved.  Enter: SPANC POOL(poolname) MAP.
  
  You will now need to know how to lay out the storage pointed to
  by the output of the above command.  Please refer to the VTAM
  Data Areas manual for the layout of the control blocks in the
  following discussion.
  
  SPANC storage is GETMAINed using the VTAM VTALLOC macro.  This
  will cause a x'10' byte SMHDR to be added to storage beginning
  at the page boundary.  If you feel you need more information
  on VTALLOC processing at this point, it is contained in the
  below in the section following this one.
  
  Immediately following the SMHDR is a PGHDR.  It contains a
  forward and backward pointer to other pages of this pool if
  any exist.  This is followed by two halfword fields.  The first
  one is a relative byte offset to the first FBQE representing
  available piece of free storage in this extent.  The offset is
  relative to the beginning of the PGHDR in VER3 and relative to
  the first byte following the PGHDR in VER4 and above.  The
  second halfword is a counter of elements in use in this extent.
  You should always expect to see a value in this counter.  When
  it goes to zero, the pool will contract and this extent will
  be freed.
  
  Once the SMHDR and PGHDR have been added, a SPANC processing
  module will go through and set up the rest of the extent for
  future use.  This consists of carving the extent into elements
  of a predefined length for each given pool and building a FBQE
  at the beginning of each element.  FBQEs are x'8' bytes long
  and are easy to spot.  They will always start on a doubleword
  boundary and the second word will contain x'FFFFFFFF'.  The
  first word is a relative offset to the next FBQE.
  
  When one of these freed elements is allocated as the result of
  a GBLK request, the first FBQE pointed to by the PGHDR is then
  converted to a GBHDR.  GBHDRs are also easy to spot in a dump.
  They are x'8' bytes long.  The first word points back to the
  PGHDR for this extent and the second word contains x'80000000',
  indicating it is now allocated.
  
  When diagnosing the contents of the extent, it is a good idea
  to start by locating the SMHDR, PGHDR and all of the allocated
  GBHDRs.  The storage immediately following the GBHDR now needs
  to be looked at in greater detail.  This is the key to finding
  what is filling up storage and completing your search argument
  to find the fixing APAR/PTF.  The storage could contain a VTAM
  control block.  Check the first byte after the GBHDR against
  the list of VTAM Control Block Id Codes in the Appendix of the
  VTAM Data Areas.  If you get a match on an Id Code and the
  storage matches up with the layout of that control you likely
  have found the problem.  Add that control block name to your
  search argument.
  
  However, the storage could contain control vectors, RU data,
  sense data or something else totally unrecognizable to you.
  Especially if the SPANC name begins with 'UTIL'.  Utility pools
  are just what their name implies.  Someplace to put stuff that
  needs to be in SPANC storage, but a specifically named SPANC
  has not yet been created for this purpose.  You may need to
  call the VTAM Support Center for further assistance.  LVL2 may
  require a VIT with OPT=SMS in order to identify the elements
  filling storage.
  
                               *****
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

• closing cancel         (DB2INFO)
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels