APAR status
Closed as canceled.
Error description
This informational APAR will describe documentation needed to diagnose most OMPROUTE problems and list some of the common problems. Before reporting a problem to support, please examine SYSLOGD, JES MSG output and the console log for errors. DOCUMENTATION: Most OMPROUTE problems are best documented by obtaining an OMPROUTE -t2 -d1 trace and by providing a dump of the TCPIP and OMPROUTE address spaces. For IPv6, add -6t2 and -6d1 to the trace options. A trace from startup is ideal because some information is only shown in the trace at startup and the time for problem determination and resolution is quicker when the trace captures the entire flow of events rather than just a small subset of events. An OMPROUTE trace from startup can be enabled by coding the trace options after the forward slash in the PARM field of the OMPROUTE catalogued procedure. An example of the recommended OMPROUTE catalogued procedure can be found at this URL: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS /f1a1b470/2.5.1?SHELF=EZ2ZO10J.bks&DT=20070604132827 If a trace cannot be enabled from startup, then the following commands can dynamically enable and disable tracing. To enable: /F omproute,TRACE=2 (TRACE6=2 for IPv6) /F omproute,DEBUG=1 (DEBUG6=1 for IPv6) To disable: /F omproute,TRACE=0 (TRACE6=0 for IPv6) /F omproute,DEBUG=0 (DEBUG6=0 for IPv6) Trace output is sent to one of the following locations: 1) destination referenced by OMPROUTE_DEBUG_FILE environment variable which is coded in the STDENV DD dataset 2) STDOUT DD - but trace output is only output to this location if OMPROUTE_DEBUG_FILE is NOT defined, and the trace is started at initialization. 3) /{TMPDIR}/omproute_debug - (TMPDIR is usually /tmp) By default OMPROUTE will create 5 debug files, each 200K in size, for a total of 1 Meg worth of trace. The size and number of trace files can be controlled with the OMPROUTE_DEBUG_CONTROL environment variable. A 1 Meg trace will only cover a few minutes, so it is highly recommended that the size and number of trace files be increased. APAR PQ76172 adds new functionality to the OMPROUTE CTRACE and changes the output destination of the OMPROUTE trace to the CTRACE facility. Use of DEBUGTRC option introduced by APAR PQ76172 is highly recommended. A dump should be taken with the following console command: /DUMP COMM=('title') /R xx,JOBNAME=(tcpip,omproute),DSPNAME=('tcpip'.*),CONT /R xx,SDATA=(ALLNUC,CSA,LPA,LSQA,PSA,RGN,SQA,SWA,TRT),END *** NOTE: the strings "tcpip" and "omproute" in the above dump command should be replaced with the jobnames for TCPIP and OMPROUTE LE User abends (U40xx abends): LE will often try to clean up the cause of the abend BEFORE reporting the U40xx abend. It is recommended you code the following LE runtime options in the OMPROUTE proc to provide abend dumps taken at the actual time of error, rather than after condition handling has occurred. TRAP(ON,NOSPIE) TERMTHDACT(UAIMM) A SYSMDUMP DD card must be coded in the OMPROUTE procedure in order to capture a usable dump. Coding a CEEDUMP DD card in addition to the SYSMDUMP DD will provide a formatted LE dump as well as an abend dump. COMMON PROBLEMS: 1) OSPF Adjacency failures, performance tuning A. Verify OMPROUTE is being dispatched and has not been swapped out. Also verify that OMPROUTE is set at a dispatching priority one less than the TCPIP address space, and one higher than any other normal application. Note that applications started via JCL with PGM=BPXBATCH may not be treated as started tasks and may not receive the proper dispatching priority. It is highly recommended that OMPROUTE be started via JCL with PGM=OMPROUTE B. Code the OMPROUTE_OPTIONS=hello_hi environment variable to treat Hello packets at a higher priority 2) OMPROUTE is DR or BDR A. This problem may occur if OMPROUTE is elected the OSPF Designated Router (DR) or Backup (BDR). OMPROUTE should never become a DR - this task is better left to a hardware router. In order to ensure that OMPROUTE never becomes a DR, it is highly recommended that ROUTER_PRIORITY=0 be coded on EVERY OSPF_INTERFACE statement. A Router Priority of zero informs routers that OMPROUTE will not become a DR. B. LE Runtime options indicate storage should be kept rather than freed. The LE default setting for HEAP management is: HEAP((32K,32K,ANYWHERE,KEEP,8K,4K),OVR) This statement allocates 32K heaps and does not free the heap (HANC control block) after it is no longer needed. To prevent storage growths, the following setting is recommended for OMPROUTE: HEAP(,,,FREE) This parameter can be specified on the JCL PARMS 3) Various routing problems can be caused because one or more interfaces are not defined in the OMPROUTE configuration file. All interfaces defined to TCPIP MUST be defined to OMPROUTE as OSPF_INTERFACE, RIP_INTERFACE, or INTERFACE. Failure to correctly define ALL interfaces will lead to a variety of routing problems. Please note that the minimal definition for any type of interface includes IP address, subnet mask, and MTU size unless the defaults are acceptable. The default subnet mask is the class mask of the IP address, and the default MTU is 576. In most cases, these values are not acceptable and will result in expected/undesired network and subnet routes, and an MTU of 576 on dynamic routes 4) Various problems may occur if the routing table is over populated. OMPROUTE can handle approximately 1000-2000 routes with tracing active. After 2000 routes, OMPROUTE performance will suffer. Factors including number of routes, system load, HFS file I/O rate (if tracing is active), and amount of traffic generated to create the 2000+ routes will all influence OMPROUTEs performance. It is highly recommended systems running OMPROUTE be placed in a OSPF stub area. 5) If using OSPF on ANY interface, then VIPA should be defined as an OSPF_INTERFACE attached to the same OSPF area as a physical interface. If not using OSPF, then VIPA interfaces should be defined as INTERFACE (even for RIP). Not following this recommendation may result in various routing problems. Also note that in OSPF environments, VIPA can be defined as an INTERFACE and will be included in OSPF if AS_BOUNDRY_ROUTING import_direct_route=yes is coded. In this case, the VIPA will appear as an AS External route which may or may not be desirable. 6) OMPROUTE display commands are not working, and OMPROUTE appears to be looping using high CPU. This may lead to adjacency loss and loss of connectivity. Loss of connectivity to the VIPA is reported often when this problem is encountered. Verify that LE apar PQ61992 is applied. 7) Stack or Network Hardware Problems A problem in OMPROUTE can be the first sign of a stack or network hardware problem. If such problems are suspected, the diagnostic approach used by L2 support will be to isolate the layer where the problem is occurring. The layers that may need to be examined include: LAYER BEST DOCUMENTATION ----------------------------------------------- APPLICATION OMPROUTE trace PRESENTATION (USS/OE) CTRACE COMP=SYSOMVS,OPTIONS=(ALL) SESSION (sockets) CTRACE COMP=SYSTCPIP,OPTIONS=(PFS, IOCTL,SESSION) TRANSPORT \ CTRACE COMP=SYSTCPIP,OPTIONS=(UDP, (stack) RAW) NETWORK / CTRACE COMP=SYSTCPIP,OPTIONS=( INTERNET) DATA LINK (VTAM/IOS) CTRACE COMP=SYSTCPIP,OPTIONS=( LCS,VTAM,VTAMDATA) and/or a VIT trace (for VTAM) and/or a CCW trace (for channel devices) PHYSICAL (hardware) sniffer trace and/or external hardware displays/traces. NOTE: 1) Contact L2 before obtaining any of the above documentation. L2 will provide a more tailored documentation request based on the symptoms of the problem. 2) ALL of the above documentation would rarely be required. 3) All CTRACEs should be filtered on the TCPIP and OMPROUTE jobnames. 4) For suspected hardware problems, the time to problem resolution can expedited by engaging both software and hardware support. Note that the diagnostic approach used by Software support will be to first eliminate any suspected software problems. 8) OMPROUTE address space is being cancelled every 5 minutes by TCPIP. For problem determination, refer to technote 1192776 at this URL: http://www-1.ibm.com/support/docview.wss?&uid=swg21192776 9) The STDENV dataset must be VB. If it is not, a) ABEND0C4 or U4083 occurs when issuing F OMPROUTE,GENERIC commands b) when running OMPROUTE traces, no data is captured in the trace files. GENERAL INFORMATION: 1) Meaning of interface capability flags on EZZ7883I message. The flags are a bitmask representation of the network capabilities of the interface: 0x01 Interface is up (required) 0x02 Interface is broadcast capable 0x04 Interface driver is in debug mode (rarely seen) 0x08 interface is in loopback only mode 0x10 Interface is a Point-to-Point interface 0x20 Interface does not support Trailer encapsulation 0x40 Interface is running (required) 0x80 Interface is ARP incapable (cannot ARP) 0x100 Interface in promiscuous mode (sees ALL frames) 0x200 Interface is receiving all multicast packets 0x400 Interface is multicast capable 0x800 Interface is point-to-multipoint (P2MP) 0x1000 Interface supports Token Ring bridging 0x2000 Interface supports extended SAP 0x4000 Interface is a VIPA Common capability flags: C51 = 0x800 + 0x400 + 0x40 + 0x10 + 0x01 = P2MP + multicast + running + P2P + up C51 is often seen on CIP/CLAW devices 463 = 0x400 + 0x40 + 0x20 + 0x03 = Multicast + running + notrailers + broadcast + up 463 is often seen in multi-access broadcast capable interfaces (10/100 Mb Ethernet, T/R, etc) 4041= 0x4000 + 0x40 + 0x01 = VIPA + running + up 4041 are VIPA interfaces 2) OROUTED to OMPROUTE conversation tool information In z/OS 1.2, OROUTED has a new command line option (-c) that creates a sample OMPROUTE configuration file from the information found in: OROUTED profile, OROUTED gateway dataset, and PROFILE.TCPIP. It is important to note that the resulting OMPROUTE configuration file generated by the OROUTED -c tool is only a sample. As documented, the conversion tool will generate an OMPROUTE configuration file that must be reviewed by the user. Further user action may be required, and is usually noted in comments in the sample OMPROUTE configuration file. . NOTE: For z/OS Commserver hints and tips go to: http://www.ibm.com/software/network/commserver/support/
Local fix
Problem summary
Problem conclusion
Temporary fix
Comments
cancel
APAR Information
APAR number
II12026
Reported component name
PA LIB INFO ITE
Reported component ID
INFOPALIB
Reported release
001
Status
CLOSED CAN
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
1999-08-22
Closed date
1999-12-09
Last modified date
2010-09-14
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Applicable component levels
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19N","label":"APARs - OS\/390 environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG32M","label":"APARs - VSE\/ESA environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":null,"label":null},"Product":{"code":"SG19O","label":"APARs - MVS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSSN3L","label":"z\/OS Communications Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG27M","label":"APARs - z\/VM environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"001","Edition":"","Line of Business":{"code":"LOB16","label":"Mainframe HW"}}]
Document Information
Modified date:
14 September 2010