When running a system with multiple TCP/IP stacks active (Common INET environment), UNIX System Services keeps a table of routing definitions to determine which stack should be used for a new outbound connection (the prerouter function). After upgrading to z/OS 1.11 (or above), the potential exists for the stack selection to be different for applications that use the DEFAULT route (also called 'route of last resort', and indicated by a subnet mask of all 0s) than had been selected on earlier releases. This can cause some applications to behave differently, including having connection failures and timeouts.
Systems running a single TCP/IP stack will not have any changes.
In releases prior to z/OS 1.11, the DEFAULT route was tracked only as a flag for each stack, indicating that the stack had a DEFAULT route defined. With z/OS 1.11, this has been changed to treat the DEFAULT route like all other subnet and network routes, including the potential for multiple duplicate routes on a stack, having an associated cost, and selection of a stack with the least cost for that route.
Selecting the stack with the least cost path to the requested destination address is generally desirable. However, on systems where multiple stacks have a DEFAULT route defined, these might exhibit different behavior with an upgrade to z/OS 1.11 or above. Where before the existence of these routes was masked by the selection logic used in earlier releases, they can now be selected causing undesirable results (including connection timeouts).
The problem occurs when one or more stacks have a DEFAULT route defined, but they do not actually have network connectivity to get to all addresses. This typically occurs when the devices used by a stack have connectivity only to a subset of the total network (such as an isolated laboratory LAN), and a DEFAULT route is coded to cover associated addresses that are not directly connected to that LAN (require going through a router to other LAN segments). If this is done using a GATEWAY or BEGINROUTES section of the TCPIP PROFILE, this route will be listed with the lowest cost. If the stack that actually does have connectivity to the entire network learns its DEFAULT route using dynamic routing (OMPROUTE), its route will have a higher cost. When this occurs, the lower cost stack will always be selected in z/OS 1.11. If the desired route is on the default stack, it would have always been selected in earlier releases.
Determining whether your system is affected:
- Examine the BPXPRMxx PARMLIB member, and find the NETWORK statement with DOMAINNAME(AF_INET) specified. If this statement has TYPE(INET) specified, this system will be unaffected by the upgrade. If it has TYPE(CINET), then continue to the next step.
Alternatively, you can do a
DISPLAY OMVS,PFSoperator command (see the BPXO046I message for a description of the output). If the PFS TYPE for AF_INET SOCKETS (typically the first or second entry) shows INET, your system is configured for a single stack and will not be affected. If it shows CINET, continue to the next step.
- Do a
DISPLAY TCPIPoperator command with no options specified. It will list the names of the TCPIP stacks that have ever been active since the last IPL. If there is only one name listed, then your system will be unaffected as long as no other stack is started.
- In the BPXPRMxx member, examine the list of
TYPE(CINET). Note which one has the
DEFAULTkeyword specified. If none of them do, then the new stack selection logic (while potentially different) will not have a significant effect.
If using the alternative
DISPLAY OMVS,PFSoutput, note the FLAGS in the second section. If one has CD, then either no default stack has been specified or the one that is specified is not active (and will have an SD flag). If one has SC, then continue with the next step.
- Do a
DISPLAY OMVS,CINEToperator command (see the BPXO047I message for a description of the output). Note which stacks in the HOME INTERFACE list have the DRS flag. If one of them is the default stack (determined above), then continue with the next step.
If the system is already running z/OS 1.11 (or above), then the DRS flag will not appear. Instead go to the end of the HOST ROUTE INFORMATION section, where there will be one or more entries with NET DESTINATION 000.000.000.000 and NET MASK 000.000.000.000. Skip step 5 and use the information from these entries for step 6.
- Display the routing table for each active stack on the system (listed in the output from step 2). This can be done either from TSO (
NETSTAT ROUTE DETAIL TCP stackname) or the operator console (
DISPLAY TCPIP,stackname,NETSTAT,ROUTE,DETAIL). Note the entries listed with Destination Default, which will be at the beginning of the report.
- If the default routes listed for the default stack all have a lower METRIC value than those for the other stacks, then your system will be unaffected by the upgrade as long as that does not change.
- Test the accuracy of the default routes by using the FTP command to connect to a server whose address will not cause selection of any other route listed by step 4. From TSO enter the command
FTP -p stackname xx.xx.xx.xx (TIMEOUT 15, first using the default stack (verify the connection works) and then using all other stacks that have a default route. If the command has a timeout making the connection, it is likely that the default route through that stack is invalid.
- Originate_Default_Route=YES is specified on the AS_Boundary_Routing or IPv6_AS_Boundary_Routing statement when using OSPF.
- The Originate_RIP_Default or IPv6_Originate_RIP_Default statement is coded when using RIP.
For systems configured for CINET, review the routing definitions before upgrading to later releases to ensure that nonexistent routes are not listed. To avoid this problem, ensure that the DEFAULT route is not configured on stacks that have limited network connectivity. For those stacks, the correct configuration is to explicitly list the subnets that are directly or indirectly connected to the devices used without coding a DEFAULT route.
When an undesired DEFAULT route is being learned through dynamic routing, identify the source of that route and update the configuration of the router advertising it. For OMPROUTE, the DEFAULT route will be advertised in one of the following conditions:
Remove this specification where this is not appropriate.
If the PTF for APAR OA36941 has been applied, an alternative method to resolve this is available. Define the static routes on secondary stacks via a BEGINROUTES statement and include the REPLACEABLE attribute on the DEFAULT routes. In this configuration, the CINET prerouter will give those routes a lesser priority than any route learned by the primary stack.
If the static routes are currently defined with a GATEWAY statement, these will need to be converted to the BEGINROUTES format first. See TechNote 1616535 for a method to assist with performing this conversion.
The stack selection logic performed by OMVS is as follows:
- The prerouter table affects only connections outbound from the z/OS system when stack affinity has not otherwise been established. Inbound connections are always associated with the stack on which the packets arrived. That is likely why only a subset of applications are affected; the others are servers that service inbound connections.
- If the application binds to a specific local address, then the stack owning that address is used for that socket from that point forward. Most client (outbound) applications do not do an explicit bind.
- On the connect() call (or sendto() call for UDP traffic), the requested remote address is used to search the prerouter table. It first finds the best match; a host route for the address (if one exists), then a subnet or network route with the most specific mask (most bits ON).
- If multiple stacks list that matching route, the one with the smallest cost (the METRIC value) is used.
- If multiple stacks have that lowest cost and one of them is the default stack, then the default stack is used. If not, the stack selected will be one of the set (it is not defined which one, probably the first encountered in the prerouter table).
- If no other route matches, then the default route is used. Here is where the difference between the release levels occurs:
- For z/OS 1.10 and earlier, the prerouter table only tracks the existence of a default route on each stack (the DRS flag). If one of those is the DEFAULT stack, then that will be used; otherwise, one of the others is selected.
- For z/OS 1.11 (and above), the prerouter table lists the default route the same way as it does all other routes (destination 0.0.0.0, mask 0.0.0.0, with a metric and associated stack). So at this point, the rules for selection become the same as described for selection of a subnet route.
|Operating System (z/OS- OS/390- MVS)||z/OS||5695SCPX1 - Z/OS UNIX SYSTEM SERVICES KERNEL AND FILE SYSTEM||z/OS||1.11, 1.12, 1.13, 2.1|