PI38376: TCP CONNECTS TO A DRVIPA FROM A TARGET SYSTEM (VIPABACKUP) CAN BE USING MSS 536 (MTU 576) UNEXPECTEDLY FOR POOR PERFORMANCE

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• When using distributed DVIPA (DRVIPA) and target systems
  (VIPABACKUPs without MOVEABLE IMMED on VIPADISTRIBUTE),
  there is potential for a TCP connection setup request (SYN)
  to use MSS 536 unexpectedly when going from a backup system
  to the distributor. OMPROUTE is used on the distributor and
  on the backup systems to advertise and learn the DRVIPA host
  routes using the OSPF protocol. While the OSPF host routes
  to the DRVIPA can have MTUs higher than the default 576
  (for example, MTU 8992 from the OSAs), a backup system can
  remain stuck with MTU 576 to result in MSS 536 for the
  outbound TCP connection setup requests.
  
  The problem occurs when an implicit host route for the DRVIPA
  is generated with the default MTU 576 instead of 65535 on a
  backup system. This is accomplished by starting the backup
  system first before the distributor. That is, the scenario is:
  
  1. Start TCPIP on a backup system (without OMPROUTE to
     simulate delay)
  
     No implicit host route for DRVIPA as VIPABACKUP is created
     on this backup system (due to MOVEABLE IMMED not coded on
     VIPADISTRIBUTE for backup).
  
  2. Start and stop the primary distributor (without OMPROUTE to
     simulate delay) to force the DRVIPA takeover on the backup
     system.
  
     The implicit host route for the DRVIPA with the default MTU
     576 is created on the backup system. The minimum MTU in RTOP
     is now set to 576. This minimum MTU in RTOP is used when
     there are multipath routes to the DRVIPA in the chain.
  
  3. Restart the primary distributor with OMPROUTE to do the
     DRVIPA takeback from the backup system.
  
  4. Start OMPROUTE on the backup system so that the OSPF host
     routes to the DRVIPA will be learned from the distributor.
  
  5. The DRVIPA implicit host route on the backup system will
     still have MTU 576 and therefore the minimum MTU in RTOP is
     stuck at MTU 576. The OSPF host routes to the DRVIPA over
     the OSAs to the primary distributor can have MTUs of 8992.
  
     When a connection setup request (SYN) is issued from this
     backup system to the DRVIPA on the distributor, the minimum
     MTU 576 from RTOP is used to result in MSS 536 for the
     outbound SYN packet. The minimium MTU from RTOP is used
     because the TCP transport layer doesn't know which multipath
     route to use for the SYN packet and the route selection to
     send the SYN packet is done later in the lower IP layer code
     path.
  
     Note: This DRVIPA implicit host route with MTU 576 is
           suppressed from view in the NETSTAT ROUTE DETAIL
           report. The only way to see this information on
           the backup system is to take a TCPIP address space
           dump and examine the IP routing table.
  
  If the distributor had been started first before starting the
  backup system, then MTU 65535 would have been set in RTOP for
  the minimum MTU for the DRVIPA. In this case, this minimum MTU
  in RTOP will eventually be reduced to the minimum of the MTUs
  among the OSPF multipath routes to DRVIPA as learned from OSPF
  (for example, MTU 8992 from the OSAs).
  
  Analysis for problem determination:
  
  One of the following commands will show the MaximumSegmentSize
  of 536 for the TCP connection on a backup target system for
  client x.x.x.x:
  
  1. TSO: NETSTAT ALL (IPADDR x.x.x.x
  2. USS: netstat -p tcp_proc -A -I x.x.x.x
  3. CON: DISPLAY TCPIP,,N,ALL,IPADDR=x.x.x.x
  
  TCPIPCS ROUTE report in a TCPIP address space dump on a backup
  target system will have the following:
  
  1. RTOP in RTE for the implicit host route for DRVIPA will have
     the minimum MTU set to 576.
  
  2. RTE for the implicit host route to the DRVIPA will also show
     MTU 576.
  
  3. RTEs for the indirect OSPF host routes to the DRVIPA on the
     distributor will have MTUs higher than 576 (e.g. 8992 for
     the OSA routing interfaces).
  
  Note: The implicit host route to the DRVIPA on the backup
        target system (that is, VIPABACKUP) is suppressed from
        view in the NETSTAT ROUTE DETAIL report. However, the
        implicit host route to the DRVIPA is not suppressed from
        view on the distributor that owns the DRVIPA. A implicit
        host route is to the DRVIPA IP address itself and has a
        zero gateway IP address to indicate no nexthop.
  

Local fix

• To get rid of the DRVIPA implicit host route with MTU 576
  dynamically on a target system without a disruptive IPL or a
  TCPIP recycle, the DRVIPA must be removed from and re-added
  to all of the distributing and target stacks by doing the
  following commands:
  
  1. Issue OBEYFILEs with these statements, first on the
     distributor and then on the target systems for the cleanups:
  
     VIPADYNAMIC
       VIPADIST DELETE x.x.x.x PORT yy DESTIP ALL
       VIPADELETE x.x.x.x
     ENDVIPADYNAMIC
  
  2. Issue the NETSTAT ROUTE command on the distributor for the
     DRVIPA to verify that it has been deleted:
  
     D TCPIP,,N,ROUTE,DETAIL,IPADDR=x.x.x.x
  
  3. Issue the NETSTAT VIPADCFG command on all target systems to
     verify that DRVIPA has been deleted:
  
     D TCPIP,,N,VIPADCFG,IPADDR=x.x.x.x
  
  4. Issue OBEYFILE on the distributor with these statements to
     re-add the DRVIPA:
  
     VIPADYNAMIC
       VIPABACKUP x.x.x.x
       VIPADIST DEFINE DISTM mmmmmm x.x.x.x PORT yy DESTIP ALL
     ENDVIPADYNAMIC
  
  5. Issue the NETSTAT ROUTE command on the distributor for the
     DRVIPA to verify that it has been added:
  
     D TCPIP,,N,ROUTE,DETAIL,IPADDR=x.x.x.x
  
  6. Issue the NETSTAT VIPADCFG command on both distributor and
     target systems for the DRVIPA to verify that it has been
     added:
  
     D TCPIP,,N,VIPADCFG,IPADDR=x.x.x.x
  
  KEYWORDS:
  VIPA DVIPA DRVIPA VIPADYNAMIC VIPADISTRIBUTE VIPABACKUP BACKUP
  MTU 576 MSS 536 OMPROUTE OSPF EZBXFDVI RTOP RTE MULTIPATH HOST
  IMPLICIT ROUTE AUTOLOG DELAYSTART GLOBALCONFIG SYSPLEXMONITOR
  DELAYJOIN SYSPLEX BACKUP TCPIP TCP SYN CONNECTION SETUP
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Users of the IBM Communications Server for z/OS Version  *
  * 2 Release 1 IP                                               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * A TCP connection can use the wrong maximum segment size      *
  * (MSS).                                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply PTF                                                    *
  ****************************************************************
  A TCP connection that flows over a multipath route with a
  destination IP address of a DVIPA can use an MSS that is too
  small. For multipath routes, the MTUs for all routes for that
  multipath route are considered, including the implicit route for
  the DVIPA. Implicit routes for DVIPAs have an MTU of 576. The
  multipath MTU always chooses the smallest MTU among all routes.
  The implicit route should not be considered when choosing an
  MTU.
  

Problem conclusion

• EZBXFDVI will be changed to ensure that implicit route MTUs are
  not used for multipath routing.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI29203 PI44962

Modules/Macros

• EZBXFDVI
  

Fix information

• Fixed component name

  TCP/IP V3 MVS

• Fixed component ID

  5655HAL00

Applicable component levels

• R210 PSY UI29203

     UP15/09/03 P F509

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.