FTP errors during data connection establishment for the z/OS FTP server

Content

During a typical FTP connection, messages flow back and forth between the FTP client and the server across the control connection (this is the connection usually established on port 21). However, whenever data is transferred, FTP opens a new connection called the data connection.


Commands which result in the opening of a data connection include LS, DIR, GET, MGET, PUT, and MPUT. When trying to establish the data connection, errors may arise, most commonly manifested as an FTP reply such as 425 Can't open data connection or 425 Unable to open data connection. Of note, data connections can be passive (the client opens a connection to the server) or active (the server opens a connection to the client), but the client controls which method is used. As a result, if the mode in which the data connection is established is related to the error, then further investigation must be done on the FTP client.

It is possible that the port the FTP server is using is not included in a range that has been defined on your firewall. Note that the FTP server will, by default, use port 20 for active connections, and will use an ephemeral port for passive connections. If the server is z/OS, see the section on Determining if you need a portrange for the FTP server, which will provide instructions on how to configure FTP to use only ports authorized on the firewall.


Steps to follow to resolve FTP errors during data connection establishment for the server:

1. Obtain a packet trace and server trace. First, start the packet trace facility. Detailed instructions for doing this can be found in Informational APAR II12014. If this is a high-volume system, be sure to reference section 6 on how to filter tracing based on IP addresses and ports.

2. Obtain an FTP server trace with options BAS and FLO. This can be done by issuing the following MVS MODIFY command prior to starting the failing data transfer (where ftp_server_jobname is the name under which the FTP server is running, such as FTPD1 or FTPSERV):

/MODIFY ftp_server_jobname,DEBUG=(FLO,BAS)

3. The output of this trace will be sent to the SYSLOGD application. If you do not have SYSLOGD configured and running correctly, reference MustGather: Collect Troubleshooting Data, Read First: Syslogd for z/OS Communications Server.

4. Recreate the problem and deactivate the traces. Instructions for stopping the packet trace are also in II12014. To stop the server trace, issue the following console MVS MODIFY command:

/MODIFY ftp_server_jobname,DEBUG=(NONE)

5. You will now need to perform trace analysis. Begin by browsing the server trace and locating the time that the timeout occurred. Obtain the timestamp from the message. Determine the port on which the data connection is being opened by finding the PORT command (if this is an active data connection) or the PASV response (if this is a passive connection). The text for both of these messages will be a string in the format of u,v,w,x,y,z. This can be interpreted as follows:

5.1 The IP address to which the data connection will be opened is u.v.w.x.
5.2 The port to which it will be opened can be calculated using the following formula:
(y x 256) + z.
5.3 At this point, you have identified the timestamp, the IP address, and the port to which the data connection will be opened.

6. Open the packet trace in IPCS. Format the trace using the command:

CTRACE COMP(SYSTCPDA) LOCAL FULL OPTIONS((SESSION))

This command formats the trace into a summary for each IP connection.

7. Find the summary that represents the failing connection by looking for an FTP connection which encompasses the time stamp from the server trace. Use the combination of the port, the IP address, and the timestamp to ensure you have the correct summary. Here is a sample of what you will see:


Local Ip Address:                             x.x.x.x          
Remote Ip Address:                            y.y.y.y          
                                                                       
                                                                       
Host:                                 Local,          Remote          
 Client or Server:                  Unknown,         Unknown          
 Port:                                   21,            1095          
 Application:                        telnet,                          
 Link speed (parm):                      10,      10 Megabits/s
                                                                       
Connection:                                                            
 First timestamp:                 2007/08/24 10:21:54.688167          
 Last timestamp:                  2007/08/24 11:46:16.591480

8. Scroll down past the summary to the individual descriptions and examine the approximate timestamp obtained from the trace.

9. Investigate the reasons why the data connection is failing. Some of the things you can look for include:

No SYN ACK being returned by the remote client, which you can see in this example:

   TcpHdr  IO F    Seq     Ack RcvWnd  Data Delta Time       TimeStamp
1      S    O    965574368          0 368640     0   0.000000 10:44:43.727489
2      S    O    965574368          0 368640     0   2.000000 10:44:45.727489
3      S    O    965574368          0 368640     0   4.000000 10:44:49.727489

Because no SYN ACK is seen inbound from the remote client, either the client is not responding, or the server’s SYN packet was lost before it reached the client. Possible causes for this lost SYN include a firewall that is preventing connections from being opened, or a router that is discarding packets. To further diagnose this problem, take a network trace at various points on the network concurrently with a packet trace. You can then compare the network trace output to the packet trace to determine on which network segment the packet was lost. For example, consider the following route between an FTP client and server:

SERVER ]--A--[ ROUTER ]--B--[ FIREWALL ]--C--[ CLIENT

If a network trace taken on segment C shows the server's SYN packet, but a network trace on segment B does not, then investigate the firewall to determine why it is dropping the packet. It could be a problem with the port range being used by the for the connection. To check this, see the section on Determining the FTP server's port range below. Similarly, if the SYN packet is seen in segment B but not A, then investigate the router. A network trace on segment A capturing the SYN packet warrants investigation on the remote TCP/IP stack. Contact the appropriate service team.

NOTE: This example shows the server opening the data connection to the client (because the SYN is sent outbound, as designated by the O in the IO column). If this connection were passive, the SYN would have been inbound. This concept will hold true for all subsequent examples.

The server's SYN resulted in a reset from the remote TCP/IP stack, another possible reason. This will appear in the packet trace as follows:

   TcpHdr  IO F        Seq        Ack RcvWnd  Data Delta Time       TimeStamp
1    S      O    965574368          0 368640     0   0.000000 10:44:43.727489
2        R  I            0  965574368 368640     0   0.000134 10:44:43.727623

This indicates that either a firewall is not allowing connections, or that the remote FTP client has not established a listen on the specified port. Investigation using network traces, as described above, will help isolate the source of the reset.

10. If you need further assistance or diagnostic support, contact IBM service with the network trace, server and packet traces available. See the informational APAR II12030 for instructions on submitting trace to IBM for analysis.


Determining if you need a portrange for the FTP server

1. Review your firewall configuration to determine if the firewall is configured to allow only ports within a certain range. If there is a range, then proceed to step 2.
If there is no range, the data connection problem is not related to portrange configurations.

2. Find the FTP.data file. If you need assistance identifying what FTP.DATA file the server is using, the hierarchical search order used by the server to find this file is detailed in the topic on TCP/IP Configuration data sets in the IP Configuration Guide.

3. Find the passivedataports statement.

4. Look for the low number and high number on the passivedataports statement. Ensure the range created by these parameters is included in the firewall's allowed portrange. If you have no passivedataports statement, add one that is within the firewall’s allowed range. The syntax for this statement is:

PASSIVEDATAPORTS (low_port, high_port)

5. If the port range is included, look at the TCP/IP port configuration using the NETSTAT command with the portlist option. This can be done from the console
with the MVS MODIFY command, from TSO, or from the OMVS shell. The syntax for these commands are:

• From the OMVS shell:
netstat -o
• From TSO:
NETSTAT PORTL

• From the console (where stackname is the jobname of the TCPIP stack for which the information is desired):
DISPLAY TCPIP,stackname,NETSTAT,PORTL

If the range is not included, add a Portrange statement that matches the passivedataports statement in the FTP.DATA file. The syntax for this is:

PORTRANGE first_port_num num_of_ports_to_reserve TCP AUTHPORT.

6. As an example, assume that your firewall allows a port range of 5000
to 6000. You can set the FTP server to use these ports using the following:

PASSIVEDATAPORTS(5000,6000)

Note that the PORTRANGE statement for FTP in the TCPIP profile corresponding
to this would be:

PORTRANGE 5000 1001 TCPAUTHPORT

A common mistake is to specify the first_port_num as 5000, and the num_of_ports_to_reserve as 1000. However, since the 5000 port is
included PORTRANGE 5000 1000 would only reserve ports 5000 to 5999.