TCP streaming workload tuning

Edit online

Streaming workloads move large amounts of data from one endpoint to the other endpoint. Examples of streaming workloads are file transfer, backup or restore workloads, or bulk data transfer. The main metric of interest in these workloads is bandwidth, but you can also look at end-to-end latency.

The primary tunables that affect TCP performance for streaming applications are the following:

  • tcp_recvspace
  • tcp_sendspace
  • rfc1323
  • MTU path discovery
  • tcp_nodelayack
  • sb_max
  • Adapter options, such as checksum offload and TCP Large Send
The following table shows suggested sizes for the tunable values to obtain optimal performance, based on the type of adapter and the MTU size:
Device Speed MTU size tcp_sendspace tcp_recvspace sb_max 1 rfc1323
Token Ring 4 or 16 Mbit 1492 16384 16384 32768 0
Ethernet 10 Mbit 1500 16384 16384 32768 0
Ethernet 100 Mbit 1500 16384 16384 65536 0
Ethernet Gigabit 1500 131072 65536 131072 0
Ethernet Gigabit 9000 131072 65535 262144 0
Ethernet Gigabit 9000 262144 131072 2 524288 1
Ethernet 10 Gigabit 1500 131072 65536 131072 0
Ethernet 10 Gigabit 9000 262144 131072 262144 1
ATM 155 Mbit 1500 16384 16384 131072 0
ATM 155 Mbit 9180 65535 65535 3 131072 0
ATM 155 Mbit 65527 655360 655360 4 1310720 1
FDDI 100 Mbit 4352 45056 45056 90012 0
Fibre Channel 2 Gigabit 65280 655360 655360 1310720 1

(1) It is suggested to use the default value of 1048576 for the sb_max tunable. The values shown in the table are acceptable minimum values for the sb_max tunable.

(2) Performance is slightly better when using these options, with rfc1323 enabled, on jumbo frames on Gigabit Ethernet.

(3) Certain combinations of TCP send and receive space will result in very low throughput, (1 Mbit or less). To avoid this problem, set the tcp_sendspace tunable to a minimum of three times the MTU size or greater or equal to the receiver's tcp_recvspace value.

(4) TCP has only a 16-bit value to use for its window size. This translates to a maximum window size of 65536 bytes. For adapters that have large MTU sizes (for example 32 KB or 64 KB), TCP streaming performance might be very poor. For example, on a device with a 64 KB MTU size, and with a tcp_recvspace set to 64 KB, TCP can only send one packet and then its window closes. It must wait for an ACK back from the receiver before it can send again. This problem can be solved in one of the following ways:
  • Enable rfc1323, which enhances TCP and allows it to overcome the 16-bit limit so that it can use a window size larger than 64 KB. You can then set the tcp_recvspace tunable to a large value, such as 10 times the MTU size, which allows TCP to stream data and thus provides good performance.
  • Reduce the MTU size of the adapter. For example, use the ifconfig at0 mtu 16384 command to set the ATM MTU size to 16 KB. This causes TCP to compute a smaller MSS value. With a 16 KB MTU size, TCP can send four packets for a 64 KB window size.

The following are general guidelines for tuning TCP streaming workloads:

  • Set the TCP send and receive space to at least 10 times the MTU size.
  • You should enable rfc1323 when MTU sizes are above 8 KB to allow larger TCP receive space values.
  • For high speed adapters, larger TCP send and receive space values help performance.
  • For high speed adapters, the tcp_sendspace tunable value should be 2 times the value of tcp_recvspace.
  • The rfc1323 for the lo0 interface is set by default. The default MTU size for lo0 is higher than 1500, so the tcp_sendspace and tcp_recvspace tunables are set to 128K.

The ftp and rcp commands are examples of TCP applications that benefit from tuning the tcp_sendspace and tcp_recvspace tunables.