System requirements for SMC-R in a shared RoCE environment

To use Shared Memory Communications over RDMA (SMC-R) with RoCE Express features operating in a shared RoCE environment, the minimum software requirement must be z/OS® Version 2 Release 1 with APARs OA44576 and PI12223 applied.

SMC-R requires RDMA over Converged Ethernet (RoCE) hardware and firmware support. The following minimum hardware requirements must be met to use SMC-R:

• You must have IBM® z13 (z13) or later systems.
• You must have one or more IBM 10GbE RoCE Express features.

  10GbE RoCE Express features are dual ports with short range (SR) optics and can be shared across multiple operating systems images or TCP/IP stacks in a central processor complex (CPC).

  Guideline: Provide two 10GbE RoCE Express features per unique physical network. For more information, see RoCE network high availability.
• You must have System z® OSA-Express for traditional Ethernet LAN connectivity.

  SMC-R does not impose any specific OSA requirements.

• You must have standard 10 GbE switches.

RoCE network configuration requirements

z/OS Communications Server supports connectivity to multiple, distinct layer 2 networks through unique physical LANs. Each unique physical network is identified by existing Ethernet standards that are based on the physical layer 2 broadcast domain. You can define a physical network ID (PNet ID) for each physical network. For more information, see Physical network considerations.

For hosts to communicate by using SMC-R, they must connect directly to the same Ethernet layer 2 LAN network. If VLANs are in use, each host must also have access to the same VLAN. For more information, see VLANID considerations.

There are restrictions on the physical distances that can be used to route RDMA frames. To understand these distance specifications and limitations, see IBM z Systems Planning for Fiber Optic Links.

RoCE network high availability

Because RoCE connections do not use IP routing and the RDMA connections to remote hosts are direct point-to-point connections that use reliable connected queue pairs (RC QPs), there is no concept of an alternative IP route to the peer. SMC-R connectivity is possible with a single 10GbE RoCE Express feature, but a loss in that single feature means that the associated TCP connections and workloads are disrupted. Therefore, redundant 10GbE RoCE Express features on both the local and remote hosts are required to achieve network high availability with SMC-R. If your TCP workloads require high availability, redundant 10GbE RoCE Express features and redundant Ethernet switches are required. The SMC-R protocol actively uses both features, rather than using one feature with the other in standby mode. For more information, see High availability considerations.

IBM 10GbE RoCE Express features also have redundant internal PCIe support structures, or PCIe internal paths, as described in Redundancy levels. To avoid another single point of failure, install each 10GbE RoCE Express feature that is managed by the same operating system with a unique internal path. For more information about how to install a 10GbE RoCE Express feature to achieve full redundancy, see your IBM service representative.

RoCE bandwidth

The 10GbE RoCE Express features provide 10 GbE ports. When redundant features are defined, SMC-R link groups can be formed by using both features, resulting in a 20 GbE logical pipe to each physical network. z/OS Communications Server uses only two features within a link group at any particular time.