A physically separate IBM® 10GbE RoCE Express® feature is provided to use RDMA over Converged Ethernet (RoCE) on System z®. This feature is used with the existing Ethernet connectivity that OSA provides. The 10GbE RoCE Express feature provides access to the same physical Ethernet fabric that is used for traditional IP connectivity. For more information about the Ethernet switch requirements for RoCE, see Setting up the environment for Shared Memory Communications over RDMA and IBM z Systems Planning for Fiber Optic Links.
The operating systems must logically group the associated 
physical ports of both the 10GbE RoCE Express and OSA adapters
based on their required physical connectivity. Each
central processor complex (CPC) connects to a physical network by
using both OSA and 10GbE RoCE Express ports. You can use two RoCE Express ports
at most to connect to a physical network at a given time, but you
can use as many OSA adapters as necessary for your network bandwidth
or usage requirements. An example of this logical grouping, using
two OSA adapters and two RoCE Express features,
is shown in Figure 1.
. Each
PFID value must match a FID value configured in the hardware configuration
definition (HCD).- In a dedicated RoCE environment, each PFID represents a unique PCHID definition of a RoCE Express feature, and only one of the two RoCE Express ports for the feature can be used at a time.
- In a shared RoCE environment, each PFID represents a virtual function (VF) usage of a RoCE Express feature, and multiple PFID values can be associated with the same physical feature and port.
The TCP/IP stack must be able to determine which physical network is connected to a particular 10GbE RoCE Express interface, so that the 10GbE RoCE Express interface can be associated with the SMC-R capable IPAQENET or IPAQENET6 interfaces that connect to that same physical network. For instance, in Figure 1, three distinct and physically separated networks can be accessed by using SMC-R communications.
The concept of a physical network identifier (PNet ID) was created to simplify this physical network configuration task. With the convention of a PNet ID, you can define a value to represent the ID or name of your physical layer 2 LAN fabric or physical broadcast domain. The System z physical ports that are to be connected to the associated physical networks are then logically associated with their respective PNet IDs. The PNet ID then becomes an attribute of the physical port of the feature or adapter, describing how this feature or adapter is physically connected to your data center network. You can specify the PNet ID in a single step within the hardware configuration definition (HCD), enabling all operating systems of all associated CPCs to dynamically learn and use this definition.
To match the 10GbE RoCE Express features with the correct OSA SMC-R capable adapters, you must define a PNet ID value for both the 10GbE RoCE Express interface (physical port) and the corresponding OSA adapters (physical port) within the HCD. The OSA ports correspond to the stack IPAQENET and IPAQENET6 interfaces. VTAM® and the TCP/IP stack then dynamically learn the PNet IDs for the 10GbE RoCE Express interface and the OSA interfaces when the 10GbE RoCE Express interface or the OSD interface is started. The 10GbE RoCE Express interface is associated with only SMC-R capable OSA interfaces that have the same PNet ID value defined.
You can use virtual LANs (VLANs) to logically separate a physical network. If you configure multiple PNet IDs for SMC-R, then you must ensure that each VLAN or subnet in your configuration does not span more than one PNet ID. The physical network that a PNet ID represents can include multiple subnets, but each subnet must correspond to a specific PNet ID.
SMC-R processing requires the use of subnet masks. For more information, see Configuring Shared Memory Communications over RDMA.
For more information about the HCD, see z/OS HCD Planning and z/OS HCD User's Guide.