Configuring Shared Memory Communications over RDMA

Use these steps to configure and begin to use Shared Memory Communications over RDMA (SMC-R).

Perform the following steps to configure SMC-R:

1. If necessary, convert IPv4 IPAQENET DEVICE, LINK, and HOME definitions to INTERFACE definitions. SMC-R processing is provided only for OSD interfaces configured with INTERFACE definitions. For more information about converting IPv4 IPAQENET DEVICE, LINK, and HOME definitions to INTERFACE definitions, see Steps for converting from IPv4 IPAQENET DEVICE, LINK, and HOME definitions to the IPv4 IPAQENET INTERFACE statement.
2. Configure the SMCR parameter on the GLOBALCONFIG statement in the TCP/IP profile. The SMCR parameter includes the following subparameters:
   • PFID specifies the PCI Express® (PCIe) function ID (PFID) value for an IBM® 10GbE RoCE Express feature that this stack uses.
     You must code at least one PFID subparameter for this stack to use SMC-R, and two PFIDs per PNet ID per stack for redundancy.

     • When the RoCE Express features operate in a dedicated RoCE environment, each RoCE Express feature must have a unique PFID value, but each TCP/IP stack that uses the RoCE Express feature specifies the same PFID value.
     • When the RoCE Express features operate in a shared RoCE environment, each TCP/IP stack that uses the same RoCE Express feature must have a unique PFID value, even if the TCP/IP stacks are defined on different LPARs.

     
   • PORTNUM specifies the 10GbE RoCE Express port number to use for each PFID.

     Configure each PFID to use only a single port. The port number can be 1 or 2; 1 is the default port number.

     
     • When the RoCE Express features operate in a dedicated RoCE environment, either port 1 or port 2 can be used for a particular 10GbE RoCE Express feature, but z/OS® Communications Server cannot be configured to use both ports of a feature. For example, specifying PFID 0018 PORTNUM 1 and PFID 0018 PORTNUM 2, even if specified on different TCP/IP stacks in the same LPAR, results in an error during 10GbE RoCE Express activation processing for the second port that is activated.
     • When the RoCE Express features operate in a shared RoCE environment, both port 1 and port 2 can be used simultaneously if the ports are associated with different PFID values. For example, assuming that PFID 0018 and PFID 0019 represent the same physical RoCE Express feature, you can specify PFID 0019 PORTNUM 1 and PFID 0018 PORTNUM 2 to use both ports.
     
   • MTU specifies the maximum transmission unit (MTU) to be used for this PFID. The default value is 1024. For more information, see RoCE maximum transmission unit.
   • FIXEDMEMORY specifies the total amount of memory, in megabytes, that can be used for the staging and remote memory buffers.

     The default value is 256 MB. To choose a value that is appropriate for your environment, see Storage considerations.

   • TCPKEEPMININTERVAL specifies the minimum time interval, in seconds, for sending keepalive probes for TCP connections that use SMC-R protocols to exchange data.

     The default value is 300 seconds. For more information, see TCP keepalive.

   The following GLOBALCONFIG statement defines two 10GbE RoCE Express features, PFID 0018 and PFID 0019. Port 2 is used on each feature, and the maximum amount of 64-bit private storage that can be used for SMC-R communications is 200 megabytes. The default values for both TCPKEEPMININTERVAL and MTU are used.

   GLOBALCONFIG SMCR 
       PFID 0018 PORTNUM 2 
       PFID 0019 PORTNUM 2 
       FIXEDMEMORY 200

   For more information about these and other SMCR subparameters on the GLOBALCONFIG statement, see z/OS Communications Server: IP Configuration Reference.
3. (Optional) Configure the SMCR parameter on the IPAQENET and IPAQENET6 INTERFACE statements with the OSD channel path ID type (CHPIDTYPE OSD).
   Tip: SMCR is the default setting on the IPAQENET and IPAQENET6 INTERFACE statements for the OSD CHPID type.
4. Associate the interfaces with the appropriate subnet or prefix.
   • For an IPv4 interface to be eligible for SMC-R, you must configure a nonzero subnet mask on the INTERFACE statement in the TCP/IP profile.
     Result: SMC-R is used only between peers whose interfaces have the same subnet value.
   • For an IPv6 interface to be eligible for SMC-R, the interface must have at least one prefix that is associated with it.
     Rule: A prefix can be associated to an IPv6 interface in any of these ways:

     • A prefix received on a router advertisement message from an attached router
     • A prefix that is configured in OMPROUTE by using the PREFIX parameter on the IPV6_OSPF_INTERFACE, IPV6_RIP_INTERFACE, or IPV6_INTERFACE statement
     • A direct static prefix route that is configured over the interface on a ROUTE statement in a BEGINROUTES block in the TCP/IP profile
     Result: SMC-R is used only between peers whose IPv6 interfaces have at least one prefix in common.
5. (Optional) If you are using VLANs for your SMC-R communications, configure the VLANID parameter on the IPAQENET and IPAQENET6 INTERFACE statements for the OSD CHPID type. For more information, see VLANID considerations.
6. (Optional) If you have a server application that primarily uses short-lived TCP connections, you might want to avoid SMC-R rendezvous processing for TCP connections that are using that server port. Configure NOSMCR on the PORT or PORTRANGE statement for the server port or ports that this server application uses. For more information, see z/OS Communications Server: IP Configuration Reference.
7. Start the IPAQENET and IPAQENET6 interfaces. When the first SMC-R capable OSD interface becomes active, z/OS Communications Server automatically starts all PFIDs that are defined in the GLOBALCONFIG statement, and associates the 10GbE RoCE Express interfaces with the OSD interfaces that have matching physical network IDs (PNet IDs). For more information about PNet IDs, see Physical network considerations.