Each stack locally allocates memory for a remote memory buffer (RMB) to receive inbound data over an SMC-R link. The sending operating system places TCP socket application data directly into the RMB that the receiving stack assigns to receive data for a TCP connection. The receiving stack then copies the data from the RMB into the receive buffer of the receiving socket application.
An RMB is partitioned into RMB elements (RMBEs) of equal size, where each RMBE is associated with a single TCP connection. In the Connection Layer Control (CLC) messages during rendezvous processing, each peer communicates the location of its local RMBE and a remote key that allows remote access to the RMB. The remote host has write access and the local host reads the data for passing to the socket application.
For z/OS® Communications Server, an RMB is a contiguous, 1-MB block of pinned 64-bit private storage. In Figure 1, an RMB is created on z/OS, into which the peer node can write, and an RMB is created on the peer node, into which z/OS can write. An RMB must be registered with the IBM® 10GbE RoCE Express® interface so that the storage is available to the remote peer.
Each RMB is associated with the reliable connected queue pairs (RC QPs) and therefore with the SMC-R link for the two communicating peers. The association of the RMB to the RC QPs ensures that only the two peers have RDMA access to this particular RMB. In addition, all RMBs that are associated with a particular SMC-R link must be accessible to the remote peer by using any SMC-R link in the link group. This accessibility enables the remote peer to use any link within the link group to place the TCP connection data into the correct RMB.
The SMC-R peers are not required to assign RMBEs of the same size for the client and the server of a specific TCP connection. The peers can use one or more RMBs for the TCP connections that are using the same SMC-R link.
When an SMC-R link group is first established, z/OS Communications Server allocates a basic set of three RMBs for the link group. The RMBE sizes for the RMBs are not defined initially, but instead are determined by the needs of the TCP connections that use the SMC-R link. z/OS Communications Server uses RMBE sizes of 32 KB, 64 KB, 128 KB, 256 KB, and greater than 256 KB. The appropriate size for the TCP connection is selected based on the receive buffer size of the local application. When the application does not explicitly set the buffer size by using SETSOCKOPT(), the default value is determined by the value of the TCPRCVBUFRSIZE parameter on the TCPCONFIG statement.
More RMBs are allocated as needed to accommodate more TCP connections or different receive buffer sizes.