Internet Protocol over InfiniBand (IPoIB)

Edit online

Internet protocol (IP) packets can be sent over an InfiniBand (IB) interface. This transport is accomplished by encapsulating IP packets of IB packets using a network interface.

In order to use IP over IB, you must install and configure the InfiniBand connection manager (ICM) driver and at least one IB device in the system. To see if an IB device is already installed, run the lsdev -C | grep iba command. The name of the fileset containing the IB interface is: devices.common.IBM.ib. The devices.chrp.IBM.lhca fileset is an example of a currently supported adapter fileset.

To configure an ICM driver, refer to Configuring an InfiniBand Communication Manager driver.

In order to create the InfiniBand interface (IB IF), the IB IF must be able to join an existing broadcast-multicast group with a user-provided PKEY (or a default PKEY = 0xFFFF is used if the user did not provide one) and a user provided Q_Key (or a default Q_Key = 0x1E is used if the user did not provide one). A broadcast-multicast group is a multicast group that the interface must join in order to send broadcast and ARP packets. If such a broadcast-multicast group does not exist, or can not be created by the interface, the creation of the IB IF will fail.

You can create or change an IB IF using the command-line interface or the SMIT user interface. The parameters required to create an IB IF are the following:
  • interface name
  • adapter name
  • port number
  • interface IP address
The following parameters are for changing the IB IF:
  • internet address
  • network mask
  • MTU size (equal to the MTU desired, less 4 bytes for the IB header)
  • state
  • Send and Receive queue size (default is 4000)
  • Multicast Queue Key
  • Super packet on or off
The following is an example of the command used to create an IB IF from the command line: 
$ /usr/sbin/mkiba -i ib0 -p 1 -A iba0 -a 1.2.3.8 [-P -1 -S "up" -m "255.255.254.0" -M 2044]
where:
Item Description
-M 2044 Maximum transmit unit.
-m "255.255.254.0" Net mask.
-p 1 Port number (defaults to 1 if not provided).
-A iba0 IB device name.
-a 1.2.3.8 IF IP address.
-i ib0 Interface name.
-P -1 Partition key (defaults to PKEY if not provided. After the interface is created, PKEY cannot be changed; the user must obtain a non-default PKEY from the network administrator.)
-S "up" Interface state.
-q 8000 Receive and Transmit Queues size (each).
-Q 0x1E Multicast Queue Key assigned to the Multicast group (defaults to Q_KEY = 0x1E if not provided).
-k "on" Superpacket will allow the interface TCP/IP MTU to be 64K. It has to be enabled in the remote host also to be able to work.
The following is an example of the command used to create an IB IF from the SMIT user interface: 
$ smitty inet

After the Network Interface Selection menu is displayed, follow this procedure:

  1. Select Add a Network Interface or Change / Show Characteristics of a Network Interface. The Add a Network Interface menu displays.
  2. In the Add a Network Interface menu, select Add an IB Network Interface. The Add an IB Network Interface menu displays.
  3. In the Add an IB Network Interface menu, make the necessary changes and press Enter.