Platform MPI error message "libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes."

Answer

Platform MPI 7.1 job running with InfiniBand has the following error message:
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.

In order to use an RDMA protocol to transfer messages, some memory buffers must be pre-pinned. The amount of memory that is pinned is a function of the number of ranks in the job. By default, each rank will pin eight 16k buffers for each remote rank. See "-rdma", "MPI_RDMA_MSGSIZE", and "MPI_RDMA_NENVELOPE" in the Platform MPI Users Guide for more information on pre-pinned buffers.

The warning is coming from the OFED driver, and indicates that the user running the job has a limit on the amount of memory that can be pinned: RLIMIT_MEMLOCK is 32768 bytes.

For sh, bash, ksh and other variants, the current limits can be checked with:
$ ulimit -a

and set to the maximum permitted with:
$ ulimit -l unlimited

For csh, tcsh, the current limits can be checked with:
% limit

And set to the maximum permitted with:
% limit memorylocked unlimited

For example:
$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 139264
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 139264
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$

Notice that the "max locked memory" is "32" Kbytes.

In general, we recommend that the limit for memlock be set to 'unlimited' for clusters that will run Platform MPI jobs using InfninBand as the interconnect. On RedHat, this limit is set in the /etc/security/limits.conf file:

* soft memlock unlimited
* hard memlock unlimited

It is necessary to restart sbatchd (LSF daemon), sshd, and xinetd (if applicable) on those nodes after the values have been changed.