Recently, users from our HPC platform reported issue of launching MPI programs inside ThinLinc. For example, a message from using IntelMPI library is like following:
Abort(1091215) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(136)…:
MPID_Init(904)…:
MPIDI_OFI_mpi_init_hook(986): OFI addrinfo() failed (ofi_init.c:986:MPIDI_OFI_mpi_init_hook:No data available)
Detailed Tracking of the Issue:
The error indicates failed initialization of MPI library on the fabric network (OFI, InfiniBand). Then, we investigate the status of the fabric driver by “fi_info” under debug level, and we got:
libfabric:1303876:verbs:fabric:vrb_get_qp_cap():465 ibv_create_qp: Cannot allocate memory(12)
libfabric:1303876:verbs:fabric:vrb_get_qp_cap():465 ibv_create_qp: Cannot allocate memory(12)
libfabric:1303876:verbs:fabric:vrb_get_qp_cap():465 ibv_create_qp: Cannot allocate memory(12)
And it implies the memory limit on the process that launches the fabric library. When checking the memory limit by “ulimit -a”, we found that the locked_mem is set to 64, the default value of Linux. However, this issue only resides inside ThinLinc, while SSH sessions is working fine and with “unlimited” parameters as we set for the HPC system.
Found Issue:
Typically, the system limits for any new user sessions, i.e., SSHD, are setup by the PAM module “pam_limits.so”, using the configuration file at “/etc/security/limits.conf”. In Redhat/CentOS 7, the new session spawned by ThinLinc is directly under the root systemd process (PID=1), and a new session is triggered by it to load the limits configuration in PAM.
However, in Redhat/CentOS 8, the mechanism becomes different. The first process spawned from ThinLinc for the user session is another “systemd” process at user level (systemd --user), which is inherit from the root systemd and with default limits (locked_mem=64). And all sub-processes, i.e., gnome-terminal, are inherit from it without invoking the pam_limits.so module. Therefore, the “ulimit” is not updated and cause issues to the fabric library.
Solution:
The limits configuration used by “systemd --user” is in the file “/usr/lib/systemd/system/user@.service”, where you can add “LimitXXX=” as the settings for “/etc/security/limits.conf”. In this issue, a line was added under “[Services]” as “LimitMEMLOCK=67108864” to solve it. More reference:
https://www.freedesktop.org/software/systemd/man/systemd.exec.html
Conclusion:
In Redhat/CentOS8, the user limits are not configured via the PAM module, but in the systemd user services. We suggest Cendio@ add this in their product manual.