Skip to content

bootstrap: let the SHMEM plugin initialize OpenSHMEM unless NVSHMEMX_INIT_WITH_SHMEM is set - #146

Open
0z5a wants to merge 1 commit into
NVIDIA:develfrom
0z5a:fix-shmem-bootstrap-self-init
Open

0z5a wants to merge 1 commit into
NVIDIA:develfrom
0z5a:fix-shmem-bootstrap-self-init

Conversation

@0z5a

@0z5a 0z5a commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #116.

Problem

nvshmemi_bootstrap() builds a bootstrap_attr_t whose initialize_shmem defaults to 0. It then
calls bootstrap_set_bootattr() with a NULL attribute when the application passed none, so the field
stays 0. With NVSHMEM_BOOTSTRAP=SHMEM and plain nvshmem_init(), the SHMEM plugin therefore skips
shmem_init(). It then calls shmem_my_pe() on an OpenSHMEM library that was never initialized.
#116 has the full trace.

Change

  • bootstrap_set_bootattr() now sets initialize_shmem from flags & NVSHMEMX_INIT_WITH_SHMEM.
    Only an application that passes that flag has called shmem_init() itself.
  • nvshmemi_bootstrap() always passes the attribute that bootstrap_init() uses.

shmem_finalize() already depends on nvshmem_initialized_shmem, so after this change the plugin
also finalizes what it initialized. The MPI plugin does the same with MPI_Init. The
NVSHMEMX_INIT_WITH_SHMEM path still passes 0 and is unchanged.

  • When the plugin initializes OpenSHMEM itself, it keeps the OpenSHMEM library loaded:
    dladdr(shmem_init) finds the library, then dlopen(..., RTLD_NOLOAD | RTLD_NODELETE) pins it.
    In this case the application usually does not link OpenSHMEM, so unloading the plugin at
    nvshmem_finalize() also unloaded it. With Open MPI's OpenSHMEM, the process then crashed at exit
    (SIGSEGV) after shmem_finalize().

Testing

Setup: devel eaf955f vs this PR. Both columns use the host library and the SHMEM plugin built from
the same tree.

Open MPI 4.1 OpenSHMEM, 2 PEs on one node (B300). Launched with mpirun; same program binaries
on both sides.

case devel this PR
NVSHMEM_BOOTSTRAP=SHMEM, nvshmem_init(), nvshmem_malloc, barrier, nvshmem_finalize() 3/3: pshmem_my_pe: SHMEM is not initialized, abort every run whose shmem_init() succeeded (3/3) passes on both PEs and exits 0
same, plugin without the library pin — 3/3 pass on both PEs, then SIGSEGV at process exit (0/2 with LD_PRELOAD=liboshmem.so)
app calls shmem_init(), then NVSHMEMX_INIT_WITH_SHMEM 3/5 pass 2/5 pass

The remaining failures appear on both sides, including the unchanged NVSHMEMX_INIT_WITH_SHMEM
path. They are SHMEM_ABORT inside Open MPI's shmem_init() itself, after a memheap "too many
segments are registered" warning and before NVSHMEM prints anything. They are an Open MPI issue on
this host.

Sandia OpenSHMEM 1.5.3 (PMI-2), 1 PE (GH200).

case devel this PR
NVSHMEM_BOOTSTRAP=SHMEM, nvshmem_init(), nvshmem_malloc, barrier, nvshmem_finalize() 3/3 segfault (SOS: Out of symmetric memory, heap size 0) 3/3 pass, exit 0
app calls shmem_init(), then NVSHMEMX_INIT_WITH_SHMEM 3/3 pass 3/3 pass

An earlier revision without the library pin gave the same picture on SOS: 4/4 segfault on devel,
4/4 pass.

At 2 PEs, SOS 1.5.3 on that system fails its own first barrier (OFI error -22) without NVSHMEM
involved, so only 1 PE was possible there.

…INIT_WITH_SHMEM is set

With NVSHMEM_BOOTSTRAP=SHMEM and nvshmem_init(), nvshmemi_bootstrap()
passes a bootstrap_attr_t whose initialize_shmem is 0, and
bootstrap_set_bootattr() is called with a NULL attribute so it never
changes it. The SHMEM plugin therefore skips shmem_init() and queries
shmem_my_pe() on an OpenSHMEM library that was never initialized.

Only an application that passes NVSHMEMX_INIT_WITH_SHMEM has already
called shmem_init(). Derive initialize_shmem from that flag and always
give bootstrap_set_bootattr() the attribute that bootstrap_init() uses.
The plugin then initializes and finalizes OpenSHMEM itself in the
environment-selected case, as the MPI plugin does with MPI.

In that case the application usually does not link the OpenSHMEM
library, so unloading the plugin at finalize also unloaded it, and the
process crashed at exit (seen with Open MPI's OpenSHMEM). Keep the
library loaded with RTLD_NODELETE when the plugin initialized it.

Fixes NVIDIA#116

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: NVSHMEM_BOOTSTRAP=SHMEM segfaults with nvshmem_init(): plugin never calls shmem_init()

1 participant