Repository navigation
Conversation
…INIT_WITH_SHMEM is set With NVSHMEM_BOOTSTRAP=SHMEM and nvshmem_init(), nvshmemi_bootstrap() passes a bootstrap_attr_t whose initialize_shmem is 0, and bootstrap_set_bootattr() is called with a NULL attribute so it never changes it. The SHMEM plugin therefore skips shmem_init() and queries shmem_my_pe() on an OpenSHMEM library that was never initialized. Only an application that passes NVSHMEMX_INIT_WITH_SHMEM has already called shmem_init(). Derive initialize_shmem from that flag and always give bootstrap_set_bootattr() the attribute that bootstrap_init() uses. The plugin then initializes and finalizes OpenSHMEM itself in the environment-selected case, as the MPI plugin does with MPI. In that case the application usually does not link the OpenSHMEM library, so unloading the plugin at finalize also unloaded it, and the process crashed at exit (seen with Open MPI's OpenSHMEM). Keep the library loaded with RTLD_NODELETE when the plugin initialized it. Fixes NVIDIA#116 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #116.
Problem
nvshmemi_bootstrap()builds abootstrap_attr_twhoseinitialize_shmemdefaults to 0. It thencalls
bootstrap_set_bootattr()with a NULL attribute when the application passed none, so the fieldstays 0. With
NVSHMEM_BOOTSTRAP=SHMEMand plainnvshmem_init(), the SHMEM plugin therefore skipsshmem_init(). It then callsshmem_my_pe()on an OpenSHMEM library that was never initialized.#116 has the full trace.
Change
bootstrap_set_bootattr()now setsinitialize_shmemfromflags & NVSHMEMX_INIT_WITH_SHMEM.Only an application that passes that flag has called
shmem_init()itself.nvshmemi_bootstrap()always passes the attribute thatbootstrap_init()uses.shmem_finalize()already depends onnvshmem_initialized_shmem, so after this change the pluginalso finalizes what it initialized. The MPI plugin does the same with
MPI_Init. TheNVSHMEMX_INIT_WITH_SHMEMpath still passes 0 and is unchanged.dladdr(shmem_init)finds the library, thendlopen(..., RTLD_NOLOAD | RTLD_NODELETE)pins it.In this case the application usually does not link OpenSHMEM, so unloading the plugin at
nvshmem_finalize()also unloaded it. With Open MPI's OpenSHMEM, the process then crashed at exit(SIGSEGV) after
shmem_finalize().Testing
Setup: devel eaf955f vs this PR. Both columns use the host library and the SHMEM plugin built from
the same tree.
Open MPI 4.1 OpenSHMEM, 2 PEs on one node (B300). Launched with
mpirun; same program binarieson both sides.
NVSHMEM_BOOTSTRAP=SHMEM,nvshmem_init(),nvshmem_malloc, barrier,nvshmem_finalize()pshmem_my_pe: SHMEM is not initialized, abortshmem_init()succeeded (3/3) passes on both PEs and exits 0LD_PRELOAD=liboshmem.so)shmem_init(), thenNVSHMEMX_INIT_WITH_SHMEMThe remaining failures appear on both sides, including the unchanged
NVSHMEMX_INIT_WITH_SHMEMpath. They are
SHMEM_ABORTinside Open MPI'sshmem_init()itself, after a memheap "too manysegments are registered" warning and before NVSHMEM prints anything. They are an Open MPI issue on
this host.
Sandia OpenSHMEM 1.5.3 (PMI-2), 1 PE (GH200).
NVSHMEM_BOOTSTRAP=SHMEM,nvshmem_init(),nvshmem_malloc, barrier,nvshmem_finalize()Out of symmetric memory, heap size 0)shmem_init(), thenNVSHMEMX_INIT_WITH_SHMEMAn earlier revision without the library pin gave the same picture on SOS: 4/4 segfault on devel,
4/4 pass.
At 2 PEs, SOS 1.5.3 on that system fails its own first barrier (
OFI error -22) without NVSHMEMinvolved, so only 1 PE was possible there.