Repository navigation
Conversation
…parallel foreign scans The proposed solution is based on three layers: * a low level Rust wrapper over PostgreSQL LWLock, not tied to SHMEM hooks; * a mechanism for requesting a LWLock tranche on startup, stored as a static global, based on the existing PgSharedMemoryInitialization trait and the pg_shmem_init! macro; * high level components that are designed to be straightforwardly used in PostgreSQL FDW routines (pgrx_*_foreign_scan).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The primary goal of this PR is to implement a mechanism for locking dynamic shared memory (DSM) provided to worker processes during a parallel foreign scan.
The proposed solution is based on three layers:
PgSharedMemoryInitializationtrait and thepg_shmem_init!macro;pgrx_*_foreign_scan).Tests
This PR implements the same test coverage of PgLwLock for the new lock mechanisms. I also provided an example implementation in
pgrx-examples/parallel_scan_lwlockthat executes parallel scans, whose workers access a shared resource (a naive integer counter) as the source of tuples for foreign tables. Parallelism is forced by not letting workers exceed a quota of produced rows (in slow testing environments, the leader may complete the query before the parallel workers finish their startup overhead).Changes since #2141
The previous PR contained an error in the locks tranche id initialization, that didn't take into account
EXEC_BACKEND-enabled builds where any forked workerexecs a brand new instance, processing its own_PG_init(), therefore reinitializing the tranche id value. Tests run concurrently evidenced this behaviour on Windows builds.The test suite was also rewritten to provide a separate lock for each test.
PG19 introduced a breaking change in LWLock C API, therefore the lower layer of LWLocks implementation in this PR required a different code path for stable versions (PG18 and older) and the currently developed one. This may be subject to changes before the release of the stable PG19 version.