Skip to content

fix: guard scan recovery on startup with a Postgres advisory lock - #2104

Merged
netomi merged 1 commit into
mainfrom
fix/scan-recovery-cross-pod-lock
Aug 27, 2026
Merged

netomi merged 1 commit into
mainfrom
fix/scan-recovery-cross-pod-lock

Conversation

@netomi

@netomi netomi commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What

ExtensionScanJobRecoveryService.recoverOnStartup() fires independently in every instance's own JVM on ApplicationReadyEvent, with no cross-pod coordination. During a rolling update with several pods starting in the same window, each one runs the whole recovery pass, which causes:

How

Guards the whole recoverOnStartup() pass with a Postgres transaction-scoped advisory lock (pg_try_advisory_xact_lock). Only the instance that acquires the lock runs recovery; the rest skip with a debug log and rely on the winner. Chose the transaction-scoped variant over the session-scoped one to avoid lock-leak risk under HikariCP connection pooling - it releases automatically on commit/rollback, with no explicit unlock call needed.

Testing

Added ExtensionScanJobRecoveryServiceTest, covering:

  • The actual Postgres lock mechanism against two real, separate JDBC connections (standing in for two pods): mutual exclusion while one holds the lock, and automatic release on commit.
  • recoverOnStartup()'s control flow: skips recovery when the lock can't be acquired, runs it when it can, and doesn't even attempt the lock when scanning is disabled.

./gradlew test --tests 'org.eclipse.openvsx.scanning.*' passes locally.

Fixes #2082

recoverOnStartup() fires independently in every instance's own JVM on
ApplicationReadyEvent, with no cross-pod coordination. During a rolling
update with several pods starting in the same window, each one runs
the whole recovery pass:

- Duplicate log lines (the reported symptom in #2082), from
  recoverStaleScans() logging the same scan on every instance.
- Genuine duplicate work: recoverPendingJobs() can re-enqueue the same
  QUEUED sync job twice, and recoverValidatingScan() can submit the
  same VALIDATING scan for scanning twice.

Guard the whole pass with a transaction-scoped Postgres advisory lock
(pg_try_advisory_xact_lock), so only one instance actually runs
recovery per startup window; the rest skip with a debug log. Chose the
transaction-scoped variant over the session-scoped one to avoid
lock-leak risk under HikariCP connection pooling - it releases
automatically on commit/rollback, no explicit unlock needed.

Fixes #2082

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@netomi
netomi requested a review from cstamas August 27, 2026 08:33
@cstamas

cstamas commented Aug 27, 2026

Copy link
Copy Markdown
Member

@netomi
netomi merged commit 2562582 into main Aug 27, 2026
5 checks passed
@netomi
netomi deleted the fix/scan-recovery-cross-pod-lock branch August 27, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ExtensionScanJobRecoveryService recovers scanning jobs multiple times

2 participants