Skip to content

fix(postgres-storage): serialize shared schema creation on its own lock - #723

Merged
blove merged 3 commits into
mainfrom
blove/pg-schema-create-race
Sep 18, 2026
Merged

blove merged 3 commits into
mainfrom
blove/pg-schema-create-race

Conversation

@blove

@blove blove commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Closes #709.

The bug

The first request to a freshly provisioned schema fails, deterministically:

error: duplicate key value violates unique constraint "pg_namespace_nspname_index" (23505)
GET /readyz -> 503 {"checks":{"requestStores":{"status":"failed"}},"status":"not_ready"}

Root cause

runMigrations takes pg_advisory_xact_lock keyed on schema.prefix.component, and the comment above it says why: the three stores get their own migrations table and their own lock so they version independently and do not serialize against each other. That is right for migrations.

But CREATE SCHEMA IF NOT EXISTS ran inside that same lock, and the schema is the one object all three components share. Three components hold three different locks, so nothing serialized the shared DDL, and IF NOT EXISTS is no more concurrency-safe than the CREATE TABLE IF NOT EXISTS on the next line: the loser raises a duplicate-key error rather than a benign no-op. The partly-created schema is why the reporter saw the checkpointer tables present, the threads and permissions tables missing, and a second run succeed.

This is the cold start after a deploy to a new environment, which is precisely what B4_PG_SCHEMA was added for.

The fix

Schema creation moves into its own short transaction under a lock keyed on the schema alone (schema:<name>). The shared DDL is serialized; the per-component migrations are not, so the independence the component key exists for is preserved.

Kept to a separate transaction deliberately. Holding a schema-wide lock for the length of a migration pass would serialize all three stores against each other, undoing the design. Committing the schema separately is safe because creating one is idempotent and additive, so a later failure leaves an empty schema rather than a wrong one.

Tests

  • Real Postgres, gated on B4_TEST_PGSTORAGE=1: the three stores the generated stores.mjs builds call ready() concurrently against a schema that does not exist. This reproduced the exact 23505 on pg_namespace_nspname_index before the change and passes after it, and it then writes and reads a thread to prove the schema is not left half-populated.
  • Statement-level, always runs: pins the ordering the fix depends on, that the schema is created under its own lock and committed before the component lock is taken. A future edit cannot quietly fold the DDL back under the component lock.

Validation

  • @b4run/postgres-storage full suite with the real-Postgres lane enabled: 9 files, 94 passed
  • @b4run/postgres-storage typecheck and lint
  • node scripts/check-changesets.mjs

Worth landing before the pending release: postgres-per-environment-schema is one of the changesets waiting in #673, so releasing without this ships a known first-run failure in the feature it belongs to.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
b4-run Ready Ready Preview Sep 18, 2026 2:04am UTC

Request Review

@blove
blove enabled auto-merge (squash) September 18, 2026 01:02

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: this PR received an intelligent (AI) code review. See the review comments on this PR.

blove and others added 3 commits September 17, 2026 19:03
The first request to a schema that does not exist yet failed with 23505 on
pg_namespace_nspname_index. runMigrations takes an advisory lock keyed on
schema, prefix and component so the three stores version independently, but
it ran CREATE SCHEMA IF NOT EXISTS inside that lock. The schema is the one
object all three share, so three components held three different locks and
raced on it, and IF NOT EXISTS is not concurrency-safe: the loser raised a
duplicate-key error rather than a benign no-op, leaving the schema partly
populated so only a second run succeeded.

Schema creation moves to its own short transaction under a lock keyed on the
schema alone. The shared DDL is serialized; the per-component migrations are
not, which is what the component key was for. Committing it separately is
safe because creating a schema is idempotent and additive.

Proven against real Postgres: three stores calling ready() concurrently on a
virgin schema reproduced the 23505 before the change and pass after it. The
statement-level test pins the ordering the fix depends on, so a future edit
cannot fold the DDL back under the component lock unnoticed.

Closes #709

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard added a second advisory lock to every cold start, which reds
hono-node-roundtrip's assertion that one cold start costs exactly one lock
per store component. That assertion is worth keeping: it is what proves
migrations run once per isolate rather than once per request.

Read the catalog first. A schema that already exists — and the default
`public` always does — needs no lock, no transaction and no DDL, so an
existing deployment pays one cheap catalog probe per component and no extra
lock. The probe is an optimization, not the correctness mechanism: two cold
starts can both read "missing" and both proceed, and the lock plus IF NOT
EXISTS is what makes that safe.

Also fixes assume-migrated's statement-sequence assertion, which pinned BEGIN
as the first statement. It now asserts on the component's own transaction,
which is what that test meant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The api-reference behavior contract for
postgres-storage.migration.instance-scoped fingerprints the assertions in
assume-migrated.test.ts, so that weakening them cannot silently leave a
documented claim unbacked. Changing those assertions for the schema guard
tripped it, which is the gate doing its job.

Repinned rather than relaxed. The claim it backs — an unflagged instance
begins a locked transaction — is still exactly what the test proves; only
the statement index moved, because the shared-schema guard opens a
transaction ahead of the component's own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch was successfully deployed

1 active and 1 inactive deployments
vercel-preview — 5bbd2a98 Deployed Sep 18, 2026 by blove via vercel-native #2070
Preview — 5bbd2a98 Deployed Sep 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

postgres-storage: concurrent CREATE SCHEMA IF NOT EXISTS races (23505) on first run with B4_PG_SCHEMA

1 participant