Skip to content

feat(postgres-storage): application document store — versioned JSONB with compare-and-swap - #742

Closed
blove wants to merge 1 commit into
mainfrom
blove/app-document-store
Closed

blove wants to merge 1 commit into
mainfrom
blove/app-document-store

Conversation

@blove

@blove blove commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Implements #734.

B4.run persists everything it owns — checkpoints, threads, permission grants — and nothing an application owns, so every stateful agent app rebuilds the same layer beside it. This adds that layer as a first-class store.

The API

import { ConflictError, createMemoryDocumentStore } from "@b4run/sdk"
import { documentStoreFromEnv } from "@b4run/postgres-storage/node"

const sessions = await documentStoreFromEnv<Session>({ name: "sessions" })

const key = await sessions.create(value)              // server-generated key, version 0
const doc  = await sessions.load(key)                 // { version, value } | undefined
await sessions.commit(key, doc.version, next)         // CAS; throws ConflictError when stale
await sessions.commit(threadId, null, binding)        // insert under a caller-supplied key
await sessions.delete(key)
Package Adds
@b4run/sdk Document, DocumentStore, ConflictError, createMemoryDocumentStore
@b4run/postgres-storage createPostgresDocumentStore, PostgresDocumentStore(Options), namingFromEnv, ResolvedNaming
@b4run/postgres-storage/node createPostgresDocumentStore (connection string), documentStoreFromEnv, closeDocumentStoresFromEnv
@b4run/testing runDocumentStoreConformance

The contract and the in-process implementation live in @b4run/sdk — the package every app already depends on, and one postgres-storage already has transitively through @b4run/permissions, so there is no new install weight. The important consequence is a single ConflictError class: two structurally identical copies would be a silent instanceof failure at the one moment the error is ever read.

One <prefix>_documents table per database, keyed (store_name, doc_key), rather than a table per named store. The store name is a bound parameter, never interpolated, so it may be any string and can never collide with a B4.run table — an app is free to call a store threads. doc_key is text, so a generated uuid and a caller-supplied thread id share one column.

commit(key, null, value) is INSERT … ON CONFLICT DO NOTHING RETURNING: the row count, not a prior SELECT, decides who claimed the binding. commit(key, n, value) is a single UPDATE … WHERE version = n RETURNING. No read-modify-write, no transaction, no lock.

The DDL trap

Shipped real migrations rather than append-only CREATE TABLE IF NOT EXISTS — and at no cost, because this repo's runMigrations already does forward-only versioned migrations under pg_advisory_xact_lock (#723) for the other three stores. That is the one deviation from the issue's requirement 5, and it strictly dominates it: CREATE TABLE IF NOT EXISTS never alters a table that already exists, which makes it a first-boot side effect rather than a schema definition.

The reasoning is written on DOCUMENTS_MIGRATIONS in packages/postgres-storage/src/schema.ts, where the next person to change the shape will read it, and states both rules:

  1. A shipped migration is frozen. A database that recorded version 1 will never run it again, so editing it gives new databases a shape older ones will never have. Change the shape by appending { version: 2, up: … } with ALTER TABLE.
  2. No column default is load-bearing. Every insert names every column and supplies every value; the DDL declares no defaults at all.

packages/postgres-storage/test/documents-ddl.test.ts pins migration 1's exact text, asserts no DEFAULT appears in any documents migration, and asserts every INSERT the store issues names all six columns — so editing history fails loudly with a message telling you to add version 2 instead. This is the exact failure the invoicing example hit: id uuid DEFAULT gen_random_uuid() in new code passed against a fresh database and wrote NULL into every database created before it.

The memory store is not a mock

createMemoryDocumentStore enforces CAS and structuredClones on the way in and on the way out, and it answers the same conformance suite as Postgres — runDocumentStoreConformance in @b4run/testing, following the existing runThreadsStoreConformance pattern. In-process on every run; against real Postgres in the B4_TEST_PGSTORAGE lane. 17 cases covering CAS, claim-once inserts, concurrent races, deep-copy in both directions, and ConflictError's payload.

Namespacing

Reuses what exists rather than inventing a parallel mechanism: the same assertIdentifier / IDENTIFIER_PATTERN (#682), the same runMigrations and its advisory lock (#723), and its own migration component so it versions independently of the other three stores. namingFromEnv reads B4_PG_SCHEMA / B4_PG_TABLE_PREFIX with the same $NAME indirection the generated stores.mjs uses, and carries the "change one, change the other" note already used between IDENTIFIER_PATTERN and @b4run/memory-pgvector. It takes its env explicitly rather than defaulting to process.env — a default parameter put an unguarded process reference into the edge-safe main entry, which test/api-reference-compatibility.test.ts caught.

Two things worth a reviewer's opinion

  • delete(key) is not in the issue's API. It is two lines in each backend and a store you cannot clean up is a leak, so it is included; say the word and it comes out.
  • Document<T> is the name the issue proposes, and it shadows the DOM Document at an import site. Happy to rename it if that bothers anyone.

First consumer

The hashbrown invoicing example is the intended first consumer and will delete roughly 340 lines when it adopts this — persistence/{types,postgres,schema,memory,from-env}.ts, of which only about 30 lines are actually about invoicing.

Test plan

Run from the worktree root, all green:

  • pnpm lint → Tasks: 29 successful, 29 total
  • pnpm build → Tasks: 26 successful, 26 total
  • pnpm typecheck → Tasks: 54 successful, 54 total
  • pnpm test → Test Files 578 total, Tests 6697 passed | 265 skipped
  • node scripts/check-docs.mjs → Docs completeness check passed.
  • pnpm check:release-inventory → pass 58, fail 0
  • pnpm pack:check → Pack check passed.

Postgres lane (Testcontainers postgres:16, the repo's existing gated lane):

  • B4_TEST_PGSTORAGE=1 pnpm --filter @b4run/postgres-storage test → Test Files 12 passed (12), Tests 133 passed (133)

That covers the conformance suite against real Postgres plus document-store-specific cases: two named stores sharing one table without seeing each other, a second store instance reattaching to a collection, eight independent stores racing a commit and a claim (one winner, seven ConflictErrors), eight concurrent cold starts all migrating, a jsonb value round-tripping unchanged, tables landing in the configured schema, and documentStoreFromEnv honouring B4_PG_SCHEMA and recovering from a failed cold start.

Scoped runs while developing:

  • pnpm --filter @b4run/testing test document-store-conformance → Tests 17 passed (17) (memory, no infrastructure)
  • pnpm --filter @b4run/web test → Test Files 36 passed (36)

Two notes on flakiness, both unrelated to this change and both passing in isolation on re-run: @b4run/workspace test/source-bundle.test.ts ("enforces 64 MiB total…") and @b4run/cli test/dev-command.test.ts ("serializes a watched edit…") each failed once in a full-suite run on a loaded machine. Re-running each suite alone: 96 passed and 19 passed.

Not released, not merged.

🤖 Generated with Claude Code

B4.run persisted everything it owns — checkpoints, threads, permission
grants — and nothing an application owns, so every stateful agent app
rebuilt the same layer beside it, and rebuilt its bugs with it. The
hashbrown invoicing example carries ~344 lines of it and only ~30 are
about invoicing.

`@b4run/sdk` now owns the contract and the in-process implementation:
`Document`, `DocumentStore`, `ConflictError`, `createMemoryDocumentStore`.
`create(value)` inserts under a generated key; `commit(key, version,
value)` replaces a document only if it is still at that version;
`commit(key, null, value)` inserts under a caller-supplied key and fails
if the row exists, which is how a binding keyed by someone else's id is
claimed exactly once across instances. One `ConflictError` class, in the
package every app already depends on, so `instanceof` holds whichever
backend produced it — two structurally identical classes would fail
silently at the one moment the error is read.

`@b4run/postgres-storage` adds `createPostgresDocumentStore`, and on the
`/node` entry `documentStoreFromEnv` / `closeDocumentStoresFromEnv`:
Postgres when `DATABASE_URL` is set, process memory when it is not,
memoized per process and per store name over one shared pool, with a
failed cold start deliberately not memoized so a database that comes back
is picked up rather than replaced by a cached error. All document stores
in a database share one `<prefix>_documents` table keyed `(store_name,
doc_key)`; the store name is a bound parameter rather than an identifier,
so an app may call a store `threads` without colliding with B4.run's own
`<prefix>_threads`. `schema`/`tablePrefix` and the `B4_PG_SCHEMA` /
`B4_PG_TABLE_PREFIX` bindings apply exactly as they do to the other three
stores; `namingFromEnv` reads them, and takes its env explicitly because
the main entry is edge-safe and `process` is undeclared on workerd.

The in-process store is not a mock. It enforces compare-and-swap and
deep-copies in and out, and `@b4run/testing` exports
`runDocumentStoreConformance` — one suite, answered by both
implementations: in-process on every run, and against real Postgres in
the `B4_TEST_PGSTORAGE` lane.

The table is created by a numbered, forward-only migration under the
existing advisory lock (#723), not by a bare `CREATE TABLE IF NOT
EXISTS`, which never alters a table that already exists and is therefore
a first-boot side effect rather than a schema definition. Two rules are
written where the next person will read them and pinned by
`test/documents-ddl.test.ts`: a shipped migration is frozen, and no
column default is load-bearing because every insert names every column.
That is the exact failure the reference implementation hit when
`id uuid DEFAULT gen_random_uuid()` reached new code — fresh databases
got the default, older ones wrote NULL.

Closes #734

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
b4-run Ready Ready Preview Sep 18, 2026 10:12pm UTC

Request Review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated approval: this PR received an intelligent (AI) code review. See the review comments on this PR.

@blove

blove commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Parked as a draft until a consumer is ready. Triage 2026-10-03: the code is sound and passed full CI, and its migration follows the rules #745/#880 later adopted, but nothing in the repo consumes the store yet (the only named consumer is the external Hashbrown invoicing app) and two API questions in the description are unanswered (keep delete; rename Document<T>, which shadows the DOM type). All 13 conflicts with main are mechanical (export lists, the restructured docs nav, page-count pins, the lastmod manifest). Rebase when a consumer commits, settling both questions then; if none appears within a release or two, close and keep #734 as the spec with a link to blove/app-document-store.

@blove

blove commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing as parked, per the 2026-10-03 triage rule (no consumer within a release or two). The work is preserved on branch blove/app-document-store (not deleted), and #734 stays open as the spec. When a consumer commits, rebase from that branch and settle the two open API questions (keep delete? rename Document<T>, which shadows the DOM type).

🤖 Generated with Claude Code

@blove blove closed this Oct 8, 2026

This branch was successfully deployed

1 active and 1 inactive deployments
Preview — 048dabfd Deployed Sep 18, 2026 by vercel[bot]
vercel-preview — 048dabfd Deployed Sep 18, 2026 by blove via vercel-native #2098
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant