Skip to content

fix: serialize migrate-on-boot across instances (coord #0041) - #169

Merged
klomkling merged 2 commits into
mainfrom
feature/migration-boot-lock
Jul 26, 2026
Merged

klomkling merged 2 commits into
mainfrom
feature/migration-boot-lock

Conversation

@klomkling

Copy link
Copy Markdown
Owner

Closes coord #41 (filed by propertiezy, target Themia.Data.Migrations).

Problem

FluentMigrator skips migrations already recorded in VersionInfo, so migrate-on-boot is a no-op once applied. The unsafe window is N instances booting simultaneously: all read VersionInfo, all see the same migration pending, and all apply it concurrently. Check-then-apply is not atomic across connections, so they collide on DDL and insert duplicate version rows. Every migrate-on-boot consumer hits this at horizontal scale.

Themia owns the runner entry point, so the fix belongs here rather than wrapped by each app.

Change

ThemiaMigrations.Run now holds the engine's session-level advisory lock over MigrateUp, taken on a dedicated connection. No API change; consumers get the fix on a version bump.

  • MigrationLock.cs (new, internal) — pg_advisory_lock / GET_LOCK / sp_getapplock
  • Explicit Npgsql / MySqlConnector / Microsoft.Data.SqlClient references (previously transitive via FluentMigrator.Runner.*)

Three decisions departed from the literal request, all deliberate:

  1. All three engines, not Postgres-only. Run is engine-generic and all three ship today; a lock that silently no-ops on 2 of 3 is a correctness cliff.
  2. On by default. An opt-in flag reproduces the "app authors get this wrong" failure the request wants absorbed.
  3. Unbounded wait. A booting instance cannot meaningfully proceed until the one ahead has finished migrating.

Two traps this turns on

The key must be scoped to the database name. The request's attached reference impl (propertiezy PR #30) uses a fixed constant. PostgreSQL advisory locks and MySQL's GET_LOCK are keyed server-wide, not per database, so a fixed key makes two unrelated Themia apps sharing one cluster queue behind each other's migrations. The key derives from the live connection.Database, which also avoids per-engine connection-string parsing. sp_getapplock is natively database-scoped but takes the same scoped name for uniformity.

Not string.GetHashCode(). It is randomized per process on .NET — every instance would derive a different key, nothing would contend, and the lock would silently protect nothing, failing in exactly the scenario it exists for while passing any same-process unit test. SHA-256, pinned with golden vectors computed independently of the implementation.

Also: CommandTimeout = 0 on the acquire — the 30s ADO.NET default would abort the wait long before a slow migration ahead had finished.

MigrationLock is kept internal on purpose: a public primitive invites RunExclusive(() => Run(...)), which self-deadlocks (the inner call opens a different connection; these are session locks).

Test plan

  • Falsified before trusting: removing the acquire makes RunExclusive_ShouldBlockASecondCaller fail (second caller runs straight through) while the cross-database test still passes — the test discriminates precisely.
  • 10/10 integration tests against real containers on all three engines — blocks-a-second-caller and no-contention-across-databases each, plus a 6-instance concurrent ThemiaMigrations.Run asserting exactly one VersionInfo row.
  • 15/14 unit tests (net10.0/net8.0) covering key derivation against golden vectors and the MySQL 64-char lock-name cap.
  • Clean full-solution build, 0 warnings (TreatWarningsAsErrors, no RS0016); full unit suite green.

Not included

Version bump and publish — the version is monorepo-shared, so the release is a whole-set decision. CHANGELOG entry sits under [Unreleased].

N instances booting at once all read VersionInfo, all see the same
migration pending, and all apply it concurrently — check-then-apply is
not atomic across connections, so they collide on DDL and duplicate the
version row. ThemiaMigrations.Run now holds the engine's session-level
advisory lock over MigrateUp on a dedicated connection.

The key is scoped to the database name because pg_advisory_lock and
GET_LOCK are keyed server-wide, not per database, and is a SHA-256
digest rather than string.GetHashCode(), which is randomized per process
and would leave every instance contending on a different key.
Ten review findings. The two that mattered most both silently voided
mutual exclusion while looking green: a lock session reaped mid-migration
(the unlock result was discarded, so nobody noticed) and connection.Database
being a connection-string echo rather than a normalised name, so Database=App
and Database=app derived unrelated keys and never contended.

- read the unlock result; warn when the lock was no longer held
- normalise the lock scope (trim + lower-case)
- fail closed on sp_getapplock: anything but a non-negative status is "not granted"
- send a positive GET_LOCK timeout; -1 is MySQL-only, not MariaDB-portable
- bound the wait (ThemiaMigrationOptions.LockTimeout) and log before blocking:
  an unbounded wait on a thread that cannot see a CancellationToken makes a
  replica crash-loop under a startup probe with no diagnostic
- unpool the lock connection so it cannot deadlock a max-pool-size-of-one app
- release best-effort on the failure path so it cannot mask the migration error
- report lock failures separately from DDL failures
- calibrate the tests' blocked-window against each engine's uncontended latency;
  a fixed 3s window let MariaDB pass with the acquire removed entirely
- run lock participants on dedicated threads; add a MariaDB container leg
@klomkling
klomkling merged commit 36923bc into main Jul 26, 2026
7 checks passed
@klomkling
klomkling deleted the feature/migration-boot-lock branch July 26, 2026 12:48
klomkling added a commit that referenced this pull request Jul 26, 2026
Migrate-on-boot is now serialized behind the target engine's session-level
advisory lock, scoped to the database (coord #41, PR #169). PATCH per the
pre-1.0 policy in CHANGELOG.md: a backwards-compatible fix plus an additive
ThemiaMigrationOptions overload, within the 0.9 milestone.
klomkling added a commit that referenced this pull request Jul 26, 2026
MINOR per the pre-1.0 policy in CHANGELOG.md — "a new module/package or a
phase boundary" — since this adds four packages. Bumping in the feature PR
follows the repo's normal pattern (#161 shipped 0.9.0, #160 shipped 0.8.8
this way); the separate release PR #172 was only needed because #169 merged
without a bump.
klomkling added a commit that referenced this pull request Jul 26, 2026
* feat: shared Data Protection key store (coord #42)

ASP.NET Core ships EF Core and Redis key repositories but no Dapper one, so
on the Themia Dapper stack the default is per-container filesystem keys: a
second instance starts, the key rings diverge, and auth cookies plus
antiforgery tokens stop round-tripping across instances. A provider gap, not
a new persistence layer (contrast the rejected #39).

Follows the Themia.Exceptional shape — neutral core with an
IDataProtectionKeyDialect seam plus one package per engine — over one
data_protection_keys schema owned by FluentMigrator.

- IDataProtectionBuilder extension, so SetApplicationName stays with the app;
  it is what keeps two apps sharing a table from reading each other's keys
- keys are per application, not per tenant: no multi-tenancy dependency
- created_at from the server clock, never the app's — a fleet disagrees
- one unparseable row is skipped, not fatal: returning nothing would fail
  every unprotect in the application
- keys are NOT encrypted at rest, matching ASP.NET's own EF/Redis providers;
  documented loudly since a DB spreads key material further than a filesystem

* fix: address review findings on the Data Protection key store (coord #42)

Ten findings from a high-effort review, two of them security.

Failing closed on an unreadable row resolves four at once. Skipping was
unsafe in two directions: XmlKeyManager writes <revocation> elements through
the same repository, so a dropped row could silently un-revoke compromised
key material; and if the whole table is unreadable the survivors are an
EMPTY ring, which Data Protection cannot tell from a fresh deployment — it
mints a new key and signs out every user while reporting healthy. It also
removes the ArgumentNullException escape on a NULL xml column and the need
for a logger at all, which was a hard constructor dependency that broke
hand-composed containers.

- implement IDeletableXmlRepository (net10 only; net8 has no such API) so
  revoked key material can actually be deleted instead of living forever
- AddSingleton, not TryAddSingleton: PersistKeysTo* is last-wins, and
  first-wins silently kept an earlier dialect while migrating a later DB
- pass ThemiaMigrationOptions through, so callers can supply the logger the
  registration-time migration needs when instances contend on the boot lock
- connectionString is required; it defaulted to null while runMigration
  defaulted to true, so the shortest legal call always threw
- SetApplicationName is NOT an isolation boundary — corrected in 4 packages,
  the CHANGELOG and the tests; it only sets the purpose-chain discriminator
- idempotency test asserted 0 == 0; now asserts a key exists and that the
  key itself, not just the count, survived the second instance

* chore: bump to 0.10.0 for the Data Protection key store

MINOR per the pre-1.0 policy in CHANGELOG.md — "a new module/package or a
phase boundary" — since this adds four packages. Bumping in the feature PR
follows the repo's normal pattern (#161 shipped 0.9.0, #160 shipped 0.8.8
this way); the separate release PR #172 was only needed because #169 merged
without a bump.
klomkling added a commit that referenced this pull request Oct 23, 2026
A user or role claim of type sid would reach the principal through the effective claims and could shadow the session id when read back; AddSession now removes any existing sid first. Refs coord #169.
klomkling added a commit that referenced this pull request Oct 23, 2026
RevokeAllForUserExceptAsync throws only for a null, blank or malformed id; a well-formed id that matches none of the user's sessions keeps nothing and revokes all. Refs coord #169.
klomkling added a commit that referenced this pull request Oct 23, 2026
RevokeAllForUserExceptAsync now throws, and revokes nothing, unless the kept id names an active refresh family of that user (unknown id, another user's family, revoked or expired). Keeping nothing would sign the caller out; the app catches and falls back to revoke-all plus a fresh pair. Refs coord #169.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant