Repository navigation
fix: serialize migrate-on-boot across instances (coord #0041) - #169
Merged
Merged
Conversation
N instances booting at once all read VersionInfo, all see the same migration pending, and all apply it concurrently — check-then-apply is not atomic across connections, so they collide on DDL and duplicate the version row. ThemiaMigrations.Run now holds the engine's session-level advisory lock over MigrateUp on a dedicated connection. The key is scoped to the database name because pg_advisory_lock and GET_LOCK are keyed server-wide, not per database, and is a SHA-256 digest rather than string.GetHashCode(), which is randomized per process and would leave every instance contending on a different key.
Ten review findings. The two that mattered most both silently voided mutual exclusion while looking green: a lock session reaped mid-migration (the unlock result was discarded, so nobody noticed) and connection.Database being a connection-string echo rather than a normalised name, so Database=App and Database=app derived unrelated keys and never contended. - read the unlock result; warn when the lock was no longer held - normalise the lock scope (trim + lower-case) - fail closed on sp_getapplock: anything but a non-negative status is "not granted" - send a positive GET_LOCK timeout; -1 is MySQL-only, not MariaDB-portable - bound the wait (ThemiaMigrationOptions.LockTimeout) and log before blocking: an unbounded wait on a thread that cannot see a CancellationToken makes a replica crash-loop under a startup probe with no diagnostic - unpool the lock connection so it cannot deadlock a max-pool-size-of-one app - release best-effort on the failure path so it cannot mask the migration error - report lock failures separately from DDL failures - calibrate the tests' blocked-window against each engine's uncontended latency; a fixed 3s window let MariaDB pass with the acquire removed entirely - run lock participants on dedicated threads; add a MariaDB container leg
This was referenced Jul 26, 2026
Merged
klomkling
added a commit
that referenced
this pull request
Jul 26, 2026
MINOR per the pre-1.0 policy in CHANGELOG.md — "a new module/package or a phase boundary" — since this adds four packages. Bumping in the feature PR follows the repo's normal pattern (#161 shipped 0.9.0, #160 shipped 0.8.8 this way); the separate release PR #172 was only needed because #169 merged without a bump.
klomkling
added a commit
that referenced
this pull request
Jul 26, 2026
* feat: shared Data Protection key store (coord #42) ASP.NET Core ships EF Core and Redis key repositories but no Dapper one, so on the Themia Dapper stack the default is per-container filesystem keys: a second instance starts, the key rings diverge, and auth cookies plus antiforgery tokens stop round-tripping across instances. A provider gap, not a new persistence layer (contrast the rejected #39). Follows the Themia.Exceptional shape — neutral core with an IDataProtectionKeyDialect seam plus one package per engine — over one data_protection_keys schema owned by FluentMigrator. - IDataProtectionBuilder extension, so SetApplicationName stays with the app; it is what keeps two apps sharing a table from reading each other's keys - keys are per application, not per tenant: no multi-tenancy dependency - created_at from the server clock, never the app's — a fleet disagrees - one unparseable row is skipped, not fatal: returning nothing would fail every unprotect in the application - keys are NOT encrypted at rest, matching ASP.NET's own EF/Redis providers; documented loudly since a DB spreads key material further than a filesystem * fix: address review findings on the Data Protection key store (coord #42) Ten findings from a high-effort review, two of them security. Failing closed on an unreadable row resolves four at once. Skipping was unsafe in two directions: XmlKeyManager writes <revocation> elements through the same repository, so a dropped row could silently un-revoke compromised key material; and if the whole table is unreadable the survivors are an EMPTY ring, which Data Protection cannot tell from a fresh deployment — it mints a new key and signs out every user while reporting healthy. It also removes the ArgumentNullException escape on a NULL xml column and the need for a logger at all, which was a hard constructor dependency that broke hand-composed containers. - implement IDeletableXmlRepository (net10 only; net8 has no such API) so revoked key material can actually be deleted instead of living forever - AddSingleton, not TryAddSingleton: PersistKeysTo* is last-wins, and first-wins silently kept an earlier dialect while migrating a later DB - pass ThemiaMigrationOptions through, so callers can supply the logger the registration-time migration needs when instances contend on the boot lock - connectionString is required; it defaulted to null while runMigration defaulted to true, so the shortest legal call always threw - SetApplicationName is NOT an isolation boundary — corrected in 4 packages, the CHANGELOG and the tests; it only sets the purpose-chain discriminator - idempotency test asserted 0 == 0; now asserts a key exists and that the key itself, not just the count, survived the second instance * chore: bump to 0.10.0 for the Data Protection key store MINOR per the pre-1.0 policy in CHANGELOG.md — "a new module/package or a phase boundary" — since this adds four packages. Bumping in the feature PR follows the repo's normal pattern (#161 shipped 0.9.0, #160 shipped 0.8.8 this way); the separate release PR #172 was only needed because #169 merged without a bump.
klomkling
added a commit
that referenced
this pull request
Oct 23, 2026
A user or role claim of type sid would reach the principal through the effective claims and could shadow the session id when read back; AddSession now removes any existing sid first. Refs coord #169.
klomkling
added a commit
that referenced
this pull request
Oct 23, 2026
RevokeAllForUserExceptAsync throws only for a null, blank or malformed id; a well-formed id that matches none of the user's sessions keeps nothing and revokes all. Refs coord #169.
klomkling
added a commit
that referenced
this pull request
Oct 23, 2026
RevokeAllForUserExceptAsync now throws, and revokes nothing, unless the kept id names an active refresh family of that user (unknown id, another user's family, revoked or expired). Keeping nothing would sign the caller out; the app catches and falls back to revoke-all plus a fresh pair. Refs coord #169.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes coord #41 (filed by propertiezy, target
Themia.Data.Migrations).Problem
FluentMigrator skips migrations already recorded in
VersionInfo, so migrate-on-boot is a no-op once applied. The unsafe window is N instances booting simultaneously: all readVersionInfo, all see the same migration pending, and all apply it concurrently. Check-then-apply is not atomic across connections, so they collide on DDL and insert duplicate version rows. Every migrate-on-boot consumer hits this at horizontal scale.Themia owns the runner entry point, so the fix belongs here rather than wrapped by each app.
Change
ThemiaMigrations.Runnow holds the engine's session-level advisory lock overMigrateUp, taken on a dedicated connection. No API change; consumers get the fix on a version bump.MigrationLock.cs(new, internal) —pg_advisory_lock/GET_LOCK/sp_getapplockNpgsql/MySqlConnector/Microsoft.Data.SqlClientreferences (previously transitive viaFluentMigrator.Runner.*)Three decisions departed from the literal request, all deliberate:
Runis engine-generic and all three ship today; a lock that silently no-ops on 2 of 3 is a correctness cliff.Two traps this turns on
The key must be scoped to the database name. The request's attached reference impl (propertiezy PR #30) uses a fixed constant. PostgreSQL advisory locks and MySQL's
GET_LOCKare keyed server-wide, not per database, so a fixed key makes two unrelated Themia apps sharing one cluster queue behind each other's migrations. The key derives from the liveconnection.Database, which also avoids per-engine connection-string parsing.sp_getapplockis natively database-scoped but takes the same scoped name for uniformity.Not
string.GetHashCode(). It is randomized per process on .NET — every instance would derive a different key, nothing would contend, and the lock would silently protect nothing, failing in exactly the scenario it exists for while passing any same-process unit test. SHA-256, pinned with golden vectors computed independently of the implementation.Also:
CommandTimeout = 0on the acquire — the 30s ADO.NET default would abort the wait long before a slow migration ahead had finished.MigrationLockis kept internal on purpose: a public primitive invitesRunExclusive(() => Run(...)), which self-deadlocks (the inner call opens a different connection; these are session locks).Test plan
RunExclusive_ShouldBlockASecondCallerfail (second caller runs straight through) while the cross-database test still passes — the test discriminates precisely.ThemiaMigrations.Runasserting exactly oneVersionInforow.TreatWarningsAsErrors, no RS0016); full unit suite green.Not included
Version bump and publish — the version is monorepo-shared, so the release is a whole-set decision. CHANGELOG entry sits under
[Unreleased].