Repository navigation
Pin that a re-added tenant's agent started before its first append projects every event - #5721
Merged
jeremydmiller merged 1 commit intoOct 23, 2026
Conversation
…ojects every event A tenant removed with RemoveTenantFromShardAsync and re-added with AddTenantToShardAsync starts fresh: a new per-tenant sequence from 1 and a new agent that catches up from the tenant's new, empty baseline. The running coordinator can start that agent before the tenant's first append commits. dynamic_tenant_lifecycle_on_shard_during_daemon leaves that ordering to a race and failed about one run in ten with "EventCount should be 3 but was 1": JasperFx's per-tenant high-water bookkeeping still held the mark the removed incarnation had reached, seeded the new agent with it, and the agent consumed that range as empty before the new sequence's first events existed. This test waits for the new agent before appending, so the race goes the same way every time. It fails on JasperFx 2.90.2 and passes on 2.91.0, which carries the fix (JasperFx/jasperfx#1026) and which master already consumes through JasperFx#5713. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 24, 2026
Closed
This was referenced Oct 31, 2026
This was referenced Nov 5, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A tenant removed with
RemoveTenantFromShardAsyncand re-added withAddTenantToShardAsync, while a coordinator keeps running, is documented to start fresh (docs/events/multitenancy.md, #4880 / #4908):Through JasperFx 2.90.2 it didn't when the coordinator started the new agent before the tenant's first append committed. JasperFx's per-tenant high-water bookkeeping still held the mark the removed incarnation had reached, and seeded the new agent with it. The agent consumed that range as empty before the new sequence's first events existed, and then applied only the events above it. Events went missing from the projection, and its progression row said it was caught up.
JasperFx/jasperfx#1026 fixed this in JasperFx 2.91.0, which master consumes since #5713. Marten has no test of its own for it.
dynamic_tenant_lifecycle_on_shard_during_daemonleaves the ordering to a race: Phase 3 appends right after the re-add, and the coordinator's 250 ms leadership cycle may or may not have started the agent by then. That race failed asreDoc!.EventCount should be 3 but was 1.Test
New file:
TenantPartitionedEventsTests/Sharded/re_added_tenant_agent_started_before_its_first_append.cs. It uses the same sharded, tenant-partitioned HotCold setup asdynamic_tenant_lifecycle_on_shard_during_daemon:Waiting for the agent before appending makes the race go the same way every time. The wait is a condition on the coordinator's daemons, read through
AllDaemonsAsync(), not a delay. It reads every daemon rather thanDaemonForDatabase(shard): when this test was written, calling that right after host start registered the shard's database id as a tenant. #5702 fixed that, and reading every daemon works either way.This PR adds only the test. It changes no production code and needs no package bump.
Red and green (net10.0, PostgreSQL 17)
re_added_tenant_agent_started_before_its_first_appenddynamic_tenant_lifecycle_on_shard_during_daemon7045d09bbwith JasperFx overridden to 2.90.2 (local only)counter.EventCount should be 3 but was 11e130e2, JasperFx 2.90.1), from the run that found thisreDoc!.EventCount should be 3 but was 17045d09bb(JasperFx 2.91.0)TenantPartitionedEventsTests.Shardedon master with this test: 42 of 42 pass.[Trait("Retry", "3")]stays ondynamic_tenant_lifecycle_on_shard_during_daemon. Its comment records failures that look like this one, but I can't show that nothing else caused them.Searched open and closed issues and PRs for "re-added tenant", the test's name, "re-added tenant high-water" and
RemoveTenantFromShardAsync. Only #4880, #4893, #4908, #5702 and #5713 come up, and none adds this test.🤖 Generated with Claude Code