Skip to content

Pin that a re-added tenant's agent started before its first append projects every event - #5721

Merged
jeremydmiller merged 1 commit into
JasperFx:masterfrom
lahma:fix/re-added-tenant-first-events
Oct 23, 2026
Merged

jeremydmiller merged 1 commit into
JasperFx:masterfrom
lahma:fix/re-added-tenant-first-events

Conversation

@lahma

@lahma lahma commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Problem

A tenant removed with RemoveTenantFromShardAsync and re-added with AddTenantToShardAsync, while a coordinator keeps running, is documented to start fresh (docs/events/multitenancy.md, #4880 / #4908):

a new per-tenant sequence starting at 1, no surviving projection state … a new agent for it that catches up from the tenant's new (empty) baseline

Through JasperFx 2.90.2 it didn't when the coordinator started the new agent before the tenant's first append committed. JasperFx's per-tenant high-water bookkeeping still held the mark the removed incarnation had reached, and seeded the new agent with it. The agent consumed that range as empty before the new sequence's first events existed, and then applied only the events above it. Events went missing from the projection, and its progression row said it was caught up.

JasperFx/jasperfx#1026 fixed this in JasperFx 2.91.0, which master consumes since #5713. Marten has no test of its own for it. dynamic_tenant_lifecycle_on_shard_during_daemon leaves the ordering to a race: Phase 3 appends right after the re-add, and the coordinator's 250 ms leadership cycle may or may not have started the agent by then. That race failed as reDoc!.EventCount should be 3 but was 1.

Test

New file: TenantPartitionedEventsTests/Sharded/re_added_tenant_agent_started_before_its_first_append.cs. It uses the same sharded, tenant-partitioned HotCold setup as dynamic_tenant_lifecycle_on_shard_during_daemon:

  1. Two tenants on one shard. The one to be re-added catches up to 2.
  2. It is removed, and the test waits until its agent is reaped.
  3. It is re-added, and the test waits until the coordinator has started its new agent. Only then are the tenant's three new events appended.
  4. The test asserts that the progression reaches 3 and that the projected document counted all 3 events.

Waiting for the agent before appending makes the race go the same way every time. The wait is a condition on the coordinator's daemons, read through AllDaemonsAsync(), not a delay. It reads every daemon rather than DaemonForDatabase(shard): when this test was written, calling that right after host start registered the shard's database id as a tenant. #5702 fixed that, and reading every daemon works either way.

This PR adds only the test. It changes no production code and needs no package bump.

Red and green (net10.0, PostgreSQL 17)

re_added_tenant_agent_started_before_its_first_append dynamic_tenant_lifecycle_on_shard_during_daemon
master 7045d09bb with JasperFx overridden to 2.90.2 (local only) 0 of 10 pass. All 10 fail with counter.EventCount should be 3 but was 1 not run
master before #5713 (1e130e2, JasperFx 2.90.1), from the run that found this 0 of 10 pass, same failure 17 of 20 pass. All 3 failures are reDoc!.EventCount should be 3 but was 1
master 7045d09bb (JasperFx 2.91.0) 10 of 10 pass 20 of 20 pass

TenantPartitionedEventsTests.Sharded on master with this test: 42 of 42 pass.

[Trait("Retry", "3")] stays on dynamic_tenant_lifecycle_on_shard_during_daemon. Its comment records failures that look like this one, but I can't show that nothing else caused them.

Searched open and closed issues and PRs for "re-added tenant", the test's name, "re-added tenant high-water" and RemoveTenantFromShardAsync. Only #4880, #4893, #4908, #5702 and #5713 come up, and none adds this test.

🤖 Generated with Claude Code

…ojects every event

A tenant removed with RemoveTenantFromShardAsync and re-added with AddTenantToShardAsync
starts fresh: a new per-tenant sequence from 1 and a new agent that catches up from the
tenant's new, empty baseline. The running coordinator can start that agent before the
tenant's first append commits. dynamic_tenant_lifecycle_on_shard_during_daemon leaves that
ordering to a race and failed about one run in ten with "EventCount should be 3 but was 1":
JasperFx's per-tenant high-water bookkeeping still held the mark the removed incarnation
had reached, seeded the new agent with it, and the agent consumed that range as empty
before the new sequence's first events existed.

This test waits for the new agent before appending, so the race goes the same way every
time. It fails on JasperFx 2.90.2 and passes on 2.91.0, which carries the fix
(JasperFx/jasperfx#1026) and which master already consumes through JasperFx#5713.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 5469576 into JasperFx:master Oct 23, 2026
32 checks passed
This was referenced Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants