Skip to content

GH-4569 Durability.NodeRole.Warmup: warm up a new release's projections inside the serving cluster - #5024

Merged
jeremydmiller merged 2 commits into
mainfrom
gh-4569-warmup-node-role
Oct 15, 2026
Merged

jeremydmiller merged 2 commits into
mainfrom
gh-4569-warmup-node-role

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #4569.

Adds opts.Durability.NodeRole = NodeRole.Warmup. A warm-up node joins the serving cluster and catches up the new release's projection versions before you cut over to it, without doing any of the serving nodes' messaging. The role is opt-in, and the default is NodeRole.Serving. The design discussion is on the issue (proposal, what changed).

Leader side

  • No schema change. The role travels as the capability marker wolverine://node-role/warmup, which a warm-up node publishes in place of the durability marker, so no store's node table changes. Blue has to be on a version that understands the role before green uses it (see the docs warning).
  • Never leads. CanLead => LeadershipEligible && NodeRole == Serving.
  • Candidates. A warm-up node is only a candidate for event subscription agents that no serving node declares. Anything else sitting on one gets detached and goes back to a serving node.
  • Separate pools. Blue/green and group affinity distribute serving nodes and warm-up nodes as two independent pools, each with its own ceiling. A single ceiling over both pools would sit below what the serving nodes carry, and the shared agents would ping-pong between blue nodes (the Blue/green warm-up: blue-only partitions ping-pong between blue nodes when they cannot fit under the per-node ceiling #4999 shape).
  • Never placed on a warm-up node:
    • DistributeEvenly, which covers the exclusive, dynamic and sticky-Postgres listener families;
    • SingularAgent (recurring schedules), which until now preferred a non-leader node;
    • durability affinity.

On the warm-up node

  • One agent family. It registers only EventSubscriptionAgentFamily, so it can't build anything else, even when a leader that predates roles asks it to.
    • The family advertises async projections only, never subscriptions. It reads them through TryCreateUsage() once per process, because the role isn't on RegisteredShardNames().
    • It refuses to build a subscription.
  • Rebuild mode. Every start of a projection shard (the first start, a resume after a rebuild or rewind, an auto-restart) goes through JasperFx 2.88.0's StartAgentInRebuildModeAsync. That means no side effects, and the shard keeps following the store and resumes from its own progress across pod restarts. Cutover is a restart as Serving, which continues Continuous from the same progress. JasperFx 2.88.0: prove a failed Rebuild-mode run can be restarted and finished (#5668) marten#5669 proves that a failed Rebuild-mode run can be restarted and finished.
  • Control traffic only. It listens to its control endpoint and that transport's reply endpoint, and nothing else. EndpointRole.System is not the boundary: the ASB retry queue is System and carries application messages, and the NATS/MQTT reply endpoints aren't System at all.
  • Nothing scheduled. There's no durable scheduled polling. Today every Balanced node polls whether or not it was assigned anything. RavenDB's dead letter replayer stays off too.

Docs

There's a new "Warming Up a New Release on Its Own Nodes" section on the Marten distribution page, with a pointer from the Polecat page. It includes the checklist of what the application has to turn off itself: hosted services, HTTP traffic, InvokeAsync from your own code, and inline side effects in EventProjection.Apply().

Tests

  • CoreTests.Runtime.Agents.warmup_node_role_assignment_4569: grid placement, 10 facts. The full CoreTests.Runtime.Agents namespace passes (505 tests, 6 skipped).
  • CoreTests.Runtime.Agents.warmup_event_subscription_agent_starts_in_rebuild_mode_4569: a warm-up node starts and resumes in Rebuild mode, and a serving node still starts Continuous.
  • PostgresqlTests.Agents.warmup_node_role_host_4569: a serving host and a warm-up host built from the same configuration. The warm-up host listens only to its control endpoint, doesn't poll for scheduled messages, publishes its role instead of the durability marker, and can't lead.
  • LeadershipElectionCompliance.a_warmup_node_never_leads_and_takes_no_ordinary_agents, which runs for every store. Postgres passes 22/22.
  • JasperFx goes to 2.88.0.

🤖 Generated with Claude Code

jeremydmiller and others added 2 commits October 14, 2026 14:27
…ns inside the serving cluster

A warm-up node joins the cluster to catch up the new release's projection versions before it is
cut over to, without doing any of the serving nodes' messaging. Opt-in, default NodeRole.Serving.

Leader side
- The role travels as the capability marker wolverine://node-role/warmup, in place of the
  durability marker, so no store's node table changes. Blue must run a version that understands it
  before green uses it.
- A warm-up node never leads (CanLead = LeadershipEligible && Serving).
- The grid makes a warm-up node a candidate only for agents no serving node declares, and detaches
  anything else sitting on one. Blue/green and group affinity distribute serving and warm-up nodes
  as two pools with their own ceilings, so the shared agents don't ping-pong between blue nodes.
- DistributeEvenly (listener families), SingularAgent (recurring schedules, which prefer a
  non-leader) and durability affinity never place on a warm-up node.

Warm-up node
- Registers only EventSubscriptionAgentFamily, so it cannot build anything else even when a leader
  that predates roles asks. The family advertises async projections only (via TryCreateUsage,
  read once), never subscriptions.
- Listens only to its control endpoint and that transport's reply endpoint: not every System
  endpoint, since the ASB retry queue is System and carries application messages.
- No durable scheduled polling; RavenDB's dead letter replayer stays off.

Not yet: starting the projection agents in Rebuild mode, resuming from their own progress,
needs JasperFx/jasperfx#1015 published first.

Docs: the warm-up section on the Marten distribution page (pointer from Polecat), including the
checklist of what the application must turn off itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Fx 2.88.0)

Every start of a projection shard on a warm-up node -- the first one, a resume after a rebuild or
rewind, an auto-restart -- goes through IProjectionDaemon.StartAgentInRebuildModeAsync
(JasperFx/jasperfx#1015): no side effects, keeps following the store, and resumes from the shard's
own progress across restarts. A serving node still starts continuous.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 7a5e420 into main Oct 15, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blue/green: let a node run projection agents without doing any messaging

1 participant