Repository navigation
GH-4569 Durability.NodeRole.Warmup: warm up a new release's projections inside the serving cluster - #5024
Merged
Conversation
…ns inside the serving cluster A warm-up node joins the cluster to catch up the new release's projection versions before it is cut over to, without doing any of the serving nodes' messaging. Opt-in, default NodeRole.Serving. Leader side - The role travels as the capability marker wolverine://node-role/warmup, in place of the durability marker, so no store's node table changes. Blue must run a version that understands it before green uses it. - A warm-up node never leads (CanLead = LeadershipEligible && Serving). - The grid makes a warm-up node a candidate only for agents no serving node declares, and detaches anything else sitting on one. Blue/green and group affinity distribute serving and warm-up nodes as two pools with their own ceilings, so the shared agents don't ping-pong between blue nodes. - DistributeEvenly (listener families), SingularAgent (recurring schedules, which prefer a non-leader) and durability affinity never place on a warm-up node. Warm-up node - Registers only EventSubscriptionAgentFamily, so it cannot build anything else even when a leader that predates roles asks. The family advertises async projections only (via TryCreateUsage, read once), never subscriptions. - Listens only to its control endpoint and that transport's reply endpoint: not every System endpoint, since the ASB retry queue is System and carries application messages. - No durable scheduled polling; RavenDB's dead letter replayer stays off. Not yet: starting the projection agents in Rebuild mode, resuming from their own progress, needs JasperFx/jasperfx#1015 published first. Docs: the warm-up section on the Marten distribution page (pointer from Polecat), including the checklist of what the application must turn off itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Fx 2.88.0) Every start of a projection shard on a warm-up node -- the first one, a resume after a rebuild or rewind, an auto-restart -- goes through IProjectionDaemon.StartAgentInRebuildModeAsync (JasperFx/jasperfx#1015): no side effects, keeps following the store, and resumes from the shard's own progress across restarts. A serving node still starts continuous. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4569.
Adds
opts.Durability.NodeRole = NodeRole.Warmup. A warm-up node joins the serving cluster and catches up the new release's projection versions before you cut over to it, without doing any of the serving nodes' messaging. The role is opt-in, and the default isNodeRole.Serving. The design discussion is on the issue (proposal, what changed).Leader side
wolverine://node-role/warmup, which a warm-up node publishes in place of the durability marker, so no store's node table changes. Blue has to be on a version that understands the role before green uses it (see the docs warning).CanLead => LeadershipEligible && NodeRole == Serving.DistributeEvenly, which covers the exclusive, dynamic and sticky-Postgres listener families;SingularAgent(recurring schedules), which until now preferred a non-leader node;On the warm-up node
EventSubscriptionAgentFamily, so it can't build anything else, even when a leader that predates roles asks it to.TryCreateUsage()once per process, because the role isn't onRegisteredShardNames().StartAgentInRebuildModeAsync. That means no side effects, and the shard keeps following the store and resumes from its own progress across pod restarts. Cutover is a restart asServing, which continues Continuous from the same progress. JasperFx 2.88.0: prove a failed Rebuild-mode run can be restarted and finished (#5668) marten#5669 proves that a failed Rebuild-mode run can be restarted and finished.EndpointRole.Systemis not the boundary: the ASB retry queue isSystemand carries application messages, and the NATS/MQTT reply endpoints aren'tSystemat all.Docs
There's a new "Warming Up a New Release on Its Own Nodes" section on the Marten distribution page, with a pointer from the Polecat page. It includes the checklist of what the application has to turn off itself: hosted services, HTTP traffic,
InvokeAsyncfrom your own code, and inline side effects inEventProjection.Apply().Tests
CoreTests.Runtime.Agents.warmup_node_role_assignment_4569: grid placement, 10 facts. The fullCoreTests.Runtime.Agentsnamespace passes (505 tests, 6 skipped).CoreTests.Runtime.Agents.warmup_event_subscription_agent_starts_in_rebuild_mode_4569: a warm-up node starts and resumes in Rebuild mode, and a serving node still starts Continuous.PostgresqlTests.Agents.warmup_node_role_host_4569: a serving host and a warm-up host built from the same configuration. The warm-up host listens only to its control endpoint, doesn't poll for scheduled messages, publishes its role instead of the durability marker, and can't lead.LeadershipElectionCompliance.a_warmup_node_never_leads_and_takes_no_ordinary_agents, which runs for every store. Postgres passes 22/22.🤖 Generated with Claude Code