Repository navigation
ateapi: Downgrade runtime-incompatible durable resumes to ROOTFS - #2880
Merged
Zoe Zhao (zoez7) merged 1 commit intoOct 19, 2026
Merged
Conversation
Previously, ResumeActor treated SNAPSHOT_FIDELITY_ROOTFS like SNAPSHOT_FIDELITY_MEMORY for sandbox runtime placement and downgraded durable snapshots on runtime-incompatible fallback workers all the way to SNAPSHOT_FIDELITY_VOLUMES because gVisor could not yet restore ROOTFS snapshots. Now that both gVisor and micro-VM workers support restoring ROOTFS snapshots, only SNAPSHOT_FIDELITY_MEMORY snapshots require a matching worker runtime: a rootfs upper layer is tied to the container image in the ActorTemplate rather than the sandbox runtime binary or host CPU features. Update requiredSnapshotRuntime to return a runtime requirement only for MEMORY snapshots, and update ensureAteletRestored to downgrade runtime-incompatible durable MEMORY restores to SNAPSHOT_FIDELITY_ROOTFS while keeping template-replaced restores downgraded to SNAPSHOT_FIDELITY_VOLUMES.
shrutiyam-glitch
marked this pull request as ready for review
October 19, 2026 15:23
Zoe Zhao (zoez7)
approved these changes
Oct 19, 2026
Bowei Du (bowei)
added
area/api-machinery
area/scheduling
area/storage
kind/feature
Benjamin Elder (BenTheElder)
pushed a commit
to BenTheElder/substrate
that referenced
this pull request
Oct 20, 2026
…nt-substrate#2881) A suspended actor with a local snapshot is pinned to its node. While that node was full, its resume failed with `ResourceExhausted` even when other nodes had room, until atelet detached the actor on its own after `--snapshot-detach-delay`. Resume now detaches the actor itself, with `DetachActor` from agent-substrate#2852, and restores it on another node. Part of agent-substrate#589. This covers a full node. A node that is gone, the scale-down case, is still open. #### How the move works Assignment still runs pinned to the node first. When it finds no room there, and a worker on another node has room, resume: 1. Calls `DetachActor` on the pinned node. atelet uploads the latest snapshot and deletes the local copies. 2. Records the detach, which marks the upload `COMPLETED` and clears the locality and `assigned_node`. 3. Assigns a worker on any node and restores from the durable snapshot. `DetachActor` is synchronous. If the snapshot isn't durable yet, the resume waits for atelet to upload it. #### What the restore keeps On a worker of the same runtime, the restore keeps the snapshot's fidelity. On a worker of another runtime, a MEMORY snapshot restores at ROOTFS, as for any durable-only actor since agent-substrate#2880. Process memory is lost, but rootfs writes and volumes are kept. ROOTFS and VOLUMES snapshots keep their fidelity on any worker of the same sandbox class. A template can't yet refuse the downgrade; agent-substrate#2818 would add that. #### When the actor doesn't move - No worker anywhere has room. The resume returns `ResourceExhausted` as before, without calling atelet. - The detach fails. The resume fails, and the actor stays SUSPENDED on its node, not crashed, since it can still resume there. - The node is gone. `DetachActor` can't reach atelet, so the resume returns `Unavailable` and the actor stays pinned. ```mermaid sequenceDiagram participant C as client participant A as ate-api-server participant L1 as atelet on node1 (full) participant O as object storage participant L2 as atelet on node2 C->>A: ResumeActor A->>A: assign on node1: no room A->>A: node2 has room A->>L1: DetachActor L1->>O: upload latest snapshot L1->>L1: delete local copies L1-->>A: state with latest_durable only A->>A: mark COMPLETED, clear locality and assigned_node A->>A: reload restore source, assign on node2 A->>L2: Restore(durable snapshot) L2->>O: download L2-->>A: ok A-->>C: RUNNING on node2 ``` #### Design choices - **Any runtime counts as room.** - Memory is an optimization for GA, so a worker that can't restore the snapshot's process memory still triggers the move, and the actor restores there at ROOTFS. - Assignment still prefers a worker that keeps it. - **No claim before the detach.** - Claiming a worker first would leave the claim and the snapshot record disagreeing if ate-api-server crashed before the detach. - In this order, every step ends in a state resume already handles. - Two actors detached for one free place race for it, and the loser has paid an upload, but it ends durable-only and its next resume schedules it onto any node without another detach. #### Also worth knowing - A concurrent update fails the record write with `Aborted`, and the next resume detaches again; atelet returns the same state. If the node has room by then, the resume restores there from the upload instead, and the record keeps the upload `PENDING`, because the detach cancelled atelet's report. - `DetachActor` errors map as the tag workflow maps `PromoteSnapshot` errors, plus atelet's `Aborted` passed through. - A resume that detaches records two scheduler samples, `no_capacity` then `assigned`. A dashboard that reads `no_capacity` as a failed resume will count these. - No API changes. - `TestResumeActor_RelocatesAfterDetach` checked `ResourceExhausted` at a point where node2 already had room, which now detaches. The check moved to before node2 exists. - The functional fake's `DetachActor` returns the named actor's last checkpointed snapshot, not the last of any actor.
2 tasks done
12 of 23 tasks
2 tasks done
Draft
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now that gVisor supports
SNAPSHOT_FIDELITY_ROOTFSalongside micro-VM (#2673), a snapshot's rootfs upper layer depends only on the container image (ActorTemplate) and not on the worker's sandbox runtime version or host CPU features.requiredSnapshotRuntime: Only impose aRequiredRuntimeplacement constraint when restoringSNAPSHOT_FIDELITY_MEMORYsnapshots;SNAPSHOT_FIDELITY_ROOTFSandSNAPSHOT_FIDELITY_VOLUMESsnapshots can schedule onto any eligible worker of the same sandbox class without runtime matching.ensureAteletRestored: When a durableMEMORYsnapshot falls back onto a runtime-incompatible worker, downgrade the restore fidelity toSNAPSHOT_FIDELITY_ROOTFS(preserving both rootfs writes and durable volumes) instead ofSNAPSHOT_FIDELITY_VOLUMES. Snapshots whoseActorTemplatewas replaced continue to downgrade toSNAPSHOT_FIDELITY_VOLUMES.workflow_resume_test.go,functionaltest/actor_test.go, anddocs/api-guide.mdto reflectROOTFSfallback semantics.Follow-up to Support splitting container rootfs and SNAPSHOT_FIDELITY_ROOTFS on gVisor #2673 and atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration #2660.