Support splitting container rootfs and SNAPSHOT_FIDELITY_ROOTFS on gVisor by nybidari · Pull Request #2673 · agent-substrate/substrate

nybidari · 2026-10-14T19:02:34Z

gVisor ActorTemplates can now set snapshotConfig.preferredFidelity: SNAPSHOT_FIDELITY_ROOTFS.
Pause and Suspend keep each container's rootfs writes and the DurableDir volumes, without process memory. Resume cold-boots with them back in place.

ateworker-gvisor saves container rootfs overlays and DurableDir volumes as two filesystem checkpoint bundles, rootfs/ and fs/:

ROOTFS checkpoints write both bundles with runsc fscheckpoint.
MEMORY checkpoints split both bundles out of the process checkpoint, so the rootfs is stored apart from the other state.
Restore passes --fs-restore-image-path for each bundle the requested fidelity uses. VOLUMES uses only fs/.
CreateActorTemplate no longer limits ROOTFS to sandboxClass: microvm, and the ROOTFS e2e cases now also run on gVisor.
Removes outdated text that limited gVisor templates to one DurableDir.

Breaking change

The default gVisor SandboxConfig moves to a newer runsc (v4). As with any gVisor version change, existing MEMORY and golden snapshots may not resume at full fidelity. Re-create golden snapshots after upgrading.
ateworker-gvisor now needs a runsc that supports multiple filesystem checkpoint bundles. A SandboxConfig pinned to an older runsc fails MEMORY checkpoints of actors that have a DurableDir.
New gVisor snapshots contain rootfs/ files, which older atelet and object-store-plugin builds reject. Upgrade atelet and the plugin before, or together with, ateworker-gvisor. Don't roll atelet back once new snapshots exist.

Fixes #2435. Completes the gVisor phase of #2447.

Unit tests pass (ateworker-gvisor, internal/resources, ateapi apivalidation)
gVisor e2e passes with the final nightly
Documentation updated

Fabricio Voznika (fvoznika) · 2026-10-14T21:46:59Z

-  # ateom-gvisor uses to save and restore durable-dir volumes.
-  - name: v3
+  # TODO: move both architectures to the first gs://gvisor nightly that
+  # includes google/gvisor@<commit> (multi-bundle filesystem checkpoint).


Missing commit

Updated with the 10/17 gVisor release and removed TODO

Benjamin Elder (BenTheElder) · 2026-10-15T00:15:48Z

+			if !errors.Is(err, fs.ErrNotExist) {
+				return nil, fmt.Errorf("while checking filesystem checkpoint manifest %q: %w", manifestPath, err)
+			}
+			if sub == resources.RootfsCheckpointSubdir && fidelity == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS {


The doc comment says MEMORY treats rootfs/ as optional so snapshots taken before the split still restore. We don't keep old snapshot formats working pre-1.0, so let's drop that and require rootfs/ for MEMORY too. A new MEMORY snapshot that is missing it should fail here with a clear error.

Updated the changes.

Dmitry Berkovich (dberkov) · 2026-10-15T03:59:22Z

-	phaseTeardown   = "teardown"
+	phaseCheckpoint   = "checkpoint"
+	phaseDurableDir   = "durable_dir"
+	phaseFSCheckpoint = "fs_checkpoint"


🤖 nit 🟢 – VOLUMES and ROOTFS both run a single runsc fscheckpoint, but record it under durable_dir and fs_checkpoint respectively. One phase name would keep the same operation in one column; ate.snapshot.fidelity already tells the two apart.

Dmitry Berkovich (dberkov) · 2026-10-17T14:41:18Z

+//     inside checkpoint.img.
+//   - Every fidelity uses `<checkpointDir>/fs` when
+//     `<checkpointDir>/fs/fscheckpoint.pb` exists.
+func fsRestoreArgs(checkpointDir string, fidelity ateompb.SnapshotFidelity) ([]string, error) {


🤖 should-fix 🟡 – On rebase this collides with a change on main: after restore, dropActorCheckpointCacheAsync now runs runsc wait -fsrestore for fsRestoreContainers(containers), which lists only durable-dir owners, before it drops the page cache and removes the restore dir. With rootfs/ restored, every container has an entry in runsc's wait map, so that list under-waits for ROOTFS and MEMORY. Under VOLUMES, waiting on a container with no durable dir fails with no filesystems restored for container. The wait list has to follow the bundles this function selects.

Rebased and fixed.

Dmitry Berkovich (dberkov) · 2026-10-17T14:41:18Z

-// reference volumes the template declares, and a ROOTFS preferred fidelity
-// needs a sandbox class that can capture rootfs changes without memory, which
-// today is the micro-VM class alone.
+// reference volumes the template declares.


🤖 nit 🟢 – This hunk is already on main. The change that removed SandboxConfig from ActorTemplate dropped the ROOTFS-needs-microvm rule and its test with it, so this and the rootfs fidelity on microvm accepted case go away on rebase.

Dmitry Berkovich (dberkov) · 2026-10-17T14:42:30Z

-snapshot of the volumes and each container's rootfs writes without guest
-memory; this template keeps `MEMORY` because the in-RAM counter is the point of
-the demo.
+The micro-VM class also serves `SNAPSHOT_FIDELITY_ROOTFS`, a snapshot of the


nit: could you please move this statement to general session, since this is available now for everyone.

Dmitry Berkovich (dberkov) · 2026-10-17T14:45:28Z

  // The structure of the checkpoint should generally be treated as opaque. For
-  // gVisor, the checkpoint consists of a checkpoint.img file that contains the
-  // memory, sentry state, and filesystem deltas.
+  // gVisor, a MEMORY checkpoint is checkpoint.img and its page files (memory


nit: Do we actually need mention the gVisor in the API explicitly?

No need, updated.

…isor Split container rootfs overlays into the rootfs/ filesystem checkpoint bundle alongside the fs/ durable-dir bundle, and enable SNAPSHOT_FIDELITY_ROOTFS for gVisor ActorTemplates. Move the default gVisor SandboxConfig to the 2026-10-17 nightly, the first to include gVisor commit b79256080dec, which saves and restores multiple filesystem checkpoint bundles.

…ed bundle - MEMORY restores now require rootfs/, like ROOTFS. Snapshots from before the split are not kept restorable before 1.0, so a missing bundle fails with a clear error. - runsc wait -fsrestore follows the bundles the restore selects: every container for ROOTFS and MEMORY, durable-dir mount owners for VOLUMES. - VOLUMES and ROOTFS both record their runsc fscheckpoint as fs_checkpoint; ate.snapshot.fidelity tells them apart. - The counter demo README describes ROOTFS in its intro, since it is no longer specific to one sandbox class.

…nt-substrate#2880) Now that gVisor supports `SNAPSHOT_FIDELITY_ROOTFS` alongside micro-VM (agent-substrate#2673), a snapshot's rootfs upper layer depends only on the container image (`ActorTemplate`) and not on the worker's sandbox runtime version or host CPU features. - **`requiredSnapshotRuntime`**: Only impose a `RequiredRuntime` placement constraint when restoring `SNAPSHOT_FIDELITY_MEMORY` snapshots; `SNAPSHOT_FIDELITY_ROOTFS` and `SNAPSHOT_FIDELITY_VOLUMES` snapshots can schedule onto any eligible worker of the same sandbox class without runtime matching. - **`ensureAteletRestored`**: When a durable `MEMORY` snapshot falls back onto a runtime-incompatible worker, downgrade the restore fidelity to `SNAPSHOT_FIDELITY_ROOTFS` (preserving both rootfs writes and durable volumes) instead of `SNAPSHOT_FIDELITY_VOLUMES`. Snapshots whose `ActorTemplate` was replaced continue to downgrade to `SNAPSHOT_FIDELITY_VOLUMES`. - **Tests & Docs**: Update `workflow_resume_test.go`, `functionaltest/actor_test.go`, and `docs/api-guide.md` to reflect `ROOTFS` fallback semantics. Follow-up to agent-substrate#2673 and agent-substrate#2660. <img width="1364" height="1688" alt="image" src="https://github.com/user-attachments/assets/7a105b64-2c30-440f-8679-5afbeb591f7d" /> - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR

nybidari force-pushed the gvisor-nightly-split-checkpoint branch 2 times, most recently from 100133a to cc9d314 Compare October 14, 2026 20:20

nybidari requested a review from Dmitry Berkovich (dberkov) October 14, 2026 20:31

nybidari force-pushed the gvisor-nightly-split-checkpoint branch from cc9d314 to b520933 Compare October 14, 2026 20:48

Fabricio Voznika (fvoznika) approved these changes Oct 14, 2026

View reviewed changes

Benjamin Elder (BenTheElder) reviewed Oct 15, 2026

View reviewed changes

Dmitry Berkovich (dberkov) reviewed Oct 15, 2026

View reviewed changes

Dmitry Berkovich (dberkov) reviewed Oct 17, 2026

View reviewed changes

nybidari added 2 commits October 17, 2026 18:58

nybidari force-pushed the gvisor-nightly-split-checkpoint branch from 5cf3dcf to e3da1a9 Compare October 17, 2026 19:09

nybidari requested review from Benjamin Elder (BenTheElder), Dmitry Berkovich (dberkov) and Fabricio Voznika (fvoznika) October 17, 2026 19:54

Dmitry Berkovich (dberkov) approved these changes Oct 17, 2026

View reviewed changes

Dmitry Berkovich (dberkov) added this pull request to the merge queue Oct 17, 2026

Merged via the queue into agent-substrate:main with commit b1e9ad1 Oct 17, 2026
15 checks passed

shrutiyam-glitch mentioned this pull request Oct 19, 2026

ateapi: Downgrade runtime-incompatible durable resumes to ROOTFS #2880

Merged

2 tasks

Davanum Srinivas (dims) mentioned this pull request Oct 21, 2026

ateom-gvisor: support multi-bundle split rootfs and durable-dir check… #2434

Closed

2 tasks

Benjamin Elder (BenTheElder) mentioned this pull request Oct 26, 2026

[bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226

Open

12 of 23 tasks

Conversation

nybidari commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nybidari commented Oct 14, 2026 •

edited

Loading