microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor by dims · Pull Request #1904 · agent-substrate/substrate

Davanum Srinivas (dims) · 2026-09-25T23:48:47Z

A micro-VM container that drops NET_BIND_SERVICE could not bind a port below 1024; the same container on gVisor could, since its network stack has no privileged-port limit.

The kata agent now receives the spec's sysctls, and ShapeMicroVM sets net.ipv4.ip_unprivileged_port_start=0, so both runtimes behave the same. Neither sets no_new_privileges; gVisor runs with --allow-suid since #2361, so setuid binaries work on both.

The capabilities e2e suite gains a drop-ALL template whose probe listens on port 80, and the probe reports NoNewPrivs, asserted off on both lanes.

Tests pass: go test -race ./internal/ocispec/..., GOOS=linux go vet on kata and ocispec, go vet ./internal/e2e/...; e2e on both lanes in CI.
Appropriate changes to documentation are included in the PR: docs/api-guide.md.

Davanum Srinivas (dims) · 2026-09-27T11:54:09Z

/assign Maya Wang (@mayawang) Benjamin Elder (@BenTheElder)

Benjamin Elder (BenTheElder) · 2026-09-28T20:43:42Z

Both stay out of Build: runsc restore compares Sysctl and Process with the checkpoint-time spec, so a shared default would break restores of older gVisor snapshots

we have no back compat right now and aggressively prefer the approach that is more correct / maintainable long-term than accruing halfway compat attempts

Davanum Srinivas (dims) · 2026-09-29T01:42:03Z

thanks Tim Hockin (@thockin) Benjamin Elder (@BenTheElder) i've removed backward compat stuff.

Maya Wang (mayawang)

Code looks good. One question on where these two settings live, following Ben's note above.

Davanum Srinivas (dims) · 2026-10-17T12:07:47Z

thanks Benjamin Elder (@BenTheElder) fixed this up as #2361 merged

Davanum Srinivas (dims) · 2026-10-22T12:22:43Z

So with this PR, OpenShell runs in micro-VM actors without any workaround. Would be great to see it merged.

w00t! thanks for confirming!

Benjamin Elder (BenTheElder)

[explicitly tracking for M3 compat/security fix]

Davanum Srinivas (dims) · 2026-10-23T15:15:09Z

Benjamin Elder (@BenTheElder) fixed the comments. thanks!

The agent honors sysctls, and net.* keys need no network namespace of their own, which is what ateom-microvm produces. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Set net.ipv4.ip_unprivileged_port_start=0 in ShapeMicroVM. gVisor has no privileged-port check and runsc ignores this sysctl, so only the guest kernel needs it. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

The probe listens on port 80, so a caps-none template proves the bind on both runtimes. The probe also reports NoNewPrivs; neither runtime sets it, and gVisor runs with --allow-suid. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

The caps-exact note claimed the probe could not come up without NET_BIND_SERVICE. Low ports bind without it on both runtimes now, so the proof is caps-none's golden bake. Move the note there and update the fixture header and the test comments for the third template. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

The caps-exact template said what it grants three times: the config, its header and an inline comment. Keep the names in the config and give each template one header that says why. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

@dims

…s kept (agent-substrate#2392) Containers ignored the image's `USER` and `WORKDIR` and ran as root in `/`, and atelet's unpack dropped the file owners image layers record. So an image that hands a directory to its user, such as jupyter's `/home/jovyan`, didn't work as built. - atelet applies each tar entry's uid/gid and keeps setuid, setgid and sticky bits. It adds `CHOWN`, `FOWNER` and `FSETID` to its `DAC_OVERRIDE`. The image cache layout moves to version 3. - The process runs as the image's `USER`, resolved as Docker does against the image's own `/etc/passwd` and `/etc/group`, and starts in its `WORKDIR` (default `/`). - The jupyter demo runs as jovyan and drops `DAC_OVERRIDE` and `--allow-root`. The `USER` and `WORKDIR` commits are @dims's from agent-substrate#1918, kept as written. This PR supersedes agent-substrate#1918. Fixes agent-substrate#2036 #### Breaking change Please add the `breaking-change` label. - After an upgrade or rollback, atelet refuses the existing image cache. Delete `/var/lib/ate/image-cache` on each node once no actor runs on it, as atelet's error says. After an upgrade, a node reboot also works. - A custom atelet securityContext needs `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `FSETID`. - Images with a `USER` stop running as root, and images with a `WORKDIR` stop starting in `/`. A non-root `USER` is expected to have no effective capabilities, so `securityContext.capabilities.add` no longer helps it. This is checked under runc and in the gVisor source, not yet inside a gVisor or microVM sandbox. - gVisor snapshots of such images don't restore, because `runsc restore` checks each container's user and working directory. Recreate those templates. MicroVM snapshots resume as they were. - Files an earlier run wrote as root into a durable dir stay root-owned. Tested: `go test -race` on atelet, imagecache, ocispec, tarutil, kata and the probe fixture, on macOS and on Linux as non-root, and imagecache with `-count=3 -shuffle=on`. The root-only imagecache and tarutil tests pass as root on a privileged Linux container. `hack/verify-all.sh` passes. The jupyter image serves `/api` under Docker as 1000:100 with every capability dropped. @dims ran the jupyter demo and a non-root node image as their users on kind, on both runtimes, before agent-substrate#1904 merged; on microVM uid 1000 couldn't bind :80 then. A new imagefs e2e starts an actor from a probe image with `USER nonroot:nonroot` and `WORKDIR /home/nonroot`. It checks uid and gid 65532, no supplementary group besides 65532, no effective capabilities, the working directory, and that the user can create a file there. CI runs it on gVisor and on microVM. At this head nothing ran in a gVisor or microVM sandbox yet: not the jupyter demo and not the imagefs e2e. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com> Signed-off-by: Davanum Srinivas <davanum@gmail.com> Co-authored-by: Davanum Srinivas <davanum@gmail.com>

Davanum Srinivas (dims) marked this pull request as ready for review September 25, 2026 23:52

Davanum Srinivas (dims) marked this pull request as draft September 26, 2026 02:19

Davanum Srinivas (dims) changed the title ~~microvm: forward Linux.Sysctl to the kata agent and allow low ports~~ microvm: parity with gVisor for sysctls, low ports and no_new_privileges Sep 26, 2026

This was referenced Sep 26, 2026

Honor a container image's USER and WORKDIR #1918

Closed

agentgateway e2e lane fails on main since #1809: ateom-for-actor certificates carry no ActorIdentity extension #1922

Closed

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch 3 times, most recently from 3eb0b33 to 4c25c22 Compare September 27, 2026 02:00

Davanum Srinivas (dims) marked this pull request as ready for review September 27, 2026 02:25

Benjamin Elder (BenTheElder) added the area/microVM label Sep 28, 2026

Bowei Du (bowei) added area/security Security related issue/pr kind/bug Something isn't working / bugfixes labels Sep 30, 2026

Maya Wang (mayawang) reviewed Sep 30, 2026

View reviewed changes

Comment thread internal/ocispec/microvm.go Resolved

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch from 4c25c22 to 1fc797b Compare October 1, 2026 11:43

Davanum Srinivas (dims) changed the title ~~microvm: parity with gVisor for sysctls, low ports and no_new_privileges~~ microvm: forward sysctls, allow low ports; set no_new_privileges on every container Oct 1, 2026

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch from 1fc797b to f605f1e Compare October 1, 2026 12:04

Benjamin Elder (BenTheElder) reviewed Oct 7, 2026

View reviewed changes

Comment thread internal/ocispec/ocispec.go Resolved

Benjamin Elder (BenTheElder) reviewed Oct 7, 2026

View reviewed changes

Comment thread internal/ocispec/microvm.go Resolved

Comment thread docs/upgrade.md Resolved

Benjamin Elder (BenTheElder) reviewed Oct 7, 2026

View reviewed changes

Comment thread docs/api-guide.md Resolved

Benjamin Elder (BenTheElder) reviewed Oct 7, 2026

View reviewed changes

Comment thread docs/api-guide.md Resolved

Benjamin Elder (BenTheElder) mentioned this pull request Oct 14, 2026

Update default runsc flags configuration #2361

Merged

2 tasks done

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch 4 times, most recently from fda5193 to 8bfff79 Compare October 17, 2026 11:59

Davanum Srinivas (dims) changed the title ~~microvm: forward sysctls, allow low ports; set no_new_privileges on every container~~ microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor Oct 17, 2026

This was referenced Oct 22, 2026

Support sandbox platforms running on Substrate #553

Open

Proposal: Agent Substrate as a delegated isolation backend under RFC 12 NVIDIA/OpenShell#3362

Open

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch from ba19b5e to ff93ea5 Compare October 22, 2026 12:31

Benjamin Elder (BenTheElder) added this to the M3 milestone Oct 23, 2026

Benjamin Elder (BenTheElder) self-assigned this Oct 23, 2026

Benjamin Elder (BenTheElder) reviewed Oct 23, 2026

View reviewed changes

Comment thread internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl Resolved

Comment thread internal/e2e/suites/capabilities/capabilities_test.go Resolved

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch from 7e0a9bc to b0f3c4e Compare October 23, 2026 03:30

Benjamin Elder (BenTheElder) reviewed Oct 23, 2026

View reviewed changes

Comment thread internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl Resolved

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch 2 times, most recently from 2bf9ea1 to 21d9bf2 Compare October 23, 2026 14:59

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch 2 times, most recently from 2c55a29 to b126e05 Compare October 26, 2026 15:17

Benjamin Elder (BenTheElder) mentioned this pull request Oct 26, 2026

[bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226

Open

12 of 23 tasks

Davanum Srinivas (dims) added 5 commits October 27, 2026 07:59

microvm: forward Linux.Sysctl to the kata agent

be19b68

The agent honors sysctls, and net.* keys need no network namespace of their own, which is what ateom-microvm produces. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

microvm: let a capability-free container bind a low port

3099fdc

Set net.ipv4.ip_unprivileged_port_start=0 in ShapeMicroVM. gVisor has no privileged-port check and runsc ignores this sysctl, so only the guest kernel needs it. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

e2e: one comment per capabilities template

259d8a7

The caps-exact template said what it grants three times: the config, its header and an inline comment. Keep the names in the config and give each template one header that says why. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Davanum Srinivas (dims) force-pushed the pr/microvm-sysctls branch from b126e05 to 259d8a7 Compare October 27, 2026 12:02

Tim Hockin (thockin) added the prio/P0 Highest priority / required for next milestone label Oct 27, 2026

Benjamin Elder (BenTheElder) approved these changes Oct 27, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 27, 2026

Tim Hockin (thockin) added the milestone-ack label Oct 27, 2026

Merged via the queue into agent-substrate:main with commit 4150e4b Oct 27, 2026
5 checks passed

Walter Fender (cheftako) mentioned this pull request Oct 27, 2026

[Bug Bash] microVM: set containerd's default masked and read-only paths #3249

Open

1 of 2 tasks

Anish Ramasekar (aramase) mentioned this pull request Oct 29, 2026

Run containers as the image's USER and WORKDIR, with layer file owners kept #2392

Merged

1 of 2 tasks

Conversation

Davanum Srinivas (dims) commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Davanum Srinivas (dims) commented Sep 27, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 28, 2026

Uh oh!

Davanum Srinivas (dims) commented Sep 29, 2026

Uh oh!

Maya Wang (mayawang) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Davanum Srinivas (dims) commented Oct 17, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 22, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Davanum Srinivas (dims) commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Davanum Srinivas (dims) commented Sep 25, 2026 •

edited

Loading