Honor a container image's USER and WORKDIR by dims · Pull Request #1918 · agent-substrate/substrate

Davanum Srinivas (dims) · 2026-09-26T11:40:09Z

USER resolves as Docker does, against the image's own /etc/passwd and /etc/group: uid:gid as is, a bare uid with its login group (gid 0 without an entry), names looked up or Run/Restore fails, group memberships as supplementary groups. No USER means root. atelet resolves it into the worker's ContainerSpec; the pause container follows the same rule through WorkloadSpec.pause_user. The process starts in the image's WORKDIR (default /), created in the actor's upper when the image lacks it.

Behavior change: images with a USER stop running as root; images with a WORKDIR stop starting in /. On gVisor no snapshot from before this restores: runsc restore checks Process.User and Process.Cwd of every container, and the pause image sets USER 65535. Micro-VM snapshots resume as they were.

Merge after #2392 (image layer owners): without it a non-root process cannot write the files its image gives it. On the micro-VM class, images that serve on a privileged port also need #1904.

Live run on kind, both classes, main + #1906 + #2392 + this PR: a node image (USER 1000, WORKDIR /app, durable dir) and the jupyter demo run as their user, write to their workdir and durable dir, and resume on demand after a suspend. On micro-VM uid 1000 cannot bind port 80 (#1904); gVisor allows it.

The actor images this repository builds with ko (the counter, egress and fspersist demos, the e2e probe and test server, glutton) bind port 80 or write at the root of their filesystem, and the default base is nonroot since #3028. .ko.yaml keeps them on the root variant of the same distroless build: they write at the root of their filesystem and into the durable-dir volumes atelet creates as root.

Tests pass: go test -race on Linux for atelet, imagecache, ocispec, ateworker-gvisor and kata.
Appropriate changes to documentation are included in the PR

Davanum Srinivas (dims) · 2026-09-27T11:53:58Z

/assign Maya Wang (@mayawang) Benjamin Elder (@BenTheElder)

… note the rootfs root dependency of kagent-upstream#125 (agent-substrate#1918)

Maya Wang (mayawang)

The USER and WORKDIR resolution looks right to me, and moving to moby/sys/user covers Ben's gid point. My main ask is a live run before this merges, since it changes who real images run as. It also needs a rebase: internal/ocispec/parity_test.go conflicts with main.

… root dependency of kagent-upstream#125 (#94) docs(fork): the egress default's upstream exit is agent-substrate#1558; note the rootfs root dependency of kagent-upstream#125 (agent-substrate#1918)

…#1918

Maya Wang (mayawang) · 2026-10-02T20:31:32Z

Two gVisor issues from Preston Holmes (@ptone) that every image this change moves to non-root will hit: #2035 (the actor rootfs / is mode 0700, so a non-root user can't resolve any path) and #2036 (files the image gives to a non-root user show up as owned by uid 0). There's a fork fix for #2036 in giantswarm#115. Neither is covered by this stack, and I think we need a plan for both before this lands, otherwise images with a USER go from running as root to not starting. Davanum Srinivas (@dims) Benjamin Elder (@BenTheElder) let me know what you think.

Maya Wang (mayawang) · 2026-10-20T18:34:09Z

One case to plan for once this lands: images whose USER names the workload's user while the entrypoint has to start as root and drop privileges itself. OpenShell's sandbox images are an example. They set a non-root USER, and its Kubernetes driver sets runAsUser: 0 on the container so the supervisor starts as root, builds the sandbox's namespaces, and runs the workload as that user.

With this PR as it stands, those images would start the supervisor as the non-root user and fail. Could Container get an override, in the spirit of Kubernetes' securityContext.runAsUser/runAsGroup? Happy to open a separate issue if that's easier to track.

(Checked on 70358cfc: today both classes still run such an image as root, so nothing is broken yet.)

Davanum Srinivas (dims) · 2026-10-20T21:05:03Z

Maya Wang (@mayawang) i will take up your offer, please open another issue. Let me rebase this and get it to green again.

Benjamin Elder (BenTheElder) · 2026-10-25T00:05:23Z

With this PR as it stands, those images would start the supervisor as the non-root user and fail. Could Container get an override, in the spirit of Kubernetes' securityContext.runAsUser/runAsGroup? Happy to open a separate issue if that's easier to track.

This makes sense.

For the moment I think we're blocked on the large PR to make layers match their image owners. They're security fixes IMO so still viable during code freeze, but we have a lot to work on.

Davanum Srinivas (dims) · 2026-10-25T16:24:33Z

Benjamin Elder (@BenTheElder) #2392 ?

Benjamin Elder (BenTheElder) · 2026-10-26T16:39:30Z

Yeah. Commented more there

rootfs and upper were 0700. On gVisor the container's / takes that mode, so a non-root process could not search its own root. 0755 is what containerd and every image root use; the 0700 actor directory above keeps other actors out. The micro-VM guest already sees 0755. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

atelet writes the OCI spec before the worker mounts the rootfs, so it cannot read the image's /etc/passwd from a mount. Image.ReadFile reads a file from the cached layers top-down, honoring recorded whiteouts and opaque dirs. Capped at 1 MiB. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Resolve USER as Docker does, against the image's own /etc/passwd and /etc/group: uid:gid as is, a bare uid with its login group (gid 0 without an entry), names looked up or Run/Restore fails, group memberships as supplementary groups. No USER means root. atelet resolves it into the worker's ContainerSpec. runsc restore checks Process.User, so a gVisor snapshot taken before this of an image that sets USER needs re-taking. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Start the process in the image's WORKDIR instead of "/", and create the directory in the actor's upper when the image lacks it. atelet resolves it into the worker's ContainerSpec. runsc restore checks Process.Cwd as well as Process.User, so gVisor snapshots from before this need re-taking. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Substrate makes no compatibility promise before v1.0.0, so a note about one release's snapshots is noise in the upgrade procedure. The API guide describes how the image's USER and WORKDIR apply. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

The actor images built from this repository write at the root of their filesystem and into the durable-dir volumes atelet creates as root. With the default base now nonroot and a container running as its image USER, they would start as uid 65532 and lose those writes, so the e2e demos break. These images stay on the root variant of the same distroless build. Signed-off-by: Davanum Srinivas <davanum@gmail.com>

Davanum Srinivas (dims) · 2026-10-29T19:59:01Z

this will be part of #2392 - thanks for picking up my commit from here Anish Ramasekar (@aramase)

@dims

…s kept (agent-substrate#2392) Containers ignored the image's `USER` and `WORKDIR` and ran as root in `/`, and atelet's unpack dropped the file owners image layers record. So an image that hands a directory to its user, such as jupyter's `/home/jovyan`, didn't work as built. - atelet applies each tar entry's uid/gid and keeps setuid, setgid and sticky bits. It adds `CHOWN`, `FOWNER` and `FSETID` to its `DAC_OVERRIDE`. The image cache layout moves to version 3. - The process runs as the image's `USER`, resolved as Docker does against the image's own `/etc/passwd` and `/etc/group`, and starts in its `WORKDIR` (default `/`). - The jupyter demo runs as jovyan and drops `DAC_OVERRIDE` and `--allow-root`. The `USER` and `WORKDIR` commits are @dims's from agent-substrate#1918, kept as written. This PR supersedes agent-substrate#1918. Fixes agent-substrate#2036 #### Breaking change Please add the `breaking-change` label. - After an upgrade or rollback, atelet refuses the existing image cache. Delete `/var/lib/ate/image-cache` on each node once no actor runs on it, as atelet's error says. After an upgrade, a node reboot also works. - A custom atelet securityContext needs `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `FSETID`. - Images with a `USER` stop running as root, and images with a `WORKDIR` stop starting in `/`. A non-root `USER` is expected to have no effective capabilities, so `securityContext.capabilities.add` no longer helps it. This is checked under runc and in the gVisor source, not yet inside a gVisor or microVM sandbox. - gVisor snapshots of such images don't restore, because `runsc restore` checks each container's user and working directory. Recreate those templates. MicroVM snapshots resume as they were. - Files an earlier run wrote as root into a durable dir stay root-owned. Tested: `go test -race` on atelet, imagecache, ocispec, tarutil, kata and the probe fixture, on macOS and on Linux as non-root, and imagecache with `-count=3 -shuffle=on`. The root-only imagecache and tarutil tests pass as root on a privileged Linux container. `hack/verify-all.sh` passes. The jupyter image serves `/api` under Docker as 1000:100 with every capability dropped. @dims ran the jupyter demo and a non-root node image as their users on kind, on both runtimes, before agent-substrate#1904 merged; on microVM uid 1000 couldn't bind :80 then. A new imagefs e2e starts an actor from a probe image with `USER nonroot:nonroot` and `WORKDIR /home/nonroot`. It checks uid and gid 65532, no supplementary group besides 65532, no effective capabilities, the working directory, and that the user can create a file there. CI runs it on gVisor and on microVM. At this head nothing ran in a gVisor or microVM sandbox yet: not the jupyter demo and not the imagefs e2e. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com> Signed-off-by: Davanum Srinivas <davanum@gmail.com> Co-authored-by: Davanum Srinivas <davanum@gmail.com>

This was referenced Sep 26, 2026

agentgateway e2e lane fails on main since #1809: ateom-for-actor certificates carry no ActorIdentity extension #1922

Closed

imagecache: make the merged rootfs root searchable by non-root #1905

Closed

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 7321a72 to 4ab3b53 Compare September 26, 2026 23:34

Davanum Srinivas (dims) mentioned this pull request Sep 27, 2026

atelet: make durable-dir volumes writable by non-root containers #1906

Merged

2 tasks done

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 4ab3b53 to 58e3ad2 Compare September 27, 2026 01:30

Davanum Srinivas (dims) marked this pull request as ready for review September 27, 2026 02:25

Quentin Bisson (QuentinBisson) mentioned this pull request Sep 28, 2026

feat(harness): run Claude Code as the image's unprivileged user when the harness runs as root giantswarm/kagent-upstream#125

Merged

Benjamin Elder (BenTheElder) reviewed Sep 28, 2026

View reviewed changes

Comment thread cmd/atelet/oci.go Resolved

Comment thread internal/ocispec/ocispec.go Resolved

Comment thread cmd/atelet/oci_test.go Resolved

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 58e3ad2 to 9ed0a39 Compare September 29, 2026 00:58

Davanum Srinivas (dims) marked this pull request as draft September 29, 2026 01:05

Davanum Srinivas (dims) marked this pull request as ready for review September 29, 2026 01:28

This was referenced Sep 29, 2026

docs(fork): egress default exits through #1558; rootfs root dependency of kagent-upstream#125 giantswarm/substrate#94

Merged

docs(fork): align the ledger with giantswarm/giantswarm#37742 after the 2026-09-30 audit giantswarm/substrate#95

Merged

Quentin Bisson (QuentinBisson) added a commit to giantswarm/substrate that referenced this pull request Sep 29, 2026

docs(fork): the egress default's upstream exit is agent-substrate#1558;…

fd5e65d

… note the rootfs root dependency of kagent-upstream#125 (agent-substrate#1918)

Bowei Du (bowei) added area/node area/security Security related issue/pr kind/feature An enhancement / feature request or implementation labels Sep 30, 2026

Maya Wang (mayawang) reviewed Sep 30, 2026

View reviewed changes

Comment thread cmd/atelet/oci.go Resolved

Comment thread docs/api-guide.md Resolved

Comment thread internal/imagecache/imagefile.go Resolved

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 9ed0a39 to 0ca0710 Compare October 1, 2026 11:51

Davanum Srinivas (dims) changed the title ~~atelet: honor a container image's own USER~~ atelet: honor a container image's USER and WORKDIR Oct 1, 2026

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 0ca0710 to d41a2fc Compare October 1, 2026 12:04

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 1, 2026

fix(imagecache): carry the rootfs root fix of agent-substrate/substrate#1918 giantswarm/substrate#101

Merged

Quentin Bisson (QuentinBisson) added a commit to giantswarm/substrate that referenced this pull request Oct 1, 2026

docs(fork): carry the rootfs root fix of agent-substrate#1918

750f3ea

This was referenced Oct 1, 2026

docs(fork): name the rootfs root commits by their SHAs on the line giantswarm/substrate#102

Merged

feat(atelet): carry the image USER and WORKDIR stack of agent-substrate/substrate#1918 giantswarm/substrate#103

Merged

Quentin Bisson (QuentinBisson) added a commit to giantswarm/substrate that referenced this pull request Oct 1, 2026

docs(fork): carry the image USER and WORKDIR stack of agent-substrate…

3deee38

…#1918

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 2, 2026

fix(imagecache): keep the owner an image layer gives its files giantswarm/substrate#115

Merged

2 tasks done

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 20, 2026

docs(fork): record the compat-migrations theme decision in its ledger headers giantswarm/substrate#516

Merged

Davanum Srinivas (dims) force-pushed the pr/image-user branch 6 times, most recently from 7f4bf05 to fc7c1bd Compare October 24, 2026 11:38

Davanum Srinivas (dims) force-pushed the pr/image-user branch from fc7c1bd to 54bfacc Compare October 26, 2026 15:19

Benjamin Elder (BenTheElder) mentioned this pull request Oct 26, 2026

[bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226

Open

12 of 23 tasks

Davanum Srinivas (dims) force-pushed the pr/image-user branch 2 times, most recently from e0bec5d to 063854a Compare October 27, 2026 12:03

Davanum Srinivas (dims) added 6 commits October 28, 2026 06:54

Davanum Srinivas (dims) force-pushed the pr/image-user branch from 063854a to 8b692a5 Compare October 28, 2026 11:02

Tim Hockin (thockin) added this to the M3 milestone Oct 29, 2026

Tim Hockin (thockin) assigned Tim Hockin (thockin) and Benjamin Elder (BenTheElder) Oct 29, 2026

Dmitry Berkovich (dberkov) added the milestone-ack label Oct 29, 2026

Davanum Srinivas (dims) closed this Oct 29, 2026

Navigation Menu

Repository navigation

Honor a container image's USER and WORKDIR - #1918

Honor a container image's USER and WORKDIR#1918
Davanum Srinivas (dims) wants to merge 6 commits into
agent-substrate:mainfrom
dims:pr/image-user

Davanum Srinivas (dims) commented Sep 26, 2026 •

edited

Loading

Uh oh!

Davanum Srinivas (dims) commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Maya Wang (mayawang) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Maya Wang (mayawang) commented Oct 2, 2026

Uh oh!

Maya Wang (mayawang) commented Oct 20, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 20, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 25, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 25, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 26, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Davanum Srinivas (dims) commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Davanum Srinivas (dims) commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Maya Wang (mayawang) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Maya Wang (mayawang) commented Oct 2, 2026

Uh oh!

Maya Wang (mayawang) commented Oct 20, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 20, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 25, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 25, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 26, 2026

Uh oh!

Davanum Srinivas (dims) commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Davanum Srinivas (dims) commented Sep 26, 2026 •

edited

Loading