Docker in Docker no longer working?

[natevw]

I have in my notes when I first started using container one of my very earliest experiments was to see if Docker-in-Docker (dind) would work. And I clearly remember it did! Here's what I was doing at the time:

container run --name my-dind --detach --rm docker.io/docker
container exec --tty --interactive my-dind sh
  docker run -it ubuntu bash

This was probably around container version 0.9.0 or so. But now with versions 1.0.0 initially and 1.3.0 now, I get only errors:

% container run --rm docker.io/docker
Certificate request self-signature ok
subject=CN=docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN=docker:dind client
/certs/client/cert.pem: OK
ip: can't find device 'nf_tables'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
ip: can't find device 'ip_tables'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
ip: can't find device 'ip6_tables'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.13 (nf_tables)
mount: permission denied (are you root?)

Is this something that was expected to have changed on the container side since #87 / #87 (comment) or did Docker change their own requirements since whatever image tag I would have been using at the time? It seems even the :dind-rootless tag runs into ± the same errors now.

I do note #2120 (comment) that perhaps the guest VM has some broken firewalling tools, but I'm not sure that would fully explain all the other errors (e.g. also missing /proc/modules and /lib/modules)?

[natevw]

Hmm, adding --cap-add ALL to the run (via #2041) gets things farther:

% container run --rm  --cap-add ALL docker.io/docker
Certificate request self-signature ok
subject=CN=docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN=docker:dind client
/certs/client/cert.pem: OK
iptables v1.8.13 (nf_tables)
time="2026-08-28T00:22:46.244063709Z" level=info msg="Starting up"
time="2026-08-28T00:22:46.245442417Z" level=info msg="containerd not running, starting managed containerd"
time="2026-08-28T00:22:46.246460625Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=59
time="2026-08-28T00:22:46Z" level=warning msg="Configuration migrated from version 2, use `containerd config migrate` to avoid migration" t="6.833µs"
time="2026-08-28T00:22:46.283084375Z" level=info msg="starting containerd" revision=aad11006b869517fcd3009450b6f82da282e1a9b version=v2.3.3

… [snip: bunch of msg="loading/skip loading plugin" and other startup stuff] …

time="2026-08-28T00:22:46.294450000Z" level=info msg="containerd successfully booted in 0.012276s"
time="2026-08-28T00:22:46.301541917Z" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
time="2026-08-28T00:22:46.301746834Z" level=info msg="CDI directory does not exist, skipping" dir=/var/run/cdi
time="2026-08-28T00:22:46.301758500Z" level=info msg="CDI directory does not exist, skipping" dir=/etc/cdi
time="2026-08-28T00:22:46.319766125Z" level=info msg="Creating a containerd client" address=/var/run/docker/containerd/containerd.sock timeout=1m0s
time="2026-08-28T00:22:46.321142875Z" level=warning msg="failed check for fsverity support" error="enable fsverity failed: operation not supported" path=/var/lib/docker/plugins/storage
time="2026-08-28T00:22:46.323609834Z" level=info msg="Loading containers: start."
time="2026-08-28T00:22:46.323683875Z" level=info msg="NRI is disabled"
time="2026-08-28T00:22:46.323694750Z" level=info msg="Starting daemon with containerd snapshotter integration enabled"
time="2026-08-28T00:22:46.326372375Z" level=info msg="Restoring containers: start."
time="2026-08-28T00:22:46.327728875Z" level=warning msg="Failed to find nft tool" error="exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:22:46.327741667Z" level=info msg="Deleting nftables IPv4 rules" error="failed to find nft tool: exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:22:46.327746459Z" level=info msg="Deleting nftables IPv6 rules" error="failed to find nft tool: exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:22:46.367129709Z" level=warning msg="unable to disable IPv6 router advertisement" error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
time="2026-08-28T00:22:46.367160667Z" level=warning msg="unable to disable IPv6 router advertisement" error="open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system"
time="2026-08-28T00:22:46.368005625Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2026-08-28T00:22:46.368170459Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2026-08-28T00:22:46.368183417Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2026-08-28T00:22:47.371648293Z" level=info msg="Daemon shutdown complete" error="failed to start daemon: Error initializing network controller: error creating default \"bridge\" network: failed to set IP forwarding '/proc/sys/net/ipv4/ip_forward' = '1': open /proc/sys/net/ipv4/ip_forward: read-only file system"
failed to start daemon: Error initializing network controller: error creating default "bridge" network: failed to set IP forwarding '/proc/sys/net/ipv4/ip_forward' = '1': open /proc/sys/net/ipv4/ip_forward: read-only file system

So maybe mostly just the #2041 networking regressions after all, plus nowadays DinD needs to running with more explicitly-granted privileges than older container builds required?

[natevw]

Live blogging here, sorry (or perhaps not sorry… ymmv…)

Circling back to more of the #2041 discussion by way of #2064 and then noting its resolution instead via #2069, combined with some feedback from the CLI argument parser, I was able to piece together a winning command:

% container run --rm  --cap-add ALL --masked-path NONE --read-only-path NONE docker.io/docker

[… snip …]

time="2026-08-28T00:37:14.889457126Z" level=info msg="Loading containers: start."
time="2026-08-28T00:37:14.889539876Z" level=info msg="NRI is disabled"
time="2026-08-28T00:37:14.889548459Z" level=info msg="Starting daemon with containerd snapshotter integration enabled"
time="2026-08-28T00:37:14.892173126Z" level=info msg="Restoring containers: start."
time="2026-08-28T00:37:14.894329251Z" level=warning msg="Failed to find nft tool" error="exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:37:14.894339834Z" level=info msg="Deleting nftables IPv4 rules" error="failed to find nft tool: exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:37:14.894343792Z" level=info msg="Deleting nftables IPv6 rules" error="failed to find nft tool: exec: \"nft\": executable file not found in $PATH"
time="2026-08-28T00:37:14.952790376Z" level=info msg="Loading containers: done."
time="2026-08-28T00:37:14.956912626Z" level=info msg="Docker daemon" commit=6a43e3d containerd-snapshotter=true storage-driver=overlayfs version=29.7.2
time="2026-08-28T00:37:14.957179126Z" level=info msg="Initializing buildkit"
time="2026-08-28T00:37:15.130452584Z" level=info msg="Completed buildkit initialization"
time="2026-08-28T00:37:15.135096167Z" level=info msg="Daemon has completed initialization"
time="2026-08-28T00:37:15.135146501Z" level=info msg="API listen on /var/run/docker.sock"
time="2026-08-28T00:37:15.135189084Z" level=info msg="API listen on [::]:2376"

So it's still upset about some "nft" tool but surely that's an issue within Docker's own docker.io/docker image itself one might assume.

Circling all the way back to one of my earliest tests, am now able to run Ubuntu in docker in container again with the extra magic:

natevw@Wilson ~ % container run --rm  --cap-add ALL --masked-path NONE --read-only-path NONE -d --name my-dind docker.io/docker
my-dind
natevw@Wilson ~ % container exec --tty --interactive my-dind sh
/ # docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
50914c2b24a1: Pull complete 
ed8299a102e9: Pull complete 
ee7f14aa75fd: Download complete 
Digest: sha256:2260313b31c8c011cd2eebe728008efac1b3982be73eb71348ea2648d2c0e09b
Status: Downloaded newer image for ubuntu:latest
root@7c39f6d4967f:/# cat /etc/issue
Ubuntu 26.04 LTS \n \l

UPDATE: in further testing the --masked-path NONE may not be necessary. So the secret sauces appear to be the --cap-add ALL and the --read-only-path NONE arguments.