Bug: RYUK container startup failure when running Docker in rootless mode

Describe the bug

When Docker is run in rootless mode, the ryuk fails to start as the docker socket mounted as a volume has the wrong permissions (nobody:nobody). This can be fixed be mounting the correct socket at /run/user/$(id -u)/docker.sock.

The overwrite can be done by setting the environment variable TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE to the above mentioned socket.

Ideally, this would be detected and configured automatically (ie. using Server.Security Options.rootless and Endpoints.docker.Host of the active context or the DOCKER_HOST environment variable).

If this is not an option, please consider documenting this case more in detail.

To Reproduce

Configure Docker to run in rootless mode (see https://docs.docker.com/engine/security/rootless/) and run the following snippet:

>>> from testcontainers.postgres import PostgresContainer
>>> with PostgresContainer() as container:
...   assert True

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

Details

# Get the operating system information (on a unix os).
$ uname -a
Linux FE-C-012RG 6.5.0-1019-oem #20-Ubuntu SMP PREEMPT_DYNAMIC Mon Mar 18 17:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info
Client: Docker Engine - Community
 Version:    26.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 19
 Server Version: 26.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.5.0-1019-oem
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.32GiB
 Name: FE-C-012RG
 ID: 547c075a-39fa-4b4a-950a-b18069861839
 Docker Root Dir: /home/CFL5FE/.local/share/docker
 Debug Mode: false
 Username: keneanung
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
# Get all python packages.
$ pip freeze
alembic==1.13.1
annotated-types==0.6.0
anyio==4.3.0
astroid==3.1.0
asyncpg==0.29.0
-e <my local repo>
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
coverage==7.4.4
cryptography==42.0.5
dill==0.3.8
docker==7.0.0
ecdsa==0.19.0
fastapi==0.110.1
fastapi-azure-auth==4.3.1
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
hvac==2.1.0
idna==3.6
importlib_metadata==7.1.0
iniconfig==2.0.0
isort==5.13.2
Mako==1.3.2
MarkupSafe==2.1.5
mccabe==0.7.0
packaging==24.0
platformdirs==4.2.0
pluggy==1.4.0
pyasn1==0.6.0
pycparser==2.22
pydantic==2.6.4
pydantic_core==2.16.3
pylint==3.1.0
pytest==8.1.1
pytest-asyncio==0.23.6
pytest-cov==5.0.0
python-jose==3.3.0
PyYAML==6.0.1
requests==2.31.0
rsa==4.9
six==1.16.0
sniffio==1.3.1
SQLAlchemy==2.0.29
starlette==0.37.2
syrupy==4.6.1
testcontainers==4.3.3
tomli==2.0.1
tomlkit==0.12.4
typing_extensions==4.11.0
urllib3==2.2.1
uvicorn==0.29.0
vault-env-gen==0.2.0
wrapt==1.16.0
yapf==0.40.2
zipp==3.18.1

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection


def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)


if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

and then i guess we will need to tweak the Reaper class a bit in core

can you confirm this approach:

mkdir test-docker-rootless-detection ; cd $_ ; python -m venv .venv && . $_/bin/activate
pip install docker
cat >  detect_rootless.py <<EOF
from docker import from_env
from docker.client import DockerClient
from docker.models.containers import Container, ContainerCollection


def is_rootless(client: DockerClient):
  info = client.info()
  sec_opts = info.get('SecurityOptions') or tuple()

  return any('rootless' in s for s in sec_opts)


if __name__ == "__main__":
  print(is_rootless(from_env()))
EOF

python detect_rootless.py

This returns the following:

$ python detect_rootless.py 
True

I also use rootless Docker and I can confirm this issue.
I can also confirm that setting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock fixes the issue. Alternatively, I have tried disabling Ryuk through TESTCONTAINERS_RYUK_DISABLED=true, which makes the tests run flawlessly as well.

@alexanderankin

Would it be possible to fix this so that testcontainers just works with rootless docker? @alexanderankin

TESTCONTAINERS_RYUK_DISABLED=true pytest works, but it's a pain to use, and we can't even just os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true' inside the python test file

Nowadays any dev slightly conscious should be using rootless docker (because it is safer, and because it reduce friction with file ownerships when mounting volumes inside an image that use the root user. So not only it is safer, but it is also easier to use...). It's a bit the future of containers, but it's already there and working. So testcontainers should be available for rootless docker without the need for complex setup

you can import the config module and set it on the dataclass there

testcontainers-python/core/testcontainers/core/config.py

Line 51 in 85d6078

ryuk_disabled: bool = RYUK_DISABLED

I'm not in a position where I'm forced to use rootless docker so I dont have easy access to an environment where I could easily test. if there is a guide or something that I can set up, this would be good prereq for contributions imo

Ah, we have confirmed that the logic i provided above works. Then all that remains is plugging it in I suppose. I'll accept PR that fixes the issue. yes.

I am not forced to use docker rootless neither, it's just that it's better than the root option, especially when you use containers a lot for development, try it you'll see :) Rootless docker can be easily enabled following these docs: https://docs.docker.com/engine/security/rootless/ usually 3 commands:

dockerd-rootless-setuptool.sh install
systemctl --user enable docker
loginctl enable-linger $UID

Unfortunately I just realized that ryuk is needed to stop the containers after running the tests. So disabling it is not a sustainable solution

But TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/run/user/$(id -u)/docker.sock works fine

An easy solution would be to just change the default RYUK_DOCKER_SOCKET depending on if rootless is detected or not here: https://github.com/testcontainers/testcontainers-python/blob/main/core/testcontainers/core/config.py#L15

I might look into this if I find sometimes and will send a PR

fix(core): Fix retrieval of the docker socket path when using rootles…

import docker

client = docker.from_env()
print(client.api.get_adapter(client.api.base_url).socket_path)

Could be used to detect the correct socket_path.

If socket_path is not defined, just use the default.

fix(core): Determine docker socket for rootless docker

fix(core): Determine docker socket for rootless docker (testcontainer…

Navigation Menu

Repository navigation

Bug: RYUK container startup failure when running Docker in rootless mode #537

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug: RYUK container startup failure when running Docker in rootless mode #537

Description

Activity

alexanderankin commented on Apr 17, 2024

alexanderankin commented on Apr 17, 2024

keneanung commented on Apr 19, 2024

flonix8 commented on Aug 20, 2024

vemonet commented on Oct 1, 2024

alexanderankin commented on Oct 1, 2024

alexanderankin commented on Oct 1, 2024

vemonet commented on Oct 2, 2024

CarliJoy commented on Feb 27, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions