Skip to content

docs(helm): list debug dumps without ls and kubectl cp - #760

Open
fjvbn2003 wants to merge 1 commit into
NVIDIA:mainfrom
fjvbn2003:docs-helm-debug-dumps-distroless
Open

fjvbn2003 wants to merge 1 commit into
NVIDIA:mainfrom
fjvbn2003:docs-helm-debug-dumps-distroless

Conversation

@fjvbn2003

Copy link
Copy Markdown

Problem

deployment/README.md ("Accessing Debug Files" and "Example Usage") tells users to inspect debug dumps with kubectl exec ... -- ls -la <dir> and to copy them with kubectl cp. The chart's default image is nvcr.io/nvidia/k8s/dcgm-exporter:4.6.1-4.8.4-distroless (deployment/values.yaml), and the distroless image has neither ls nor tar, so both commands fail.

I ran the commands against 4.5.3-4.8.2-distroless (the image GPU Operator v26.3.3 deploys) on Kubernetes v1.34.10:

$ kubectl exec <pod> -c nvidia-dcgm-exporter -- ls -la /tmp
... exec: "ls": executable file not found in $PATH

$ kubectl cp <ns>/<pod>:/etc/dcgm-exporter/ ./x -c nvidia-dcgm-exporter
... exec: "tar": executable file not found in $PATH

$ for c in cat tar ls find cp; do kubectl exec <pod> -- bash -c "command -v $c >/dev/null && echo $c=yes || echo $c=no"; done
cat=no tar=no ls=no find=no cp=no

$ kubectl exec <pod> -c nvidia-dcgm-exporter -- \
    bash -c 'shopt -s nullglob; printf "%s\n" /etc/dcgm-exporter/*'
/etc/dcgm-exporter/1.x-compatibility-metrics.csv
/etc/dcgm-exporter/dcp-metrics-included.csv
/etc/dcgm-exporter/default-counters.csv

On main, the distroless stage copies only sh, bash, env, lshw, setcap, ldconfig and the entrypoint from the helper image (hack/licenses/stage-container-runtime.sh), on top of nvcr.io/nvidia/distroless/cc. So I expect 4.6.1-4.8.4-distroless to behave the same way, but I have not run that tag.

Change

  • "From within the pod": state that the distroless image has no ls, cat or tar, and list the files with bash -c 'shopt -s nullglob; printf "%s\n" <dir>/*'.
  • "Copy files": explain that kubectl cp needs tar in the container, and point users to the hostPath directory on the node, which is already described in item 1.
  • Use the same listing command in the two "Check if debug files are being created" examples.

Documentation only. No chart templates or values change.

Testing

The listing command and both failures were run on a cluster as shown above. I did not enable debugDump on that cluster, so I listed a directory that the image already contains instead of a real dump directory.

This change was drafted with AI assistance and reviewed by me.

🤖 Generated with Claude Code

The debug dump section tells users to run `ls` through `kubectl exec`
and to copy the dumps with `kubectl cp`. The chart's default image is
the `-distroless` variant, which has no `ls`, `cat`, or `tar`, so both
commands fail:

  exec: "ls": executable file not found in $PATH
  exec: "tar": executable file not found in $PATH

The image does ship bash, so list the files with a glob and printf, and
point users to the hostPath directory on the node for copying.

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant