Skip to content

perf(docker): slow steady-state supervisor healthcheck - #4802

Merged
TaylorMutch merged 1 commit into
NVIDIA:mainfrom
lunarwhite:perf/4729-docker-supervisor-health-interval/lunarwhite
Oct 21, 2026
Merged

TaylorMutch merged 1 commit into
NVIDIA:mainfrom
lunarwhite:perf/4729-docker-supervisor-health-interval/lunarwhite

Conversation

@lunarwhite

Copy link
Copy Markdown
Contributor

Summary

Keep the Docker supervisor healthcheck's 250 ms start_interval for fast readiness detection and raise the steady-state interval from 250 ms to 30 s, as proposed in the triage.

Related Issue

Closes #4729

Changes

  • Give the start interval its own constant (250 ms) and set SUPERVISOR_HEALTH_INTERVAL_NS to 30 s. Timeout, retries, and the 60 s start period are unchanged.
  • Move the HealthConfig into docker_supervisor_healthcheck() and pin all three timing values in a unit test.
  • Note: Docker uses the start interval only within the 60 s start period, so a supervisor that becomes ready later (for example after a policy repair) is detected up to 30 s later. Public Ready comes from the supervisor session and is unaffected.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Additionally, one idle sandbox, Docker 29.7.2 on WSL2, gateway built from main vs this branch:

main This PR
Health probes per minute 185 2
Supervisor container CPU 16.3% 0.4%
dockerd CPU 4.7% 0.3%
sandbox start (3 runs) 1.7–2.8 s 1.8–2.7 s
Policy repaired 91 s into startup: to Ready / to Docker healthy 1.4 s / 1.2 s 1.0 s / 29.1 s

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@TaylorMutch TaylorMutch self-assigned this Oct 21, 2026
@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test f7bb874

@TaylorMutch
TaylorMutch added this pull request to the merge queue Oct 21, 2026
Merged via the queue into NVIDIA:main with commit 20cb306 Oct 21, 2026
81 checks passed
@lunarwhite
lunarwhite deleted the perf/4729-docker-supervisor-health-interval/lunarwhite branch October 21, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(driver-docker): 250 ms supervisor healthcheck costs ~12% of a CPU core per idle sandbox

2 participants