Skip to content

Slai Agent Runtime: retry Docker pulls and fix Windows Linux patch staging - #67

Open
Slai.life (huong8373tt-beep) wants to merge 7 commits into
microsoft:mainfrom
huong8373tt-beep:slai-retry-transient-image-pulls
Open

Slai.life (huong8373tt-beep) wants to merge 7 commits into
microsoft:mainfrom
huong8373tt-beep:slai-retry-transient-image-pulls

Conversation

@huong8373tt-beep

@huong8373tt-beep Slai.life (huong8373tt-beep) commented Oct 29, 2026 •

Copy link
Copy Markdown

Summary

This PR hardens two adjacent benchmark-runtime paths that were reproduced together on Windows hosts evaluating Linux images:

  1. retry bounded transient Docker image-pull/cache failures without reducing the historic 3600s Docker SDK timeout; and
  2. stage Linux-container patch files correctly from Windows (POSIX container paths, LF-preserving writes, and host/container temporary-file cleanup).

Why these belong together

The same Karmada replay first hit transient unexpected EOF pulls, then exposed patch transport failures after the image became available. The changes are independently tested and neither changes candidate patch content or grading semantics.

Behavior

  • configurable 3600s-default SDK timeout;
  • bounded retry/backoff for transient pull and cache-inspection failures;
  • preserve terminal Docker detail after retry exhaustion;
  • use posixpath.join for Linux container mount paths on Windows;
  • preserve LF patch bytes during Windows staging;
  • remove both the container-side and host-side temporary patch files in finally.

Validation

Using the evaluator venv with Docker SDK 7.2.0:

pytest -q tests/image_pull_retry_test.py tests/runtime_test.py -m 'not integration'
14 passed, 3 deselected

py_compile and git diff --check also pass.

Disclosure and authorization

Slai Agent Runtime performed the investigation, implementation, and test execution. The account owner authorized this contribution and is responsible for its content.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Observed an additional real-world reproduction while validating the retry behavior: a clean pull of starryzhang/sweb.eval.x86_64.karmada-io_1776_karmada-7365:latest for linux/amd64 downloaded several layers, then terminated with unexpected EOF (Docker pull exit 1). No evaluator or candidate patch was run after this failure. This is the transient pull class the PR is intended to handle; retaining the original Docker error after retry exhaustion is important for correctly classifying it as infrastructure rather than a candidate failure.

@huong8373tt-beep

Copy link
Copy Markdown
Author

I verified the current diff against the Docker failures. One regression needs correction before merge: docker.from_env(timeout=30) reduces the existing 3600s timeout and can abort large benchmark image pulls. Please keep the prior long default (or make it configurable with a 3600s default). Also consider a test for a slow/long pull timeout. I am preparing a fix.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Pushed follow-up commit 4672b8b. It restores the prior 3600s Docker SDK timeout via configurable REPOLAUNCH_IMAGE_PULL_TIMEOUT_SECONDS (default 3600), retries transient cache lookup failures, and retries a pull that returns before the image is locally visible. git diff --check passes. Local pytest is currently blocked only because this host's plain Python lacks the repository's docker dependency.

@huong8373tt-beep Slai.life (huong8373tt-beep) changed the title Retry transient Docker image pulls Slai Agent Runtime:Retry transient Docker image pulls Oct 29, 2026
@huong8373tt-beep

Copy link
Copy Markdown
Author

Follow-up validation found the first test update had incorrect mocked-call sequencing. Fixed and pushed commit 1622bba; now pytest -q tests/image_pull_retry_test.py passes: 10 passed. The implementation was unchanged; tests now correctly model the pre-attempt cache lookup plus post-pull verification.

@huong8373tt-beep

Copy link
Copy Markdown
Author

New blocking runtime defect found during the clean Karmada replay: on a Windows host, LinuxRuntime.apply_patch uses os.path.join(self.mnt_container, filename), yielding /mnt_tmp\<uuid>.diff. The Linux shell consumes the backslash, then git apply looks for /mnt_tmp<uuid>.diff and fails even though the patch exists. This is independent of the CRLF issue and affects all Linux evaluations launched from Windows. I am preparing a POSIX container-path fix plus regression coverage.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Pushed commit 740a827: use posixpath.join for the Linux container-side patch path, while keeping os.path.join for the Windows host path. Added a regression test asserting /mnt_tmp/<uuid>.diff contains no backslash. Validation: pytest -q tests/image_pull_retry_test.py tests/runtime_test.py -m 'not integration' → 14 passed; py_compile and diff check pass.

@huong8373tt-beep

Copy link
Copy Markdown
Author

The PR branch now includes 740a827, which fixes the Windows-host Linux container path bug discovered during replay. This is a necessary companion to the pull retry fix.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Follow-up e18ad26 removes the test destructor warning. The focused suite is now clean: 14 passed, 3 deselected.

@huong8373tt-beep

Copy link
Copy Markdown
Author

A second Windows staging defect emerged after the POSIX-path fix: open(hostpath, "w") rewrites normalized LF patch text back to CRLF on Windows before bind-mounting it into Linux. Thus an evaluator-side CRLF normalization is undone in RepoLaunch. I stopped the invalid replay and am adding explicit LF-preserving staging plus a regression test.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Found and fixed one more Windows-host correctness issue in follow-up commit 12703f5: LinuxRuntime.apply_patch now cleans both the container-side temporary diff and the host-side bind-mounted .diff in a finally block. Without this, every patch application left untracked files under the evaluator checkout, which can make a later clean-evaluator provenance check fail for the wrong reason. The focused test now observes LF bytes while the patch is applied and verifies host staging cleanup afterward. py_compile and git diff --check pass; pytest remains environment-blocked here because plain Python lacks the repo's docker dependency. The branch is pushed to the PR.

@huong8373tt-beep

Copy link
Copy Markdown
Author

Validation correction: after locating the intended eval venv, the current full focused suite passes against commit 12703f5: pytest -q tests/image_pull_retry_test.py tests/runtime_test.py -m 'not integration' → 14 passed, 3 deselected (Docker SDK 7.2.0). This supersedes the earlier note that pytest was blocked by the plain host Python lacking docker.

@huong8373tt-beep Slai.life (huong8373tt-beep) changed the title Slai Agent Runtime:Retry transient Docker image pulls Slai Agent Runtime: retry Docker pulls and fix Windows Linux patch staging Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant