Skip to content

(janitor/flaky-test) allow graceful deployment interruption - #8905

Open
kilo-code-bot[bot] wants to merge 1 commit into
mainfrom
kilo/janitor-flaky-deploy-interruption
Open

kilo-code-bot[bot] wants to merge 1 commit into
mainfrom
kilo/janitor-flaky-deploy-interruption

Conversation

@kilo-code-bot

@kilo-code-bot kilo-code-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Evidence

  • CI run 40050039408, merge-group SHA 10a3f4721dab: handles interruption by aborting deployment and releasing its lease failed at scripts/deploy.test.mjs:565 with 0 !== 1 because deploy_aborted was never recorded.
  • CI run 40059919755, unrelated merge-group SHA 113c2a348097: the same named test failed with the identical signature while the other 26 deployment tests passed.
  • The test and fixture were unchanged across those SHAs. The same test later passed with all 27 deployment tests green in CI run 40133290500 and CI run 40135659358, without a relevant test or production change. This is 2 observed failures with one recurring signature, followed by repeated green runs rather than a consistent regression.

Root cause and fix

The deploy runner sends SIGTERM to Wrangler and escalates to SIGKILL after 250 ms. Under CI scheduling pressure, the fake Wrangler process could be killed before its Node SIGTERM handler recorded deploy_aborted. The parent deploy still exited as interrupted, making the outcome timing-dependent.

The graceful-shutdown window is now 1 second. Wrangler's close event still cancels the forced-kill timer immediately, so normal cleanup does not wait for the full window. This preserves real graceful-abort behavior instead of modifying the test, adding a retry, or weakening its assertions. The test remains event-synchronized on deploy_started and still requires both deploy_aborted and lease release.

Validation

  • Unmodified baseline: focused test passed 32/32 runs at 8-way concurrency; total runtimes increased from ~0.5 s to 2–5 s under load, demonstrating the scheduling pressure behind the 250 ms race.
  • After the fix: focused test passed 64/64 runs at 8-way concurrency.
  • pnpm --filter auto-routing-benchmark test: 53 files / 909 Vitest tests passed, plus all 27 deployment tests.
  • pnpm --filter auto-routing-benchmark typecheck: passed.
  • pnpm --filter auto-routing-benchmark lint: passed.
  • Formatting and git diff --check: passed.

@kilo-code-bot kilo-code-bot Bot added the janitor Automated dead-code/duplication cleanup label Oct 26, 2026
@kilo-code-bot
kilo-code-bot Bot enabled auto-merge October 26, 2026 09:24
@kilo-code-bot

kilo-code-bot Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Single-line change extending the graceful SIGTERM-to-SIGKILL escalation window in runWrangler from 250 ms to 1000 ms; the timer is still cleared on child close, so normal deploys and existing timeouts are unaffected, and the abort paths only incur a bounded extra wait after failure is already determined.

Assumptions: [INFERENCE] The 1 s grace window was validated by the author's local stress runs (64/64) rather than by CI; no repository test asserts the escalation duration itself, so no test needs updating.

Files Reviewed (1 file)
  • services/auto-routing-benchmark/scripts/deploy.mjs

Reviewed by deepseek-flash-latest · Input: 29.4K · Output: 2.3K · Cached: 235.9K

Review guidance: REVIEW.md from base branch main

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

janitor Automated dead-code/duplication cleanup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants