[iscekic]

Problem

Benchmark lanes kept dying with benchmark_scheduler_wait reason=ambiguous_execution, with no deploy involved.

Root cause, from read-only Axiom and D1 evidence:

1. The Workers runtime cancels in-flight benchmark queue invocations (Outcome canceled, no exception, no log line). None of our error handling runs.
2. The invocation was awaiting the container's /run. A Durable Object request keeps running after its caller is canceled, so the container finished the case and wrote its answer into the Durable Object's still-open connection. It therefore treated the response as delivered and never posted /container/case-result. The canceled /run events end when the container answers, well after their caller was canceled. Outside rollouts, almost no container results are recovered.
3. Queues redelivers the message within about a minute, but startChunk refused an already active claim, so the redelivery was dropped silently.
4. When the 20-minute lease expired, the re-claim found a paid-ledger row with no output and failed the lane. The output had never been posted, so waiting for it alone would only have held the slot longer.

Fix

• Root fix: BenchRunnerContainer relays the caller's cancellation to the container request (withCallerCancellation), and the worker sets the enable_request_signal compatibility flag. When the caller goes away, the container sees its request close, finishes the case, and posts the outcome as fix(auto-routing): survive container rollouts and make scheduler limits admin-configurable #7614 intended. The runtime never passes an incoming request's own signal on to a subrequest, even when you pass it explicitly, so the abort goes through a new AbortController.
• Bounded same-token resume: a redelivery resumes the executing claim it still owns, keeping the running lease.
  • Completed cases are skipped and recoverable output is reused.
  • A ledgered case without output keeps its slot and provider lease. It re-sends its own claim every 30 seconds until the case timeout (180s) plus the container delivery window (90s) has passed since the case's latest paid call started. An empty-output retry restarts that clock in the ledger. Then the lane fails as before.
  • The paid-execution ledger still admits a case once, so a resumed or duplicate delivery never repeats inference.
  • Rechecks mint no user token and warm no container; those now wait for the first case that actually needs a paid call.
• Ledger start time: scheduler_executions.started_at records when the paid call began. The column is additive, generated with pnpm db:generate, and defaults to 0, so existing rows count as past their deadline.
• The README covers the resume, the wait, and the relay.

What enable_request_signal changes in this worker

The flag only makes an incoming request's signal abort when its client disconnects; without it the signal never fires. request_signal_passthrough stays off, so no incoming signal reaches a subrequest implicitly. In this worker:

• BenchRunnerContainer: relays the signal to the container request. This is intended.
• @cloudflare/containers: already hands request.signal to its container-start wait. That wait now stops when the caller is gone, and returns a 500 that nobody reads.
• Hono routes, /container/case-result and outbound fetches: unaffected. No handler reads request.signal, and none forwards an incoming request to fetch.
• Queue, cron and alarm handlers: unaffected; they have no incoming Request.

Tests

• caller-cancellation.workerd.test.ts runs a fixture Durable Object in real workerd, through wrangler's unstable_startWorker, using the service's compatibility date and flags from wrangler.jsonc. A caller starts a subrequest and leaves mid-flight:
  • Relayed through withCallerCancellation, the upstream connection closes.
  • Forwarded directly, as @cloudflare/containers does on its own, it stays open until answered.
• run-redelivery.test.ts runs production migrations on real SQLite with processJob and acceptContainerCaseResult:
  • Output arrives: a paid case's invocation is canceled, and the redelivery keeps the slot and queues a recheck. The container then posts the outcome using the callback it was given. The recheck grades that output, completes the chunk, and hands the slot to the next claim. The container is called once.
  • Nothing posts: the lane is still quarantined one second before the deadline and fails at the deadline (4.5 minutes, not after the 20-minute lease). Then the slot goes to another lane.
• bench-runner-container.test.ts checks that the container class forwards a request whose own signal aborts with the caller's.
• caller-cancellation.test.ts covers a caller that is already gone. run-process-job.test.ts and scheduler.test.ts are updated for the resume, recheck, ledger start-time and retry-restart behavior, and the scheduler smoke now asserts that a resumed delivery cannot begin a ledgered paid case again.

The new workerd, redelivery, and container-class tests fail on current main: the redelivery is a no-op, the relay never closes, and the forwarded signal is the incoming one. They pass here.

Verification

Run on a work VM for services/auto-routing-benchmark: pnpm typecheck, pnpm lint, pnpm exec vitest run (all suites pass), pnpm smoke:scheduler, oxfmt --list-different on the changed files, and pnpm types. worker-configuration.d.ts changes only in its config hash.

Known and out of scope

The cancellations themselves are Cloudflare-side. Most end within seconds after a 5-minute wall-clock boundary, in batches that share an end time, and other workers' queue consumers in the account show the same alignment. No limit was involved (wall and CPU time far below the limits, no exceeded* outcomes), and no container exits, Durable Object resets or D1 errors coincide with them. This PR makes lanes survive those cancellations; it does not prevent them.

Not verified until deploy: the local workerd run shows the relay closing the container request at the moment of cancellation. That production's runtime propagates a canceled queue invocation to the Durable Object the same way is inferred, not observed. After deploy, benchmark_container_result_recovered should follow canceled invocations, and ambiguous_execution should drop.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental commit fixes all three prior findings with tests; no new issues were found in the changed cancellation, ledger-restart, and test code.

Files Reviewed (8 files)

• services/auto-routing-benchmark/README.md
• services/auto-routing-benchmark/src/caller-cancellation.test.ts
• services/auto-routing-benchmark/src/caller-cancellation.ts
• services/auto-routing-benchmark/src/caller-cancellation.workerd.test.ts
• services/auto-routing-benchmark/src/run-process-job.test.ts
• services/auto-routing-benchmark/src/run.ts
• services/auto-routing-benchmark/src/scheduler.test.ts
• services/auto-routing-benchmark/src/scheduler.ts

Previous Review Summary (commit 64d2c70)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 64d2c70)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The core fix is well-covered by tests; the main gap is that the fixed 270s paid-output wait can end before a canceled empty-output retry can post its outcome, causing a lane failure to discard a legitimately paid result.

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	2

Issue Details (click to expand)

WARNING

File	Line	Issue
services/auto-routing-benchmark/src/run.ts	1117	PAID_OUTPUT_WAIT_MS covers one CLI call, but runPaidCase can issue two; a cancel during the empty-output retry can post past the deadline, failing the lane and discarding the paid output.

SUGGESTION

File	Line	Issue
services/auto-routing-benchmark/src/caller-cancellation.ts	10	An already-aborted caller signal is not relayed (AbortSignal does not replay abort to late listeners).
services/auto-routing-benchmark/src/caller-cancellation.workerd.test.ts	21	upstream.listen() runs before the try, so a worker-startup failure leaks the listening socket.

Files Reviewed (21 files)

• services/auto-routing-benchmark/README.md - 0 issues
• services/auto-routing-benchmark/migrations/0022_burly_ben_grimm.sql - 0 issues (add-column with constant default, safe on SQLite/D1)
• services/auto-routing-benchmark/migrations/meta/0022_snapshot.json - generated
• services/auto-routing-benchmark/migrations/meta/_journal.json - generated
• services/auto-routing-benchmark/scripts/scheduler-smoke.ts - 0 issues
• services/auto-routing-benchmark/src/bench-runner-container.test.ts - 0 issues
• services/auto-routing-benchmark/src/bench-runner-container.ts - 0 issues
• services/auto-routing-benchmark/src/caller-cancellation.ts - 1 issue
• services/auto-routing-benchmark/src/caller-cancellation.workerd.test.ts - 1 issue
• services/auto-routing-benchmark/src/cli-runner.ts - 0 issues
• services/auto-routing-benchmark/src/db-schema.ts - 0 issues
• services/auto-routing-benchmark/src/run-process-job.test.ts - 0 issues
• services/auto-routing-benchmark/src/run-redelivery.test.ts - 0 issues
• services/auto-routing-benchmark/src/run.ts - 1 issue
• services/auto-routing-benchmark/src/scheduler.test.ts - 0 issues
• services/auto-routing-benchmark/src/scheduler.ts - 0 issues
• services/auto-routing-benchmark/test/fixtures/caller-cancellation/worker.ts - 0 issues
• services/auto-routing-benchmark/test/fixtures/caller-cancellation/wrangler.jsonc - 0 issues
• services/auto-routing-benchmark/test/stubs/cloudflare-containers.ts - 0 issues
• services/auto-routing-benchmark/worker-configuration.d.ts - generated hash only
• services/auto-routing-benchmark/wrangler.jsonc - 0 issues

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 42.4K · Output: 12K · Cached: 673.7K}

_{Review guidance: REVIEW.md from base branch main}