Skip to content

fix(cloud-agent-wrapper): stop gated spawns closing reused fds - #7764

Merged
iscekic merged 78 commits into
mainfrom
test/wrapper-flakes
Oct 16, 2026
Merged

iscekic merged 78 commits into
mainfrom
test/wrapper-flakes

Conversation

@iscekic

@iscekic iscekic commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Three wrapper tests failed in about 1 of 10 CI runs:

  • native upload episode projection > keeps archive and diagnostic episodes independent, ... got a second stderr line.
  • adopting a repository snapshot against a real repository > ... never saw session.ready.
  • local worktree recovery > restores additions, ... had git init killed by SIGPIPE.

They share one cause: a file descriptor closed twice. Bun 1.3.14 closes the parent end of an extra stdio pipe (fd 3 or above) two times. The net.Socket that wraps it closes it first. When the subprocess is collected, Bun closes the same number again. By then another descriptor uses that number, for example the stdout pipe of a later tar or git. Bun 1.4.0 fixed this (oven-sh/bun#33828).

spawnGated used an fd 3 pipe as its start gate. setup-cgroup.test.ts runs it. A GC a few files later then closed a pipe in whichever test was running. In every failed CI run I checked, setup-cgroup.test.ts ran 1 to 5 files before the failing file. Bun's file order changes with each checkout, so the failing test changed too.

The fix moves the gate to the child's stdin. Bun closes the stdin pipe once.

  • Owned children (the Kilo server and tools) keep that pipe open as their stdin, as before.
  • Setup commands get stdin closed after start, so they read EOF, as they did with ignore. A setup spawn with any other stdin now throws, because the gate takes stdin.

The production sandbox image runs Bun 1.3.12, which has the same bug. There, too, each gated spawn could later close an unrelated descriptor. This change fixes that as well.

A new test in setup-cgroup.test.ts checks that descriptors opened after a gated setup command stay open through GC. It failed 10 of 10 runs before the fix and passes 10 of 10 after. No existing test or assertion changed.

Verification

All runs were on gcloud VMs (8 vCPU, Bun 1.3.14), with 4 to 8 loops in parallel.

  • Before the fix, the CI command bun test src --path-ignore-patterns='**/control-plane/process.test.ts' failed 5 of 75 runs. Every failure was the upload episode test, 3 files after setup-cgroup.test.ts, with tar spawn error in the log. bun test src failed 1 of 20 runs, the same way. The CI command with --randomize failed 4 of 20 runs. Those failures were the snapshot adoption tests and setup-cgroup.test.ts itself.
  • After the fix, the CI command passed 60 of 60 runs, the CI command with --randomize passed 20 of 20, and bun test src passed 15 of 15.
  • A descriptor probe opened 64 descriptors after 5 gated spawns, then ran GC. With the old gate, GC closed 1 of them in every run, on both the placed path and the failed-placement path. With the stdin gate it closed 0. The result was the same on Bun 1.3.14 and on the sandbox image's Bun 1.3.12.
  • workload-isolation.real.test.ts ran in a private-cgroup Docker container (4 GiB, Kilo 7.8.1). The owned-tool test passed. The two Kilo tests fail with 401 Unauthorized on both the base and this branch, so that failure was already there. The gated Kilo server still started and answered.
  • tsgo --noEmit, oxlint, oxfmt --check, and check:duplication pass.

Visual Changes

N/A

Reviewer Notes

  • Moving to Bun 1.4 would also fix this. That is a much larger change: the sandbox base image (Bun 1.3.12), the CI pin, and @types/bun would all have to move.
  • The parent writes only start\n to the gate. So the gate shell cannot take any stdin data that belongs to the command.

Test, build, and workspace jobs needed typecheck, lint, format-check, and drizzle-check to pass first. That chain put the slowest gate in front of the slowest suite. Each job now needs only the change detection, so all jobs start together. The Chrome e2e job no longer waits for verify.
The web Jest suite now runs in six shards and the mobile Vitest suite in four. A fan-in job keeps the required check name test in each workflow; it fails when any shard fails or is cancelled. The root script tests and the Postgres regressions run in their own test-scripts job.
scripts/affected-workspaces.mjs selects the changed workspaces and every workspace that reads them, through workspace dependencies or relative paths. A change outside every workspace, or to shared config, selects all of them. Main pushes and the merge queue always select all workspaces.

Typecheck runs in web, trpc and packages groups, and lint in trpc and rest groups, as parallel jobs. Fan-in jobs keep the required typecheck and lint check names.
Drizzle check runs only when the db package is affected, and the ai-gateway build only when ai-gateway is affected. The mobile suite also runs when the dependency graph reaches kilo-app. Format check installs only the root package. The mobile tests no longer build the tRPC types, which hold only declarations.
The ai-gateway build copies the web app's .env files, so the detector now treats ai-gateway as a reader of apps/web. A failed or empty graph detection in kilo-app CI runs the mobile suite instead of skipping it.
Only apps/web/.env and apps/web/.env.test are build inputs of ai-gateway, not the whole web workspace.
kilo-app CI typechecks the app whenever the app is affected, and its job name is also typecheck. The root typecheck job no longer repeats that work. The tRPC group keeps the extension, which no other job covers on an unrelated pull request.
The plugin shared-schema and openclaw pin checks read only manifests, so the changes job runs them. The typecheck fan-in job no longer checks out the repository or installs Node.\n\nThe fan-in jobs for test, typecheck and lint report a failure in the changes job as a failure. Before, a failed changes job left those required checks skipped.
A change to typecheck-all.sh or lint-all.sh now runs only that check for every workspace, instead of every job. The workspace test matrix skips apps/mobile, whose sharded suite runs in kilo-app CI. The script and Postgres test job keeps the large runner the old test job used.
apps/mobile no longer enters the matrix, so its harness SDK build step could not run.
Per-shard times were uneven, and the slowest shard set the test check time.
The builder cleanup test drives two concurrent claims against the same rows. It failed once on the default runner.
The i18n-leftover job built the tRPC types and the harness SDK, but its oxlint config is not type-aware. The root lint groups no longer lint apps/mobile, which the kilo-app CI lint job checks with the same command; the tRPC lint group now covers only the extension and skips the harness SDK build. The typecheck tRPC group runs only when the extension is affected.
A change to lint-all.sh runs the lint check for every workspace. The kilo-app CI lint job does not run that script, so the tRPC lint group now lints apps/mobile in that case, with the harness SDK build it needs.
The cloud-agent-next job ran only on its own paths, so a packages/db or worker-utils change skipped the tests of a dependent. On pull requests the job now also runs when the affected-workspace detector selects cloud-agent-next.
A composite action restores packages/trpc/dist from the cache or builds it. The key hashes every repository file in the declaration program (tsgo --listFilesOnly), the build configuration, and pnpm-lock.yaml, so a hit restores what the build would write. The kilo-app CI typecheck and lint jobs and the tRPC typecheck and lint groups use it. A change to the action or the key script checks every workspace.
The changes job ran nine wiring tests and two plugin checks before it detected changes, and every other job waited for it. They now run in a ci-wiring job that starts at the same time. The required test job needs ci-wiring, so a failure still blocks the merge. stacked-ci.test.mjs stays in changes, where its own guard requires it.
The six mobile shards took between about 90 and 140 seconds, and the slowest shard set the test check time.
The web typecheck group writes tsgo build info to a fixed path, and CI caches it per commit with a lockfile-scoped restore key. tsgo rechecks every file whose hash changed, so a restored file only skips unchanged work.
The job ran test:all, which chains the unit, Workers, SDK provenance and wrapper suites, then the Postgres tests. It now runs as three matrix parts: unit, Workers, and the other checks. The same suites run; none is dropped.
The workspace test matrix, drizzle-check and the secure-proxy build install the target workspace, its workspace dependencies and the root package, not the whole monorepo.
A composite action mounts the pnpm store on a sticky disk and points pnpm at it. It replaces the setup-node pnpm cache in the CI, kilo-app CI and extension CI jobs, which downloaded and unpacked the whole store on every job.
The kiloclaw suite runs the tests of its nested controller workspace, which needs tar-stream. The directory filter selects the workspace and every workspace under it.
…i-gateway builds

Required checks match exact names, so test (<workspace>), drizzle-check, harness-sdk and the two ai-gateway builds never blocked a merge. The required test job now needs them, and fails when any of them fails or is cancelled. Skipped jobs still pass.
Bun 1.3.14 closes the parent end of an extra stdio pipe twice: the net.Socket closes it, then the collected subprocess closes the same number again. spawnGated used an fd 3 pipe as its start gate, so after setup-cgroup.test.ts a later GC closed a tar or git pipe in another test. Gate on stdin instead, which Bun closes once.
@kilo-code-bot

kilo-code-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change since the previous review is limited to the spawnInCgroup doc comment in setup-cgroup.ts, which now explicitly documents that only a placement failure falls back to a plain spawn while any non-ignore stdio[0] throws; this resolves the prior advisory finding with no new issues.

Files Reviewed (1 file)
  • services/cloud-agent-next/wrapper/src/control/setup-cgroup.ts

Notes

  • Prior finding (doc comment contradicted the new stdin guard) is resolved: the comment at lines 81-84 now states "a placement failure falls back to a plain spawn" and that any other stdio[0] throws, and the author confirmed the throw is intentional because the gate now owns stdin and a silent unmanaged fallback would drop containment without a signal.
  • The comment-only change matches the behavior at setup-cgroup.ts:95 (if (stdin !== 'ignore') throw new Error('A setup command takes no stdin');) and the try/catch fallback that follows.
Previous Review Summary (commit 72f3be0)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 72f3be0)

Status: 1 Issue Found | Recommendation: Merge (advisory note only)

Executive Summary

The fd-double-close fix (gate on the child's stdin instead of an extra fd 3 pipe) is correct and leak-free; the only finding is a minor contract/behavior note on the new setup-stdin guard.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1

Review Notes

The change replaces the fd 3 start gate with the child's stdin. I verified the key invariants:

  • GATE_SCRIPT now reads the gate line from stdin, matching stdio: 'pipe' / ['pipe', ...] at both call sites; no fd 3 is created, so Bun cannot close a reused descriptor number when collecting the subprocess.
  • releaseGate switching from end('start\n') to write('start\n') is required: for owned children the gate is the child's stdin now, and the previous stdio[0] pipe was likewise never written or ended, so the child still sees an open, empty stdin. Both paths close the pipe exactly once (releaseChildStreams destroys stdin; the setup path calls gate.end() explicitly), so no double close or leak is introduced.
  • The setup try/catch still ends the gate, destroys stdout/stderr, kills the gated child, and falls back to a plain spawn; the placement-failure test at setup-cgroup.test.ts:66 still exercises that path.

No memory leaks found: the removed child.gate reference is fully superseded by child.process.stdin, whose stream is already destroyed in releaseChildStreams, and the new test cleans up its sentinel descriptors in afterEach.

Findings

services/cloud-agent-next/wrapper/src/control/setup-cgroup.ts:94 — the new if (stdin !== 'ignore') throw runs before the try, so it bypasses the documented "any failure falls back to a plain spawn" contract. Not reachable by current callers (setup never passes stdinFd), but it converts a graceful degradation into a rejection if one ever does. Advisory only.

Files Reviewed (4 files)
  • services/cloud-agent-next/wrapper/src/control/gated-spawn.ts
  • services/cloud-agent-next/wrapper/src/control/owned-processes.ts
  • services/cloud-agent-next/wrapper/src/control/setup-cgroup.ts - 1 issue
  • services/cloud-agent-next/wrapper/src/control/setup-cgroup.test.ts

Reviewed by deepseek-v4.1-flash · Input: 35.1K · Output: 2.6K · Cached: 176.4K

Review guidance: REVIEW.md from base branch ci/speed-20-blocking-checks

@iscekic iscekic changed the title test(cloud-agent-wrapper): stop gated spawns closing reused fds fix(cloud-agent-wrapper): stop gated spawns closing reused fds Oct 16, 2026
Base automatically changed from ci/speed-20-blocking-checks to main October 16, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants