Skip to content

fix(cli): upload release assets one at a time with retries - #14868

Merged
kirillk merged 1 commit into
mainfrom
fix/release-asset-upload-retry
Oct 6, 2026
Merged

kirillk merged 1 commit into
mainfrom
fix/release-asset-upload-retry

Conversation

@kirillk

@kirillk kirillk commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Issue

No issue filed — this was found by triaging the failed v7.8.6 publish dispatch directly (run 37520899871).

What Problem This Solves

CLI and VS Code releases are still blocked. #14860 fixed the TUI worker hang that failed the previous four dispatches, but the v7.8.6 dispatch immediately after it failed somewhere else: build-cli died after the full ~15-minute build, before it could even upload its artifact, so the validate-cli-* gate #14860 was about never ran.

HTTP 422: Validation Failed (https://uploads.github.com/repos/Kilo-Org/kilocode/releases/405082514/assets?label=&name=kilo-linux-x64-musl.tar.gz)
ReleaseAsset.name already exists
ShellError: Failed with exit code 1

The draft release was left holding 8 of 26 assets.

This is not a stale asset from one of the earlier failed dispatches. Those left their own drafts on different tags (v7.8.4 and v7.8.5, 26 assets each), which share no asset namespace with v7.8.6. Every asset involved here was created by this run:

19:41:44  run 37520899871 starts
19:42:38  draft release v7.8.6 (id 405082514) created by this run's `version` job
19:42:51  build-cli starts
19:45:47  5 assets created at once          <- gh's 5 concurrent upload workers
19:46:02  kilo-linux-x64-musl.tar.gz finishes uploading (updated_at)
19:46:20  HTTP 422 "ReleaseAsset.name already exists" for that same file
19:46:23  build-cli fails

The 18s between the asset completing and the 422 is the mechanism in the open: the asset row is created when the POST opens and flips to uploaded when the body lands, so attempt 1 committed at 19:46:02, lost its response, and gh re-streamed the whole body via req.GetBody before GitHub rejected the name it had just taken. The duplicate it died on was one it created 33 seconds earlier. The telling detail is that kilo-linux-x64-musl.tar.gz — the asset the error names as a duplicate — is the first asset listed on that release. It had already uploaded successfully in the same command.

Why This Change Was Made

build.ts published the whole set with a single command:

await $`gh release upload v${Script.version} ${[...archives, ...evidence.files]} --clobber`

gh/pkg/cmd/release/shared/upload.go runs that through one errgroup with 5 workers, and retries an individual upload whenever shouldRetry matches — which includes errNetwork, i.e. any transport failure:

g, gctx := errgroup.WithContext(ctx)
g.SetLimit(numWorkers)
...
return backoff.Retry(func() error {
    _, err := uploadAsset(ctx, httpClient, uploadURL, a)
    if err == nil || shouldRetry(err) { return err }
    return backoff.Permanent(err)
}, backoff.WithContext(backoff.WithMaxRetries(bo, 3), ctx))

A dropped connection is not a rejection. With ~1 GB of CLI archives going out 5 at a time, a transfer commits server-side and the response is lost in transit; gh retries and collides with the asset it just created. 422 is not retryable, so it becomes backoff.Permanent, the command fails, and gctx cancels every upload still in flight. --clobber cannot help, because it only deletes assets that existed when the command started — never a ghost created by the same invocation.

So one hiccup on one of 26 assets discarded the other 18 and failed the release. Retrying the whole command is not a fix either: it costs another full build, and it only has to get unlucky once across ~26 large transfers.

The fix is to stop making the assets share a fate.

Changes:

  • script/kilocode/release.ts uploads one asset per gh invocation, so a failure is scoped to its own file. Each attempt is independently retryable (4 attempts, 5s apart), and --clobber deletes the ghost or half-written asset left by the previous attempt, so "already exists" resolves itself instead of being fatal.
  • A file that exhausts its attempts is reported at the end alongside every other failure, rather than cancelling assets that would have succeeded.
  • Concurrency drops from gh's 5 to 3. Saturating the runner's uplink is what produces the dropped connections in the first place, so the parallelism that made this fast is also what made it fail.
  • After uploading, the published set is confirmed with gh release view --json assets. An upload that exits 0 but leaves an asset in starter state now fails the job instead of silently shipping an incomplete release. A listing that cannot be read is logged and skipped, so confirmation adds no new failure mode of its own.
  • Call sites: packages/opencode/script/build.ts (the 26-asset upload that failed), packages/kilo-vscode/script/publish.ts (VSIX + sidecars, same shape), and packages/opencode/script/publish.ts (distribution evidence — kept non-fatal, since it records a release that already shipped and must not fail an npm publish that cannot be undone).
  • publish.yml's Re-upload signed Windows release zips step has the identical failure mode on a Windows runner that never checks out the repo, so it gets the same per-file retry loop in pwsh instead of importing the helper.

The helper lives in script/kilocode/, so no shared upstream file gains any logic; build.ts and publish.ts only swap one call each behind their existing kilocode_change markers.

User Impact

Releases stop being blocked by a single flaky transfer. Users get the v7.8.6+ CLI and VS Code builds that have been stuck since v7.8.3 (2026-10-01).

Nothing about the shipped artifacts changes — same archives, same SBOM sidecars, same asset names. What changes is that a transient upload failure now costs one retry of one file instead of a 15-minute build and a half-populated draft release, and that a release which ends up with missing or incomplete assets fails loudly rather than proceeding to publish.

Screenshots / Video

N/A - no visual changes. This is release tooling only.

Evidence

All commands below were run by the agent in this worktree.

  • bun test ./script/kilocode/release.test.ts: 6 pass, 0 fail. Covers both handshake-free paths that matter — recovery from the real 422, a permanent failure not cancelling siblings, incomplete/absent assets, the concurrency bound, an unreadable listing, and an empty set never shelling out.
  • The regression test was verified to fail without the fix, not just to pass with it: forcing ATTEMPTS = 1 makes a dropped connection that still created the asset recovers on the next attempt fail, which is run 37520899871 reproduced. The fake commits the asset and then reports the exact 422 payload from that run, because that ordering is the whole bug.
  • Verified the real gh seam rather than assuming its shape, against a throwaway draft release (zz-upload-probe-14868, deleted afterwards; drafts create no git tag, and I confirmed no zz-* release or remote tag remains):
    • the real pusher/lister commands resolved the draft by tag and published all 3 probe assets;
    • re-running the whole set when every name already exists succeeded — this is the exact state that killed the real run, and --clobber per invocation absorbs it;
    • gh release view --json assets returned state: "uploaded" for each, which is what the confirmation step reads;
    • a name that never lands was caught by confirmation and reported as not published (state=absent).
  • GH_REPO=Kilo-Org/kilocode gh release view v7.8.6 --json assets against the live failed draft: returns the 8 partial assets, all state: "uploaded" — confirming the asset named in the error had genuinely landed before gh retried it.
  • bun run test:script:ci: 177 pass, 0 fail (the suite that schedules script/kilocode/*.test.ts in CI).
  • bun run typecheck in packages/opencode and packages/kilo-vscode: clean.
  • bun run knip in packages/kilo-vscode: clean.
  • bun run script/check-opencode-annotations.ts (default and --worktree), check-workflows.ts, check-test-ci.ts, check-md-table-padding.ts, check-kilocode-duplication.ts: all pass.
  • bunx prettier --check on all touched files: clean. bunx oxlint on touched files: 0 errors.
  • CI on this PR: green, including unit tests, all unit (linux/macos/windows) shards, typecheck, Check kilocode_change annotations, and Kilo Code Review.

How to Test

Reviewer test steps

  1. bun test ./script/kilocode/release.test.ts — expect 6 passing.
  2. To see the bug the first test pins, set ATTEMPTS = 1 in script/kilocode/release.ts and re-run: a dropped connection that still created the asset recovers on the next attempt fails with the v7.8.6 error text.
  3. The real proof is the next publish dispatch. The per-file log line (upload: kilo-linux-x64.tar.gz) and the closing upload: N asset(s) published to vX.Y.Z make the step's progress and completeness readable in the job log, which the previous single-command form did not.

Blocked checks and substitute verification

  • The publish workflow cannot be exercised from a PR; it is workflow_dispatch-only and goes on to publish. Substitute verification was the throwaway draft release probe described in Evidence, which exercises the real gh commands, real draft-by-tag resolution, and the real already-exists condition — everything except the 1 GB transfer volume that triggers the flakiness.
  • The pwsh loop in publish.yml runs only on a Windows release runner and was not executed. It is a literal transcription of the same retry shape and is not on the path that failed; the Bun helper carries the behaviour this PR is actually fixing.

Dependency

Depends on #14860 (merged) — that fixed the TUI worker hang which blocks the validate-cli-* gate later in the same workflow. This PR fixes the failure that now happens before that gate; both are needed for a dispatch to get through.

No changeset: release tooling with no user-facing behaviour change of its own.

Cleanup, not blocking: the failed dispatches left draft releases v7.8.4 and v7.8.5 (26 assets each) and v7.8.6 (8 partial assets). They are separate tags so they will not collide with the next bump, but they should be deleted.

Checklist

  • Issue linked above, or exception explained
  • Tests/verification described
  • Screenshots/video included for visual changes, or marked N/A
  • Changeset considered for user-facing changes
  • I personally reviewed the diff and can explain the changes, including any AI-assisted work.

The v7.8.6 publish dispatch failed in build-cli after the full build with
"HTTP 422: ReleaseAsset.name already exists" on kilo-linux-x64-musl.tar.gz,
leaving 8 of 26 assets on the draft release.

gh release upload --clobber uploads the whole set concurrently through one
errgroup and retries an individual upload when the transport fails. A dropped
connection does not mean GitHub rejected the asset: the upload commits
server-side and gh's own retry then collides with the asset it just created.
422 is not retryable, so gh fails the command and the shared errgroup context
cancels every upload still in flight.

Upload one asset per invocation instead. Each attempt is then independently
retryable, and --clobber clears the ghost asset from the previous attempt, so
"already exists" resolves itself. A file that exhausts its attempts is reported
alongside every other failure rather than discarding assets that succeeded, and
the published asset set is confirmed afterwards so an upload left in `starter`
state is not mistaken for a complete release.
@kilo-code-bot

kilo-code-bot Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Reviewed the new per-asset upload helper and its two gh boundaries, the four call-site conversions, the pwsh retry loop, and the test suite. No bugs, security, performance, or fork-hygiene issues found on changed lines.

Notes (not blocking):

  • release.ts retry/concurrency/confirmation logic is correct: attempts are per-file, --clobber re-clears the ghost asset before each retry, workers pull from a shared queue without double-shifting, and the failures set prevents double-reporting a name in the confirmation step.
  • The post-upload gh release view confirmation is deliberately advisory-on-unreadable-listing (logged and skipped), so it adds no new failure mode beyond the intended starter/absent check.
  • New files live in script/kilocode/, and the edits to shared packages/opencode/script/*.ts stay inside existing kilocode_change blocks, so upstream merge surface is unchanged.
  • publish-jetbrains*.yml still call gh release upload ... --clobber for a much smaller set; out of scope here.
Files Reviewed (6 files)
  • .github/workflows/publish.yml
  • packages/kilo-vscode/script/publish.ts
  • packages/opencode/script/build.ts
  • packages/opencode/script/publish.ts
  • script/kilocode/release.test.ts
  • script/kilocode/release.ts

Reviewed by deepseek-v4.1-flash · Input: 49.1K · Output: 24K · Cached: 469.6K

Review guidance: REVIEW.md from base branch main

@kirillk
kirillk merged commit b3b6824 into main Oct 6, 2026
34 checks passed
@kirillk
kirillk deleted the fix/release-asset-upload-retry branch October 6, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants