Repository navigation
fix(cli): upload release assets one at a time with retries - #14868
Merged
Merged
Conversation
The v7.8.6 publish dispatch failed in build-cli after the full build with "HTTP 422: ReleaseAsset.name already exists" on kilo-linux-x64-musl.tar.gz, leaving 8 of 26 assets on the draft release. gh release upload --clobber uploads the whole set concurrently through one errgroup and retries an individual upload when the transport fails. A dropped connection does not mean GitHub rejected the asset: the upload commits server-side and gh's own retry then collides with the asset it just created. 422 is not retryable, so gh fails the command and the shared errgroup context cancels every upload still in flight. Upload one asset per invocation instead. Each attempt is then independently retryable, and --clobber clears the ghost asset from the previous attempt, so "already exists" resolves itself. A file that exhausts its attempts is reported alongside every other failure rather than discarding assets that succeeded, and the published asset set is confirmed afterwards so an upload left in `starter` state is not mistaken for a complete release.
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Reviewed the new per-asset upload helper and its two Notes (not blocking):
Files Reviewed (6 files)
Reviewed by deepseek-v4.1-flash · Input: 49.1K · Output: 24K · Cached: 469.6K Review guidance: REVIEW.md from base branch |
lambertjosh
approved these changes
Oct 6, 2026
This was referenced Oct 7, 2026
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
No issue filed — this was found by triaging the failed v7.8.6 publish dispatch directly (run 37520899871).
What Problem This Solves
CLI and VS Code releases are still blocked. #14860 fixed the TUI worker hang that failed the previous four dispatches, but the v7.8.6 dispatch immediately after it failed somewhere else:
build-clidied after the full ~15-minute build, before it could even upload its artifact, so thevalidate-cli-*gate #14860 was about never ran.The draft release was left holding 8 of 26 assets.
This is not a stale asset from one of the earlier failed dispatches. Those left their own drafts on different tags (
v7.8.4andv7.8.5, 26 assets each), which share no asset namespace withv7.8.6. Every asset involved here was created by this run:The 18s between the asset completing and the 422 is the mechanism in the open: the asset row is created when the POST opens and flips to
uploadedwhen the body lands, so attempt 1 committed at19:46:02, lost its response, and gh re-streamed the whole body viareq.GetBodybefore GitHub rejected the name it had just taken. The duplicate it died on was one it created 33 seconds earlier. The telling detail is thatkilo-linux-x64-musl.tar.gz— the asset the error names as a duplicate — is the first asset listed on that release. It had already uploaded successfully in the same command.Why This Change Was Made
build.tspublished the whole set with a single command:gh/pkg/cmd/release/shared/upload.goruns that through oneerrgroupwith 5 workers, and retries an individual upload whenevershouldRetrymatches — which includeserrNetwork, i.e. any transport failure:A dropped connection is not a rejection. With ~1 GB of CLI archives going out 5 at a time, a transfer commits server-side and the response is lost in transit; gh retries and collides with the asset it just created. 422 is not retryable, so it becomes
backoff.Permanent, the command fails, andgctxcancels every upload still in flight.--clobbercannot help, because it only deletes assets that existed when the command started — never a ghost created by the same invocation.So one hiccup on one of 26 assets discarded the other 18 and failed the release. Retrying the whole command is not a fix either: it costs another full build, and it only has to get unlucky once across ~26 large transfers.
The fix is to stop making the assets share a fate.
Changes:
script/kilocode/release.tsuploads one asset perghinvocation, so a failure is scoped to its own file. Each attempt is independently retryable (4 attempts, 5s apart), and--clobberdeletes the ghost or half-written asset left by the previous attempt, so "already exists" resolves itself instead of being fatal.gh release view --json assets. An upload that exits 0 but leaves an asset instarterstate now fails the job instead of silently shipping an incomplete release. A listing that cannot be read is logged and skipped, so confirmation adds no new failure mode of its own.packages/opencode/script/build.ts(the 26-asset upload that failed),packages/kilo-vscode/script/publish.ts(VSIX + sidecars, same shape), andpackages/opencode/script/publish.ts(distribution evidence — kept non-fatal, since it records a release that already shipped and must not fail an npm publish that cannot be undone).publish.yml'sRe-upload signed Windows release zipsstep has the identical failure mode on a Windows runner that never checks out the repo, so it gets the same per-file retry loop in pwsh instead of importing the helper.The helper lives in
script/kilocode/, so no shared upstream file gains any logic;build.tsandpublish.tsonly swap one call each behind their existingkilocode_changemarkers.User Impact
Releases stop being blocked by a single flaky transfer. Users get the v7.8.6+ CLI and VS Code builds that have been stuck since v7.8.3 (2026-10-01).
Nothing about the shipped artifacts changes — same archives, same SBOM sidecars, same asset names. What changes is that a transient upload failure now costs one retry of one file instead of a 15-minute build and a half-populated draft release, and that a release which ends up with missing or incomplete assets fails loudly rather than proceeding to publish.
Screenshots / Video
N/A - no visual changes. This is release tooling only.
Evidence
All commands below were run by the agent in this worktree.
bun test ./script/kilocode/release.test.ts: 6 pass, 0 fail. Covers both handshake-free paths that matter — recovery from the real 422, a permanent failure not cancelling siblings, incomplete/absent assets, the concurrency bound, an unreadable listing, and an empty set never shelling out.ATTEMPTS = 1makesa dropped connection that still created the asset recovers on the next attemptfail, which is run 37520899871 reproduced. The fake commits the asset and then reports the exact 422 payload from that run, because that ordering is the whole bug.ghseam rather than assuming its shape, against a throwaway draft release (zz-upload-probe-14868, deleted afterwards; drafts create no git tag, and I confirmed nozz-*release or remote tag remains):pusher/listercommands resolved the draft by tag and published all 3 probe assets;--clobberper invocation absorbs it;gh release view --json assetsreturnedstate: "uploaded"for each, which is what the confirmation step reads;not published (state=absent).GH_REPO=Kilo-Org/kilocode gh release view v7.8.6 --json assetsagainst the live failed draft: returns the 8 partial assets, allstate: "uploaded"— confirming the asset named in the error had genuinely landed before gh retried it.bun run test:script:ci: 177 pass, 0 fail (the suite that schedulesscript/kilocode/*.test.tsin CI).bun run typecheckinpackages/opencodeandpackages/kilo-vscode: clean.bun run knipinpackages/kilo-vscode: clean.bun run script/check-opencode-annotations.ts(default and--worktree),check-workflows.ts,check-test-ci.ts,check-md-table-padding.ts,check-kilocode-duplication.ts: all pass.bunx prettier --checkon all touched files: clean.bunx oxlinton touched files: 0 errors.unit tests, allunit (linux/macos/windows)shards,typecheck,Check kilocode_change annotations, andKilo Code Review.How to Test
Reviewer test steps
bun test ./script/kilocode/release.test.ts— expect 6 passing.ATTEMPTS = 1inscript/kilocode/release.tsand re-run:a dropped connection that still created the asset recovers on the next attemptfails with the v7.8.6 error text.publishdispatch. The per-file log line (upload: kilo-linux-x64.tar.gz) and the closingupload: N asset(s) published to vX.Y.Zmake the step's progress and completeness readable in the job log, which the previous single-command form did not.Blocked checks and substitute verification
publishworkflow cannot be exercised from a PR; it isworkflow_dispatch-only and goes on to publish. Substitute verification was the throwaway draft release probe described in Evidence, which exercises the realghcommands, real draft-by-tag resolution, and the real already-exists condition — everything except the 1 GB transfer volume that triggers the flakiness.publish.ymlruns only on a Windows release runner and was not executed. It is a literal transcription of the same retry shape and is not on the path that failed; the Bun helper carries the behaviour this PR is actually fixing.Dependency
Depends on #14860 (merged) — that fixed the TUI worker hang which blocks the
validate-cli-*gate later in the same workflow. This PR fixes the failure that now happens before that gate; both are needed for a dispatch to get through.No changeset: release tooling with no user-facing behaviour change of its own.
Cleanup, not blocking: the failed dispatches left draft releases
v7.8.4andv7.8.5(26 assets each) andv7.8.6(8 partial assets). They are separate tags so they will not collide with the next bump, but they should be deleted.Checklist