Skip to content

ci(helm): retry chart pushes when GHCR briefly misses a new manifest - #4578

Merged
matthewgrossman merged 1 commit into
mainfrom
fix/4577-helm-oci-push-retry/matthewgrossman
Oct 19, 2026
Merged

matthewgrossman merged 1 commit into
mainfrom
fix/4577-helm-oci-push-retry/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Retry Helm chart pushes to GHCR. helm push uploads the manifest by digest and then fetches it back to apply the version tag. GHCR sometimes returns 404 for a manifest it just accepted, so the single push fails. That failed the v0.1.5-pre.2 Release Tag run after Release Qualification passed, and failed Release Dev on Oct 5.

Related Issue

Closes #4577

Changes

  • Both push steps in .github/actions/release-helm-oci/action.yml (versioned and SHA-pinned charts) push through a small push_chart function. It tries up to four times, waiting 10, 20, then 30 seconds between attempts.
  • Each retry logs a ::warning::, so we can see how often GHCR does this.
  • A persistent failure still fails the step after the fourth attempt.
  • A retry is idempotent. The blobs and manifest are already in GHCR, so Helm skips the upload and only applies the tag.

Testing

  • shellcheck passes on both extracted run scripts.
  • Ran the push step against a stub helm: no failures (2 calls, exit 0); two transient failures (4 calls, 2 warnings, exit 0); four consecutive failures (fails after 4 attempts, exit 1).
  • Not exercised against GHCR; the next Release Dev run on main will use it.

Evidence that the failure is transient: rerunning the failed job (attempt 3) succeeded with no changes and tagged both charts 0.1.5-pre.2. That rerun repackaged the charts, so it pushed a new manifest (helm-chart@sha256:b4d88075…) and left attempt 2's manifest (sha256:01fe3bb5…) untagged. The in-step retry here re-pushes the same archive, so GHCR already has the blobs and manifest and only the tag is applied.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

helm push uploads the chart manifest by digest, then fetches it back to
apply the version tag. GHCR sometimes returns 404 for a manifest it just
accepted, failing the push with 'failed to perform "Tag" on
destination: <digest>: not found'. This failed Release Dev on Oct 5 and
the v0.1.5-pre.2 Release Tag run after qualification passed.

Retry each chart push up to four times with a short backoff. A retry
finds the blobs and manifest already present and only applies the tag.

Closes #4577

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@matthewgrossman
matthewgrossman marked this pull request as ready for review October 19, 2026 17:30
@matthewgrossman
matthewgrossman added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 834b14f Oct 19, 2026
83 checks passed
@matthewgrossman
matthewgrossman deleted the fix/4577-helm-oci-push-retry/matthewgrossman branch October 19, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(helm): chart pushes fail when GHCR briefly misses a new manifest

2 participants