Skip to content

ci(helm): chart pushes fail when GHCR briefly misses a new manifest #4577

Closed

Description

@matthewgrossman

User Story

As a release captain, I want Helm chart publication to survive transient GHCR errors, so that a qualified pre-release or dev build isn't marked failed and doesn't need a manual rerun.

Problem Statement

.github/actions/release-helm-oci pushes each chart with a single helm push. Helm 4 uploads the chart manifest by digest, then fetches it back by digest to apply the version tag (oras-go ExtendedCopy followed by Tag). GHCR sometimes returns 404 for a manifest it has just accepted, and the push fails:

Error: failed to perform "Tag" on destination: sha256:01fe3bb54286974ffbe8420d00699b998bb54c4da8471fc270a3433552e57bde: not found

The step doesn't retry, so the job fails and leaves an untagged chart version in GHCR. Because the push loop stops at the first failure, any later charts aren't published either.

Impact / Why This Matters

  • The v0.1.5-pre.2 Release Tag run (38644199462, attempt 2) passed Release Qualification, then failed at Release Helm Chart (OCI). The run is marked failed, a pre-release failure alert went to Slack, and the 0.1.5-pre.2 charts weren't published.
  • Release Dev failed the same way on Oct 5 (job 112014316972), that time on the workspace chart.
  • A stable release hitting this would need a manual rerun of a publish job.

Upstream reports the same failure against Quay in helm/helm#31348, closed as stale without a fix.

Acceptance Criteria

  • A transient failure of a chart push is retried a bounded number of times with backoff in both the versioned and SHA-pinned push steps.
  • Retries log a warning, so their frequency is visible.
  • A persistent failure still fails the step.

Logs

Release Helm Chart (OCI), job 116150688298: both charts packaged, the duplicate-version check passed, and the first helm push (helm-chart-0.1.5-pre.2.tgz) failed with the error above. GHCR has an untagged helm-chart version created at 2026-10-14T23:00:33Z.

Activity

  1. added a commit that references this issue on Oct 19, 2026
    834b14f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions