[xavpaice]

fix(kubernetes): alias canonical pause image for SecureBuild packages (sc-140326)

Shortcut: https://app.shortcut.com/replicated/story/140326

Summary

Airgapped Kubernetes upgrades that step onto 1.36.5 or 1.37.1 fail deterministically on every OS. kubeadm's upgrade plan runs its upgrade-health-check Job with the canonical pause image registry.k8s.io/pause:3.10.2, but since the SecureBuild switch (#6167) the package loads pause as docker.io/kurlsh/pause:3.10.2 and nothing aliases it back — so in airgap the Job cannot pull, never completes, and the CreateJob preflight is fatal.

Observed in staging release run STAGING-release-v2026.10.18-1-cd94753d5-20261026211729: "Upgrade from k8s 1.31 to latest - Airgap" failed 7/7 supported legs with kurl_upgrade at ErrImagePull: registry.k8s.io/pause:3.10.2. 1.36.4 was unaffected (its Manifest shipped pause under the canonical name directly). Online upgrades and fresh installs are unaffected.

Changes

• scripts/common/kubernetes.sh — kubernetes_load_images now calls kubernetes_alias_canonical_pause_image, which reads the package Manifest's image pause line and tags it (ctr images tag --force, or docker tag on docker-era clusters) to the canonical registry.k8s.io/pause:<tag> when it ships under a non-canonical repository. No-op for already-canonical references (pre-SecureBuild packages), digest suffixes are stripped, and untaggable references get a logWarn rather than a failure.
• scripts/common/kubernetes-test.sh — new tests for the alias (kurlsh pause → canonical tag, digest-pinned → tag stripped, canonical pause → no-op). The "multi" kubeadm-image-overrides fixture's etcd line is also corrected to match the shipped 1.36.5 file (docker.io/kurlsh/etcd:v3.6.15).

The alias is derived from the Manifest at load time rather than added to kubeadm-image-overrides: pause is not a kubeadm-managed static pod, and a pause-only overrides file would make kubernetes_configure_kubeadm_images create an empty patch dir and kustomization stanza on every init/join for future versions that have no securebuild-<version> dir.

Known follow-up (out of scope, filed as sc-140366): in multi-node airgap clusters the upgrade-health-check Job can schedule on a node that has not loaded the target package yet, so CreateJob can still fail there — that gap predates SecureBuild (any pause tag bump) and is untouched by this PR.

UAT

1. make test-shell (or CI build-test): kubernetes-test.sh 31/31, script-test.sh 40/40, add-minor-test.sh 14/14.
2. Re-queue the staging "Upgrade from k8s 1.31 to latest - Airgap" spec against a build from this branch and confirm the upgrade-health-check-* Job completes on the 1.36.5 (and 1.37.1) steps; instance logs should show no ErrImagePull for registry.k8s.io/pause:3.10.2.
3. Optionally on an airgap VM: after loading the 1.36.5 package, sudo ctr -n k8s.io images ls | grep pause shows both docker.io/kurlsh/pause:3.10.2 and registry.k8s.io/pause:3.10.2.

[doomer-ai]

Doomer review
Review completed.

_{Automatic approval: Enabled}

[doomer-ai]

This pull request is closed. No approval action remains.

Open the review and findings

Reviewed commit: b384a90

[banjoh]

This alias only exists on the node that runs kubernetes_load_images, which in upgrade_kubernetes_local_master is the local primary (scripts/common/upgrade.sh:426). The upgrade-health-check Job has no nodeSelector or affinity, only a control-plane toleration (cmd/kubeadm/app/phases/upgrade/health.go in v1.37.1), so in a multi-node airgap cluster its pod can land on a worker or another primary that hasn't loaded the target package yet. That node has no registry.k8s.io/pause:3.10.2, and kubeadm upgrade plan still fails CreateJob.

This gap predates SecureBuild (any pause tag bump hits it), so it could be a follow-up. The simplest fix I can see is to add CreateJob to the ignored preflight errors for plan when AIRGAP=1. kubeadm upgrade apply already runs with --ignore-preflight-errors=all for the same reason.

local ignore_preflight="$KUBERNETES_UPGRADE_IGNORE_PREFLIGHT_ERRORS"
if [ "$AIRGAP" = "1" ]; then
    ignore_preflight="${ignore_preflight:+$ignore_preflight,}CreateJob"
fi
( set -x; kubeadm upgrade plan "v${targetK8sVersion}" --ignore-preflight-errors="$ignore_preflight" )

[xavpaice]

Good catch, and thanks for checking the Job's scheduling semantics. You're right — on multi-node airgap the health-check pod can land on a node that has not loaded the target package, alias or no alias. Since that gap predates SecureBuild (any pause tag bump) and this PR is scoped to the deterministic single-node regression blocking the staging release, I've filed it as a follow-up with your --ignore-preflight-errors=CreateJob-in-airgap sketch (consistent with apply's existing --ignore-preflight-errors=all): https://app.shortcut.com/replicated/story/140366