Skip to content

Kubernetes control-plane restart fails after node IP rotation #2156

Closed
#2289
Closed
Kubernetes control-plane restart fails after node IP rotation#2156
#2289

Description

@mxschmitt
mxschmitt
opened on Aug 26, 2026

Summary

A single-node Kubernetes cluster created by the Apple container K8s plugin
does not restart after its VM receives a different IPv4 address. Address
rotation is expected; the Kubernetes provider does not preserve the kind
bootstrap contract or configure an address-independent in-VM API endpoint.

Environment

  • macOS on Apple Silicon
  • apple/container 1.3.0 (d6de569)
  • kindest/node:v1.35.5
  • One VM with both control-plane and worker roles
  • No workload or custom DNS configuration

Minimal reproduction

container k8s create --name k8s-repro
container exec k8s-repro cat /kind/old-ipv4
container stop k8s-repro
container k8s start --name k8s-repro

On a clean runtime, the first ordinary restart changed
192.168.64.2 -> 192.168.64.3 and failed before the K8s plugin could perform
its readiness checks:

INFO: detected IPv4 address: 192.168.64.3
INFO: detected old IPv4 address: 192.168.64.2
error: unable to read config from "/kind/kubeadm.conf":
open /kind/kubeadm.conf: no such file or directory

Root cause

There are two independent restart requirements:

  1. The plugin runs kubeadm with /etc/kubernetes/kubeadm-config.yaml, while a
    kind node expects its provider to create /kind/kubeadm.conf for restart
    reconciliation. The kind-owned file was never created; kubeadm did not
    consume or delete it.
  2. Without controlPlaneEndpoint, kubeadm persists the VM address in clients
    such as admin.conf and kube-proxy. In an ablation test, adding only the kind
    file let kind repair API-server state, but kube-proxy continued dialing the
    retired address and CoreDNS remained unready.

Expected behavior

The node should recover after intentional address rotation without requiring a
sticky IP. kind should reconcile address-dependent advertise/certificate
state, while persistent clients use an address-independent endpoint.

Validated fix

PR #2158 writes and uses /kind/kubeadm.conf directly and configures
controlPlaneEndpoint: 127.0.0.1:6443 for clients inside the current one-VM
cluster. After rotation, kind rewrites the node advertise address to the new
VM address while in-VM clients retain loopback. Host kubeconfig export remains
unchanged.

The fix is intentionally limited to newly created single-node clusters. It
does not add sticky addressing, DNS, a proxy, broad start-time rewriting, or
migration for clusters created by older releases. A future multi-node topology
will need a stable shared DNS, VIP, or load-balancer endpoint.

Activity

  1. mxschmitt commented on Aug 26, 2026

    @mxschmitt
    mxschmitt
    on Aug 26, 2026
    Author

    Superseded prototype

    The earlier fix/k8s-restart-ip-change prototype and its broad start-time
    address rewriting have been superseded by PR #2158. Clean-room and ablation
    testing showed that the rewriting could not migrate a stock cluster because
    kind fails before the plugin regains control; it was also unnecessary for new
    fixed clusters.

    The final patch is one focused, signed commit. It writes and uses kind's
    /kind/kubeadm.conf directly and gives the current single-node cluster a
    stable in-VM control-plane endpoint. The prototype commits referenced by the
    original version of this comment should not be used for review.

  2. mxschmitt commented on Aug 26, 2026

    @mxschmitt
    mxschmitt
    on Aug 26, 2026
    Author

    Final clean-room and ablation result

    A clean stock 1.3.0 cluster with no workload, address bumper, or custom DNS
    failed on its first ordinary restart (192.168.64.2 -> 192.168.64.3) because
    kind could not read /kind/kubeadm.conf.

    Adding only that file passed kind's first gate, but admin.conf and kube-proxy
    still used the retired address. kube-proxy logged no route to host, CoreDNS
    remained unready, and k8s start could not complete. Both the kind bootstrap
    contract and an address-independent client endpoint are therefore required.

    PR #2158 now implements only those two requirements:

    • write and use /kind/kubeadm.conf directly;
    • use 127.0.0.1:6443 for clients inside the current one-VM cluster.

    The rebased, GitHub-Verified commit 86735d0 passed the focused unit tests and
    the exact integration regression twice. The final run rotated
    192.168.64.5 -> 192.168.64.7, verified the loopback endpoint, verified that
    kind rewrote advertiseAddress to the new VM address, completed k8s start,
    and observed the node Ready.

    The regression uses an intervening Alpine allocation to advance the rotating
    allocator; it does not consume the released node address. Existing clusters
    created without the kind file are not claimed to be migrated.

  3. guanchzhou commented on Sep 4, 2026

    @guanchzhou
    guanchzhou
    on Sep 4, 2026
    Contributor

    Independent reproduction on container 1.3.1 (release, Apple Silicon). Same kube-proxy half of this bug is still present on a cluster that was created days earlier and then stopped/started after the node IP rotated (192.168.64.5 → 192.168.64.3).

    What we observed that 1.3.1 does rewrite

    After container start k8s-dev + container k8s start --name k8s-dev, these on-disk files already pointed at the new address:

    • /etc/kubernetes/kubelet.conf
    • /etc/kubernetes/controller-manager.conf
    • /etc/kubernetes/scheduler.conf

    kubectl from the host also worked once the serving cert was restored (see below). The control plane looked healthy from outside.

    What it misses: the in-cluster kube-proxy ConfigMap

    kubectl get cm -n kube-system kube-proxy -o jsonpath='{.data.kubeconfig\.conf}'
# still: server: https://192.168.64.5:6443

    kube-proxy then loops on:

    dial tcp 192.168.64.5:6443: connect: no route to host

    It never syncs, so there are no Service rules at all. Every pod call to https://10.96.0.1:443 times out. That is easy to misread as an application bug.

    In our case Envoy Gateway certgen logged generated certificates and then:

    failed to get server groups: Get "https://10.96.0.1:443/api": dial tcp 10.96.0.1:443: i/o timeout

    A busybox pod against https://10.96.0.1:443/version timed out the same way. After patching the ConfigMap to https://192.168.64.3:6443 and kubectl rollout restart daemonset/kube-proxy -n kube-system, the same pod retrieved v1.37.0.

    Related: serving cert deleted, not regenerated

    On the same restart, kube-apiserver crash-looped on:

    open /etc/kubernetes/pki/apiserver.crt: no such file or directory

    CA and the rest of /etc/kubernetes/pki survived. The stored kubeadm config still had advertiseAddress: 192.168.64.5 while the live node was .3. The plugin rewrote the apiserver manifest advertise-address to .3 and dropped the stale serving cert, but did not regenerate it. Workaround inside the node:

    sed -i 's/192.168.64.5/192.168.64.3/' /etc/kubernetes/kubeadm-config.yaml
kubeadm init phase certs apiserver --config /etc/kubernetes/kubeadm-config.yaml

    Environment

    • container CLI version 1.3.1 (build: release)
    • existing single-node cluster k8s-dev (kindest/node v1.37.0, 2 CPU / 4 GiB at the time)
    • macOS Apple Silicon
    • no custom DNS

    PR 2158's 127.0.0.1:6443 in-VM client endpoint would have avoided the kube-proxy ConfigMap trap for new clusters. Existing clusters created without /kind/kubeadm.conf still hit this on restart, matching the "not claimed to be migrated" note on that PR.

  4. katiewasnothere
    closed this as completedin #2289on Oct 13, 2026

  5. mxschmitt commented on Oct 14, 2026

    @mxschmitt
    mxschmitt
    on Oct 14, 2026
    Author

    Retested current main (17ea964, including #2289) on macOS/arm64 with a fresh single-node cluster. After ordinary container stop / container start, its IP changed from 192.168.64.2 to .4. The API certificate and local node files were repaired, but kube-system/kube-proxy's kubeconfig.conf still used https://192.168.64.2:6443. kube-proxy logged no route to host, CoreDNS remained unready, and Service routing failed although the node became Ready.

    For new zero-worker clusters, bootstrapping with controlPlaneEndpoint: 127.0.0.1:6443 avoids that persistent VM address. The small Service/DNS rotation regression passed, and latest try-playwright's five selected JavaScript API/UI checks passed before and after rotation once startup settled. This does not migrate existing clusters or solve multi-node discovery.

    I understand #2290 deliberately removed k8s start and currently recommends recreation. I am narrowing #2158 to this remaining single-node endpoint change, dropping the now-superseded kind-entrypoint/config-path changes and leaving the command UX unchanged. Would that scoped repair and its regression fit the intended direction while cluster lifecycle support is being redesigned?

  6. jglogan commented on Oct 29, 2026

    @jglogan
    jglogan
    on Oct 29, 2026
    Contributor

    @mxschmitt we'll be working on the broader problem in #2290. I think what you propose for #2158 is pretty much what kind does as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions