Repository navigation
Use node image overlay during scale up #1922
Description
(This isn't strictly an openshift/os change, but more how the node image is used.)
Starting in 4.19, we switched the installer to directly overlay the node image on top of the initial boot to avoid paying for another reboot (see openshift/installer#8742).
We could consider doing this as well for the scale up case where saving a reboot is also extremely valuable. With the bootimage skew work, we can add enforcements that this feature can only be turned on if the bootimage used is within some version, and that also limits CI coverage needed for this.
There would be some major caveats around this that users would have to accept before turning this on. Primarily the kernel mismatch. Again, I think skew limitations allow us to keep this testable, but any operator that has a tighter binding on the node image kernel than our skew check (or any drivers) wouldn't be usable on the first boot.
There's also soft-reboot nowadays of course. It'd be slower than the overlay approach but probably not by much. Worth looking at the timing. But we do still need to keep the overlay approach anyway for the live case (but ideally eventually extend bootc's soft-reboot feature to support this use case).
Probably simplest would be to move all the node pulling and overlaying logic out of the openshift/installer repo and into the base image where it can be more easily reused.
Activity
travier commentedon Apr 3, 2026
It's unlikely that soft-reboot will work as in most cases, after a few weeks, the new image will have a new kernel and soft-reboot refuses to operate in this case. Doing a manual mount of the kernel modules like we do for the live case could be an option but I don't know if bootc folks will accept that and it's risky.
sdodson commentedon Apr 7, 2026
Two things, I hadn't realized that the rebootless pivot kept the kernel from the boot image though I should've. I agree that in the situations where elastic scaling time is a priority we can forego with the caveats/limits you're putting forth. Perhaps a MachineConfig flag that determines whether or not kernel version convergence is required, it could be set on the pool level and perhaps only allowed for a pool that has boot image management enabled?
Have we looked at how we're chunking the images lately? It'd probably be good to take a look at that since the advent of layered builds and perhaps optimize specifically for z-stream updates and just assume that if folks care about scale up time they're leveraging boot image management to ensure that images are matched at least to the RHCOS X.Y release.
jlebon commentedon Apr 7, 2026
Yeah, I've thought about passing the node image through chunkah though the gain there is probably going to be minimal given that the base RHCOS image itself was already well-split.
But to repeat something I've said elsewhere, we should definitely sanity-check/add CI checks verifying that the layer refs embedded in the bootimages are still getting good dedupe with the initial node image pull. We had issues with that initially that basically negated any potential gains there because of layer recompression in the node image pipeline.
openshift-ci commentedon Jul 7, 2026
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting
/lifecycle frozen.If this issue is safe to close now please do so with
/close./lifecycle stale
openshift-ci commentedon Aug 7, 2026
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting
/remove-lifecycle rotten.Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting
/lifecycle frozen.If this issue is safe to close now please do so with
/close./lifecycle rotten
/remove-lifecycle stale
openshift-ci commentedon Sep 6, 2026
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting
/reopen.Mark the issue as fresh by commenting
/remove-lifecycle rotten.Exclude this issue from closing again by commenting
/lifecycle frozen./close not-planned
openshift-ci commentedon Sep 6, 2026
@openshift-ci[bot]: Closing this issue, marking it as "Not Planned".
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.