Repository navigation
Turning on composefs breaks pivot from old bootimages #1678
Description
Updated description:
Any bootimage with a libostree older than v2023.4 will be too old to know how to create composefs images when upgrading. And so once we upgrade, if composefs = yes in prepare-root.conf, ostree-prepare-root in the new initramfs will fail because it sees that composefs should be enabled and yet no composefs image exists. We currently set composefs = maybe in prepare-root.conf because of this: https://github.com/coreos/rhel-coreos-config/blob/2c3f8527babb935d8d71b505bc764dbab8163b69/manifest-el9-shared.yaml#L28-L34
This ticket tracks being able to switch it back to yes.
Original description follows.
Attached the full secondboot journal.
journal.txt
The specific error is:
Dec 09 17:13:23 localhost systemd[1]: Mounted /sysroot.
Dec 09 17:13:23 localhost systemd[1]: Starting OSTree Prepare OS/...
Dec 09 17:13:23 localhost ostree-prepare-root[675]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/8bd789c8d973d9910e8cc1fc477dd20ac5e975e0663085eeb06a95bfcdba5199.0
Dec 09 17:13:23 localhost ostree-prepare-root[675]: sysroot.readonly configuration value: 1 (fs writable: 1)
Dec 09 17:13:23 localhost ostree-prepare-root[675]: ostree-prepare-root: composefs: failed to mount: No such file or directory
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Dec 09 17:13:23 localhost systemd[1]: Failed to start OSTree Prepare OS/.
Activity
jlebon commentedon Dec 10, 2024
This doesn't look like the first boot of the machine. Which AMI is this BTW? Is this from the CoreOS pipeline, or somewhere else? Let's try to reproduce it outside of OpenShift (presumably by booting the AMI, doing an
rpm-ostree rebaseorbootc switchand rebooting?).Prashanth684 commentedon Dec 10, 2024
Yes this is not the first boot. This log is from the e2e test that is run on OKD release nightlies, where the Openshift installer boots the machine with fcos AMis and then on second boot, pivots to scos.
Prashanth684 commentedon Dec 11, 2024
When i tried booting with the AMI that the Openshift installer uses -
39.20231101.3.0, I see this issue. But when booting with a newer AMI -40.20240602.3.0, I don't see the issue anymore and the rebase works. openshift/installer#8640 updates the AMI for fcos and also tests the okd scos e2e.[-]ostree-prepare-root: composefs fails to boot latest scos image[/-][+]Turning on composefs breaks pivot from old bootimages[/+]jlebon commentedon Dec 16, 2024
So the issue here is that any bootimage with a libostree older than v2023.4 will be too old to know how to create composefs images when upgrading. And so once we upgrade,
ostree-prepare-rootin the new initramfs fails because it sees that composefs should be enabled and yet no composefs image exists.This should be fixed by ostreedev/ostree#3353, which would allow ratcheting in composefs by letting
ostree-prepare-roottolerate missing composefs images.But clearly, longer-term we want to be able to rely on composefs just working the first time. This would definitely be fixed as we keep rolling out the bootimages update effort (openshift/enhancements#1496), but we may decide to do a shorter-term tactical fix for it if we're blocked on it.
common.yaml: disable composefs on all arches
jlebon commentedon Dec 16, 2024
For now, let's back out composefs: #1700
common.yaml: disable composefs on all arches
openshift-bot commentedon Mar 17, 2025
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting
/lifecycle frozen.If this issue is safe to close now please do so with
/close./lifecycle stale
openshift-bot commentedon Apr 16, 2025
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting
/remove-lifecycle rotten.Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting
/lifecycle frozen.If this issue is safe to close now please do so with
/close./lifecycle rotten
/remove-lifecycle stale
dustymabe commentedon Apr 21, 2025
composefs was changed back to
maybein #1735so it's enabled, but the
maybeallows older bootimages.dustymabe commentedon Apr 21, 2025
longer term fix for this is:
manifest-el9-shared: Update comment for composefs=maybe
openshift-bot commentedon May 22, 2025
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting
/reopen.Mark the issue as fresh by commenting
/remove-lifecycle rotten.Exclude this issue from closing again by commenting
/lifecycle frozen./close
openshift-ci commentedon May 22, 2025
@openshift-bot: Closing this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
jlebon commentedon May 22, 2025
/lifecycle frozen