dynamic overlays

This overlaps with #7 some.

Here, the basic idea is something like:

bootc config add [--root=/etc] examplecorp.com/config:latest

Where config.yml is a standard Kubernetes ConfigMap. By default, we "mount" the keys to /etc. Then, bootc upgrade looks for updates to all provided configmaps - if any change, it triggers the same upgrade logic as the base image.

We also fetch and handle secret objects in the same way. It'd be cool though to support something like handling encrypted secrets (and configmaps) which need to be decrypted via a key (which could be in a TPM or so).

We also need to think carefully about file permissions; mode 0644 for all configmap files and 0600 for secrets may make sense. In addition we could support special annotations to override these.

(This should also work to be invoked immediately after bootc install to have it ready on the first boot, i.e. we also have a --root argument or so)

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Also, another variant of this is that we should support fetching all this stuff from the cloud user data too.

So this design would work in an obvious way outside of Kubernetes - we're using a standard API type, but we're not depending on an API server or kubelet, etc.

But, in a Kubernetes context, it would make total sense to fetch this data from the API server instead of just running http. (And potentially use proper watches to react to events instead of polling). I'm thinking we probably want to make the mechanism to fetch this data pluggable; ultimately it might be that there's /var/lib/bootc/config.d and a privileged container image could e.g. use kubelet's credentials to fetch from the API server instead, and just hand things down to bootc.

This intersects with openshift/machine-config-operator#3327 too.

This intersects with openshift/machine-config-operator#3327 too.

A hugely interesting topic here of course is whether we should have a declarative way to set this stuff up. Today for example we have an imperative interface in bootc switch to specify the root image.

But...following the kubelet "static pod" model, and building on the pod analogy above, perhaps we have
/etc/bootc/host.yaml which is a subset of the pod spec, and allows declaratively writing the attached configmaps.

(This topic also relates to coreos/rpm-ostree#2326 )

It may also be interesting here to try to support something like an annotation which says the configmap changes should be live-applied by default - in the general case we'd probably need some sort of "post-apply hook" to ask affected services to reload. Or, such services could use inotify of course (at the risk of seeing partial states).

To implement this I'm thinking:

Add a kubernetes feature (on by default)
Use the https://docs.rs/k8s-openapi/latest/k8s_openapi/ crate
Support basic fetch via http, and investigate OCI artifact support (going to need to be driven into skopeo most likely)
Write configmaps as ostree commits (map name to ref, etc.); although we may want to actually split into "metadata" and "contents"; metadata changes to the configmap shouldn't appear as changes probably, except things like the potential update semantics
Create merged commit by applying configmap commits into /usr/etc to start - and here we have the interesting question of handling conflicts (i.e. what happens if my configmap has /etc/sudoers and the host has that file too? I'd probably vote that we error by default and require an annotation to overwrite?)
bootc update looks for changes to base and all configmaps

Now...what many people will want is a read-only /etc; if we have that, it becomes logically easier to have each configmap be a mounted tmpfs or whatever, which we can more easily implement individual live-apply semantics.

Trying to play around with this, the ergonomics are definitely somewhat annoying because configmaps can't have / in keys. If one wanted to e.g. write drop a a CA certificate into /etc/pki/ca-trust/source/anchors and also a local registry mirror config in /etc/containers/registries.conf.d it'd require separate configmaps with separate mountpoints. Maybe we just accept that for now.

Edit: Or, we could accept a List object of configmaps.

Add support for configmaps

cli: Add an `edit` verb

And outside of a Kubernetes API, the notion of ConfigMap doesn't exist, so I don't see how it applies practically to a standalone machine running Fedora.

https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.html exists and is usable without an API server. See also https://www.redhat.com/en/blog/running-containers-cars which argues for reusing container-ecosystem-adjacent tooling without an api server.

Do you plan to support other APIs?
I mean, Podman kube play supports Pod, Deployment, PersistentVolumeClaim, ConfigMap, Secret and DaemonSet. Which ones would you add to bootc?

Also, the Kubernetes doc says that ConfigMap are limited to 1MiB:

A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB. If you need to store settings that are larger than this limit, you may want to consider mounting a volume or use a separate database or file service.

Podman kube play supports Pod, Deployment, PersistentVolumeClaim, ConfigMap, Secret and DaemonSet. Which ones would you add to bootc?

I think bootc's role is just scoped to host management, so Pod/Deployment/DaemonSet are clearly out of scope, right? Those things should be handled by podman. Whether we do mapping around PersistentVolume for the host seems more open, but my instincts say that's just confusing and we should point administrators to systemd .mount units. ConfigMap and Secret to me are the most obvious things to support.

Files have a lot of metadata such as file mode, paths, owners, etc. Rather than encode all of this into an arbitrary JSON blob and stuff it inside a ConfigMap, why not use the metadata part of the ConfigMap to store these values as annotations?

Here's a rough idea of some annotations that could be used for this purpose:

Annotation Name:	Description:
`bootc.coreos.io/baseDir`	The base directory where each file in the ConfigMap will be created, e.g. /etc. This is required because the ConfigMap spec does not allow slashes in the name portion of the files it creates.
`bootc.coreos.io/baseDirMode`	The octal file mode (`0755`, `0644`, etc.) that the deepest directory in the path should have. Given `/etc/a/deeply/nested/base`, this would affect the `./base` portion of the directory mode.
`bootc.coreos.io/fileMode`	The octal file mode that each file in the ConfigMap should be created with (e.g., `0755`, `0644`, etc.). Additional file modes such as sticky bit, executable, et. al. should be supported. If multiple files within the same directory need to have different modes, more than one ConfigMap will be needed.
`bootc.coreos.io/userOwner`	The username or UID of the file's owner.
`bootc.coreos.io/groupOwner`	The group name or GID of the file's owner.

(From an organizational perspective note bootc is part of the containers/ GH org, not coreos/)

bootc.coreos.io/baseDir

Right, this one already exists in the WIP code...I think maybe what we should do is make a shared hackmd doc and refine it into a spec, then update the first comment here or so?

As far as the other metadata...maybe. So far, there hasn't been a use case for it in Kubernetes, right? I think specifically having executables in configmaps is probably something we should think of as an anti-pattern - we have container images for that. This reduces the need significantly for fileMode. The use cases for user/group owners also seem like they need very specific motivation.

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Would it make sense to enable only the configmap to be updated without updating the bootc image as well or would these actions/lifecycles be bound together somehow?

Would you be able to use an image manifest to store the config maps with the config descriptor could be used to describe how all of the layers should be applied to the file system?

These could then be customized per-architecture and distributed as an image index so a configuration file could be applied to the bootc installations regardless of the architecture.

Here are some of the ways we've discussed so far in how configs could be packaged:

configmap (YAML hosted somewhere or in an OCI artifact)
FROM scratch container image
systemd-confext (I guess... as an image file hosted somewhere or in an OCI artifact)

ISTM like those are completely orthogonal to how they're bound to the host (in the case of systemd-confext, it could be none at all, but... it could also be "managed" like the other options would be). Even in the managed option, there's e.g. configured in /etc (local state) or in /usr (closer to LBIs).

I've mentioned this before OOB, but just to surface it here, from the CoreOS side we're very interested in this capability because there the default entrypoint is not to do a custom build, but the day-2 config management is still extremely useful. IOW, we want this so as to reduce the need for either having to reprovision (current Ignition model) or having to maintain a CI pipeline (bootc model).

We had some in-person discussions related to this last week. Just wanted to jot down some of what was mentioned:

Strongly related to dynamic overlays #22 (comment) and as mentioned in on-cluster builds enhancement openshift/enhancements#1515, this has important implications for on-cluster layering in OpenShift. Currently we're building a derived images containing MachineConfig content. This means that as soon as a user changes a MachineConfig, we have to trigger a new build. If we instead have a separate way to apply config changes, it means a lot less churn in how often we need to rebuild the full image.
Following on the previous point, this might be out of scope for this issue but it'd be really nice if this just extended to /var as well. This would allow the MCO to get out of the business of writing files directly almost entirely without breaking backwards compatibility with what is currently possible with MachineConfigs. Obviously the implementation details would be different from the /etc case. E.g. we'd need to also think about reconciliation there, etc...

Can we rename this issue to e.g.:

Config management decoupled from OS updates

I think the configmap approach is just one potential implementation of this, but we should discuss which implementation is the most appropriate.

I think I'd prefer to keep this issue focused on a bootc-oriented deliverable for this.

But it absolutely makes sense to consider things that aren't configmaps or that wouldn't require changes to bootc...we could have a tracker issue somewhere for that? Could be a discussion? On this topic we do have https://docs.fedoraproject.org/en-US/bootc/dynamic-reconfiguration/ today and wherever we take that issue it'd make sense to add that content there (e.g. specifically the advantages/disadvantages of ansible, a custom agent writing files directly, maybe also systemd-confext)

I think I'd prefer to keep this issue focused on a bootc-oriented deliverable for this.

Agree with this, though that doesn't contradict

I think the configmap approach is just one potential implementation of this, but we should discuss which implementation is the most appropriate.

right?

Hmm, perhaps? I mean it kind of depends on the scope of solutions. I initially took your suggestion as being very broad potentially (again things like ansible or whatever).

But do you have an outline of something concrete that might be within bootc's scope that isn't configmap-like?

@jmarrero

Whoops, just realized I never got back to you here.

Briefly, I think configmaps are not rich enough for the purpose of configuring a whole OS. #22 (comment) captures it well I think (and while I agree e.g. executables or non-root uid/gids are not prevalent, they're certainly not uncommon). My understanding also from the k8s side is that configmaps tend to be abused for things that would be better represented as a proper CRD where you can have proper validation and types, etc...

As a first approach, I like better @jmarrero's idea of just shoving configs in a container image. Yes, it's awkward, but also builds easily, composes nicely (you can FROM a config image and add more things), etc... All the things around ownership, file modes, paths, etc... are all first class in tar. And it's trivial enough really that you can assemble it directly rather than actually doing a container build. (And actually... that's something I'm thinking could make sense for the MCO to do.)

Navigation Menu

Repository navigation

dynamic overlays #22

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

dynamic overlays #22

Description

Activity

cgwalters commented on Jan 18, 2023

cgwalters commented on Jan 18, 2023

cgwalters commented on Feb 1, 2023

cgwalters commented on Feb 2, 2023

cgwalters commented on Feb 11, 2023

cgwalters commented on Feb 17, 2023

25 remaining items

fabiendupont commented on Nov 2, 2023

cgwalters commented on Dec 1, 2023

cheesesashimi commented on Dec 8, 2023

cgwalters commented on Dec 8, 2023

arewm commented on Jun 4, 2024

jlebon commented on Oct 4, 2024

jlebon commented on Oct 4, 2024

jlebon commented on Oct 17, 2024

jlebon commented on Feb 11, 2025

cgwalters commented on Feb 11, 2025

jlebon commented on Feb 11, 2025

cgwalters commented on Feb 11, 2025

jlebon commented on Mar 20, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions