Repository navigation
Dev container metadata in image labels #18
Description
Unlike some developer centric formats, a non-goal for devcontainer.json is to become yet another multi-container orchestrator format. Instead, its goal is to enable development in containers regardless of how they are orchestrated. With this in mind, allowing image labels be used to set devcontainer.json properties could have significant advantages. In particular, there is a problem with images and devcontainer.json files needing to be paired but are deployed and managed separately, which can lead to disconnects. This is important in scenarios like container runtime settings required to enable debugging (e.g. cap_add: SYS_PTRACE), but applies broadly to any settings the images was purpose built to support.
In particular:
- When the dev container CLI is used to build the image, these labels can added automatically.
- Labels can be added to images using a Dockerfile, command line, an orchestrator format, or another image build system like Buildpacks.
This second point also makes it easier to use the metadata when some other mechanism is used to build the container images - whether an orchestrator or another non-Docker CLI like pack for Buildpacks.
The json-based nature of devcontainer.json would main this fairly straight forward. To simplify just reusing json blobs in automation scenarios, we can also support a whole structure being added:
LABEL dev.containers.metadata='{ "userEnvProbe": "loginInteractiveShell", "customizations": { "vscode": { "extensions": ["ms-python.python", "ms-toolsai.jupyter" ] } } }'For hand editing, common and less complex properties could be referenced directly.
LABEL dev.containers.metadata.userEnvProbe="loginInteractiveShell"
LABEL dev.containers.metadata.customizations.vscode.extensions= '["ms-python.python","ms-toolsai.jupyter"]'More complex any type properties could then be encoded json.
LABEL dev.containers.metadata.customizations.vscode.settings="{\"some.setting\": \"some-value\"}"In the common case, these labels would be automatically added by the dev container CLI to the image when it is pre-built. (devcontainer build ...), but manual entry would enable these additions to be embedded in orchestrator formats as well.
services:
app:
build:
context: .
dockerfile: Dockerfile
labels:
- "dev.containers.metadata.userEnvProbe=loginInteractiveShell"
- "dev.containers.metadata.customizations.vscode.extensions=[\"ms-python.python\",\"ms-toolsai.jupyter\"]"Any tool that supports the dev container spec would then look for these image labels - regardless of whether they are on a pre-built image or one built by an orchestrator. The reference implementation would then illustrate how to make this happen.
We should be able make this work with any general, lifecycle, or tool specific property.
Dev container features
The dev container CLI enables pre-building images that include the proposed dev container features concept. However, there's a couple things here to consider. First, we can include an image label that indicates whether the build step is already done for the image. This avoids accidentally re-firing of the same feature multiple times based on reused configuration.
LABEL dev.containers.features.built='["docker-in-docker","github-cli"]'Second, we can render out the devcontainer.json properties that tie to the feature in the resulting image. e.g., extensions would include those that were indicated by the feature. Properties like capAdd we've talked about as higher level properties, so these could be handled the same way.
Processing then is always: build (optional), read labels with metadata, run. You can use devcontainer.json as you use it today, but the labels could also come from somewhere else.
Beyond providing processing consistency, this avoids a possible issue where the referenced feature in devcontainer.json updates, but the pre-built image does not. Problems here could be extremely difficult to track down when they happen otherwise.
Multi-container
Finally - this should help with the [one-to-many problem at least tactically or when there is not a specific integration to an orchestrator format. For pre-building, each image can have a separate devcontainer.json or Dockerfile with the labels - with the dev container CLI enabling features. These can then be referenced in an orchestrator directly. Even when the dev container CLI is not used, at least a subset of properties could be included in the orchestrator format.
Net-net, the resulting image should have labels on it that explains how it should be set up. Orchestrator formats can influence processing where they support adding properties. In all cases, the dev container CLI will inspect the image and make the needed adjustments.
Activity
jkeech commentedon Mar 21, 2022
Embedding devcontainer metadata in the generated images as labels seems like a really valuable feature 👍. A couple quick comments/questions:
.microsoftportion of the label namespace and just go with something likecom.devcontainersince we aim to make this more of an open/neutral specification.Chuxel commentedon Mar 21, 2022
John Keech (@jkeech) There may be a maximum (I need to check), but it's large. From what I've seen, things like Buildpacks add pretty giant json structures as labels - far bigger than what a devcontainer.json would include.
The reason for considering separate properties is simplify the process of hand editing and additions into a Dockerfile when you just have one or two. Format wise, json is quote heavy, which means lots of escaping. We could support both I suppose. e.g.
com.microsoft.devcontainercould be the full json blob.Yeah, we need to use a domain we own (or more accurately, the spec owns), but agreed that the
.microsoftbit we'd ideally get rid the.microsoftpart as we open the spec. I talked to Brigit Murtaugh (@bamurtaugh) about one option to consider. I just didn't put something else down since I didn't want to set false expectations - we can update the summary once we have something.joshspicer commentedon Mar 21, 2022
If the limit is not too high, it seems cool to me to store the entire partial devcontainer.json as a label, and be able to "reverse engineer" the devcontainer components of an image without too much manual copy/pasting.
Doing both also makes sense to me for flexibility.
Chuxel commentedon May 27, 2022
This repository has a concrete example of where this could be useful. It uses Buildpacks to generate dev container images w/config. https://github.com/chuxel/devpacks
The image build is performed in a way that is not easily to update (since creating a Buildpack is not easy), and does not have an out-of-box way to generate secondary outputs like devcontainer.json. Allowing information to be on labels on the image results in a completely independent workflow being able to participate in devcontainer creation. It also covers #2 and illustrates setting container capabilities for Go (specifically ptrace).
To John Keech (@jkeech)'s point, I just used the label
dev.containers.metadatafor a merged json structure. There's a little CLI to bridge gaps right now - but it's pretty simple, so I think it's got a great value for effort ratio. 😄chrmarti commentedon Jul 18, 2022
Goal
Record feature metadata in prebuilt images, such that, the image and the built-in features can be used with a devcontainer.json (image-, Dockerfile- or Docker Compose-based) that does not repeat the features or feature metadata. Other tools should be able to record the same metadata without necessarily using features themselves.
Feature metadata relevant for using the feature when it is already part of the image:
mounts,init,privileged,capAdd,securityOpt,entrypointandcustomizations.Proposal
We investigated adding a devcontainer.json to the image as a label on the image. This devcontainer.json would contain the merged feature metadata (same merge logic as when starting a dev container, e.g.,
privilegedistruewhen it istruefor one or more features). This approach has the following difficulties:runArgsfor the image- or Dockerfile-based devcontainer.json to recordinit,privileged,capAddandsecurityOptfeature metadata, butrunArgsdoes not apply when the user's devcontainer.json uses Docker Compose."privileged": truein the feature metadata can be recorded in the devcontainer.json as"runArgs": [ "--privileged" ].customizations. (E.g., there is currently no VS Code-specific code in the CLI required to handle that.)"foo"might have an array of values as customization and two features might have different values for it:{ "customizations": { "foo": { "bar": ["baz"] } } }{ "customizations": { "foo": { "bar": ["xyz"] } } }{ "customizations": { "foo": { "bar": ["baz", "xyz"] } } }{ "customizations": { "foo": { "bar": ["xyz"] } } }.mounts) and the correct way to merge can depend on what the user intends to do.To avoid these hurdles we propose to add metadata to the image with the following structure using one entry per feature:
To simplify adding this metadata for other tools, we also support having a single top-level object with the same properties omitting the
idproperty.We already have the merge logic for merging the metadata from multiple features originating from a devcontainer.json and can use the same logic for also merging this new metadata from the container image.
Notes
Chuxel commentedon Jul 20, 2022
Christof Marti (@chrmarti) Beyond recoding feature metadata, the goal was also to enable alternate build systems like Buildpacks or Nixpacks or even Docker Compose itself without the ability to output a devcontainer.json file to contribute dev container metadata. This is also important and should not be required to use features under the hood (and in some cases can't use them). Are you thinking the feature metadata is in addition to the main proposal? Or would you express the metadata in this situation like a feature even though no such feature exists? If the idea would be to express the metadata as a feature, we should probably clarify how this should be expressed - in particular how we would think about the "id" in this situation.
Christof Marti (@chrmarti) FYI - Raised #2 a while back for this bit. Agree 100% we should do it as a part of this.
jkeech commentedon Jul 20, 2022
This is an important part. Even when using the
devcontainer buildcommand, you could have the followingdevcontainer.json:{ "build": { "dockerfile": "Dockerfile" }, "features": { "node": "latest" }, "postStartCommand": "npm install", "customizations": { "vscode": { "extensions": [ "my-publisher.myextension" ] } },The resulting image metadata should have all of the feature contributions that need to get applied at runtime as well as the lifecycle hooks like
postStartCommandand customizations in the devcontainer.json. That way the end user can just reference the resulting image and get everything working as expected in the same way is directly running the originaldevcontainer.json.Instead of cherry-picking which properties from devcontainer.json and from features would need to get applied at runtime and carried forward in the image metadata, it might be better to include the full contents. That way we don't have to continually update the list of properties that apply after the image is built as new properties are added to the spec.
Chuxel commentedon Jul 21, 2022
Yeah, postCreateCommand was used in the Buildpacks prototype I did. It is partly why I raised #60. It also assumed
#2 was done for the reasons mentioned above since one of the buildpacks was for Go and needed ptrace.
chrmarti commentedon Jul 21, 2022
The difficulty with adding an entire devcontainer.json is that the correct merge logic for a property depends on its semantic. E.g., when merging string properties of the same name I would generally guess that the last one wins, but with
postStartCommandwe actually want to collect them and run them all. Also some properties from the devcontainer.json (e.g., the Dockerfile path) might not make sense as metadata on an image.To simplify use of the metadata by other tools we support a single object and an array of the same type of object (updating
idto the optionalfeaturehere and adding the lifecycle properties):E.g.:
[ { "postStartCommand": "npm install", "customizations": { "vscode": { "extensions": [ "my-publisher.myextension" ] } } }, { "feature": "devcontainers/features/node@1.2", "customizations": { "vscode": { "extensions": [ "dbaeumer.vscode-eslint" ] } } }, // ... ]Chuxel commentedon Jul 21, 2022
Christof Marti (@chrmarti) Alexandru Dima (@alexdima) as we just chatted, this could work if we also did #2 so that we can encourage people not to use things like "runArgs" in the pre-build scenario given we know how common these are. We can update all of our samples as well. (In some ways it's tech-debt to back-port what we already did for features - the reasons for the properties is the same.) Totally good with the array idea and merging later.
For the list of properties available in the label though, looking at devcontainer.json, I'd propose we support all General and Lifecycle properties. In particular, these will matter if you pre-build your image that is not in the list above:
remoteUser- This one is in every devcontainer.json we've published and has major impacts.userEnvProbe- Has a m major effect on what ends up in the container.remoteEnv- Used to update the PATH since it can reference containerEnvoverrideCommand- Some images will not work without this being set.portsAttributes/otherPortsAttributes- Users that name their ports or set HTTPS as the protocol should be respectedThese are a bit debatable, but I think customers would expect if you built from a devcontainer.json, they would be applied:
forwardPortsshutdownActionupdateRemoteUserUIDThese are not applicable to Dev Container Features (#60), but that's because they are inputs to constructing the image. Here we are dealing with a fully constructed image that either needs to container dev container metadata, or was created as a build from a devcontainer.json file.
Or put another way, it may be easier to just say we don't support Scenario Specific Properties. These are really the ones that would differ between orchestrator rather than being about the contents of the image itself. John Keech (@jkeech) made a pretty good point there as I think about it.
chrmarti commentedon Jul 22, 2022
We have to decide on how to merge each property. Going by the above list:
featuredevcontainers/features/node@1.2initbooleantrueif at least one istrue,falseotherwise.privilegedbooleantrueif at least one istrue,falseotherwise.capAddstring[]capAddarrays without duplicates.securityOptstring[]securityOptarrays without duplicates.entrypointstringmounts(string | { type, src, dst })[]onCreateCommandstring | string[]updateContentCommandstring | string[]postCreateCommandstring | string[]postStartCommandstring | string[]postAttachCommandstring | string[]customizationsremoteUserstringuserEnvProbestring(enum)remoteEnvcontainerEnvoverrideCommandbooleanportsAttributesotherPortsAttributesforwardPorts(number | string)[]shutdownActionstring(enum)updateRemoteUserUIDbooleanDoes it make sense, e.g., for
overrideCommandto have a different logic thaninit?12 remaining items
chrmarti commentedon Sep 1, 2022
Actually, this needs clarification: We need to make sure features work independently of their order, e.g., if a feature sets
"privileged" true, another feature should not be able to override with"privileged": falsebecause that would potentially break the first feature (e.g., withgodebugging would no longer work). To retain the "last set value wins" semantic in the image metadata we can remove any"privileged": false(and similar flags) from the feature metadata at the time we add the metadata to the image.chrmarti commentedon Sep 1, 2022
I agree. I was thinking specifically about the
customizationsproperty where the correct merging strategy also depends on the semantics of the properties (like with built-in properties) and these are not known to the CLI. E.g. with:and
you might want to merge the permissions both as
writebecause that will satisfy the requirements of both configurations.Chuxel commentedon Sep 1, 2022
Christof Marti (@chrmarti) Yeah I think whatever logic we put in place should be emitted by the CLI. I think documenting a default behavior and then mentioning that tools can override this behavior is a good starting point. To your point, the default on booleans should be "or". For the CLI, generally this indicates that we may need a way for implementors to contribute their own merge semantics where needed so they can alter default behaviors in the CLI. The CLI can also emit all details like it does now for features if an implementing client wants to do something on its own. But, for merged view, we could create a function interface that for a "customizations" name and a given property takes in two values to perform a merge. If no function for a given customization is available, it drops to the default logic. This would be something that could be contributed in addition to a schema update, and we could document that fact
That keeps things simple for the majority, but allows for exceptions, rather than forcing complexity on everyone unequivocally. I can see conceptually that there could be customizations that are also used by more than one implementor, so this also keeps logic consistent for those scenarios.
chrmarti commentedon Sep 6, 2022
For image metadata and/or feature metadata? The previous suggestion seemed to be that the "last one set wins" in the image metadata and "
trueif at least one istrue,falseotherwise" in the feature metadata.The CLI is currently the "API". If we were to make it extensible we would either have to accept other (small) CLIs as extensions or start having a JavaScript (or so) extension API. We have carefully avoided any need for either so far as that would introduce more complexity. I don't see a pressing need to change that and suggest we first see how far we get without.
chrmarti commentedon Sep 6, 2022
On merging booleans: Maybe
overrideCommandandupdateRemoteUserUIDonly make sense in the devcontainer.json and not in the feature metadata and therefore "last wins" is an appropriate merge strategy because that allows for overriding later. Forinitandprivilegedtheorstrategy seems to make more sense because that is about including the required behavior or permission for a feature.chrmarti commentedon Sep 6, 2022
Updated proposal (I have removed the previously included discussion of alternatives for clarity):
Goal
Record feature metadata in prebuilt images, such that, the image and the built-in features can be used with a devcontainer.json (image-, Dockerfile- or Docker Compose-based) that does not repeat the features or feature metadata. Other tools should be able to record the same metadata without necessarily using features themselves.
Feature metadata relevant for using the feature when it is already part of the image:
mounts,init,privileged,capAdd,securityOpt,entrypointandcustomizations.Proposal
We propose to add metadata to the image with the following structure using one entry per feature and devcontainer.json (see table below for the full list):
To simplify adding this metadata for other tools, we also support having a single top-level object with the same properties.
The metadata is added to the image as a
devcontainer.metadatalabel with a JSON string value representing the above array or single object.Merge Logic
To apply the metadata together with a user's devcontainer.json at runtime the following merge logic by property is used:
idghcr.io/devcontainers/features/node:1initbooleantrueif at least one istrue,falseotherwise.privilegedbooleantrueif at least one istrue,falseotherwise.capAddstring[]capAddarrays without duplicates.securityOptstring[]securityOptarrays without duplicates.entrypointstringmounts(string | { type, src, dst })[]onCreateCommandstring | string[]updateContentCommandstring | string[]postCreateCommandstring | string[]postStartCommandstring | string[]postAttachCommandstring | string[]customizationsremoteUserstringuserEnvProbestring(enum)remoteEnvcontainerEnvoverrideCommandbooleanportsAttributesotherPortsAttributesforwardPorts(number | string)[]shutdownActionstring(enum)updateRemoteUserUIDbooleanhostRequirementscpus,memory,storageNotes
edgonmsft commentedon Sep 7, 2022
Sounds good to me, only the following comments:
chrmarti commentedon Sep 8, 2022
It might be best to keep
userEnvProbeat its defaultloginInteractiveShell(unless the user explicitly requests otherwise) to ensure tools depending on the shell startup scripts work. So maybe this is another case where we don't want features to have the same flexibility as the devcontainer.json.Chuxel commentedon Sep 8, 2022
Edmundo Gonzalez (@edgonmsft) Christof Marti (@chrmarti) Yeah, any value that is not set should remain at its default across the board. The same is true for booleans, so there's really three states: true, false, unset. This mirrors how devcontainer.json itself is processed.
chrmarti commentedon Sep 8, 2022
Chuck Lantz (@Chuxel) Do you agree that features should not be able to change
userEnvProbe? Changing it seems to have too broad an impact to allow for individual features to control it.Chuxel commentedon Sep 8, 2022
Christof Marti (@chrmarti) Image labels should be able to do it for sure given you can do this from devcontainer.json, but Features is an interesting question given the default is
loginInteractiveShellwhich is the most aggressive and what I'd typically expect Features would want (since they may need to alter rc files). In the past when the default wasnoneI would have said that they absolutely need to support it - but things are better here now.Since the user may extend an image that already has the value set, we'll still need to support processing more than one value. However, given that, adding it into Features later would be pretty trivial I'd assume - so omitting for now is probably the right thing to do until we get feedback otherwise.
Unrelated: One other thing we may want to document is how variables will be applied - I'm assuming these would generally just appear as the variables and processed at the end when the container is coming up (rather than when the label is applied). Is that how you were thinking about it as well?
chrmarti commentedon Sep 9, 2022
I agree, that seems to make the most sense.
chrmarti commentedon Sep 14, 2022
Posted the proposal with the discussed clarifications in PR #95 for review.
eitsupi commentedon Nov 3, 2022
Hello. What is the reason why the label is
devcontainer.metadatainstead ofdev.containers.metadata?dev.containers.metadataseems more natural as to the OCI spec Annotations.https://github.com/opencontainers/image-spec/blob/ff0257194e20f7fd17ca21c67a2e725f21c4d4f2/annotations.md