OCI Image Deprecation: vendor support

[ValentinViennot]

Summary

This proposal discusses a solution to improve the deprecation of vendor-supported products distributed as OCI images. The effectiveness of the proposed solution depends on OCI tooling and registries implementing derived features.

We intend to introduce new OCI annotations to add the “End of Support” and “End of Life” dates information to the OCI image metadata. This would allow vendors to provide a standardised product-level support timeline information that can then be used by registries, marketplaces, and image tooling.

Background

Container images are immutable and are often distributed in a decentralised manner. As a consequence, deprecating outdated versions and controlling their distribution is a difficult and yet-to-be-resolved issue [1].

Vendor-supported software usually follows the following lifecycle: start of development, general availability, end of life, end of support.

Let’s say “Ubuntu 16.04” (docker.io/library/ubuntu:16.04) reaches end of life, there is no way of warning downstream users that they should consider moving away from this container image.

We identified three solution areas:

• scanning tools flagging outdated images and informing on continuity options
• registries flagging outdated images and documenting the continuity path
• built-in “time bomb” binary (either with hardcoded EOL date or API call)

Additionally, we split into two types of deprecation:

• known-in-advance for products with planned and predictable vendor support
• mutable products without support timeline and release on release deprecation

Motivation

There’s currently no good story to OCI images deprecation, including on popular public registries such as Docker Hub [1].

From a vendor point of view, this can lead to bad reputation with use of outdated images and difficulty to track versions in use. However, purely retiring an image by removing old tags would potentially break downstream users and could lead to ever worse scenarios.

From an end-user perspective, it’s a risk and a potential liability to keep using unsupported or abandoned versions of OCI images without being aware of it.

We believe that a solution could be implemented at the OCI specification level leading to subsequent implementations in OCI-compliant solutions such as registries and image tooling. In short, we need a plan for what happens to the dead images.

Proposal

Support annotations

We propose to extend the org.opencontainers.image annotations with information regarding the support and deprecation timelines. It is expected that the maintenance and support are provided by the entity identified in org.opencontainers.image.vendor.

org.opencontainers.image.support.end-of-life

Date after which the software won’t receive new features/updates from the publisher. Users should stop using this OCI image and move to a newer version. Publishers may or may not continue to ship security updates. On this day, the OCI image is deprecated.

• This MAY be specified (optional field)
• This SHOULD be specified for maintained software with predictable end of life
• If not empty,
  • this MUST be a valid ISO 8601 (YYYY-MM-DD) date format [2]
  • org.opencontainers.image.vendor MUST provide contact information to the entity responsible for support
  • org.opencontainers.image.support.info SHOULD provide necessary information for the end user to access the support services

org.opencontainers.image.support.end-of-support

Date after which the software won’t receive service, support, or security updates from the publisher. Users must stop using this OCI image. On this day, the OCI image is considered outdated, dangerous to use, and might become unavailable.

• This MAY be specified (optional field)
• This SHOULD be specified only when support.end-of-life is also specified
• If not empty,
  • this MUST be a valid ISO 8601 (YYYY-MM-DD) date format [2]

org.opencontainers.image.support.info

URL or text to provide more information about vendor support and upgrade options.

• This MAY be specified (optional field)
• This SHOULD be displayed by tooling advertising the deprecation status

Limitations

• Later change in support dates: usually, this should only be for extending the support timeline as any initial support commitment must be honoured by org.opencontainers.image.vendor
• Mutable deprecation: this proposal addresses product-level support, but the image content being immutable, one would have to find a way to constantly update their OCI images as new builds with patches become available
• Images built upon (“FROM”) deprecated OCI images: downstream products based on a deprecated image wouldn’t include the support information and might be shipping deprecated content without any warning
• Dependance on tooling and registries to implement solutions using this spec: the success of this specification in solving the issues identified is entirely dependent on OCI-compliant toolings implementing efficient solutions (e.g., registries displaying the information on their UIs and CLIs, build/pull/run CLIs like Docker and Podman to warn the end user trying to use a deprecated image…)

Future Work

Dynamic deprecation for unpredictable product support and for “build on build” image deprecation (i.e. when a new build is released, to deprecate the previously tagged and now replaced content) (— I have a proposal in progress, but any suggestions are welcome!)

References

[1] docker/hub-feedback#907
[2] https://www.iso.org/iso-8601-date-and-time-format.html

Signed-off-by: Valentin Viennot [email protected]

[Jamstah]

Hi Valentin, I'm not a maintainer just FYI, but have some opinions here I thought worth sharing. It's a great problem area and I agree that outdated software is a problem in most environments, and welcome proposals to make it easier to manage.

However, I'd be worried about anything attempting to rely on immutably set future dates as a measure of supportability for an image. You can have some faith that an organisation like Canonical could maintain support as promised, but I would find it hard to trust these dates regardless of the vendor. I feel like the only thing they could tell you is when you're unlikely to be able to receive support (the date has passed) although even then, as you say, support periods can be extended. I would be hesitant to rely on them to tell me my image was supported.

Secondly, what does it mean that an image is supported? Does it mean security fix updates will be provided? Does it mean bugs will be fixed? What kind of bugs? Do you have to pay for support? There are a lot of unanswered questions.

I'd also like to propose an alternative - a model along the lines of Certificate Revocation Lists. If an image were to include a link to a CRL style API that lists the digests of outdated images (although that would need to be defined more precisely) then images could be checked by scanning tools, registries, or other clients, especially as the lists could be decentralised to avoid tracking (which as you mention is generally frowned upon!)

[sudo-bmitch]

Most of my suggestions are to align this with the verbosity of the other annotations. I've added this to this week's OCI discussion, feel free to join in. Note, I'm not a maintainer here, so I defer to them for any decisions on this.

[sudo-bmitch]

The DCO check is also throwing this error:

Author: Valentin Viennot, Committer: Valentin Viennot; Expected "Valentin Viennot [email protected]", but got "Valentin Viennot [email protected]".

Not sure if it's picking up the "noreply" email from your local Git environment, or your GitHub account.

[SteveLasker]

Hi @ValentinViennot,
Great scenario, and something that's been a hot topic around when an artifact should be considered EOL, and what actions should be taken.
The interesting part is when would that value be known? Do you really know what the EOL be when you create the image (artifact)? You might have some idea, but software EOL is often extended, or reduced as it may be surperceeded by a "better product" or it may have insecure aspects that only became known after it was shipped.

This basically makes it interesting when you can add the annotation, and how would users consume it.
We've been doing a bunch of work around the secure supply chain, where you can add information to an artifact (image) after it was built, and after it was pushed to a registry. This started with additional signatures to provide proof that some additional process was taken. Such as importing content into a private registry, after it was scanned. You can also add a SBOM, a scan result or many other things.
To support these enhancements to an artifact, we created the ORAS Artifact spec which enables you to add signatures, sboms, and... additional annotations.
Because the ORAS Artifact manifest doesn't require blobs|layers, you can push a reference type manifest that is just an annotation.
You can then use the referrers to pull the references, including the annotations.

{
  "mediaType": "application/vnd.oras.artifact.manifest.v1+json",
  "artifactType": "org.cncf.oras.manifest.annotations",
  "blobs": [],
  "subjectManifest": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:73c803930ea3ba1e54bc25c2bdc53edd0284c62ed651fe7b00369da519a3c333",
    "size": 16724
  },
  "annotations": {
    "io.cncf.oras.artifact.support.end-of-life": "<date>",
    "io.cncf.oras.artifact.redirect-to": "registry.example.com/product:v3"
  }
}

It would be interesting to hear more about your thoughts on the timing issues for additional annotations.

[vbatts]

was just coming to comment, and saw that Lasker said similar.

Making these annotations as first-class keys, I do not think this workflow will ultimately provide the flexibility desired. If there were an extension on support, it could not be modified without changing the image digest. If there were a change in the URL, it would change the image digest.

Instead, working with our Referrers WorkingGroup, such that this kind of support information could be attached to an image digest, then future updates to this information would not change the image digest itself.

[ValentinViennot]

Thanks for the prompt replies, and apologies for the long lag!

a model along the lines of Certificate Revocation Lists

@Jamstah, I'm glad you suggest this option as this is the WIP proposal I mentioned in Future work! A significant advantage I also see there is the ability to depreciate at the image level (instead of product level). However, the work involved seems big enough that I considered product-level depreciation valuable enough on its own to split proposals and start with the easier one.

[ValentinViennot]

Thanks @sudo-bmitch for the suggestions. All addressed! (and indeed, my local config wasn't right)

[ValentinViennot]

@SteveLasker @vbatts, I did not know about the Referrers WorkingGroup! Sounds exciting and much more relevant for this scenario indeed. I will have a look there.

(edit: from what I understand, the ORAS artefact spec would even enable to dynamically append to an image "this image is now deprecated, go look there instead".)

[sudo-bmitch]

The reference type working group makes a lot of sense for a way to update these annotations on existing images without modifying the image. The ORAS artifact spec is one of the options we are considering, but even if accepted, it would be quite some time before registries are updated to support the new artifact media type and APIs.

More importantly, once we have reference types sorted, I believe we would still want to define these annotations. The question may be more about how runtimes and build tooling should check for these annotations. Do we want to support annotations on any of the index, image, and any associated reference types? If they are only set on an index, they may be missed if running an image by the image digest. Requiring a check of artifacts before building or running any image could result in a lot of overhead in certain situations (depending on our implementation, there could be lots of other references to the same image that need to be filtered through on the client side, pulling each one).

I've also been working on tooling that makes setting annotations on an existing image a lot easier (regctl image mod). Which means you could set a short end-of-life, and extend it periodically, without rebuilding the image and modifying the layers. Only the manifest digest would change. That also forces users building/running by digest to periodically update their digest, which they should be doing anyway.

@vbatts what are your thoughts on approving this now to get the annotations defined for some use cases today, since we'd still want these same annotations to use it via reference types later?

[ValentinViennot]

it would be quite some time before registries are updated to support the new artifact media type and APIs.

@sudo-bmitch, Yes, that's one of the issues I have in mind. The idea was to first go with a quick and easy proposal, even if it only covers specific product-level support situations. Later, to explore something more like CRLs. An incremental approach in which reference types can base on this initial work to solve the dynamic depreciation is appealing!

Which means you could set a short end-of-life, and extend it periodically, without rebuilding the image and modifying the layers.

I like the idea! I'm trying to think about how we could encourage this behaviour. The argument is also that, from looking at an OCI image's manifest, you could tell which one has vendor support and for how long. In the proposal, "End of life" is meant to be shorter and could be what you propose, and "End of support" is specified only when long-term enterprise support is available.

[ValentinViennot]

@SteveLasker, some thoughts on your questions:

when would that value be known? Do you really know what the EOL be when you create the image (artifact)? [...] software EOL is often extended, or reduced [...]

I agree that this proposal covers only a limited subset of "End of life" scenarios, particularly the long-term supported products with an organisation behind them. It doesn't solve the generic depreciation problem (which we might solve with something in the spirit of Certificates Revocation Lists or reference types).
We know the EOL date from the publication day for this specific vendor support scenario. Often, we might extend it, and I don't see this as an issue: The EOL date is an annotation on an immutable packaging index. It's information concerning product-level support, and we would expect end-users to update their images regularly to benefit from security updates. In that sense, I like @sudo-bmitch's proposal of having short EOL timelines and refreshing the EOL date in new Manifest upgrades even without changing the image content.
Vendor support, as proposed here, feels like a commitment to me and should never be shortened. However, I assume this could still happen in some scenarios.

Your question made me realise that I forgot to specify in the initial PR that the all org.opencontainers.image.support.* information is about the org.opencontainers.image.title product for its org.opencontainers.image.version as supported by org.opencontainers.image.vendor.

how would users consume it.

Now that I clarified my PR's missing part, this part should also make more sense. The idea is first to help end-users avoid using outdated content and make the best decision from the start (as in, picking images with support timelines compatible with their project). But the information – in this proposal – is really to cover product-level support for long-term vendor-supported products.
(edit: the expectation is for this info to be used by a registry UI/CLI, scanning tools, build tools, etc.)

the ORAS Artifact manifest [...]

The problem with OCI resource depreciation is more extensive, and the reference types seem like a cleaner approach to solve it.
Thanks again for sharing this; I'm looking into it with the team now, and we're amazed!

[SteveLasker]

Vendor support, as proposed here, feels like a commitment to me and should never be shortened.

I tend to agree that "support shouldn't be shortened", unless an exploit makes the product unsupportable. There are likely other cases, such as acquisition or the company ceases to exist.

...set a short end-of-life, and extend it periodically, without rebuilding the image and modifying the layers.

I'd actually suggest this is a leftover of x509 and CRL hitting their limits. Requiring a client to continually check back, with continued extensions sets expectations that unless there's an extension, there's an expiration. This suggests that all products would set some short life to ___, whether it's an EOL or another valid state. This will require more entries to continue to extend, more clients to continue to check for that extension. Not all products will know this, but many will try and add this information as it's "part of the automation". I'd suggest this will be too much info and too much chatter.
The other approach is to simply update the stream of information when something actually changes. When a product is built, it' doesn't have an EOL, as it's simply unknown when the next version will be available. And, just because a new version is available, doesn't mean I should be forced to use it. Remember, the Solarwinds exploit was an update.

it would be quite some time before registries are updated to support the new artifact media type and APIs.

I continue to hear this, which I struggle with. k8s is far more complex as millions of instances exist across clouds and on-prem instances. k8s continues to make major changes. ipv6 as an example. Yes, it does take time, but there's a start date, and folks engage on moving things forward. I'm hopeful to see the output of the working group evolve the standards. There are far fewer registry instances than k8s.

Isn't this really an example of:

Standard evolve through incubation, and execution, with parallel standards efforts.
In line with that, ACR has implemented ORAS Artifacts, AWS has support coming and Zot also has support.

The annotation idea came from early implementations that solve these problems as well as the filtering and sorting on the reference types.
I'm hoping we'll see more implementations, and continue to learn the new capabilities we can add as registries are at the core, empowering the cloud native ecosystem of containers, the secure supply chaing and all sorts of other artifact types.

[adrosa]

I like proposal but org.opencontainers.image.support.end-of-support seems to imply that this particular image will last until that date. this is not the case. for example their are ~70 versions of ubuntu:bionic named like ubuntu:bionic-20220427

1. add a .state field (see bellow)
2. add .ref to clarify what image this promise is for
3. add .until to promise support that state
4. add .build that allows user to pull a particular build
5. add .update that recommends update path

state names

we use these lifecycle states based on Microsoft Lifecycle Policy:
preview - not yet officially supported and no commitment to ever be supported ( should warn user )
mainstream - general availability and active development
deprecated - no longer in active development, will receive security updates ( should warn user )
unsupported - will no longer receive quality or security updates ( should error without --allow-unsupported )
removed - this is no longer available for use and has been removed from a product or service.
( image doesn't include empty and fails to start )

extra states

might make sense to add 2 additional states
development - in development before public preview aka private preview. no promises at all. ( error unless --allow-preview )
extended-support - only supported for customers with special agreement

Proposed additions

org.opencontainers.image.support.state - the lifecycle/support state
org.opencontainers.image.support.until - (optional) a promise that this .ref will not be down-supported at least until date specified.
org.opencontainers.image.support.ref- (optional) the ref the until this promise is for
org.opencontainers.image.support.build - (optional) pulling this ref gives you same build with possibly different labels.
org.opencontainers.image.support.update - (optional) recommended version to update to, non breaking if possible

example

org.opencontainers.image.support.state = "stable"
org.opencontainers.image.support.until = "2028-05-11"
org.opencontainers.image.support.ref= "docker.io/library/ubuntu:bionic"
org.opencontainers.image.support.build = "docker.io/library/ubuntu:bionic-20220427"
org.opencontainers.image.support.update = "docker.io/library/ubuntu:bionic"

note

FROM should mask org.opencontainers.image.support.*

tools should warn if image includes org.opencontainers.image.support.ref that doesn't match the image ref.
maybe tools should add the .from themselves?

[sudo-bmitch]

I like proposal but org.opencontainers.image.support.end-of-support seems to imply that this particular image will last until that date. this is not the case. for example their are ~70 versions of ubuntu:bionic named like ubuntu:bionic-20220427

1. add a `.state` field (see bellow)

2. add `.ref` to clarify what image this promise is for

3. add `.until` to promise support that state

4. add `.build` that allows user to pull a particular build

5. add `.update` that recommends update path

A lot of this is confusing to me since the annotation is on the manifest, and the manifest will have a digest. So if someone pulls by digest or one of multiple tags to the same digest, this implies the behavior should be different for each. It also has a lot in common with the Docker schema v1 images that tightly integrated the tag into the manifest, which made copying images to other registries or retagging an image would create a new manifest.

[ecki]

While I think the topic is important, I am not sure a image annotation cuts it. There is this „image is supported“ notion, where supported means vendor will give you updated(!) new images. So by definition, no matter o a image is eol or not, you will need to install a new one.

so maybe this should more focus on „we invent to roll a new version no later than“? And also make eos dates for stable tags.

btw, which reference specs it is no problem to attach „revocation“ or „update notice“ to existing images, that looks like a better road to travel? Especially as it can also be used by other entities (like „this image is in the company registry for reproduceabilty reasons but the governance department says „do not use use or new product builds“.

[sudo-bmitch]

With a rebase the shorter link syntax will work. The reference types working group has been merged, but I don't think that affects this too much. EOS/EOL data could be attached as an annotation only referrer, but I'd lean towards pushing a short lived manifest and periodically extending the date with a new manifest. Either way that would involve defining the annotations so I'm comfortable approving with the minor change.

[sudo-bmitch]

Suggested change

	* **org.opencontainers.image.support.end-of-life** Date and time when the image stops receiving new features and bug fixes (security updates may continue). (Date-time as defined by [RFC 3339](https://tools.ietf.org/html/rfc3339#section-5.6).)
	* **org.opencontainers.image.support.end-of-life** Date and time when the image stops receiving new features and bug fixes (security updates may continue). (Date-time as defined by [RFC 3339][rfc3339].)

[toddysm]

Question about this one:

Date and time when the image stops receiving new features and bug fixes

Considering that images are immutable, this is a little confusing. Wouldn't this be a completely new image anyway? Also, doesn't EOL mean also no security patches?

[sudo-bmitch]

Suggested change

	* **org.opencontainers.image.support.end-of-support** Date and time when the image stops receiving support, including new features, bug fixes, and security updates. (Date-time as defined by [RFC 3339](https://tools.ietf.org/html/rfc3339#section-5.6).)
	* **org.opencontainers.image.support.end-of-support** Date and time when the image stops receiving support, including new features, bug fixes, and security updates. (Date-time as defined by [RFC 3339][rfc3339].)

[toddysm]

Re-reading this thread again after @SteveLasker sent it to me back in April, a few things that are not clear (at least to me) on the purpose of those annotations and how they should be used. Here is a real scenario:

As @adrosa mentioned above, there are ~70 images that have the ubuntu:bionic tag. Those are images produced every month if not more often that include updates to the Bionic version of the product (security fixes, bug fixes, etc.). So, the question that comes to mind is: "Are the org.opencontainers.image.support.* annotations for the Ubuntu Bionic product or for each individual image (like ubuntu:bionic-20220427)"?

Let's assume that those are annotations on the Bionic product. This is LTS with 5 years of support (right?), which means that every image will have annotation for EOS 2023-04-something (correct?). This means that I should feel OK to use ubuntu:bionic-20220427 until 2023-04, which is misleading and not very helpful. (I will certainly miss on many vulnerability fixes :)).

On the other hand, annotating each image like ubuntu:bionic-20220427 with such information has a certain value. Let's say for example, that after Canonical publishes ubuntu:bionic-20220531 they also add annotation to ubuntu:bionic-20220427 that it is deprecated. Then, if I am using ubuntu:bionic-20220427 (let's say in production deployment), I can check the annotation every time before deployment and if it is deprecated (or outdated) I can make a decision to report it or even block it from deployment. This is much closer to what @sudo-bmitch proposed above with shorter lifespan of the images and has a certain value.

Having a standardized way to set those annotations across registries (using the Referrers) and synching it across registries would be great because it can open a lot of opportunities for automation, policy enforcement, etc.

Thoughts?

[sudo-bmitch]

So, the question that comes to mind is: "Are the org.opencontainers.image.support.* annotations for the Ubuntu Bionic product or for each individual image (like ubuntu:bionic-20220427)"?

The annotations are set on individual manifests, and you'll want to include each platform specific manifest in a multi-platform image. Pushing an artifact with refers can be used to update an image without changing it's digest. However, you need to upload a separate artifact manifest for each image manifest because the refers field only points to a single descriptor. So if you have hundreds of images, pushing the EOS/EOL data as a separate artifact would be hundreds of artifact manifests to push and maintain. You would also need to delete the old artifact when pushing a change, and it's not clear how that would be handled by mirrors (how do they know the artifact was deleted upstream rather than the artifact was locally created for the mirror).

On the plus side with an annotation only artifact using refers, you wouldn't need to update any SBOM or signatures associated with the image since the image digest didn't change. But tooling would need to know to check that annotation only artifact, and there isn't a standard defined for that yet. Also the EOS/EOL artifact could get separated from the image when copying the image. So I think there are a number of more complex use cases not covered by the existing tooling.

For the less complex use cases, I don't see a need to hold back defining these annotations while solutions for the complex use cases are found.

[toddysm]

@sudo-bmitch Is there another thread that discusses all those use cases and how they can be handled? For example, this one:

You would also need to delete the old artifact when pushing a change, and it's not clear how that would be handled by mirrors

I had a discussion with @SteveLasker about not deleting the old one but just pushing the most recent one. Though, then the question is how the consumer of that information queries the registry for the most recent one.

[sudo-bmitch]

"The most recent one" assumes that only a single source can define an annotation only artifact. What if there are multiple sources? How would you merge them if multiple sources define the same annotation? How would you specify to delete an annotation? I worry there are too many unknowns trying to make a generic solution that extends the artifact list on a manifest.

The best I can suggest is making a new artifact type, that happens to define everything in annotations, and define the policies for that type of artifact separate from the manifest and it's annotations. I don't think new artifact types something OCI is ready to define (that's left up to the community). But if we did, that would probably need a new working group.

[johnsonshi]

Yes, with the reference types working group merged, I agree with the approach of:

• having an annotation-only artifact manifest that contains end-of-life information
• this annotation-only artifact manifest refers to an image
• the annotation-only artifact manifest should have a well-defined artifactType
• the well-defined artifactType for end-of-life annotation-only manifest should define the semantics around multiple pushes, whether to delete old annotation-only manifests, how to merge, etc.

[ValentinViennot]

On the other hand, annotating each image like ubuntu:bionic-20220427 with such information has a certain value. Let's say for example, that after Canonical publishes ubuntu:bionic-20220531 they also add annotation to ubuntu:bionic-20220427 that it is deprecated.

This solution would be the easiest, indeed.
With the reference type proposal merged, it could even refer the user to the new OCI artefact that replaces the old one?

(However, as you said, it does not answer the question, "until when is the product supported?")

[toddysm]

@ValentinViennot and I started looking at the different scenarios for using this type of metadata. It is going beyond retiring images. Would be happy to collaborate with more folks on this.

[tianon]

Just for the sake of being explicit, I have concerns about the specific annotations being proposed (essentially echoing @vbatts early in the thread; #903 (comment)); given the immutable nature of images, I'd unfortunately be a NACK on this one. 🙇

(Some of the discussion here about "what's the 'latest' ubuntu:bionic tag" has some very strong TUF vibes though, and coupling that with a way to say "this tag is no longer going to be updated" is definitely an interesting thought IMO.)

[cgwalters]

Making these annotations as first-class keys, I do not think this workflow will ultimately provide the flexibility desired. If there were an extension on support, it could not be modified without changing the image digest. If there were a change in the URL, it would change the image digest.

The thing is that in reality one needs to keep making new container images for a stable tag to e.g. pull in glibc security fixes. So given that, any "support extension" changes could easily be done as part of just...updating the image for security or other bugfixes. Could someone running a "distroless" image with a statically linked app binary inside get away with not updating it for months or even years? Maybe...but I'm pretty skeptical.

A lot of this discussion IMO got way too bike-sheddy and I'll xref #1152 (comment) here - systemd already "standardized" SUPPORT_END which from my PoV is simple and adequate as a starting point.