Skip to content

Begin adjustments on the imported specs #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vbatts merged 1 commit into from
Apr 14, 2016
Merged

Begin adjustments on the imported specs #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 108 additions & 155 deletions manifest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,124 +4,101 @@ draft = true
+++
<![end-metadata]-->

# Image Manifest Version 2, Schema 2

This document outlines the format of of the V2 image manifest, schema version 2.
The original (and provisional) image manifest for V2 (schema 1), was introduced
in the Docker daemon in the [v1.3.0
release](https://github.com/docker/docker/commit/9f482a66ab37ec396ac61ed0c00d59122ac07453)
and is specified in the [schema 1 manifest definition](./manifest-v2-1.md)
# Image Manifest

This second schema version has two primary goals. The first is to allow
multi-architecture images, through a "fat manifest" which references image
manifests for platform-specific versions of an image. The second is to
move the Docker engine towards content-addressable images, by supporting
an image model where the image's configuration can be hashed to generate
an ID for the image.
There are three main goals of the Image Manifest Specification.
The first goal is content-addressable images, by supporting an image model where the image's configuration can be hashed to generate a unique ID for the image and its components.
The second goal is to allow multi-architecture images, through a "fat manifest" which references image manifests for platform-specific versions of an image.
The third goal is to be translatable to the [OpenContainers/runtime-spec](https://github.com/opencontainers/runtime-spec)

# Media Types

The following media types are used by the manifest formats described here, and
the resources they reference:
The following media types are used by the manifest formats described here, and the resources they reference:

- `application/vnd.docker.distribution.manifest.v1+json`: schema1 (existing manifest format)
- `application/vnd.docker.distribution.manifest.v2+json`: New image manifest format (schemaVersion = 2)
- `application/vnd.docker.distribution.manifest.list.v2+json`: Manifest list, aka "fat manifest"
- `application/vnd.docker.image.rootfs.diff.tar.gzip`: "Layer", as a gzipped tar
- `application/vnd.docker.container.image.v1+json`: Container config JSON
- `application/vnd.oci.image.manifest.list.v1+json`: Manifest list, aka "fat manifest"
- `application/vnd.oci.image.manifest.v1+json`: Image manifest format
- `application/vnd.oci.image.rootfs.tar.gzip`: "Layer", as a gzipped tar archive
- `application/vnd.oci.image.serialization.v1+json`: Container config JSON

## Manifest List

The manifest list is the "fat manifest" which points to specific image manifests
for one or more platforms. Its use is optional, and relatively few images will
use one of these manifests. A client will distinguish a manifest list from an
image manifest based on the Content-Type returned in the HTTP response.
The manifest list is the "fat manifest" which points to specific image manifests for one or more platforms.
Copy link
Contributor

@stevvooe stevvooe Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can deprecate the term "fat manifest". I'm not sure if it helps the understanding here.

While the use of a manifest list is OPTIONAL for image providers, image consumers SHOULD be prepared to process them.
A client will distinguish a manifest list from an image manifest based on the Content-Type returned in the HTTP response.

## *Manifest List* Field Descriptions

- **`schemaVersion`** *int*
This field specifies the image manifest schema version as an integer. This
schema uses the version `2`.

This REQUIRED property specifies the image manifest schema version.
Copy link
Contributor

@stevvooe stevvooe Apr 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reset will make the format incompatible with Docker's manifest format. I'd rather we maintain this at 2 and deprecate the field.

Copy link
Member Author

@vbatts vbatts Apr 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k. changed.

On Fri, Apr 8, 2016 at 2:21 PM, Stephen Day [email protected]
wrote:

In manifest.md
#12 (comment)
:

Manifest List Field Descriptions

- schemaVersion int

  • This field specifies the image manifest schema version as an integer. This
  • schema uses the version 2.
  • This REQUIRED property specifies the image manifest schema version.

This reset will make the format incompatible with Docker's manifest
format. I'd rather we maintain this at 2 and deprecate the field.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/opencontainers/image-spec/pull/12/files/c146488d55d702f2c7103da26ac5a20dfc5f34c1#r59068108

Copy link
Contributor

@philips philips Apr 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need a note as to why it is at 2. Leaving it at 2 is fine by me.

Copy link
Member Author

@vbatts vbatts Apr 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps as a part of the compatibility matrix/table

On Fri, Apr 8, 2016 at 2:43 PM, Brandon Philips [email protected]
wrote:

In manifest.md
#12 (comment)
:

Manifest List Field Descriptions

- schemaVersion int

  • This field specifies the image manifest schema version as an integer. This
  • schema uses the version 2.
  • This REQUIRED property specifies the image manifest schema version.

We will need a note as to why it is at 2. Leaving it at 2 is fine by me.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/opencontainers/image-spec/pull/12/files/c146488d55d702f2c7103da26ac5a20dfc5f34c1#r59079019

This schema uses the version `2`.

- **`mediaType`** *string*

The MIME type of the manifest list. This should be set to
`application/vnd.docker.distribution.manifest.list.v2+json`.
This REQUIRED property contains the MIME type of the manifest list.
For this version of the specification, this MUST be set to `application/vnd.oci.image.manifest.list.v1+json`.

- **`manifests`** *array*

The manifests field contains a list of manifests for specific platforms.
This REQUIRED property contains a list of manifests for specific platforms.
While the property MUST be present, the size of the array MAY be zero.

Fields of a object in the manifests list are:

- **`mediaType`** *string*

The MIME type of the referenced object. This will generally be
`application/vnd.docker.image.manifest.v2+json`, but it could also
be `application/vnd.docker.image.manifest.v1+json` if the manifest
list references a legacy schema-1 manifest.

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.

- **`digest`** *string*
Fields of each object in the manifests list are:
Copy link
Collaborator

@duglin duglin Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to assume that there is no in-lining of the referenced images - meaning, we're not defining a way for someone to download a single entity that contains the fat manifest plus all referenced images. Was this intentional? I think it would be good to allow for the sharing of single entities instead of forcing people to always break things up.

Copy link
Member Author

@vbatts vbatts Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you saying that the manifests array is only used when referencing multiple objects, otherwise the other mediatypes be sub-structures of in this manifest-list document?

Copy link
Collaborator

@duglin duglin Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't really a statement about the manifests array itself, its just that this is where I was in the doc when it dawned on me that downloading this manifest json by itself feels like the main usecase being promoted - so I started to wonder about the "big bundle" case. So, should we consider creating a blob (tar) containing this manifest and all referenced images as files in there?

Copy link
Member Author

@vbatts vbatts Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the confusion around having 0 mapping to even an OPTIONAL api, like
#15

On Wed, Apr 13, 2016 at 11:03 AM, Doug Davis [email protected]
wrote:

In manifest.md
#12 (comment)
:

- - mediaType string

  •    The MIME type of the referenced object. This will generally be

  •    `application/vnd.docker.image.manifest.v2+json`, but it could also

  •    be `application/vnd.docker.image.manifest.v1+json` if the manifest

- list references a legacy schema-1 manifest.

- - size int

  •    The size in bytes of the object. This field exists so that a client

  •    will have an expected size for the content before validating. If the

  •    length of the retrieved content does not match the specified length,

- the content should not be trusted.

    • digest string
  • Fields of each object in the manifests list are:

This wasn't really a statement about the manifests array itself, its just
that this is where I was in the doc when it dawned on me that downloading
this manifest json by itself feels like the main usecase being promoted -
so I started to wonder about the "big bundle" case. So, should we consider
creating a blob (tar) containing this manifest and all referenced images as
files in there?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/opencontainers/image-spec/pull/12/files/8046ab707b00e89eb9d48b0634ecf23a7c57ef27#r59564875

Copy link
Contributor

@stevvooe stevvooe Apr 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do embedding, it should probably be with a data attribute, with base64 encoded data. There are problems with pulling the full tar up into something like that, but intermediates, like manifests and config, can see gains from this.


- **`mediaType`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).
This REQUIRED property contains the MIME type of the referenced object.
(i.e. `application/vnd.oci.image.manifest.v1+json`)

- **`platform`** *object*
- **`size`** *int*

The platform object describes the platform which the image in the
manifest runs on. A full list of valid operating system and architecture
values are listed in the [Go language documentation for `$GOOS` and
`$GOARCH`](https://golang.org/doc/install/source#environment)
This REQUIRED property specifies the size in bytes of the object.
This field exists so that a client will have an expected size for the content before validating.
If the length of the retrieved content does not match the specified length, the content should not be trusted.

- **`architecture`** *string*
- **`digest`** *string*

The architecture field specifies the CPU architecture, for example
`amd64` or `ppc64le`.
The digest of the content, as defined by the [Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).

- **`os`** *string*
- **`platform`** *object*

The os field specifies the operating system, for example
`linux` or `windows`.
This REQUIRED property describes the platform which the image in the manifest runs on.
A full list of valid operating system and architecture values are listed in the [Go language documentation for `$GOOS` and `$GOARCH`](https://golang.org/doc/install/source#environment)

- **`os.version`** *string*
- **`architecture`** *string*

The optional os.version field specifies the operating system version,
for example `10.0.10586`.
This REQUIRED property specified the CPU architecture, for example `amd64` or `ppc64le`.

- **`os.features`** *array*
- **`os`** *string*

The optional os.features field specifies an array of strings,
each listing a required OS feature (for example on Windows
`win32k`).
This REQUIRED property specifies the operating system, for example `linux` or `windows`.

- **`variant`** *string*
- **`os.version`** *string*

The optional variant field specifies a variant of the CPU, for
example `armv6l` to specify a particular CPU variant of the ARM CPU.
This optional property specifies the operating system version, for example `10.0.10586`.

- **`features`** *array*
- **`os.features`** *array*

The optional features field specifies an array of strings, each
listing a required CPU feature (for example `sse4` or `aes`).
This OPTIONAL property specifies an array of strings, each specifying a mandatory OS feature (for example on Windows `win32k`).

- **`variant`** *string*

This OPTIONAL property specifies the variant of the CPU, for example `armv6l` to specify a particular CPU variant of the ARM CPU.

- **`features`** *array*

This OPTIONAL property specifies an array of strings, each specifying a mandatory CPU feature (for example `sse4` or `aes`).

## Example Manifest List

*Example showing a simple manifest list pointing to image manifests for two platforms:*
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"mediaType": "application/vnd.oci.image.manifest.list.v1+json",
"manifests": [
{
"mediaType": "application/vnd.docker.image.manifest.v2+json",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7143,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
"platform": {
Expand All @@ -130,7 +107,7 @@ image manifest based on the Content-Type returned in the HTTP response.
}
},
{
"mediaType": "application/vnd.docker.image.manifest.v2+json",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7682,
"digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
"platform": {
Expand All @@ -147,96 +124,89 @@ image manifest based on the Content-Type returned in the HTTP response.

# Image Manifest

The image manifest provides a configuration and a set of layers for a container
image. It's the direct replacement for the schema-1 manifest.
The image manifest provides a configuration and a set of layers for a container image.

## *Image Manifest* Field Descriptions

- **`schemaVersion`** *int*
This field specifies the image manifest schema version as an integer. This
schema uses version `2`.

This REQUIRED property specifies the image manifest schema version.
This schema uses version `2`.

- **`mediaType`** *string*

The MIME type of the manifest. This should be set to
`application/vnd.docker.distribution.manifest.v2+json`.
This REQUIRED property contains the MIME type of the image manifest.
For this version of the specification, this MUST be set to `application/vnd.oci.image.manifest.v1+json`.

- **`config`** *object*

The config field references a configuration object for a container, by
digest. This configuration item is a JSON blob that the runtime uses
to set up the container. This new schema uses a tweaked version
of this configuration to allow image content-addressability on the
daemon side.
The config field references a configuration object for a container, by digest.
This configuration item is a JSON blob that the runtime uses to set up the container.
This new schema uses a tweaked version of this configuration to allow image content-addressability on the daemon side.

Fields of a config object are:

- **`mediaType`** *string*
The MIME type of the referenced object. This should generally be
`application/vnd.docker.container.image.v1+json`.

This REQUIRED property contains the MIME type of the referenced object.
(i.e. `application/vnd.oci.image.serialization.v1+json`)

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.


This REQUIRED property specifies the size in bytes of the object.
This field exists so that a client will have an expected size for the content before validating.
If the length of the retrieved content does not match the specified length, the content should not be trusted.

- **`digest`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).
The digest of the content, as defined by the [Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).

- **`layers`** *array*

The layer list is ordered starting from the base image (opposite order of schema1).

Fields of an item in the layers list are:

- **`mediaType`** *string*
The MIME type of the referenced object. This should
generally be `application/vnd.docker.image.rootfs.diff.tar.gzip`.

This REQUIRED property contains the MIME type of the referenced object.
(i.e. `application/vnd.oci.image.rootfs.tar.gzip`)

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.


This REQUIRED property specifies the size in bytes of the object.
This field exists so that a client will have an expected size for the content before validating.
If the length of the retrieved content does not match the specified length, the content should not be trusted.

- **`digest`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).
The digest of the content, as defined by the [Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).

## Example Image Manifest

*Example showing an image manifest:*
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"mediaType": "application/vnd.oci.image.serialization.v1+json",
"size": 7023,
"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.oci.image.rootfs.tar.gzip",
"size": 32654,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.oci.image.rootfs.tar.gzip",
"size": 16724,
"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.oci.image.rootfs.tar.gzip",
"size": 73109,
"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
}
Expand All @@ -246,39 +216,22 @@ image. It's the direct replacement for the schema-1 manifest.

# Backward compatibility

The registry will continue to accept uploads of manifests in both the old and
new formats.

When pushing images, clients which support the new manifest format should first
construct a manifest in the new format. If uploading this manifest fails,
presumably because the registry only supports the old format, the client may
fall back to uploading a manifest in the old format.

When pulling images, clients indicate support for this new version of the
manifest format by sending the
`application/vnd.docker.distribution.manifest.v2+json` and
`application/vnd.docker.distribution.manifest.list.v2+json` media types in an
`Accept` header when making a request to the `manifests` endpoint. Updated
clients should check the `Content-Type` header to see whether the manifest
returned from the endpoint is in the old format, or is an image manifest or
manifest list in the new format.

If the manifest being requested uses the new format, and the appropriate media
type is not present in an `Accept` header, the registry will assume that the
client cannot handle the manifest as-is, and rewrite it on the fly into the old
format. If the object that would otherwise be returned is a manifest list, the
registry will look up the appropriate manifest for the amd64 platform and
linux OS, rewrite that manifest into the old format if necessary, and return
the result to the client. If no suitable manifest is found in the manifest
list, the registry will return a 404 error.

One of the challenges in rewriting manifests to the old format is that the old
format involves an image configuration for each layer in the manifest, but the
new format only provides one image configuration. To work around this, the
registry will create synthetic image configurations for all layers except the
top layer. These image configurations will not result in runnable images on
their own, but only serve to fill in the parent chain in a compatible way.
The IDs in these synthetic configurations will be derived from hashes of their
respective blobs. The registry will create these configurations and their IDs
using the same scheme as Docker 1.10 when it creates a legacy manifest to push
to a registry which doesn't support the new format.
The registry will continue to accept uploads of manifests in both the old and new formats.
Copy link
Contributor

@stevvooe stevvooe Apr 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some thought, we may need to word this in a way that doesn't assume the existence of a registry. We can use the Accept/Allow headers to frame a fallback story, but we should talking about expressing compatibility in transport agnostic manner.

The behavior here is going to be very important to ensure we have a specification that is beneficial to the community of users. Part of the problem here is that content-addressability (content security, really), needs to be balanced against backwards compatibility and usability.

Copy link
Contributor

@philips philips Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevvooe I think we can just refer to the non-existent mime-type table: #14

Then add concrete language around handling those mime-types in the optional distribution layer: #15

Copy link
Contributor

@stevvooe stevvooe Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really need to add distribution-agnostic fallback. A mimetype table and optional distribution layer aren't really sufficient for describing fallback that will work across transports. Otherwise, we risk losing compatibility across transports.

Perhaps, this section should describe the structure of the table.

Copy link
Contributor

@philips philips Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevvooe It is a bit hard to design with a strawman of "distribution-agnostic". Do you have a concrete example?

Copy link
Contributor

@stevvooe stevvooe Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if "distribution-agnostic" or "transport-agnostic" is a "strawman". It simply means to describe mediatype fallback outside of the context of http, which may refer to Accept headers. A concrete example would be if application/vnd.oci.image.manifest.v1+json is encountered, and only application/vnd.docker.image.manifest.v2+json is supported, what should be done?

Another issue is expressing a fallback precedence. For example, one might declare they can handle application/vnd.oci.image.manifest.v1+json and application/vnd.docker.image.manifest.v2+json but only supports the latter. Both are provided. Should the client convert or fallback to the format they support natively?

Copy link
Contributor

@philips philips Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example transport in mind? I am confused what you are trying to ask for here. Perhaps we can move it to a separate issue?

Do you just want language like what is in the HTTP spec about quality factors?

Quality factors allow the user or user agent to indicate the relative degree of preference for that media-range, using the qvalue scale from 0 to 1 (section 3.9). The default value is q=1.

The example

       Accept: audio/*; q=0.2, audio/basic
SHOULD be interpreted as "I prefer audio/basic, but send me any audio type if it is the best available after an 80% mark-down in quality."

Copy link
Contributor

@stevvooe stevvooe Apr 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section discusses fallback in the context of http. Let's remove the transport-specific items and describe what was here generically.

I'm really confused as to why this is an unreasonable request for this stage of the specification process. If you believe this is outside the scope of this PR, we can file a separate issue, but let's make sure it is followed up.


When pushing images, clients which support the new manifest format should first construct a manifest in the new format.
If uploading this manifest fails, presumably because the registry only supports the old format, the client may fall back to uploading a manifest in the old format.

When pulling images, clients indicate support for this new version of the manifest format by sending the
`application/vnd.oci.image.manifest.v1+json` and
`application/vnd.oci.image.manifest.list.v1+json` media types in an `Accept` header when making a request to the `manifests` endpoint.
Updated clients should check the `Content-Type` header to see whether the manifest returned from the endpoint is in the old format, or is an image manifest or manifest list in the new format.

If the manifest being requested uses the new format, and the appropriate media type is not present in an `Accept` header, the registry will assume that the client cannot handle the manifest as-is, and rewrite it on the fly into the old format.
If the object that would otherwise be returned is a manifest list, the registry will look up the appropriate manifest for the amd64 platform and linux OS, rewrite that manifest into the old format if necessary, and return the result to the client.
If no suitable manifest is found in the manifest list, the registry will return a 404 error.

One of the challenges in rewriting manifests to the old format is that the old format involves an image configuration for each layer in the manifest, but the new format only provides one image configuration.
To work around this, the registry will create synthetic image configurations for all layers except the top layer.
These image configurations will not result in runnable images on their own, but only serve to fill in the parent chain in a compatible way.
The IDs in these synthetic configurations will be derived from hashes of their respective blobs.
The registry will create these configurations and their IDs using the same scheme as Docker 1.10 when it creates a legacy manifest to push to a registry which doesn't support the new format.
Loading