Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
284 changes: 284 additions & 0 deletions manifest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
<!--[metadata]>
+++
draft = true
+++
<![end-metadata]-->

# Image Manifest Version 2, Schema 2

This document outlines the format of of the V2 image manifest, schema version 2.
The original (and provisional) image manifest for V2 (schema 1), was introduced
in the Docker daemon in the [v1.3.0
release](https://github.com/docker/docker/commit/9f482a66ab37ec396ac61ed0c00d59122ac07453)
and is specified in the [schema 1 manifest definition](./manifest-v2-1.md)

This second schema version has two primary goals. The first is to allow
multi-architecture images, through a "fat manifest" which references image
manifests for platform-specific versions of an image. The second is to
move the Docker engine towards content-addressable images, by supporting
an image model where the image's configuration can be hashed to generate
an ID for the image.

# Media Types

The following media types are used by the manifest formats described here, and
the resources they reference:

- `application/vnd.docker.distribution.manifest.v1+json`: schema1 (existing manifest format)
- `application/vnd.docker.distribution.manifest.v2+json`: New image manifest format (schemaVersion = 2)
- `application/vnd.docker.distribution.manifest.list.v2+json`: Manifest list, aka "fat manifest"
- `application/vnd.docker.image.rootfs.diff.tar.gzip`: "Layer", as a gzipped tar
- `application/vnd.docker.container.image.v1+json`: Container config JSON

## Manifest List

The manifest list is the "fat manifest" which points to specific image manifests
for one or more platforms. Its use is optional, and relatively few images will
use one of these manifests. A client will distinguish a manifest list from an
image manifest based on the Content-Type returned in the HTTP response.

## *Manifest List* Field Descriptions

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why italics for Manifest List? (I might expect code font)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much of this is just carry over styling that has yet to be visited.


- **`schemaVersion`** *int*

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

personally I find inline YAML + comments is more readable than markdown bullets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brendandburns example?

Personally I am OK as-is and we can reformat later.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider making this a string with proper versioning semantics since I suspect we may have minor releases that are fully backwards compatible while major releases don't necessarily have to be backwards compatible. Just a single int doesn't allow us to convey that info.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duglin This already represents the minor version of the spec. Major version is represented within the mediatype. Typically, unless explicitly identified, differing mediatypes are considered incompatible, while a revision here indicate the inclusion of new fields.

Generally, you want to avoid having a manifest know too much about its version. This allows the mediatype to act like a "lense" into the interpretation of a manifest.


This field specifies the image manifest schema version as an integer. This
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we go the appc or ociruntime route (acVersion, ociVersion) - this allows it to double as a magic marker.

Also +1 on using a SemVer string

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this raises the question that now ociVersion is unambiguous in the runtime-specs project.. perhaps one of you maintainers @philips @vbatts can fix that? ;-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonboulle Would it be possible to maintain both? This can be fixed from the perspective of docker interpretation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I'm still churning on the the schemaVersion carried over here, but it definitely has me reconsidering the ociVersion in runtime-spec, and how that file is distinguished (should it now have a mediatype?). ociVersion is not really clear in this regard.

schema uses the version `2`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'2' ? Even though the MIME types use "v1" ? Seems kind of confusing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, I am fine starting at v2 in the mime types

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh. I hadn't gotten to all these references yet, and figured starting the mime type at 2 would be confusing for a v1.0 release :-)


- **`mediaType`** *string*

The MIME type of the manifest list. This should be set to
`application/vnd.docker.distribution.manifest.list.v2+json`.

- **`manifests`** *array*

The manifests field contains a list of manifests for specific platforms.

Fields of a object in the manifests list are:

- **`mediaType`** *string*

The MIME type of the referenced object. This will generally be
`application/vnd.docker.image.manifest.v2+json`, but it could also
be `application/vnd.docker.image.manifest.v1+json` if the manifest
list references a legacy schema-1 manifest.

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.

- **`digest`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rather define this within the scope of the spec

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, we can file an issue and leave this as-is for now, I think.


- **`platform`** *object*

The platform object describes the platform which the image in the
manifest runs on. A full list of valid operating system and architecture
values are listed in the [Go language documentation for `$GOOS` and
`$GOARCH`](https://golang.org/doc/install/source#environment)

- **`architecture`** *string*

The architecture field specifies the CPU architecture, for example
`amd64` or `ppc64le`.

- **`os`** *string*

The os field specifies the operating system, for example
`linux` or `windows`.

- **`os.version`** *string*

The optional os.version field specifies the operating system version,
for example `10.0.10586`.

- **`os.features`** *array*

The optional os.features field specifies an array of strings,
each listing a required OS feature (for example on Windows
`win32k`).

- **`variant`** *string*

The optional variant field specifies a variant of the CPU, for
example `armv6l` to specify a particular CPU variant of the ARM CPU.

- **`features`** *array*

The optional features field specifies an array of strings, each
listing a required CPU feature (for example `sse4` or `aes`).

## Example Manifest List

*Example showing a simple manifest list pointing to image manifests for two platforms:*
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.image.manifest.v2+json",
"size": 7143,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
"platform": {
"architecture": "ppc64le",
"os": "linux",
}
},
{
"mediaType": "application/vnd.docker.image.manifest.v2+json",
"size": 7682,
"digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270",
"platform": {
"architecture": "amd64",
"os": "linux",
"features": [
"sse4"
]
}
}
]
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the complete example, it helps see everything all in one shot.
Couple of questions:

  • should there be some kind of URI pointing to where the referenced image can be downloaded from? (optional?)
  • should we allow for referenced images to be "in-lined"? It would be really nice if we could "export" a fat-manifest and share it by just passing around a single file instead of a set of them. People will probably end up tar'ing them up anyway, so we might as well formalize it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed we need to discuss how we find the referenced images using a discovery mechanism. I think this will be a longer discussion. Please file an issue @duglin.

Currently distribution is out of scope for the image spec. But, this is something the TOB needs to look at fixing soon. I think we should skip this discussion on the initial import pass.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be some kind of URI pointing to where the referenced image can be downloaded from? (optional?)

This defeats the purpose of providing content addresses. Pushing down the fetch content also over-couples the description of the image from the distribution of the image.

should we allow for referenced images to be "in-lined"? It would be really nice if we could "export" a fat-manifest and share it by just passing around a single file instead of a set of them. People will probably end up tar'ing them up anyway, so we might as well formalize it.

This was considered. However, it burdens verification of the "in-lined" manifest. The identity becomes dependent on the ability to re-serialize the embedded format. Re-serialization should be avoided at all costs in a robust content-addressable system.

```

# Image Manifest

The image manifest provides a configuration and a set of layers for a container
image. It's the direct replacement for the schema-1 manifest.

## *Image Manifest* Field Descriptions

- **`schemaVersion`** *int*
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like above, let's make this a string (x.y).


This field specifies the image manifest schema version as an integer. This
schema uses version `2`.

- **`mediaType`** *string*

The MIME type of the manifest. This should be set to
`application/vnd.docker.distribution.manifest.v2+json`.

- **`config`** *object*

The config field references a configuration object for a container, by
digest. This configuration item is a JSON blob that the runtime uses
to set up the container. This new schema uses a tweaked version
of this configuration to allow image content-addressability on the
daemon side.

Fields of a config object are:

- **`mediaType`** *string*

The MIME type of the referenced object. This should generally be
`application/vnd.docker.container.image.v1+json`.

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.

- **`digest`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).

- **`layers`** *array*

The layer list is ordered starting from the base image (opposite order of schema1).

Fields of an item in the layers list are:

- **`mediaType`** *string*

The MIME type of the referenced object. This should
generally be `application/vnd.docker.image.rootfs.diff.tar.gzip`.

- **`size`** *int*

The size in bytes of the object. This field exists so that a client
will have an expected size for the content before validating. If the
length of the retrieved content does not match the specified length,
the content should not be trusted.

- **`digest`** *string*

The digest of the content, as defined by the
[Registry V2 HTTP API Specificiation](https://docs.docker.com/registry/spec/api/#digest-parameter).

## Example Image Manifest

*Example showing an image manifest:*
```json
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 7023,
"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32654,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 16724,
"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 73109,
"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
}
],
}
```

# Backward compatibility
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably makes sense to drop this section?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean we intend to drop compatibility?


The registry will continue to accept uploads of manifests in both the old and
new formats.

When pushing images, clients which support the new manifest format should first
construct a manifest in the new format. If uploading this manifest fails,
presumably because the registry only supports the old format, the client may
fall back to uploading a manifest in the old format.

When pulling images, clients indicate support for this new version of the
manifest format by sending the
`application/vnd.docker.distribution.manifest.v2+json` and
`application/vnd.docker.distribution.manifest.list.v2+json` media types in an
`Accept` header when making a request to the `manifests` endpoint. Updated
clients should check the `Content-Type` header to see whether the manifest
returned from the endpoint is in the old format, or is an image manifest or
manifest list in the new format.

If the manifest being requested uses the new format, and the appropriate media
type is not present in an `Accept` header, the registry will assume that the
client cannot handle the manifest as-is, and rewrite it on the fly into the old
format. If the object that would otherwise be returned is a manifest list, the
registry will look up the appropriate manifest for the amd64 platform and
linux OS, rewrite that manifest into the old format if necessary, and return
the result to the client. If no suitable manifest is found in the manifest
list, the registry will return a 404 error.

One of the challenges in rewriting manifests to the old format is that the old
format involves an image configuration for each layer in the manifest, but the
new format only provides one image configuration. To work around this, the
registry will create synthetic image configurations for all layers except the
top layer. These image configurations will not result in runnable images on
their own, but only serve to fill in the parent chain in a compatible way.
The IDs in these synthetic configurations will be derived from hashes of their
respective blobs. The registry will create these configurations and their IDs
using the same scheme as Docker 1.10 when it creates a legacy manifest to push
to a registry which doesn't support the new format.
Loading