Cache can't be exported to Quay.io

[mbarbero]

Similar to #1143

When exporting cache to quay.io, it fails with

error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:c2aba47e903ef19d459785c7e5750ef7da0f6f86657d9b40c329d5268dfe2185": unexpected status: 401 Unauthorized

The error is the same with both modes: mode=max or mode=min

buildctl" build \
    --progress=plain \
    --frontend=dockerfile.v0 \
    --local context="${context}" \
    --local dockerfile="$(dirname "${dockerfile}")" \
    --opt filename="$(basename "${dockerfile}")" \
    --output "type=image,\"name=${name}\",push=${push}" \
    --export-cache "type=registry,mode=max,ref=${image}:${tag}-buildcache" \
    --import-cache "type=registry,ref=${image}:${tag}-buildcache" \
    "${@}"

When I do not --export-cache, images are pushed properly to quay.io so the credentials are correct.

[mbarbero]

I should add that the cache export works OK with the docker.io registry.

[cpuguy83]

I have not done any real investigation here, but this sounds like quay does not support cache manifests.
You may want to try an inline cache? This may still not work. I know I've had issues with some other registries with it.

[mbarbero]

You may want to try an inline cache? This may still not work. I know I've had issues with some other registries with it.

Issue is that inline cache does not support max mode, which is a requirement in my case.

[cpuguy83]

Speaking with someone else, they are having trouble pushing 2 tags... e.g. -t foo -t bar, getting the same 401 error.
Maybe it's not even a cache manifest issue, and the 401 is legit... like not authorized to push 2 tags with the same auth token?

[mbarbero]

I've tried to push 2 tags at the same time with a cache type=local and it works. So the issue is not about quay not supporting pushing 2 tags at once.

[tonistiigi]

#1550

[mbarbero]

Coming back at this issue as using quay.io may be something we're interested in again. I tried with latest buildkit (0.8.2), and I still cannot export cache to quay. Both

--export-cache "type=registry,mode=max,oci-mediatypes=false,ref=${image}:${tag}-buildcache"

and

--export-cache "type=registry,mode=min,oci-mediatypes=false,ref=${image}:${tag}-buildcache" \

lead to 400 BAD REQUEST

> exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:d0227d38f60d9887de4933d638b4ac36538d320cd9ea6b02e5b87c6ef34b786b": unexpected status: 400 BAD REQUEST

while

--export-cache "type=registry,mode=max,oci-mediatypes=true,ref=${image}:${tag}-buildcache"

and

--export-cache "type=registry,mode=min,oci-mediatypes=true,ref=${image}:${tag}-buildcache" \

both lead to 415 UNSUPPORTED MEDIA TYPE

> exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:d0227d38f60d9887de4933d638b4ac36538d320cd9ea6b02e5b87c6ef34b786b": unexpected status: 415 UNSUPPORTED MEDIA TYPE

So it seems that the OCI mediatypes support did not fix this issue. Any idea?

[jedevc]

We can reproduce this internally, using oci-mediatypes=true on the cache (which is the default) does not solve the issue.

BuildKit's debug logs print out the whole message from quay.io:

{
  "errors": [
    {
      "code": "MANIFEST_INVALID",
      "detail": {
        "message": "failed to parse manifest: manifest data does not match schema: 'platform' is a required property\n\nFailed validating 'required' in schema['properties']['manifests']['items']:\n    {'properties': {'annotations': {'additionalProperties': True,\n                                    'description': 'The annotations, if '\n                                                   'any, on this '\n                                                   'descriptor',\n                                    'type': 'object'},\n                    'digest': {'description': 'The content addressable '\n                                              'digest of the manifest in '\n                                              'the blob store',\n                               'type': 'string'},\n                    'mediaType': {'description': 'The MIME type of the '\n                                                 'referenced manifest',\n                                  'enum': ['application/vnd.oci.image.manifest.v1+json',\n                                           'application/vnd.oci.image.index.v1+json'],\n                                  'type': 'string'},\n                    'platform': {'description': 'The platform object '\n                                                'describes the platform '\n                                                'which the image in the '\n                                                'manifest runs on',\n                                 'properties': {'architecture': {'description': 'Specifies '\n                                                                                'the '\n                                                                                'CPU '\n                                                                                'architecture, '\n                                                                                'for '\n                                                                                'example '\n                                                                                'amd64 '\n                                                                                'or '\n                                                                                'ppc64le.',\n                                                                 'type': 'string'},\n                                                'features': {'description': 'specifies '\n                                                                            'an '\n                                                                            'array '\n                                                                            'of '\n                                                                            'strings, '\n                                                                            'each '\n                                                                            'listing '\n                                                                            'a '\n                                                                            'required '\n                                                                            'CPU '\n                                                                            'feature '\n                                                                            '(for '\n                                                                            'example '\n                                                                            'sse4 '\n                                                                            'or '\n                                                                            'aes).',\n                                                             'items': {'type': 'string'},\n                                                             'type': 'array'},\n                                                'os': {'description': 'Specifies '\n                                                                      'the '\n                                                                      'operating '\n                                                                      'system, '\n                                                                      'for '\n                                                                      'example '\n                                                                      'linux '\n                                                                      'or '\n                                                                      'windows',\n                                                       'type': 'string'},\n                                                'os.features': {'description': 'specifies '\n                                                                               'an '\n                                                                               'array '\n                                                                               'of '\n                                                                               'strings, '\n                                                                               'each '\n                                                                               'listing '\n                                                                               'a '\n                                                                               'required '\n                                                                               'OS '\n                                                                               'feature '\n                                                                               '(for '\n                                                                               'example '\n                                                                               'on '\n                                                                               'Windows '\n                                                                               'win32k)',\n                                                                'items': {'type': 'string'},\n                                                                'type': 'array'},\n                                                'os.version': {'description': 'Specifies '\n                                                                              'the '\n                                                                              'operating '\n                                                                              'system '\n                                                                              'version, '\n                                                                              'for '\n                                                                              'example '\n                                                                              '10.0.10586',\n                                                               'type': 'string'},\n                                                'variant': {'description': 'Specifies '\n                                                                           'a '\n                                                                           'variant '\n                                                                           'of '\n                                                                           'the '\n                                                                           'CPU, '\n                                                                           'for '\n                                                                           'example '\n                                                                           'armv6l '\n                                                                           'to '\n                                                                           'specify '\n                                                                           'a '\n                                                                           'particular '\n                                                                           'CPU '\n                                                                           'variant '\n                                                                           'of '\n                                                                           'the '\n                                                                           'ARM '\n                                                                           'CPU',\n                                                            'type': 'string'}},\n                                 'required': ['architecture', 'os'],\n                                 'type': 'object'},\n                    'size': {'description': 'The size in bytes of the '\n                                            'object. This field exists so '\n                                            'that a client will have an '\n                                            'expected size for the content '\n                                            'before validating. If the '\n                                            'length of the retrieved '\n                                            'content does not match the '\n                                            'specified length, the content '\n                                            'should not be trusted.',\n                             'type': 'number'},\n                    'urls': {'description': 'This OPTIONAL property '\n                                            'specifies a list of URIs from '\n                                            'which this object MAY be '\n                                            'downloaded. Each entry MUST '\n                                            'conform to RFC 3986. Entries '\n                                            'SHOULD use the http and https '\n                                            'schemes, as defined in RFC '\n                                            '7230.',\n                             'items': {'type': 'string'},\n                             'type': 'array'}},\n     'required': ['mediaType', 'size', 'digest', 'platform'],\n     'type': 'object'}\n\nOn instance['manifests'][0]:\n    {'annotations': {'buildkit/createdat': '2022-11-04T14:23:52.088237395Z',\n                     'containerd.io/uncompressed': 'sha256:e51020b79aa31ba0068429ef6d68b917b5d953ee4c8c0c860599a89a13292377'},\n     'digest': 'sha256:0a6daaf7fd60ecab61c0767d25a027d62a655f29209e0e6add18ea4131f9416f',\n     'mediaType': 'application/vnd.oci.image.layer.v1.tar+gzip',\n     'size': 27903893}"
      },
      "message": "manifest invalid"
    }
  ]
}

Unpacking the schema we get:

{
  "properties": {
    "annotations": {
      "additionalProperties": true,
      "description": "The annotations, if any, on this descriptor",
      "type": "object"
    },
    "digest": {
      "description": "The content addressable digest of the manifest in the blob store",
      "type": "string"
    },
    "mediaType": {
      "description": "The MIME type of the referenced manifest",
      "enum": [
        "application/vnd.oci.image.manifest.v1+json",
        "application/vnd.oci.image.index.v1+json"
      ],
      "type": "string"
    },
    "platform": {
      "description": "The platform object describes the platform which the image in the manifest runs on",
      "properties": {
        "architecture": {
          "description": "Specifies the CPU architecture, for example amd64 or ppc64le.",
          "type": "string"
        },
        "features": {
          "description": "specifies an array of strings, each listing a required CPU feature (for example sse4 or aes).",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "os": {
          "description": "Specifies the operating system, for example linux or windows",
          "type": "string"
        },
        "os.features": {
          "description": "specifies an array of strings, each listing a required OS feature (for example on Windows win32k)",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "os.version": {
          "description": "Specifies the operating system version, for example 10.0.10586",
          "type": "string"
        },
        "variant": {
          "description": "Specifies a variant of the CPU, for example armv6l to specify a particular CPU variant of the ARM CPU",
          "type": "string"
        }
      },
      "required": [
        "architecture",
        "os"
      ],
      "type": "object"
    },
    "size": {
      "description": "The size in bytes of the object. This field exists so that a client will have an expected size for the content before validating. If the length of the retrieved content does not match the specified length, the content should not be trusted.",
      "type": "number"
    },
    "urls": {
      "description": "This OPTIONAL property specifies a list of URIs from which this object MAY be downloaded. Each entry MUST conform to RFC 3986. Entries SHOULD use the http and https schemes, as defined in RFC 7230.",
      "items": {
        "type": "string"
      },
      "type": "array"
    }
  },
  "required": [
    "mediaType",
    "size",
    "digest",
    "platform"
  ],
  "type": "object"
}

Which fails on instance['manifests'][0]:

{
  "annotations": {
    "buildkit/createdat": "2022-11-04T14:23:52.088237395Z",
    "containerd.io/uncompressed": "sha256:e51020b79aa31ba0068429ef6d68b917b5d953ee4c8c0c860599a89a13292377"
  },
  "digest": "sha256:0a6daaf7fd60ecab61c0767d25a027d62a655f29209e0e6add18ea4131f9416f",
  "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
  "size": 27903893
}

Quay is treating the platform field as required, which the OCI image-spec marks as optional. It looks like there's something wrong with Quay's OCI implementation, if someone wants to follow that up with upstream I think that would be the right way forwards.

Alternatively, for a buildkit-side fix, we could also take the same approach we take for attestation manifests, and explicitly attach unknown/unknown OS/Architecture onto descriptors.

WDYT @crazy-max @tonistiigi?