"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

MikeMcC399 · 2024-06-04T09:33:20Z

Issue

The CircleCI workflow job "Push Factory Image" is failing to get the status of cypress/factory:4.0.1 from Docker Hub cypress/factory.

The error is

message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',

Examining the manifest of cypress/factory:4.0.1 with the following command shows that it differs in format from the manifest for cypress/factory:4.0.0 (and from cypress/base, cypress/browsers and cypress/included)

docker manifest inspect cypress/factory:4.0.1

shows

   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",

instead of

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

Logs

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1959/workflows/3c12d584-8378-4f1d-8722-bcc504b21e2d/jobs/59620 from PR

shows

npm WARN exec The following package was not found and will be installed: [email protected]
npm WARN deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Glob versions prior to v9 are no longer supported
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
got an error other than image not found
RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest
    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)
    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)
    at Object.onceWrapper (node:events:632:28)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  jse_shortmsg: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  jse_info: {},
  message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  [stack]: 'InvalidContentError: unsupported schema version 2 in cypress/factory:4.0.1 manifest\n' +
    '    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)\n' +
    '    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)\n' +
    '    at Object.onceWrapper (node:events:632:28)\n' +
    '    at IncomingMessage.emit (node:events:530:35)\n' +
    '    at endReadableNT (node:internal/streams/readable:1696:12)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)',
  body: {
    code: 'InvalidContent',
    message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest'
  },
  restCode: 'InvalidContent',
  cause: <ref *1> [Function: we_cause] {
    [length]: 1,
    [name]: 'we_cause',
    [arguments]: null,
    [caller]: null,
    [prototype]: { [constructor]: [Circular *1] }
  },
  name: 'InvalidContentError',
  statusCode: 400
}
exiting with code 1
Docker hub has image cypress/factory:4.0.1 or not responding
We should stop in this case

Analysis

cypress/factory:4.0.1 was published through PR fix: remove obsolete environment variable npm_config_unsafe_perm #1078

Manifest

cypress/factory:4.0.1

$ docker manifest inspect cypress/factory:4.0.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:bbf8a2e6fda64adb6c87e9793493240667e9e3a43874e71563f5963076942b85",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:7cfe1c11128c9e4a119b7ad3a6bb4ce13a705cc8f00436f4174a964e102fe4fd",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:936a49c206ffcdfeb0faed320876c164b671f862cd97e815a8cfa9ba3c61eff6",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:fdb5a0baf2cb8b69d44a4d435b0ff6fdd1b6877de919cf0a74bc4aae0c93a4e1",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

cypress/factory:4.0.0

$ docker manifest inspect cypress/factory:4.0.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:7a9498698dc9ceacb1a3e93cc602436f6e135c734a701681ef0fe7304e2cc601",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:abbafb58e45003f37c1e2a0466c36af8707917a7b682eed9546ed928b2ceeec3",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Log Push Factory Image for 4.0.1

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1944/workflows/d32dcb15-f148-4cf7-a4d1-3b1f48ab6141/jobs/59206

Check for existence

npm WARN exec The following package was not found and will be installed: [email protected]
npm WARN deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated [email protected]: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated [email protected]: Glob versions prior to v9 are no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0
Docker hub says image cypress/factory:4.0.1 does not exist

building Docker image for target factory

#14 exporting to image
#14 exporting manifest sha256:265f8134ce492f9219dd8c71ff531f33fa38e2e4254873e981ef94bba7427103 0.0s done
#14 exporting config sha256:a8eb011e2c5ca149e4f573a8b902f10596a378dbcebc94eb67ea1a95a21911fa 0.0s done
#14 exporting manifest sha256:206ac86fb72641f54d01c8068bafa8e216169cd4ce433aa408de646176e885eb 0.0s done
#14 exporting config sha256:de63602eb3ffcd58ce5dbde4fa501245b01748e3e17b91a78406557327465997 0.0s done
#14 exporting manifest list sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.0s done
#14 pushing layers
#14 pushing layers 5.4s done
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.5s done
#14 pushing layers 0.2s done
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.3s done
#14 DONE 25.3s

The text was updated successfully, but these errors were encountered:

MikeMcC399 · 2024-06-04T09:41:15Z

@jennifer-shehane

I don't understand why this issue has occurred and it is in an area where I have no experience! I can't tell if was a CircleCI glitch or something else.

If it is not addressed it will prevent any new cypress/factory images from being published.

Depending on the root cause it may also affect the automatic publication of cypress/base, cypress/browsers and cypress/included images.

MikeMcC399 · 2024-06-04T13:09:08Z

There is a possibility that the image cypress/factory:4.0.1 is corrupted through 3 parallel-running PRs which each checked if there was an image cypress/factory:4.0.1 present, found there was not, and then each built one and published it:

fix: remove obsolete environment variable npm_config_unsafe_perm #1078
Merged May 31, 2024 4:06 PM GMT+2 Built cypress/factory:4.0.1
ci: update circleci runner #1079
Merged May 31, 2024 4:07 PM GMT+2 Built cypress/factory:4.0.1
docs: update contributing image size section #1080
Merged May 31, 2024 4:10 PM GMT+2 Built cypress/factory:4.0.1

The 2nd of the PRs #1079 also effectively bumped the Docker client version from Docker 20.10.18 to 26.0.2. This might have all been too much in one go!

Perhaps cypress/factory:4.0.1 should be manually deleted so that it can be recreated?

MikeMcC399 · 2024-06-04T14:24:19Z

It seems there were some breaking changes in https://github.com/docker/buildx

We may need to revert PR ci: update circleci runner #1079

MikeMcC399 · 2024-06-04T15:06:25Z

I don't see any quick and reliable fix, so I have submitted PR Revert "ci: update circleci runner (#1079)" #1094

After this is merged it will require a manual delete of the incompatible cypress/factory:4.0.1 image

jennifer-shehane · 2024-06-04T15:58:31Z

K, let me know if that works and if the deletion is required.

MikeMcC399 · 2024-06-04T16:12:32Z

@jennifer-shehane

Sorry that a revert was necessary, but at least only one image was caught up in this. I only noticed by accident when I was looking into another failure, as the workflow itself does not return an error. It's only visible when you examine the logs manually.

Status

PR Revert "ci: update circleci runner (#1079)" #1094 has merged successfully
https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1967/workflows/63050ae3-636c-46a2-a570-c5a14187d2e1/jobs/59867 is showing success

BUT in "Check if image for factory exists or Docker hub does not respond" the incompatibility shows up as expected:

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1967/workflows/63050ae3-636c-46a2-a570-c5a14187d2e1/jobs/59867/parallel-runs/0/steps/0-103?invite=true#step-103-1209_95

RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest

So, yes please, deletion of cypress/factory:4.0.1 on Docker Hub is needed. I don't think it checks on AWS, so I'm guessing that it will just overwrite there.

jennifer-shehane · 2024-06-04T16:48:07Z

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

MikeMcC399 · 2024-06-04T16:58:34Z

@jennifer-shehane

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

Thank you! That should be good, and the next merge to master will test that out.

In the meantime I checked by hand and docker-image-not-found is happy again! 😃

$ docker-image-not-found --repo cypress/factory:4.0.1
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0

It does however still fail when no tag is defined, which defaults to latest, and latest is still linked to the problematic manifest. I expect that will get overwritten at the next release. We'll see!

MikeMcC399 · 2024-06-04T17:10:49Z

Closing as resolved. I will be monitoring what happens next!

Edit:

PR chore: release cypress 13.11.0 related docker images and update factory #1097 merged and released images successfully

MikeMcC399 added the bug label Jun 4, 2024

MikeMcC399 self-assigned this Jun 4, 2024

MikeMcC399 mentioned this issue Jun 4, 2024

Revert "ci: update circleci runner (#1079)" #1094

Merged

MikeMcC399 closed this as completed Jun 4, 2024

MikeMcC399 mentioned this issue Jun 4, 2024

Migrate to current Docker version #1095

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024 •

edited

Loading

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

jennifer-shehane commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

jennifer-shehane commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024 •

edited

Loading

"Push Factory Image" failure checking cypress/factory:4.0.1 #1093

"Push Factory Image" failure checking cypress/factory:4.0.1 #1093

Comments

MikeMcC399 commented Jun 4, 2024

Issue

Logs

Analysis

Manifest

cypress/factory:4.0.1

cypress/factory:4.0.0

Log Push Factory Image for 4.0.1

Check for existence

building Docker image for target factory

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024 • edited Loading

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

jennifer-shehane commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

Status

jennifer-shehane commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024

MikeMcC399 commented Jun 4, 2024 • edited Loading

"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

MikeMcC399 commented Jun 4, 2024 •

edited

Loading

MikeMcC399 commented Jun 4, 2024 •

edited

Loading