[bitnami/cosign] Docker Content Trust fails for cosign versions > 1

[ChristopherZellermann]

Name and Version

bitnami/cosign:2

What architecture are you using?

amd64

What steps will reproduce the bug?

Activate Docker Content Trust

export DOCKER_CONTENT_TRUST=1

Try to pull recent bitnami/cosign versions

docker pull bitnami/cosign:2.0.2
docker pull bitnami/cosign:2

What is the expected behavior?

Pull the bitnami/cosign:2.0.2 image

What do you see instead?

No valid trust data for 2.0.2
No valid trust data for 2

Additional information

It works with older versions

docker pull bitnami/cosign:1
Pull (1 of 1): bitnami/cosign:1@sha256:b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256
docker.io/bitnami/cosign@sha256:b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256: Pulling from bitnami/cosign
Digest: sha256:b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256
Status: Image is up to date for bitnami/cosign@sha256:b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256
Tagging bitnami/cosign@sha256:b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256 as bitnami/cosign:1
docker.io/bitnami/cosign:1

Note that the 'latest' tag is still referencing a v1 version of cosign.

[carrodher]

Thanks for reporting this issue. We've been investigating this issue and it seems something related to multi-arch (ARM and AMD) and Docker CLI.

According to our findings, it seems the docker client when inspecting the signatures for a multi-arch image, does not seems to know which architecture to pull for inspection, for example:

$ docker trust inspect --pretty bitnami/cosign:2

No signatures for bitnami/cosign:2

Administrative keys for bitnami/cosign:2

  Repository Key:	56315934a0b27dd0a9425979132551cf5a3cae16b7f1d1888ccc088948c4fbab
  Root Key:	0c70c2c70c46c85abcdf140585e6a4dbde9d9f3a4d6540a8d7c05d9ea4d7c494

But if we request the signature of one of the two architectures by its sha256. In this case, amd:

$ docker trust inspect --pretty bitnami/cosign@sha256:4ba139936195738785b7b5bf9c54f6c8000fe6e55cf061cd6ba85a9ef5c9f6ec

Signatures for bitnami/cosign@sha256:4ba139936195738785b7b5bf9c54f6c8000fe6e55cf061cd6ba85a9ef5c9f6ec

SIGNED TAG             DIGEST                                                             SIGNERS
1                      b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256   (Repo Admin)
1-debian-10            458d37103a3bce6ffa5fb5bf45b5ff581399f959b39eb5360e268dca2cea99da   (Repo Admin)
1-debian-11            b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256   (Repo Admin)
....... WAY MORE TAGS .......
1.13.1-debian-11-r40   3bfc24807dfef64c948553e9024c6dc2aa53b8db4c1982efd61d6a319713555e   (Repo Admin)
1.13.1-debian-11-r41   10493b6530ad863e8a1bf4725991d488e86fca7161e8a0ad060d91d259dd65e8   (Repo Admin)
1.13.1-debian-11-r42   cfae3edc84f6ef13efb876d5c4010f014d821f750befa1977019523d235f2ca1   (Repo Admin)
1.13.1-debian-11-r43   b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256   (Repo Admin)
latest                 b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256   (Repo Admin)

Administrative keys for bitnami/cosign@sha256:4ba139936195738785b7b5bf9c54f6c8000fe6e55cf061cd6ba85a9ef5c9f6ec

  Repository Key:	56315934a0b27dd0a9425979132551cf5a3cae16b7f1d1888ccc088948c4fbab
  Root Key:	0c70c2c70c46c85abcdf140585e6a4dbde9d9f3a4d6540a8d7c05d9ea4d7c494

we see all are signed and the same happens with arm, so it seems that the manifest index is indeed not signed and that’s the one being retrieved by the tag.

In the end, it seems an issue related to Docker CLI and multi-arch images (whose support was recently added to the Bitnami containers), there are already issues created for this topic, for instance, docker/buildx#313

[ChristopherZellermann]

Thank you for the quick reply.

What I don't understand is why the docker trust inspect command finds signatures for tags like 1.13.1 but not 2

$ docker trust inspect --pretty bitnami/cosign:1.13.1

Signatures for bitnami/cosign:1.13.1

SIGNED TAG   DIGEST                                                             SIGNERS
1.13.1       b31c7f4419c8793a90100424204be9b795d814a6c77b5cb2211410dac440c256   (Repo Admin)

Administrative keys for bitnami/cosign:1.13.1

  Repository Key:	56315934a0b27dd0a9425979132551cf5a3cae16b7f1d1888ccc088948c4fbab
  Root Key:	0c70c2c70c46c85abcdf140585e6a4dbde9d9f3a4d6540a8d7c05d9ea4d7c494

vs

$ docker trust inspect --pretty bitnami/cosign:2

No signatures for bitnami/cosign:2


Administrative keys for bitnami/cosign:2

  Repository Key:	56315934a0b27dd0a9425979132551cf5a3cae16b7f1d1888ccc088948c4fbab
  Root Key:	0c70c2c70c46c85abcdf140585e6a4dbde9d9f3a4d6540a8d7c05d9ea4d7c494

Unless I am mistaken, bitnami/cosign:1.13.1 and bitnami/cosign:2 are both multi-arch images (both exists for amd64 and arm64).

[carrodher]

1.13.1 is a rolling tag which is pointing to multi-arch immutable tags in the latest versions (for instance 1.13.1-debian-11-r44 or 1.13.1-debian-11-r67) but non-multi-arch in previous versions (for instance 1.13.1-debian-11-r0 or 1.13.1-debian-11-r43)

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.