Default image output in buildx v0.10 cannot be pushed to Yandex Cloud Container Registry

[prvshvnth]

exporting to image
exporting manifest sha256:633baadaa74008615f4fd0d19eba63a696a847c6ec3b7058a883204418b68192 0.0s done
exporting config sha256:77db7689ffdb022a9a9d03ccf73f86aa17f2c4ecd7103e6440c5495df7fc20f6 done
exporting attestation manifest sha256:4d8160c1d02c9276d0a44f813f9ffe7006754e7b160da44c716b956d43b97a53 0.0s done exporting manifest list sha256:c16a5d9f9dabeaf910344ed6a032bc53841b88dc44a51869ff94204ae53488e done
pushing layers
pushing layers 14.4s done
pushing manifest for registry.url/namspace/image:bb8432e8cce2b213e21251732c7d3edcb7c58d3@sha256:c16a5d99dabeaf910344ed6a032bc53841b838dc44a51869ff94204ae53488e 0.1s done
ERROR: failed to push registry.url/namspace/image:bb8432e8cce2b213e21251732c7d3edcb7cb58d3: failed commit on ref "manifest-sha256:633baadaa74008615f4fd0d19eba63a696a847c6ec3b7058a883204418b68": unexpected status: 400 Bad Request

buildx failed with: ERROR: failed to solve: failed to push <registry_url>/<namespace>/<imagename>:341aa798b8365346c3e32b2024bb62d99652f4a6: failed commit on ref "manifest-sha256:d6d1f20cd9c061daec67cf71af16544bae42f4f1652bb0771d78c6ad8cc8b336": unexpected status: 400 Bad Request

Below are the steps:

  - name: Checkout repository
    uses: actions/checkout@v3
  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v2
  - name: Login to DockerHub
    uses: docker/login-action@v2
    with:
      registry: ${{ env.DOCKER_REGISTRY }}
      username: ${{ secrets.DOCKER_USERNAME }}
      password: ${{ secrets.DOCKER_PASSWORD }}
  - name: Build and push
    uses: docker/build-push-action@v3
    with:
      context: .
      file: ./tools/path/Dockerfile
      push: true
      tags: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_NAMESPACE }}/${{ env.DOCKER_IMAGE }}:${{ env.GITHUB_SHA }}
      outputs: type=image,oci-mediatypes=true,push=true

[jedevc]

Since you've redacted the <registry_url>, we can't tell where this error comes from - the 400 is coming from the registry, but it's not clear why.

I assume it's a self-hosted/internal registry? What registry software is it using?

[langovoi]

After GitHub Actions updated Ubuntu image (actions/runner-images/pull/6942, buildx was upped to v.0.10.0) — I get same error.

If I lock buildx version to v0.9.1 by action docker/setup-buildx-action@v2 it works well.

I use cr.yandex container registry. It is Yandex Cloud Container Registry.

[tapokshot]

@langovoi i use same cloud and get same error, but after your response i finally fix that problem, thanks

[pimvandenbroek]

Issue doesn't seem to be limited to Yandex alone, I'm having the same issue with GCR. @langovoi 's solution fixed it for me

[jedevc]

@pimvandenbroek do you have log messages from the push to GCR? We've been successfully pushing the new provenance attestations to GCR without too much of a problem 👀

[pimvandenbroek]

The initial error we received was: "registry cache exporter requires ref". I unfortunately don't have any other logs anymore.
After applying the version pin, everything worked again.

[jedevc]

@pimvandenbroek I think this is unrelated to this issue, looks like moby/buildkit@c5242ba is the source of the error message that was introduced in the newest BuildKit. I think this is related to a generic caching issue, instead of a GCR-specific issue; could you open a separate issue/discussion or share on #buildkit in the community slack?

[pimvandenbroek]

Sure, I will check that out.
The reason I replied here, is because up untill yesterday it was working fine. Then the Github Action Runner was updated, included buildx v.0.10.0, and it got broken.
When pinning the version back to 0.9.1 it started working again. Buildkit wasn't updated as far as I know.
The buildkit error message might be related or it might not, but the fact that reverting to buildx 0.9.1 fixed the issue tells me that I am in the right place.

[crazy-max]

@pimvandenbroek Doesn't just setting provenance: false with the build-push action (or --provenance false with buildx cmd) solves the issue instead of rolling back to 0.9.1?

[pimvandenbroek]

@crazy-max Not sure, I will have to test.
You mean the following right:

    -  name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
            provenance: false

[crazy-max]

@pimvandenbroek This is a build option so in build-push-action:

      -
        name: Build and push
        uses: docker/build-push-action@v3
        with:
          push: true
          provenance: false
          tags: user/app:latest

Or if you invoke buildx directly then docker buildx build --provenance false ....

Not sure, I will have to test.

Please that would help thx!

[pimvandenbroek]

Clear.
I'll try that block, as the buildx command is generated by code, and I currently don't have access to that part. I'll keep you posted

[langovoi]

@crazy-max I tested cr.yandex with provenance: false in docker/build-push-action@v3 and removed pin to v0.9.1 in docker/setup-buildx-action@v2.

It works.

[pimvandenbroek]

@crazy-max tested it, however apparently, the environment where the buildx command is generated is using an older version of buildkit and we received the following error:
buildx failed with: ERROR: attestations are not supported by the current buildkitd
I'll have to dive into that part early next week

[crazy-max]

@crazy-max tested it, however apparently, the environment where the buildx command is generated is using an older version of buildkit and we received the following error: buildx failed with: ERROR: attestations are not supported by the current buildkitd I'll have to dive into that part early next week

Interesting, if you could add docker buildx inspect before the build command in your script we could see why it's not supported. Maybe it's using BuildKit backed by the Docker Engine (default).

[violen]

Same Issue here:
buildx 0.9.1 works but the update by Docker Desktop to buildx 0.10.0 breaks Pushing to Sonatype Nexus Docker Registry (v3.38.1-01)

Error shown in Nexus is:

org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/imagename/manifests/latest
java.lang.NullPointerException: Cannot get property 'digest' on null object

buildx 0.10.0 seems to break the manifest generation.

Any Idea how to downgrade to buildx 0.9.1 ?

The Suggestion disabling the provenance via docker buildx --provenance false seems to work also with Sonartype Nexus.

Edit: @crazy-max here is the log output as requested.
buildx-out.log

I will try to get a nexus oss update to version 3.40.0+ as mentioned in the ticket you have linked.

[crazy-max]

@violen Can you post the logs of the build command please?

If you can also enable debug logs and post BuildKit logs that would be handy:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          buildkitd-flags: --debug

It seems Nexus support OCI format since 3.23.0: https://issues.sonatype.org/browse/NEXUS-21087.

Looks also related to this issue using docker-app which build and push OCI compliant images: https://github.com/docker/app#sharing-the-app-on-the-hub.

Best guess is Nexus does not fully support OCI spec (Index spec in particular).

Edit: Yes can confirm Nexus does not fully support the OCI spec

• https://issues.sonatype.org/browse/NEXUS-34847
• push cache to registry also fails: https://issues.sonatype.org/browse/NEXUS-27494

[pimvandenbroek]

@crazy-max Finally had the time to do a bit of a deepdive, and got the following result.
docker buildx inspect returned the following:

pablo - Name:          pablo
pablo - Driver:        docker-container
pablo - Last Activity: 2023-01-31 14:02:37 +0000 UTC
pablo - Nodes:
pablo - Name:      pablo
pablo - Endpoint:  unix:///var/run/docker.sock
pablo - Status:    running
pablo - Buildkit:  v0.11.2
pablo - Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Buildx / Buildkit:

BuildKit version
  builder-0f0b68c2-ebf1-440e-85e5-b8e5a95380060: moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.11.2 944939944ca4cc58a11ace4af714083cfcd9a3c7

Buildx version
  /usr/bin/docker buildx version
  github.com/docker/buildx 0.10.0+azure-1 876462897612d36679153c3414f7689626251501

We've added --provenance false to the buildx build command like so:
"docker" "buildx" "build" "--pull" "--output" "type=image,name=eu.gcr.io/REPO/IMAGE:TAG,push=true" "--provenance=false" "--build-arg" "COMMIT_HASH=c9253bd3a363134d9c45593e9d0ee2826885d552" "--cache-from" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest" "--cache-to" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest,mode=max" "-f" "k8s/Dockerfile" "."

However, unfortunately we are currently getting:
ERROR: failed to solve: error writing manifest blob: failed commit on ref "sha256:38d4502df1efa339d0b44820d3b561af63ea43fa353faad8229a0e81428a00e3": unexpected status: 400 Bad Request

Maybe it is really obvious, but I'm unable to find the solution

[crazy-max]

@pimvandenbroek I don't think this error is related to the image being pushed but the cache. Does it work if you remove

"--cache-to" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest,mode=max"?

[pimvandenbroek]

@crazy-max Yes, without cache-to everything is running fine.

[pimvandenbroek]

Does this mean that --cache-to isn't working in this version? Or do you perhaps know of a workaround so we can still use it?

[crazy-max]

Looks related to issues with Quay moby/buildkit#1440 and Harbor moby/buildkit#2479 (comment)

If you could post the BuildKit logs too that would be handy but best guess is Yandex does not properly support OCI mediatypes. Can you try with:

"--cache-to" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest,mode=max,oci-mediatype=false"

[pimvandenbroek]

@crazy-max Unfortunately, that didn't work either:

"docker" "buildx" "build" "--pull" "--output" "type=image,name=eu.gcr.io/REPO/IMAGE:tag,push=true" "--provenance" "false" "--build-arg" "COMMIT_HASH=feece02abfb339131ac1821ccad437862d293e3b" "--cache-from" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest" "--cache-to" "type=registry,ref=eu.gcr.io/REPO/IMAGE:latest,mode=min,oci-mediatype=false" "-f" "k8s/Dockerfile" "k8s"
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 0.5s done
#1 creating container buildx_buildkit_tag
#1 creating container buildx_buildkit_tag 0.5s done
#1 DONE 1.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 29B
#2 transferring dockerfile: 65B done
#2 DONE 0.0s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 ...
#5 [auth] library/alpine:pull token for registry-1.docker.io
#5 DONE 0.0s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.8s
#6 [1/2] FROM docker.io/library/alpine@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a
#6 resolve docker.io/library/alpine@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a done
#6 DONE 0.0s
#7 importing cache manifest from eu.gcr.io/REPO/IMAGE:latest
#7 ...
#8 [auth] REPO/IMAGE:pull token for eu.gcr.io
#8 DONE 0.0s
#7 importing cache manifest from eu.gcr.io/REPO/IMAGE:latest
#7 DONE 3.0s
#6 [1/2] FROM docker.io/library/alpine@sha256:f271e74b17ced29b915d351685fd4644785c6d1559dd1f2d4189a5e851ef753a
#6 sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9 3.37MB / 3.37MB 0.2s done
#6 extracting sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9
#6 extracting sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9 0.1s done
#6 DONE 0.3s
#9 [2/2] RUN echo "hello"
#0 0.095 hello
#9 DONE 0.1s
#10 exporting to image
#10 exporting layers 0.0s done
#10 exporting manifest sha256:28a7fa76848982633add699ceb9298436b7b47739c7283305a90077db4cc085d done
#10 exporting config sha256:ee1628b850162fc8f713cfff44eaa1300163f1a47143c72163f96ee58503e629 done
#10 pushing layers
#10 ...
#11 [auth] REPO/IMAGE:pull,push token for eu.gcr.io
#11 DONE 0.0s
#10 exporting to image
#10 pushing layers 2.3s done
#10 pushing manifest for eu.gcr.io/REPO/IMAGE:tag@sha256:28a7fa76848982633add699ceb9298436b7b47739c7283305a90077db4cc085d
#10 pushing manifest for eu.gcr.io/REPO/IMAGE:tag@sha256:28a7fa76848982633add699ceb9298436b7b47739c7283305a90077db4cc085d 0.7s done
#10 DONE 3.1s
#12 exporting content cache
#12 preparing build cache for export
#12 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#12 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.2s done
#12 writing layer sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9
#12 writing layer sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9 0.2s done
#12 writing config sha256:9d4fd633816360389de7c5b214258bc238039775740843cfd1a8988335095b4d
#12 writing config sha256:9d4fd633816360389de7c5b214258bc238039775740843cfd1a8988335095b4d 2.0s done
#12 writing manifest sha256:82f8f3e3f3209c5179a044d71fee90746657150bc259c7b24173bb3ace61a773
#12 preparing build cache for export 2.5s done
    1) Build and push image
    - Build and push image with provenance
    - Cache image
  Help
#12 writing manifest sha256:82f8f3e3f3209c5179a044d71fee90746657150bc259c7b24173bb3ace61a773 0.2s done
#12 ERROR: error writing manifest blob: failed commit on ref "sha256:82f8f3e3f3209c5179a044d71fee90746657150bc259c7b24173bb3ace61a773": unexpected status: 400 Bad Request
------
 > exporting content cache:
------
ERROR: failed to solve: error writing manifest blob: failed commit on ref "sha256:82f8f3e3f3209c5179a044d71fee90746657150bc259c7b24173bb3ace61a773": unexpected status: 400 Bad Request

[crazy-max]

@pimvandenbroek Can you create another BuildKit container with --buildkitd-flags '--debug' flag and post the logs of the container please? Like in moby/buildkit#2479 (comment)

[pimvandenbroek]

@crazy-max

time="2023-02-03T09:43:49Z" level=debug msg="do request" request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=buildkit/v0.11 request.method=HEAD spanID=cc7745aaeb3d9bd9 traceID=e7a2f5c820057c4947c93e8f9b473275 url="https://eu.gcr.io/v2/REPO/IMAGE/manifests/latest"
  time="2023-02-03T09:43:49Z" level=debug msg="fetch response received" response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.header.content-length=855 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Fri, 03 Feb 2023 09:43:49 GMT" response.header.docker-content-digest="sha256:3e238896d240821df797f0f4d2f1d7577553e654eec9a66da8dc3cc7692d86ea" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" spanID=cc7745aaeb3d9bd9 traceID=e7a2f5c820057c4947c93e8f9b473275 url="https://eu.gcr.io/v2/REPO/IMAGE/manifests/latest"
  time="2023-02-03T09:43:49Z" level=debug msg="do request" request.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json request.header.user-agent=buildkit/v0.11 request.method=PUT spanID=cc7745aaeb3d9bd9 traceID=e7a2f5c820057c4947c93e8f9b473275 url="https://eu.gcr.io/v2/REPO/IMAGE/manifests/latest"
  time="2023-02-03T09:43:49Z" level=debug msg="fetch response received" response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.header.cache-control=private response.header.content-type=application/json response.header.date="Fri, 03 Feb 2023 09:43:49 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="Docker Registry" response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="400 Bad Request" spanID=cc7745aaeb3d9bd9 traceID=e7a2f5c820057c4947c93e8f9b473275 url="https://eu.gcr.io/v2/REPO/IMAGE/manifests/latest"
  time="2023-02-03T09:43:49Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"MANIFEST_INVALID\",\"message\":\"Failed to parse manifest for request \\\"/v2/REPO/IMAGE/manifests/latest\\\": Failed to deserialize application/vnd.docker.distribution.manifest.list.v2+json.\"}]}" resp="&{400 Bad Request 400 HTTP/1.1 1 1 map[Alt-Svc:[h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000] Cache-Control:[private] Content-Type:[application/json] Date:[Fri, 03 Feb 2023 09:43:49 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] {0xc000966940} -1 [chunked] false true map[] 0xc000547900 0xc0000eb130}" spanID=cc7745aaeb3d9bd9 traceID=e7a2f5c820057c4947c93e8f9b473275
  time="2023-02-03T09:43:49Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:5e1cf7823d402156c4e2edc2a27246dd41cf869dfa0840518206d212f8af6c74\": unexpected status: 400 Bad Request"
  error writing manifest blob: failed commit on ref "sha256:5e1cf7823d402156c4e2edc2a27246dd41cf869dfa0840518206d212f8af6c74": unexpected status: 400 Bad Request
  1 v0.11.2 buildkitd --debug
  github.com/moby/buildkit/cache/remotecache.(*contentCacheExporter).Finalize
  	/src/cache/remotecache/export.go:140
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1.1
  	/src/solver/llbsolver/solver.go:605
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
  	/src/solver/llbsolver/solver.go:913
  github.com/moby/buildkit/solver.(*Job).InContext
  	/src/solver/jobs.go:611
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext
  	/src/solver/llbsolver/solver.go:909
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
  	/src/solver/llbsolver/solver.go:586
  golang.org/x/sync/errgroup.(*Group).Go.func1
  	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594
  
  1 v0.11.2 buildkitd --debug
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:576
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:2440
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1340
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1713
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594

[crazy-max]

Failed to deserialize application/vnd.docker.distribution.manifest.list.v2+json

Looks to be an issue with Yandex registry 😟

[pimvandenbroek]

@crazy-max I'm not using Yandex, I'm using GCR

[crazy-max]

Ah I'm confused, thought it was same issue as OP 😣

Maybe related to moby/buildkit#1143 (comment) in your case.

Also switching to GAR is recommended: https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr

[pimvandenbroek]

That's odd, as the issues began when 0.10.0 was pushed.
As we have way too many builds a the moment, just migrating to GAR isn't possible just yet, we'll have to plan a migration.

For now I think we'll disabled caching, or maybe revert to 0.9.1 untill we have the time to migrate to GAR.

Thanks for your help untill now. If you do have any ideas to solve this (without migrating to GAR), I'm all ears :)

[rdhatt]

As a data point: also repro'ing this issue running a buildx build --push ... with Artifactory container registry version 6.23.38

ERROR: failed to solve: failed to push ... failed commit on ref "manifest-sha256...": unexpected status: 400 Bad Request

We noted a new line in the output from buildx v0.10.0 while exporting the image: exporting attestation manifest ...

Looking at the docs for the --provenance option, there didn't appear anyway to turn it off. So we reverted to buildx version v0.9.1, which fixed our issue.

Returning to buildx v0.10.0 and using the undocumented --provenance false option also fixes our issue. Will this option be documented or is intentionally undocumented?

It appears OCI support came to Artifactory 7.11.1. We're upgrading in a few weeks but until then can't confirm/deny if Artifactory fully complies with the index spec.

I've attached the buildkit log. One discrepancy I noted is the logs mention request.header.user-agent=buildkit/v0.11 while the output of docker buildx version is github.com/docker/buildx v0.10.0-docker 876462897612d36679153c3414f7689626251501

buildkitlog.txt

[jedevc]

Returning to buildx v0.10.0 and using the undocumented --provenance false option also fixes our issue. Will this option be documented or is intentionally undocumented?

😱 oops, yup, that's an oversight, I've PRed some docs in.

I've attached the buildkit log. One discrepancy I noted is the logs mention request.header.user-agent=buildkit/v0.11 while the output of docker buildx version is github.com/docker/buildx v0.10.0-docker 8764628

Version mismatch is because buildkit and buildx releases do not necessarily line up. It's slightly unfortunate they're so close to each other, since it means it's easy to mistake one for the other - but this is expected, those are both part of the latest release series for their respective projects.

[ljy-life]

@pimvandenbroek Hello, you can configure an environment variable to solve this problem, it's same like --provenance false

export BUILDX_NO_DEFAULT_ATTESTATIONS=1

[sleeperss]

Hello

I have a similar issue on gitea :

ERROR: failed to solve: failed to push ****gitea_url****/****repo_orga****/****image_name****:latest: unexpected status from HEAD request to https://****gitea_url****/v2/****repo_orga****/****image_name****/blobs/sha256:7c6fed81b558a4c5d4dacc0e9fb078dbc9fb8b789a485833408237f3c46dafff: 400 Bad Request

unfortunatly provenance: false does not change anything :'(
If I remove the buildx step everything works as expected.

Using the following pipeline (My server needs a client certificate) :

name: Build docker-ci
on: 
  push:
    tags:
      - v*
  pull_request:
    branches:
      - master


defaults:
  run:
    shell: bash

jobs:
  Build:
    env:
      internal_address: ${{ secrets.internal_address }}
      PACKAGES_USER: ${{ secrets.PACKAGES_USER }}
      PACKAGES_ACCESS_TOKEN: ${{ secrets.PACKAGES_ACCESS_TOKEN }}
      CERT: ${{ secrets.CERT }}
      CERT_KEY: ${{ secrets.CERT_KEY }}
    name: ⛏️ Build
    steps:
      - name: 🔎 Checkout code
        uses: actions/checkout@v3   

      - name: 🪴 Setup environment
        uses: ./.gitea/actions/setup_env 
        
      - name: ⛏️ Build Dockerfile
        uses: ./.gitea/actions/build_docker

name: '🪴 Setup environment'
description: '🪴 Setup environment'
runs:
  using: 'composite'
  steps:
    - shell: bash
      name: '🪴 Setup environment'
      run: |
        apt update
        apt install -y nodejs npm ca-certificates curl gnupg
        install -m 0755 -d /etc/apt/keyrings

        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
        chmod a+r /etc/apt/keyrings/docker.gpg

        echo \
          "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
           focal stable" \
          > /etc/apt/sources.list.d/docker.list

        apt update -y

        apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

name: 'Build Dockerfile'
description: 'Build Dockerfile'
runs:
  using: 'composite'
  steps:
    - name: 🪴 Set BuildX
      uses: docker/[email protected]
    - name: 🪴 Set Docker credentials
      shell: bash
      run: | 
        mkdir -p /etc/docker/certs.d/****gitea_url****/
        echo "${{ env.CERT }}" > /etc/docker/certs.d/****gitea_url****/client.cert
        echo "${{ env.CERT_KEY }}" > /etc/docker/certs.d/****gitea_url****/client.key
    - name: 🔑 Login to Docker Registry
      uses: docker/login-action@v2
      with:
        registry: https://****gitea_url****
        username: ${{ env.PACKAGES_USER  }}
        password: ${{ env.PACKAGES_ACCESS_TOKEN  }}
    - uses: docker/build-push-action@master
      with:
        context: .
        file: "Dockerfile"
        tags: ****gitea_url****/****repo_orga****/****image_name****:latest
        push: ${{ github.ref_type == 'tag' }}

[vfiset]

@sleeperss have you tried a more recent version ? The latest is 3.0.0. We had similar problems back when 0.10 came out and since we upgraded past 1.1.0 (IIRC) we did not have these problems anymore.

[sleeperss]

@vfiset I tried v1 v2 v3 and master, unfortunatly, each of them leads to the same results :(

[vfiset]

Is it possible that the gitea_url does not support HEAD requests ? Try to do a similar HEAD request on the same endpoint. Then a GET.

[sleeperss]

Do you have a curl example that I can try to run ?

[sleeperss]

It is definitly linked to the client certificat,

If I disable ssl_verify_client on my reverse proxy, evrything works correctly.