Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'docker push' w/ docker-registry on reencrypt route with SSL certs fails with "unauthorized: authentication required" #14249

Closed
dmsimard opened this issue May 18, 2017 · 4 comments
Closed

'docker push' w/ docker-registry on reencrypt route with SSL certs fails with "unauthorized: authentication required" #14249

dmsimard opened this issue May 18, 2017 · 4 comments
Assignees
@miminar
Labels
component/imageregistry kind/bug Categorizes issue or PR as related to a bug. priority/P2

Comments

@dmsimard
Copy link

dmsimard commented May 18, 2017 •
edited
Loading

In summary, a docker-registry service on a reencrypt route (with proper certificates) and self-signed certificates mounted in /etc/secrets will let you login successfully but pushes will fail with a client-side error: unauthorized: authentication required

Switching the route to passthrough and re-mounting /etc/secrets with the proper certificates lets you both log in and push.

Reproducible every time when deploying origin 1.5 with openshift-ansible on CentOS mostly following instructions provided for standalone registry setup.

Version
oc v1.5.0
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.1.6:8443
openshift v1.5.0
kubernetes v1.5.2+43a9be4
Steps To Reproduce

  1. Create proper certificates

  2. Configure openshift-ansible to set up route host, termination and ssl certs, for example:

openshift_hosted_registry_routehost: registry.domain.tld
openshift_hosted_registry_routetermination: reencrypt
openshift_hosted_registry_routecertificates:
  certfile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-cert.pem"
  keyfile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-privkey.pem"
  cafile: "/etc/letsencrypt/live/registry.domain.tld/registry.domain.tld-chain.pem"

  1. Run openshift-ansible to set up a standalone registry+console as per documentation

  2. Create user and project

  3. Retrieve user token and login successfully

docker login -p token -e unused -u unused registry.domain.tld
  1. Pull an image from docker hub and try to push it to the project
docker pull alpine
docker tag docker.io/alpine registry.domain.tld/project/alpine
docker push registry.domain.tld/project/alpine

Push fails with:

# docker push registry.domain.tld/test/alpine
The push refers to a repository [registry.domain.tld/test/alpine]
e154057080f4: Pushing [==================================================>] 4.016 MB
unauthorized: authentication required
Current Result

Login works but push fails

Expected Result

Login and push both work

Additional Information

Here's a side-by-side diff of route/svc/pod export as well as master-config.yml for the exact same deployment configuration except one is passthrough (left), the other is reencrypt (right):
https://www.diffchecker.com/wzWtOFmJ

Here's the docker-registry logs on a successful docker login:

10.128.0.1 - - [18/May/2017:17:00:53 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:00:53.526338597Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=8556f42f-bf65-413e-9d76-1faafbd51ca9 http.request.method=GET http.request.remoteaddr="10.128.0.1:37868" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:53.52642273Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=8556f42f-bf65-413e-9d76-1faafbd51ca9 http.request.method=GET http.request.remoteaddr="10.128.0.1:37868" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:53.954144006Z" level=info msg="response completed" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=e7e72bda-6c08-4212-93f2-e72ae7e2d624 http.request.method=GET http.request.remoteaddr="10.128.0.1:37872" http.request.uri="/openshift/token?account=unused&client_id=docker&offline_token=true" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json" http.response.duration=3.257955ms http.response.status=200 http.response.written=117 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:00:53 +0000] "GET /openshift/token?account=unused&client_id=docker&offline_token=true HTTP/1.1" 200 117 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:00:54.336259375Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=ed5212a5-f786-4b4d-bdac-93221798a762 http.request.method=GET http.request.remoteaddr="10.128.0.1:37876" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:00:54.339158083Z" level=info msg="response completed" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=ed5212a5-f786-4b4d-bdac-93221798a762 http.request.method=GET http.request.remoteaddr="10.128.0.1:37876" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.233502ms http.response.status=200 http.response.written=2 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:00:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"

Here's the docker-registry logs on a failed docker push:

time="2017-05-18T17:01:01.12733397Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=b363194a-7507-4891-8556-ad401ff5c106 http.request.method=GET http.request.remoteaddr="10.128.0.1:37900" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
time="2017-05-18T17:01:01.127436592Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=b363194a-7507-4891-8556-ad401ff5c106 http.request.method=GET http.request.remoteaddr="10.128.0.1:37900" http.request.uri="/v2/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:01 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:01.509111676Z" level=info msg="response completed" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=2e668396-a92b-484c-8ca9-405b93d8d2b7 http.request.method=GET http.request.remoteaddr="10.128.0.1:37904" http.request.uri="/openshift/token?account=unused&scope=repository%3Atripleo%2Fcentos%3Apush%2Cpull" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.contenttype="application/json" http.response.duration=3.634646ms http.response.status=200 http.response.written=117 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:01 +0000] "GET /openshift/token?account=unused&scope=repository%3Atripleo%2Fcentos%3Apush%2Cpull HTTP/1.1" 200 117 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:02.01261792Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.01812253Z" level=debug msg="Origin auth: checking for access to repository:tripleo/centos:pull" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.020825074Z" level=debug msg="Origin auth: checking for access to repository:tripleo/centos:push" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.023760248Z" level=debug msg="(*linkedBlobStore).Writer" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.030031724Z" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/startedat\")" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=6.198524ms trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=abbf9ec5-71c8-4186-9c67-05db49d2ed50 trace.line=95 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.030201053Z" level=debug msg="filesystem.Writer(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/data\", false)" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=86.493µs trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=cc01145a-bc40-40ff-841e-0e9821252df5 trace.line=124 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.035808284Z" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/tripleo/centos/_uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8/hashstates/sha256/0\")" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.auth.user=dmsimard openshift.auth.userid=d5ad0e0c-3be7-11e7-86af-fa163e58c44f openshift.logger=registry trace.duration=5.454152ms trace.file="/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=a98da2cc-1181-4afa-aa41-1c9e8a711217 trace.line=95 vars.name="tripleo/centos" 
time="2017-05-18T17:01:02.036032606Z" level=info msg="response completed" go.version=go1.7.4 http.request.host=trunk.registry.rdoproject.org http.request.id=a12a91cc-703f-4144-9e86-280c08dd1b4a http.request.method=POST http.request.remoteaddr="10.128.0.1:37908" http.request.uri="/v2/tripleo/centos/blobs/uploads/" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" http.response.duration=27.502021ms http.response.status=202 http.response.written=0 instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry 
10.128.0.1 - - [18/May/2017:17:01:02 +0000] "POST /v2/tripleo/centos/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
time="2017-05-18T17:01:02.408410116Z" level=debug msg="authorizing request" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=54b68a50-9ff4-45fb-b3dc-9b4667694cde http.request.method=PATCH http.request.remoteaddr="10.128.0.1:37912" http.request.uri="/v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" vars.uuid=37fe4ee0-2245-4c89-8f9e-30c74f16b1e8 
time="2017-05-18T17:01:02.408521996Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.4 http.request.host="trunk.registry.rdoproject.org:443" http.request.id=54b68a50-9ff4-45fb-b3dc-9b4667694cde http.request.method=PATCH http.request.remoteaddr="10.128.0.1:37912" http.request.uri="/v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9" http.request.useragent="docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))" instance.id=07e4a1d7-5dcd-4acc-9106-6c21f08f9958 openshift.logger=registry vars.name="tripleo/centos" vars.uuid=37fe4ee0-2245-4c89-8f9e-30c74f16b1e8 
10.128.0.1 - - [18/May/2017:17:01:02 +0000] "PATCH /v2/tripleo/centos/blobs/uploads/37fe4ee0-2245-4c89-8f9e-30c74f16b1e8?_state=fapf89621-4skE0-D_L9JrmM1gpUNTB3_hhcq9BG8OR7Ik5hbWUiOiJ0cmlwbGVvL2NlbnRvcyIsIlVVSUQiOiIzN2ZlNGVlMC0yMjQ1LTRjODktOGY5ZS0zMGM3NGYxNmIxZTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMThUMTc6MDE6MDIuMDIzNzk4MjEzWiJ9 HTTP/1.1" 401 208 "" "docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/4.10.14-200.fc25.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.3 \\(linux\\))"
@dmsimard
Copy link
Author

@abutcher @aweiteka fyi, as per discussed

@pweil- pweil- added component/imageregistry kind/bug Categorizes issue or PR as related to a bug. priority/P2 labels May 19, 2017
@dmsimard dmsimard mentioned this issue May 19, 2017
Merged
@thomasmckay thomasmckay mentioned this issue May 31, 2017
Closed
@abutcher abutcher mentioned this issue Jun 21, 2017
Merged
@mjudeikis
Copy link
Contributor

Does this has a workaround now?

@mjudeikis
Copy link
Contributor

Bizarre thing is that some of the layers managed to land in the filesystem:

drwxr-sr-x. 17 1000000000 1000000000 4096 Aug 18 12:36 .                                                                                                                                                                                             
drwxr-sr-x.  3 1000000000 1000000000   22 Aug 18 12:23 ..                                                                                                                                                                                            
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 13e33fe4-12eb-4924-9bbf-cb72a79a6caa                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 150af01b-7c2a-44b9-9c68-a30bc15a8933                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 29aa3a9e-dc4a-4d91-8fce-4be78b094015                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:31 450ec39a-a94c-4b23-beac-cd68892f360e                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 50ef1402-e1b1-4c68-af34-b34db4bce3df                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 7037d840-c5cf-4716-b98c-17cea1d862e3                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 7f57b5ef-fb9b-4e2d-8a76-0c15b157224d                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 882b3a66-45d9-41d7-83b2-abceb7fc21cb                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 9be1f361-d503-4653-ac42-41d42b9c21a4                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 c20ae65b-b249-4e0e-a95e-3fae9a9f494f                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:33 d16cd403-c808-4e14-b035-ea3f89a23597                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:23 d20250d6-d738-4f70-bc94-0f29bfe6a030                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 dcc1b319-0ecf-4c14-a344-f047fea6fb9e                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:27 e5eb03da-d316-4439-a0f7-1d18f38005a9                                                                                                                                                          
drwxr-sr-x.  3 1000000000 1000000000   53 Aug 18 12:36 f8caac91-12bf-4adc-af2f-d54691968457                                                                                                                                                          
sh-4.2$ pwd                                                                                                                                                                                                                                          
/registry/docker/registry/v2/repositories/library/rhel-tools/_uploads

and docker cli:

[root@ocpt00236 ~]# docker push  containers.registry.company.local/library/rhel-tools:latest
The push refers to a repository [containers.registry.company.local/library/rhel-tools]
d466c001c314: Pushing [==================================================>]  1.19 GB
bb4f52dd78f6: Pushing [==================================================>] 10.24 kB
f1f88d1c363a: Pushing [==================================================>] 195.9 MB/195.9 MB
unauthorized: authentication required

Logs:


10.217.4.1 - - [18/Aug/2017:12:36:49 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/882b3a66-45d9-41d7-83b2-abceb7fc21cb?_state=mvg6sxRQPHJ6kjsn_He6J6ksLpsFMx1NR2iNjnSb2wN7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiODgyYjNhNjYtNDVkOS00MWQ3LTgzYjItYWJjZWI3ZmMyMWNiIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyODY0NTY4OVoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"
--
  | time="2017-08-18T12:36:50.127546721Z" level=debug msg="authorizing request" go.version=go1.7.6 http.request.host="containers.registry.companys.local:443" http.request.id=cc250e82-ea11-4e2d-bc01-a167e43ba89f http.request.method=PATCH http.request.remoteaddr="10.217.2.1:46762" http.request.uri="/v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=f8caac91-12bf-4adc-af2f-d54691968457
  | time="2017-08-18T12:36:50.12770762Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.6 http.request.host="containers.registry.comapny.local:443" http.request.id=cc250e82-ea11-4e2d-bc01-a167e43ba89f http.request.method=PATCH http.request.remoteaddr="10.217.2.1:46762" http.request.uri="/v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=f8caac91-12bf-4adc-af2f-d54691968457
  | time="2017-08-18T12:36:50.132647441Z" level=debug msg="authorizing request" go.version=go1.7.6 http.request.host="containers.registry.company.local:443" http.request.id=8765f19b-d178-4166-97b4-e123174f7ae2 http.request.method=PATCH http.request.remoteaddr="10.217.0.1:52708" http.request.uri="/v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=dcc1b319-0ecf-4c14-a344-f047fea6fb9e
  | time="2017-08-18T12:36:50.132767552Z" level=error msg="error authorizing context: authorization header required" go.version=go1.7.6 http.request.host="containers.registry.company.local:443" http.request.id=8765f19b-d178-4166-97b4-e123174f7ae2 http.request.method=PATCH http.request.remoteaddr="10.217.0.1:52708" http.request.uri="/v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))" instance.id=77d360ab-b4d0-42a2-a6fd-14097bb6c08f openshift.logger=registry vars.name="library/rhel-tools" vars.uuid=dcc1b319-0ecf-4c14-a344-f047fea6fb9e
  | 10.217.0.1 - - [18/Aug/2017:12:36:50 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/dcc1b319-0ecf-4c14-a344-f047fea6fb9e?_state=t24iCUNB4RE4IT740ZVaG4pkelQzRU_jCcPk5H1t5wh7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZGNjMWIzMTktMGVjZi00YzE0LWEzNDQtZjA0N2ZlYTZmYjllIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyMjE1NjkyOFoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"
  | 10.217.2.1 - - [18/Aug/2017:12:36:50 +0000] "PATCH /v2/library/rhel-tools/blobs/uploads/f8caac91-12bf-4adc-af2f-d54691968457?_state=9nBbILDnjl1MwyPGVu1sgaXC6hJeh9KFkvg0gAqD81Z7Ik5hbWUiOiJsaWJyYXJ5L3JoZWwtdG9vbHMiLCJVVUlEIjoiZjhjYWFjOTEtMTJiZi00YWRjLWFmMmYtZDU0NjkxOTY4NDU3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDEyOjM2OjQ5LjMyNzQ4MjA2OFoifQ%3D%3D HTTP/1.1" 401 216 "" "docker/1.12.6 go/go1.7.6 kernel/3.10.0-514.26.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.12.6 \\(linux\\))"

@mfojtik mfojtik assigned miminar and unassigned mfojtik Aug 23, 2017
@miminar
Copy link

miminar commented Sep 5, 2017

Resolved in 3.7 via #15694.

See bz#1471707 for more details.

If you need it in earlier release, please clone the bugzilla with a desired target release.

@miminar miminar closed this as completed Sep 5, 2017
rlaurika added a commit to CSCfi/pouta-openshift-cluster that referenced this issue Oct 30, 2017
@rlaurika
Fix registry and registry console certs
3341c7c 
The registry is by default deployed using a certificate signed by the
OpenShift CA. As this is not generally recognized, create a new
re-encrypting edge route to the registry that uses a proper certificate.
This way it is possible to login to the registry normally.

Using the re-encrypting route triggers a bug in the Docker registry that
is fixed in a newer version. To work around this bug, manually update
the Docker image used for the registry to a newer one. See:
openshift/origin#14249 and also
openshift/origin#11391.

The registry console also needs to be reconfigured with the new route to
the registry. Make this configuration change using the oc_env module
from openshift-ansible. For this to work, add modules from the
lib_openshift role into the library path.

Replace the certificate of the registry console with a proper recognized
certificate so that warnings are not shown when accessing the console
from a browser.

Write tests for checking correct connectivity to the routes used for the
registry and the registry console. These should verify that there are no
certificate issues with the endpoints.
rlaurika added a commit to CSCfi/pouta-openshift-cluster that referenced this issue Oct 30, 2017
@rlaurika
Fix registry and registry console certs
028aaf7 
The registry is by default deployed using a certificate signed by the
OpenShift CA. As this is not generally recognized, create a new
re-encrypting edge route to the registry that uses a proper certificate.
This way it is possible to login to the registry normally.

Using the re-encrypting route triggers a bug in the Docker registry that
is fixed in a newer version. To work around this bug, manually update
the Docker image used for the registry to a newer one. See:
openshift/origin#14249 and also
openshift/origin#11391.

The registry console also needs to be reconfigured with the new route to
the registry. Make this configuration change using the oc_env module
from openshift-ansible. For this to work, add modules from the
lib_openshift role into the library path.

Replace the certificate of the registry console with a proper recognized
certificate so that warnings are not shown when accessing the console
from a browser.

Write tests for checking correct connectivity to the routes used for the
registry and the registry console. These should verify that there are no
certificate issues with the endpoints.

Split the registry config changes into their own playbook from the
post-install playbook to keep things tidy.
rdoproject pushed a commit to rdo-infra/rdo-container-registry that referenced this issue Nov 6, 2017
Update RDO registry playbooks to deploy OpenShift 3.7
c9daecf 
This commits allows to deploy the RDO registry with OpenShift 3.7.

It currently uses the "rdo-test" branch because there is still one unmerged
pull request upstream that hasn't merged yet upstream.

Delta from OpenShift 3.5 that is interesting to us:

- Significant improvements for registry and image pruning see [1][2][3][4]
- docker-registry can now use reencrypt routes [5]
- Metrics and logging were deployed by default in 3.5, this is no longer the
  case in 3.7, avoiding an unnecessary impact on performance. [6]
- We're now deploying a persistent volume for the docker-registry service on
  the local filesystem.

[1]: openshift/origin#13671
[2]: openshift/origin#16717
[3]: openshift/origin#16656
[4]: openshift/origin#17020
[5]: openshift/origin#14249
[6]: openshift/openshift-ansible@660bafe

Change-Id: I5c364a1aab883b6af061051bf190ce857bf2e1f9
@bparees bparees mentioned this issue Jul 3, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@miminar miminar

Labels
component/imageregistry kind/bug Categorizes issue or PR as related to a bug. priority/P2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mfojtik @dmsimard @miminar @mjudeikis @pweil-