Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minikube connection to internal registry is refused #4604

Closed
kameshsampath opened this issue Jun 26, 2019 · 36 comments
Closed

Minikube connection to internal registry is refused #4604

kameshsampath opened this issue Jun 26, 2019 · 36 comments
Labels
area/registry registry related issues kind/support Categorizes issue or PR as a support question. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.

Comments

@kameshsampath
Copy link

When I try to push images to internal registry from container (build via Tekton piplines) I get a connection refused error. This is working well with minikube v1.1.1.

The exact command to reproduce the issue:
Following https://github.com/kameshsampath/minikube-helpers/tree/master/registry/README.adoc.

The full output of the command that failed:
[build-greeter-jvm : buildah-build-push] Head http://dev.local/v2/rhdevelopers/pipeline-greeter/blobs/sha256:9ce2ac72ea3a3f426d08fe7ed892b6983f6ded93b9f82d8f87b8083760f26bec: dial tcp 10.111.212.82:80: connect: connection refused

The output of the minikube logs command:

When I inspected registry logs I got this

registry-p8bmz registry 10.1.0.20 - - [25/Jun/2019:08:54:32 +0000] "HEAD /v2/rhdevelopers/pipeline-greeter/blobs/sha256:49d9522d5abfe1bf7e9646b02e1125fec6376add018a3a1a9a5ab875a3e52b68 HTTP/1.1" 404 157 "" "Buildah/1.7"

The operating system version:

  • minikube version: v1.2.0
  • macOS Mojave 10.14.5
  • vm-driver: hyperkit or virtualbox

Related: #4553
@castlemilk
Copy link
Contributor

castlemilk commented Jun 27, 2019 •
edited
Loading

I've found the issue gets resolved if you remove the registry-proxy DaemonSet. when you enable the registry addon, it creates a Service which is loadbalancing across both the registry-proxy and the registry. Configuration is here: https://github.com/kubernetes/minikube/blob/master/deploy/addons/registry/

@kameshsampath
Copy link
Author

thanks @castlemilk - I suspected the same let me try deleting the daemonset

@castlemilk castlemilk mentioned this issue Jun 28, 2019
Closed
@tstromberg
Copy link
Contributor

I can't tell from #4627 - do we think this issue has been fixed now?

If not, what is required to do so?

@tstromberg tstromberg added triage/needs-information Indicates an issue needs more information in order to work on it. area/registry registry related issues labels Jul 17, 2019
@kumarom
Copy link
Contributor

kumarom commented Jul 18, 2019 •
edited
Loading

@tstromberg I think, #4627 does not resolves this issue..

@castlemilk
Copy link
Contributor

I was experiencing the exact same issue with Tekton, the merged changes in #4639, should hopefully resolve this and ensure testing validates connectivity from outside the minikube cluster (on port 5000) as well as internal access to the registry (on port 80)

kumarom added a commit to kumarom/minikube that referenced this issue Jul 18, 2019
@kumarom
Fix regression caused by registry-proxy
743c64f 
This PR fixes kubernetes#4604 by adding a new selector to concerned svc/rc only.
This also reverts `kubernetes.io/minikube-addons` to `registy` for registry-proxy
so that addon manager can deploy registry-proxy when registry addon is enabled.
@kumarom kumarom mentioned this issue Jul 18, 2019
Merged
@kumarom
Copy link
Contributor

kumarom commented Jul 18, 2019

@tstromberg @castlemilk @kameshsampath and others, I think #4805 should fix this issue.

kumarom added a commit to kumarom/minikube that referenced this issue Jul 18, 2019
@kumarom
Fix regression caused by registry-proxy
55cd455 
This PR addresses kubernetes#4604 by adding a new selector to concerned svc/rc only.
This also reverts `kubernetes.io/minikube-addons` to `registy` for registry-proxy
so that addon manager can deploy registry-proxy when registry addon is enabled.
kumarom added a commit to kumarom/minikube that referenced this issue Jul 18, 2019
@kumarom
Fix regression caused by registry-proxy
d76d874 
This PR addresses kubernetes#4604 by adding a new selector to concerned svc/rc only.
This also reverts `kubernetes.io/minikube-addons` to `registy` for registry-proxy
so that addon manager can deploy registry-proxy when registry addon is enabled.
@tstromberg tstromberg added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels Jul 18, 2019
@kameshsampath
Copy link
Author

kameshsampath commented Jul 18, 2019 via email

@nicolaferraro
Copy link
Contributor

Deleting the daemonset isn't really a workaround, as it gets recreated by the addon. Does anyone know a proper workaround until the fix gets released?

@kameshsampath
Copy link
Author

kameshsampath commented Jul 30, 2019 via email

@nicolaferraro
Copy link
Contributor

nicolaferraro commented Jul 30, 2019 •
edited
Loading

Nicola Staying with 1.1.1 ;)

That's a good idea ;)

@tstromberg
Copy link
Contributor

Has anyone tested this with v1.3.1? I'm curious if this issue can be closed.

@kameshsampath
Copy link
Author

kameshsampath commented Aug 20, 2019 via email

@tstromberg tstromberg added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. triage/needs-information Indicates an issue needs more information in order to work on it. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Sep 16, 2019
@tstromberg
Copy link
Contributor

Any luck confirming whether or not this issue still exists with v1.3.1?

@kameshsampath
Copy link
Author

@tstromberg - still same issue :( , if you can deploy the demo https://github.com/kameshsampath/pipeline-helloworld after enabling https://github.com/kameshsampath/minikube-helpers/registry you will see it just times out at the stage of pushing the image to local registry. I tried 1.3.1, 1.2.0 both same result. Works as usual with 1.1.1

@castlemilk
Copy link
Contributor

Not sure about those repos, but I can successfully make requests to the internal registry now via say: curl http://registry.kube-system.

Try setting your Tekton PipelineResource as follows:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: helloworld-image
spec:
  type: image
  params: 
  - name: url
    value: "registry.kube-system:80/tutorial/helloworld"

@kameshsampath
Copy link
Author

@castlemilk ah thats a good tip . I was simply aliasing the same via my helper.

@Javatar81
Copy link

@castlemilk I checked your example with Minikube 1.3.1 (registry add-on enabled and registry-proxy active) and Tekton 0.6.0. It worked and uses the registry service. It is important setting port 80 in the url (it does not work without setting any port) because this matches the registry service port.

@kameshsampath
Copy link
Author

kameshsampath commented Sep 19, 2019 via email

@tstromberg tstromberg added kind/support Categorizes issue or PR as a support question. and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels Sep 20, 2019
@tstromberg
Copy link
Contributor

I can't tell from the comments - was there a resolution found here?

@kameshsampath
Copy link
Author

kameshsampath commented Oct 11, 2019 •
edited
Loading

@castlemilk - were you able to use the image without setting insecure-regsitry in minkube start ?
when run the command like kubectl run greeter --image=registry.kube-system:80/rhdevelopers/knative-tutorial-greeter I get image pull backoff and describing the pod I get the error

Events:
  Type     Reason     Age        From               Message
  ----     ------     ----       ----               -------
  Normal   Scheduled  <unknown>  default-scheduler  Successfully assigned knativetutorial/greeter-78bbb96d8d-4xfk4 to minikube
  Normal   Pulling    11s        kubelet, minikube  Pulling image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter"
  Warning  Failed     11s        kubelet, minikube  Failed to pull image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.kube-system:80/v2/: dial tcp: lookup registry.kube-system on 192.168.64.1:53: no such host
  Warning  Failed     11s        kubelet, minikube  Error: ErrImagePull
  Normal   BackOff    11s        kubelet, minikube  Back-off pulling image "registry.kube-system:80/rhdevelopers/knative-tutorial-greeter"
  Warning  Failed     11s        kubelet, minikube  Error: ImagePullBackOff

@kameshsampath
Copy link
Author

@tstromberg @castlemilk - I am able to run with localhost:5000/<image-name> but when using registry-kubesystem:80/<image-name> result in lookup errors :(.

@kameshsampath
Copy link
Author

@tstromberg - I feel I still need to use my registry helper https://github.com/kameshsampath/minikube-helpers/tree/master/registry to make it resolve the registry.kube-system dns name to go to registry.kube-system.svc.cluster.local. It works with vm-driver virtualbox

@medyagh
Copy link
Member

medyagh commented Oct 14, 2019

@kameshsampath have you tried with other drivers? could you tell me what your helper does, is there any way we could integrate into minikube?

@kameshsampath
Copy link
Author

@medyagh - i see that my helper consistently working with 1.4.0 with both hyperkit and virtualbox.

This is what my helper does

  1. Define a list of dns names (via configmap) that you wish to you use for internal registry e.g. dev.local, example.com
  2. Run a daemonset to add dns names to /etc/hosts of the node matching the CLUSTER-IP of registry in kube-system that was enabled via addons
  3. Finally patch the coredns to resolve the dns names to registry.kube-system.cluster.local address

After these 3 you can now push and pull to the internal registry using the custom domain names.

is there any way we could integrate into minikube?

I was thinking to make this an addon, so we can easily enable/disable this ?

@medyagh
Copy link
Member

medyagh commented Nov 4, 2019

@kameshsampath that is a good idea ! please feel free to make a PR for it !

@medyagh
Copy link
Member

medyagh commented Dec 16, 2019 •
edited
Loading

@kameshsampath do you still have interests to make a PR for this and to clarify is the daemonset is inside minikube?

@kameshsampath
Copy link
Author

kameshsampath commented Dec 17, 2019 via email

@kameshsampath
Copy link
Author

kameshsampath commented Feb 16, 2020 •
edited
Loading

@medyagh - I have the initial draft for the addon, lets discuss this addon in new issue for clarity

New Issue #6656 and related PR #6657

@kameshsampath kameshsampath mentioned this issue Feb 16, 2020
Closed
@tstromberg
Copy link
Contributor

@kameshsampath - PR looks good, but could use a quick README file so that users understand what it is actually doing.

@kameshsampath
Copy link
Author

@tstromberg - thinking where is the right place to add that, just in the addons folder ?

I also got another question on whats the best practice to do the following while disabling the addon:

  1. I patch the coreDNS CM, wondering if there is way I could do a merge like kubectl does so that if we disable addon the update is reversed ?

  2. The /etc/hosts files get updated for entries, just thinking that we need to remove them while disabling

@kameshsampath
Copy link
Author

@tstromberg - updates done on the PR. thanks

@kameshsampath
Copy link
Author

@tstromberg - I think this issue has popped up again with 1.8.2, after applying the registry-aliases addon or https://github.com/kubernetes/minikube/tree/master/deploy/addons/registry-aliases, then tying to push image from within a pod fails with 404 :(. Any thoughts ?

@tstromberg
Copy link
Contributor

No thoughts, but a 404 does sound like a very different than "connection refused".

@tstromberg
Copy link
Contributor

tstromberg commented May 28, 2020 •
edited
Loading

Anyone still running into this with minikube v1.10.x or newer?

@kameshsampath
Copy link
Author

kameshsampath commented May 29, 2020 via email

@sharifelgamal
Copy link
Collaborator

OK, looks like this is safe to close. Please reopen if the issue return.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/registry registry related issues kind/support Categorizes issue or PR as a support question. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@tstromberg @kameshsampath @nicolaferraro @medyagh @castlemilk @kumarom @Javatar81 @sharifelgamal