investigate requirements for support for rootless

[owenthereal]

Steps to reproduce the issue:

I'm running rootless Docker on Fedora 32:

$ minikube start --driver docker --alsologtostderr -v=1
W1019 22:55:24.847725    7914 root.go:252] Error reading config file at /home/owen/.minikube/config/config.json: open /home/owen/.minikube/config/config.json: no such file or directory
I1019 22:55:24.848503    7914 out.go:191] Setting JSON to false
I1019 22:55:24.865576    7914 start.go:103] hostinfo: {"hostname":"mbp-oinux","uptime":1250,"bootTime":1603172074,"procs":362,"os":"linux","platform":"fedora","platformFamily":"fedora","platformVersion":"32","kernelVersion":"5.8.15-201.fc32.x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostid":"7eb75122-5f3a-4d0a-a3db-57f3818fe31c"}
I1019 22:55:24.867438    7914 start.go:113] virtualization: kvm host
I1019 22:55:24.868454    7914 out.go:109] 😄  minikube v1.14.0 on Fedora 32
😄  minikube v1.14.0 on Fedora 32
I1019 22:55:24.868756    7914 notify.go:126] Checking for updates...
I1019 22:55:24.868861    7914 driver.go:288] Setting default libvirt URI to qemu:///system
W1019 22:55:24.939450    7914 docker.go:105] docker version returned error: exit status 1
I1019 22:55:24.939982    7914 out.go:109] ✨  Using the docker driver based on user configuration
✨  Using the docker driver based on user configuration
I1019 22:55:24.940050    7914 start.go:272] selected driver: docker
I1019 22:55:24.940082    7914 start.go:680] validating driver "docker" against <nil>
I1019 22:55:24.940139    7914 start.go:691] status for docker: {Installed:true Healthy:false Running:false NeedsImprovement:false Error:"docker version --format {{.Server.Os}}-{{.Server.Version}}" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Fix:Start the Docker service Doc:https://minikube.sigs.k8s.io/docs/drivers/docker/}
I1019 22:55:24.940499    7914 out.go:109]

W1019 22:55:24.940742    7914 out.go:145] 💣  Exiting due to PROVIDER_DOCKER_NOT_RUNNING: "docker version --format -" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
💣  Exiting due to PROVIDER_DOCKER_NOT_RUNNING: "docker version --format -" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
W1019 22:55:24.941099    7914 out.go:145] 💡  Suggestion: Start the Docker service
💡  Suggestion: Start the Docker service
W1019 22:55:24.941407    7914 out.go:145] 📘  Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/
📘  Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/
I1019 22:55:24.941482    7914 out.go:109]

$ echo $DOCKER_HOST
unix:///run/user/1000/docker.sock

$ docker version
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46
 Built:             Wed Sep 16 16:58:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46
  Built:            Wed Sep 16 17:04:43 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

As you can see my rootless Docker is running on unix:///run/user/1000/docker.sock but minikube is trying to talk to the one on unix:///var/run/docker.sock

Full output of failed command:

See above

Full output of minikube start command used, if not already included:

See above

Optional: Full output of minikube logs command:

``` minikube logs 🤷 There is no local cluster named "minikube" 👉 To fix this, run: "minikube start" ```

[afbjorklund]

Not sure if rootless docker is enough to run the docker-in-docker node, but it should at least give a more obvious message...

There are some similar issues on Windows, when people have both Hyper-V and WSL versions of the docker daemon VMs.

[afbjorklund]

Are you able to run kind with this docker setup ?

https://github.com/kubernetes-sigs/kind

AkihiroSuda commented on Jul 15

Work in progress for kind: kubernetes-sigs/kind#1727

[ilya-zuyev]

Wondering, if we need to support non-default docker endpoints anyway. Running docker server on a socket/fd different from /var/run/docker.sock is a perfectly valid configuration.

[afbjorklund]

I'm not sure that the container will start with rootless docker, it seems to have the same issues as running on chromeos etc:

INFO: remounting /sys read-only
+ mount -o remount,ro /sys
mount: /sys: permission denied.

[afbjorklund]

@ilya-zuyev :

Wondering, if we need to support non-default docker endpoints anyway. Running docker server on a socket/fd different from /var/run/docker.sock is a perfectly valid configuration.

Moving /var/run/docker.sock to a weird location is quite different from trying to run rootless docker, though.

It's a feature that could be supported, but it wouldn't be a very common setup. Maybe if you want two ?

[afbjorklund]

This issue is similar to #8719 (for podman)

And my comment was #8719 (comment)

So I think it would be better off handled in usernetes.

There was some similar "root" discussions in #8257

[ilya-zuyev]

@afbjorklund

Wondering, if we need to support non-default docker endpoints anyway. Running docker server on a socket/fd different from /var/run/docker.sock is a perfectly valid configuration.

Moving /var/run/docker.sock to a weird location is quite different from trying to run rootless docker, though.

True. It seems we have two issues here (please correct me if I'm wrong):

1. Support custom docker endpoints (Allow minikube to run in a non-default docker host #9463). Check if DOCKER_* environment variables are set and use it instead of defaults
2. Support rootless docker (this issue). It, probably, depends on 1 since using /var/run/* requires root permissions and we have to define DOCKER_HOST.

[afbjorklund]

The issues aren't so much about "custom docker endpoint", that's pretty much a given if not using the standard socket...

Issue 1 is about using a remote (not local) Docker daemon, Issue 2 is about using a rootless (not root) Docker daemon.

1. DOCKER_HOST="tcp://192.168.99.100:2376"

2. DOCKER_HOST="unix:///run/user/1000/docker.sock"

[owenthereal]

Are you able to run kind with this docker setup ?

Nope, I wasn't able to kubernetes-sigs/kind#1902. And it looks like k8s can't run on rootless docker yet. But as you mentioned, there are two issues here. It would be nice that minikube can respect the DOCKER_HOST env var.

[ilya-zuyev]

The issues aren't so much about "custom docker endpoint", that's pretty much a given if not using the standard socket...

Issue 1 is about using a remote (not local) Docker daemon, Issue 2 is about using a rootless (not root) Docker daemon.

1. DOCKER_HOST="tcp://192.168.99.100:2376"
2. DOCKER_HOST="unix:///run/user/1000/docker.sock"

Both require minikube to use non-default DOCKER_HOST to work with daemon correctly, and right now we just set DOCKER_HOST to "" internally. This should be fixed in #9463.

[afbjorklund]

They will both still fail though.

[afbjorklund]

@medyagh : rootless minikube seems unlikely to make it to the 1.16 release, so we should probably remove this issue from it ?

it will however be something for next year's (2021) roadmap, to keep an eye on the Docker 20 and Kubernetes 1.20 progress

[medyagh]

invetigation done, will be done in this issue #10836

[shubhamphegade]

Try the below commands sequentially:
minikube config set driver docker
minikube delete
minikube start --driver=docker

hope it help as it worked for me