issue with docker-env on restart

[medyagh]

with latest release 1.8.1

start minikube and eval docker env
build an image with docker

docker build -t med .

minikube stop
start minikube again (either stop and start or just start while it is running )

minikube correctly tells you that the docker-env port is changed and u need to re-eval

😄  minikube v1.8.1 on Darwin 10.13.6
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the docker driver based on existing profile
⌛  Reconfiguring existing host ...
📌  Noticed that you are using minikube docker-env:
❗  After minikube restart the dockerd ports might have changed. To ensure docker-env works properly.
Please re-eval the docker-env command:

	'minikube -p minikube docker-env'


🔄  Starting existing docker container for "minikube" ..

try re-eval:

minikube -p minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:32862"
export DOCKER_CERT_PATH="/Users/medmac/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

$ docker ps
error during connect: Get https://127.0.0.1:32862/v1.40/containers/json: EOF

$ docker images
error during connect: Get https://127.0.0.1:32862/v1.40/images/json: EOF

inside minikube docker is working

medmac@~/workspace/minikube (priyawadhwa-no-overwrite-tar) $ minikube ssh

docker@minikube:~$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
med                                       latest              acc98787071f        4 minutes ago       5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        4 weeks ago         116MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        4 weeks ago         161MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        4 weeks ago         171MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        4 weeks ago         94.4MB
alpine                                    latest              e7d92cdc71fe        7 weeks ago         5.59MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago        41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago        288MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB



docker@minikube:~$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
4de4320c1003        70f311871ae1           "/coredns -conf /etc…"   41 seconds ago      Up 39 seconds                           k8s_coredns_coredns-6955765f44-wkpds_kube-system_45860336-99a5-4dea-a9f4-c3fab77ba4ca_1
bed5d32193e9        70f311871ae1           "/coredns -conf /etc…"   41 seconds ago      Up 39 seconds                           k8s_coredns_coredns-6955765f44-bxgjl_kube-system_94ef63c6-fa15-4890-be74-1699cc8943ba_1
821606804a3d        aa67fec7d7ef           "/bin/kindnetd"          42 seconds ago      Up 40 seconds                           k8s_kindnet-cni_kindnet-8hmxz_kube-system_6f8467df-13af-4357-8e2e-514de6e111e5_1
eb6cea5fb099        ae853e93800d           "/usr/local/bin/kube…"   42 seconds ago      Up 40 seconds                           k8s_kube-proxy_kube-proxy-nqfz8_kube-system_a71b451d-cbf7-48b9-9fbe-c10c8b813404_1
353d27f63e12        k8s.gcr.io/pause:3.1   "/pause"                 43 seconds ago      Up 41 seconds                           k8s_POD_kube-proxy-nqfz8_kube-system_a71b451d-cbf7-48b9-9fbe-c10c8b813404_1
4f09085c0e43        k8s.gcr.io/pause:3.1   "/pause"                 43 seconds ago      Up 40 seconds                           k8s_POD_coredns-6955765f44-wkpds_kube-system_45860336-99a5-4dea-a9f4-c3fab77ba4ca_1
a30ef02dc9e9        k8s.gcr.io/pause:3.1   "/pause"                 43 seconds ago      Up 41 seconds                           k8s_POD_kindnet-8hmxz_kube-system_6f8467df-13af-4357-8e2e-514de6e111e5_1
256877385a70        k8s.gcr.io/pause:3.1   "/pause"                 43 seconds ago      Up 40 seconds                           k8s_POD_coredns-6955765f44-bxgjl_kube-system_94ef63c6-fa15-4890-be74-1699cc8943ba_1
41a167d96dc4        k8s.gcr.io/pause:3.1   "/pause"                 43 seconds ago      Up 41 seconds                           k8s_POD_storage-provisioner_kube-system_b403ab81-c4a5-4de9-909c-35afa89c9b1c_1
eb4980461f7f        d109c0821a2b           "kube-scheduler --au…"   51 seconds ago      Up 49 seconds                           k8s_kube-scheduler_kube-scheduler-m01_kube-system_e3025acd90e7465e66fa19c71b916366_1
b8e714a1a7ad        303ce5db0e90           "etcd --advertise-cl…"   51 seconds ago      Up 48 seconds                           k8s_etcd_etcd-m01_kube-system_3ec997b76fb6ed3b78da8e0b5676dac4_1
4abc62d5c27c        b0f1517c1f4b           "kube-controller-man…"   51 seconds ago      Up 50 seconds                           k8s_kube-controller-manager_kube-controller-manager-m01_kube-system_383600d5a5a800bd3691e94db5ea25cc_1
931a95c8ffed        90d27391b780           "kube-apiserver --ad…"   51 seconds ago      Up 49 seconds                           k8s_kube-apiserver_kube-apiserver-m01_kube-system_e6cd06b3d06392a262c0cef19b318176_1
a5a1ccf09acf        k8s.gcr.io/pause:3.1   "/pause"                 52 seconds ago      Up 50 seconds                           k8s_POD_etcd-m01_kube-system_3ec997b76fb6ed3b78da8e0b5676dac4_1
55d7dd7c760d        k8s.gcr.io/pause:3.1   "/pause"                 52 seconds ago      Up 50 seconds                           k8s_POD_kube-scheduler-m01_kube-system_e3025acd90e7465e66fa19c71b916366_1
95021c93b4e0        k8s.gcr.io/pause:3.1   "/pause"                 53 seconds ago      Up 51 seconds                           k8s_POD_kube-controller-manager-m01_kube-system_383600d5a5a800bd3691e94db5ea25cc_1
5c0ca89921ac        k8s.gcr.io/pause:3.1   "/pause"                 53 seconds ago      Up 51 seconds                           k8s_POD_kube-apiserver-m01_kube-system_e6cd06b3d06392a262c0cef19b318176_1
docker@minikube:~$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED              SIZE
med                                       latest              acc98787071f        About a minute ago   5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        4 weeks ago          116MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        4 weeks ago          161MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        4 weeks ago          171MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        4 weeks ago          94.4MB
alpine                                    latest              e7d92cdc71fe        7 weeks ago          5.59MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago         41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago         78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago         288MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago          742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago          80.8MB
docker@minikube:~$ exit
logout
medmac@~/workspace/minikube (priyawadhwa-no-overwrite-tar) $ docker ps
error during connect: Get https://127.0.0.1:32862/v1.40/containers/json: EOF
medmac@~/workspace/minikube (priyawadhwa-no-overwrite-tar) $ minikube -p minikube docker-env
medmac@~/workspace/minikube (priyawadhwa-no-overwrite-tar) $ docker images
error during connect: Get https://127.0.0.1:32862/v1.40/images/json: EOF

[medyagh]

could be related to preload ? this is 1.8.1 and in that release the preload for docker driver was broken but I tested with PR that fixes the preload and that one also has this problem
#6985

cc
@tstromberg
@priyawadhwa

[josedonizetti]

@medyagh The error is because kic is not reflecting the port change on the internal docker.

When you start kic for the first time it will map the internal docker port to an external port, local to the host:

7d0f1fe8a665        gcr.io/k8s-minikube/kicbase:v0.0.7   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   127.0.0.1:32776->22/tcp, 127.0.0.1:32775->2376/tcp, 127.0.0.1:32774->8443/tcp   minikube

external port: 32775
internal port: 2376

The internal port does match the dockerd inside minikube:

sudo lsof -i -P -n | grep LISTEN
dockerd    509   root    6u  IPv6 421720      0t0  TCP *:2376 (LISTEN)

After stopping and starting again, even by doing the minikube docker-env, the mapping kic is doing is no longer valid.

7d0f1fe8a665        gcr.io/k8s-minikube/kicbase:v0.0.7   "/usr/local/bin/entr…"   4 minutes ago       Up About a minute   127.0.0.1:32779->22/tcp, 127.0.0.1:32778->2376/tcp, 127.0.0.1:32777->8443/tcp   minikube

external port: 32778
internal port: 2376

If you look into the port of the internal docker, you gonna see it is no longer 2376

sudo lsof -i -P -n | grep LISTEN
dockerd    938   root    6u  IPv6 450777      0t0  TCP *:32777 (LISTEN)

[medyagh]

systemd file for VM

$ cat /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target  minikube-automount.service docker.socket
Requires= minikube-automount.service docker.socket 

[Service]
Type=notify



# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=hyperkit --insecure-registry 10.96.0.0/12 
ExecReload=/bin/kill -s HUP 

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

systemd file for kic

$ cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify



# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
#  Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 
ExecReload=/bin/kill -s HUP 

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

[medyagh]

interestingly systemctl restart wont change the port

so it seems that if the restart happens through systemctl it will keep the port

docker@d1:~$ sudo lsof -i -P -n | grep LISTEN | grep dockerd
dockerd    528   root    6u  IPv6 1081021      0t0  TCP *:2376 (LISTEN)
docker@d1:~$ sudo systemctl restart dockerd
Failed to restart dockerd.service: Unit dockerd.service not found.
docker@d1:~$ sudo systemctl restart docker 
docker@d1:~$ sudo lsof -i -P -n | grep LISTEN | grep dockerd
dockerd   7048   root    6u  IPv6 1110477      0t0  TCP *:2376 (LISTEN)
docker@d1:~$ sudo systemctl restart docker
docker@d1:~$ sudo lsof -i -P -n | grep LISTEN | grep dockerd
dockerd   8340   root    6u  IPv6 1146837      0t0  TCP *:2376 (LISTEN)