Check KVM existing networks so that docker custom network doesn't overlap

[thinkeng]

😄 Ubuntu 18.04 上的 minikube v1.16.0
✨ 根据现有的配置文件使用 docker 驱动程序
💨 For improved Docker performance, enable the overlay Linux kernel module using 'modprobe overlay'
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...

❗ minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4, but successfully downloaded kicbase/stable:v0.0.15-snapshot4 as a fallback image
🤷 docker "minikube" container is missing, will recreate.
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
E1219 12:28:29.867577 1566 network_create.go:81] error while trying to create network create network minikube 192.168.49.0/24: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc --label=created_by.minikube.sigs.k8s.io=true minikube: exit status 1
stdout:

stderr:
Error response from daemon: cannot create network b94ded520b4e0b15d70972bf6b0757711ae6a30210ddb7aa22782000bb3deef1 (br-b94ded520b4e): conflicts with network 117eb69f8b5394440df0ac7aebcfd65f91197d72607b76d79c4ba38757278fd0 (br-117eb69f8b53): networks have overlapping IPv4
❗ Unable to create dedicated network, this might result in cluster IP change after restart: un-retryable: create network minikube 192.168.49.0/24: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc --label=created_by.minikube.sigs.k8s.io=true minikube: exit status 1
stdout:

stderr:
Error response from daemon: cannot create network b94ded520b4e0b15d70972bf6b0757711ae6a30210ddb7aa22782000bb3deef1 (br-b94ded520b4e): conflicts with network 117eb69f8b5394440df0ac7aebcfd65f91197d72607b76d79c4ba38757278fd0 (br-117eb69f8b53): networks have overlapping IPv4

🤦 StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 kicbase/stable:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb31649c16: exit status 125
stdout:
0f776c824edd3e2d66d45e95db564621b0606f92e6fbddf3dc7319d8cc20f137

stderr:
WARNING: Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "rootfs_linux.go:58: mounting \"/lib/modules\" to rootfs \"/var/lib/docker/overlay2/57fbbb497007e7192d2c6ed7148d1117a700a7a19ed3fd192c882ae49b75e8c0/merged\" at \"/var/lib/docker/overlay2/57fbbb497007e7192d2c6ed7148d1117a700a7a19ed3fd192c882ae49b75e8c0/merged/usr/lib/modules\" caused \"device or resource busy\""": unknown.

🤷 docker "minikube" container is missing, will recreate.

🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
E1219 12:31:58.465139 1566 network_create.go:81] error while trying to create network create network minikube 192.168.49.0/24: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc --label=created_by.minikube.sigs.k8s.io=true minikube: exit status 1
stdout:

stderr:
Error response from daemon: cannot create network ff62039f0f64721c2d0baae63ea7e69c2d36c729de2bba28b21a09f1d883a7e3 (br-ff62039f0f64): conflicts with network 117eb69f8b5394440df0ac7aebcfd65f91197d72607b76d79c4ba38757278fd0 (br-117eb69f8b53): networks have overlapping IPv4
❗ Unable to create dedicated network, this might result in cluster IP change after restart: un-retryable: create network minikube 192.168.49.0/24: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc --label=created_by.minikube.sigs.k8s.io=true minikube: exit status 1
stdout:

stderr:
Error response from daemon: cannot create network ff62039f0f64721c2d0baae63ea7e69c2d36c729de2bba28b21a09f1d883a7e3 (br-ff62039f0f64): conflicts with network 117eb69f8b5394440df0ac7aebcfd65f91197d72607b76d79c4ba38757278fd0 (br-117eb69f8b53): networks have overlapping IPv4

😿 Failed to start docker container. Running "minikube delete" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 kicbase/stable:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb31649c16: exit status 125
stdout:
d430bde6cd36ed26afaf33cc4cc40cb7d0ebca3781bbec144632e452f88f60ee

stderr:
WARNING: Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "rootfs_linux.go:58: mounting \"/lib/modules\" to rootfs \"/var/lib/docker/overlay2/824fb74641df220f839307f7abbe30cd27f68a08383f71b5acf070395ad768f8/merged\" at \"/var/lib/docker/overlay2/824fb74641df220f839307f7abbe30cd27f68a08383f71b5acf070395ad768f8/merged/usr/lib/modules\" caused \"device or resource busy\""": unknown.

❌ Exiting due to GUEST_PROVISION: Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 kicbase/stable:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb31649c16: exit status 125
stdout:
d430bde6cd36ed26afaf33cc4cc40cb7d0ebca3781bbec144632e452f88f60ee

stderr:
WARNING: Your kernel does not support memory limit capabilities or the cgroup is not mounted. Limitation discarded.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "rootfs_linux.go:58: mounting \"/lib/modules\" to rootfs \"/var/lib/docker/overlay2/824fb74641df220f839307f7abbe30cd27f68a08383f71b5acf070395ad768f8/merged\" at \"/var/lib/docker/overlay2/824fb74641df220f839307f7abbe30cd27f68a08383f71b5acf070395ad768f8/merged/usr/lib/modules\" caused \"device or resource busy\""": unknown.

😿 If the above advice does not help, please let us know:
👉 https://github.com/kubernetes/minikube/issues/new/choose

[medyagh]

Thank you for taking the time to create this issue,
I am curious if you have KVM installed or Podman ?
one thing I can suspect is that minikube is overlapping with a network that is in KVM or podman

do you mind providing the output of the following commands ?

virt-host-validate

and also
virsh -c qemu:///system list --all
and aslo

virsh net-list

and also

if that is the case, minkube should do better and check for existing networks.

also, I wonder if there are other docker networks that overlap with this

docker network ls

u can inspect the networks using

docker network inspect "name"

[priyawadhwa]

Hey @niuxiongwen friendly ping, are you still seeing this issue?

[medyagh]

this is a bug in minikube and we should fix this this seems to be because of the overlaping kvm and docker network.

[medyagh]

place in the code https://github.com/medyagh/minikube/blob/a278e45dbfd11dc567ba3646453f2b51a07a8b9c/pkg/drivers/kvm/network.go#L39

[prezha]

/assign

[medyagh]

also https://github.com/medyagh/minikube/blob/a278e45dbfd11dc567ba3646453f2b51a07a8b9c/pkg/drivers/kic/oci/network_create.go#L35