Minikube start after previous delete fails with "network ... is already being used by a cni configuration"

[adamish]

Steps to reproduce the issue:

1. minikube start --driver=podman
2. minikube delete
3. minikube start --driver=podman

Full output of minikube start command used

😄  minikube v1.17.1 on Redhat 8.2
✨  Using the podman (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating podman container (CPUs=2, Memory=3800MB) ...| E0205 22:30:38.549998  304006 network_create.go:85] error while trying to create network create network minikube 192.168.49.0/24: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube: exit status 125
stdout:

stderr:
Error: network 192.168.49.0/24 is already being used by a cni configuration

The cause is that minikube delete doesn't clean up the network it created with podman, and the subsequent start command then sees the network and complains about it.

This can be worked around with the following, however this isn't obvious

sudo podman network rm minikube

[BLasan]

@adamish try running minikube delete --all instead of minikube delete

[alekonko]

@BLasan i have same problem. Cluster seems to work correctly.

Steps to reproduce the issue:

1. minikube delete --all
2. minikube config set driver podman
3. minikube start --driver=podman --container-runtime=cri-o -p minikube --cpus=4 --memory=12000MB

Full output of failed command:

Full output of minikube start command used, if not already included:
😄 minikube v1.17.1 on Fedora 33
✨ Using the podman (experimental) driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating podman container (CPUs=4, Memory=12000MB) ...- E0207 12:41:09.265224 831751 network_create.go:85] error while trying to create network create network minikube 192.168.49.0/24: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube: exit status 125
stdout:

stderr:
Error: network 192.168.49.0/24 is already being used by a cni configuration

❗ Unable to create dedicated network, this might result in cluster IP change after restart: un-retryable: create network minikube 192.168.49.0/24: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube: exit status 125
stdout:

stderr:
Error: network 192.168.49.0/24 is already being used by a cni configuration

🎁 Preparing Kubernetes v1.20.2 on CRI-O 1.19.1 ...
▪ Generating certificates and keys ...
🔗 Configuring CNI (Container Networking Interface) ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass

❗ /usr/local/bin/kubectl is version 1.18.2-0-g52c56ce, which may have incompatibilites with Kubernetes 1.20.2.
▪ Want kubectl v1.20.2? Try 'minikube kubectl -- get pods -A'
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Optional: Full output of minikube logs command:

[BLasan]

@BLasan i have same problem. Cluster seems to work correctly.

Steps to reproduce the issue:

1. minikube delete --all
2. minikube config set driver podman
3. minikube start --driver=podman --container-runtime=cri-o -p minikube --cpus=4 --memory=12000MB

Full output of failed command:

Full output of minikube start command used, if not already included:
minikube v1.17.1 on Fedora 33
Using the podman (experimental) driver based on user configuration
Starting control plane node minikube in cluster minikube
Creating podman container (CPUs=4, Memory=12000MB) ...- E0207 12:41:09.265224 831751 network_create.go:85] error while trying to create network create network minikube 192.168.49.0/24: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube: exit status 125
stdout:

stderr:
Error: network 192.168.49.0/24 is already being used by a cni configuration

Unable to create dedicated network, this might result in cluster IP change after restart: un-retryable: create network minikube 192.168.49.0/24: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube: exit status 125
stdout:

stderr:
Error: network 192.168.49.0/24 is already being used by a cni configuration

Preparing Kubernetes v1.20.2 on CRI-O 1.19.1 ...
Generating certificates and keys ...
Configuring CNI (Container Networking Interface) ...
Booting up control plane ...
Configuring RBAC rules ...
Verifying Kubernetes components...
Enabled addons: storage-provisioner, default-storageclass

/usr/local/bin/kubectl is version 1.18.2-0-g52c56ce, which may have incompatibilites with Kubernetes 1.20.2.
Want kubectl v1.20.2? Try 'minikube kubectl -- get pods -A'
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Optional: Full output of minikube logs command:

I think we need to run the command sudo podman network rm minikube along with the cluster deletion. @medyagh Please take a look

[afbjorklund]

Related to #9705 - we were relying on docker labels to do the cleanup, but they were either buggy or missing in podman...

The correct implementation would be to clean up the volumes and networks explicitly, and then only do labels as a fallback.

[BLasan]

Related to #9705 - we were relying on docker labels to do the cleanup, but they were either buggy or missing in podman...

The correct implementation would be to clean up the volumes and networks explicitly, and then only do labels as a fallback.

Can I take up this issue?

[BLasan]

/assign

[alekonko]

hi,
after cleaning podman bridge with "sudo podman network rm minikube", the startup errors are gone.
So before it worked as it used an old cni.

Thanks a lot

[BLasan]

hi,
after cleaning podman bridge with "sudo podman network rm minikube", the startup errors are gone.
So before it worked as it used an old cni.

Thanks a lot

Yuh. we need to clean up those configurations from the minikube clean up process instead of running the command manually :)

[BLasan]

@medyagh Can we close this issue sir?

[sharifelgamal]

Yeah, this issue should be fixed at HEAD and will be included in the next release. I'll close this for now.