unify Default CNI for VM and containers

[medyagh]

currently we have two CNIs

https://github.com/kubernetes/minikube/blob/a9558856c54cc0b3b6c46a7ef96960f521a72fb6/pkg/minikube/bootstrapper/kubeadm/default_cni.go

• defaultCNIConfig
• kicCNIConfig

kicCNIConfig is only used by containerd and crio only on docker driver.
I have a feeling by tweaking the defaultCNIConfig we could get rid of the kicCNIConfig.

the defaultCNIConfig has "subnet": "10.1.0.0/16",

// The config is being written to /etc/cni/net.d/k8s.conf.
const defaultCNIConfig = `
{
  "cniVersion": "0.3.0",
  "name": "rkt.kubernetes.io",
  "type": "bridge",
  "bridge": "mybridge",
  "mtu": 1460,
  "addIf": "true",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.1.0.0/16",
    "gateway": "10.1.0.1",
    "routes": [
      {
        "dst": "0.0.0.0/0"
      }
    ]
  }
}
`

but for kic drivers we add 10.244.0.0/16

🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16

The Error if we delete kic overlay for cotainerd on docker driver.

medmac@~/workspace/minikube (remove_overlay) $ kc get pods -A
NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
kube-system   coredns-66bff467f8-l7qkl           0/1     ContainerCreating   0          96s
kube-system   coredns-66bff467f8-vf6bm           0/1     ContainerCreating   0          96s
kube-system   etcd-minikube                      1/1     Running             0          112s
kube-system   kube-apiserver-minikube            1/1     Running             0          112s
kube-system   kube-controller-manager-minikube   1/1     Running             0          112s
kube-system   kube-proxy-t424s                   1/1     Running             0          96s
kube-system   kube-scheduler-minikube            1/1     Running             0          112s
kube-system   storage-provisioner                1/1     Running             0          110s

and here is the descripbe which says:
'failed to set bridge addr: could not add IP address to "cni0": permission denied'

medmac@~/workspace/minikube (remove_overlay) $ kc describe pod coredns-66bff467f8-vf6bm -n kube-system
Name:                 coredns-66bff467f8-vf6bm
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 minikube/172.17.0.2
Start Time:           Sun, 05 Apr 2020 00:28:49 -0700
Labels:               k8s-app=kube-dns
                      pod-template-hash=66bff467f8
Annotations:          <none>
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-66bff467f8
Containers:
  coredns:
    Container ID:
    Image:         k8s.gcr.io/coredns:1.6.7
    Image ID:
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-4pcrq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-4pcrq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-4pcrq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Warning  FailedScheduling        37s   default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               37s   default-scheduler  Successfully assigned kube-system/coredns-66bff467f8-vf6bm to minikube
  Warning  FailedCreatePodSandBox  34s   kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f12bcbeb30c919a8cd36b189f562b436689f1910a9863417894da16fe5f730b8": failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  17s   kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "63493acdabb0936dd539423740917614ff0037b85f3ae214797df4c980de8a51": failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  0s    kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1f1461a15f879b63c09ca6432f50704fe9d1af3af220c8af0169d20b03259558": failed to set bridge addr: could not add IP address to "cni0": permission denied

The Error if we delete kic overlay for crio

8-ttxg5_kube-system_bc731488-5665-4038-b5c4-f0c004033182_0(d00b42a39dd99b43b935861a415e15d5e79faf074f08aaccdff7813ece173571): failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  2s    kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-ttxg5_kube-system_bc731488-5665-4038-b5c4-f0c004033182_0(99ba05ba72455dc22c5a6f41da697020eda88200db897a8e924a20997cd4fea0): failed to set bridge addr: could not add IP address to "cni0": permission denied
medmac@~/workspace/minikube (remove_overlay) $ ./out/minikube  profile list
|----------|-----------|---------|------------|------|---------|---------|
| Profile  | VM Driver | Runtime |     IP     | Port | Version | Status  |
|----------|-----------|---------|------------|------|---------|---------|
| minikube | docker    | crio    | 172.17.0.2 | 8443 | v1.18.0 | Running |
|----------|-----------|---------|------------|------|---------|---------|

[medyagh]

I believe only @josedonizetti is expert enough in networking to handle this !

[medyagh]

related: #7354

[tstromberg]

This may get resolved by work on #7459

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.