Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unify Default CNI for VM and containers #7428

Closed
medyagh opened this issue Apr 5, 2020 · 7 comments
Closed

unify Default CNI for VM and containers #7428

medyagh opened this issue Apr 5, 2020 · 7 comments
Assignees
Labels
area/cni CNI support kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@medyagh
Copy link
Member

medyagh commented Apr 5, 2020

currently we have two CNIs

https://github.com/kubernetes/minikube/blob/a9558856c54cc0b3b6c46a7ef96960f521a72fb6/pkg/minikube/bootstrapper/kubeadm/default_cni.go

  • defaultCNIConfig
  • kicCNIConfig

kicCNIConfig is only used by containerd and crio only on docker driver.
I have a feeling by tweaking the defaultCNIConfig we could get rid of the kicCNIConfig.

the defaultCNIConfig has "subnet": "10.1.0.0/16",

// The config is being written to /etc/cni/net.d/k8s.conf.
const defaultCNIConfig = `
{
  "cniVersion": "0.3.0",
  "name": "rkt.kubernetes.io",
  "type": "bridge",
  "bridge": "mybridge",
  "mtu": 1460,
  "addIf": "true",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.1.0.0/16",
    "gateway": "10.1.0.1",
    "routes": [
      {
        "dst": "0.0.0.0/0"
      }
    ]
  }
}
`

but for kic drivers we add 10.244.0.0/16

🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16

The Error if we delete kic overlay for cotainerd on docker driver.

medmac@~/workspace/minikube (remove_overlay) $ kc get pods -A
NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
kube-system   coredns-66bff467f8-l7qkl           0/1     ContainerCreating   0          96s
kube-system   coredns-66bff467f8-vf6bm           0/1     ContainerCreating   0          96s
kube-system   etcd-minikube                      1/1     Running             0          112s
kube-system   kube-apiserver-minikube            1/1     Running             0          112s
kube-system   kube-controller-manager-minikube   1/1     Running             0          112s
kube-system   kube-proxy-t424s                   1/1     Running             0          96s
kube-system   kube-scheduler-minikube            1/1     Running             0          112s
kube-system   storage-provisioner                1/1     Running             0          110s

and here is the descripbe which says:
'failed to set bridge addr: could not add IP address to "cni0": permission denied'

medmac@~/workspace/minikube (remove_overlay) $ kc describe pod coredns-66bff467f8-vf6bm -n kube-system
Name:                 coredns-66bff467f8-vf6bm
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 minikube/172.17.0.2
Start Time:           Sun, 05 Apr 2020 00:28:49 -0700
Labels:               k8s-app=kube-dns
                      pod-template-hash=66bff467f8
Annotations:          <none>
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-66bff467f8
Containers:
  coredns:
    Container ID:
    Image:         k8s.gcr.io/coredns:1.6.7
    Image ID:
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-4pcrq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-4pcrq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-4pcrq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Warning  FailedScheduling        37s   default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               37s   default-scheduler  Successfully assigned kube-system/coredns-66bff467f8-vf6bm to minikube
  Warning  FailedCreatePodSandBox  34s   kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f12bcbeb30c919a8cd36b189f562b436689f1910a9863417894da16fe5f730b8": failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  17s   kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "63493acdabb0936dd539423740917614ff0037b85f3ae214797df4c980de8a51": failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  0s    kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1f1461a15f879b63c09ca6432f50704fe9d1af3af220c8af0169d20b03259558": failed to set bridge addr: could not add IP address to "cni0": permission denied

The Error if we delete kic overlay for crio

8-ttxg5_kube-system_bc731488-5665-4038-b5c4-f0c004033182_0(d00b42a39dd99b43b935861a415e15d5e79faf074f08aaccdff7813ece173571): failed to set bridge addr: could not add IP address to "cni0": permission denied
  Warning  FailedCreatePodSandBox  2s    kubelet, minikube  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-ttxg5_kube-system_bc731488-5665-4038-b5c4-f0c004033182_0(99ba05ba72455dc22c5a6f41da697020eda88200db897a8e924a20997cd4fea0): failed to set bridge addr: could not add IP address to "cni0": permission denied
medmac@~/workspace/minikube (remove_overlay) $ ./out/minikube  profile list
|----------|-----------|---------|------------|------|---------|---------|
| Profile  | VM Driver | Runtime |     IP     | Port | Version | Status  |
|----------|-----------|---------|------------|------|---------|---------|
| minikube | docker    | crio    | 172.17.0.2 | 8443 | v1.18.0 | Running |
|----------|-----------|---------|------------|------|---------|---------|
@medyagh
Copy link
Member Author

medyagh commented Apr 5, 2020

I believe only @josedonizetti is expert enough in networking to handle this !

@medyagh
Copy link
Member Author

medyagh commented Apr 7, 2020

related: #7354

@priyawadhwa priyawadhwa added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/feature Categorizes issue or PR as related to a new feature. labels Apr 8, 2020
@tstromberg
Copy link
Contributor

This may get resolved by work on #7459

@tstromberg tstromberg added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. area/cni CNI support and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels May 28, 2020
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 26, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 25, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cni CNI support kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

6 participants