update flanneld and cni to support ipv6 nat

[KhooHaoYit]

Summary

This PR update flanneld and cni to support ipv6 nat
It also include update to migrate from etcdv2 to v3

The motive of this PR is to let microk8s able to communicate with service over ipv6 with some extra configuration in non-ha cluster

Changes

Update flanneld to v0.21.2 and cni to v1.2.0
Added flannel-cni-plugin components because it got moved out of cni (containernetworking/plugins#633)
Added etcdutl migrate to migrate old etcdv2 data to v3

Testing

Tested etcdv2 to v3 migration by upgrading from 1.25/stable#v1.25.6 to 1.26.1-dirty

Test if IPv6 works

snap install microk8s_v1.26.1_amd64.snap --classic --dangerous
microk8s disable ha-cluster --force
microk8s enable dns
nano /var/snap/microk8s/current/args/flannel-network-mgr-config
The content should be:

{
  "Network": "10.1.0.0/16",
  "Backend": {
    "Type": "vxlan"
  },
  "EnableIPv6": true,
  "IPv6Network": "fd01::/64"
}

reboot
microk8s.kubectl run -it --rm ipv6 --image=ubuntu -- bash
apt update && apt install curl -y && curl ipv6.icanhazip.com
It should return the host ipv6 after running the last command

Possible Regressions

Multi-cluster environment isn't tested yet

Checklist

• Read the contributions page.
• Submitted the CLA form, if you are a first time contributor.
• The introduced changes are covered by unit and/or integration tests.

Notes

• Reimplemented etcdutl migrate which got removed in Split etcdctl into etcdctl (public API access) & etcdutl (direct surgery on files) etcd-io/etcd#12971
• cni needs to update to at least v1.0.1 due to this requirement
• The flannel-network-mgr-config configuration documentation can be found here
• These commands can be followed to enable ipv6 ingress which is based on this discussion:
  nano /var/snap/microk8s/current/args/kube-proxy
  Change --cluster-cidr=10.1.0.0/16 to --cluster-cidr=10.1.0.0/16,fd01::/64
nano /var/snap/microk8s/current/args/kube-apiserver
  Change --service-cluster-ip-range=10.152.183.0/24 to --service-cluster-ip-range=10.152.183.0/24,fd98::/108
nano /var/snap/microk8s/current/args/kube-controller-manager
  Add --service-cluster-ip-range=10.152.183.0/24,fd98::/108 and --cluster-cidr=10.1.0.0/16,fd01::/64

[KhooHaoYit]

On the CLA form what should I write in the Please add the Canonical Project Manager or contact field??

[neoaggelos]

Hi @KhooHaoYit, thanks for this PR! The team will look into it!

Feel free to use my name as the Canonical contact name.

[neoaggelos]

I wonder if it would be possible to get to the latest flannel version instead of staying at 0.17.0. We do require the etcd v2 API with https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/run-flanneld-with-args#L33.

Can you test whether we can change that to use ETCDCTL_API=3 instead (and perhaps update the etcdctl commands below accordingly as needed? We should then be able to update flanneld without issue. Thanks!

NOTE: It would be highly appreciated to do the above, but I am also fine with skipping it if you don't feel like it is important for you. Thanks again for the PR!

[neoaggelos]

Also, I see this on the cla-check failure (gmail username redacted):

Checking the following users on Launchpad:
- <your_gmail_account>@gmail.com ✕ (has no Launchpad account)

[KhooHaoYit]

Weird, I think I might've submitted the wrong Launchpad id
Which one is the Launchpad id in this screenshot??

[neoaggelos]

That would be the username, thus khoohaoyit

[KhooHaoYit]

I have resubmitted it just now, try running the cla-check pipeline again

[KhooHaoYit]

Am I missing somthing??
I feel like all the field should be correct

[KhooHaoYit]

Also I have encountered the same bug in #2986 when testing the code
It might be due to having the cluster offline for too long and when it came back online, ingress-nginx keeps saying port 80 is used
The bug might be reproducible via snap restore # with a 1 day old snapshot

Not sure if it's intended or not but there might be another bug where running snap install microk8s after microk8s stop might start up some services

[neoaggelos]

@KhooHaoYit thanks again for the work in this PR!

Unfortunately, I see the CLA check is still failing, not sure why. Let me see if there is something wrong in the check itself.

[neoaggelos]

@KhooHaoYit Can you please confirm what your launchpad ID is? Please share the page of your launchpad account, e.g. https://launchpad.net/~aggkolaitis.

Thanks a lot!

[KhooHaoYit]

@neoaggelos I have created the launchpad ID just now, iirc the Don’t know your Launchpad id? redirected me to https://login.ubuntu.com/ instead of https://login.launchpad.net/ soo my account wasn't registered

Can try running the CLA check again

[KhooHaoYit]

@neoaggelos Do I have to sign the Ubuntu Code of Conduct too or the CLA might take time to process??

[neoaggelos]

Great! Looks like now the account is there. Can you sign the CLA with the Launchpad ID? Looks like it was missed previously since the account did not exist.

Checking the following users on Launchpad:
- [email protected] ✕ (has not signed the CLA)

[KhooHaoYit]

@neoaggelos I have signed the CLA and received a confirmation email just now after filling my email into the Launchpad ID field

The CLA pipeline should work now

[neoaggelos]

Hi @KhooHaoYit,

Thanks a lot for the incredible work with this PR! I have taken the liberty to make a few changes:

• Adjust the run-flanneld-with-args script to only setup etcd in case the etcd-endpoints arguments is set. This enables more scenarios, including using flannel using the Kubernetes Subnet Manager, effectively using the kube-apiserver as store instead of etcd.
• Revert the etcdv2 -> etcdv3 migration, as this is not required for the rest of the work done on this PR. The migration is quite a breaking change which should not be done in a single release, since it may unexpectedly break user workflows that might be depending on etcdv2. Instead, I believe this should done over the course of a few releases, following a deprecation policy (add a deprecation notice, and allow for people to adjust their workflows for a few releases).

I have kept your original branch in https://github.com/canonical/microk8s/tree/KhooHaoYit/etcdv2-removal, which can be used to start incorporating this change in a separate PR. This is even more important given the timeline of the 1.27 MicroK8s release next month.

Merging this PR, and it will be part of the 1.27 release next month. Thanks again!