Add aws-k8s-1.20 variant with Kubernetes 1.20 support #1437

etungsten · 2021-04-01T18:43:54Z

Issue number:
#1429
#1386

Description of changes:

For reference, the addition of 1.19 was in #1256.

Version of kubelet packaged: v1.20.6

Author: Erikson Tung <[email protected]>
Date:   Wed Mar 31 17:37:47 2021 -0700

    Add aws-k8s-1.20 variant with Kubernetes 1.20

Author: Erikson Tung <[email protected]>
Date:   Thu Apr 1 10:59:47 2021 -0700

    kubernetes-1.20: constrain ip_local_port_range
    
    Constrains ephemeral ports for k8s-1.20 variants

Regenerated the 0001-always-set-relevant-variables-for-cross-compiling.patch patch.

Added license clarification for "github.com/storageos/go-api"

Testing done:

Pods run OK
Conformance testing with 1.20 control plane and x86_64 aws-k8s-1.20 nodes - all passed
Conformance testing with 1.20 control plane and aarch64 aws-k8s-1.20 nodes - all passed
Confirm the EBS CSI driver still works.
Check kubelet logs and system logs in general, compared it with the aws-k8s-1.19 variant kubelet logs and found no abnormalities.

Confirmed that the ephemeral port range has been constrained:

[ec2-user@ ~]$ sysctl -a | grep net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

packages/Cargo.toml

packages/kubernetes-1.20/kubelet-sysctl.conf

packages/kubernetes-1.20/kubernetes-1.20.spec

tjkirch · 2021-04-05T23:11:37Z

packages/kubernetes-1.20/0001-always-set-relevant-variables-for-cross-compiling.patch

+-    case "${platform}" in
+-      "linux/arm")
+-        export CGO_ENABLED=1
+-        export CC=${KUBE_LINUX_ARM_CC:-arm-linux-gnueabihf-gcc}


Hmm. In previous versions, these were hardcoded, so it made sense to unilaterally override them with the -bottlerocket- versions we need. Is there any reason we'd want to allow the $KUBE_LINUX_ARM_CC-type overrides in our replacement code? I don't have background on these.

@bcressey can you take a look at this?

Upstream is now a bit closer to what we want, which is to always use our known cross compilers. We're still missing a way to express "always cross-compile, even if we're on linux/amd64". I expect we could come up with a patch to bridge what's left.

For now, your port of the patch looks good.

packages/kubernetes-1.20/clarify.toml

packages/kubernetes-1.20/kubelet-config

packages/kubernetes-1.20/kubelet.service

sources/models/src/aws-k8s-1.20

packages/kubernetes-1.20/kubelet-config

etungsten · 2021-04-16T18:54:28Z

Push above rebases onto develop.
Create separate models files for aws-k8s-1.20.

etungsten · 2021-04-19T21:35:37Z

Push above addresses majority of @tjkirch comments.

Replays several changes:
- k8s settings: kube-reserved, eviction-hard, cpu-manager-policy, allow-unsafe-sysctls #1388
- k8s: Improve kubeReserved to handle Null max-pods #1472
Fixes the license info in clarify.toml
Adds aws-k8s-1.20 to the list of variants in the README
Updates documentation for the sysctl file.
Changes the prefix of the destination filename for the sysctl-conf file.

For #1465, we decided offline that this will be the first variant where we introduce cpuManagerPolicy: static as a hardcoded default.

etungsten · 2021-04-19T21:48:14Z

Push above rebases onto develop to get changes from #1408 and #1494

etungsten · 2021-04-19T21:55:58Z

Push above applies changes from #1408 to the aws-k8s-1.20 variant.

etungsten · 2021-04-19T21:59:13Z

Push above adds the missing Cargo.lock changes.

etungsten · 2021-04-19T22:02:37Z

Push above applies changes from #1494 to the aws-k8s-1.20 variant.

etungsten · 2021-04-19T22:47:25Z

Push above adopts #1295 for aws-k8s-1.20

etungsten · 2021-04-19T23:57:51Z

Push above updates the kubernetes-1.20 package to Kubernetes v1.20.6

variants/README.md

sources/models/src/aws-k8s-1.20/mod.rs

sources/models/README.md

sources/models/src/lib.rs

etungsten · 2021-04-22T00:42:42Z

Push above addresses @tjkirch 's comments.

Splits out kernel lockdown default setting into its own separate shared default toml file.
Symlinks aws-k8s-1.20's 50-aws-k8s.toml to aws-k8s-1.19's 50-aws-k8s.toml
Fixes documentation

I ran sonobuoy conformance testing for kubelet v1.20.6 and everything still passed. The testing description in the PR is up to date.

etungsten · 2021-04-22T21:11:57Z

Push above adds a commit that adds systemd.unified_cgroup_hierarchy=1 to aws-k8s-1.20's kernel parameters.

I've built and image and verified that it works:

bash-5.0# dmesg
[    0.000000] Linux version 5.4.105 (builder@buildkitsandbox) (gcc version 9.3.0 (Buildroot 2020.02.2)) #1 SMP Tue Apr 20 00:03:24 UTC 2021
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz systemd.unified_cgroup_hierarchy=1 ....

The host is running with a unified cgroup hierarchy (cgroupv2)

bash-5.0# mount -l 
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel)

kubelet is running fine, the node joins the cluser fine, can run pods.

variants/aws-k8s-1.20/Cargo.toml

arnaldo2792

👢

etungsten · 2021-04-23T16:34:38Z

Push above integrates changes from #1513

Constrains ephemeral ports for k8s-1.20 variants

Adds `systemd.unified_cgroup_hierarchy=1` to the kernel parameters for the aws-k8s-1.20 variant.

etungsten · 2021-04-23T17:02:22Z

Push above rebases onto current develop to be on top of the changes from #1513.

Tested things they still work.

jhaynes linked an issue Apr 1, 2021 that may be closed by this pull request

Kubernetes 1.20 support #1429

Closed

etungsten added this to the next+1 milestone Apr 1, 2021

etungsten force-pushed the k8s-1.20 branch from b021051 to ba610aa Compare April 5, 2021 20:46

jhaynes modified the milestones: next+1, next Apr 5, 2021

etungsten removed this from the next milestone Apr 5, 2021

etungsten linked an issue Apr 5, 2021 that may be closed by this pull request

Setting for local_port_range overlaps with very common listen ports #1386

Closed

tjkirch reviewed Apr 5, 2021

View reviewed changes

tjkirch reviewed Apr 9, 2021

View reviewed changes

packages/kubernetes-1.20/kubelet-config Show resolved Hide resolved

etungsten force-pushed the k8s-1.20 branch from ba610aa to 0cfd29b Compare April 16, 2021 18:53

etungsten force-pushed the k8s-1.20 branch from 0cfd29b to 9c9e2cb Compare April 19, 2021 21:16

etungsten force-pushed the k8s-1.20 branch 2 times, most recently from 05c123d to 18e75f4 Compare April 19, 2021 21:47

etungsten force-pushed the k8s-1.20 branch from 18e75f4 to 2bc575e Compare April 19, 2021 21:54

etungsten force-pushed the k8s-1.20 branch from 2bc575e to e48737f Compare April 19, 2021 21:58

etungsten force-pushed the k8s-1.20 branch from e48737f to 2edc277 Compare April 19, 2021 22:01

etungsten force-pushed the k8s-1.20 branch from 2edc277 to cc696d0 Compare April 19, 2021 22:46

etungsten force-pushed the k8s-1.20 branch from cc696d0 to 92abae6 Compare April 19, 2021 23:54

etungsten marked this pull request as ready for review April 20, 2021 02:18

etungsten requested review from bcressey, tjkirch and webern April 20, 2021 02:19

etungsten requested a review from arnaldo2792 April 20, 2021 02:19

webern reviewed Apr 20, 2021

View reviewed changes

variants/README.md Show resolved Hide resolved

tjkirch reviewed Apr 21, 2021

View reviewed changes

sources/models/src/aws-k8s-1.20/mod.rs Show resolved Hide resolved

sources/models/README.md Outdated Show resolved Hide resolved

sources/models/src/lib.rs Outdated Show resolved Hide resolved

etungsten force-pushed the k8s-1.20 branch from 92abae6 to d7cb4fa Compare April 22, 2021 00:29

etungsten requested review from tjkirch and webern April 22, 2021 00:42

tjkirch approved these changes Apr 22, 2021

View reviewed changes

webern approved these changes Apr 22, 2021

View reviewed changes

tjkirch reviewed Apr 22, 2021

View reviewed changes

variants/aws-k8s-1.20/Cargo.toml Show resolved Hide resolved

bcressey approved these changes Apr 22, 2021

View reviewed changes

arnaldo2792 approved these changes Apr 23, 2021

View reviewed changes

etungsten force-pushed the k8s-1.20 branch from 22f03b7 to c2b21a7 Compare April 23, 2021 16:32

etungsten requested a review from tjkirch April 23, 2021 17:00

etungsten added 3 commits April 23, 2021 10:01

Add aws-k8s-1.20 variant with Kubernetes 1.20

40b4d4f

kubernetes-1.20: constrain ip_local_port_range

7b5e0a8

Constrains ephemeral ports for k8s-1.20 variants

aws-k8s-1.20: enable unified cgroup hierarchy

07f5e84

Adds `systemd.unified_cgroup_hierarchy=1` to the kernel parameters for the aws-k8s-1.20 variant.

etungsten force-pushed the k8s-1.20 branch from c2b21a7 to 07f5e84 Compare April 23, 2021 17:01

tjkirch approved these changes Apr 23, 2021

View reviewed changes

etungsten merged commit acf04d9 into bottlerocket-os:develop Apr 23, 2021

etungsten deleted the k8s-1.20 branch April 23, 2021 17:26

arnaldo2792 mentioned this pull request Apr 26, 2021

enable kernel lockdown in integrity mode #813

Closed

etungsten mentioned this pull request Jun 15, 2021

Add aws-k8s-1.21, vmware-k8s-1.21 variants #1612

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add aws-k8s-1.20 variant with Kubernetes 1.20 support #1437

Add aws-k8s-1.20 variant with Kubernetes 1.20 support #1437

etungsten commented Apr 1, 2021 •

edited

Loading

tjkirch Apr 5, 2021

etungsten Apr 19, 2021

bcressey Apr 21, 2021 •

edited

Loading

etungsten commented Apr 16, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 22, 2021

etungsten commented Apr 22, 2021 •

edited

Loading

arnaldo2792 left a comment

etungsten commented Apr 23, 2021

etungsten commented Apr 23, 2021

Add aws-k8s-1.20 variant with Kubernetes 1.20 support #1437

Add aws-k8s-1.20 variant with Kubernetes 1.20 support #1437

Conversation

etungsten commented Apr 1, 2021 • edited Loading

tjkirch Apr 5, 2021

Choose a reason for hiding this comment

etungsten Apr 19, 2021

Choose a reason for hiding this comment

bcressey Apr 21, 2021 • edited Loading

Choose a reason for hiding this comment

etungsten commented Apr 16, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 19, 2021

etungsten commented Apr 22, 2021

etungsten commented Apr 22, 2021 • edited Loading

arnaldo2792 left a comment

Choose a reason for hiding this comment

etungsten commented Apr 23, 2021

etungsten commented Apr 23, 2021

etungsten commented Apr 1, 2021 •

edited

Loading

bcressey Apr 21, 2021 •

edited

Loading

etungsten commented Apr 22, 2021 •

edited

Loading