Add two kubelet settings - topologyManagerPolicy and topologyManagerScope #1659

gthao313 · 2021-07-20T00:28:28Z

Issue number:
#1597

Description of changes:
Pass two new arguments to kubelet topologyManagerPolicy and topologyManagerScope

expose kubernetes.topology-manager-policy as a setting
expose kubernetes.topology-manager-scope as a setting
add migration for kubernetes.topology-manager-policy and kubernetes.topology-manager-scope

Testing done:

Test behavioral effect of new setting topology-manager-policy
Test behavioral effect of new setting topology-manager-scope
Test migration process - cpu-manager-policy cpu-manager-reconcile-policy

`topology-manager-policy`

launching instance with the AMI which contains new setting and check if topology-manager-policy has been configured expectantly.

Step1 check if topology-manager-policy has been configured expectantly:

Test with topology-manager-policy= none

1737 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"

Test with default ("none")

1737 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"

Test with topology-manager-policy= best-effort

1730 container_manager_linux.go:314] "Initializing Topology Manager" policy="best-effort" scope="container"

Test with topology-manager-policy= restricted

1729 container_manager_linux.go:314] "Initializing Topology Manager" policy="restricted" scope="container"

Test with topology-manager-policy= single-numa-node

1736 container_manager_linux.go:314] "Initializing Topology Manager" policy="single-numa-node" scope="container"

Step2: Launched nginx pods/containers

`topology-manager-scope`

launching instance with the AMI which contains new setting and check if topology-manager-scope has been configured expectantly.

Step1 check if topology-manager-scope has been configured expectantly:

Test with topology-manager-scope = container

1726 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"

Test with topology-manager-scope = pod

1725 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="pod"

Test with default

1737 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"

Step2: Launched nginx pods/containers

Migration test:

upgrade
Step1: Upgrade to v1.2.0

bash-5.0# updog check-update -a --json
[
  {
    "variant": "aws-k8s-1.21",
    "arch": "x86_64",
    "version": "1.2.0",
    "max_version": "1.2.0",
.....

bash-5.0# updog update -i 1.2.0 -r -n
Starting update to 1.2.0

Step2: Specify new setting topology-manager-policy and topology-manager-scope through control container

apiclient set -j '{"kubernetes": {"topology-manager-policy": "restricted"}}'
apiclient set -j '{"kubernetes": {"topology-manager-scope": "pod"}}'

<atastore/current/live/settings/kubernetes/topology-manager-policy
"restricted"
<e/current/live/settings/kubernetes/topology-manager-scope
"pod"

journalctl -u kubelet
4488 container_manager_linux.go:314] "Initializing Topology Manager" policy="restricted" scope="pod"

downgrade
Step1: Check migration binary

ls -al /var/lib/bottlerocket-migrations
-rw-r--r--.  1 root root 477627 Jul 20 00:18 e9968ee60b449f1767b025034356dfca9c37ffe34e6a97fa8c176f169b698af1.migrate_v1.2.0_kubelet-topology-manager.lz4

Step2: Downgrade to previous verison

signpost rollback-to-inactive
reboot

Step3: Check if topology-manager-policy and topology-manager-scope have been removed

cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/topology-manager-policy
cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/topology-manager-policy: No such file or directory
cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/topology-manager-scope
cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/topology-manager-scope: No such file or directory

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

webern · 2021-07-20T17:52:45Z

With some previous settings I have advocated for excluding them from the kube config file when not set rather than 'hardcoding' a default. Should we do that again here? @tjkirch @bcressey My thinking is that it would be better to defer default behavior to kubelet rather than be the ones that determine what the default is.

bcressey · 2021-07-23T04:26:23Z

It would be better to defer default behavior to kubelet rather than be the ones that determine what the default is.

Agreed.

gthao313 · 2021-07-23T19:10:40Z

Push above remove 'hardcode' defaults and update correct release version.

README.md

packages/kubernetes-1.16/kubelet-config

sources/models/src/modeled_types/mod.rs

gthao313 · 2021-07-26T18:31:19Z

Push above removed those two new settings in packages/kubernetes-1.16/kubelet-config since kubernetes-1.16 will be deprecated in next release.

arnaldo2792 · 2021-07-27T21:29:36Z

Release.toml

@@ -59,3 +59,6 @@ version = "1.1.4"
    "migrate_v1.1.3_kubelet-cpu-manager.lz4",
 ]
 "(1.1.3, 1.1.4)" = []
+"(1.1.4, 1.1.5)" = [


Shouldn't this be 1.2.0? Since this is a new feature @jhaynes / @tjkirch?

We should have all unreleased migrations under the same version pair so that, when we test upgrading and downgrading, all the migrations will run. In the event of a 'patch' level release, it may be necessary to edit Release.toml in the event that some migrations need to go into the patch.

As for what the unreleased version represented in Release.toml should be, I'm not sure. I don't think it matters as long as we agree (i.e. there are a few PRs open that should all agree).

pass topology-manager-scope argument to kubelet

pass topology-manager-policy argument to kubelet

…gerScope adds migration for two new settings `settings.kubernetes.topology-manager-scope` and `settings.kubernetes.topology-manager-policy`

gthao313 · 2021-07-30T22:46:53Z

Push above fix release version (change 1.1.5 to 1.2.0) and pass migration test.

arnaldo2792

🥾

gthao313 requested review from bcressey, jpculp, webern and etungsten July 20, 2021 00:28

gthao313 force-pushed the kubelet-settings branch from 81c8a93 to 95f2858 Compare July 23, 2021 19:04

gthao313 marked this pull request as draft July 23, 2021 19:04

gthao313 force-pushed the kubelet-settings branch from 95f2858 to f0629ee Compare July 23, 2021 19:07

gthao313 marked this pull request as ready for review July 23, 2021 19:10

bcressey approved these changes Jul 25, 2021

View reviewed changes

README.md Show resolved Hide resolved

packages/kubernetes-1.16/kubelet-config Outdated Show resolved Hide resolved

sources/models/src/modeled_types/mod.rs Outdated Show resolved Hide resolved

sources/models/src/modeled_types/mod.rs Outdated Show resolved Hide resolved

bcressey changed the title ~~Add twoKubelet settings topologyManagerPolicy and topologyManagerScope~~ Add two kubelet settings - topologyManagerPolicy and topologyManagerScope Jul 25, 2021

gthao313 force-pushed the kubelet-settings branch from f0629ee to 2759b93 Compare July 26, 2021 18:27

gthao313 requested a review from bcressey July 26, 2021 18:32

etungsten approved these changes Jul 27, 2021

View reviewed changes

jpculp approved these changes Jul 27, 2021

View reviewed changes

arnaldo2792 reviewed Jul 27, 2021

View reviewed changes

webern approved these changes Jul 27, 2021

View reviewed changes

bcressey approved these changes Jul 29, 2021

View reviewed changes

gthao313 added 3 commits July 30, 2021 22:30

kubelet: add setting for configuring TopologyManagerScope

d0de57c

pass topology-manager-scope argument to kubelet

kubelet: add setting for configuring TopologyManagerPolicy

1c60879

pass topology-manager-policy argument to kubelet

migrations: add migrations for topologyManagerPolicy and topologyMana…

8af98dc

…gerScope adds migration for two new settings `settings.kubernetes.topology-manager-scope` and `settings.kubernetes.topology-manager-policy`

gthao313 force-pushed the kubelet-settings branch from 2759b93 to 8af98dc Compare July 30, 2021 22:45

gthao313 requested a review from arnaldo2792 July 30, 2021 22:47

arnaldo2792 approved these changes Jul 30, 2021

View reviewed changes

gthao313 merged commit 4fc0c2b into bottlerocket-os:develop Jul 30, 2021

jhaynes linked an issue Aug 2, 2021 that may be closed by this pull request

Add kubelet options - topologyManagerPolicy and topologyManagerScope #1597

Closed

bcressey mentioned this pull request Apr 26, 2022

Support for kubelet config options: imageGCHighThresholdPercent, imageGCLowThresholdPercent #2065

Closed

diranged mentioned this pull request Apr 4, 2023

Add in support for the --memory-manager-policy flag? #2977

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add two kubelet settings - topologyManagerPolicy and topologyManagerScope #1659

Add two kubelet settings - topologyManagerPolicy and topologyManagerScope #1659

gthao313 commented Jul 20, 2021 •

edited

Loading

webern commented Jul 20, 2021

bcressey commented Jul 23, 2021

gthao313 commented Jul 23, 2021

gthao313 commented Jul 26, 2021

arnaldo2792 Jul 27, 2021

webern Jul 27, 2021

gthao313 commented Jul 30, 2021

arnaldo2792 left a comment

Add two kubelet settings - topologyManagerPolicy and topologyManagerScope #1659

Add two kubelet settings - topologyManagerPolicy and topologyManagerScope #1659

Conversation

gthao313 commented Jul 20, 2021 • edited Loading

topology-manager-policy

topology-manager-scope

Migration test:

webern commented Jul 20, 2021

bcressey commented Jul 23, 2021

gthao313 commented Jul 23, 2021

gthao313 commented Jul 26, 2021

arnaldo2792 Jul 27, 2021

Choose a reason for hiding this comment

webern Jul 27, 2021

Choose a reason for hiding this comment

gthao313 commented Jul 30, 2021

arnaldo2792 left a comment

Choose a reason for hiding this comment

gthao313 commented Jul 20, 2021 •

edited

Loading

`topology-manager-policy`

`topology-manager-scope`