Pass additional arguments to the kubelet

[itomaldonado]

What I'd like:
Running on AWS + EKS, the bottlerocket worker nodes are in an ASG and currently start properly and become ready. BBut we would like to be able to pass in extra args to the kubelet (similar to what the eks images can do), for example:

--eviction-hard=memory.available<560Mi
--allowed-unsafe-sysctls net.core.somaxconn,net.ipv4.ip_local_port_range

Any alternatives you've considered:
Building our own image, although seems a bit cumbersome when all that is needed is to add one or one or two extra args to kubectl.

[zmrow]

Thanks for opening this issue! We don't currently have a good solution for arbitrary arguments like this, but it's something that's on our radar.

[ugurarpaci]

This will be pretty useful! Adding these flags will satisfy the benchmark tools like kube-bench

[itomaldonado]

@zmrow that's fair, I would love to follow along an issue or something that tracks that functionality, if there is one (and could be linked here). If so you can also feel free to close this. If there isn't one, could this one track this functionality?

As @ugurarpaci mentioned there are quite a few things we could control in our clusters by just changing kubelet flags instead of having to build an image for each.

[bcressey]

We don't have plans to allow arbitrary arguments to kubelet, but we can and should model arguments like this as settings.

The reason we'd like to avoid arbitrary arguments is so that we can better test and predict kubelet behavior across upgrades and downgrades; otherwise removing the argument or changing how it's interpreted could break the system in ways that we can't anticipate.

This is especially a concern when we introduce support for cross-variant upgrades, for example from Kubernetes 1.18 to 1.19. Having a model for the settings also lets us specify a migration path for them - causing the setting to go away on upgrade and reappear on downgrade, or moving it from a command-line argument into the kubelet config.

Are these the only two flags you're looking for, or are there others?

[svedatbegar]

As an addition to parameters mentioned above, we use kube-reserved parameters to reduce allocatable resources on nodes so there's always enough CPU, memory, ephemeral storage and pid for stability. So far I couldn't see a way to set them.

[supasteev0]

Just wanted to share my experience in testing bottlerocket AMI:
We currently use the following kubelet args with the standard EKS AMI:

• allowedUnsafeSysctls: ["net.*"]
• cpuManagerPolicy: "static"

The later is a must have for us as we handle CPU pinning for some of our pods via a Guaranteed Qos class.
Would love to be able to do the same with the Bottlerocket AMI but for that we'd need to be able to configure those args in the toml file.

[backjo]

We'd also like to see system-reserved added

[diranged]

Just adding support here - we have a number of settings around the eviction rules that we want to tweak. While I understand the argument that you want Bottlerocket to be testable and predictable, I would argue that it really isn't your place to dictate which Kubelet settings that we can implement or not. Sure, you can set defaults that are testable for your purposes, but the users should be able to make the determination of which settings are critical. Otherwise you are going to be in a constant battle playing catchup. :/

[anguslees]

My 2c (and I'm nothing to do with BR team, so this is a suggestion only):

--eviction-hard=memory.available<560Mi (or some similar value) should just be set by default in BR. Whatever the right 'reserved' memory/cpu is, it's a fixed/known quantity (calculated using max-pods) for a fixed/known BR image. Once we start using the 'right' value, there should be no reason to adjust it afaics? (And we should probably use system-reserved rather than eviction-hard, or some combination of both.)

--allowed-unsafe-sysctls net.core.somaxconn,net.ipv4.ip_local_port_range is definitely something determined by local policy choices and should be exposed as a settings option.

--cpu-manager-policy we could just unilaterally decide to enable (set to static always), since it's really a feature maturity rather than policy question. If we want to allow people to 'opt in' to things like this ahead of the BR-team's unilateral decision then we could maybe introduce a 'feature flags' concept like upstream k8s (possibly using the exact same flags as upstream k8s).

I think my meta-suggestion is that these options are better considered separately, since the desired solution might be different. I think we should close/split this specific github issue, because I don't think treating them the same just because they happen to be kubelet command line flags is appropriate.

---

Otoh, I share the underlying "I just want to control my own destiny" sentiment behind this feature request. I look forward to 'BR as a Service', where I can trivially build my own BR variant AMI by applying my own arbitrary patch/overlay, and voiding all warranties in the process. I could probably do that right now using some CDK and a codebuild pipeline 🤔 ...

[jhaynes]

We've added kube-reserved, eviction-hard, cpu-manager-policy and allow-unsafe-sysctls and they'll be available in our next release. The only remaining item is system-reserved which is being tracked in #1447 . I'm going to resolve this and we can cut new issues for individual (or related groups) of kubelet settings.

[awoimbee]

We don't have plans to allow arbitrary arguments to kubelet

This means that as an advanced user I can't test a config before it is validated, tested, merged by the bottlerocket team. E.g. we could have had experimental swap support in EKS 2 years ago, to first test and then formally add to Bottlerocket.

Note that it would be OK and expected for this escape hatch to:

• be buried deep in the docs, with warnings
• be unsupported (voids any kind of upgrade policy or support plan, arbitrary closure of github issues, ...)