kubelet: add image GC threshold settings

[mchaker]

Issue number:
Closes #2216
Closes #2217
Closes #2065

Description of changes:
Adds two kubelet settings: imageGCHighThresholdPercent and imageGCLowThresholdPercent.
Also adds validation for those two settings.

From the kubernetes repo (https://pkg.go.dev/k8s.io/kubernetes/pkg/kubelet/apis/config#KubeletConfiguration):

// imageGCHighThresholdPercent is the percent of disk usage after which
// image garbage collection is always run. The percent is calculated as
// this field value out of 100.

// imageGCLowThresholdPercent is the percent of disk usage before which
// image garbage collection is never run. Lowest disk usage to garbage
// collect to. The percent is calculated as this field value out of 100.

Some Q&A:

1. Why validate twice: at the apiserver level, and the template rendering level?
1. There are two kinds of invalidity for the settings this PR adds: independent invalidity and interdependent invalidity. Independent invalidity would be a value outside of the defined 0-100 threshold. Interdependent invalidity is when the High threshold percent is lower than the Low threshold percent, and is only calculable by comparing (or knowing) both settings at the same time.
1. The independent invalidity can and should be checked as soon as possible and stop the user from entering invalid values. That kind of validation can and is done at the apiserver level.
1. The interdependent invalidity is only able to be checked when the template is about to be rendered -- when both setting values are known (since the user can provide only one of the two new settings in an apiserver request, and these settings also have default kubelet values to initially compare against).
1. Why does your template code contain ~ characters at each end of the helper function call?
1. That is a handlebars feature that removes whitespace (including newlines) before or after that ~ character.
1. In the case of this PR, the helper function image_gc_threshold_percent actually handles the rendering of necessary whitespace after writing out the setting values (specifically: the newlines after each setting), so the template rendering engine/handlebars should remove the existing newline it sees at the end of the helper function call (after the }} at the end of the line). Therefore, the ~ feature of handlebars was used.

Testing done:

• Upgrade/downgrade for variant: aws-k8s-1.22
• [ ] Upgrade/downgrade for variant: metal 1.22
• Rendering and setting updates for variant: vmware 1.22
• Manual testing of valid and invalid values, in varying order, on a bottlerocket aws-k8s-1.22 node in a k8s 1.22 cluster

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[mchaker]

Latest force push: consolidate branch history. Summary of changes below.

apiserver and schnauzer: Add validation for new kubelet settings imageGCHighThresholdPercent and imageGCLowThresholdPercent.

Description

• apiserver: add new setting types to the model with basic apiserver-level validation
• schnauzer: add template rules and more in-depth validation for new settings

What settings?

in the kubelet-config template:

• imageGCHighThresholdPercent and imageGCLowThresholdPercent

How and why

Add value validation to both the apiserver and template (via schnauzer), since these settings are interdependently validated. Meaning, since the High Image GC Threshold Percent must be greater than the Low Image GC Threshold Percent, both settings must be checked against each other. That is not possible at the apiserver level, or at least is much easier to do in the template rendering level (once all the values from the datastore are ready to be written).

[mchaker]

Latest force push: remove setting validation from template rendering layer. Remove helper function in template rendering layer. Remove interdependent validation entirely, but keep independent validation. Clarify kubelet config template (leave setting names and defaults in the template, instead of in Rust code).

[mchaker]

Latest force push: remove remaining unused constants.

[mchaker]

Latest force push: clean up overkill tests (remove unnecessary for loop), remove unnecessary comments, squashed commits to 1 commit, and chose to keep the {{#if}} path of settings validation instead of also using the default helper function

[mchaker]

Latest force push: fix commit message

[bcressey]

Code change LGTM.

For writing commit messages, I tend to roughly follow this guide with some minor changes:
https://cbea.ms/git-commit/

Minor changes:

• I don't capitalize the subject
• I start with a "subsystem: blah" tag unless the change is very cross-cutting

So for this commit I would say:

kubelet: add image GC threshold settings

And combine your current subject and body into a new body wrapped at 70 characters.

[mchaker]

Code change LGTM.

For writing commit messages, I tend to roughly follow this guide with some minor changes: https://cbea.ms/git-commit/

Minor changes:

• I don't capitalize the subject
• I start with a "subsystem: blah" tag unless the change is very cross-cutting

So for this commit I would say:

kubelet: add image GC threshold settings

And combine your current subject and body into a new body wrapped at 70 characters.

Thank you @bcressey ! I've clarified the commit message and PR title.

[etungsten]

Can you please update the PR description to reflect the latest state of the PR? Some parts of it is out of date.

[mchaker]

Last 3 git force-pushes: added README documentation for the new kubernetes settings and updated Release.toml to include the latest migration.

[mchaker]

Latest force push: remove linter formatting-on-save unrelated changes

[mchaker]

Latest force push: Fixed README.md getting over-tuned by my linter. Also recovered my PR from having too many commits squashed into it accidentally.

[etungsten]

🍨

[mchaker]

Latest force push: had to rebase my branch/PR due to merge conflicts with develop/HEAD

[bcressey]

This doesn't sound correct; my reading of the documentation is that the garbage collector will run less frequently - not always, but not never - if disk usage percentage falls between high and low.

I would also avoid phrases like "In short". Maybe just strike the second sentence from each since you're introducing an abstract concept - whether or not the disk is too full or empty enough - to what would otherwise be a precise description of how the setting controls the kubelet garbage collector.

[mchaker]

You are correct.

EDIT: Will use the following, as per the kubelet config documentation:

settings.kubernetes.image-gc-high-threshold-percent: the percent of disk usage after which image garbage collection is always run.

settings.kubernetes.image-gc-low-threshold-percent: the percent of disk usage before which image garbage collection is never run.

[mchaker]

Latest force push: make README documentation accurate to kubelet behavior as per its documentation

[mchaker]

Latest force push: merge in the latest from develop