don't use bootconfig for systemd's unified cgroup hierarchy

[bcressey]

In #3886 I switched to using a bootconfig drop-in to select the cgroup hierarchy for systemd, as part of the larger effort to eradicate conditional compilation tracked in #3883.

The bootconfig approach works for newly launched nodes, but I've realized it won't work for in-place updates. The bootconfig data we generate in rpm2img is stored on Bottlerocket's private partition, which isn't touched during updates. What's more, the usual crutch of fixing this in a migration can't be used, because migrations run after the new image has been booted, and systemd will have already set up the cgroup hierarchy by then.

The outcome here will be that a system that was using the unified hierarchy would switch back to the legacy hierarchy after an in-place update - not good!

I'm mulling over alternatives. So far my best idea is:

1. Change systemd to default to the unified cgroup hierarchy. (This is the right long-term change regardless.)
2. In rpm2img, extend the default kernel command line with systemd.unified_cgroup_hierarchy=0 if the unified feature flag is not set.

This would preserve the ability for newer variants to switch back to the legacy hierarchy. The downside is that older variants (aws-ecs-1, aws-k8s-1.23 through 1.25) won't be able to use bootconfig to enable the unified hierarchy. aws-ecs-1 doesn't support bootconfig, so that's no loss. Support for the unified hierarchy in kubelet wasn't marked as GA until 1.25, so 1.23 and 1.24 shouldn't be affected either.

That just leaves the case of nodes running aws-k8s-1.25, using in-place updates, and attempting to use bootconfig to enable the unified hierarchy. My guess is that this is a niche scenario that may not affect any nodes in the wild.

[bcressey]

While testing #3935 I discovered that it's actually possible to use bootconfig to handle the aws-k8s-1.25 case I was worried about.

Kernel command line parameters are divided into two sets - one for the kernel, one for init - and normally these trump anything specified in bootconfig. The kernel will pass unrecognized parameters along to init, under the assumption that they might be intended for that process. However, if bootconfig has a value for an init parameter of the same name, then that will take precedence over the value passed to init, because all init parameters come after all kernel parameters.

So if systemd.unified_cgroup_hierarchy=0 is set as a kernel parameter in the variant definition - even though it is an init parameter - it can be overridden by a bootconfig parameter at runtime.