[CompatibilityVersion] Unconventional use of feature gates

[siyuanfoundation]

Throughout the code base, instead of just using the feature gates to check if a feature is enabled or not, there are several unconventional use of feature gates:

• kubernetes/staging/src/k8s.io/component-base/logs/api/v1/kube_features.go

  Lines 64 to 65 in df030f3

  	LoggingAlphaOptions: {Default: false, PreRelease: featuregate.Alpha},
  	LoggingBetaOptions: {Default: true, PreRelease: featuregate.Beta},

  are used to promote new format options configurable.

• kubernetes/pkg/features/kube_features.go

  Lines 98 to 116 in df030f3

  	CPUManagerPolicyAlphaOptions featuregate.Feature = "CPUManagerPolicyAlphaOptions"
  	// owner: @fromanirh
  	// beta: see below.
  	//
  	// Allow fine-tuning of cpumanager policies, experimental, beta-quality options
  	// Per https://groups.google.com/g/kubernetes-sig-architecture/c/Nxsc7pfe5rw/m/vF2djJh0BAAJ
  	// We want to avoid a proliferation of feature gates. This feature gate:
  	// - will guard *a group* of cpumanager options whose quality level is beta.
  	// - is thus *introduced* as beta
  	// - will never graduate to stable.
  	// See https://groups.google.com/g/kubernetes-sig-architecture/c/Nxsc7pfe5rw/m/vF2djJh0BAAJ
  	// for details about the removal of this feature gate.
  	CPUManagerPolicyBetaOptions featuregate.Feature = "CPUManagerPolicyBetaOptions"
  	// owner: @fromanirh
  	//
  	// Allow the usage of options to fine-tune the cpumanager policies.
  	CPUManagerPolicyOptions featuregate.Feature = "CPUManagerPolicyOptions"

  are used to allow for new options to fine-tune the cpumanager policies

• kubernetes/pkg/features/kube_features.go

  Lines 684 to 700 in df030f3

  	TopologyManagerPolicyAlphaOptions featuregate.Feature = "TopologyManagerPolicyAlphaOptions"
  	// owner: @PiotrProkop
  	// kep: https://kep.k8s.io/3545
  	//
  	// Allow fine-tuning of topology manager policies with beta options.
  	// This feature gate:
  	// - will guard *a group* of topology manager options whose quality level is beta.
  	// - is thus *introduced* as beta
  	// - will never graduate to stable.
  	TopologyManagerPolicyBetaOptions featuregate.Feature = "TopologyManagerPolicyBetaOptions"
  	// owner: @PiotrProkop
  	// kep: https://kep.k8s.io/3545
  	//
  	// Allow the usage of options to fine-tune the topology manager policies.
  	TopologyManagerPolicyOptions featuregate.Feature = "TopologyManagerPolicyOptions"

  are used to allow for new options to fine-tune the topology manager policies.

These feature gate do not progress in lifecycle, while the underlying available options progress over the stages.
As it is right now, we would not be able to emulate these options in compatibility mode. For example, if a new option is moved from Beta to GA feature from 1.32 to 1.33, this option would be available as a GA option when we are emulating 1.32.

Ideally, we would want these new options to be tracked by a dedicated feature gate for each option.
If that's not feasible, we would need a mechanism to track when these options are moved from one stage to another.
Otherwise, these options are not emulatible, and we will always have the options at the binary version, even when the emulation version is different.

[siyuanfoundation]

/cc @yongruilin
/triage accepted

[thisisharrsh]

/sig node
/sig architecture

[thisisharrsh]

/kind feature