Refactor cloud provider creation options

[elmiko]

What type of PR is this?

/kind cleanup
/kind api-change

What this PR does / why we need it:

This patch series changes an argument to the NewCloudProvider function to use an AutoscalerOptions struct instead of AutoscalingOptions. This change allows cloud providers to have more control over the core functionality of the cluster autoscaler.

In specific, this patch series also adds a method named RegisterScaleDownNodeProcessor to the AutoscalerOptions so that cloud providers can inject a custom scale down processor.

Lastly, this change adds a custom scale down processor to the clusterapi provider to help it avoid removing the wrong instance during scale down operations that occur during a cluster upgrade.

Which issue(s) this PR fixes:

Fixes #8494

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[elmiko]

this is an alternate solution instead of #8531

[elmiko]

refactored slightly to take suggestions:

• changed register function to util style
• moved cloud provider init to the end of default init
• added TODO for future work

[elmiko]

updated to revert the public scoping on combined mixed node scaled down processor.

[jackfrancis]

/lgtm

/assign @towca

[k8s-ci-robot]

@elmiko: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-autoscaler-e2e-azure-master	51a0514	link	false	/test pull-cluster-autoscaler-e2e-azure-master

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[elmiko]

@jackfrancis is the azure test failure something that i introduced?

[towca]

Thanks for incorporating my feedback, LGTM! Leaving the hold so that you can confirm the e2e test before submitting, feel free to unhold.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: elmiko, jackfrancis, towca

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [jackfrancis,towca]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jackfrancis]

E2E failures are not related to this change: #8681

/hold cancel

[sbueringer]

Thx @elmiko and everyone contributing to this!! Really appreciate it

[jackfrancis]

/kind bug

[jackfrancis]

/cherry-pick cluster-autoscaler-release-1.34

[k8s-infra-cherrypick-robot]

@jackfrancis: #8583 failed to apply on top of branch "cluster-autoscaler-release-1.34":

Applying: refactor core.AutoscalerOptions in a new package
Using index info to reconstruct a base tree...
M	cluster-autoscaler/core/autoscaler.go
Falling back to patching base and 3-way merge...
Auto-merging cluster-autoscaler/core/autoscaler.go
CONFLICT (content): Merge conflict in cluster-autoscaler/core/autoscaler.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 refactor core.AutoscalerOptions in a new package

In response to this:

/cherry-pick cluster-autoscaler-release-1.34

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.