✨ Cancel instance refresh on any relevant change to ASG instead of blocking until previous one is finished (which may have led to failing nodes due to outdated join token)

[AndiDog]

(reopened from #5173 because I didn't have enough permissions to force-push)

What type of PR is this?

What this PR does / why we need it:

Changing any relevant spec.* for an AWSMachinePool triggers rolling of nodes via ASG instance refresh. If another change happens shortly afterwards, it has to wait until the first rollout is done, and will then trigger another instance refresh. But it is neither necessary nor desired to roll all worker nodes twice in such a case, and it's much slower. Instead, cancel the first pending instance refresh, wait until another one can be started, and apply the latest change as soon as possible with the second instance refresh.

This change has been running fine in Giant Swarm's CAPA fork for almost a year at the time of opening this PR.

Checklist:

• squashed commits
• includes documentation
• includes emoji in title
• adds unit tests
• adds or updates e2e tests

Release note:

Cancel instance refresh on any relevant change to ASG instead of blocking until previous one is finished

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dlipovetsky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AndiDog]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[AndiDog]

@richardcase I think you have some machine pool experience and could maybe review this?

[AndiDog]

Rebased onto AWS SDK Go v2 changes

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[AndiDog]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[AndiDog]

Rate limit errors in the test, trying again

/test pull-cluster-api-provider-aws-e2e

[fiunchinho]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 524f880dcd8a5eff0e7f608006e4e86f3fc3786c

[richardcase]

@AndiDog - do you think there is any benefit in adding an e2e that covers this scenario?

[AndiDog]

@AndiDog - do you think there is any benefit in adding an e2e that covers this scenario?

@richardcase I'd say the feature is not that critical to require an E2E test. The behavior is well-covered in exp/controllers/awsmachinepool_controller_test.go (and I'm saying that as a huge fan of those quickly-running, mock-based unit tests because they're much easier to write).