Fix the backport of #4512 into release-2.2 branch

[cnmcavoy]

What type of PR is this?
/kind regression

What this PR does / why we need it: Regression from main branch, backport of #4512 into release-2.2 failed and introduced errors and missed adding a unit test. See conversation on slack by @philjb : https://kubernetes.slack.com/archives/CD6U2V71N/p1697658532510839

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4511

Special notes for your reviewer:

Checklist:

• squashed commits
• includes documentation
• adds unit tests
• adds or updates e2e tests

Release note:

Fixed: Additional Tags specified on the AWSCluster will now propagate to root storage volumes.

[richardcase]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[philjb]

@richardcase @cnmcavoy - how does this work now? Is there a new release of 2.2? aka 2.2.5?

Thanks for making this fix - I'd been meaning to do it. I just yesterday double checked the behavior of main and the 2.2.4 release. 2.2.4 with only the partial cherry pick is broken still with regards to #4512. The code in this PR should fix pushing tags to storage volumes on the 2.2 branch line.

Same thing said differently, based on my (local) testing yesterday, #4512 should not be added to 2.2.4 release notes as the issue isn't fixed there. This pr should fix it in the next release of 2.2.

[philjb]

looks good to me.

I tested (tilt based test setup) that someone with a cluster with CAPA 2.2.4 can upgrade to this patch (future 2.2.5) and it works properly. I was concerned that the prevAnnotation annotation might not push tag updates through after upgrades but it does.

EDIT: for more context, prevAnnotations isn't even used when updating tags (which is good - prevents issues with an annotation's state being wrong - but it does mean more API tag requests - on every reconcile i believe); prevAnnotations is used when deciding when to update the annotation which says which tags were last applied.

[richardcase]

@philjb - once this merges we can thinking about doing a v2.2.5 release.

[richardcase]

We just need some to review and provide the lgtm

[AndiDog]

/lgtm