azurerm_kubernetes_cluster: Support for upgrade path from Azure CNI to Azure CNI Overlay

[jkroepke]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

This PR adds support upgrade-path from Azure CNI to Azure CNI Overlay

Ref: https://learn.microsoft.com/en-us/azure/aks/azure-cni-overlay?tabs=kubectl#azure-cni-cluster-upgrade

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevent documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

The existing test TestAccKubernetesCluster_networkPluginModeUpdate cover this scenario.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

=== RUN   TestAccKubernetesCluster_networkPluginModeUpdate
=== PAUSE TestAccKubernetesCluster_networkPluginModeUpdate
=== CONT  TestAccKubernetesCluster_networkPluginModeUpdate
--- PASS: TestAccKubernetesCluster_networkPluginModeUpdate (1500.61s)
PASS

Observing that the upgrade is done in-place:

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_kubernetes_cluster: Support for upgrade path from Azure CNI to Azure CNI Overlay [GH-00000]

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #26259

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[jkroepke]

@stephybun could you please take a look here?

[uyless]

Hey, any news here?

[stephybun]

Apologies for taking so long to circle back to this @jkroepke. I was umm'ing and ahh'ing over whether this was OK or not since on the surface the change looks small and harmless but this would still require a more in-depth investigation which turned out to be correct.

I left an explanation on my findings in-line, we want to be careful not to introduce issues for users that are using other networking modes. Details are in the comment, let me know if you have any questions!

[stephybun]

Although this will work for upgrading from Azure CNI to Azure CNI Overlay, it will cause issues for users that are using other networking configurations which we still need to support.

pod_cidr and pod_cidrs are both Optional and Computed because there are instances where the API will assign values to these if they're omitted e.g. when using kubenet, this compounds with the fact that these value change checks performed in the CustomizeDiff compare the old state value and the new state value and not whether the value in the users configuration has changed.

Ultimately this will end up triggering a ForceNew for users that are still using kubenet but haven't specified pod_cidr or pod_cidrs (and potentially other scenarios!).

I would appreciate if you could add some test cases (or extend existing test cases) to ensure that we're not triggering a ForceNew in situations where we shouldn't be, those should also help you to refine the condition needed here to enable the behaviour you're after without introducing unwanted side effects.