Fix spurious resource replacements during API version upgrades with refresh

[EronWright]

Problem

When upgrading resources to a new API version using aliases and running pulumi up --refresh, the provider incorrectly reported spurious property changes and marked resources for replacement, even though the actual Azure resource hadn't changed.

Example scenario:

• User has a ManagedCluster in state using API version 2024-01-02-preview
• User upgrades provider to v3 which defaults to 2024-10-02-preview
• User runs pulumi up --refresh
• Provider incorrectly shows properties being added and marks resource for replacement

Reported in: #4400

Root Cause

The provider uses "input-input diffing" to detect changes:

1. Convert old state outputs → inputs (using schema)
2. Convert new Azure outputs → inputs (using schema)
3. Diff these to find actual changes

The bug: When API version changed via alias, the provider used the NEW schema to normalize BOTH old and new state. This caused properties that only exist in the new schema to appear as "additions".

Concrete Example

Old state (deployed with v2.90.0 using API 2024-01-02-preview):

{
  "dnsPrefix": "test-aks-4400",
  "location": "eastus",
  "agentPoolProfiles": [...],
  "azureApiVersion": "2024-01-02-preview"
}

New Azure response (read with API 2024-10-02-preview):

{
  "dnsPrefix": "test-aks-4400",
  "location": "eastus",
  "agentPoolProfiles": [...],
  "kind": "Base",                           // New property in 2024-10-02-preview
  "networkProfile": {
    "podLinkLocalAccess": "IMDS"           // New property in 2024-10-02-preview
  },
  "azureApiVersion": "2024-10-02-preview"
}

Before fix: Both states normalized with 2024-10-02-preview schema

• Old state projection: {dnsPrefix, location, agentPoolProfiles} (missing new properties)
• New state projection: {dnsPrefix, location, agentPoolProfiles, kind, networkProfile}
• Diff: Shows kind and networkProfile.podLinkLocalAccess being added → triggers replacement ❌

After fix: Schema-aware normalization

• Old state projection with 2024-01-02-preview schema: {dnsPrefix, location, agentPoolProfiles}
• New state projection with 2024-10-02-preview schema: {dnsPrefix, location, agentPoolProfiles} (new properties dropped as defaults)
• Diff: No changes ✅

Solution

Modified the Read method in provider.go to detect API version changes and use schema-aware normalization:

1. Detect API version change: Compare azureApiVersion from state vs current metadata
2. Look up old metadata: Find resource metadata for the old API version
3. Use correct schemas:
   • Old state → old schema → inputs projection
   • New state → new schema → inputs projection
4. Diff the correct projections: Now comparing apples-to-apples

Key Code Changes

API Version Detection (lines 1376-1392):

// Check if API version has changed (e.g., via alias during upgrade).
var oldApiVersion string
var oldRes *resources.AzureAPIResource
if azureApiVersion, ok := oldState["azureApiVersion"]; ok && azureApiVersion.IsString() {
    oldApiVersion = azureApiVersion.StringValue()
    if oldApiVersion != res.APIVersion {
        // API version has changed. Try to look up the old resource metadata.
        logging.V(5).Infof("%s: API version changed from %s to %s", label, oldApiVersion, res.APIVersion)
        oldRes, err = k.lookupResourceWithAPIVersion(urn, oldApiVersion)
        if err != nil {
            logging.V(5).Infof("%s: could not look up old resource metadata for API version %s: %v", label, oldApiVersion, err)
            oldRes = nil
        }
    }
}

Schema-Aware Normalization (lines 1460-1477):

// Use old resource metadata if API version changed
resForDefaults := res
if oldRes != nil {
    resForDefaults = oldRes
}
removeDefaults(*resForDefaults, plainOldState, previousInputs.Mappable())

// Project old outputs using old schema if API version changed
var oldInputProjection map[string]interface{}
if oldRes != nil {
    // API version changed: use old schema for converting old state
    oldInputProjection = k.converter.SdkOutputsToSdkInputs(oldRes.PutParameters, plainOldState)
} else {
    // Same API version: use current schema
    oldInputProjection = k.converter.SdkOutputsToSdkInputs(res.PutParameters, plainOldState)
}

Testing

✅ Unit tests:

• Added TestReadWithApiVersionMismatch for Read() with API version mismatch
• Validates old schema lookup via lookupResourceWithAPIVersion()
• Confirms azureApiVersion output update after Read
• Added TestApiVersionToVersionPart for API version conversion utilities

✅ E2E regression test:

• Added TestUpgradeAksApiVersion_2_90_0 replay-based test
• Validates v2.90.0 → v3 upgrade scenario (versioned → unversioned types)
• Uses recorded GRPC interactions from real v2.90.0 ManagedCluster deployment
• Asserts no replacements during provider upgrade preview
• Comprehensive documentation in test-programs/upgrade-aks-api-version/README.md

✅ Manual testing:

• Deployed ManagedCluster with v2.90.0 using API version 2024-01-02-preview
• Upgraded to v3 provider (defaults to 2024-10-02-preview)
• Ran pulumi up --refresh - no spurious replacements ✅
• Verified azureApiVersion updated correctly
• Properties like kind and networkProfile.podLinkLocalAccess handled correctly as defaults

Additional Enhancements

User-facing warning for API version changes:

• Added warning in Diff() when API version mismatch is detected between state and provider metadata
• Suggests running pulumi refresh to align schemas and update resource state
• Helps users understand when refresh operations are beneficial after provider upgrades

Code simplifications:

• Leveraged existing openapi.ApiToSdkVersion() and openapi.SdkToApiVersion() utilities for format conversion
• Used resources.ParseToken() and resources.BuildToken() helpers for type token manipulation
• Improved error messages to distinguish between missing resources vs unavailable API versions

Files Changed

• provider/pkg/provider/provider.go - Core fix, helper methods, and user warnings
• provider/pkg/provider/provider_test.go - Unit test for Read() with API version mismatch
• provider/pkg/provider/resource_lookup_test.go - Unit tests for API version conversion
• provider/pkg/provider/provider_e2e_test.go - E2E regression test infrastructure
• provider/pkg/provider/test-programs/upgrade-aks-api-version/ - E2E test program and recordings
• CLAUDE.md - Documentation on testing API version upgrades

Fixes #4400

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

[codecov]

Codecov Report

❌ Patch coverage is 56.36364% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.47%. Comparing base (39412ea) to head (53c9387).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
provider/pkg/provider/provider.go	56.36%	12 Missing and 12 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4429      +/-   ##
==========================================
+ Coverage   59.44%   59.47%   +0.03%     
==========================================
  Files          91       91              
  Lines       11480    11531      +51     
==========================================
+ Hits         6824     6858      +34     
- Misses       4020     4024       +4     
- Partials      636      649      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[EronWright]

✅ Fix Verified - Spurious Replacement Eliminated

Successfully tested the improved fix (commit 87fd4f5) using the blind variant reproduction scenario. The fix eliminates the spurious replacement bug reported in #4400.

Test Setup

Starting State (Critical Mixed Condition):

• Type token: azure-native:containerservice/v20241002preview:ManagedCluster (new)
• azureApiVersion in state: 2024-01-02-preview (old)
• This mixed state was created via "blind update" (pulumi up without --refresh after API version upgrade)

Provider Version: Built from eronwright/fix-api-version-upgrade-refresh branch with simplified fix

Test Results

Original Bug Behavior (v2.90.0)

 ~  azure-native:containerservice/v20241002preview:ManagedCluster test-cluster refresh [diff: +kind,networkProfile~agentPoolProfiles]
 ++ azure-native:containerservice/v20241002preview:ManagedCluster test-cluster create replacement
 +- azure-native:containerservice/v20241002preview:ManagedCluster test-cluster replace
 -- azure-native:containerservice/v20241002preview:ManagedCluster test-cluster delete original
Resources:
    +-1 to replace

Problem: Provider read resource using NEW API version (2024-10-02-preview) and compared it with state containing OLD API version outputs, causing false positive property differences.

Fixed Behavior (This PR)

 ~  azure-native:resources:ResourceGroup aks-test-rg refresh 
 ~  azure-native:containerservice/v20241002preview:ManagedCluster test-cluster refresh 
Resources:
    3 unchanged

Solution: Provider now correctly looks up resource metadata using the OLD API version (2024-01-02-preview) from state's azureApiVersion field when performing refresh operations. This ensures schema consistency during cross-API-version comparisons.

State Evolution

Before Refresh:

{
  "type": "azure-native:containerservice/v20241002preview:ManagedCluster",
  "azureApiVersion": "2024-01-02-preview"
}

After pulumi up --refresh:

{
  "type": "azure-native:containerservice/v20241002preview:ManagedCluster",
  "azureApiVersion": "2024-10-02-preview"
}

The API version was seamlessly upgraded during refresh without triggering any spurious replacements. ✨

Key Improvements in This Revision

1. Simplified implementation: Uses existing openapi.ApiToSdkVersion() conversion function instead of custom parsing logic (~45 lines of code eliminated)
2. Added ARM path validation: Verifies resource identity by checking that the looked-up resource has the same ARM path
3. Better maintainability: Leverages well-tested conversion infrastructure that handles all API version format edge cases

Test Logs

• Baseline bug reproduction: /tmp/preview-refresh-blind-variant.log
• Fixed behavior: /tmp/preview-refresh-FIXED-no-run-program.log
• Final verification: /tmp/up-refresh-FIXED.log

---

The fix is working as intended and ready for review!

[pulumi-bot]

This PR has been shipped in release v3.11.0.