[azcosmos] unblock cross-region failover from synchronous gem.Update

[tvaron3]

What & Why

Three retry paths in the Cosmos client retry policy (cosmos_client_retry_policy.go) used to gate the cross-region retry on a successful synchronous gem.Update(ctx, true) against the account's global endpoint. During a regional outage the global FQDN typically resolves to the same regional FE pool we just marked unavailable (Cosmos accounts in a region share FEs), so the forced refresh stalls on a connect timeout and surfaces that error instead of attempting the failover.

This PR also fixes two interlocking bugs in attemptRetryOnNetworkError that prevented the failover from actually routing to another region even when the refresh succeeded, and tightens a few related correctness issues.

Changes

• attemptRetryOnNetworkError (connection error) — dropped the forced gem.Update. MarkEndpointUnavailable* already calls locationCache.updateLocked which recomputes readEndpoints / writeEndpoints with the unavailable endpoint demoted; invalidate() arms the next non-force Update for real topology changes. Also left retryCount at 0 after the failover — getPrefAvailableEndpointsLocked demotes unavailable endpoints to the tail of the route list rather than removing them, so retryCount=1 (after retryCount += 1) re-indexed back to the just-marked unhealthy endpoint via ResolveServiceEndpoint(1 % 2).
• attemptRetryOnEndpointFailure (HTTP 403 / WriteForbidden) — replaced the blocking gem.Update with a fire-and-forget asyncForceRefreshGEM helper. The helper uses a CompareAndSwap gate so a retry storm cannot leak a goroutine per attempt, runs on context.Background() so a near-expired caller deadline cannot abort the refresh, and has panic recovery + error logging.
• attemptRetryOnRequestTimeout (HTTP 408) — removed both MarkEndpointUnavailableForRead and the forced gem.Update. A 408 is a per-request signal (the gateway accepted the request but did not finish in time), not regional unhealthiness; demoting the whole region for unavailableLocationExpirationTime based on one slow request would penalize concurrent reads. The outer loop's retryCount++ already advances the location index for the cross-region retry.
• URL-key normalization in locationCache — MarkEndpointUnavailable* was called with the full request URL (path/query/RawPath), but the unavailability map was keyed by url.URL struct equality while availReadEndpointsByLocation / availWriteEndpointsByLocation only stored scheme+host. Marks were silently written under a key nothing else looked up, so isEndpointUnavailableLocked always returned false and the demote was a no-op. Now normalized to scheme+host via a new internal endpointKey helper.
• locationCache.resolveServiceEndpoint — now takes mapMutex.RLock. Reads of locationInfo / enableMultipleWriteLocations race the writes in update/updateLocked, which the new async refresh paths surface deterministically under go test -race.

Repro that motivated the fix

Reproduced live by running azcosmos_perf in AKS with iptables OUTPUT DROP rules applied via kubectl debug --profile=netadmin to block the workload account's westus2 FE IPs (20.42.169.199, 20.42.170.147, 20.9.156.133).

Account topology:

• Workload account: write=westus2, reads=[westus2, westus]
• Client args: --application-region westus2 --preferred-regions westus

For 10 minutes:

• ss -tan inside the pod showed every goroutine in SYN_SENT to a blocked westus2 IP. Zero TCP attempts to 40.112.241.99 (the westus replica).
• Per-op error rate ≈ 15/min/op across concurrency=50 → ~18–22 s wall-clock per failed op, matching the budgeted same-region retries (3 × (5s + 1s backoff)) plus one failover attempt whose gem.Update itself timed out.
• Effective throughput went to zero and stayed there for the full window.

After the fix, the cache mutation from MarkEndpointUnavailable* is enough for the next iteration's ResolveServiceEndpoint to return westus, and the cross-region retry actually executes.

Regression tests

• TestConnectionErrorReadFailsOverWhenGlobalEndpointIsUnreachable — pins the GEM mock to net.DNSError so any forced refresh would fail; asserts the 5th-attempt 200 from the failover server is returned. Pre-fix the request surfaces a DNS error instead.
• TestFix3a_WriteRetrySucceedsWhenGEMRefreshFails — 403 retry returns (true, nil) even when GEM transport returns 500 on every call.
• TestFix3b_RequestTimeoutDoesNotMarkEndpointUnavailable — after a 408 retry, locationUnavailabilityInfoMap is empty and the GEM transport was never invoked.
• TestFix3c_RequestTimeoutDoesNotBlockOnStalledGEM — with a 10-minute hanging GEM transport, attemptRetryOnRequestTimeout returns within 5 s.
• TestFix3d_AsyncRefreshCASGateCoalescesRetryStorm — 50 concurrent asyncForceRefreshGEM() calls produce far fewer than concurrency/2 HTTP refreshes.
• TestFix3e_AsyncRefreshIgnoresCallerContextCancellation — an already-cancelled caller context still triggers a successful background refresh (proves context.Background()).
• TestFix3_WriteRetryForceRefreshesGEM updated to use require.Eventually since the refresh is now async and the singleflight + CAS gate legitimately coalesce concurrent forced refreshes.

Backwards compatibility

• No public API changes.
• The single-master-write branch of attemptRetryOnNetworkError already had a comment explicitly noting that no gem.Update is needed here (added in [Cosmos] Fix excess GetDatabaseAccount calls and eliminate data-plane fallback to customer endpoint (#25468) #26815). This change extends the same reasoning to the read / multi-master-write failover branch and to the 403 / 408 paths.

CHANGELOG

Entry added under 1.5.0-beta.7 → Bugs Fixed.

Testing

go test -count=1 -timeout 300s -race ./...
ok  github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos    71.863s

[Copilot]

Pull request overview

This PR adjusts the azcosmos client retry policy so that cross-region failover for connection errors is no longer blocked by a forced global-endpoint metadata refresh (which can fail during regional gateway outages), ensuring the failover retry actually occurs.

Changes:

• Removed the synchronous gem.Update(ctx, true) gating from attemptRetryOnNetworkError after marking an endpoint unavailable.
• Added a regression test covering failover behavior when the global endpoint is unreachable.
• Added a CHANGELOG entry under 1.5.0-beta.7 describing the fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
sdk/data/azcosmos/cosmos_client_retry_policy.go	Stops forcing a GEM refresh on the connection-error failover path so failover isn’t blocked by unreachable metadata endpoint.
sdk/data/azcosmos/cosmos_client_retry_policy_test.go	Adds a regression test ensuring read failover proceeds even when the global endpoint can’t be reached.
sdk/data/azcosmos/CHANGELOG.md	Documents the bug fix in the unreleased changelog.

[tvaron3]

PR Deep Review summary

I found three blocking correctness risks and several targeted test/robustness gaps. The main concern is that the new demote-to-tail routing only works when the next retry indexes 0; the 403 path still increments retryCount, and the connection-error path does not reset a pre-existing non-zero retryCount. I also found remaining unlocked location-cache readers exposed by the new async refresh path.

I could not run local Go tests because go is not installed in this environment (go: command not found).

---

_{⚠️ AI-generated review — may be incorrect. Agree? → resolve the conversation. Disagree? → reply with your reasoning.}

[kushagraThapar]

Thanks for the rapid turnaround on top of #26815 and the excellent live-AKS repro narrative — iptables OUTPUT DROP + ss -tan showing zero TCP attempts to the replica IP is exactly the level of forensic detail that made the three interlocking bugs visible. The URL-key normalization catch was particularly nice. Core fix (MarkEndpointUnavailable* cache mutation is sufficient; synchronous refresh is the trap) reads cleanly and aligns with .NET on the demote-to-tail routing model. LGTM 🚀

[kushagraThapar]

LGTM, thanks @tvaron3