Diagnostics: Adds Hedging Detection API (HedgingStarted, GetRequestedRegions and GetRespondedRegions)

[NaluTripician]

Diagnostics: Adds hedging detection API (HedgingStarted / GetRequestedRegions / GetRespondedRegions)

Closes #5867 (partial — covers .NET; Java and Python tracked separately).

Summary

Adds a focused, hot-path-friendly public surface on CosmosDiagnostics so callers can observe per-operation hedging behavior without parsing the ToString() JSON:

• virtual bool HedgingStarted() — true iff the SDK dispatched ≥ 1 hedge arm.
• virtual IReadOnlyList<RequestedRegion> GetRequestedRegions() — every region the SDK dispatched to, in observed order, tagged with a RequestedRegionReason.
• virtual IReadOnlyList<string> GetRespondedRegions() — every region that produced a response, in arrival order, duplicates allowed.
• New types: RequestedRegion (readonly struct, IEquatable<>, case-insensitive name equality) and RequestedRegionReason enum (Initial, OperationRetry, RegionFailover, Hedging, CircuitBreakerProbe, TransportRetry).

All members are virtual with safe defaults (false / empty list); custom subclasses of CosmosDiagnostics are unaffected.

Why this design

• State lives on TraceSummary.HedgingDetectionState, not as a TraceDatum. This means the Diagnostics.ToString() JSON shape is unchanged, while the API remains O(1). Mirrors the existing RegionsContacted pattern.
• Cross-handler signaling via RequestMessage.Properties[__CosmosDB_HedgingDetection_NextDispatchReason] — Properties is already deep-cloned per hedge arm by RequestMessage.Clone() and is shared with DocumentServiceRequest.Properties, so the upstream sites (ClientRetryPolicy, CrossRegionHedgingAvailabilityStrategy) can signal the downstream dispatch site (TransportHandler) without adding a new internal contract.
• Region resolution at dispatch via GlobalEndpointManager.GetLocation(locationEndpointToRoute) — DocumentServiceRequest.RequestContext.RegionName is not populated until after the append site.
• No phantom Hedging entries — hedge arms (requestNumber > 0) are only tagged inside CloneAndSendAsync, which is invoked only after the previous threshold delay elapses without primary cancellation.

See openspec/changes/hedging-detection-api/design.md for the full design write-up.

Wiring

Site	Change
CrossRegionHedgingAvailabilityStrategy.CloneAndSendAsync	For requestNumber > 0 set Properties[KEY] = Hedging.
ClientRetryPolicy.OnBeforeSendRequest	After RouteToLocation, when retryContext != null set Properties[KEY] to RegionFailover or OperationRetry.
TransportHandler.ProcessMessageAsync	After ToDocumentServiceRequest, append a RequestedRegion, then remove the property.
ClientSideRequestStatisticsTraceDatum	At all 3 response-record sites, also append the region to GetRespondedRegions.

AC coverage

AC	Status	Mechanism
AC1	✅	HedgingDetectionState.HedgingStarted flag, flipped on first Hedging append.
AC2	✅	Hedge arm tag set only inside requestNumber > 0; threshold delay gates dispatch.
AC3	✅	Single-region happy path: Initial requested, region recorded as responded.
AC4	✅	ClientRetryPolicy tags OperationRetry for same-region retries.
AC5	✅	ClientRetryPolicy tags RegionFailover when RetryRequestOnPreferredLocations.
AC6	⏸ Deferred	CircuitBreakerProbe reserved in enum but not populated — PPCB rerouting decision happens inside Microsoft.Azure.Cosmos.Direct's storeProxy.ProcessMessageAsync, after the pre-dispatch append site. Tracked as follow-up requiring a Direct-package change.
AC7	⏸ Deferred	TransportRetry reserved but per-channel transport retries inside Direct are not surfaced at this layer in v1.
AC8	✅	Hedge winner appears both in GetRequestedRegions (Hedging) and GetRespondedRegions.
AC9	✅	Default virtuals on CosmosDiagnostics return false / empty — CosmosDiagnosticsBackwardCompatTests.
AC10	✅	IReadOnlyList<T> returns immutable snapshot arrays.
AC11	✅	State off the trace tree → ToString() JSON shape unchanged.
AC12	✅	RequestedRegion is a readonly struct with case-insensitive name equality, ==/!=, custom GetHashCode, ToString.
AC13	✅	No phantom Hedging entries — see design §5.
AC14	✅	Duplicates in GetRespondedRegions allowed and preserved.
AC15	✅	Lock-protected snapshots; concurrency test (8 writers × 500 iterations).
AC16	⏸ Tracked	Live multi-region smoke test deferred to a follow-up emulator/live-account test; requires an environment outside the per-commit CI matrix.

Files changed

Public surface (src/Diagnostics)

• CosmosDiagnostics.cs — 3 new virtual methods with SE-013 <remarks>.
• CosmosTraceDiagnostics.cs — overrides delegate to HedgingDetectionState.
• RequestedRegion.cs (new) — readonly struct.
• RequestedRegionReason.cs (new) — enum.

Internal state (src/Tracing)

• HedgingDetectionState.cs (new) — lock-protected per-trace state.
• TraceSummary.cs — eager HedgingDetectionState property.

Wiring

• Routing/AvailabilityStrategy/CrossRegionHedgingAvailabilityStrategy.cs
• ClientRetryPolicy.cs
• Handler/TransportHandler.cs
• Tracing/TraceData/ClientSideRequestStatisticsTraceDatum.cs

Tests

• Diagnostics/RequestedRegionTests.cs (6 tests)
• Diagnostics/HedgingDetectionStateTests.cs (8 tests incl. concurrency)
• Diagnostics/CosmosDiagnosticsBackwardCompatTests.cs (3 tests, AC9)

Contract / changelog / OpenSpec

• tests/Microsoft.Azure.Cosmos.Tests/Contracts/DotNetSDKAPI.net6.json
• changelog.md
• openspec/changes/hedging-detection-api/{proposal.md,design.md,tasks.md,specs/hedging-detection/spec.md}

Validation

• dotnet build Microsoft.Azure.Cosmos\src\Microsoft.Azure.Cosmos.csproj -c Debug — clean ✅
• dotnet build Microsoft.Azure.Cosmos\src\Microsoft.Azure.Cosmos.csproj -c Release — clean ✅
• New unit tests — 17 / 17 ✅
• GA contract enforcement test — 107 / 107 ✅

Open questions / follow-ups

• AC6 / CircuitBreakerProbe — full PPCB tagging needs a small change in Microsoft.Azure.Cosmos.Direct's DocumentServiceRequestContext to flag probe requests. Suggested follow-up: surface an IsPpcbProbe flag and append a CircuitBreakerProbe-tagged entry at the rerouting site.
• AC7 / TransportRetry — same Direct-package consideration for per-channel transport retries.
• AC16 — live multi-region smoke test; can either reuse the existing Microsoft.Azure.Cosmos.EmulatorTests infra or add a single live-account test under an environment-gated test category.
• Java / Python issue threads should be opened separately referencing issue [Feature] Hedging Detection API — public accessors on CosmosDiagnostics #5867 and the cross-SDK invariants in internal-spec.md.

cc @kirankumarkolli — reviewer of record per the internal spec.

[NaluTripician]

Diagnostics sample — what callers will see

Attaching a sample of how the new surface looks at call-sites and what the values look like for a few realistic dispatch shapes. The Diagnostics.ToString() JSON shape is unchanged (state lives off the trace tree on TraceSummary.HedgingDetectionState), so the only observable difference is the three new accessors on CosmosDiagnostics.

Caller usage

ItemResponse<Book> response = await container.ReadItemAsync<Book>(id, pk);
CosmosDiagnostics diagnostics = response.Diagnostics;

bool hedged = diagnostics.HedgingStarted();
IReadOnlyList<RequestedRegion> requested = diagnostics.GetRequestedRegions();
IReadOnlyList<string>          responded = diagnostics.GetRespondedRegions();

logger.LogInformation(
    "Hedging started: {Hedged}; requested=[{Req}]; responded=[{Resp}]",
    hedged,
    string.Join(", ", requested),                 // uses RequestedRegion.ToString() → "East US:Initial"
    string.Join(", ", responded));

// Non-exhaustive enum — always include a default arm
foreach (RequestedRegion r in requested)
{
    switch (r.Reason)
    {
        case RequestedRegionReason.Initial:             /* first dispatch */              break;
        case RequestedRegionReason.OperationRetry:      /* same-region retry */           break;
        case RequestedRegionReason.RegionFailover:      /* cross-region failover */       break;
        case RequestedRegionReason.Hedging:             /* hedge arm dispatched */        break;
        case RequestedRegionReason.CircuitBreakerProbe: /* PPCB probe (reserved in v1) */ break;
        case RequestedRegionReason.TransportRetry:      /* reserved in v1 */              break;
        default:                                        /* future-proof */                break;
    }
}

Scenarios

1. Single-region happy path (no hedging configured)

HedgingStarted()      = false
GetRequestedRegions() = [ East US:Initial ]
GetRespondedRegions() = [ East US ]

2. Same-region retry (e.g. 410 Gone → SDK retries primary)

HedgingStarted()      = false
GetRequestedRegions() = [ East US:Initial, East US:OperationRetry ]
GetRespondedRegions() = [ East US, East US ]

3. Region fail-over (preferred-locations list exhausts primary, fails over)

HedgingStarted()      = false
GetRequestedRegions() = [ East US:Initial, West US:RegionFailover ]
GetRespondedRegions() = [ East US, West US ]

4. Hedging configured, primary wins under threshold (no hedge arm dispatched)

HedgingStarted()      = false           // ← AC1: registered ≠ dispatched
GetRequestedRegions() = [ East US:Initial ]
GetRespondedRegions() = [ East US ]

Note: HedgingStarted() == false here does not mean hedging was disabled — the primary just won under the configured threshold so no hedge arm ever fan-out. To check configuration, inspect CosmosClientOptions.AvailabilityStrategy / RequestOptions.AvailabilityStrategy directly.

5. Hedging configured, hedge arm wins (primary slow / failed)

HedgingStarted()      = true
GetRequestedRegions() = [ East US:Initial, West US:Hedging ]
GetRespondedRegions() = [ West US, East US ]   // hedge winner first; late primary response preserved (AC14)

6. Hedging configured, two hedge arms, last responds (worst case)

HedgingStarted()      = true
GetRequestedRegions() = [ East US:Initial, West US:Hedging, North Europe:Hedging ]
GetRespondedRegions() = [ North Europe, West US, East US ]

Equality / formatting on RequestedRegion

RequestedRegion a = new RequestedRegion("East US", RequestedRegionReason.Hedging);
RequestedRegion b = new RequestedRegion("east us", RequestedRegionReason.Hedging);

a == b           // true (case-insensitive on name, exact on reason)
a.ToString()     // "East US:Hedging"

Backward-compat note (AC9)

Any customer-authored subclass of CosmosDiagnostics that does not override the new virtuals returns:

HedgingStarted()      = false
GetRequestedRegions() = []   // Array.Empty<RequestedRegion>()
GetRespondedRegions() = []   // Array.Empty<string>()

This is also what the SDK returns for diagnostics objects produced before this version's plumbing populated them.

Diagnostics.ToString() JSON shape

Unchanged. State is on TraceSummary.HedgingDetectionState, not in the trace tree, so existing dashboards / log parsers that consume the JSON are unaffected.

[kundadebdatta]

🟡 Recommendation (F2) No end-to-end tests cover the wiring this PR adds.

The 17 added tests cover HedgingDetectionState and RequestedRegion in isolation plus AC9 backward-compat defaults — but nothing verifies the integration that is the entire ask of the PR:

• ClientRetryPolicy.OnBeforeSendRequest actually writes OperationRetry vs RegionFailover for the two branches of RetryRequestOnPreferredLocations.
• CrossRegionHedgingAvailabilityStrategy.CloneAndSendAsync writes Hedging only on requestNumber > 0.
• TransportHandler.AppendDispatchedRegion resolves a region, appends an entry, and removes the property.
• An end-to-end hedge fan-out via the existing mock pipeline produces the expected HedgingStarted()/GetRequestedRegions()/GetRespondedRegions() sequence.

AC1–AC5 are essentially asserted by inspection only. CrossRegionHedgingAvailabilityStrategyTests is the natural home for the integration coverage, and the gap is what would have caught F3 (hedge-arm retry overwrite) before merge.

[kundadebdatta]

🟡 Recommendation (F4) Thin-client / PPAF / per-partition routed dispatches produce no entry in GetRequestedRegions().

GlobalEndpointManager.GetLocation(uri) walks AvailableWriteEndpointByLocation and AvailableReadEndpointByLocation and returns null for any URI not in those maps. That includes:

• Thin-client endpoints (ResolveThinClientEndpoint in ClientRetryPolicy.cs:274-277 returns endpoints from ThinClientReadEndpoints/ThinClientWriteEndpoints, which GetLocation does not consult).
• PPAF / PPCB-rerouted URIs.
• Any per-partition routed endpoint.

For those operations AppendDispatchedRegion bails silently — but ClientSideRequestStatisticsTraceDatum.RecordResponse/RecordHttpResponse still call AppendResponded because the responded region is derived from a different source. Net result: GetRespondedRegions() has entries but GetRequestedRegions() is empty for thin-client and PPCB scenarios — a surprising asymmetry that the XML docs do not disclose.

Suggested fix (lowest cost): add a remark to the XML doc on CosmosDiagnostics.GetRequestedRegions and HedgingStarted calling out that thin-client / PPAF dispatches are not currently included, and cross-reference AC6/AC7. Better: fall back to serviceRequest.RequestContext.RegionName when GetLocation returns null. Best: make GlobalEndpointManager aware of thin-client endpoints.