Routing: Adds Gateway Account Property Flag for Dynamic Hedging Control

[NaluTripician]

Description

Adds a Gateway-controlled disableCrossRegionalHedging account property that lets on-call engineers dynamically disable PPAF cross-region hedging without rolling back PPAF entirely.

When the flag is true, the SDK clears the AvailabilityStrategy on the next account-properties refresh; toggling it back to false restores the customer-configured strategy or rebuilds the SDK default.

Type of change

• Bug fix / mitigation lever (non-breaking; adds an operational escape hatch)
• New feature
• Breaking change

Spec

• openspec/changes/ppaf-dynamic-hedging-control/ (proposal/design/spec/tasks)
• Reference PR: [Internal] Spec: Adds Gateway Account Property Flag for Dynamic Hedging Control #5719 (spec only)

Changes

• AccountProperties: new internal DisableCrossRegionalHedging (nullable bool) bound to JSON key disableCrossRegionalHedging.
• GlobalEndpointManager: OnEnablePartitionLevelFailoverConfigChanged is now Action<bool, bool>; tracks lastKnownDisableCrossRegionalHedging so refresh fires on either flag change.
• DocumentClient:
  • Captures the initial gateway flag during GetInitializationTaskAsync and stashes any customer-configured AvailabilityStrategy when the flag is true at startup.
  • InitializePartitionLevelFailoverWithDefaultHedging short-circuits when the flag is true.
  • UpdatePartitionLevelFailoverConfigWithAccountRefresh reconciles both PPAF state and the disable flag via ApplyHedgingStrategyForCurrentState.
  • Customer strategies are stashed/restored across toggles; SDK-default strategies are not stashed (rebuilt deterministically from PPAF state).
• 10 new unit tests in GatewayHedgingOverrideTests.
• Updates 2 existing serializer contract tests that pin the AccountProperties JSON shape.

Behavior

State	Result
flag true, customer strategy present	Strategy stashed, AvailabilityStrategy=null
flag true, SDK default present	AvailabilityStrategy=null (no stash; deterministic)
Toggle false, strategy stashed	Restored from stash
Toggle false, no stash, PPAF on	SDK default rebuilt
DisablePartitionLevelFailoverClientLevelOverride=true	Gateway flag ignored entirely

Validation

This change is a runtime mitigation lever for a production feature (PPAF cross-region hedging), so correctness across the toggle lifecycle and CI green across the full SDK matrix were both treated as gating signals.

Unit-test coverage (10 new tests in GatewayHedgingOverrideTests)

Each row in the Behavior table above has a dedicated test, plus dedicated coverage for the JSON contract and the init-time path:

JSON contract / forward-compat (AccountProperties)

• ..._DisableCrossRegionalHedging_True_Deserializes -- flag true round-trips.
• ..._False_Deserializes -- flag false round-trips.
• ..._Absent_DefaultsToNull -- missing key deserializes as null (i.e. "no opinion"), preserving existing behavior for older accounts.
• ..._Unknown_FieldDoesNotBreakDeserialization -- unknown sibling fields don't break deserialization (forward-compat guard for future Gateway additions).

Init-time path (InitializePartitionLevelFailoverWithDefaultHedging)

• ..._FlagTrue_SkipsApplyingDefaultStrategy -- when the flag is true at startup, the SDK default hedging strategy is not installed.
• ..._FlagFalse_AppliesDefaultStrategy -- when the flag is false, the SDK default is installed as before (no regression for the normal path).

Refresh-time reconciliation (UpdatePartitionLevelFailoverConfigWithAccountRefresh)

• UpdateConfig_FlagToggleOn_StashesCustomerStrategyAndClearsAvailabilityStrategy -- flipping the flag to true clears ConnectionPolicy.AvailabilityStrategy and stashes the customer's original reference (verified with AreSame, not just structural equality).
• UpdateConfig_FlagToggleOff_RestoresStashedCustomerStrategy -- flipping back to false restores the exact same customer strategy instance and clears the stash, validating the full enable -> disable -> re-enable cycle from task 6.6.
• UpdateConfig_FlagTrue_DoesNotStashSDKDefaultStrategy -- when only the SDK default is present, the flag clears it but does not stash it; toggling back rebuilds a fresh SDK default. This guards the "stash only customer-configured strategies" invariant called out in the design.
• UpdateConfig_ClientLevelOverrideDisabled_FlagIsIgnored -- when DisablePartitionLevelFailoverClientLevelOverride=true, the Gateway flag is ignored end-to-end and the customer strategy is left untouched (client-level override remains absolute).

The tests use compile-time-safe internal test-only properties (DisableCrossRegionalHedgingForTests, CustomerConfiguredAvailabilityStrategyForTests) instead of reflection, so any future rename of the underlying fields breaks the build rather than silently passing.

Contract tests

The two existing tests that pin the AccountProperties JSON shape (CosmosJsonSerializerUnitTests, CosmosSerializerCoreTests) were updated and re-run, confirming the new optional property is the only change to the wire contract.

Review-feedback hardening (commit 816e334)

A second pass addressed correctness concerns that aren't easy to assert from a unit test but materially raise confidence:

• Thread safety: introduced hedgingStrategyLock to serialize the (flag, stash, ConnectionPolicy.AvailabilityStrategy) mutation sequence, so concurrent refresh callbacks cannot interleave stash/clear/restore steps.
• Init-time race: moved the initial flag capture and customer-strategy stash from GetInitializationTaskAsync into InitializeGatewayConfigurationReaderAsync, before the GlobalEndpointManager background-refresh loop starts and before the change-event handler is subscribed -- eliminating the theoretical window where a refresh-driven event could be overwritten by a later init-time assignment.
• Early-return guard: UpdatePartitionLevelFailoverConfigWithAccountRefresh no-ops when neither PPAF enablement nor the hedging flag changed, protecting future direct callers from spurious AvailabilityStrategy mutation.
• Shadowing guard: renamed locals (isEnabled -> latestIsEnabled, disableCrossRegionalHedging -> latestDisableCrossRegionalHedging) so a future edit that drops a this. prefix can't silently flip semantics.
• Loud-fail diagnostic: ApplyHedgingStrategyForCurrentState emits a TraceWarning if it would silently drop a non-default AvailabilityStrategy while a previously-stashed customer strategy is still held, so this edge case is observable in support logs rather than invisible.

CI

All 31 required checks are green on the final commit, including:

• Microsoft.Azure.Cosmos.Tests (full unit-test run) and Microsoft.Azure.Cosmos.Tests Coverage.
• Full EmulatorTests matrix: Query/ChangeFeed/ReadFeed/Batch/Client Telemetry, MultiMaster, MultiRegion, ThinClient, Others, and the Flaky lane.
• Both preview lanes (Preview Tests Release, Preview Flag Release, Preview Encryption EmulatorTests Release).
• Encryption and Encryption.Custom emulator tests (project-ref and package-ref).
• Static Analysis, CodeQL, and all Analyze (...) jobs.
• PerformanceTests Release and CosmosBenchmark (no perf regression in the hot path; the new logic only runs on account-refresh and at init, not per-request).

Live long-running soak (~9.3 hours, 837 K ops, 22 toggle cycles)

In addition to the unit/contract tests and CI matrix, we ran the dedicated PpafHedgingValidator harness end-to-end against a real PPAF-enabled account in our internal test environment. While the harness ran the workload + fault injection, a separate operator script toggled disableCrossRegionalHedging on the account every 15 minutes by alternating UpdateGlobalDatabaseAccount.ps1 -PropertyValue true and RemoveGlobalDatabaseAccountConfigurationOverrides.ps1 -- exactly the on-call mitigation gesture this PR is designed to support.

The 12 h budgeted run was cut short by an OS auto-update at 2026-05-13 03:44 UTC after 09:18:16 of wall time / 837,303 ops / 22 full toggle cycles captured. The artifacts (samples.csv, events.jsonl, requests.jsonl, report.md, timeline.png) cover the full 9.3 h window.

Headline result -- hedging efficacy under live chaos (post-warmup samples, fault windows only)

Gateway flag	SDK ConnectionPolicy.AvailabilityStrategy (in-process)	Samples	Mean p99	Total hedges
disableCrossRegionalHedging = false (hedging enabled)	SdkDefault (CrossRegionHedging)	79	706 ms	3,889
disableCrossRegionalHedging = true (hedging disabled)	null	93	1,454 ms	65

When the gateway flag flipped to false, the SDK rebuilt the hedging strategy and hedges fired against the faulted North Central US primary -- cutting p99 by ~750 ms (~51%). When the gateway flag flipped to true, the strategy cleared and the ~60x drop in hedge count demonstrates the disable path actually disabled hedging. The 65 residual hedges in the null bucket are in-flight requests from the prior SdkDefault window completing with their pre-flip hedges -- by construction the SDK cannot retroactively cancel those.

Toggle-cadence alignment

After SDK warmup (~20:16 UTC onward), strategy_state -- read directly from in-process ConnectionPolicy.AvailabilityStrategy via reflection -- flipped between Null and SdkDefault on a perfect 15-minute cadence, exactly matching the external Start-Sleep -Seconds 900 toggle script. Post-warmup cycle ratio: 52% Null / 48% SdkDefault (effectively 50/50). The timeline.png flag panel shows the gateway flag and the SDK-observed flag overlay one another across all 36 transitions.

Other signals

• 0 errors across 837,303 operations.
• 4,590 hedged requests total; 3,898 of them hedged away from the faulted primary region (i.e. the SDK followed the fault, which is the whole point of PPAF hedging).
• report.md tabulates 33 of 36 detected transitions as PASS. The 3 strict-comparator misses (No DocumentQuery Linq to SQL evaluation #4 / skip/take? #8 / Preserve some of the way creating a query worked in v2 #26 -- 48 / 47 / 65 hedges) are all false -> true boundary noise from the in-flight tail described above; the strict numerical comparator does not yet treat a sub-poll-interval in-flight tail as a pass, but the underlying SDK behavior is correct.

Validator-side caveats (harness bugs, not product code)

The GatewayPropertyProbe and SdkStateInspector reflective probes regressed mid-run, so per-sample gw_flag / sdk_flag / sdk_lastKnown columns were stuck at 0 for the whole run. Transition timestamps in report.md are therefore synthesized from strategy_state (Null -> flag=true, SdkDefault -> flag=false) by the harness's --rebuild-from-csv path, which is accurate to the 30 s poll boundary. strategy_state itself is a direct in-process read of ConnectionPolicy.AvailabilityStrategy and is the authoritative signal -- and as the table above shows, it tracks the external toggle cadence perfectly and yields a decisive separation in both p99 latency and hedge count between the two flag states.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[kundadebdatta]

Please add changelog entry as well.

[kundadebdatta]

Waiting on author.

[NaluTripician]

Addressed in d9041ca. Added an [Unreleased] section at the top of changelog.md (matching the Keep a Changelog format the file's header references) with the PR #5829 entry under Added:

### <a name="unreleased"/> [Unreleased]

#### Added

- [5829](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/5829) Routing: Adds Gateway Account Property Flag for Dynamic Hedging Control. Introduces a Gateway-controlled `disableCrossRegionalHedging` account property that lets operators dynamically disable PPAF cross-region hedging (and any customer-configured `AvailabilityStrategy`) without rolling back PPAF entirely. The SDK reconciles `ConnectionPolicy.AvailabilityStrategy` against the Gateway-supplied flag on each account-properties refresh; toggling the flag back to `false` restores the customer-configured strategy or rebuilds the SDK default.

The release author can rename Unreleased to the actual version (e.g., 3.60.0-preview.1 or 3.60.0) at release time. Let me know if you'd prefer the entry be slotted directly into a specific existing version section instead.

[kushagraThapar]

Hi @NaluTripician — thanks for the careful hardening work in commit 816e334fc and the second round in d9041cae, the lock discipline + init-time relocation closed off the most concerning windows, and the changelog entry at HEAD is great. A few collaborative follow-ups from a fresh-eyes pass on commit 95938a5c2. None of these duplicate Debdatta's open N1/N2/N3 (replies on those are tracked separately).

---

🟡 M1 — Per-request RequestOptions.AvailabilityStrategy may bypass the new operator override

Could you please help me understand the scope of this PR vs the unchecked tasks 4.1 (RequestInvokerHandler enforcement) and 6.3 (per-request test) in openspec/changes/ppaf-dynamic-hedging-control/tasks.md?

Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs:132 resolves the per-request strategy as:

AvailabilityStrategy strategy = request.RequestOptions?.AvailabilityStrategy
    ?? this.client.DocumentClient.ConnectionPolicy.AvailabilityStrategy;

So a customer that sets AvailabilityStrategy on RequestOptions per-call would keep speculating cross-region after the gateway flag is true — which seems to contradict the operator-mitigation intent.

Two options to consider, either is fine — mostly want the scope to be unambiguous before merge:

(a) Implement task 4.1 — add an internal DocumentClient.IsHedgingDisabledByGateway accessor (read under hedgingStrategyLock) and have RequestInvokerHandler.AvailabilityStrategy(...) return null when set; or
(b) Explicitly downscope — strike the per-request scenarios from spec.md, mark task 4.1 / 6.3 as out of scope in tasks.md, and note in the PR description that per-request enforcement is a follow-up.

Happy to defer to (b) if that's the agreed scope.

---

🟡 M4 — Java parity gap (informational; happy to be told there's already a tracking issue)

I am curious whether there is a Java tracking issue for this — could you please help me understand whether Java is intentionally lagging on the gateway-driven knob?

I checked Azure/azure-sdk-for-java main and found:

$ grep -rn --include="*.java" "disableCrossRegionalHedging" \
    sdk/cosmos/azure-cosmos/src sdk/cosmos/azure-cosmos-tests/src
(no matches)

$ grep -n "IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF" \
    sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java
333:    private static final String DEFAULT_IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF = "true";
334:    private static final String IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF = "COSMOS.IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF";
335:    private static final String IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF_VARIABLE = "COSMOS_IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF";

Java currently has the related-but-distinct client-side JVM lever COSMOS.IS_READ_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF, but no server-driven override. Without one, an operator flipping the gateway flag will get inconsistent behavior across .NET vs Java clients on the same account — which is exactly what this knob seems designed to avoid. Java already has the GEM perPartitionAutomaticFailoverConfigModifier callback plumbing (GlobalEndpointManager.java:60), so mirroring this would be additive rather than new infrastructure. Python has no SDK-default PPAF hedging at all (default availability_strategy=False), so the gap there is largely cosmetic and probably worth a single note in spec.md calling Python out of scope.

---

Thanks again — happy to chat through any of these if helpful.

[NaluTripician]

Thanks @kushagraThapar — addressed both M1 and M4 in 6433c5afbc67c0ee31772737c760c604a3443ec2.

Re: M1 (Per-request RequestOptions.AvailabilityStrategy bypasses the operator override)

Going with option (a). The shipped spec.md (Requirement: Gateway flag disables all hedging when true → Scenario: PPAF account with request-level hedging override and flag set to true) already requires this to be blocked, and tasks 4.1 / 6.3 were exactly the unchecked items. Done in three small pieces:

1. DocumentClient — added internal bool IsHedgingDisabledByGateway that reads this.disableCrossRegionalHedging under hedgingStrategyLock (read-only, atomic).
2. RequestInvokerHandler.AvailabilityStrategy(RequestMessage) — short-circuits to null when IsHedgingDisabledByGateway is true, before resolving any per-request or client-level strategy. Operator override now has absolute precedence regardless of where the strategy was configured.
3. GatewayHedgingOverrideTests — added RequestInvokerHandler_FlagTrue_PerRequestStrategy_ReturnsNull covering task 6.3 (and the symmetric RequestInvokerHandler_FlagFalse_HonorsPerRequestStrategy for regression protection).

Tasks 4.1 and 6.3 ticked off in tasks.md.

Re: M4 (Java parity)

Status:

• Java: confirmed the grep — disableCrossRegionalHedging is not yet in azure-sdk-for-java. Filed Cosmos: Add gateway-driven disableCrossRegionalHedging handling (mirror .NET PR #5829) azure-sdk-for-java#49192 (mirror the .NET PR; reuse the existing perPartitionAutomaticFailoverConfigModifier callback plumbing in GlobalEndpointManager.java). Acceptance criteria explicitly include the HasValue guard (M2 in Java too) and the per-request precedence guarantee (M1 in Java too).
• Python: agreed it's cosmetic — no SDK-default PPAF hedging, so the per-request override path that motivates this knob doesn't exist there.

Added a ## Cross-SDK Parity section to proposal.md recording this status: .NET implements the gateway-driven knob; Java has a tracking issue and is expected to mirror; Python is explicitly out of scope. Keeps the in-tree design record honest about what's shipping where and avoids the inconsistent-behavior-across-clients trap you flagged.