CrossRegionHedgingAvailabilityStrategy: Fixes ArgumentNullException race condition in hedging cancellation

[FabianMeiswinkel]

CrossRegionHedgingAvailabilityStrategy: Fixes ArgumentNullException race condition in hedging cancellation

Bug

Multiple production customers reported unobserved ArgumentNullException: Value cannot be null. (Parameter 'request') crashes originating from CrossRegionHedgingAvailabilityStrategy.RequestSenderAndResultCheckAsync. The exception was surfaced as a TaskScheduler_UnobservedTaskException, crashing the process. The affected code paths were:

• ContainerCore.ReadItemAsync → ReadItemStreamAsync → ProcessItemStreamAsync → RequestInvokerHandler.SendAsync → CrossRegionHedgingAvailabilityStrategy.ExecuteAvailabilityStrategyAsync

Root Cause

A race condition caused by passing the wrong CancellationToken to the sender delegate:

1. ExecuteAvailabilityStrategyAsync creates a hedgeRequestsCancellationTokenSource (linked to the app-provided CT) to coordinate hedge request lifecycle.
2. CloneAndSendAsync clones the request inside a using block and calls RequestSenderAndResultCheckAsync.
3. Bug: RequestSenderAndResultCheckAsync called sender.Invoke(request, cancellationToken) with the application-provided CancellationToken — not hedgeRequestsCancellationTokenSource.Token.
4. When hedge Region B returned a final result (e.g., 200 OK), hedgeRequestsCancellationTokenSource.Cancel() was called.
5. But the in-flight sender for Region A still held the app CT (e.g., CancellationToken.None), which was never cancelled.
6. The CloneAndSendAsync using block exited, disposing the cloned request.
7. The Region A sender continued executing with a reference to the now-disposed request → ArgumentNullException: Value cannot be null. (Parameter 'request').

A secondary issue: when the application CT was cancelled (e2e timeout), the hedge timer (linked to app CT) would fire, and the old code would blindly continue the loop attempting to clone and send new requests on a cancelled path.

Fix

Two changes in CrossRegionHedgingAvailabilityStrategy.cs:

1. Pass hedgeRequestsCancellationTokenSource.Token to sender.Invoke() instead of the app CT

This ensures that when any hedge gets a final result and calls hedgeRequestsCancellationTokenSource.Cancel(), all in-flight senders immediately see their CT cancelled and stop before the cloned request is disposed. The CancellationTokenSource and CancellationToken parameters were also consolidated into a single hedgeRequestsCancellationTokenSource parameter passed through CloneAndSendAsync → RequestSenderAndResultCheckAsync.

2. Add do/while loop to handle spurious timer completions on app CT cancellation

When the app CT is cancelled (e2e timeout), the hedge timer fires via the linked CTS. The old code would continue the loop and try to clone a new request. The do/while loop now detects applicationProvidedCancellationToken.IsCancellationRequested and falls through to consolidate existing request outcomes instead of spawning new hedges.

Tests Added (8 new unit tests)

Test	Validates
HedgeCancellationCancelsInFlightRequests_NoNullRef	Slow primary request's CT is cancelled when a hedge returns a final result — core regression test
SenderReceivesHedgeCancellationToken_NotAppToken	Captures the actual CT passed to each sender and asserts all are from the hedge CTS, not the app CT
AppCancellationDuringHedging_DoesNotSpawnNewHedgeRequests	E2e timeout (app CT cancelled) does not spawn new hedge requests — validates the do/while loop fix
MultiRegionHedging_RequestNotAccessedAfterDisposal	Verifies the cloned request is still accessible when cancellation fires — exact scenario from the crash reports
HedgeCancellation_StreamRequest_NoNullRef	Tests the stream-based code path (ReadItemStreamAsync) from the NullRef2/NullRef3 stack traces
PrimaryRequestFinalResult_NoAdditionalHedgesSent	Fast primary response skips hedging entirely
AllHedgesTransientError_ReturnsLastResponse	All regions return transient errors — strategy returns last response without NullRef
ConcurrentHedgingRequests_NoNullRef	Stress test: 50 concurrent hedging requests with random delays — no NullRef under concurrency

Closing issues

To automatically close an issue: closes #5623

[github-actions]

All good!

[Copilot]

@kundadebdatta I've opened a new pull request, #5625, to work on those changes. Once the pull request is ready, I'll request review from you.