[Internal] Tests: Fixes test reliability across flaky and nightly tests

[NaluTripician]

Summary

This PR improves the reliability of tests across the nightly rolling pipeline and the flaky-tagged test suite. Changes are targeted at the root causes of intermittent failures identified from rolling test data analysis.

Changes

Tier 1: High-Impact Fixes (>5% failure rate in nightly rolling)

AvailabilityStrategyNoTriggerTest (10.8% failure rate — 23/212 runs)

File: CosmosAvailabilityStrategyTests.cs

• Root cause: Injected response delay (200ms) barely exceeded the hedging threshold (150ms), leaving only 50ms margin. On loaded CI agents, timing jitter caused inconsistent behavior.
• Fix: Increased delay to 500ms and threshold to 300ms, providing a 200ms safety margin.

DistributedTransaction E2E Tests (8.6% failure rate — 16/187 runs, 4 tests)

File: DistributedTransactionE2ETests.cs
Tests: ValidateConflictResponseReturnsErrorStatus, ValidateHappyPathRequestAndResponse, ValidateMixedOperationsRequestStructure, ValidateResponseDeserializesCorrectly

• Root cause: All 4 tests share a TestInitialize that creates a new container via BaseCosmosClientHelper.TestInit() (which calls DeleteAllDatabasesAsync). Concurrent test execution caused resource contention on the emulator.
• Fix: Added [DoNotParallelize] attribute and retry logic with exponential backoff around container creation.

AvailabilityStrategyAllFaultsTests (5.2% failure rate — 11/212 runs)

File: CosmosAvailabilityStrategyTests.cs

• Root cause: 90 DataRow combinations with a 100ms hedging threshold that was too tight for CI environments.
• Fix: Increased hedging threshold from 100ms to 200ms.

AppCancellationDuringHedging_DoesNotSpawnNewHedgeRequests (5.4% failure rate — 9/168 runs)

File: AvailabilityStrategyUnitTests.cs

• Root cause: Threshold of 10ms and app cancellation at 15ms gave only 5ms margin. Task.Delay precision at these scales is unreliable on loaded CI agents.
• Fix: Increased threshold from 10ms to 100ms, thresholdStep from 10ms to 100ms, and cancellation delay from 15ms to 150ms (10x margin increase).

Tier 2: Medium-Impact Fixes (1-3% failure rate)

CircuitBreaker Integration Tests (2.8% and 1% failure rates)

File: CosmosItemIntegrationTests.cs

• Root cause: 3000ms replication delay insufficient for multi-region data propagation on slow networks.
• Fix: Increased replication delay from 3s to 5s for both ReadItemAsync_WithCircuitBreakerEnabled...ThirdRegion and ReadItemAsync_WithCircuitBreakerDisabled... tests.

ValidatesCongestionControlAsync (0.3% failure rate — 1/359 runs)

File: BatchAsyncStreamerTests.cs

• Root cause: Fixed Task.Delay(2000) followed by immediate assertion on semaphore count. On slow CI, background semaphore release may not have completed.
• Fix: Replaced with polling loop (200ms intervals, 10s timeout) that waits for the semaphore count to reach the expected value.

Controller_ShouldReleasesLease_IfObserverExits (0.3% failure rate — 1/359 runs)

File: PartitionControllerTests.cs

• Root cause: 100ms delay insufficient for async observer completion and lease release.
• Fix: Increased delay from 100ms to 500ms.

Flaky-Tagged Test Improvements

CosmosHttpClientCoreTests (RetryTransientIssuesTestAsync, RetryTransientIssuesForQueryPlanTestAsync)

• Root cause: Delays only 0.1s beyond timeout policy thresholds, causing race conditions.
• Fix: Increased delay margins to ~1s beyond policy thresholds. Added [Timeout(120000)] (2 min) to both tests.

GlobalEndpointManagerTest (EndpointFailureMockTest)

• Root cause: 3s delay with 2s UnavailableLocationsExpirationTimeInSeconds left only 1s margin.
• Fix: Increased delay to 5s (3s margin). Added [Timeout(30000)].

CosmosAuthorizationTests (TestTokenCredentialBackgroundRefreshAsync)

• Root cause: Unbounded polling loop (while (NumTimesInvoked == 1) await Task.Delay(500)) could spin indefinitely.
• Fix: Added bounded polling with Stopwatch and 20s timeout with descriptive assertion message.

ClientTelemetryTests (all 12 test methods)

• Root cause: No [Timeout] attributes on any tests in this flaky class, allowing individual tests to consume the entire 60-minute job budget.
• Fix: Added [Timeout(300000)] (5 min) to all 12 test methods.

EndToEndTraceWriterBaselineTests (QueryAsync, TypedPointOperationsAsync)

• Root cause: No [Timeout] attributes despite being marked [TestCategory("Flaky")].
• Fix: Added [Timeout(300000)] (5 min) to both tests.

Test Impact

Category	Tests Improved	Expected Improvement
Nightly rolling (>5% failure)	4 test groups	Eliminate timing races
Nightly rolling (1-3% failure)	4 tests	Reduce replication/sync failures
Flaky-tagged tests	8 tests across 6 files	Bounded timeouts prevent job overruns

Pipeline Duration Impact

The [Timeout] additions to flaky-tagged tests prevent any single test from consuming the entire 60-minute pipeline job timeout. Previously, a hanging flaky test with retryCountOnTaskFailure: 4 could consume the full budget across retries.

[github-actions]

All good!

[Pilchie]

It would be so much better if we could convert these delays into deterministic processing via ManualResetEvent's.

[NaluTripician]

Thanks for the feedback @Pilchie! Agreed — the latest commits (054aa8e and 7c831e1) already converted several tests to deterministic patterns:

• **\AppCancellationDuringHedging**: Now uses \TaskCompletionSource\ with \ct.Register()\ — fully deterministic, no timing dependency
• **\PartitionControllerTests**: Now polls with \Mock.Verify\ in a bounded loop instead of a fixed \Task.Delay(100ms)\
• **\GlobalEndpointManagerTest**: Now polls for the endpoint to switch back instead of a fixed sleep

For the remaining delay-based tests (replication delays in circuit breaker tests, timeout policy threshold tests), deterministic replacements are harder since they depend on actual time-based SDK behavior (timeout policies fire via real timers, cross-region replication needs real propagation time). Happy to explore further if you have specific tests in mind.

[NaluTripician]

Closing this PR in favor of 5 smaller, impact-grouped PRs for independent validation:

High Impact:

• [Internal] Tests: Fixes Availability Strategy emulator test timing margins #5713 — Availability Strategy timing fixes (78 failures)
• [Internal] Tests: Adds DoNotParallelize to DistributedTransaction E2E tests #5711 — DistributedTransaction DoNotParallelize (64 failures)
• [Internal] Tests: Fixes AppCancellationDuringHedging with deterministic synchronization #5712 — AppCancellationDuringHedging deterministic fix (29 failures)

Medium Impact:

• [Internal] Tests: Fixes circuit breaker replication delay for CI reliability #5714 — Circuit Breaker replication delay (11 failures)

Low Impact (bundled):

• [Internal] Tests: Adds timeouts and polling loops for low-impact test reliability #5715 — Timeouts + polling loops across 7 files (~14 failures)

Dropped: CosmosItemThinClientTests.cs whitespace-only changes (no functional impact).