Repository navigation
De-flake ShardRegionQueriesSpecs: async cluster formation, realistic Ask timeouts, proxy-side logging - #8618
Merged
Aaronontheweb merged 1 commit intoSep 23, 2026
Conversation
… asking with 1-3 s timeouts ShardRegion_should_support_GetEntityLocation_query_remotely timed out on its first Ask through a freshly started proxy region on a 2-vCPU Windows agent. The Sys-side association to the proxy system was dropped (EndpointDisassociatedException, address gated for 5 s) while that Ask was in flight, and the 3 s timeout died inside the gate. Even without a dropped link the first message through a new region can need the region's registration retries, which settle at 2 s apart, so 3 s had no slack. - Entity Asks use Dilated(10s), GetEntityLocation queries Dilated(3s). - The region start and the 2-node cluster formation move from the ctor's synchronous AwaitAssert into InitializeAsync (StartAsync + JoinAsync), disposing the class if that throws, since xUnit will not. - The proxy system's log is wired to the test output: why the proxy side closed the association is not visible in this run's log, and the next failure should show it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ShardRegionQueriesSpecs.ShardRegion_should_support_GetEntityLocation_query_remotelyfailed on a 2-vCPU Windows agent (in a #8606 build, which doesn't touch sharding or remoting) withAskTimeoutException: Timeout after 3.00 secondson its firstAskthrough a new proxy region.What happened
Sysand the proxy system associateSysmarks the proxy node UNREACHABLEEndpointDisassociatedException: Disassociated; the address is gated for 5 sAskruns and dies inside the gatestable-after = 20sbefore acting, so it couldn't have acted this early.Disassociate(Unknown). The proxy system's log wasn't connected to the test output, so this run can't say why. The root cause of the disconnect is still open. This PR adds the logging needed to see it next time.A healthy link would not have saved the test either. The first message through a new region waits on the region's registration retries (250 ms, 500 ms, 1 s, then every 2 s;
GetShardHomeis re-sent on that tick). One lost message plus a gate cycle costs about 7–8 s, so 3 s had no slack.Change
Asks now useDilated(10s), andGetEntityLocationqueries useDilated(3s), in all three tests.StartAsyncplus the 2-node cluster formation move from the constructor's synchronousAwaitAssertintoInitializeAsync, usingJoinAsync, which waits for the realMemberUp. It follows the pattern inClusterShardingLeaseSpec. If setup throws, the class disposes itself, because xUnit v3 doesn't dispose a class whoseInitializeAsyncfailed.InitializeLogger(_proxySys, "[proxy]")sends the proxy system's log to the test output.Verification (local)