Skip to content

De-flake HubsDocTests dynamic merge: async expectation with an explicit 10s timeout - #8664

Merged
Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/deflake-hubs-doc-dynamic-merge
Sep 29, 2026
Merged

Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/deflake-hubs-doc-dynamic-merge

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Symptom

DocsExamples.Streams.HubsDocTests.Hubs_must_demonstrate_creating_a_dynamic_merge failed on the Windows agent (build 132091, net10.0, 8 s) with OperationCanceledException thrown from ExpectMsgAllOf. That PR only touched extension loading, so the failure was unrelated to it.

Root cause

The test called the sync ExpectMsgAllOf(params T[]), which cancels a token after the 3 s default. The stack shows the throw at cancellationToken.ThrowIfCancellationRequested() in InternalReceiveNAsync: the first element was received, but only at the deadline. So the stream delivered late, not never. Two details point to a stalled agent: the test took 8 s in total when the wait was 3 s, and it normally runs in about 30 ms.

MergeHub has no race here. Producers that start before the hub stage enqueue their Register/Element/Deregister events in FIFO order, and the hub drains them on its first pull. Both elements always arrive. ExpectMsgAllOf does not depend on order. Pekko's HubsDocSpec is the same code with the same default timeout.

Fix

  • Make the test async Task and use await ExpectMsgAllOfAsync(TimeSpan.FromSeconds(10), ...). It waits on the real signal (both elements reaching the TestActor), no longer blocks a pool thread (half the pool minimum on a 2-vCPU agent), and has an explicit budget that costs nothing when the test passes.
  • The #region merge-hub example code is unchanged.

Verification

  • Repro before the fix: a temporary Thread.Sleep in the test's consumer delegate made the old test fail every time with OperationCanceledException. At 2985 ms, 2 of 4 runs matched the CI stack exactly (throw at TestKitBase_Receive.cs:825).
  • With the same delay (1600 ms and 2985 ms, 4 runs each), the fixed test passed every time. Then I removed the delay.
  • Dynamic merge test: 30/30 passes. Whole HubsDocTests class: 10/10 passes, plus 10/10 with DOTNET_PROCESSOR_COUNT=2.

…it 10s timeout

The test used the sync ExpectMsgAllOf with the 3s default. On a stalled
Windows agent the first element reached the TestActor right at the
deadline and the expectation threw OperationCanceledException. MergeHub
always delivers both elements, so wait on them asynchronously with a
10s budget instead.
@Aaronontheweb
Aaronontheweb merged commit 6ef46df into akkadotnet:dev Sep 29, 2026
16 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/deflake-hubs-doc-dynamic-merge branch September 29, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant