Skip to content

De-flake InboxSpec: receive through ReceiveAsync so a deadline failure surfaces as TimeoutException - #8564

Merged
Aaronontheweb merged 2 commits into
fix/cluster-deathwatch-spec-resolve-end-actorfrom
fix/inbox-spec-receive-async
Sep 10, 2026
Merged

Aaronontheweb merged 2 commits into
fix/cluster-deathwatch-spec-resolve-end-actorfrom
fix/inbox-spec-receive-async

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

Stacked on #8563.

What changes

InboxSpec receives through ReceiveAsync instead of the blocking Receive, and the two facts that assert a timeout assert the exact exception type again.

  • The 1000-message drain, the selective receive, and the queued-queries fact await ReceiveAsync.
  • Inbox_have_maximum_queue_size asserts TimeoutException with Assert.ThrowsAsync.
  • Inbox_have_a_default_and_custom_timeouts gets its exact-type assertions back. Fix flaky tests: InboxSpec, ShardRegionSpec, and scheduler interval test #8035 had loosened them to accept an AggregateException as well, through a helper that is now deleted.
  • ReceiveWhere calls stay as they are; there is no async overload, and their timeouts never expire in these facts. The queued-queries fact runs them under Task.Run with awaited delays instead of thread sleeps.
  • The negative-deadline fact asserts TimeoutException exactly. A deadline already in the past makes the inbox actor kick itself at once and answer with that one exception type, so nothing else can come out.

No change to Inbox itself. It is deprecated and slated for removal in #7946.

Why

Build 131326, Linux unit tests, on a PR that touched a different spec: Inbox_have_maximum_queue_size expected TimeoutException and got an AggregateException wrapping TimeoutException("Deadline passed").

The sync Receive(timeout) blocks on task.Wait(timeout). The inbox actor schedules its own deadline for the same duration and faults the task with a Status.Failure carrying the timeout. When the actor's deadline fires before the wall-clock wait gives up, Task.Wait throws the aggregate, and the unwrap code written for the faulted case, right after the wait, never runs. Which side wins is a race between the scheduler and the wall clock over the same second. The same test was reported flaky in 2014 (#340), and the sibling fact in 2025 (#7934), which was closed as won't-fix because the Inbox is going away; #8035 then loosened the sibling's assertion instead.

ReceiveAsync is the Ask with the deadline and no wall-clock wait, so awaiting it surfaces the actor's timeout directly. That removes the race rather than tolerating either outcome, and it is the async form the touched-file rule asks for.

How it was checked

dotnet build src/core/Akka.Tests -c Release -warnaserror clean. The InboxSpec class run ten times: 7 passed each time. The grep for blocking receives, waits, results, and synchronous throw assertions on the file returns only ReceiveAsync and ReceiveWhere calls. Test-only change. No ledger entry.

Second commit

From the adversarial review: the two Task.Factory.StartNew blocks with thread sleeps became Task.Run with awaited delays, and the third timeout fact's any-exception assertion became the exact type, which the negative-deadline path produces deterministically. InboxSpec run five times: 7 of 7 each time.

…e surfaces as TimeoutException, and restore the exact-type assertions

Inbox.Receive(timeout) waits on the wall clock while the inbox actor schedules its own deadline for the same duration; when that deadline wins the race, Task.Wait rethrows the actor's Status.Failure(TimeoutException) wrapped in AggregateException instead of the bare TimeoutException the tests expect. Awaiting ReceiveAsync instead surfaces the exception directly with no race, so this migrates every InboxSpec receive call to the async API and drops the IsTimeoutException either-type workaround added by an earlier fix.
…ct exception type on the negative-deadline fact

Task.Run + Task.Delay staggers the two ReceiveWhere calls without parking
a pool thread; ReceiveWhere itself has no async overload and isn't in the
race, so it stays as-is.

Traced the negative-deadline fact through Inbox.Actor.cs's Kick handling
and FutureActorRef<object>.TellInternal: a deadline already in the past
always produces a bare TimeoutException("Deadline passed"), never
anything else, so ThrowsAnyAsync<Exception> is tightened to
ThrowsAsync<TimeoutException>.
@Aaronontheweb
Aaronontheweb force-pushed the fix/inbox-spec-receive-async branch from f1ce4d7 to bedcdad Compare September 10, 2026 14:54
@Aaronontheweb
Aaronontheweb merged commit 989e299 into dev Sep 10, 2026
4 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/inbox-spec-receive-async branch September 10, 2026 15:27
Aaronontheweb added a commit that referenced this pull request Sep 12, 2026
…e surfaces as TimeoutException (#8564)

* De-flake InboxSpec: receive through ReceiveAsync so a deadline failure surfaces as TimeoutException, and restore the exact-type assertions

Inbox.Receive(timeout) waits on the wall clock while the inbox actor schedules its own deadline for the same duration; when that deadline wins the race, Task.Wait rethrows the actor's Status.Failure(TimeoutException) wrapped in AggregateException instead of the bare TimeoutException the tests expect. Awaiting ReceiveAsync instead surfaces the exception directly with no race, so this migrates every InboxSpec receive call to the async API and drops the IsTimeoutException either-type workaround added by an earlier fix.

* InboxSpec: queued-queries fact awaits delays instead of sleeping; exact exception type on the negative-deadline fact

Task.Run + Task.Delay staggers the two ReceiveWhere calls without parking
a pool thread; ReceiveWhere itself has no async overload and isn't in the
race, so it stays as-is.

Traced the negative-deadline fact through Inbox.Actor.cs's Kick handling
and FutureActorRef<object>.TellInternal: a deadline already in the past
always produces a bare TimeoutException("Deadline passed"), never
anything else, so ThrowsAnyAsync<Exception> is tightened to
ThrowsAsync<TimeoutException>.

(cherry picked from commit 989e299)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant