fix(tests): deterministically gate two Windows-flaky actor cold-start tests - #1572
Merged
Aaronontheweb merged 2 commits intoJul 3, 2026
Merged
Conversation
Both tests failed only on the Windows CI runner (Ubuntu + macOS green) with timeouts, not assertion-logic failures. Root cause in both: a fixed short timeout racing a persistent-actor cold start under ThreadPool starvation. - DiscordSessionBindingContractTests.Approval_response_sends_feedback: the default 3s AwaitAssert poll raced the binding actor's full cold start (recovery -> init -> hydrate -> active -> unstash -> render). Under CPU starvation the poll loop got only ~2 attempts before the deadline. Gate on `await pipeline.Created.WaitAsync(ct)` first -- a linear await on the real readiness signal, matching the Reminder_delivery_* / Stashes_messages_during_init siblings -- so the 3s poll only covers the fast in-process output tail. - SessionMemoryObserverActorTests.DistillMemories_only_persists_accepted_proposals_for_future_dedup: the 5s ExpectMsg budget was consumed by first-persistent-actor journal cold start + recovery (commands stash until RecoveryCompleted). Gate on an empty RecordAcceptedDistillationProposals Ask -- answered immediately post-recovery with no Persist and no state change, a side-effect-free readiness ack -- so the behavioral windows are measured from a warm actor. No production change: stash-until-RecoveryCompleted is the correct Akka synchronization; this is a test cold-start-budget artifact. No Thread.Sleep / Task.Delay introduced -- both fixes block on real signals.
Aaronontheweb
enabled auto-merge (squash)
July 3, 2026 19:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two actor tests flake only on the Windows CI runner (Ubuntu + macOS stay green in the same job). They surfaced most recently on the
Test-windows-latestjob of #1539, but neither test's file — nor any actor source — is touched by that PR, so these are pre-existing flakes that can red-X any PR:DiscordSessionBindingContractTests.Approval_response_sends_feedback→
Assert.Contains() Failure: Filter not matched in collectionafterAwaitAssert failed, timeout [00:00:03] is over after [2] attemptsSessionMemoryObserverActorTests.DistillMemories_only_persists_accepted_proposals_for_future_dedup→
Timeout 00:00:05 while waiting for a message of type SessionDistillationCompletedRoot cause (same class in both)
A fixed short timeout racing a persistent-actor cold start under ThreadPool starvation. The assertions are correct; the deadlines were sized for the work, not the worst-case scheduling latency on a saturated Windows runner. The
2 attempts in 3.66sdetail is the tell — the poll loop's retry continuation itself was starved, so it wasn't "the work took 3s," it was "the harness couldn't run."DiscordSessionBindingActoris aReceivePersistentActor. Output only renders afterrecovery → InitializePipeline → hydration → Active → UnstashAll; everyOutputReceivedbeforeActiveis stashed. The firstAwaitAssertAsync(default 3s, fromakka.test.default-timeout) had no readiness gate, so it polledGetPostedTexts()while that entire cold start was still in flight. CI logged "session pipeline initialized" ~3 ms after the poll gave up.SessionMemoryObserverActoris aReceivePersistentActorwhose command handlers stash untilRecoveryCompleted. It's an early persistent actor, so the in-memory journal/snapshot plugin cold start + recovery consumed the 5sExpectMsgbudget before the (fast)FakeChatClientdistillation ran. CI loggedsession_observer_recovery_completeright at the deadline.This is not a production defect — stash-until-
RecoveryCompletedis the correct Akka.Persistence synchronization the parent legitimately relies on. It's a test cold-start-budget artifact, so the fix lives in the tests.Fix — gate on real readiness signals (no widened timeouts, no sleeps)
await pipeline.Created.WaitAsync(ct)before the poll — a linear await on the signalRecordingSessionPipelinealready exposes (completes insideCreateAsync, once recovery + init are done). This is byte-for-byte the pattern theReminder_delivery_*andStashes_messages_during_initsiblings already use. The remaining tail is in-process, millisecond-scale, and the 3s poll now covers only that.Askan emptyRecordAcceptedDistillationProposals([])first. That command is answered immediately post-recovery with noPersistand no state mutation (empty-list early return), so it's a side-effect-free readiness ack; the generous 30s ceiling absorbs cold start without polling and the behavioral 5s windows are then measured from a warm actor. Mirrors the existingReminderManagerActorTestsgenerous-Askprecedent and CLAUDE.md's "Ask<Ack>so callers know a state transition occurred" rule.Neither fix introduces
Thread.Sleep/Task.Delay; both block on a real signal.Validation
dotnet build src/Netclaw.Actors.Tests— clean (0 warnings/errors)DiscordSessionBindingContractTests+SessionMemoryObserverActorTestsclasses — 83/83 passdotnet slopwatch analyze— 0 issues./scripts/Add-FileHeaders.ps1 -Verify— all headers presentScope / follow-up
This fixes exactly the two tests that failed. Two independent specialist passes (Akka lifecycle + .NET concurrency) flagged that ~40 sibling tests across
SessionBindingContractTests,DiscordSessionBindingContractTests, andSessionMemoryObserverActorTestsshare the same anti-pattern (a fixed poll/ExpectMsgracing first-actor cold start), and thatDistillMemories_records_accepted_proposals_for_recovery_after_ackis a near-twin of the observer flake. I kept this PR tightly scoped to the observed failures rather than rewriting the family. If the maintainers want suite-wide insurance, the low-risk lever is a class-levelakka.test.timefactoron those base test classes (the concurrency pass's recommendation) — happy to open a separate PR for that.