De-flake MNTR specs: stop inner waits from consuming their enclosing Within budget

[Aaronontheweb]

Second batch of deterministic MNTR de-flakes. All four commits share one root cause family: inner waits are allowed to consume the budget of the block that was supposed to bracket them, so the work that follows inherits little or no time and fails instantly. No timeout was widened to fix any of these.

The mechanism

AwaitAssertAsync resolves its bound as var max = RemainingOrDilated(duration) — it does not clamp to the enclosing Within. ExpectMsgAsync/EnterBarrierAsync with no explicit duration resolve via RemainingOrDefault, i.e. the enclosing Within's remainder, clamped at zero. So a phase written as Within(30s) (dilated to 90s at timefactor = 3) can have a single inner operation legally claim all 90s, after which every later expectation and barrier in that phase gets ≈0.

That produces both CI signatures we chased: Assert.Equal() Expected: 5, Actual: 4 (a convergence wait given no time to converge in) and Timeout 00:00:00 (an expectation given a zero budget).

Commits

1. Fix double-dilated timeouts — four call sites passed Dilated(x) into TestKit methods that dilate again (RemainingOrDilated → Dilated), squaring the factor. StressSpec.cs:842 turned a 3s wait into 27s, matching the Timeout 00:00:27 while waiting for ActorIdentity failures in Azure build 129447. Waits get shorter and retries ~3x denser. Sites: StressSpec, ClusterShardingQueriesSpec (x3).

2. Correct the timefactor note in StressSpec — the config comment claimed Within(max) is not scaled by akka.test.timefactor and that bounds were hand-sized to compensate. That is false: TestKitBase.WithinAsync applies Dilated(max) and every overload routes through it. That false premise is what invited the pre-multiplied bounds and pre-dilated durations fixed in (1).

3. StressSpec: resolve the result aggregator once per step — each phase looked the same aggregator up to three times (CreateResultAggregatorAsync, ReportResult, AwaitClusterResultAsync), each a retrying remote Identify bounded at AwaitAssertAsync(..., 30s) → 90s, i.e. the whole phase. Whichever retried consumed the phase and starved the AwaitMembersUpAsync(size, timeout: RemainingOrDefault) that followed. The aggregator is one actor per Step, created before the result-aggregator-created-<Step> barrier, so it provably exists after that barrier — resolve once and reuse, and on the creating node keep the local ActorOf reference so it never resolves its own actor remotely. One lookup per node per phase instead of three.

4. De-flake ClusterShardingFailureSpec — both phases wrapped 5 barriers, a deliberate 1s ExpectNoMsg, a journal blackhole/restore cycle, 8 ExpectMsg<Value> calls and 2 AwaitAssert loops in a single WithinAsync(20s). EnterBarrierAsync also derives its timeout from the ambient remainder (MultiNodeSpec.cs:603), so barriers silently traded their 30s default for scraps. Remove the wrappers so barriers get their real bound back, give each ordinary expectation an explicit bound, and make the one recovery-critical check a retrying AwaitAssert bounded by the spec's own coordinator-failure-backoff/shard-failure-backoff/journal-timeout constants. Same fix shape as f13a8fa for this bug class. Shrinking the original Within to 4s reproduces the exact CI three-node signature, confirming the mechanism rather than correlating with it.

Verification

• StressSpec: passes locally, 10/10 nodes, 2m35s.
• ClusterShardingFailureSpec: both Persistent and DData variants pass, including under sustained CPU saturation.
• CircuitBreaker/sharding-queries builds clean.

Test-only changes. A related framework bug found during this work (TestKitBase_Expect.cs:207 double-dilates) is filed separately since it is shipped TestKit code.