Skip to content

De-flake ORMultiValueDictionary_WithValueDeltas_LargeDataSet: replicate with WriteLocal and poll the replicas - #8611

Merged
Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/ddata-large-dataset-writelocal
Sep 23, 2026
Merged

Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/ddata-large-dataset-writelocal

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

ReplicatorSpecs.ORMultiValueDictionary_WithValueDeltas_LargeDataSet failed on the AOT stack's CI (#8604, Windows unit tests) with UpdateTimeout on its second WriteMajority update. The failure has nothing to do with that PR. This is the third de-flake of this test. The first two (#5686, #7899) raised the timeout; this one removes the cause.

Cause

  • ORMultiValueDictionary.SetItems clears and re-adds every element (the same as JVM ORMultiMap.put). Each write in this test ships the full 20,000-element set, delta included.
  • With WriteMajority, ReadWriteAggregator resends the full state to the primary and secondary after Timeout/5, which is 2 s here. On a 2-vCPU agent one hop already took about 2.5 s. So each write sent roughly three full payloads between three in-process ActorSystems sharing one thread pool.
  • The failing run's log shows heartbeats 2–5 s late, nodes flapping UNREACHABLE, and WriteAcks arriving 5–9 s after the aggregator had timed out.
  • Reachability does not gate WriteMajority; the flapping was a symptom of the CPU load.

Change

  • Both updates now use WriteLocal.
  • The test polls each replica's local copy (Get with ReadLocal, inside AwaitAssertAsync) until the entries equal the expected set, with a Dilated(30s) bound. The data still replicates across all three nodes, through delta propagation (every 400 ms) and gossip.
  • The synchronous ExpectMsg<Changed> calls, each of which blocked a pool thread for up to 10 s, are gone.

The test still catches the #5663 quadratic SetItems/merge regression it was written for. It no longer covers WriteMajority with a 20,000-element payload; the rest of ReplicatorSpecs still covers WriteMajority.

Verification (local)

  • 20/20 passes at 7–11 s each.
  • 10/10 passes at 10–15 s each, with every core pegged by busy loops.
  • The full ReplicatorSpecs class passes, 7/7.

…h WriteLocal and poll the replicas

SetItems clears and re-adds every element, so each update ships the full
20k-element set. Under WriteMajority the write aggregator resends that
full state to the primary and secondary acks every Timeout/5, and a hop
already took ~2.5s under CI load, so each write fanned out roughly three
full-state payloads across three ActorSystems sharing one thread pool -
causing UpdateTimeout flakiness on 2-vCPU CI agents. Two earlier fixes
(akkadotnet#5686, akkadotnet#7899) just raised the timeout. This switches the updates to
WriteLocal and polls each replica's local copy until it converges,
avoiding WriteMajority's resend amplification while still catching the
akkadotnet#5663 quadratic SetItems/merge regression the test guards.
@Aaronontheweb
Aaronontheweb merged commit edf98c6 into akkadotnet:dev Sep 23, 2026
15 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/ddata-large-dataset-writelocal branch September 23, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant