Skip to content

De-flake Bugfix5962Spec.SBR_Should_work_with_channel_executor - #8398

Merged
Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/de-flake-bugfix5962spec
Jul 17, 2026
Merged

Aaronontheweb merged 1 commit into
akkadotnet:devfrom
Aaronontheweb:fix/de-flake-bugfix5962spec

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Member

Root-caused via paired Akka.NET + concurrency specialist analysis after it failed #8396's Windows run (pre-existing flake; that PR doesn't touch Akka.Cluster).

Root cause: MemberUp structurally cannot arrive before the 1s periodic-tasks-initial-delay leader tick, but the spec gated on a raw, non-dilated TestLatch.Ready(1s) — a 100–400ms margin that Windows CI load eats. Secondary: non-dilated 1s ResolveOne racing fire-and-forget cluster-extension init, and a fixed DotNetty port (15508) exposed to cross-process bind collisions.

Fix (test-local, regression coverage for #5962 fully preserved): dilated AwaitAssertAsync around ResolveOne; TaskCompletionSource(RunContinuationsAsynchronously) + dilated wait replaces the latch (moved outside the EventFilter block — no continuation inlining onto the member-listener's dispatcher); explicit dilated 2s dead-letter quiet window (> SBR's 1s self-tick, so a dead resolver still trips it); port = 0 + explicit Join(SelfAddress).

Validation: 50/50 consecutive Release runs (two 25× loops), clean build, format clean on the touched file.

… member-up signal

Root cause: MemberUp cannot arrive before the 1s periodic-tasks-initial-delay
leader tick, but the spec waited on a raw non-dilated TestLatch.Ready(1s) --
a 100-400ms margin Windows CI load routinely eats. Also a non-dilated 1s
ResolveOne raced fire-and-forget cluster-extension init, and a fixed DotNetty
port (15508) could collide across processes on shared agents.

- ResolveOne wrapped in dilated AwaitAssertAsync (still fails if the downing
  provider is dead -- the akkadotnet#5962 regression signal)
- TestLatch -> TaskCompletionSource(RunContinuationsAsynchronously) awaited
  with a dilated timeout OUTSIDE the EventFilter block (no continuation
  inlining onto the member-listener's channel-executor thread)
- EventFilter.DeadLetter quiet window via the explicit dilated 2s overload
  (> SBR's 1s self-tick, so a dead resolver is still detected; test ~3s
  faster)
- port = 0 + explicit cluster.Join(SelfAddress) replaces fixed port/seed-nodes

Verified 50/50 consecutive Release runs (net10.0).

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - although we should probably deprecate and remove the ChannelExecutor

@Aaronontheweb
Aaronontheweb merged commit 5f225ed into akkadotnet:dev Jul 17, 2026
11 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/de-flake-bugfix5962spec branch July 17, 2026 15:11
Aaronontheweb added a commit that referenced this pull request Sep 12, 2026
… member-up signal (#8398)

Root cause: MemberUp cannot arrive before the 1s periodic-tasks-initial-delay
leader tick, but the spec waited on a raw non-dilated TestLatch.Ready(1s) --
a 100-400ms margin Windows CI load routinely eats. Also a non-dilated 1s
ResolveOne raced fire-and-forget cluster-extension init, and a fixed DotNetty
port (15508) could collide across processes on shared agents.

- ResolveOne wrapped in dilated AwaitAssertAsync (still fails if the downing
  provider is dead -- the #5962 regression signal)
- TestLatch -> TaskCompletionSource(RunContinuationsAsynchronously) awaited
  with a dilated timeout OUTSIDE the EventFilter block (no continuation
  inlining onto the member-listener's channel-executor thread)
- EventFilter.DeadLetter quiet window via the explicit dilated 2s overload
  (> SBR's 1s self-tick, so a dead resolver is still detected; test ~3s
  faster)
- port = 0 + explicit cluster.Join(SelfAddress) replaces fixed port/seed-nodes

Verified 50/50 consecutive Release runs (net10.0).

(cherry picked from commit 5f225ed)
Aaronontheweb added a commit that referenced this pull request Sep 12, 2026
Adds a `1.5.72 TBD` section covering every user-visible change in this PR, including the
non-blocking Cluster startup fix and a migration note for #8574's write-timeout key change.

Review follow-up (Opus review, findings 2, 3, 5, 10, 11):
- The #8359 bullet now states outright that the cluster's internal actor tree
  (/system/cluster/core, /system/cluster/core/daemon, /system/cluster/heartbeatReceiver) is
  created after Cluster.Get() returns rather than before it - the one sharp edge of this change,
  and previously the only thing this note left unsaid - and says code resolving those paths
  right after Cluster.Get() should await JoinAsync/RegisterOnMemberUp or a cluster event instead.
- Says outright that akka.actor.creation-timeout is no longer consulted by Akka.Cluster at all
  (stronger and more accurate than "no longer blocks on"), while noting Akka.Cluster.Sharding
  still uses it to bound its own start-up asks, so nobody reads this as license to revert that
  setting.
- Replaced "a small dedicated dispatcher pool" with wording that also covers a starved default
  pool, since the deadlock reproduces on either.
- The #8580 bullet no longer says "during startup" - the routing through /system/cluster is
  permanent for the extension's lifetime (two extra local mailbox hops on the low-traffic command
  path), and now says a `Leave`/`LeaveAsync()` call issued after the node has already shut down
  produces a dead-lettered ClusterUserAction.Leave, relevant to dead-letter alerting.
- Adds a Testing bullet for #8398 (De-flake Bugfix5962Spec), the dev de-flake this branch had been
  missing for the spec #8359 makes newly racy.
Aaronontheweb added a commit that referenced this pull request Sep 12, 2026
The #8359 cherry-pick brought in ClusterStartupFuzzSpec.cs, which uses
Task.IsCompletedSuccessfully at two call sites, and the #8398 cherry-pick (5f225ed, applied
earlier in this branch as the Opus-review follow-up for finding 1) uses the non-generic
`new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously)` /
`TrySetResult()` in Bugfix5962Spec.cs. On dev, Akka.Cluster.Tests no longer targets net48, so
both compiled fine there. On v1.5, Akka.Cluster.Tests still targets net48 alongside net10.0:
`Task.IsCompletedSuccessfully` does not exist on .NET Framework 4.8 (CI build 131554 failed with
CS1061 on every net48 lane), and the non-generic `TaskCompletionSource` does not exist before
.NET 5 (CS0305).

- ClusterStartupFuzzSpec.cs: switched both call sites to
  `joinTask.Status == TaskStatus.RanToCompletion`, the exact equivalent available on all target
  frameworks (the `using System.Threading.Tasks;` needed for `TaskStatus` was already present).
- Bugfix5962Spec.cs: switched to `TaskCompletionSource<Done>` / `TrySetResult(Done.Instance)`,
  the same pattern already used by ClusterDaemon's own `_clusterPromise`/`_selfExiting` fields -
  available on every target framework this project builds for. No behavior change: `Done`
  carries no data, so the task's completion is still the only thing the spec observes.

Verified:
- `dotnet build src/core/Akka.Cluster.Tests -c Release -f net48 -warnaserror` and
  `-f net10.0 -warnaserror`: both 0 warnings/errors.
- `dotnet test -f net10.0 --filter FullyQualifiedName~ClusterStartupFuzzSpec` passes both tests:
  aggressive run completed=72, kills(delayed=0, immediate=0), pass=72 (reachedUp=72); moderate
  run completed=60, kills(delayed=0, immediate=0), pass=60 (reachedUp=60).
- `dotnet test -f net10.0 --filter FullyQualifiedName~Bugfix5962Spec`: passes.
This was referenced Oct 4, 2026
This was referenced Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant