Skip to content

fix(tests): deterministically gate the DoHealthChecksAsync reentrancy-guard test#3331

Merged
jeremydmiller merged 1 commit into
mainfrom
fix/reentrancy-guard-test-race
Jul 7, 2026
Merged

fix(tests): deterministically gate the DoHealthChecksAsync reentrancy-guard test#3331
jeremydmiller merged 1 commit into
mainfrom
fix/reentrancy-guard-test-race

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

What

CoreTests.Runtime.Agents.leader_election_self_visibility_tests.reentrancy_guard_prevents_concurrent_DoHealthChecksAsync failed in full-suite net10.0 runs with:

Expected to receive exactly 1 call matching: TryAttainLeadershipLockAsync(any CancellationToken)
Actually received 2 matching calls

while passing in isolation and in full-suite net9.0 runs.

Diagnosis

The reentrancy guard in NodeAgentController.DoHealthChecksAsync (from #2999) was never broken — the race was in the test's synchronization. The test parked the winning call inside the guard with await Task.Yield() in the mocked TryAttainLeadershipLockAsync, assuming the other nine competing calls would launch before the winner resumed. Task.Yield() gives no such guarantee: the yielded continuation lands on an xUnit sync-context worker thread and can finish the entire mocked election path — releasing the guard — before the test thread even starts the second call. That second call then legitimately re-enters the guard and acquires the lock a second time. A warm thread pool plus net10.0 scheduling made the window reliable in full-suite runs; cold isolated runs and net9.0 timing happened to stay on the lucky side.

Fix (test only)

Park the winner on TaskCompletionSources instead — the mock signals entered and awaits release — so the guard is provably held while the nine competitors run. The test now also asserts:

  • competitors bounce off the guard with the AgentCommands.Empty singleton and never reach TryAttainLeadershipLockAsync
  • the winner still elects leadership after release
  • new coverage: the finally block frees the guard — a subsequent health check goes all the way through again

Verification

  • Full CoreTests suite, net10.0 (the previously failing configuration): 1860 passed, 0 failed
  • Test class in isolation: green on net10.0 and net9.0

🤖 Generated with Claude Code

…-guard test

reentrancy_guard_prevents_concurrent_DoHealthChecksAsync failed in
full-suite net10.0 runs with 'expected 1 call, actually received 2' on
TryAttainLeadershipLockAsync, while passing in isolation and on net9.0.

The guard itself was never broken. The test parked the winning call
inside the guard with 'await Task.Yield()' in the mocked lock
acquisition, assuming the other nine competing calls would launch before
the winner resumed. That is a timing gamble: the yielded continuation
lands on an xUnit sync-context worker and can finish the entire mocked
election path — releasing the guard — before the test thread starts the
second call, which then legitimately re-enters the guard and takes the
lock a second time. A warm thread pool plus net10.0 scheduling made that
window reliable in full-suite runs.

Park the winner on TaskCompletionSources instead (signal 'entered',
await 'release') so the guard is provably held while the competitors
run, assert they bounce off with AgentCommands.Empty, then release the
winner and additionally verify the finally block frees the guard for a
subsequent health check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 0298d27 into main Jul 7, 2026
43 of 47 checks passed
This was referenced Jul 9, 2026
This was referenced Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant