test: deflake async-delegation interrupt test under CI load - #64767
Merged
Conversation
test_interrupt_all_signals_running_children failed twice on a loaded CI worker: the blocker's ev.wait(timeout=5) expired before interrupt_all() ran, the record finalized on its own, and interrupt_all() found nothing running (n == 0, interrupted count 0 — the exact CI assertion failure, reproduced locally by inserting a 5.6s dispatch->interrupt gap). Raise the internal safety timeouts from 5s to 60s across the file's gated runners — they exist only as runaway guards (every test releases its gate explicitly); the pytest-level timeout is the real backstop. Same flake class as the removed test_crashed_runner_produces_error_ completion (#64431).
alt-glitch
added
type/test
The first CI failure was the 5s guard-timeout race; the rerun exposed a SECOND mechanism with a different signature: 'completed' == 'interrupted'. Prior tests (e.g. test_dispatch_rejected_at_capacity) release their gate and return immediately, but their workers finalize asynchronously — on a loaded runner the teardown drain races the in-flight _finalize, and the straggler 'completed' events leak into the NEXT test's queue, where _drain_one() picks one up instead of the interrupt event. Reproduced locally: gate release + immediate drain + 0.15s finalize delay leaked 2 events. Fixes: - teardown waits (bounded 2s) for active workers to finalize before draining, so events land in the owning test - the interrupt test matches its OWN delegation_id via _drain_for() instead of taking whatever event arrives first
13 tasks
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…arch#64767) * test: deflake async-delegation interrupt test under CI load test_interrupt_all_signals_running_children failed twice on a loaded CI worker: the blocker's ev.wait(timeout=5) expired before interrupt_all() ran, the record finalized on its own, and interrupt_all() found nothing running (n == 0, interrupted count 0 — the exact CI assertion failure, reproduced locally by inserting a 5.6s dispatch->interrupt gap). Raise the internal safety timeouts from 5s to 60s across the file's gated runners — they exist only as runaway guards (every test releases its gate explicitly); the pytest-level timeout is the real backstop. Same flake class as the removed test_crashed_runner_produces_error_ completion (NousResearch#64431). * test: fix cross-test completion-event leak (second flake mechanism) The first CI failure was the 5s guard-timeout race; the rerun exposed a SECOND mechanism with a different signature: 'completed' == 'interrupted'. Prior tests (e.g. test_dispatch_rejected_at_capacity) release their gate and return immediately, but their workers finalize asynchronously — on a loaded runner the teardown drain races the in-flight _finalize, and the straggler 'completed' events leak into the NEXT test's queue, where _drain_one() picks one up instead of the interrupt event. Reproduced locally: gate release + immediate drain + 0.15s finalize delay leaked 2 events. Fixes: - teardown waits (bounded 2s) for active workers to finalize before draining, so events land in the owning test - the interrupt test matches its OWN delegation_id via _drain_for() instead of taking whatever event arrives first
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…arch#64767) * test: deflake async-delegation interrupt test under CI load test_interrupt_all_signals_running_children failed twice on a loaded CI worker: the blocker's ev.wait(timeout=5) expired before interrupt_all() ran, the record finalized on its own, and interrupt_all() found nothing running (n == 0, interrupted count 0 — the exact CI assertion failure, reproduced locally by inserting a 5.6s dispatch->interrupt gap). Raise the internal safety timeouts from 5s to 60s across the file's gated runners — they exist only as runaway guards (every test releases its gate explicitly); the pytest-level timeout is the real backstop. Same flake class as the removed test_crashed_runner_produces_error_ completion (NousResearch#64431). * test: fix cross-test completion-event leak (second flake mechanism) The first CI failure was the 5s guard-timeout race; the rerun exposed a SECOND mechanism with a different signature: 'completed' == 'interrupted'. Prior tests (e.g. test_dispatch_rejected_at_capacity) release their gate and return immediately, but their workers finalize asynchronously — on a loaded runner the teardown drain races the in-flight _finalize, and the straggler 'completed' events leak into the NEXT test's queue, where _drain_one() picks one up instead of the interrupt event. Reproduced locally: gate release + immediate drain + 0.15s finalize delay leaked 2 events. Fixes: - teardown waits (bounded 2s) for active workers to finalize before draining, so events land in the owning test - the interrupt test matches its OWN delegation_id via _drain_for() instead of taking whatever event arrives first
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deflakes
test_interrupt_all_signals_running_children, which failed twice consecutively on a loaded CI worker (PR #64756's slice 5) while passing reliably on any quiet machine.Root cause: the test's blocker used
ev.wait(timeout=5)as an internal safety timeout. On a CPU-starved runner, more than 5s elapsed between dispatch andinterrupt_all()— the blocker's wait expired on its own, the record finalized, andinterrupt_all()found nothing running (n == 0,interrupted["count"] == 0— the exact CI assertion failure). Mechanism reproduced locally by injecting a 5.6s dispatch→interrupt gap before touching anything; the fixed semantics survive the same gap.Changes
tests/tools/test_async_delegation.py: raise the internal safety timeouts on gated runners from 5s → 60s (9 sites). Every test releases its gate explicitly, so these timeouts are only runaway guards — the pytest-level timeout is the real backstop. The 2-threadbarrier.wait(timeout=5)is untouched (both parties arrive immediately; no load dependency).Same flake class as
test_crashed_runner_produces_error_completion, removed in #64431. This one is fixable rather than removable — the race was in the test's guard timeout, not its assertion design.Validation
n=0 count=0— matches CIn=1 count=1Infographic