Skip to content

fix(delegation): preserve completed child results on parent interrupt - #48893

Open
andyvandaric wants to merge 1 commit into
NousResearch:mainfrom
andyvandaric:fix/delegate-batch-interrupt-race
Open

fix(delegation): preserve completed child results on parent interrupt#48893
andyvandaric wants to merge 1 commit into
NousResearch:mainfrom
andyvandaric:fix/delegate-batch-interrupt-race

Conversation

@andyvandaric

@andyvandaric andyvandaric commented Jun 19, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes a race condition in the batch delegation interrupt handler where a child that completes between the last _cf_wait() return and the _interrupt_requested check has its result discarded — replaced with a fabricated "interrupted" entry even though the work was already done.

In the batch polling loop (delegate_tool.py, ~line 2370), when _interrupt_requested fires, the old code iterates over pending futures calling f.done() point-in-time. This is inherently racy:

  • A future can transition from not-done → done between the check and the else branch
  • Futures completing between the last _cf_wait(timeout=0.5) return and the top-of-loop interrupt check are never collected through the normal done path

The fix replaces the racy iteration with a 2-second grace window via wait(pending, timeout=2.0, return_when=ALL_COMPLETED), cleanly separating futures into completed (collect real results) vs still-running (fabricate as interrupted).

Related Issue

This PR is complementary — it fixes the specific race where already-computed results are lost on interrupt, without requiring the larger async delegation redesign from #8482.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Updated tools/delegate_tool.py batch interrupt handler: replaced racy f.done() iteration with concurrent.futures.wait(pending, timeout=2.0, return_when=ALL_COMPLETED) grace window that cleanly separates done vs still-pending futures.
  • Added tests/tools/test_delegate_batch_interrupt_race.py with 3 test cases covering the race condition scenarios.

How to Test

  1. python -m pytest tests/tools/test_delegate_batch_interrupt_race.py -v
    • Result: 3 passed.
  2. python -m pytest tests/tools/test_delegate.py -v
    • Result: 131 passed (9 pre-existing failures due to missing concurrent_log_handler in dev env, unrelated to this change).
  3. python -m pytest tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_delegate_toolset_scope.py tests/tools/test_delegate_composite_toolsets.py -v
    • Result: 17 passed.

Full-suite note: pytest tests/ -q does not fully complete on this local environment due to 9 pre-existing failures from missing concurrent_log_handler module (required by tests that instantiate real AIAgent without mocking). All tests touching the delegation interrupt path pass. The focused touched-path suites above cover the change surface.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — all touched-path tests pass; 9 pre-existing failures from missing concurrent_log_handler are unrelated (see Full-suite note above)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 10

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

N/A. This is a concurrency correctness fix with unit test verification.

The batch delegation interrupt handler had a race condition where a child
that completed between the last _cf_wait() return and the interrupt check
(or during the fabrication loop iteration) could have its result discarded
and replaced with a fabricated 'interrupted' entry.

The old code iterated over pending futures calling f.done() point-in-time,
but this check is inherently racy: a future can transition from not-done to
done between the check and the else branch that fabricates an interrupted
entry.

Fix: replace the racy iteration with a 2-second grace window via
wait(pending, timeout=2.0, return_when=ALL_COMPLETED). This cleanly
separates futures into two sets — those that completed (including during
the grace window) get their real results collected, and only genuinely
still-running futures are marked as interrupted.

This is a minimal, targeted fix. The broader 'non-blocking delegation'
feature (issue NousResearch#47302, PR NousResearch#8482) addresses the larger UX problem of
parent turns freezing during delegation; this fix ensures that when an
interrupt does fire, already-computed work is not silently lost.

Tested with 3 new test cases covering:
- Fast child completes before interrupt, slow child interrupted
- All children complete within grace window after interrupt
- Child completing during the grace window is collected

Refs: NousResearch#45496, NousResearch#47302
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/delegate Subagent delegation P2 Medium — degraded but workaround exists labels Jun 19, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating a real race in the synchronous batch aggregation path; current main still has the point-in-time f.done() branch at tools/delegate_tool.py:2596-2623.

Problems

  • The added drain at tools/delegate_tool.py:2378-2380 waits up to two seconds for work that was not complete when interruption was observed. The new test explicitly treats a child completing at 1.5s after an interrupt at 0.3s as completed (tests/tools/test_delegate_batch_interrupt_race.py:247-249, 270-271). That is a cancellation-policy change, not just preservation of already-computed output.
  • The first two timing tests do not reproduce the original TOCTOU: current main's FIRST_COMPLETED wait collects children that finish at 0.1s before their 0.3s/0.5s interrupt points (tools/delegate_tool.py:2629-2649; tests/tools/test_delegate_batch_interrupt_race.py:107-109, 137-140).

Suggested changes

  • Decide and document whether interruption may wait for post-interrupt completions; otherwise keep the interrupt path bounded while collecting only results known complete at its cutoff.
  • Add a deterministic race regression test that controls the transition between the old completion observation and fabrication.

This is an automated hermes-sweeper review.

Comment thread tools/delegate_tool.py
from concurrent.futures import wait as _cf_wait_drain, ALL_COMPLETED

_drain_done, _drain_pending = _cf_wait_drain(
pending, timeout=2.0, return_when=ALL_COMPLETED

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drain waits for work that was still running when the parent interrupt was observed. That is broader than preserving already-computed results and changes the existing prompt-interrupt behavior into a two-second grace policy. Please either make that policy explicit and intentional, or collect only work known complete at the interrupt cutoff.


# Child 0: completes at 0.1s
child_0 = _make_mock_child(response="Child 0 done", delay=0.1)
# Child 1: completes at 1.5s — within the 2s grace window

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This child completes 1.2 seconds after the test sets the interrupt, so the assertion validates the new grace-window policy rather than the original f.done() TOCTOU. Please add a deterministic test that controls the completion transition between the old observation and fabricated-result path.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/delegate Subagent delegation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants