fix(a2a): orphan sweep honors A2A_REPLY_TIMEOUT - #106982
Closed
gaoanze888 wants to merge 6 commits into
Closed
gaoanze888 wants to merge 6 commits into
gaoanze888 wants to merge 6 commits into
Conversation
…ork revision (token lacks workflow scope)
The watchdog orphan sweep used a hardcoded _ORPHAN_TIMEOUT = 300s while the caller's reply deadline already honored A2A_REPLY_TIMEOUT via _reply_timeout(). A turn taking 350-600s was therefore marked TASK_STATE_FAILED by the sweep before the configured reply deadline arrived, and once terminal the TaskStore.complete guard silently discarded the real (eventually-produced) reply — raising A2A_REPLY_TIMEOUT had no effect, contradicting the docs. Extract the sweep into _sweep_orphans() and resolve the window via _reply_timeout() so the sweep window matches the configured reply deadline: a long-running turn survives long enough for its real reply to complete the task. _ORPHAN_TIMEOUT is removed; _WATCHDOG_INTERVAL stays. Fixes NousResearch#106972
Author
|
Closing this as a duplicate — thanks to @alt-glitch's triage I see the orphan-sweep / reply-timeout mechanism already has three open PRs in flight:
Five PRs for a one-symbol fix is just noise for the maintainers, and #100312 is both first and more correct, so I'd rather not split reviewer attention. The four regression tests I added here (default-window sweep, the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #106972. The A2A orphan-task sweep used a hardcoded
_ORPHAN_TIMEOUT = 300while the caller's reply deadline already honoredA2A_REPLY_TIMEOUTvia_reply_timeout(). A turn taking 350–600s was markedTASK_STATE_FAILEDby the sweep before the configured reply deadline arrived, and once terminal theTaskStore.completeguard silently discarded the real (eventally-produced) reply — so raisingA2A_REPLY_TIMEOUThad no effect, contradicting the docs that present it as the fix for "Replies time out on long tasks."Root cause
plugins/platforms/a2a/adapter.py:_ORPHAN_TIMEOUT = 300(hardcoded) fedself.tasks.fail_orphans(_ORPHAN_TIMEOUT)in_watchdog_loop. Separately,_reply_timeout()readsos.getenv("A2A_REPLY_TIMEOUT", "300")— but only for the active caller's own wait deadline, never for the orphan sweep.plugins/platforms/a2a/protocol.py:fail_orphansmarks any task older thantimeout_secondsasSTATE_FAILED;complete()then hasif rec["state"] in TERMINAL_STATES: return None, so a later legitimate reply is silently dropped.Fix
Extract the sweep into
_sweep_orphans()and resolve the window via_reply_timeout()so the sweep window matches the configured reply deadline: a long-running turn survives long enough for its real reply to complete the task (the reply arrives → task goesSTATE_COMPLETED→ the sweep'snot in TERMINAL_STATESguard skips it)._ORPHAN_TIMEOUTis removed;_WATCHDOG_INTERVALis unchanged.This is option (1) from @leyhux101-bit's report (make
_ORPHAN_TIMEOUTrespectA2A_REPLY_TIMEOUT). With it, the docs become accurate as written; no doc change is needed.Tests
New
TestOrphanSweepRespectsReplyTimeoutintests/plugins/test_a2a_plugin.py(4 tests):STATE_FAILED+ metrics++A2A_REPLY_TIMEOUT=600preserves a 400s-old task (under the new 600s window; would be swept under the old hardcoded 300) — this is the A2A: A2A_REPLY_TIMEOUT does not affect the orphan-task sweep (hardcoded 300s) — late replies get silently discarded #106972 scenario_reply_timeoutexcept branch)Mutation-verified: reverting
_sweep_orphansto the oldorphan_timeout = 300makes the regression test fail (400s-old task swept at 300 even withA2A_REPLY_TIMEOUT=600); restored, all 4 pass. Full a2a suite: 159 passed (4 new + 155 existing), 18 integration deselected.ruff checkclean.Note
Huge credit to @leyhux101-bit for the precise root-cause writeup (the split between
_reply_timeout()and the hardcoded_ORPHAN_TIMEOUTis the whole story). The reporter offered to open this PR — I put it up to unblock the fix, but if you'd rather land your own implementation @leyhux101-bit I'm happy to close this in favor of yours.The diff also carries unrelated
.github/workflowschanges — that's a fork/main ↔ origin/main sync artifact (my fork is behind on two workflow files and I lack a workflow-scope token to fully resync); the actual code change is only the two files listed above.