fix(gateway): drain timeout=0 must still wait for in-flight cron/API work - #82282
Closed
JonthanaHanh wants to merge 1 commit into
Closed
JonthanaHanh wants to merge 1 commit into
JonthanaHanh wants to merge 1 commit into
Conversation
…work When ``restart_drain_timeout`` is 0 (the default), the drain phase returns immediately with ``timed_out=True`` — even when cron jobs or API-server runs are the only active work. Cron jobs live outside ``_running_agents`` and have no pre-stop protection via ``restart_after_turn_timeout``, so the zero-budget drain kills them with no grace period. Apply a minimum drain budget (30s) when the only active work is background (cron/API) and the configured timeout is 0. Chat sessions still get the immediate drain behavior when timeout=0 (they are protected by the pre-stop ``restart_after_turn_timeout`` wait). Fixes NousResearch#82161
1 task
Collaborator
|
Closing with credit: the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
restart_drain_timeoutis 0 (the default), the drain phase returns immediately withtimed_out=True— even when cron jobs or API-server runs are the only active work. Cron jobs live outside_running_agentsand have no pre-stop protection viarestart_after_turn_timeout, so the zero-budget drain kills them with no grace period.Root Cause
In
_drain_active_agents()(gateway/run.py:9381), the checkif timeout <= 0: return snapshot, Truefires even when only background work (cron/API) is active. The early-return at line 9376 (not self._running_agents and cron_count == 0 and api_count == 0) correctly falls through when cron is running, but the zero-timeout check immediately kills it.The design intent of
restart_drain_timeout=0is "don't wait for chat sessions" (they are protected by the pre-stoprestart_after_turn_timeout). But cron jobs have no pre-stop protection — they run on the scheduler's thread pool, entirely outside_running_agents.Fix
Apply a minimum drain budget (30s) when the only active work is background (cron/API) and the configured timeout is 0. Chat sessions still get the immediate drain behavior when timeout=0.
Changes
gateway/run.py: Add_MIN_DRAIN_FOR_BACKGROUND_WORKconstant and use it whentimeout <= 0with active cron/API work.tests/gateway/test_update_cron_drain.py: Add regression testtest_drain_zero_timeout_still_waits_for_cron_jobs.Fixes #82161