fix(gateway): keep timeout cleanup in the routed profile - #113676
Open
poijygfdyy wants to merge 2 commits into
Open
poijygfdyy wants to merge 2 commits into
poijygfdyy wants to merge 2 commits into
Conversation
poijygfdyy
marked this pull request as ready for review
September 17, 2026 02:24
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Invariant
Every timeout/abandon cleanup for a routed gateway turn must execute under the same
contextvars.Context/ profile scope as the turn it owns. Neither the event-loop-independent inactivity watchdog nor the asyncio fallback reaper may fall back to the process-launchHERMES_HOMEwhile reaping that turn's processes.Current-main RED
Audited against upstream
mainat6005aa1fd9aac8b1024ace50fec8cd1c85a04bae. A final refresh immediately before publication still shows the same main SHA.gateway/run_turn.pycrosses two barethreading.Threadboundaries from a routed turn:_run_agent_start_turn_worker()starts_watch_gateway_turn_inactivityon a fresh daemon thread._run_agent_await_turn_worker()starts_abandon_timed_out_gateway_turnon a fresh daemon thread when the loop-side inactivity poll reaches the timeout first.Fresh Python threads start with an empty
Context, so both paths can enter timeout cleanup without the caller's profile-scopedHERMES_HOME. The cleanup reaches profile-sensitive process-registry/checkpoint/result/config paths.The first commit adds focused regressions for both production spawn boundaries. Each binds process
HERMES_HOMEto a launch home, enters a different served-profile override, drives the real watchdog/reaper spawn site, and requires the child to observe the served home. The bare-thread mechanism observes the launch home instead.A standalone real-thread mechanism probe reproduces the same boundary deterministically: caller=
served, bare child=launch; running the same body throughcopy_context().runyields child=served.Fix
Capture the caller's full Context at each existing timeout-cleanup spawn boundary and execute the unchanged body inside it:
and likewise for
_abandon_timed_out_gateway_turn.Timeout thresholds, inactivity detection, run-generation ownership gates, process baselines, cleanup locking, hard-interrupt behavior, and reaping semantics are unchanged. This is the same profile-sensitive thread-hop pattern already used elsewhere in the repository.
Collision audit
Refreshed immediately before marking this ready across open/recent/historical PRs, issues, commits, exact function names, root-cause terms, and
gateway/run_turn.py.gateway/run_turn.pywatchdog/timeout-reaper thread hops as residual profile-scope leaks after its multiplex audit; it did not modify them./stop//newand API-disconnect reaper spawn sites (gateway/run_agent_cache.py,gateway/platforms/api_server.py) by preserving the caller Context, but not these timeout paths._watch_gateway_turn_inactivity,_abandon_timed_out_gateway_turn copy_context, andgateway/run_turn.py+ timeout/profile Context found no competing implementation._watch_gateway_turn_inactivity copy_contextreturned no commit; recent_abandon_timed_out_gateway_turncommits concern hard-stop/reaping semantics rather than Context propagation.Final diff
Two files, two commits, preserving
RED regression → production fix:gateway/run_turn.py: only the two existing thread targets are wrapped incopy_context().run.tests/gateway/test_timeout_reaper_profile_scope.py: deterministic regressions for both timeout-cleanup spawn boundaries.The final diff was audited after implementation; no unrelated runtime behavior is included.
Validation
6005aa1fd9aac8b1024ace50fec8cd1c85a04bae)This contribution environment does not have a repository checkout and outbound GitHub clone/DNS access is unavailable, so repository pytest/Ruff results are not claimed. Exact head
872cb33ce27b105b0ab0f4b828d4ed6b7dd13cdchas upstreamCI,Docker Build, Test, and Publish, andNix flake checkruns, but all currently concludeaction_required; no jobs executed and no CI pass or failure is claimed.