fix(gateway): keep abandoned-turn reapers in the session profile - #112845
Closed
poijygfdyy wants to merge 3 commits into
Closed
poijygfdyy wants to merge 3 commits into
poijygfdyy wants to merge 3 commits into
Conversation
Contributor
|
Independently reproduced the boundary on this branch (Linux, CPython 3.11, pytest):
Both spawn sites now snapshot on the caller's thread, which is the correct place for Two non-blocking notes:
|
Contributor
Author
|
Thanks for the independent check. Two scope decisions here:
The functional scope remains only profile Context propagation at the two abandoned-turn reaper spawn sites. |
poijygfdyy
force-pushed
the
fix/gateway-reaper-profile-scope
branch
from
September 16, 2026 17:17
b909f45 to
b39ea69
Compare
Collaborator
|
Landed on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Abandoned-turn cleanup can run on bare
threading.Threadreapers after a/stop-style interrupt or API-server turn abandonment. In a multiplexed gateway, those threads start without the caller'scontextvars.Context, so process-registry cleanup can resolveget_hermes_home()to the launch profile instead of the served session profile.That breaks the profile-isolation invariant for process checkpoint/result/config reads and writes during cleanup.
Fixes #112968.
Invariant
Every abandoned-turn reaper must execute under the same profile scope as the turn it is cleaning up. A reaper must never fall back to the process launch
HERMES_HOMEmerely because finalization crossed a thread boundary.Deterministic RED
tests/gateway/test_reaper_profile_scope.pydrives the real_interrupt_running_turnpath with launch home A and a context-local served home B, then observesprocess_registry.kill_started_since()from the spawned reaper thread.On the unpatched path the thread is bare, so it starts with an empty
Contextand observes A. The regression requires B.Fix
Capture the caller context at the thread-spawn boundary and execute
_reap_gateway_turn_processes(...)throughcopy_context().run(...).This PR applies that mechanism to the two abandoned-turn reaper sites:
gateway/run_agent_cache.py—/stop//new/ eviction interrupt reaper;gateway/platforms/api_server.py— API turn-abandonment reaper.Generation/epoch guards, process ownership, reaper timing, and
_reap_gateway_turn_processesitself are unchanged.The earlier unrelated
aiohttp server→aiohttp web serverwording was removed and folded out of the final history. Current headb39ea694d9ad3f06a4c452de2f8fcd75bc4174f1is three commits (RED test+ the two production spawn-site fixes) and the final diff is only the two Context propagation sites plus the focused regression.Collision / consolidation alignment
Searched current code plus open/recent PRs, issues, and commits for
_reap_gateway_turn_processes, process-registry profile scope,copy_context, and the affected files/root cause.gateway/run_agent_cache.py=1197ab4a330a23e3358b414bf60db5d19ea4140b;gateway/platforms/api_server.py=02a1bd0081b7ef87f53aaf6029b3d660aaa340ba. There is no remaining implementation-quality gap requiring this fix to land only through the consolidation PR.Validation
This PR is ready for review.