Skip to content

fix(gateway): keep timeout cleanup in the routed profile - #113676

Open
poijygfdyy wants to merge 2 commits into
NousResearch:mainfrom
poijygfdyy:fix/gateway-timeout-reaper-profile-context
Open

poijygfdyy wants to merge 2 commits into
NousResearch:mainfrom
poijygfdyy:fix/gateway-timeout-reaper-profile-context

Conversation

@poijygfdyy

@poijygfdyy poijygfdyy commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Invariant

Every timeout/abandon cleanup for a routed gateway turn must execute under the same contextvars.Context / profile scope as the turn it owns. Neither the event-loop-independent inactivity watchdog nor the asyncio fallback reaper may fall back to the process-launch HERMES_HOME while reaping that turn's processes.

Current-main RED

Audited against upstream main at 6005aa1fd9aac8b1024ace50fec8cd1c85a04bae. A final refresh immediately before publication still shows the same main SHA.

gateway/run_turn.py crosses two bare threading.Thread boundaries from a routed turn:

  1. _run_agent_start_turn_worker() starts _watch_gateway_turn_inactivity on a fresh daemon thread.
  2. _run_agent_await_turn_worker() starts _abandon_timed_out_gateway_turn on a fresh daemon thread when the loop-side inactivity poll reaches the timeout first.

Fresh Python threads start with an empty Context, so both paths can enter timeout cleanup without the caller's profile-scoped HERMES_HOME. The cleanup reaches profile-sensitive process-registry/checkpoint/result/config paths.

The first commit adds focused regressions for both production spawn boundaries. Each binds process HERMES_HOME to a launch home, enters a different served-profile override, drives the real watchdog/reaper spawn site, and requires the child to observe the served home. The bare-thread mechanism observes the launch home instead.

A standalone real-thread mechanism probe reproduces the same boundary deterministically: caller=served, bare child=launch; running the same body through copy_context().run yields child=served.

Fix

Capture the caller's full Context at each existing timeout-cleanup spawn boundary and execute the unchanged body inside it:

threading.Thread(
    target=copy_context().run,
    args=(_watch_gateway_turn_inactivity,),
    ...,
).start()

and likewise for _abandon_timed_out_gateway_turn.

Timeout thresholds, inactivity detection, run-generation ownership gates, process baselines, cleanup locking, hard-interrupt behavior, and reaping semantics are unchanged. This is the same profile-sensitive thread-hop pattern already used elsewhere in the repository.

Collision audit

Refreshed immediately before marking this ready across open/recent/historical PRs, issues, commits, exact function names, root-cause terms, and gateway/run_turn.py.

Final diff

Two files, two commits, preserving RED regression → production fix:

  • gateway/run_turn.py: only the two existing thread targets are wrapped in copy_context().run.
  • tests/gateway/test_timeout_reaper_profile_scope.py: deterministic regressions for both timeout-cleanup spawn boundaries.

The final diff was audited after implementation; no unrelated runtime behavior is included.

Validation

  • Current-main source audit (6005aa1fd9aac8b1024ace50fec8cd1c85a04bae)
  • Deterministic real-thread RED mechanism: bare child loses the caller Context
  • Deterministic copied-Context GREEN mechanism: child observes the served Context
  • Test-only regression committed before production changes
  • Pre-implementation collision search
  • Fresh pre-publication PR/issue/commit/function/file collision search
  • Final two-file diff audited
  • Exact-head upstream workflows created
  • Repository pytest / Ruff / full suite executed in this environment
  • Hosted CI executed

This contribution environment does not have a repository checkout and outbound GitHub clone/DNS access is unavailable, so repository pytest/Ruff results are not claimed. Exact head 872cb33ce27b105b0ab0f4b828d4ed6b7dd13cdc has upstream CI, Docker Build, Test, and Publish, and Nix flake check runs, but all currently conclude action_required; no jobs executed and no CI pass or failure is claimed.

@alt-glitch alt-glitch added type/test Test coverage or test infrastructure P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Sep 17, 2026
@poijygfdyy
poijygfdyy marked this pull request as ready for review September 17, 2026 02:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/profiles Multi-profile isolation, HERMES_HOME scoping comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants