Skip to content

fix(gateway): carry the profile scope into pre-turn hygiene compression - #78400

Closed
Drexuxux wants to merge 1 commit into
NousResearch:mainfrom
Drexuxux:fix/hygiene-compression-profile-scope
Closed

Drexuxux wants to merge 1 commit into
NousResearch:mainfrom
Drexuxux:fix/hygiene-compression-profile-scope

Conversation

@Drexuxux

@Drexuxux Drexuxux commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

The multiplexed inbound handler runs the whole message inside
_profile_runtime_scope, which installs the routed profile's HERMES_HOME
override and its secret scope. Both are contextvars — the helper's own
docstring says so:

set_hermes_home_override … Contextvar, so it propagates into the agent
worker thread via copy_context().

The automatic pre-turn hygiene compression handed _compress_context to a bare
executor hop:

_hyg_future = loop.run_in_executor(
    None,
    lambda: _hyg_agent._compress_context(...),
)

loop.run_in_executor(None, fn) starts the worker with an empty context.
So inside that thread:

  • get_hermes_home() fell back to the default profile — the compressor
    resolved config, model and skills from the wrong home;
  • get_secret read process-global os.environ, which under multiplexing may
    hold a different profile's credentials (the first-writer-wins YAML→env
    bridge). The compressor's aux-client provider resolution therefore ran
    unscoped: it either failed closed (automatic compression silently stopped
    working for multiplexed profiles) or resolved against another profile's keys.

This is a known failure mode — the other caller was already fixed

gateway/slash_commands.py routes the manual /compress through
_run_in_executor_with_context and says exactly why at the call site:

_run_in_executor_with_context (not a bare run_in_executor): the profile
secret scope installed by the wrapper is a contextvar, and the
default-executor hop would drop it — the compressor's aux-client provider
resolution would then read credentials unscoped and fail closed under
multiplexing.

The automatic path never got the same treatment — and it runs unattended on
every inbound message whose session crosses the compression threshold, so it
fires far more often than the slash command.

How it was found

Grepped the repo for its own declared invariants (chokepoint,
must go through, single source of truth, not a bare …) and then looked for
call sites that bypass them. The /compress comment names the rule; the hygiene
path was the remaining violation.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • gateway/run.py — the hygiene hop now runs through copy_context().run,
    so the worker inherits the caller's contextvars. The executor stays the
    default one, so pool behaviour and the caller's progress-aware Future
    polling (.done() / .result() / .cancel()) are unchanged.
  • gateway/run.py — extracted as GatewayRunner._spawn_hygiene_compression
    rather than patching the lambda in place. The call site sits ~660 lines into
    _handle_message_with_agent; AGENTS.md asks for exactly this extraction so
    the contract can be exercised for real instead of asserted against the
    god-file's source text.
  • tests/gateway/test_hygiene_compression_profile_scope.py — 5 tests.

Single-profile gateways never enter _profile_runtime_scope, so their
behaviour is byte-identical.

How to Test

  1. Run a multiplexed gateway with two profiles whose .env files hold
    different auxiliary-provider keys.
  2. Drive a session on the non-default profile past the compression threshold so
    pre-turn hygiene compression fires.
  3. Before: the compression worker resolved get_hermes_home() to the
    default profile and read credentials from process-global os.environ.
    After: it resolves both from the routed profile.

Test Results

New file tests/gateway/test_hygiene_compression_profile_scope.py — 5 tests.
They drive the real _profile_runtime_scope and the real spawn helper, with
a recording stand-in agent that captures what the worker thread actually sees;
the contextvar loss is a property of the hop itself, so mocking it away would
test nothing:

Test Asserts
test_worker_sees_the_profile_home get_hermes_home() inside the worker is <root>/profiles/coder
test_worker_sees_the_profile_secret_not_the_process_env get_secret returns the profile .env value, not the conflicting os.environ one
test_arguments_and_result_are_passed_through_unchanged messages / approx_tokens / commit_fence and the return value are untouched
test_returns_a_future_the_progress_wait_can_poll the .done()/.result()/.cancel() interface the caller's inactivity wait depends on is preserved
TestSingleProfileGatewayUnchanged with no scope installed, the launch home still wins

Red-without-fix, with only the copy_context() hop reverted:

2 failed, 3 passed

With the fix:

5 passed in 5.46s

Regression sweep — whole tests/gateway/ directory, both sides on the same
main:

baseline (change stashed):   72 failed, 4771 passed
with this fix:               73 failed, 4775 passed

The failing files are the same pre-existing set (feishu, runtime_footer, update,
discord, systemd, …) — none of them touch this code path. The one-test delta is
test_telegram_start_polling_timeout.py::test_initial_connect_succeeds_on_current_generation_progress,
which I verified fails identically with and without this change: it is a
pre-existing main regression from e05eba26a, where a new
_polling_conflict_recovery_generation attribute was added to __init__ but
the suite's _bare_adapter() helper (which builds the adapter via __new__)
was not updated.

The multiplexed inbound handler runs the whole message inside
_profile_runtime_scope, which installs the routed profile's HERMES_HOME
override and its secret scope as contextvars -- its own docstring notes they
reach the agent worker "via copy_context()".

The automatic pre-turn hygiene compression handed _compress_context to a bare
loop.run_in_executor(None, fn). A bare executor hop starts the worker with an
EMPTY context, so inside it get_hermes_home() fell back to the default profile
and get_secret read process-global os.environ -- which under multiplexing may
hold a different profile's credentials. The compressor's aux-client provider
resolution therefore ran unscoped: it either failed closed (automatic
compression silently stopped working for multiplexed profiles) or resolved
against the wrong profile's keys and config.

gateway/slash_commands.py already routes the manual /compress through
_run_in_executor_with_context for exactly this reason, and says so in a comment
at the call site. The automatic path -- which runs unattended on every inbound
message whose session crosses the compression threshold, and so fires far more
often than the slash command -- never got the same treatment.

Spawn the hop through copy_context().run. The executor stays the default one,
so pool behaviour and the caller's progress-aware Future polling are unchanged.

Extracted as _spawn_hygiene_compression rather than fixing the lambda in place:
the call site sits ~660 lines into _handle_message_with_agent, and AGENTS.md
asks for exactly this extraction so the contract can be exercised directly
instead of asserted against the god-file's source text.
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery area/compression Context compression and continuation sessions area/profiles Multi-profile isolation, HERMES_HOME scoping P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Aug 4, 2026
@teknium1

teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Resolved on main by #100950 (merge c5c9aa8d44), which lands exactly this change — copy_context().run around the hygiene run_in_executor hop — plus the sibling codex app-server hygiene hop and an UnscopedSecretError → abort-not-truncate classification.

You identified and fixed this a month before it showed up in a user's debug bundle, @Drexuxux, and this PR was missed in the pre-merge duplicate sweep (my error: it surfaced on the post-merge sweep). You are the first submitter of this fix; noting that here for the record. Closing as implemented on main.

@teknium1 teknium1 closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions area/profiles Multi-profile isolation, HERMES_HOME scoping comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants