fix(gateway): multiplexed hygiene compaction no longer loses the profile secret scope (salvage #100849) - #100950
Merged
Merged
Conversation
…submit Some bundled CPython runtime builds strip stdlib ThreadPoolExecutor's copy_context() propagation, so work submitted to the daemon pool runs in a bare context. Under the multiplexed gateway this dropped the profile secret scope in pool workers: the context-compression timeout fence resolved auxiliary provider keys (SURPLUS_API_KEY) with UnscopedSecretError, silently degrading LLM compression to lossy deterministic summaries and driving re-read loops in affected sessions. Restore stdlib semantics in submit() by snapshotting the caller's context and running the callable inside it (a no-op re-application on runtimes that already propagate). Mirrors the gateway's _run_in_executor_with_context pattern. Tests: daemon pool worker sees caller contextvars; scoped get_secret works in a daemon-pool worker under multiplex while scoped misses still fail closed (no env leak).
… multiplexing Session-hygiene compaction ran _compress_context on a bare loop.run_in_executor(None, ...) worker. Under gateway.multiplex_profiles the profile secret scope and HERMES_HOME override are ContextVars installed by the per-turn _profile_runtime_scope, and a bare worker starts with an empty Context — so the summary model's get_secret(<PROVIDER>_API_KEY) failed closed with UnscopedSecretError on EVERY hygiene pass and compaction silently degraded to a lossy truncation (#100849 debug bundle: 'Failed to generate context summary: get_secret(SURPLUS_API_KEY) called with no profile secret scope active'). - gateway/run.py: run both hygiene executor hops (detached-agent path and codex app-server path) inside copy_context().run, keeping the default executor so a fence-cancelled hung summary never occupies a gateway agent-work slot. - agent/context_compressor.py: UnscopedSecretError is a missing-credential class failure — abort and preserve the session instead of dropping the middle window for a placeholder summary (same carve-out as 401/402/403). - tools/daemon_pool.py: correct the salvaged docstrings — stdlib ThreadPoolExecutor only propagates contextvars from 3.14; nothing is stripped from the bundled runtime. - tests: hygiene worker inherits caller ContextVars (fails on bare run_in_executor); UnscopedSecretError classified as access failure. Live A/B (real get_secret in a run_in_executor worker, multiplex on, profile .env scope installed): main -> UnscopedSecretError; fixed -> scoped value.
Contributor
૮ >ﻌ< ა ci reviewran on 6ab1250 — chore: retrigger CI (zero-job dispatch failure, auto-heal)
|
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multiplexed-gateway session-hygiene compaction can reach the summary model again: the hygiene worker now inherits the caller's profile secret scope instead of failing closed with
UnscopedSecretErrorand silently truncating the middle of the conversation on every pass (salvages #100849 by @MattMaximo).Root cause:
gateway/run.pyhygiene ran_compress_contexton a bareloop.run_in_executor(None, ...)worker. Undergateway.multiplex_profilesthe profile secret scope andHERMES_HOMEoverride are ContextVars installed per turn by_profile_runtime_scope; a bare worker starts with an empty Context, soget_secret("<PROVIDER>_API_KEY")in the summary path raisedUnscopedSecretError, the summary was "unavailable", and the compressor fell through to the lossy placeholder-and-drop path. In the reporter's debug bundle every hygiene pass took that route (697→642,645→426messages, no LLM summary).Changes
gateway/run.py: both hygiene executor hops (detached-agent path and codex app-server path) run insidecopy_context().run. Default executor kept on purpose — a fence-cancelled hung summary must never occupy a gateway agent-work slot.agent/context_compressor.py:UnscopedSecretErrorclassified as a missing-credential failure → compress() aborts and preserves the session unchanged (same carve-out as 401/402/403), instead of dropping the middle window for a placeholder.tools/daemon_pool.py(salvaged from fix(tools): propagate caller contextvars in DaemonThreadPoolExecutor.submit #100849, @MattMaximo):DaemonThreadPoolExecutor.submitpropagates the caller's contextvars by default, so every pool consumer is safe even withoutpropagate_context_to_thread. Docstring corrected: stdlib only does this from Python 3.14; nothing is stripped from the bundled runtime.run_in_executor),UnscopedSecretError→ access-failure class, plus the salvaged daemon-pool / secret-scope tests.Validation
get_secret("SURPLUS_API_KEY")in arun_in_executorworker, multiplex on, profile.envscope installed (/tmp/hyg_ab.py)UnscopedSecretError_is_summary_access_or_quota_error(UnscopedSecretError(...))False→ truncationTrue→ abort, session preservedtest_hygiene_worker_inherits_caller_contextvarswithcopy_context().runremovedtests/gateway/test_codex_hygiene_compaction.py,tests/tools/test_daemon_pool.py,tests/agent/test_secret_scope.py,-k access_failureNote on #100849's stated mechanism: the daemon pool consumer (
conversation_compression._run_compress_with_timeout) already wrapped its worker withpropagate_context_to_thread; the scope was lost one layer up in gateway hygiene, which never touches the daemon pool. The pool change is kept as defense in depth for othersubmit()callers.Related: #76574 (secret-scope residuals), #100697 / #100709 (unscoped-probe log noise).
Infographic