fix(gateway): standalone gateway turns keep resolving API keys after a hosted-room activation (#112878, salvage #112884) - #113163
Merged
Conversation
… after hosted activation A native hosted room running a second profile calls tui_gateway.launch_profile_policy.activate_multi_profile_hosting() inside the messaging gateway process, so get_secret() fails closed for every unscoped read afterwards. A gateway with gateway.multiplex_profiles: false never bound a scope (the config flag was the only gate), so its next ordinary turn died in _resolve_session_agent_runtime with UnscopedSecretError / "Hermes could not read this profile's API key" until restart (#112878). Builds on the predicate from #112884: instead of re-entering the routed-profile scope (which rebuilds credentials from .env alone and would drop a key injected by systemd / `op run`), the standalone branch binds the launch profile's OWN scope — launch_secret_scope() (.env + external sources over the env frozen at activation) plus its terminal policy — exactly what the tui_gateway already binds for launch-profile RPC bodies. One predicate, GatewayTurnMixin ._standalone_launch_scope(), no-op while the process is single-profile. Whole class: every standalone entry point now runs under it — the foreground / background turn wrappers, busy, goals and heartbeat-restore paths already routed through _profile_scope_for_source, and the primary adapter's message, busy-session and platform-event handlers (slash commands such as /model, /status and /compress run inside _handle_message and were equally stranded). Cron already binds its own per-fire scope. The guard is not weakened: reads outside any scope still fail closed and a secondary never sees the launch env. Tests: tests/gateway/test_standalone_gateway_launch_scope.py — real activation helper, real server-bound secondary scope entered and left, then the standalone turn and handler resolve both the .env key and the env-injected key (red on origin/main with UnscopedSecretError); control: no activation → nullcontext and no scope in the handler. The #112884 test moves here in trimmed form. Co-authored-by: Lyti4 <205342405+Lyti4@users.noreply.github.com> Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
…activation flips the guard Records the #112878 rule in gateway/AGENTS.md so a new standalone path gates on _standalone_launch_scope rather than the config flag.
૮ >ﻌ< ა ci reviewran on fe87767 — docs(gateway): standalone gateways bind the launch scope onc all good! |
15 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A standalone messaging gateway (
gateway.multiplex_profiles: false) keeps resolving its API keys after a native hosted room has run another profile in the same process, instead of failing every later turn with "Hermes could not read this profile's API key" until restart.Fixes #112878
Salvages #112884 (@KoNit-K) — predicate cherry-picked; diagnosis and minimal diff by the reporter @Lyti4 (co-authored).
Changes
gateway/run_turn.py::_profile_scope_for_source— standalone branch now returns_standalone_launch_scope(): a no-op until the process hosts another profile home (agent.secret_scope.is_multiplex_active()), then the launch profile's own runtime scope. Config-multiplex behaviour unchanged.gateway/run_turn.py::_standalone_launch_scope— the ONE predicate for every standalone path.gateway/run_adapters.py::_standalone_scoped— the primary adapter's message, busy-session and platform-event handlers (_primary_*_handler) run under the same scope, decided per event (activation happens after the adapters were wired). This covers slash commands (/model,/status,/compress) that run inside_handle_messageand were equally stranded.tui_gateway/launch_profile_policy.py::launch_profile_runtime_scope— bindslaunch_secret_scope(home)(.env+ external sources over the env frozen at activation) plus the launch terminal policy overlaunch_terminal_env(); the same mapping the TUI gateway already binds for launch-profile RPC bodies. Not a.env-only rebuild: a key injected by systemd /op runhas no file to rebuild it from and would otherwise vanish.gateway/AGENTS.md— rule recorded.tests/gateway/test_standalone_gateway_launch_scope.py(2 invariants).Not changed: the guard itself (reads outside any scope still fail closed; a secondary never sees the launch env), cron (
cron/scheduler.py::run_jobalready binds its own per-fire scope),_media_delivery_scope_for_source(path validation only, no credentials).Root cause
_profile_scope_for_sourcegated the scope binding on the config flag only, whiletui_gateway.launch_profile_policy.activate_multi_profile_hosting()(called by the in-process hosted-room worker) flips the process-wide fail-closed guard independently — so after a room turn the standalone gateway's own turns read credentials unscoped andget_secretraisedUnscopedSecretError.Validation
Live in-process probe (temp
HERMES_HOME, realactivate_multi_profile_hosting(), realtui_gateway.server._session_profile_runtime_scopefor a temp secondary entered and left, then a standalone runner withmultiplex_profiles=False):9796235nullcontext,.envkey + env-injected key resolve_profile_scope_for_source)UnscopedSecretErrorfor every key.envkey and env-injected key resolve; terminal policy bound_primary_message_handler/ busy / platform-event //bgbackground after activationUnscopedSecretError_resolve_profile_home_for_source(default)with multiplexing offRed-on-base A/B:
test_standalone_turn_binds_launch_profile_scope_after_hosted_activationfails on origin/main withUnscopedSecretError(control passes); green here.scripts/run_tests.sh— 9 files (new file, multiplex background-task / primary-token / interactive-auth / session-db scope, tui_gateway multi-profile hosting fail-closed + transitions + terminal-scope entrypoints, hosted-room gateway lifecycle) → 60 passed, 0 failed.Dropped hunks
tests/gateway/test_multiplex_background_task_scope.py, mocked_resolve_profile_home_for_source) is replaced by the topical file above; it re-drives the same case through the real launch home plus the env-injected key and the handler entry point.Infographic
(image generation returned HTTP 429 twice in this lane)
Review follow-up
Independent review (batch113) — 3 MINOR findings, all verified on head
fe87767e; none fixed in this PR (reasons below), all listed as residuals.gateway/run_turn.py::_standalone_launch_scope(mid-turn activation race) — residual, verified. Probe on head: inside_standalone_launch_scope()pre-activation reads{'OPENAI_API_KEY': 'injected-key', 'INJECTED_ONLY': 'injected'}; afteractivate_multi_profile_hosting()in the same body everyget_secret→UnscopedSecretError; the next turn is fine. The suggested swap (bindlaunch_profile_runtime_scopeunconditionally) is not a clearly-safe one-liner:launch_terminal_env()callscapture_launch_env(), so binding it while single-profile freezes the launch env at the first standalone turn instead of at activation — probe: a value set inos.environafter that first turn (BRIDGED_LATER) isNonein the launch scope post-activation, contradicting the module invariant "frozen at activation". A faithful mirror of the TUI'sweb_server_profiles._profile_scope(secret scope only, pre-activation) needs a split secret/terminal context manager plus a per-turnbuild_profile_secret_scope(.env + external secret sources read on every standalone turn) — left for a follow-up.gateway/run.py::_bridge_max_turns_from_config/_current_max_iterations— residual (see below).cron/scheduler.py::run_job/run_agent_cache.py::_run_release_in_profile_scope/api_server.py::_profile_scope— residual, pre-existing and outside this PR's diff; the AGENTS.md sentence should be read as covering turns/handlers.Known residuals
gateway/run_turn.py::_standalone_launch_scope— The standalone scope is decided at body entry (nullcontext whileis_multiplex_active()is False), so a standalone turn already in flight when a hosted room activates the guard still dies withUnscopedSecretErrormid-turn — the TUI gateway avoided exactly this by bindinglaunch_secret_scopefor EVERY launch-profile body. Suggested fix: bind the launch secret scope unconditionally in the standalone branch (secret scope only pre-activation to keepcapture_launch_envat activation time).gateway/run.py::_bridge_max_turns_from_config/_current_max_iterations— Once the standalone gateway binds a scope after activation,gateway/platforms/_shared.profile_scoped()turns True, so the per-turn config.yaml → env re-bridge ofagent.max_turns/sessions.*is skipped and_current_max_iterationsreads the staleHERMES_MAX_ITERATIONS— a config edit after a hosted-room activation is ignored until restart (probe: max_turns 7 → activate, edit to 9 → still 7). Suggested fix: treat a launch-profile scope like a routed one in_current_max_iterations(readagent.max_turnsfromget_process_hermes_home()/config.yamlwhenprofile_scoped()and no override), or let the launch scope keep the bridge.cron/scheduler.py::run_job/gateway/run_agent_cache.py::_run_release_in_profile_scope/gateway/platforms/api_server.py::_profile_scope— AGENTS.md now says every standalone path after activation binds '.env over the frozen launch env', but the standalone gateway's cron fires, agent eviction/release, and the API server's default-profile scope still bindbuild_profile_secret_scope/_profile_runtime_scope(launch home)— .env only — so a systemd/op-injected key isNonethere after activation (probe:INJECTED_ONLY→ None vs'injected'via_profile_scope_for_source). Pre-existing, outside this diff; either narrow the AGENTS.md sentence to turns/handlers or route those sites throughlaunch_profile_runtime_scopein a follow-up.