fix(gateway): routed multiplex profiles get their own terminal cwd/backend/docker config; container boot honors config multiplex_profiles (#68559 #85413, salvage #99225 #85437) - #101242
Conversation
A multiplexed Hermes process (gateway.multiplex_profiles, unified dashboard/TUI, or cron) serves several profiles at once, but terminal.* resolved through process-global TERMINAL_* env vars bridged ONCE at startup from the launch profile (gateway/run.py ~2700-2760) plus the one-shot _ensure_terminal_env_bridged() guard. Every routed profile therefore inherited the launch profile's backend, cwd, docker volumes, SSH target and shared-container key: a local profile ran inside another profile's docker sandbox (or a docker profile escaped to the host), and a container labeled profile A carried profile B's RW bind mounts. Fix: an authoritative per-profile terminal policy seam, mirroring agent/secret_scope.py: - tools/terminal_scope.py: ContextVar holding the routed profile's COMPLETE effective TERMINAL_* policy (defined defaults <- profile .env TERMINAL_* <- config.yaml terminal:). While bound, terminal_env() resolves ONLY from it - an omitted key yields the defined default, never os.environ. Unreadable/malformed policy installs a refusal scope; terminal_tool / execute_code refuse instead of running under ambient launch-process policy (fail closed). - Installed at every in-process profile boundary: gateway _profile_runtime_scope, tui_gateway session/build/turn scopes, cron per-job fire. The unscoped single-process path is byte-identical. - Every terminal.* consumer reads through the scope: terminal_tool (_get_env_config, _resolve_container_task_id shared key, orphan reaper lifetime, degraded mode), gateway/platforms/base.py docker media translation (volumes, shared key, persistence), runtime_cwd / agent_init / skill_utils / code_execution_tool / file_tools cwd anchors, prompt_builder / browser_tool / env_probe backend checks, gateway footer, @-refs and slash-command cwd. env_probe resolves the backend in the caller's context, since the probe worker thread does not inherit the ContextVar. Salvage of #99225 onto current main: adds the three ambient reads the PR missed (tools/file_tools.py TERMINAL_CWD, tools/browser_tool.py and tools/env_probe.py TERMINAL_ENV; shape from #79117) and trims the test module to the leak matrix driven through the real gateway boundary, omitted-key defaults, refusal, and boundary reset. Fixes #68559 Fixes #94200 Fixes #101132 Fixes #95470 Co-authored-by: x7peeps <9640837+x7peeps@users.noreply.github.com> Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com> Co-authored-by: ExitMaster <292490062+ExitMaster@users.noreply.github.com>
hermes_cli/container_boot.py resolved multiplexing from the GATEWAY_MULTIPLEX_PROFILES env var only, while the gateway runtime resolves env -> config.yaml -> default. A deployment enabling multiplex_profiles via config.yaml alone therefore auto-started every named profile's gateway slot at boot, which then crash-looped in the double-bind guard against the multiplexing default gateway. Resolve through load_gateway_config().multiplex_profiles (the shared resolver, so env override precedence is preserved) and fall back to the env var only when config loading fails. Salvage of #85437 (test module trimmed to config-only + env-override). Fixes #85413
Documents the per-profile terminal.* resolution and fail-closed refusal behavior introduced by the terminal scope seam (#68559 class).
૮ >ﻌ< ა ci reviewran on 3165a91 — chore: map contributor email for muhifni (#99225 salvage)
|
andrexibiza
left a comment
There was a problem hiding this comment.
Reviewed exact head 3165a9198f14fb80f94b08ae178ce9ec7f7e7b35 against base 3a980a431b28633a5b79c462f654dc100dc1c598, including the 21-file diff, the new tools/terminal_scope.py, tests/tools/test_terminal_scope_multiplex.py, the container-boot slice/tests, the prompt/system-prompt call path, current exact-head CI, commit authorship, and the adjacent multiplex PR graph.
There is a lot that is right here. Moving terminal authority out of process-global TERMINAL_* and into a per-profile ContextVar is the correct class-level shape; the scope projection is explicit, omission does not fall back to ambient os.environ, the mutation-capable terminal/code-exec sinks have a hard refusal gate, and the salvage preserves the original authorship from muhifni (#99225) and 1052326311 (#85437) instead of laundering the work into a rewrite. Current-head CI, Docker, and Nix are all green. That is real work and the core isolation direction is strong.
I still have two runtime blockers, plus two merge gates.
BLOCKER 1 — the refusal scope currently escapes the terminal boundary and can kill an otherwise non-terminal agent turn.
tools/terminal_scope.py::terminal_env() raises TerminalPolicyUnavailable whenever the active mapping is a TerminalPolicyRefusal. _profile_runtime_scope() deliberately installs that refusal and still yields, with the stated contract that terminal execution is refused rather than ambient authority being used.
But agent/prompt_builder.py::_tenv_read() calls terminal_env() without handling that typed refusal, and build_environment_hints() immediately uses _tenv_read("TERMINAL_ENV"). agent/system_prompt.py::build_system_prompt_parts() then calls _r.build_environment_hints() without a guard, and build_system_prompt() calls build_system_prompt_parts() directly. So a malformed/unreadable profile terminal policy is not currently “chat still works; terminal refuses”: on a system-prompt build/rebuild it raises out of prompt assembly before the model call.
The new refusal test does not cover this boundary; it installs the refusal and calls terminal_tool() directly. Please add a real routed-agent/system-prompt regression with malformed profile config.yaml proving (a) a non-terminal turn can still build its prompt and reach the model path, while (b) terminal_tool/execute_code remain refused. The clean semantic split is: execution surfaces hard-refuse; observational/prompt surfaces return no scoped hint on policy-unavailable, never ambient process values.
BLOCKER 2 — env_probe maps “policy unavailable” to local, which is the opposite side of the same refusal shape.
tools/env_probe.py::_resolve_terminal_backend() catches every exception from terminal_env() and returns "local". Under a refusal scope that turns epistemic state unavailable into a concrete host-local backend, after which get_environment_probe_line() is allowed to probe/cache the host toolchain. We must not reconstruct a concrete backend from an authority-resolution failure. Return an explicit unavailable/refused sentinel (or simply suppress the probe) so refusal neither crashes the turn (blocker 1) nor silently becomes ambient/local (this blocker).
These two need to be fixed together; otherwise one call path fails closed too broadly while another fails open informationally. The invariant should be simple: policy unavailable never widens/reconstructs terminal authority; only actual execution is denied.
MERGE GATE — the multiplex campaign has live file collisions and one semantic-doc contradiction that need an explicit order. These are complementary, not duplicates:
- #101244 (profile secret/config scope) also touches
gateway/run.py. Its newly addeddocs/design/multiplexing-gateway.mdcurrently lists “Terminal / sandbox env (TERMINAL_*)” as process-global under Known limitations. If this PR lands, that row becomes stale immediately. Either land this first and rebase/update #101244, or land #101244 first and update that design doc here. - #101245 (cron multiplex isolation) collides on
cron/scheduler.pyandwebsite/docs/user-guide/multi-profile-gateways.md; whichever lands second needs a real rebase plus cron scope-composition tests because both branches alter ContextVar/thread behavior at the same fire/delivery boundary. - #101246 collides on
gateway/run.pyandgateway/slash_commands.py; #101247 collides ongateway/run.py. Again: complementary slices, but exact-current-head green on each branch is not proof of the composed merge graph. - #99225 and #85437 remain open fork PRs. This branch is correctly a superseding salvage of those slices, not an independent reimplementation; keep their authorship/credit intact and close/supersede them only after this replacement actually lands.
MERGE GATE — every-commit CI is not yet proved. The exact current head is green across CI / Docker / Nix, which is good. The three preceding commits in this four-commit stack (80bff8a74fdcaba94c3849b565956dd1cf19c30d, ed3da7ea4ac0bba64f370b83f54b92b86c8f1947, efa8fecda06e3a696f7c9cf6ab5584ac45744cbd) currently have no hosted workflow runs attached. Under the every-commit gate, this is a proof gap rather than a code finding; please attach/replay equivalent receipts before calling the stack fully green.
Finally, this branch modifies two existing godfiles (gateway/run.py is already >20k lines at the touched hunk; cron/scheduler.py >7k). Under the sub-2K acceptance gate I cannot call that ownership clean. The new terminal mechanism itself is correctly extracted into a bounded module; please route the remaining boundary edits through the owning extraction graph / declare the merge order rather than treating further growth in those owners as neutral.
Once the refusal semantics are corrected and the composed-graph/CI gates are closed, I think the underlying terminal-scope architecture is the right fix for this defect class. The contributor-credit handling and the effort to salvage the strongest prior work rather than overwrite it are especially worth preserving.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
NousResearch#25119, NousResearch#85355, and NousResearch#90058 closed without merge. Tekium salvage PRs NousResearch#82162, NousResearch#101247, and NousResearch#101242 cover those gaps on upstream main.
Summary
Under a multiplexed gateway/dashboard/cron process every routed profile inherited the launch profile's terminal.* (backend,
cwd, docker volumes, shared-container key, SSH target) because terminal config was bridged once into process-global
TERMINAL_* env and read via os.getenv; separately, container boot read GATEWAY_MULTIPLEX_PROFILES from env only, so a
config.yaml-only opt-in crash-looped named gateways. This lands a per-turn authoritative terminal-policy ContextVar
(mirroring secret_scope) installed at every profile boundary, and routes boot through the shared gateway config resolver.
Changes
fallthrough under scope; malformed policy -> refusal scope, terminal_tool/execute_code refuse (fail closed)
suppressed while scoped; unscoped single-process path byte-identical
gateway media translation (base.py), runtime_cwd/agent_init/skill_utils/code_execution/file_tools cwd anchors,
prompt_builder/browser_tool/env_probe backend checks, footer/@-ref/slash cwd
Validation
/, multiplex active, real _profile_runtime_scope / reconcile_profile_gateways.
Credits
Commits by @muhifni (#99225) and @1052326311 (#85437) with authorship preserved. Co-authored-by @x7peeps (#68611),
@100yenadmin (#79117), @ExitMaster (#70600). Supersedes #79117 #85084 #94890 #78289 #70600 #80546 #84584 #88635 #90058
#95471 #86114 — thanks to all submitters; @a-yeyang for the #85413 RCA, @Adolanium for the #95470 repro.
Fixes #68559
Fixes #94200
Fixes #101132
Fixes #95470
Fixes #85413
Infographic