Skip to content

fix(gateway): routed multiplex profiles get their own terminal cwd/backend/docker config; container boot honors config multiplex_profiles (#68559 #85413, salvage #99225 #85437) - #101242

Merged
teknium1 merged 4 commits into
mainfrom
salvage/mux-terminal-scope
Sep 2, 2026
Merged

teknium1 merged 4 commits into
mainfrom
salvage/mux-terminal-scope

Conversation

@teknium1

@teknium1 teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Under a multiplexed gateway/dashboard/cron process every routed profile inherited the launch profile's terminal.* (backend,
cwd, docker volumes, shared-container key, SSH target) because terminal config was bridged once into process-global
TERMINAL_* env and read via os.getenv; separately, container boot read GATEWAY_MULTIPLEX_PROFILES from env only, so a
config.yaml-only opt-in crash-looped named gateways. This lands a per-turn authoritative terminal-policy ContextVar
(mirroring secret_scope) installed at every profile boundary, and routes boot through the shared gateway config resolver.

Changes

  • tools/terminal_scope.py: complete per-profile TERMINAL_* projection (defaults <- .env <- config.yaml); no os.environ
    fallthrough under scope; malformed policy -> refusal scope, terminal_tool/execute_code refuse (fail closed)
  • Installed in gateway _profile_runtime_scope, tui_gateway session/build/turn scopes, cron per-job fire; env bridge
    suppressed while scoped; unscoped single-process path byte-identical
  • All terminal.* consumers read through the scope: terminal_tool (_get_env_config, shared key, reaper, degraded mode),
    gateway media translation (base.py), runtime_cwd/agent_init/skill_utils/code_execution/file_tools cwd anchors,
    prompt_builder/browser_tool/env_probe backend checks, footer/@-ref/slash cwd
  • hermes_cli/container_boot.py: multiplex_profiles from load_gateway_config() with env fallback on load failure
  • Docs note in multi-profile-gateways.md

Validation

Probe Before (origin/main) After
routed local profile, launch=docker: terminal_tool env_type/cwd docker / launch cwd local / profile cwd
docker_volumes / shared key / container task id launch mounts / alpha-shared / shared:alpha-shared [] / '' / default
media translation mounts + sandbox candidates launch mounts, shared_alpha-shared none, profile_bee
file_tools cwd, runtime_cwd, browser local, env_probe backend leak / leak / False / docker profile / profile / True / local
malformed profile config.yaml ran under launch policy refused: "terminal policy unavailable"
boot with config multiplex_profiles: true, no env coder slot 'started' (crash loop) 'registered' + down file
Tests: 92 passed across 8 files; sabotage runs fail as expected. Live repro: real-import harness against isolated
HERMES_HOME + profiles/

/, multiplex active, real _profile_runtime_scope / reconcile_profile_gateways.

Credits

Commits by @muhifni (#99225) and @1052326311 (#85437) with authorship preserved. Co-authored-by @x7peeps (#68611),
@100yenadmin (#79117), @ExitMaster (#70600). Supersedes #79117 #85084 #94890 #78289 #70600 #80546 #84584 #88635 #90058
#95471 #86114 — thanks to all submitters; @a-yeyang for the #85413 RCA, @Adolanium for the #95470 repro.

Fixes #68559
Fixes #94200
Fixes #101132
Fixes #95470
Fixes #85413

Infographic

mux-terminal-scope

muhifni and others added 4 commits September 2, 2026 04:22
A multiplexed Hermes process (gateway.multiplex_profiles, unified
dashboard/TUI, or cron) serves several profiles at once, but terminal.*
resolved through process-global TERMINAL_* env vars bridged ONCE at
startup from the launch profile (gateway/run.py ~2700-2760) plus the
one-shot _ensure_terminal_env_bridged() guard. Every routed profile
therefore inherited the launch profile's backend, cwd, docker volumes,
SSH target and shared-container key: a local profile ran inside another
profile's docker sandbox (or a docker profile escaped to the host), and a
container labeled profile A carried profile B's RW bind mounts.

Fix: an authoritative per-profile terminal policy seam, mirroring
agent/secret_scope.py:

- tools/terminal_scope.py: ContextVar holding the routed profile's
  COMPLETE effective TERMINAL_* policy (defined defaults <- profile .env
  TERMINAL_* <- config.yaml terminal:). While bound, terminal_env()
  resolves ONLY from it - an omitted key yields the defined default,
  never os.environ. Unreadable/malformed policy installs a refusal
  scope; terminal_tool / execute_code refuse instead of running under
  ambient launch-process policy (fail closed).
- Installed at every in-process profile boundary: gateway
  _profile_runtime_scope, tui_gateway session/build/turn scopes, cron
  per-job fire. The unscoped single-process path is byte-identical.
- Every terminal.* consumer reads through the scope: terminal_tool
  (_get_env_config, _resolve_container_task_id shared key, orphan
  reaper lifetime, degraded mode), gateway/platforms/base.py docker
  media translation (volumes, shared key, persistence), runtime_cwd /
  agent_init / skill_utils / code_execution_tool / file_tools cwd
  anchors, prompt_builder / browser_tool / env_probe backend checks,
  gateway footer, @-refs and slash-command cwd. env_probe resolves the
  backend in the caller's context, since the probe worker thread does
  not inherit the ContextVar.

Salvage of #99225 onto current main: adds the three ambient reads the PR
missed (tools/file_tools.py TERMINAL_CWD, tools/browser_tool.py and
tools/env_probe.py TERMINAL_ENV; shape from #79117) and trims the test
module to the leak matrix driven through the real gateway boundary,
omitted-key defaults, refusal, and boundary reset.

Fixes #68559
Fixes #94200
Fixes #101132
Fixes #95470

Co-authored-by: x7peeps <9640837+x7peeps@users.noreply.github.com>
Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com>
Co-authored-by: ExitMaster <292490062+ExitMaster@users.noreply.github.com>
hermes_cli/container_boot.py resolved multiplexing from the
GATEWAY_MULTIPLEX_PROFILES env var only, while the gateway runtime
resolves env -> config.yaml -> default. A deployment enabling
multiplex_profiles via config.yaml alone therefore auto-started every
named profile's gateway slot at boot, which then crash-looped in the
double-bind guard against the multiplexing default gateway.

Resolve through load_gateway_config().multiplex_profiles (the shared
resolver, so env override precedence is preserved) and fall back to the
env var only when config loading fails.

Salvage of #85437 (test module trimmed to config-only + env-override).

Fixes #85413
Documents the per-profile terminal.* resolution and fail-closed refusal
behavior introduced by the terminal scope seam (#68559 class).
@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

૮ >ﻌ< ა ci review

ran on 3165a91 — chore: map contributor email for muhifni (#99225 salvage)

⚠️ Warnings

OSV vulnerability scan · View job

13 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 4m30s vs 4m32s (-0.7%). 7 job(s) slower, 5 faster, 3 unchanged.

  • Python tests / Run tests: -36.0s
  • Docs Site / docs-site-checks: +22.0s
  • Profile artifact check / Reject profile archives: +8.0s
  • Python lints / Windows footguns (blocking): +7.0s
  • Check no case-colliding filenames / check-case-collisions: +7.0s

@andrexibiza andrexibiza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 3165a9198f14fb80f94b08ae178ce9ec7f7e7b35 against base 3a980a431b28633a5b79c462f654dc100dc1c598, including the 21-file diff, the new tools/terminal_scope.py, tests/tools/test_terminal_scope_multiplex.py, the container-boot slice/tests, the prompt/system-prompt call path, current exact-head CI, commit authorship, and the adjacent multiplex PR graph.

There is a lot that is right here. Moving terminal authority out of process-global TERMINAL_* and into a per-profile ContextVar is the correct class-level shape; the scope projection is explicit, omission does not fall back to ambient os.environ, the mutation-capable terminal/code-exec sinks have a hard refusal gate, and the salvage preserves the original authorship from muhifni (#99225) and 1052326311 (#85437) instead of laundering the work into a rewrite. Current-head CI, Docker, and Nix are all green. That is real work and the core isolation direction is strong.

I still have two runtime blockers, plus two merge gates.

BLOCKER 1 — the refusal scope currently escapes the terminal boundary and can kill an otherwise non-terminal agent turn.

tools/terminal_scope.py::terminal_env() raises TerminalPolicyUnavailable whenever the active mapping is a TerminalPolicyRefusal. _profile_runtime_scope() deliberately installs that refusal and still yields, with the stated contract that terminal execution is refused rather than ambient authority being used.

But agent/prompt_builder.py::_tenv_read() calls terminal_env() without handling that typed refusal, and build_environment_hints() immediately uses _tenv_read("TERMINAL_ENV"). agent/system_prompt.py::build_system_prompt_parts() then calls _r.build_environment_hints() without a guard, and build_system_prompt() calls build_system_prompt_parts() directly. So a malformed/unreadable profile terminal policy is not currently “chat still works; terminal refuses”: on a system-prompt build/rebuild it raises out of prompt assembly before the model call.

The new refusal test does not cover this boundary; it installs the refusal and calls terminal_tool() directly. Please add a real routed-agent/system-prompt regression with malformed profile config.yaml proving (a) a non-terminal turn can still build its prompt and reach the model path, while (b) terminal_tool/execute_code remain refused. The clean semantic split is: execution surfaces hard-refuse; observational/prompt surfaces return no scoped hint on policy-unavailable, never ambient process values.

BLOCKER 2 — env_probe maps “policy unavailable” to local, which is the opposite side of the same refusal shape.

tools/env_probe.py::_resolve_terminal_backend() catches every exception from terminal_env() and returns "local". Under a refusal scope that turns epistemic state unavailable into a concrete host-local backend, after which get_environment_probe_line() is allowed to probe/cache the host toolchain. We must not reconstruct a concrete backend from an authority-resolution failure. Return an explicit unavailable/refused sentinel (or simply suppress the probe) so refusal neither crashes the turn (blocker 1) nor silently becomes ambient/local (this blocker).

These two need to be fixed together; otherwise one call path fails closed too broadly while another fails open informationally. The invariant should be simple: policy unavailable never widens/reconstructs terminal authority; only actual execution is denied.

MERGE GATE — the multiplex campaign has live file collisions and one semantic-doc contradiction that need an explicit order. These are complementary, not duplicates:

  • #101244 (profile secret/config scope) also touches gateway/run.py. Its newly added docs/design/multiplexing-gateway.md currently lists “Terminal / sandbox env (TERMINAL_*)” as process-global under Known limitations. If this PR lands, that row becomes stale immediately. Either land this first and rebase/update #101244, or land #101244 first and update that design doc here.
  • #101245 (cron multiplex isolation) collides on cron/scheduler.py and website/docs/user-guide/multi-profile-gateways.md; whichever lands second needs a real rebase plus cron scope-composition tests because both branches alter ContextVar/thread behavior at the same fire/delivery boundary.
  • #101246 collides on gateway/run.py and gateway/slash_commands.py; #101247 collides on gateway/run.py. Again: complementary slices, but exact-current-head green on each branch is not proof of the composed merge graph.
  • #99225 and #85437 remain open fork PRs. This branch is correctly a superseding salvage of those slices, not an independent reimplementation; keep their authorship/credit intact and close/supersede them only after this replacement actually lands.

MERGE GATE — every-commit CI is not yet proved. The exact current head is green across CI / Docker / Nix, which is good. The three preceding commits in this four-commit stack (80bff8a74fdcaba94c3849b565956dd1cf19c30d, ed3da7ea4ac0bba64f370b83f54b92b86c8f1947, efa8fecda06e3a696f7c9cf6ab5584ac45744cbd) currently have no hosted workflow runs attached. Under the every-commit gate, this is a proof gap rather than a code finding; please attach/replay equivalent receipts before calling the stack fully green.

Finally, this branch modifies two existing godfiles (gateway/run.py is already >20k lines at the touched hunk; cron/scheduler.py >7k). Under the sub-2K acceptance gate I cannot call that ownership clean. The new terminal mechanism itself is correctly extracted into a bounded module; please route the remaining boundary edits through the owning extraction graph / declare the merge order rather than treating further growth in those owners as neutral.

Once the refusal semantics are corrected and the composed-graph/CI gates are closed, I think the underlying terminal-scope architecture is the right fix for this defect class. The contributor-credit handling and the effort to salvage the strongest prior work rather than overwrite it are especially worth preserving.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/tools Tool registry, model_tools, toolsets tool/browser Browser automation (CDP, Playwright) tool/code-exec execute_code sandbox tool/file File tools (read, write, patch, search) tool/terminal Terminal execution and process management backend/docker Docker container execution area/config Config system, migrations, profiles area/docker Docker image, Compose, packaging area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 2, 2026
@teknium1
teknium1 merged commit 4a7f228 into main Sep 2, 2026
44 checks passed
@teknium1
teknium1 deleted the salvage/mux-terminal-scope branch September 2, 2026 12:34
cursor Bot pushed a commit to pebble-tech/hermes-agent that referenced this pull request Sep 22, 2026
cursor Bot pushed a commit to pebble-tech/hermes-agent that referenced this pull request Sep 23, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 24, 2026
cursor Bot pushed a commit to pebble-tech/hermes-agent that referenced this pull request Sep 27, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 28, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 29, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
vKongv added a commit to pebble-tech/hermes-agent that referenced this pull request Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles area/docker Docker image, Compose, packaging area/profiles Multi-profile isolation, HERMES_HOME scoping backend/docker Docker container execution comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/browser Browser automation (CDP, Playwright) tool/code-exec execute_code sandbox tool/file File tools (read, write, patch, search) tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

6 participants