Skip to content

fix(gateway): bind profile cwd in multiplexed sessions - #80546

Closed
dyreckt wants to merge 1 commit into
NousResearch:mainfrom
dyreckt:fix/multiplex-profile-cwd-complete
Closed

dyreckt wants to merge 1 commit into
NousResearch:mainfrom
dyreckt:fix/multiplex-profile-cwd-complete

Conversation

@dyreckt

@dyreckt dyreckt commented Aug 6, 2026

Copy link
Copy Markdown

What does this PR do?

In a multiplexed gateway (multiplex_profiles: true), startup bridges only the primary profile's terminal.cwd to the process-global TERMINAL_CWD env var. Sessions routed to a secondary profile resolved their cwd from that process-global latch, so:

  • the runtime footer showed the wrong profile's workspace,
  • @file:/@folder: reference preprocessing resolved against the wrong profile's workspace,
  • the terminal tool seeded its per-session cwd from whichever profile made the first terminal call after a gateway restart — a coin flip that stamped one profile's workspace process-wide for every lane.

This PR binds the routed profile's cwd through the existing session ContextVar and routes every consumer through it, making cwd resolution deterministic per profile. It builds on #70600 and addresses the gaps raised in its review (the @-reference preprocessor and the runtime footer still read os.environ["TERMINAL_CWD"] directly), plus two further defects found while verifying the fix in production:

  1. _prepare_profile_scoped_inbound_message_text ran after _set_session_env() and its pin/clear wiped the ContextVar, so the footer fell back to process-global TERMINAL_CWD.
  2. _resolve_profile_home_for_source() fell back to get_active_profile_name(), which honors the context-local profile override. Messaging tasks spawn with copy_context(), so an unrouted/default-lane turn could inherit a previously routed profile's scope and resolve to the wrong profile home. It now falls back to the gateway process's own home.

Single-profile gateways are deliberately unchanged: _session_cwd_for_source() returns an empty override when multiplex_profiles is off, preserving existing process-level cwd behavior.

Related Issue

Builds on / supersedes #70600 (review feedback addressed; happy for that branch to be pulled in here or vice versa — whichever is easier for maintainers).

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • gateway/run.py
    • _set_session_env() resolves the routed profile's terminal.cwd via new _session_cwd_for_source() and binds it through the session ContextVar (set_session_vars(cwd=...))
    • Runtime footer reads get_session_cwd_override() before falling back to TERMINAL_CWD
    • @-reference preprocessor resolves its cwd from the session override first
    • _resolve_profile_home_for_source() falls back to the gateway process home instead of get_active_profile_name() (context-scope leak via copy_context())
  • agent/runtime_cwd.py — expose get_session_cwd_override()
  • tools/terminal_tool.py — seed the per-session cwd from the ContextVar override instead of the process-global TERMINAL_CWD latch
  • tests/gateway/test_multiplex_profile_cwd.py — 8 regression tests: per-profile cwd isolation, footer, @-reference resolution, terminal-tool seeding
  • tests/gateway/test_multiplex_credential_isolation.py — regression test: unrouted source resolves to the gateway home even under an inherited profile scope

How to Test

  1. Configure a multiplexed gateway with two profiles that have different terminal.cwd values, and a profile_routes entry routing one chat/topic to the secondary profile.
  2. Send a message in the routed chat: the runtime footer must show the secondary profile's workspace, and terminal calls must start in that cwd.
  3. Send a message in an unrouted (default-lane) chat immediately after: footer and cwd must show the primary profile's workspace — no leakage from the routed turn.
  4. Regression suite: python -m pytest tests/gateway/test_multiplex_profile_cwd.py tests/gateway/test_multiplex_credential_isolation.py -q → 13 passed.
  5. Verified in production on macOS 26 with a live multiplexed gateway (Telegram topics routed to two secondary profiles): footers and terminal cwd now resolve deterministically per profile across restarts.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate — builds on fix(gateway): bind profile cwd in multiplexed sessions #70600 with its review gaps addressed
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — full scripts/run_tests.sh run: only pre-existing failures on clean main (unrelated to cwd), no new failures from this change
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 26.6 (Apple Silicon), Python 3.13

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A (docstrings only)
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config changes)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — pathlib throughout, no signal/process-group calls; scripts/check-windows-footguns.py clean
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A (no schema changes)

Screenshots / Logs

Production footer after fix — routed secondary profile session shows its own workspace instead of the primary profile's:

before (routed lane):  cwd ~/<primary>/workspace            ← wrong profile
after  (routed lane):  cwd ~/<primary>/profiles/<secondary>/workspace/<project>  ✓
after  (default lane): cwd ~/<primary>/workspace            ✓ no cross-lane leak

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery tool/terminal Terminal execution and process management area/config Config system, migrations, profiles area/sessions Session lifecycle, resume, persistence, history area/profiles Multi-profile isolation, HERMES_HOME scoping P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 6, 2026
@dyreckt
dyreckt force-pushed the fix/multiplex-profile-cwd-complete branch from 2c411f1 to ad7bce7 Compare August 9, 2026 06:52
In a multiplexed gateway, startup bridges only the primary profile's
terminal.cwd to process-global TERMINAL_CWD. Sessions routed to a
secondary profile resolved their cwd from that process-global latch, so
the runtime footer, @-reference preprocessing, and the terminal tool all
used the wrong profile's workspace — and whichever profile made the
first terminal call after a restart stamped its workspace process-wide
for every lane.

- gateway/run.py: _set_session_env() resolves the routed profile's
  terminal.cwd and binds it through the session ContextVar; the runtime
  footer and @-reference preprocessor read the session-cwd override
  instead of os.environ["TERMINAL_CWD"];
  _resolve_profile_home_for_source() falls back to the gateway process's
  own home rather than get_active_profile_name(), which can leak a
  previously routed turn's profile scope through copy_context()
- agent/runtime_cwd.py: expose get_session_cwd_override()
- tools/terminal_tool.py: seed the per-session cwd from the ContextVar
  override instead of the process-global TERMINAL_CWD latch
- tests: multiplex profile cwd isolation, footer and @-reference
  coverage, unrouted-source home resolution under an inherited profile
  scope, and profile-resolution fallback tests repointed at the new
  gateway-home seam

Builds on NousResearch#70600 and addresses its review gaps (@-reference preprocessor
and runtime footer now honor the session-cwd override).

Co-authored-by: ExitMaster <292490062+ExitMaster@users.noreply.github.com>
@teknium1

teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this PR. Merged via #101242 (4a7f228) on current main — routed multiplex profiles get their own terminal cwd/backend/docker config; container boot honors config multiplex_profiles.

#101242 won as the consolidated fix because it covers the whole multiplex-profile bug class in one change (with tests) rather than the single symptom addressed here; this PR is superseded by it.
Your PR is referenced in #101242's body as prior work on this bug.

If anything from your original change is still missing on main >= 4a7f228, please open a fresh PR/issue against main and tag it. Thanks again.

@teknium1 teknium1 closed this Sep 2, 2026
@neurosovereign

Copy link
Copy Markdown

Production evidence for this bug class, from a deployment that ran gateway.multiplex_profiles with six profiles over E2EE Matrix through 2026-08-11 (we've since moved to per-profile gateways for operational reasons, so we no longer exercise this path — but we hit it hard while we did, and we've maintained the patch ever since).

The bug as it presented: every Matrix session bound to a non-launch profile resolved its workspace from the launch profile's TERMINAL_CWD — wrong AGENTS.md, wrong project context, wrong relative paths in terminal/file tools. Our launch profile was effectively a system account, so all five user-facing profiles inherited the wrong cwd on every session. We shipped a local patch the same day (2026-08-11) and have carried it since.

One semantic point the PRs in this class differ on: the cwd source must be the session profile's own terminal.cwd from that profile's config.yaml — not get_hermes_home() (the profile home is ~/.hermes/profiles/<name>, the state dir, not the workspace; #90058 pins exactly that), and not the routed scope alone. Profiles with disjoint configured workspaces would regress in the opposite direction under home-pinning. The resolver also needs placeholder filtering (./auto/cwd resolve to home via the config bridge) and existence validation, failing open to legacy resolution so single-profile behavior is untouched.

Our carry, rebased onto v0.21.0's _SESSION_CWD plumbing: resolve the session profile's terminal.cwd in _set_session_env and pass it as cwd= into set_session_vars — ~20 lines of gateway change plus the resolver, behavioral tests included: neurosovereign/hermes-agent@c4ccd911. Tests green on current main (tests/agent/test_runtime_cwd.py, tests/gateway/test_session_env.py).

Honest limitation of the minimal variant: it does not cover the direct os.environ["TERMINAL_CWD"] readers flagged on #70600 (@-reference preprocessing, runtime footer, terminal-tool seeding) — which is why we track this PR rather than submitting ours as a fifth duplicate. If the maintainer prefers the small seam fix first and reader-migration second, the commit above is available as that increment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants