Skip to content

fix(gateway): bind profile cwd in multiplexed sessions - #70600

Closed
ExitMaster wants to merge 1 commit into
NousResearch:mainfrom
ExitMaster:fix/multiplex-profile-cwd
Closed

ExitMaster wants to merge 1 commit into
NousResearch:mainfrom
ExitMaster:fix/multiplex-profile-cwd

Conversation

@ExitMaster

Copy link
Copy Markdown

Summary

Fix multiplexed gateway sessions inheriting the primary profile's process-global terminal cwd instead of the routed profile's configured cwd.

Root cause

Gateway startup bridges only the active profile's terminal.cwd into process-global TERMINAL_CWD. A routed secondary profile loaded its own context files, but terminal commands still inherited the primary process cwd. The terminal environment is also cached across sessions, so changing only initial environment configuration does not prevent cwd bleed.

Changes

  • resolve a routed profile's local terminal.cwd inside the existing profile runtime scope
  • bind that value through the existing session cwd ContextVar
  • expose the explicit ContextVar override to terminal infrastructure
  • lazily seed the terminal's canonical per-session cwd record on the first terminal call
  • key cwd state by the durable gateway/topic session key, not the transient execution ID
  • preserve explicit per-session cwd changes instead of resetting them each turn
  • keep single-profile and non-local backend behavior unchanged
  • fail closed to existing process cwd when profile config is missing, malformed, or uses a non-local backend

Regression coverage

  • explicit secondary-profile cwd
  • placeholder cwd resolution without inheriting primary cwd
  • existing session cd state preservation
  • single-profile behavior
  • non-local backend behavior
  • two multiplexed sessions sharing a cached local environment without cwd bleed

Verification

  • targeted regression: 6 passed
  • all gateway multiplex and cwd-related tests: 291 passed
  • py_compile
  • ruff check
  • git diff --check

Scope

This PR intentionally addresses the local terminal backend only. Profile-specific SSH/container backend configuration remains outside this change because those backends use different path semantics and process-global backend configuration.

Model used

OpenAI Codex gpt-5.6-sol; the implementation was manually constrained by TDD, source inspection, and executable regression tests.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery tool/terminal Terminal execution and process management area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state P2 Medium — degraded but workaround exists labels Jul 24, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the multiplexed cwd leak to the session binding seam. The core premise remains valid on current main: GatewayRunner._set_session_env() at gateway/run.py:20237 passes no cwd= to set_session_vars(), while agent/runtime_cwd.py:60 then falls back to process-global TERMINAL_CWD.

Problems

  • The new ContextVar binding would not cover direct process-env readers. gateway/run.py:15307 reads os.environ["TERMINAL_CWD"] during inbound @-reference preprocessing, before session binding at gateway/run.py:15641; gateway/run.py:16885 does the same for the runtime footer. A routed profile can still use or display the primary profile cwd on those paths.

Suggested changes

  • Route these consumers through resolve_agent_cwd() or bind the routed cwd before preprocessing, then add regressions for @ references and the enabled footer.
  • Salvage the focused change onto the current _set_session_env() at gateway/run.py:20237; the method moved during 1a3a9de630's TurnRunner extraction.

Automated hermes-sweeper review.

Comment thread gateway/run.py
@@ -17179,9 +17180,69 @@ def _set_session_env(self, context: SessionContext) -> list:
session_key=context.session_key,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This binds the resolver used by agent startup and terminal seeding, but gateway/run.py:15307 still reads process-global TERMINAL_CWD before this session binding for @-reference expansion, and the footer does so at line 16885. Please route those paths through the scoped resolver or establish this cwd before preprocessing, with coverage.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/sessions Session lifecycle, resume, persistence, history labels Jul 30, 2026
@dyreckt

dyreckt commented Aug 2, 2026 •

Copy link
Copy Markdown

Verified production repro + completed rebase with the review gaps covered

I hit this exact bug in production on v0.19.1: a Telegram topic routed to a secondary profile (mco) ran on the routed profile's model and loaded its skills, but the runtime footer and tools resolved to the primary profile's workspace (~/hermes/workspace) instead of the routed profile's terminal.cwd (~/.hermes/profiles/xxx/workspace/xxxxxxxx). Session logs confirmed routing was correct; only the cwd resolution was wrong — matching your root-cause analysis exactly.

I rebased your fix onto current main and addressed the two gaps from the sweeper review:

  • @-reference preprocessor (_prepare_inbound_message_text) now reads the session-cwd override. Because that preprocessor runs before _set_session_env(), I also pin the routed cwd early in _prepare_profile_scoped_inbound_message_text so @file:/@folder: references resolve against the routed profile's workspace.
  • Runtime footer now uses the session-cwd override instead of process-global TERMINAL_CWD, so the status line shows the routed profile's cwd.
  • Two regression tests added for both paths, alongside your original six.

Verification (branch fix/multiplex-profile-cwd-complete, pushed to dyreckt/hermes-agent — based on current main d0b87dad779, one commit on top):

  • tests/gateway/test_multiplex_profile_cwd.py: 8 passed (6 original + 2 new)
  • Related footer/context-ref suites: 24 passed
  • Full tests/gateway/: 4659 passed; 55 failures in test_telegram_noise_filter.py are pre-existing on clean main (confirmed identical without this branch) and unrelated to cwd
  • py_compile, ruff check, git diff --check: clean

The branch is a drop-in superset of this PR's commit — if it's easier for the maintainers, the completed branch can be pulled into this PR, or the additional hunks can be applied on top of yours. Either way the fix is now complete and ready to merge.

@dyreckt

dyreckt commented Aug 6, 2026

Copy link
Copy Markdown

Update: the completed branch now also includes two further fixes found while verifying in production — a ContextVar wipe in _prepare_profile_scoped_inbound_message_text that broke the footer fallback, and _resolve_profile_home_for_source() leaking an inherited profile scope for unrouted sources. Opened as #80546 (single squashed commit on current main, co-authored); this PR's branch can still be pulled in instead if preferred — whichever is easier.

@dyreckt

dyreckt commented Aug 20, 2026 •

Copy link
Copy Markdown

Just to clarify authorship on the record: #70600 is @ExitMaster's PR, and dyreckt's #80546 builds on it (same fix, plus the review gaps: runtime footer, @-reference preprocessor, terminal-tool seeding, and a copy_context() home-resolution leak, with regression tests). Both remain open; the recommendation is for maintainers to consolidate on one — #80546 has the fuller coverage, but #70600 deserves credit as the original. No closure action intended from dyreckt's side on this PR.

@dyreckt

dyreckt commented Aug 20, 2026 •

Copy link
Copy Markdown

(Note: the previous "Superseded by #80546" line was posted in error by an automated close attempt; #70600 remains open and is @ExitMaster's PR — dyreckt's #80546 builds on it and both stay open for maintainer consolidation.)

@vszgdcn8cj-ctrl

Copy link
Copy Markdown

Thanks for this — I hit the same bug and can confirm the root cause. Two gaps remain after this PR that I wanted to flag, since both bit me on a multiplexed gateway where the secondary profile is reached over the API server rather than a messaging adapter:

1. The API-server path is not covered. _bind_api_server_session in gateway/platforms/api_server.py (around line 7087, the documented "single structural chokepoint" for every /p/<profile>/v1/... route) calls set_session_vars(...) without cwd= (and without profile= — related: #92674). So resolve_context_cwd() still falls through to the process-global TERMINAL_CWD, which is bridged from the root config only (gateway/run.py ~2614). Result: a request to /p/second/v1/chat/completions loads the default profile's AGENTS.md. The same _session_cwd_for_source logic this PR adds to _set_session_env needs to be applied there too.

2. File tools never consult the session cwd contextvar. tools/file_tools.py::_authoritative_workspace_root / _resolve_base_dir (~277-367) resolve in this order: terminal session record → registered task override → $TERMINAL_CWD → process cwd. Neither file_tools.py nor terminal_tool.py imports agent.runtime_cwd. This PR seeds the terminal's per-session cwd record lazily on the first terminal call, so a profile that has the terminal toolset disabled (common for a restricted profile) never gets seeded, and relative read_file / write_file on the secondary profile still land in the default profile's workspace. Locally I worked around it by also calling register_task_env_overrides(...) with the profile's cwd in the API-server binding, which makes the "registered task override" branch resolve correctly.

Minimal repro:

# ~/.hermes/config.yaml
gateway: { multiplex_profiles: true }
terminal: { cwd: /home/u/.hermes/workspace }
# ~/.hermes/profiles/second/config.yaml
terminal: { cwd: /home/u/.hermes/profiles/second/workspace }
disabled_toolsets: [terminal]

Put a different AGENTS.md in each workspace, then POST /p/second/v1/chat/completions asking the model to read_file AGENTS.md and write_file note.txt. Both hit the default workspace. Verified on 2584b7c (v0.20.5, macOS 26.5, Python 3.11); the relevant functions are unchanged on current main.

Happy to open a separate issue for the file-tools half if you'd prefer to keep this PR scoped.

dyreckt added a commit to dyreckt/hermes-agent that referenced this pull request Aug 24, 2026
In a multiplexed gateway, startup bridges only the primary profile's
terminal.cwd to process-global TERMINAL_CWD. Sessions routed to a
secondary profile resolved their cwd from that process-global latch, so
the runtime footer, @-reference preprocessing, and the terminal tool all
used the wrong profile's workspace — and whichever profile made the
first terminal call after a restart stamped its workspace process-wide
for every lane.

- gateway/run.py: _set_session_env() resolves the routed profile's
  terminal.cwd and binds it through the session ContextVar; the runtime
  footer and @-reference preprocessor read the session-cwd override
  instead of os.environ["TERMINAL_CWD"];
  _resolve_profile_home_for_source() falls back to the gateway process's
  own home rather than get_active_profile_name(), which can leak a
  previously routed turn's profile scope through copy_context()
- agent/runtime_cwd.py: expose get_session_cwd_override()
- tools/terminal_tool.py: seed the per-session cwd from the ContextVar
  override instead of the process-global TERMINAL_CWD latch
- tests: multiplex profile cwd isolation, footer and @-reference
  coverage, unrouted-source home resolution under an inherited profile
  scope, and profile-resolution fallback tests repointed at the new
  gateway-home seam

Builds on NousResearch#70600 and addresses its review gaps (@-reference preprocessor
and runtime footer now honor the session-cwd override).

Co-authored-by: ExitMaster <292490062+ExitMaster@users.noreply.github.com>
@teknium1

teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this PR. Merged via #101242 (4a7f228) on current main — routed multiplex profiles get their own terminal cwd/backend/docker config; container boot honors config multiplex_profiles.

#101242 won as the consolidated fix because it covers the whole multiplex-profile bug class in one change (with tests) rather than the single symptom addressed here; this PR is superseded by it.
You are credited via Co-authored-by / in the PR body of #101242 as noted there.

If anything from your original change is still missing on main >= 4a7f228, please open a fresh PR/issue against main and tag it. Thanks again.

@teknium1 teknium1 closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/terminal Terminal execution and process management type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants