Skip to content

fix(gateway): per-profile hooks, MCP discovery, webhook skills, and scoped browser/Hindsight threads under multiplex (#92672 #95518 #86402 #92608 #67277, salvage #92682 #95542 #86472) - #101255

Merged
teknium1 merged 9 commits into
mainfrom
salvage/mux-tools-scope-b
Sep 2, 2026
Merged

teknium1 merged 9 commits into
mainfrom
salvage/mux-tools-scope-b

Conversation

@teknium1

@teknium1 teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Under multiplex_profiles, several process-global startup/background paths still ran under the launch home only: hooks.* and mcp_servers registered once from the root config, the webhook route rendered skills before entering the routed profile's scope (and skill_commands scanned an import-time-frozen SKILLS_DIR), and the browser janitor / Hindsight background threads were bare threads that hit fail-closed get_secret with no scope. This PR routes each through the profile's own scope — contextvars propagation where the spawner is scoped, per-session owner scope for the process-global janitor — with no os.environ fallthrough anywhere.

Changes

  • Gateway registers each served profile's shell hooks + outbound webhooks inside _start_one_profile_adapters; idempotence keys are home-scoped; force-reload re-registers per home; webhook payloads carry profile.
  • Boot MCP discovery runs once per served profile via copy_context(); /reload-mcp runs under the requesting profile and only tears down/rediscovers that profile's servers (shutdown_mcp_servers(scope=), registry.deregister(scope=)).
  • Browser inactivity janitor records each session's owner home, restores that scope around teardown, force-reaps (incl. cloud-provider close) after 3 failures; activity touch never resets the counter.
  • Hindsight writer/daemon-start/prefetch threads inherit the spawner's context.
  • Webhook adapter wraps route script/prompt/skill resolution in the routed profile's _profile_runtime_scope; skill_commands/skill_utils resolve the skills root at call time.

Validation

Item Before (origin/main) After
hooks profile b callbacks [], payload.profile None shell + outbound registered for b, payload.profile b
MCP boot/reload connect ['srv-default'] only ['srv-beta','srv-default'], beta overlay has only srv-beta tools
browser 4× UnscopedSecretError, session still tracked session reaped under owner scope
hindsight UnscopedSecretError in writer/daemon threads, log in launch home secrets resolve, log under profiles/p1
webhook skills ['/skill-a'], skill-b not injected ['/skill-b'], skill-b prompt injected
442 targeted tests pass; each fix sabotage-verified; ruff clean.

Credits

@chelsealong (#92682, commits preserved); @fangliquanflq (#95542, #86472); @vKongv (#85204 first submitter); @roraag (#88525); @KIAgent01 (#81816 first submitter); @Parker-Fawcett (#93028); @JuaniLezcano (#67280 / issue #67277); @webtecnica (#67293). Reporters: @vszgdcn8cj-ctrl, @asw-86-svg, @nniejadlik, @zbabiarz, @jirathip-k, @nino339.

Fixes #92672
Fixes #92674
Fixes #95518
Fixes #86402
Fixes #100738
Fixes #92608
Fixes #94933
Fixes #67277

Infographic

mux-tools-scope-b

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 2c68301 — chore: retrigger CI (zero-job dispatch failure, auto-heal)

⚠️ Warnings

OSV vulnerability scan · View job

13 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 4m37s vs 5m44s (-19.5%). 4 job(s) slower, 11 faster,

  • OS-specific tests / Windows-only tests: -86.0s
  • Docs Site / docs-site-checks: -54.0s
  • Check no case-colliding filenames / check-case-collisions: -38.0s
  • Check contributors / check-attribution: -33.0s
  • Python lints / Windows footguns (blocking): -28.0s

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins comp/cli CLI entry point, hermes_cli/, setup wizard tool/browser Browser automation (CDP, Playwright) tool/mcp MCP client and OAuth tool/memory Memory tool and memory providers tool/skills Skills system (list, view, manage) area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 2, 2026
@webtecnica

Copy link
Copy Markdown
Contributor

Thanks for the credit on #67293 — glad the webhook skills-scope fix made it into the multiplex cleanup.

@teknium1
teknium1 force-pushed the salvage/mux-tools-scope-b branch 5 times, most recently from b9381ca to bb3f378 Compare September 2, 2026 13:23
chelsealong and others added 9 commits September 2, 2026 06:54
…ebhooks

Multiplex gateway startup only ever calls agent.shell_hooks/
outbound_webhooks register_from_config() once, against the
root/default profile's config, before any profile scope exists.
_start_one_profile_adapters() discovers Python plugins per profile
but never registered that profile's own declarative `hooks:` block,
so a secondary profile's shell hooks (e.g. a deny-writes gate) and
outbound webhooks silently never fire.

Load and register each profile's own config inside its
_profile_runtime_scope, and key the module-level idempotence sets in
shell_hooks.py/outbound_webhooks.py by resolved Hermes home so two
profiles configuring an identical hook/webhook both register on
their own plugin manager instead of the second being dropped as a
duplicate of the first.

Fixes #92672
re_register_config_hooks() cleared the entire process-global idempotence
set on every force-reload, so a profile-local plugin force-reload dropped
another live profile's ledger key without touching its still-registered
callback — the next registration call for that profile then appended a
duplicate. Scope the clear to the reloading profile's own home, and give
outbound webhooks the same force-reload restoration shell hooks already
had, since unload() wipes both from the shared _hooks dict.
Under a multiplexed gateway every profile's outbound webhooks share one
delivery worker, so receivers could not tell which profile fired an
event. Add a top-level `profile` field to the payload, resolved at fire
time from the bound Hermes home via get_active_profile_name() ("default"
outside profiles). Documents the field in the wire-format section.

Reported by @vszgdcn8cj-ctrl.

Fixes #92674
…e scope

The inactivity janitor is one process-global thread started by whichever
profile first opens a browser, so under `gateway.multiplex_profiles` it runs
with no secret scope: `cleanup_browser` -> `is_camofox_mode` ->
`get_secret("CAMOFOX_URL")` raises UnscopedSecretError, the session entry is
never removed, and the same failure repeats every 30s while the Chromium
daemon leaks.

- `_update_session_activity` records the owning Hermes home per session;
  `_cleanup_inactive_browser_sessions` re-enters that owner's
  `set_hermes_home_override` + `build_profile_secret_scope` around each
  teardown (`_session_owner_scope`, mirroring `_profile_runtime_scope`).
  copy_context at thread spawn would pin the first profile's secrets onto
  every other profile's teardown; there is no os.environ fallthrough.
- 3 consecutive failures -> `_force_reap_browser_session`, which skips the
  failing `close` round-trips but still closes the cloud provider session
  and kills the local daemon via the shared `_release_session_resources`
  tail (extracted from `_cleanup_single_browser_session`, unchanged).
  An activity touch does not reset the failure budget.

Fixes #86402
Fixes #100738

Co-authored-by: fangliquanflq <fangliquan@qq.com>
…s under multiplex

Under multiplex_profiles the Hindsight provider's writer, daemon-start and
prefetch threads were spawned as bare threading.Thread, so they started with
an empty contextvars Context: no profile secret scope and no HERMES_HOME
override. get_secret() fails closed there, so the local_embedded daemon never
booted and every retain raised UnscopedSecretError, even though the spawning
thread (initialize()/sync_turn() inside the gateway's copy_context'd turn) had
the scope all along.

Spawn each thread with contextvars.copy_context().run so the child inherits
the spawner's scope + home override. No environ fallback, no re-parsed .env.
The shared hindsight-loop thread needs no wrap: coroutines submitted via
run_coroutine_threadsafe already run in the submitter's context per call.

Fixes #92608
Fixes #94933

Co-authored-by: KIAgent01 <297567825+KIAgent01@users.noreply.github.com>
Co-authored-by: Parker Fawcett <259203091+Parker-Fawcett@users.noreply.github.com>
A `/p/<profile>/webhooks/<route>` request resolved the profile from the URL
but ran the route script, prompt render and `skills:` lookup with no
profile scope — the runner only enters `_profile_runtime_scope` later,
around `handle_message` — so routed webhooks loaded the launch (default)
profile's skills and logged "Skill not found" for the routed profile's own.

- gateway/platforms/webhook.py: add `_profile_scope(profile)` (nullcontext
  when no prefix was resolved; `_profile_runtime_scope(get_profile_dir(p))`
  otherwise, same helper the runner uses) and wrap the script / render /
  skill-injection block in it. Bare routes are unchanged.
- agent/skill_commands.py: `scan_skill_commands` scanned the import-time
  `SKILLS_DIR` (frozen to the launch home), so even a correctly scoped call
  listed default's skills; the #88023 home-keyed cache alone could not fix
  that. Use the call-time `_skills_dir()` there and at the two other
  SKILLS_DIR-relative sites in the module.
- agent/skill_utils.py: `normalize_skill_lookup_name` used the same frozen
  root, so a routed profile's absolute skill_dir was rejected by
  `skill_view` ("must be a relative path within the skills directory").
  Resolve against `_skills_dir()` — the root `skill_view` itself enforces.

Fixes #67277

Co-authored-by: Juani Lezcano <tky.juani@gmail.com>
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
…plex

A multiplexed gateway ran `discover_mcp_tools()` once, unscoped, at boot
and again on `/reload-mcp`, so only the launch profile's `mcp_servers`
ever connected; secondary profiles' servers never registered, and a
`/reload-mcp` from any profile tore down every profile's connections.

- `_discover_gateway_mcp_tools()`: under multiplex, run discovery once per
  served profile inside `_profile_runtime_scope`, carried into the
  executor via `copy_context()` (same shape as
  `_run_in_executor_with_context`). Single-profile path unchanged.
- `_execute_mcp_reload()`: enter the requesting profile's scope when the
  caller (e.g. button-confirm callback) did not; shut down / rediscover /
  report only that profile's servers; refresh only that profile's cached
  agents.
- `shutdown_mcp_servers(scope=)`: scoped teardown keyed by the new
  `_server_scope_keys` ownership map; leaves the shared MCP loop running
  while other profiles' servers are live. Unscoped call keeps the full
  historical behavior.
- MCP tools register into the owning profile's registry overlay
  (`registry.register(scope=...)`), and `registry.deregister()` gains a
  matching `scope=` kwarg. Plugin callers still cannot name another
  profile's scope; the plugin-vs-global guard is unchanged for them.

Fixes #95518

Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: Kong <mgongzai@gmail.com>
Co-authored-by: roraag <232666910+roraag@users.noreply.github.com>
@teknium1
teknium1 force-pushed the salvage/mux-tools-scope-b branch from 7161807 to 2c68301 Compare September 2, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/browser Browser automation (CDP, Playwright) tool/mcp MCP client and OAuth tool/memory Memory tool and memory providers tool/skills Skills system (list, view, manage) type/bug Something isn't working

Projects

None yet

4 participants