fix(gateway,tools): multiplexed profiles keep their own max_turns, fallback chain, hooks, auth, caches and paths (salvage #56315 #56508 #63962, fixes #95685) - #108453
Merged
Conversation
૮ >ﻌ< ა ci reviewran on e10797d — ci: retrigger (zero-job dispatch on 155683e)
|
alt-glitch
added
type/bug
teknium1
force-pushed
the
fix/mux-process-memo
branch
from
September 11, 2026 21:10
67a8092 to
7096421
Compare
… isolation gateway/hooks.py::HOOKS_DIR is resolved once at import time via get_hermes_home(), which is a context-local ContextVar under the multiplexed gateway (multiple profiles sharing one process, each owning its own Gateway/HookRegistry instance). Freezing the path at import time pins every later HookRegistry.discover_and_load() call to whichever profile's HERMES_HOME was active when this module was first imported -- so a later-starting profile silently discovers and executes the FIRST profile's hook handlers (arbitrary Python code, not just data) against its own live event context, including session_id/message/response text. Same bug class already fixed for cache dirs, skills_hub, rich_sent_store, and the OAuth/auth.json/sessions.json/checkpoint/sticker-cache paths. Add a per-call resolver, following the established "respect an existing test monkeypatch of the constant, otherwise re-resolve through get_hermes_home()" pattern so the existing test seam in tests/gateway/test_hooks.py keeps working unmodified. (cherry picked from commit 1e4f96a)
…, hooks, aux auth and media policy One multiplexed gateway process serves every profile, but several per-turn reads still went through state frozen from the LAUNCH profile: - `_current_max_iterations` re-bridged `agent.max_turns`/`sessions.*` from the module constant `_hermes_home` into one process-wide HERMES_MAX_ITERATIONS, so every secondary ran with the default profile's turn budget. A routed turn (HERMES_HOME override) now resolves `agent.max_turns` from its own config. - `_refresh_fallback_model` read `_hermes_home/config.yaml` into one runner-wide slot, so secondaries fell back through the default's provider/model with their own keys. It now reads the active gateway home and keeps a last-known-good chain per home. - `_load_prefill_messages` resolved relative paths against the launch home. - `agent/auxiliary_client._AUTH_JSON_PATH` was an import-time constant, so a secondary's compression/title/vision calls authenticated to Nous with the default profile's token when it had no pool entry. Resolved per call via `hermes_cli.auth._auth_file_path()` (patched constant still wins in tests). - `gateway/hooks.HOOKS_DIR` was frozen at import and one `HookRegistry` was loaded outside any profile scope, so secondaries' `hooks/` never ran and the default profile's handlers received every profile's messages, responses and user ids. `HOOKS_DIR` now resolves per call (salvaged from #56508) and the runner holds one registry per served home, picked from the active scope at emit time and front-loaded under each secondary's startup scope. - Shell-hook subprocesses inherited the launch `os.environ` (default HERMES_HOME and the default profile's secrets). They now get the routed HERMES_HOME via `build_subprocess_env`, scrubbed under multiplexing, and the stdin payload carries `profile` so one script can tell which profile fired it. - Media-delivery policy (`gateway.strict`, `media_delivery_allow_dirs`, `trust_recent_files*`) was bridged once into env at startup and read from env per delivery; under a HERMES_HOME override the validator now reads the routed profile's config. Single-profile runs keep the env-bridge contract. Audit: /tmp/mux_audit F3, F4, F6 (auth.json half), F7, F12 (media). Live repro (temp HERMES_HOME A with profiles/B): before, B saw max_iterations 7, fallback A/fallback, TOKEN_A, A's hooks, strict=A; after, all B's values.
tools/checkpoint_manager.py's CHECKPOINT_BASE and gateway/sticker_cache.py's CACHE_PATH are resolved once at import time via get_hermes_home(), which is a context-local ContextVar under the multiplexed gateway (multiple profiles sharing one process). Freezing the path at import time pins every later checkpoint/cache read-write to whichever profile's HERMES_HOME was active when the module was first imported -- the same bug class already fixed for cache dirs, skills_hub, rich_sent_store, and (this session) the OAuth/auth.json/ sessions.json paths. CheckpointManager is "owned by AIAgent" per-instance, but its methods read the frozen module constant directly instead of taking the store root from the instance, so a profile's CheckpointManager can read/write code-edit checkpoints into a different profile's store. Add a per-call resolver for each path, following the established "respect an existing test monkeypatch of the constant, otherwise re-resolve through get_hermes_home()" pattern so the extensive existing test seams in tests/tools/test_checkpoint_manager.py and tests/gateway/test_sticker_cache.py keep working unmodified. (cherry picked from commit 03ae075) (cherry picked from commit b850c4b18e2ae2158a97c6cb87bd2057918b8170)
… and schema paths under multiplex
Under `gateway.multiplex_profiles` one gateway process serves every profile
under ~/.hermes/profiles/NAME/; each routed turn runs with a context-local
HERMES_HOME override while `os.environ` still holds the DEFAULT profile's
values. Anything evaluated once at import, or memoised in a single unkeyed
module slot, therefore freezes the LAUNCH profile's value and leaks it into
every other profile's turns. This lands the tools-side half of that class:
- tools/process_registry.py, tools/environments/{modal,singularity}.py:
`_checkpoint_path()` / `_snapshot_store()` resolve `get_hermes_home()` at
call time (same seam as `tools/skills_tool._skills_dir`, so the existing
`monkeypatch.setattr(CHECKPOINT_PATH)` test sites keep working). Completes
the checkpoint_manager / sticker_cache half cherry-picked from #56315.
- plugins/platforms/feishu/feishu_comment_rules.py: `_MtimeCache` is now
path-keyed (accepts a Path or a zero-arg resolver, one (mtime, data) slot
per resolved path) with `invalidate()`; `_rules_file()` / `_pairing_file()`
resolve the routed profile's files. Proposed in #63962.
- tools/tool_output_limits.py, tools/browser_tool.py, tools/browser_camofox.py:
the process-lifetime config caches are dicts keyed by `hermes_home_key()`;
the `_X_resolved` flags and the lifecycle reset keep their shape.
tools/file_tools.py drops its private `file_read_max_chars` memo and reads
the already mtime+path-cached `load_config_readonly()`.
- hermes_time.py: `get_timezone_name()`; when `is_multiplex_active()` the
env `HERMES_TIMEZONE` (bridged from the default profile's config at gateway
startup) is ignored in favour of the routed profile's config.yaml. Both
sandbox TZ sites (code_execution_env/_tool) now use it.
- tools/cronjob_tools.py, tools/tts_tool.py, tools/skill_manager_tool.py:
the static schema text is profile-neutral and `dynamic_schema_overrides=`
rebuilds the `display_hermes_home()` / create-dir hint per
`get_definitions()`, so a routed profile's model sees its own paths.
Refs #95685.
Co-authored-by: Nathan Shan <nathanielcrush51@gmail.com>
(cherry picked from commit 6d3fc6b07b3155c6196b1fd61a829283f1d7855c)
teknium1
force-pushed
the
fix/mux-process-memo
branch
from
September 11, 2026 22:35
7096421 to
e10797d
Compare
This was referenced Sep 11, 2026
teknium1
added a commit
that referenced
this pull request
Sep 12, 2026
… unified init path The unified runtime copied _init_registries_and_clocks and the cron bootstrap from a pre-fix snapshot: HookRegistry() (one process-wide registry, so served secondaries' hooks/ never load and fire the launch profile's handlers) and default_profile="default" with _multiplex_profile_homes (a --profile <name> multiplexer's own jobs treated as a secondary's). Restore ProfileHookRegistries and _cron_tick_profile_homes / runner._primary_profile_name from main (#108453, #108428).
8 of 9 tasks
Open
12 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under
gateway.multiplex_profiles, a secondary profile's turns now run with its ownagent.max_turns, fallback chain, gateway hooks, Nousauth.json, media-delivery policy, checkpoint/snapshot stores, tool limits, browser timeouts, sandbox timezone and tool-schema paths — instead of the launch profile's values frozen into module constants and process caches at import.Changes
Gateway / agent side (
fix(gateway))gateway/run.py::_current_max_iterations— a routed turn (HERMES_HOME override) readsagent.max_turnsfrom its own config instead of the process-wideHERMES_MAX_ITERATIONSbridge filled from_hermes_home.gateway/run_config_loaders.py::_refresh_fallback_model— reads the active gateway home and keeps a last-known-good chain per home (was one runner-wide slot from the launch home);_load_prefill_messagesrelative paths likewise.agent/auxiliary_client.py—_AUTH_JSON_PATHresolved per call viahermes_cli.auth._auth_file_path(); a secondary's compression/title/vision calls stop authenticating with the default profile's Nous token.gateway/hooks.py—HOOKS_DIRresolved per call (salvaged from security(gateway): re-resolve hooks directory per call to fix profile isolation #56508, @srojk34's commit kept) and a newProfileHookRegistriesholds oneHookRegistryper served home, picked from the active scope at emit time. Secondaries'hooks/now fire; the default's handlers no longer see other profiles' messages/responses/user ids.agent/shell_hooks.py::_spawn— child env viabuild_subprocess_env(scrub_secrets=is_multiplex_active()): routedHERMES_HOME, default-profile secrets scrubbed under multiplexing; stdin payload gainsprofile. Single-profile runs keep the process env byte-for-byte.gateway/media_policy.py(+platforms/base.py,media_fetch.py) —media_delivery_strict/allow_dirs/trust_recent*read the routed profile's config under an override; the env bridge remains the contract when no override is active.Tools side (
fix(tools), + cherry-picked #56315 by @srojk34)tools/process_registry.py::_checkpoint_path,tools/checkpoint_manager.py::_resolve_checkpoint_base,gateway/sticker_cache.py::_resolve_cache_path, modal/singularity_snapshot_store()— resolved per call; existingmonkeypatch.setattr(<CONSTANT>)test seams preserved (*_AT_IMPORTpattern fromtools/skills_tool._skills_dir).plugins/platforms/feishu/feishu_comment_rules.py—_MtimeCacheis path-keyed withinvalidate();_rules_file()/_pairing_file()follow the routed profile (second half of fix(feishu): preserve profile scope for SDK-thread callbacks #63962, credit @nateEc).tools/tool_output_limits.py,tools/browser_tool.py,tools/browser_camofox.py— caches keyed byhermes_home_key();tools/file_tools.pydrops its privatefile_read_max_charsmemo for the already path+mtime-cachedload_config_readonly().hermes_time.py::get_timezone_name— under multiplexing the routed profile'stimezonebeats the envHERMES_TIMEZONEbridged from the default; bothexecute_codesandboxTZsites use it.tools/cronjob_tools.py,tools/tts_tool.py,tools/skill_manager_tool.py— static schema text is profile-neutral;dynamic_schema_overridesrebuilds thedisplay_hermes_home()path perget_definitions()([Bug]: Tool schema f-strings call display_hermes_home() at import time, freezing a stale profile path into tool descriptions #95685).Docs:
multi-profile-gateways.md(what follows the routed profile) andhooks.md(profilepayload field).Validation
Live repro (
/tmp/mux_audit/fix-process-memo/repro.py: temp HERMES_HOME A +profiles/Bwith different values everywhere; warm every cache under A, thenset_hermes_home_override(B)):_current_max_iterations()_refresh_fallback_model()A/fallbackB/fallback_read_nous_auth()tokenTOKEN_ATOKEN_Bhook_Ahook_Bprocesses.json/checkpoints// snapshot storesA/…profiles/B/…enabledtool_output.max_bytes/file_read_max_charscommand_timeoutTZ/hermes_timeAmerica/New_YorkAsia/Tokyoprofiles/B21 FAIL → 0 FAIL.
Tests:
tests/gateway/test_multiplex_process_memo_scope.py(4, red on base),tests/agent/test_shell_hooks.py::TestRoutedProfileEnv,tests/tools/test_multiplex_tool_memo_scope.py(3), feishu/timezone invariants, salvagedtests/test_profile_isolation_runtime.pyclasses trimmed to ≤2 each.scripts/run_tests.sh tests/gateway tests/agent tests/tools tests/plugins: all green except pre-existing failures also red onorigin/mainin this environment (test_modal_snapshot_isolation×2 — lazy-install gate,test_web_tools_config×2,test_execution_flag_detection×1) and one unrelated retry-flake (test_compression_stall_fallback_78981).Root cause: the multiplexer runs every profile in one process whose module-level
get_hermes_home()constants,os.environbridges and unkeyed caches were populated once from the launch profile.Audit refs:
/tmp/mux_audit/audit-memoF3, F4, F6 (auth.json half), F7, F9, F10, F12;audit-startupF9.Credits: @srojk34 — earliest fixes for hooks dir (#56508) and checkpoint/sticker paths (#56315), both cherry-picked; @nateEc (Nathan Shan) — Feishu comment-rules half of #63962, co-authored; @rawnets — reporter of #95685.
Fixes #95685. Addresses #56315, #56508, #63962.
Infographic