Cron, kanban, ledger and adapter background paths follow the owning profile under multiplex (#119858, #119859, #119973, #119242; salvage #119974 #120045 #120019 #120009 #112888) - #120103
Merged
Conversation
Contributor
૮ >ﻌ< ა ci reviewran on 58d5ad0 — chore: retrigger CI (zero-job startup failure, merge_train a debug infoCI timingsCI timings · View report · View jobWall time 6m35s vs 5m41s (+15.8%). 7 job(s) slower, 4 faster, 2 unchanged.
|
This was referenced Sep 23, 2026
alt-glitch
added
type/bug
This was referenced Sep 23, 2026
teknium1
force-pushed
the
fix/s5-routed-cron
branch
from
September 23, 2026 13:36
841fb2e to
3764ee5
Compare
teknium1
force-pushed
the
fix/s5-routed-cron
branch
from
September 23, 2026 14:40
3764ee5 to
1690f51
Compare
…nch state.db A multiplexed gateway connects each served profile's adapter inside _profile_runtime_scope(<profile home>). The receive loop an adapter starts while connecting inherits that home override, so every final reply the bot sends is recorded from it. The ledger resolved its path through get_hermes_home(), which follows the override, and the rows landed in profiles/<name>/state.db. The boot sweep (sweep_recoverable) and the boot flood-timer arming (pending_retries) run in the launch context and open the launch state.db, so they never saw those rows. A served bot's reply cut off by a crash or SIGKILL between finalize and platform ACK was never redelivered, a flood-refused reply that spanned a restart was never retried, and resume_pending was not cleared for a session whose answer sat in the ledger. The ledger is meant to be one shared store: the boot sweep already scopes rows by (platform, adapter_profile), and the profile purge terminalizes rows in the shared store. _db_path now resolves from get_process_hermes_home(), as the gateway's other process-level files do (gateway.status). It deliberately skips the get_hermes_home() fallback that lifecycle_ledger uses when HERMES_HOME is unset: a default gateway started in the foreground has no HERMES_HOME, and that fallback would follow the override again. Rows an earlier build already wrote to a profile's state.db stay where they are.
…ted_by when HERMES_PROFILE unpinned Board records (comment author, task creator) were written as the generic "worker" whenever the dispatcher did not pin HERMES_PROFILE, even though the active Hermes profile was resolvable. Add _persisted_identity(): environment (HERMES_PROFILE_NAME / HERMES_PROFILE) first, else the active profile derived from HERMES_HOME via hermes_cli.profiles, else "worker". Wire it into _handle_comment author, _handle_create created_by, and the own-comment skip filter in inject_new_comments_from_env so a worker's own notes never re-enter its live turn as fake operator steering. Identity is never taken from tool args: board records are injected into future workers' prompts, so a caller-supplied author override could forge an authoritative-looking directive (see #19713). Tests: regression tests for env present, env absent with active profile, and no profile; injection echo guard without env profile.
…he launch env One resolver, hermes_cli.profiles.current_profile_name(): the HERMES_HOME override (a multiplexed cron tick or routed gateway turn) names the profile first; the dispatcher's HERMES_PROFILE pin is consulted only when no override is bound; the process home last. #112888 added the HERMES_HOME-derived fallback but kept the env pin FIRST, so under a multiplexer whose launch process carries a HERMES_PROFILE the served profile's writes were still re-labelled with the host's name. The same resolver replaces the per-module copies in hermes_cli/kanban.py, kanban_specify.py, cron/lifecycle_guard.py and the kanban notify-target default. Tests trimmed to two invariants (A->B->A under the override; control pin + generic absence). Closes #119859 Supersedes #112888
… the ticking home Regression guard for #119858. The fix itself is already on main: 3b0fe0c pinned the child HERMES_HOME to the override-aware source home and 786c0e3 moved the spawn onto served_profile_child_env(target_home=home, inherit_credentials=True). No existing test asserted the OWN-profile (no -p) leg A->B->A under multiplex; these two do (red at 786c0e3~1, green on main).
… profiles Every "does this task serve a ROUTED home" decision (serves_routed_profile, _is_process_home, _is_routed_home, env_loader._process_hermes_home) compares the home override with get_process_hermes_home(), which read os.environ["HERMES_HOME"] live. A host that mirrors the served profile into that env var per turn (hermes-webui) made every served profile look like the launch one: MCP registry scope None, bare cross-profile connection names, launch residue kept in served child envs, the launch GATEWAY_ALLOW_ALL_USERS grant seeded into the served scope. set_multiplex_active(True) now pins the launch home (hermes_constants. pin_process_hermes_home; first pin wins, an embedding host may pin explicitly) and get_process_hermes_home() returns the frozen value while multiplex is active. Standalone hermes -p x gateway run (multiplex inactive) keeps following the env. No os.environ fallthrough is added anywhere. Closes #119242
…otice and resumes deferred sessions A multiplexed gateway defers two things for a platform that is offline at boot: the planned-restart online notice (the marker keeps the target owed "for its reconnect", #112109) and restart-interrupted sessions (left resume_pending "for the reconnect watcher"). The owed set spans every served profile's home channel, and a served profile's session never falls back to the default bot, so both wait on that profile's own reconnect. Only the primary reconnect (_install_reconnected_adapter) acted on either. _run_secondary_profile_reconnect published the adapter, redelivered failed obligations and returned: the notice never went out and the marker outlived the outage, and the deferred sessions stayed stranded until the freshness window aged them out. The secondary reconnect now does what the primary does, through one shared helper for the notice replay (still lock-serialized, delivered targets recorded, so a concurrent primary replay cannot double-send).
…on policy Under multiplex (and the Desktop/dashboard backend) every record is formatted on the log QueueListener thread, after the profile scope that produced it is gone. _ProfileRoutingFileHandler still writes a routed profile's records to <profile>/logs/*.log, but RedactingFormatter ran there with no home override, so _redact_enabled() returned the LAUNCH profile's import-time snapshot and the vault scrub keyed on the launch home. A launch profile with security.redact_secrets: false (or HERMES_REDACT_SECRETS=false) therefore wrote every routed profile's credentials raw into that profile's agent.log/errors.log/gateway.log, and a routed profile's own opt-out was ignored. The routing handler now binds the record's stamped home while a record for another profile is formatted. Launch-profile records are untouched. With no live secret scope (that thread has none), _redact_enabled reads the profile's own .env for HERMES_REDACT_SECRETS, as its scope would; without it the first call there cached a config-only answer for the process.
The launch-opt-out case pins the routing handler's home binding; the .env case pins the listener-thread .env read in _redact_enabled and also exercises the binding. The config.yaml case duplicated the binding proof. Salvaged from #120019
A multiplexed API server mirrors every route under /p/<profile>/ and
authenticates each mirror with that profile's key, but it kept one
ResponseStore at the home it was constructed in. Conversation names are
client-chosen strings ("main", "my-project"), so another profile's key
could post `conversation: <name>`, receive that profile's transcript,
instructions and session id as its agent's context, become the
conversation's tip (the owner's next turn replayed the intruder's
messages), and GET or DELETE the owner's responses by id.
The adapter now resolves the store from the request's profile home, as
the SessionDB cache already does: the construction home keeps
self._response_store (and its response_store.db), every other routed
home gets its own <home>/response_store.db, opened on first use and
closed on disconnect. The stream state captures its store when the
request starts, so a snapshot written after the scope ends (disconnect)
still lands in the right one.
Rows a secondary profile wrote into the shared store before this change
stay there, visible to the construction home's profile only; they are
not migrated.
…tated HERMES_HOME `test_language_is_per_profile_under_multiplex` switched profiles by rewriting `os.environ["HERMES_HOME"]` after `set_multiplex_active(True)`. That is the T1 standalone contract (environ IS the profile); a multiplexed turn binds `set_hermes_home_override`, and since the launch home is pinned at activation (#119242) a later env mutation is deliberately ignored. The two assertions are unchanged; only the profile-switch mechanism now matches production.
…lip routed-profile decisions Symptom: a host that serves several profiles from one process and mirrors the active turn's profile into `os.environ["HERMES_HOME"]` for legacy readers (Hermes WebUI does this on every chat turn, next to the context-local override) makes every launch-home decision see the served profile as the launch profile. Two profiles that both configure `atlassian` with different credentials share whichever MCP connection came first: a READ_ONLY_MODE=false profile ends up calling a read-only server (nesquena/hermes-webui#7721). The same misjudgement leaves the launch residue in the served profile's child env, seeds the launch profile's bridged allow-all grant into the served profile's secret scope, and lets the served profile's `terminal.*` config bridge into the shared process env. Cause: four launch-home checks compare the task's override with `get_process_hermes_home()`, which reads `HERMES_HOME` live: `agent.secret_scope.serves_routed_profile` (keys the MCP ledger via `_mcp_registry_scope`, #108352 / #111481, and the check_fn cache, #111151), `agent.secret_scope._is_process_home`, `tools.environments.local._is_routed_home` and `hermes_cli.env_loader._process_hermes_home`. Under the mirror the two sides are equal for every turn. Change: `hermes_constants.pin_process_hermes_home(path | None)` lets the host record the home it serves as its own; `get_routing_process_hermes_home()` returns the pin when set, else `get_process_hermes_home()`; the four checks compare against it. The pin is deliberately NOT folded into `get_process_hermes_home()`: `get_hermes_home()` falls back to it for tasks carrying no override (MCP loop, spawners), and the host's mirror exists precisely so those readers see the served profile. Only "is this task routed / is this the launch home" changes. Unpinned, behaviour is byte-for-byte the old one; hosts that never mutate `HERMES_HOME` need not call it. `activate_multi_profile_hosting()` is not the seam for this: it flips `get_secret` fail-closed process-wide and freezes the launch env, which an embedding host cannot adopt as a bug fix. Tests (2 invariants, parametrized over the four checks plus the MCP ledger key; red on main, green here): pinned + mirrored env -> the served home is routed and the launch home is not, the MCP key is `(home_key, name)`, `get_process_hermes_home()` still follows the env var; never pinned or pinned-then-cleared -> old semantics, including "a mirrored env var IS the launch home". `tests/conftest.py` resets the pin per test so the module-global cannot leak between files. Live repro (WebUI + a stdio FastMCP server named `atlassian` in two profiles, one gated by READ_ONLY_MODE): base -> one ledger key `'atlassian'`, the write profile lists only the read-only tools; fixed -> `(<read_home_key>, 'atlassian')` and `(<write_home_key>, 'atlassian')`, each profile lists its own tools. Docs: `gateway/AGENTS.md` § Profile scope (one launch-home identity) and the isolation table in `website/docs/user-guide/multi-profile-gateways.md`. Also maps the author e-mail under contributors/emails/ (attribution check). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…it under multiplex Builds on tancou's #119129 (cherry-picked above): the pin now lives in get_routing_process_hermes_home() and only the four routed-profile DECISIONS read it. get_process_hermes_home()/get_hermes_home() keep following HERMES_HOME, so an env-only home switch in a multiplexed process resolves as before. - set_multiplex_active(True) pins the launch home only when no host pin exists, and set_multiplex_active(False) releases only the pin it created itself. A transient toggle (gateway_migrate._multiplex_read_mode, cron external-worker restore) no longer drops an embedding host's explicit pin_process_hermes_home(launch). - profiles._cleanup_gateway_service binds set_hermes_home_override(profile_dir) beside the env write. Under the previous head, DELETE /api/profiles/<x> from a multi-profile dashboard resolved get_service_name() against the pinned launch home -> bare `hermes-gateway`, and disabled/stopped/unlinked the HOST multiplexer's unit. Same path serves rename_profile. Tests (red on the previous head): explicit pin survives True->False; env readers follow the env while pinned; two-home delete removes hermes-gateway-victim and leaves hermes-gateway.
teknium1
force-pushed
the
fix/s5-routed-cron
branch
from
September 23, 2026 14:52
8581282 to
58d5ad0
Compare
This was referenced Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under a multiplexed gateway, seven background execution points read state from the LAUNCH profile instead of the profile they act for; after this PR each follows the owning profile.
Closes #119858
Closes #119859
Closes #119973
Closes #119242
Salvages #119129 (@tancou, cherry-picked with authorship)
Supersedes #119867
Supersedes #112888
Supersedes #119974
Supersedes #120045
Supersedes #120019
Supersedes #120009
What changes
deliver: bot-chat(own profile, no-p) — [Bug]: multiplexed crondeliver: bot-chat(own profile, no -p) runs the delivery turn against the gateway default HERMES_HOME #119858. The class fix is already onmain(3b0fe0c pinned the child'sHERMES_HOMEto the override-aware source home; 786c0e3 moved the spawn ontoserved_profile_child_env(target_home=home, inherit_credentials=True)). What was missing was a guard for the own-profile leg: two invariant tests drive_deliver_to_bot_chatA→B→A underset_multiplex_active(True)+ the home override and assert the captured child env (tests/cron/test_bot_chat_own_profile_multiplex_home.py). fix(cron): scope bot-chat delivery to job profile #119867 re-threaded asource_homeparameter throughrun_one_job→_save_compose_deliver→_deliver_to_bot_chat→bot_chat_delivery; that is per-call-site plumbing for a seamget_hermes_home()already resolves correctly, so it is superseded with credit, not cherry-picked.hermes_cli.profiles.current_profile_name(): the boundHERMES_HOMEoverride (multiplexed tick / routed turn) names the profile FIRST; the dispatcher'sHERMES_PROFILE/HERMES_PROFILE_NAMEpin is consulted only when no override is bound; the process home last.tools/kanban_tools.py(comment author,created_by, own-comment skip, notify-target default),hermes_cli/kanban.py::_profile_author,hermes_cli/kanban_specify.py::_profile_authorandcron/lifecycle_guard.py::_current_profile_nameall use it. fix(kanban): persist active-profile identity in comment author / created_by when HERMES_PROFILE unpinned #112888 (maximalang) is cherry-picked for the_persisted_identityseam and the own-comment-skip fix, then re-ordered: its env-first resolution still re-labelled a served profile's writes with the host'sHERMES_PROFILE(the A→B→A test fails on its head with'launch-host' == 'alpha').gateway/delivery_ledger.py::_db_pathopens the LAUNCH home'sstate.db(get_process_hermes_home()), so rows a served profile's receive loop records under its override land where the boot sweep (sweep_recoverable/pending_retries) reads.adapter_profilekeeps the per-bot routing. Chosen over "sweep every served home" because the ledger is a host-process recovery record (one gateway, one boot sweep), matching gateway/AGENTS.md § Profile scope and the Delivery Reliability docs (updated in the same commit)._run_secondary_profile_reconnectnow does what_install_reconnected_adapterdoes: replays the owed planned-restart notice (_schedule_planned_restart_replay, one helper for both callers) and_schedule_resume_pending_sessions(platform=). The notice send and the resume dispatch bind their own scope downstream (_served_home_channel_configs/ per-turn scope), matching the primary path._ProfileRoutingFileHandler.emitbinds the record's stamped home before formatting on the QueueListener thread, soRedactingFormatterapplies the ROUTED profile'ssecurity.redact_secretspolicy and vault, not the launch profile's. Noos.environfallthrough.ResponseStoreper routed home (<home>/response_store.db, cached and closed like theSessionDBcache keyed bystr(home)), so/p/<profile>/v1/responsesconversation names never resolve another profile's transcript.hermes_constants.pin_process_hermes_home()+get_routing_process_hermes_home();serves_routed_profile(),_is_process_home(),_is_routed_home()andenv_loader._process_hermes_home()compare against the routing home, so a host that mirrors the served profile intoos.environ["HERMES_HOME"]per turn (hermes-webui) can no longer re-label the launch home (MCP registry scope stays per-profile; launch residue is stripped from served child envs; the launchGATEWAY_ALLOW_ALL_USERSgrant is not seeded into the served scope).set_multiplex_active(True)auto-pins when no host pin exists andset_multiplex_active(False)releases only that auto-pin.get_process_hermes_home()/get_hermes_home()keep following the env: only the four decisions are frozen, so env-only home switches in the same process keep working. T1 standalone (hermes -p x gateway run) unchanged.Root cause (one sentence)
os.environ, module globals and env-derived defaults hold the launch profile's values; every one of these paths read them where the served profile's ContextVar home override should have been the source.Live A/B (two fake homes under scratch,
set_multiplex_active(True), A→B→A)a27b13056e7)run_one_job,run_jobpatched,_run_bot_chat_turncaptured)HERMES_HOME=own profile, key=sk-<name>, noLAUNCH_ONLY_MARKER, no-p— already correct on main_profile_cron_scope,HERMES_PROFILEunsetworker/worker/workeralpha/beta/alphaprofiles/alpha/state.db,profiles/beta/state.db;boot sweep claimed: []state.db; claimed[('ob-alpha','alpha'), ('ob-beta','beta')]_is_routed_home(served)=False,_is_process_home(served)=True,env_loaderhome = servedTrue/False/ launch_is_routed_home(served)=False,serves_routed_profile()=FalseRed on base → green on head
test_bot_chat_own_profile_multiplex_home.py::test_an_own_profile_bot_chat_turn_runs_in_the_ticking_profiles_home786c0e3f9dc~1(KeyError: 'ANTHROPIC_API_KEY'), PASS ona27b130…::test_an_explicit_target_profile_still_wins_over_the_ticking_profile786c0e3f9dc~1, PASS ona27b130test_kanban_persisted_identity.py::test_a_served_profiles_tick_authors_board_records_as_that_profile'launch-host' == 'alpha'; also FAIL on #112888's head)…::test_a_dispatched_worker_keeps_its_pinned_identity_and_an_unnamed_caller_stays_generictest_delivery_ledger_process_home.py::test_boot_sweep_redelivers_a_reply_recorded_under_a_served_profile_scope[hermes-home-env|platform-default]test_planned_restart_notice_multiplex.py::test_a_served_profiles_reconnect_replays_the_owed_noticetest_restart_resume_pending.py::test_served_profile_reconnect_resumes_what_boot_deferredtest_hermes_logging.py::test_routed_records_follow_their_own_profiles_redaction_policy[launch-opt-out|routed-env-opt-out]test_api_server_response_store_profile_scope.py::test_a_profile_key_never_reaches_another_profiles_responses[json|sse]test_secret_scope_pinned_launch_home.py::test_a_per_turn_env_mirror_cannot_relabel_the_launch_home_under_multiplexserved == launch)…::test_a_standalone_profile_process_keeps_following_its_envExisting tests:
tests/tools/test_kanban_persisted_identity.py(added by #112888, never on main) was rewritten to two invariants;tests/test_hermes_logging.pyparametrize trimmed 3→2 cases (routed-config-opt-outredundant with the binding proof). No assertion of a pre-existing main test was changed.tests/conftest.pyresets the new_PINNED_PROCESS_HOMElatch per test next to_MULTIPLEX_ACTIVE.Not covered
HERMES_HOMEswitchers that still run in-process without an override:hermes_cli/gateway.py::_sync_hermes_home_from_systemd_unit(sudo--systemadoption; single-profile CLI process) andhermes_cli/update_cmd_fleet.pyrestore aroundrefresh_systemd_unit_if_needed(same). Both are safe because the pin only affects routing decisions, notget_hermes_home();profiles._cleanup_gateway_service(reachable from a multiplexed dashboard) binds the override explicitly.deliver: bot-chat(own profile, no -p) runs the delivery turn against the gateway default HERMES_HOME #119858's issue text points at the pinned releasev2026.8.31; the fix landed on main before this PR (3b0fe0c, 786c0e3). This PR adds the missing invariant tests; the issue closes as implemented-on-main with those SHAs.Infographic
Co-authored-by: John Paul Soliva soliva.johnpaul@icloud.com
Co-authored-by: maximalang maximal04040404@gmail.com
Existing test changed (justification)
tests/agent/test_i18n.py::test_language_is_per_profile_under_multiplexswitched profiles by rewritingos.environ["HERMES_HOME"]afterset_multiplex_active(True). That is the standalone (T1) contract; a multiplexed turn bindsset_hermes_home_override, and with the launch home pinned at activation (#119242) a later env mutation is ignored on purpose. Both assertions (frthende) are unchanged — only the profile-switch mechanism now matches production (CI red on the first push was exactly this).Review follow-up
Independent review of the first head (
841fb2ee7d76) returned one blocker and two majors; all three fixed on head3764ee576c8ebdefb969b2ccf1029a7f701f294a, red-on-head tests for each. Suitestests/hermes_cli tests/agent tests/gateway: 29405 passed, 4 failed — each red identically on a pristine origin/main worktree (checkout-identityexternalrows, a load TimeoutError,test_run_agent.pyat the 300 s per-file cap under load 80; 267/267 alone).841fb2ee)profiles._cleanup_gateway_service/rename_profileswitch home env-only; with the launch home pinned intoget_process_hermes_home(),DELETE /api/profiles/<x>from a multi-profile dashboard resolvedget_service_name()to barehermes-gatewayand disabled/stopped/unlinked the HOST multiplexer's unitget_process_hermes_home()intoget_routing_process_hermes_home()(#119129's design), and_cleanup_gateway_servicealso bindsset_hermes_home_override(profile_dir)beside the env write (thegateway_migrate._home_envshape).rename_profilecalls the same helper.tests/hermes_cli/test_profile_delete_service_name.py— pinned launch A, delete B:hermes-gateway-victimdisabled+unlinked,hermes-gateway.serviceuntouchedset_multiplex_active(False)unpinned unconditionally, dropping an embedding host's explicitpin_process_hermes_home(launch)on transient toggles (_multiplex_read_mode, cron worker restore)_AUTO_PINNED_HOME); deactivation releases only that auto-pintest_secret_scope_pinned_launch_home.py::test_an_explicit_host_pin_survives_a_transient_multiplex_togglefix(profiles): pin the launch home…, includes his 10-case parametrized test, docs,contributors/emails); this PR's own pin test rewritten on top. Body:Salvages #119129; comment posted on #119129tests/agent/test_serves_routed_profile_pin.py(10 cases, from #119129)HERMES_HOMEswitchers not enumeratedprofiles.pygrowth on the facade_cleanup_gateway_service; no new behaviour appended to the facadeget_hermes_home()now follows the env under multiplex again (the pin is routing-only), so the reviewer's minor about override-less readers ignoring env mutation no longer applies;test_multiplex_activation_pins_the_launch_home_but_env_readers_still_follow_the_envasserts it.