Served-profile children and background threads run with their own profile's env/scope; kanban never kills a recycled PID - #111617
Conversation
… profile's env
Under gateway.multiplex_profiles (and the Desktop/dashboard backend serving named
profiles) os.environ holds the LAUNCH profile's .env. Five spawn sites built a
child's env from it while acting for another profile, so the child saw the
launch profile's HERMES_HOME (bot_relay, key_cmd), its credentials, HERMES_MODEL
and TERMINAL_* policy, and none of the served profile's own .env:
- tui_gateway/server.py _SlashWorker: pinned HERMES_HOME but kept the launch
base with tier-2 credentials + settings.
- tools/bot_relay.py delivery_env (relay RPC + --run-delivery): dict(os.environ).
- tools/browser_tool.py _build_browser_env: re-added BROWSERBASE/FIRECRAWL/
BROWSER_USE keys from os.environ after the scrub.
- plugins/platforms/a2a/adapter.py _forward_to_profile: {**os.environ}.
- agent/command_token_source.py _mint: key_cmd helper inherited os.environ.
tools.environments.local.served_profile_child_env is the one builder: pin the
target home, drop the launch profile's .env residue and bridged TERMINAL_*
(strip_launch_profile_env), and for children that legitimately run with the
profile's credentials (agent worker, token helper) overlay the target profile's
own secrets - what a standalone `hermes -p X` loads itself, never a sibling's.
The browser keeps the provider scrub and re-adds only its passthrough keys via
get_secret. Outside multiplex the env is unchanged.
Live proof from inside the child (launch A, served B, multiplex on): all five
children print HERMES_HOME == B, see B_MARKER=b from B's .env and do not see
A_MARKER; the browser child gets B's FIRECRAWL_API_KEY. On base every one leaked
A_MARKER and lacked B_MARKER; bot_relay and key_cmd also had A's HERMES_HOME.
…s profile scope The profile scope (HERMES_HOME override, secret scope, terminal policy) is a contextvar bundle bound per turn. A bare threading.Thread / Timer / gRPC callback starts with an empty context and resolves the LAUNCH profile: - agent/title_generator.py: the auto-title thread read auxiliary.title_generation (model, language, provider key) from the default profile's config and billed the default's key for a secondary's session. Spawn via agent.memory_provider.spawn_context_thread (copy_context). - tui_gateway/session_lifecycle.py: every teardown caller is a bare Timer (ws-orphan reap), the idle-reaper thread, atexit _shutdown_sessions, the session.close pool RPC, superseded_by_resume or compute_host flush - none carries a scope, yet on_session_end / commit_memory_session / agent.close -> shutdown_memory_provider read the provider's config + credentials at call time. Under multiplex they failed closed (tail never committed, #110622 class); on the Desktop backend a secondary's transcript went to the launch profile's memory tenant. _finalize_session and _teardown_session now bind _session_profile_runtime_scope(session) around those blocks, which covers every spawn site through the single chokepoint. - plugins/platforms/google_chat/adapter.py: Pub/Sub callbacks run on the gRPC SubscriberClient's threads and run_coroutine_threadsafe copies THAT empty context onto the loop task, so _dispatch_message and everything under it (attachment cache, per-user OAuth token store via _acquire_user_chat_api -> _load_per_user_chat_api, TTS keys, delivery ledger, bot-id cache) resolved the launch profile. connect() captures its scope; _on_pubsub_message and _submit_on_loop run under a per-callback copy of it. spawn_context_thread gains a kwargs passthrough for the title thread's callbacks.
…ingerprint tasks.worker_pid outlives a reboot; afterwards the number can belong to any process. _pid_alive answered from bare existence, so reclaim_stale_claims kept extending the claim of a "live" stranger (stuck running task), and enforce_max_runtime / _terminate_reclaimed_worker SIGTERM'd then SIGKILL'd it. _set_worker_pid now records gateway.status.get_process_start_time(pid) as tasks.worker_started_at (additive column, NULL on legacy rows). _worker_alive (pid, started_at) is the liveness check every reader uses (reclaim, defer, reconcile, crash sweep, max-runtime, archive, reopen invalidation); a live pid whose fingerprint disagrees is a recycled PID: treated as dead, never signalled (termination reports pid_recycled). Legacy rows without a fingerprint keep the existence answer until their next spawn. Row hermes_cli/kanban_db_dispatch.py:458 (lane4_high) confirmed by tracing: the SIGKILL at :461 was gated only on _pid_alive.
૮ >ﻌ< ა ci reviewran on 8317ddb — fix(kanban): worker liveness and kills require the spawn-tim
|
alt-glitch
added
type/bug
andrexibiza
left a comment
andrexibiza
left a comment
There was a problem hiding this comment.
Reviewed exact head 8317ddbcbfb54d725bccde58ba41a39348b288e7 against current main 4d55ca91656ac5f83e1506679b7f81e0238e5e16 and the current synthetic merge result. The branch is 3 commits ahead / 95 behind its actual merge base (f5a457ad5bebd9d78bbf35ffaaf1c33866a03ca6); the only current-main overlap in the changed-file set is tools/environments/local.py, so I inspected the composed result rather than treating the PR head in isolation.
The central direction is right: one served-profile child-env builder, context-preserving background work, teardown scoped at the lifecycle chokepoint, and PID identity carried with Kanban ownership are much better shapes than more call-site predicates. I found four remaining P1 authority failures before this is safe to land.
P1 — routed child env still retains launch-only credentials that did not come from .env / a recorded source
served_profile_child_env(..., inherit_credentials=True) starts from hermes_subprocess_env(inherit_credentials=True). Under multiplex that still contains profile credentials injected directly into the launch process by systemd/Compose/the shell. strip_launch_profile_env() removes launch dotenv/source provenance, but it cannot remove a launch-only credential that was never recorded in those sets. If profile B does not define the same name, the A credential remains in the child after B's scope is overlaid.
That is the inverse of get_secret()'s multiplex contract: a scoped miss must not fall back to ambient process credentials. The child boundary needs the same semantics — start a routed credential-bearing child from a credential-scrubbed/non-profile base and then overlay B's authoritative scope (including managed policy), rather than retaining unknown ambient credentials first. I left the exact counterexample inline.
P1 — Desktop/dashboard served profiles still fall through the process-global multiplex predicate
The new helper delegates cleanup to strip_launch_profile_env(), which is deliberately a no-op unless is_multiplex_active() is true. That is correct for its original cron contract, but not for this PR's claimed Desktop/backend coverage: tui_gateway._session_profile_runtime_scope() installs the profile home + secret scope without turning on process-global multiplex semantics. A routed B child in that backend can therefore keep A's dotenv/settings, and _build_browser_env() can additionally resolve an absent B passthrough key through ambient A because get_secret() falls through when multiplex is inactive.
Open #111481 is complementary, not a duplicate: it is addressing the same “served routed profile even when the process flag is off” identity boundary for MCP and now has a serves_routed_profile()-style predicate. This PR needs equivalent routed-home authority for child env construction rather than keying isolation solely to the gateway-wide multiplex flag.
P1 — the persisted Kanban fingerprint is not reboot-stable on Linux
The stated threat is specifically a DB row surviving reboot. On Linux, gateway.status.get_process_start_time() preferentially stores /proc/<pid>/stat field 22, which is clock ticks since this boot. It resets when the machine boots. Persisting only (pid, start_ticks) therefore does not prove incarnation across a reboot: an unrelated process can later receive the same PID and the same boot-relative start tick and pass _start_times_agree().
The repo already has the missing shape in gateway/drain_control.current_instantiation_epoch(), which composes Linux boot_id with PID-1 start specifically because durable state can survive machine/container restart. Kanban needs a restart-stable incarnation witness as part of the persisted worker identity (or an absolute creation-time witness with equivalent semantics), and the hostile test should hold PID/start-time constant while changing the boot/instantiation identity and prove that no signal is sent.
P1 — a failed fingerprint capture creates a brand-new “legacy” kill-authority row
get_process_start_time() returns Optional[int]. _set_worker_pid() persists None when the lookup fails, while _pid_recycled() treats started_at is None as “legacy row, never recycled” and falls back to bare PID existence. So a transient lookup failure on a new worker silently recreates the exact unsafe authority this change is meant to remove.
That conflicts with the repository-wide destructive-process contract already landed through #99558 / #89614: missing or unavailable process identity is refusal, not permission to signal. Legacy migrated rows may need compatibility behavior, but a post-migration spawn whose fingerprint could not be captured must be distinguishable from that legacy state and must never authorize TERM/KILL. I left an inline witness for the None path.
Class closure / “other side of the shape”
The two sweeps are useful, but they also mean the publication claim is currently too broad. The PR body itself records 15 child-process residuals and 29 thread/timer residuals in the same defect class while the title/body say a served profile gets its own env in every child and its own scope on every background thread. Several residuals are not hygiene-only: Singularity child execution, skills-hub/GitHub auth fallback, Google Meet, Photon, inline skill shell expansion, credits/review workers, TTS, late MCP refresh, and process-registry reapers can cross a profile authority boundary.
I do not think every low-risk residual has to be stuffed into this carrier. I do think the security/identity-bearing residuals need explicit owned follow-up carriers/interlocks before this can be described as class closure; otherwise narrow the publication contract to the seams actually closed here. #109417 is the natural campaign-level tracker.
Topology / provenance
- #111187 is merged complementary foundation, not duplicate work: it owns routed cron-fire multiplex semantics, launch-residue tracking, managed-env precedence, and the current
strip_launch_profile_env()behavior that this PR composes with. It also explains why the current-main overlap inlocal.pymatters. - #111082 is merged complementary scope work: gateway shutdown/session-end and cron cleanup acquire the owning profile scope; this PR handles distinct TUI/Desktop lifecycle and other background seams.
- #111481 is open complementary routed-profile identity work for the Desktop/dashboard surface where the global multiplex flag is false. Its home-routing predicate is directly relevant to the second blocker above.
- #99558 / #89614 are the merged destructive-PID authority class. The Kanban addition extends that class to persisted task workers, but must preserve its fail-closed rule for missing identity.
- I found no competing PR carrying this exact combination of child-env + TUI/background + Kanban-worker changes. The three surviving commits are all authored by the current PR author; there is no contributor salvage/history collision to reconcile in this carrier.
Verification state
At the exact reviewed head, Docker 34938453586 and Nix 34938453614 are green; CI 34938454224 is still queued after reruns. More importantly, the first two surviving commits (7b8a44e53fcb5f4a7dfe70dc58ba4e8b148362f5, 13377543c1dafef36a41f35128a5fc85846de46f) have no workflow runs attached, so this is not every-commit proven green even if the head CI finishes successfully.
Required inverse witnesses before merge:
- launch A has an ambient-only credential (not in A
.envor source provenance), B omits it, multiplex on → B child never receives A's value; - Desktop/dashboard serves B with the multiplex flag off and
target_home != process_home→ no A dotenv/settings/credential fallback in slash worker/browser/helper children; - same PID + same Linux start tick but different boot/instantiation identity → worker is foreign and never signalled;
- fingerprint capture returns
Nonefor a newly spawned worker → no later reclaim/timeout path may signal that PID; - recompose on current main and establish exact-head and every-surviving-commit green evidence.
Once those authority boundaries are closed, the core refactor is a good reusable seam for the rest of the sweep.
| snapshot.""" | ||
| from agent.secret_scope import build_profile_secret_scope, current_secret_scope | ||
| from hermes_constants import get_hermes_home_override | ||
| env = dict(base) if base is not None else hermes_subprocess_env(inherit_credentials=inherit_credentials) |
There was a problem hiding this comment.
P1 — ambient-only launch credentials can survive into the served profile. inherit_credentials=True starts from the launch process's provider/tool credentials, and strip_launch_profile_env() only knows names with dotenv/source provenance. A key injected by systemd/Compose/the shell (for example ambient OPENAI_API_KEY=A that is not in A's .env or recorded source names) survives here; if B does not define that key, the later B-scope overlay never removes it. Under multiplex, get_secret() deliberately treats that exact scoped miss as no credential, not ambient fallback. Build routed credential-bearing children from a credential-scrubbed base and then overlay the target profile's authoritative scope, or otherwise prove every non-global credential name is removed before the overlay. Please add the inverse test with an ambient-only A key and B missing it.
There was a problem hiding this comment.
Fixed on head 42e01d9 (#112685, daa084775245a): a routed target scrubs every Tier-1/2 credential from the base regardless of provenance before B's scope overlay — test_served_profile_child_env_authority.py::test_ambient_only_launch_credential_never_reaches_a_routed_child (ambient-only OPENAI_API_KEY on A, B lacks it, mux on; red on base).
| target = str(target_home or get_hermes_home_override() or "") | ||
| if target: | ||
| env["HERMES_HOME"] = target | ||
| strip_launch_profile_env(env, target) |
There was a problem hiding this comment.
P1 — this predicate misses the Desktop/dashboard served-profile case. strip_launch_profile_env() is intentionally a no-op when is_multiplex_active() is false, but the TUI/Desktop backend serves named profiles by installing HERMES_HOME/secret scope without enabling the gateway-wide multiplex flag. In that topology a B slash/helper child reaches this line with target != process_home, the strip does nothing, and A's dotenv/settings remain; browser passthrough also has an ambient get_secret() fallback on a B miss. The authority test needs to be “this task serves a routed home,” not only “the whole gateway is multiplexing.” #111481 is complementary work on exactly that routed-home predicate for MCP. Add a mux-flag-off B witness here.
There was a problem hiding this comment.
Fixed on head 42e01d9 (#112685, daa084775245a): the strip/scrub now keys on target != process home, not is_multiplex_active(); the browser passthrough resolves from the bound scope only under serves_routed_profile() — …::test_routed_home_with_multiplex_flag_off_gets_no_launch_residue (flag-off B witness; red on base).
| carrying them. The fingerprint is what lets every later liveness/kill decision tell OUR worker | ||
| from a process that recycled the PID after a reboot.""" | ||
| from gateway.status import get_process_start_time | ||
| started_at = get_process_start_time(int(pid)) |
There was a problem hiding this comment.
P1 — this persisted fingerprint is not reboot-stable on Linux, so it does not prove the threat model in the docstring. gateway.status.get_process_start_time() first returns /proc/<pid>/stat field 22 on Linux: boot-relative clock ticks. The DB survives reboot; that counter does not. A later boot can therefore produce an unrelated process with the same PID and the same boot-relative tick value, and _start_times_agree() accepts it. Persist an incarnation witness with the process start (for example the existing gateway.drain_control.current_instantiation_epoch() / Linux boot_id, or an equivalent restart-stable creation-time identity), then require both to match before liveness extends a claim or any signal is authorized. Hostile test: same PID + same start tick, different boot identity => zero signals.
There was a problem hiding this comment.
Fixed on head 42e01d9 (#112685, 9cb9fd0d0cb9c): fingerprint is now "<current_instantiation_epoch()>|<start>" (boot_id + PID-1 start), both halves must match — test_kanban_worker_pid_fingerprint.py::test_same_pid_and_start_tick_on_another_boot_is_foreign (same PID + same tick, other boot id → zero signals; red on base). Live: real sleeper child left untouched on the reboot-shaped row, SIGTERM'd on the matching one.
| def _pid_recycled(pid: Optional[int], started_at: Optional[int]) -> bool: | ||
| """True when a live ``pid`` is NOT the process fingerprinted at spawn (or the fingerprint can no | ||
| longer be read). Signalling it would hit a stranger. ``None`` fingerprint = legacy row, never recycled.""" | ||
| if started_at is None or not pid: |
There was a problem hiding this comment.
P1 — None is safe only for truly legacy rows, but _set_worker_pid() can create it today. get_process_start_time() is optional; if a new spawn's lookup fails, this PR persists worker_started_at=NULL, then this branch deliberately treats the live PID as our worker and the timeout/reclaim paths can signal it by bare number. That recreates the #89614/#99558 authority failure for a post-migration row. Preserve legacy compatibility separately, but a new unverified spawn must fail closed for destructive signaling. Add a test that forces fingerprint capture to None, then presents a live/reused PID and proves TERM/KILL is never issued.
There was a problem hiding this comment.
Fixed on head 42e01d9 (#112685, 9cb9fd0d0cb9c): a failed capture persists 'unverified', never NULL — held while live, never signalled by timeout/stale-claim/manual reclaim/archive/reaper, reclaimed once gone; NULL stays legacy-only — …::test_unverified_fingerprint_capture_never_authorizes_a_signal (red on base).
kvnloo
left a comment
kvnloo
left a comment
There was a problem hiding this comment.
Review — PR #111617 (teknium1)
"Served-profile children and background threads run with their own profile's env/scope; kanban never kills a recycled PID"
Verdict: COMMENT, concurring with andrexibiza's blockers. Read his review first — not repeating his four P1s (ambient-credential retention, multiplex-inactive fallthrough, reboot-unstable fingerprint, None-fingerprint legacy authority). Verified the mechanics independently against base; the new angles below are all his-review-adjacent gaps, not duplicates.
What I verified as sound
- Fingerprint plumbing is consistent across every reclaim path:
release_stale_claims,reclaim_task,detect_stale_running,reconcile_orphaned_running,_reclaim_dead_workers,enforce_max_runtime,archive_task,_set_status_direct,invalidate_descendants_for_parent_reopenall threadworker_started_atthrough. No path still kills on bare PID existence. _start_times_agreeis effectively exact equality on integer clock ticks (abs(r - cur) <= 0.001) — no tolerance slop that could merge two incarnations.build_profile_secret_scopecomposes.env+ external sources + managed env last, so the target overlay inserved_profile_child_envis complete when a target is bound.- google_chat: per-callback
ctx.copy()is the right call (aContextcan't be entered concurrently), andctx_boundcopies at spawn time — the auto-title thread now genuinely carries the turn's scope instead of an empty context. - Gateway multiplex connects adapters inside
_profile_runtime_scope(run_adapters.py:1016), so the_scope_ctxcapture inconnect()is correct on that path.
New finding 1 (medium): _mint fails open to launch credentials when no scope is bound
served_profile_child_env(inherit_credentials=True) with no target_home resolves the target from the ambient get_hermes_home_override() (tools/environments/local.py). When nothing is bound — which is exactly the background-thread situation this PR is fixing elsewhere — target is "", the overlay degrades to current_secret_scope() or {}, and the base is hermes_subprocess_env(inherit_credentials=True), i.e. the launch environ's provider credentials (os.environ.copy() minus Tier-1).
_mint (agent/command_token_source.py) calls it with no target_home, and CommandTokenSource.__call__ runs on whatever thread first needs the token — TTL refresh can fire minutes later on an arbitrary worker thread. An unscoped refresh mints the token with the launch profile's credentials while it's cached as the served profile's provider key: the op read / vault kv get helper signs in as the wrong profile, silently.
The docstring promises "never the multiplexer's launch environ", but unlike get_secret — which fails closed with UnscopedSecretError under multiplex — this helper fails open. Suggest: when multiplex is active and neither an override nor a scope is bound, raise instead of minting; or require target_home for credential-bearing children so the safe usage is the only usage.
New finding 2 (P2/nit): the fingerprint check and the signal are still check-then-act
_terminate_reclaimed_worker (kanban_db_dispatch.py) does _pid_recycled(pid, started_at) → then kill(pid, SIGTERM) as separate syscalls; enforce_max_runtime has the same shape. A recycle landing in that window still signals a stranger. The window is tiny and unexploitable in practice, but the PR title claims "never kills a recycled PID" — strictly, the guarantee is "never kills a detected recycled PID". Linux has the atomic primitive for this: pidfd_open + pidfd_send_signal. Worth naming as the residual even if it's accepted as-is.
Nit: archive_task clears worker_pid but not worker_started_at
The UPDATE nulls worker_pid/claim_lock/claim_expires but leaves the stale fingerprint on the row. Harmless today (NULL pid → _pid_alive false; every pid set goes through _set_worker_pid), but the column is part of the kill-authority tuple now — clear it with the rest so no future path can read a fingerprint that outlived its pid.
Question (not a finding): _scope_ctx on the TUI/Desktop backend
Verified the gateway multiplex path connects inside the profile scope, but the google_chat change is shared with the TUI/Desktop backend. If adapters there connect at startup under the launch scope and later serve a named profile, _scope_ctx permanently captures launch and every Pub/Sub callback resolves the wrong profile — the exact bug, persisting. The None fallback (contextvars.copy_context() on the gRPC thread = empty context) degrades the same way, silently. Is connect() guaranteed to run under the served profile's scope on every backend, or should the capture assert/fail-closed when it isn't?
Note extending andrexibiza's P1 #1 (not a new P1)
delivery_env (tools/bot_relay.py) calls served_profile_child_env(base=os.environ, ...) — the base replaces the hermes_subprocess_env snapshot, so this call site is the raw multiplexer environ minus strip_launch_profile_env (a no-op when multiplex is inactive), with target secrets overlaid. It's the strongest concrete instance of the ambient-credential retention he flagged, and the docstring's "never the multiplexer's raw os.environ" overclaims for exactly this call.
|
Follow-up for the four P1s: #112685 (head
Also: |
|
Follow-up: #112685 (head
|
…, with or without the multiplex flag Two authority gaps in served_profile_child_env (#111617 review, andrexibiza P1 #1/#2, kvnloo finding 1): - The base was hermes_subprocess_env(inherit_credentials=True) = the launch environ's provider credentials; strip_launch_profile_env only knows names with .env/source provenance, so a key systemd/Compose/the shell injected into the launch process survived into profile B's child whenever B did not define the same name. Now a ROUTED target scrubs every Tier-1/Tier-2 credential from the base regardless of provenance before B's own scope is overlaid (the child boundary gets get_secret's contract: a scoped miss is no credential, never ambient fallback). The launch profile's own child keeps its env. bot_relay's base=os.environ goes through the same scrub. - strip_launch_profile_env / the scrub keyed on is_multiplex_active(); the Desktop and dashboard backends serve ?profile=B by installing the HERMES_HOME override without that flag, so B's slash worker / helper children kept A's .env and settings. The authority test is now "is the target a routed home" (target != process home). - _build_browser_env resolved the passthrough keys via get_secret, which falls through to os.environ on a scoped miss while multiplexing is inactive: a routed B with no Firecrawl key got A's. Under serves_routed_profile() the bound scope is the only source. - served_profile_child_env(inherit_credentials=True) with no target and no scope bound under multiplex minted with the launch credentials (key_cmd TTL refresh on a worker thread); it now raises UnscopedSecretError like get_secret. tests/tui_gateway/test_served_profile_child_env_authority.py: ambient-only A key + B missing it (mux on), flag-off routed B (helper child + browser), real child observation. 3/3 red on base.
… fingerprint never authorizes a signal #111617 review (andrexibiza P1 #3/#4, kvnloo nit): - worker_started_at persisted only gateway.status.get_process_start_time(): on Linux that is /proc/<pid>/stat field 22, clock ticks since THIS boot. The threat is a row surviving a reboot, and that counter does not, so an unrelated process on a later boot with the same PID and the same tick value passed _start_times_agree(). The fingerprint is now "<gateway.drain_control.current_instantiation_epoch()>|<start>" (boot_id + PID-1 start, the witness the drain marker already uses); both halves must match. Integer values on rows written before this change keep the start-time-only comparison. - A failed capture persisted NULL, which _pid_recycled treats as the legacy pre-fingerprint row and falls back to bare PID existence - a new spawn silently recreated the #89614/ #99558 kill authority. A failed capture now persists UNVERIFIED_WORKER_FINGERPRINT: the claim is held while the PID is live (never released beside it, never SIGTERM/SIGKILLed by timeout, stale-claim, manual reclaim, archive or the terminal reaper) and reclaimed once it is gone. NULL stays legacy-only. - Every tasks UPDATE that nulls worker_pid nulls worker_started_at too (archive_task and the reclaim/timeout/reopen paths): the fingerprint is part of the kill-authority tuple and must not outlive its pid. Live (real sleeper child): reboot-shaped row (same pid, same tick, other boot id) -> reclaimed to ready, child untouched; matching fingerprint -> SIGTERM delivered, exit -15. tests/hermes_cli/test_kanban_worker_pid_fingerprint.py: +2 hostile tests, both red on base. Not changed: the check-then-act window between _pid_recycled and kill (kvnloo P2) is real but needs pidfd_open/pidfd_send_signal (Linux 5.3+) to close atomically; left as the documented residual of "never kills a DETECTED recycled PID".
…nv restart requirement kvnloo (#111620 review) asked for the operator-visible statement that once a serve / dashboard process hosts a second profile, the launch profile's env-only credentials are frozen at activation and a rotation in the process env needs a restart. Also states the routed-child rule with and without the multiplex flag (#111617). Adds encoding='utf-8' to the probe child in the child-env authority test (windows-footguns lint).
4 participants
A profile served by a multiplexed gateway / Desktop backend now gets its own env in every child process it spawns and its own scope on every background thread that works for it; the kanban dispatcher never treats a recycled PID as its worker.
Symptom → change → behaviour
Root cause (one sentence): under
gateway.multiplex_profilesos.environholds the LAUNCH profile's.env, and five spawn sites built a child env from it while three background paths started bare threads with an empty contextvars context — so profile B's children and workers ran with profile A's home, keys and settings._SlashWorker(tui_gateway/server.py) for a B sessionHERMES_MODEL,TERMINAL_*as fallbacktools/bot_relay.py::delivery_env, RPC +--run-delivery)dict(os.environ): A's HERMES_HOME, keys, settingsserved_profile_child_env)tools/browser_tool.py::_build_browser_env)os.environ(A's)get_secret→ B's or none_forward_to_profilehermes chatchild{**os.environ}+ pinned homekey_cmdtoken helper (agent/command_token_source.py::_mint)agent/title_generator.py)auxiliary.title_generationconfig, language and API key for B's sessionspawn_context_thread— B's config/keytui_gateway/session_lifecycle.py_finalize_session/_teardown_session) from ws-orphan Timer, idle reaper, atexit,session.close,superseded_by_resume, compute_host flushon_session_end/commit_memory_session/agent.close()ran unscoped: fail-closed under multiplex (tail never committed), launch tenant on the Desktop backend_session_profile_runtime_scope(session)at the single chokepoint — covers every spawn site, incl. the one-linesession.closeRPC inmethods_session.py(no change needed there)_on_pubsub_message,_submit_on_loop)_dispatch_message, attachment cache, per-user OAuth token store (send→_send_file→_acquire_user_chat_api→_load_per_user_chat_api), TTS key, delivery ledger, bot-id cache resolved Aconnect()captures its scope; each callback and each loop hand-off runs under a copy of ithermes_cli/kanban_db_dispatch.py)_pid_alive(pid)gated claim-extend and SIGTERM/SIGKILL → after a reboot a recycled PID kept a task stuckrunningor got killedworker_started_atfingerprint (gateway.status.get_process_start_time) recorded at spawn;_worker_alive(pid, started_at)everywhere; mismatched pid → dead, never signalled (pid_recycledin the termination record); legacy NULL rows keep the old answer until their next spawnChanges
tools/environments/local.py::served_profile_child_env— the one builder the five spawn sites use: pin target home,strip_launch_profile_env, and for children that run with the profile's credentials (agent worker, token helper, relay/A2A turn) overlay the target profile's own secrets (what a standalonehermes -p Xloads itself). Browser keeps the provider scrub.agent/memory_provider.py::spawn_context_threadgainskwargs=.hermes_cli/kanban_db_connect.py: additivetasks.worker_started_atcolumn;invalidate_descendants_for_parent_reopenterminations tuples carry it; dashboard plugin updated.Live repro — child env, from INSIDE the child (temp HOME, launch A + served B, multiplex on)
A's
.env:A_MARKER=a HERMES_MODEL=a-model TERMINAL_ENV=docker FIRECRAWL_API_KEY=a-firecrawl; B's.env:B_MARKER=b FIRECRAWL_API_KEY=b-firecrawl. Each child is apython -cprobe printing its own environ.Base (
f5a457ad5beb) — every child leaks A, none sees B; relay + key_cmd even get A's HERMES_HOME:Head — B's home, B's marker, no A residue; browser gets B's Firecrawl key:
Live repro — threads (same two homes; A
title_generation.language=en, Bzh;TITLE_KEYa-key/b-key)(teardown = real
_schedule_ws_orphan_reapTimer →_teardown_popped_session; title = realmaybe_auto_title.)Tests (6, all red on base by source swap, green on head)
tests/tui_gateway/test_served_profile_child_env.py— real_SlashWorkerspawn observed from the child;key_cmdhelper + browser env under_profile_runtime_scope(B).tests/tui_gateway/test_background_thread_profile_scope.py—maybe_auto_titlethread; ws-orphan reap Timer → teardown under B.tests/hermes_cli/test_kanban_worker_pid_fingerprint.py— recycled PID reclaimed without a signal (max-runtime + stale-claim paths); matching fingerprint extends the claim and is signalled.scripts/run_tests.sh tests/tools tests/agent tests/tui_gateway tests/plugins tests/hermes_cli/test_kanban*.py: 1776 files, 23.4k tests; remaining reds are pre-existing env-reds identical on base (test_hindsight_provider×8 = missinghindsight_client_apimodule,test_delegate_timeout_cleanup,test_execution_flag_detection).Sweep 1 — child-process spawn sites (
rg subprocess.(Popen|run|check_output|call)|create_subprocess_(exec|shell)|os.environ.copy()|env=os.environ, 736 hits)Totals: FIXED 4 hits (the 5 sites;
_mint's Popen is one rg hit, bot_relay's builder has no spawn of its own) · builder-routed 72 · cannot run for a served profile 578 (standalone CLI/installer/scripts/evals, or the child reads nothing profile-dependent) · tests-tree 67 · residual 15 (below). Full 736-row table: https://gist.github.com/teknium1/72c7296b2df742448e15cd11ecc11227FIXED / residual / builder-routed rows:
agent/command_token_source.pyplugins/platforms/a2a/adapter.pyhermes chatspawn — listed FIXED sitetui_gateway/methods_bot_relay.pytui_gateway/server.pyagent/skill_preprocessing.pycmdin skill: env=delegated_child_subprocess_env() -> None (inherits os.environ); 69 commenthermes_cli/git_credentials.pyhermes_cli/goals.pyhermes_cli/kanban_pr_acceptance.pyhermes_cli/web_routers/profiles.py<name> setupinherits backend os.environ (launch .env); no env kwarghermes_cli/worktree_ops.pygh pr listinherits os.environ (GH_TOKEN/GITHUB_TOKEN) from cron worktree GC under multiplexplugins/google_meet/process_manager.pyplugins/platforms/photon/adapter.pyplugins/platforms/telegram/adapter.pytools/environments/base_output.pytools/environments/singularity.pytools/skills_hub_github.pygh auth tokenwith NO env kwarg: gh returns launch os.environ GH_TOKEN/GITHUB_TOKEN when served profile has no PATagent/copilot_acp_client.pyagent/secret_sources/base.pyagent/secret_sources/command.pyagent/shell_hooks.pyagent/transports/codex_app_server.pyagent/vault_backends/base.pycli.pycron/scheduler.pycron/scheduler_delivery.pycron/scheduler_script.pygateway/platforms/webhook_filters.pygateway/run_inbound.pygateway/run_shutdown.pyhermes_cli/bang_shell.pyhermes_cli/dep_ensure.pyhermes_cli/kanban_db_dispatch.pyhermes_cli/main_dashboard.pyhermes_cli/main_tui_launch.pyhermes_cli/onepassword_secrets_cli.pyhermes_cli/secrets_cli.pyhermes_cli/web_server.pyhermes_cli/web_server_gateway.pyhermes_cli/web_server_memory.pyplugins/memory/byterover/__init__.pyplugins/web/ddgs/provider.pytools/bot_mode_dm.pytools/browser_lightpanda.pytools/browser_tool_install.pytools/browser_tool_real_profile.pytools/browser_tool_session.pytools/browser_use_cli.pytools/checkpoint_manager.pytools/code_kernel.pytools/computer_use/cua_backend.pytools/computer_use/cua_backend_daemon.pytools/computer_use/cua_backend_session.pytools/computer_use/doctor.pytools/computer_use/permissions.pytools/environments/local.pytools/file_operations_search.pytools/lazy_deps.pytools/process_registry.pytools/tts_command_provider.pytools/voice_mode.pytui_gateway/host_supervisor.pytui_gateway/methods_tools.pyResidual (unscoped, not this PR — same class, follow-up):
apptainer exec instance://… bash -cinherits os.environ — launch profile's API keys/HERMES_HOME/TERMINAL* reach the apptainer client and (per apptainer default host-env passthrough) the sandbox shell; unlike docker/ssh there is no scope-resolved passthrough/unset handlinggh auth tokenwith no env kwarg -> gh returns GH_TOKEN/GITHUB_TOKEN from os.environ (launch profile's token) and it is used as profile B's GitHub credential<name> setup; no env kwarg — terminal +hermes -p <name> setupchild inherit the LAUNCH profile's .env credentials / HERMES_HOME etc. (Windows)sh -lc '<name> setup'with no env kwarg — child inherits launch profile os.environ (creds, HERMES_HOME, TERMINAL_*) while configuring profilegh auth tokenwith env=noninteractive_git_env() (dict(os.environ)); leaks launch-profile GH_TOKEN/GITHUB_TOKEN/GH_CONFIG_DIR to gh and the resulting token authenticates X's clone (note: get_secret path above it IS scoped; only the gh fallback is raw)gh api …with no env kwarg; gh reads GH_TOKEN/GITHUB_TOKEN/GH_CONFIG_DIR from the launch profile's os.environ, not profile X's scopegh pr list— inherits launch profile GH_TOKEN/GITHUB_TOKEN for profile B's job-workdir repos (read-only, low severity)preprocess-> expand_inline_shell -> run_inline_shell(bash -c, env=delegated_child_subprocess_env()) -> None outside kanban => inherits launch os.environ — skill !cmdsnippets see launch profile HERMES_HOME/creds/TERMINAL_*Sweep 2 — threads / timers / executors (
rg threading.(Thread|Timer)(|run_in_executor(|loop.call_soon_threadsafe|asyncio.to_thread(over gateway/ tools/ agent/ tui_gateway/ plugins/ cron/ hermes_cli/, 604 hits)Totals: FIXED 3 (title thread, ws-orphan Timer, idle-reaper thread → teardown; the google_chat and atexit/RPC sites are not Thread/Timer lines so are not rg hits) · scope-carrying 349 · reads nothing profile-dependent 123 · process-global by design 100 · residual 29 (below). Full table in the gist.
FIXED / residual / process-global rows:
agent/title_generator.pytui_gateway/server.pytui_gateway/session_lifecycle.pyagent/credits_tracker.pyagent/review_idle_queue.pycron/scheduler.pygateway/platforms/api_server.pygateway/run_agent_cache.pygateway/run_turn.pyhermes_cli/approval_transport.pyhermes_cli/local_runtime/endpoint.pyhermes_cli/model_catalog.pyhermes_cli/observability/relay_shared_metrics.pyhermes_cli/plugins_dispatch.pyhermes_cli/web_routers/audio.pyhermes_cli/web_routers/messaging.pyhermes_cli/web_routers/models.pyplugins/memory/honcho/oauth_flow.pyplugins/platforms/a2a/adapter.pyplugins/platforms/email/adapter.pytools/async_delegation.pytools/discord_tool.pytools/terminal_tool.pytools/tts_tool_lifecycle.pytools/tts_tool_speaker.pytui_gateway/methods_voice.pytui_gateway/prompt_turn.pytui_gateway/server.pytui_gateway/session_notifications.pyagent/agent_init.pyagent/azure_identity_adapter.pyagent/curator.pyagent/models_dev.pyagent/monitoring/gateway_health_export.pyagent/plugin_stream_hooks.pycron/scheduler_provider.pycron/scheduler_thread.pygateway/control_socket.pygateway/kanban_watchers_common.pygateway/run.pygateway/run_startup.pygateway/shutdown_watchdog.pyhermes_cli/banner.pyhermes_cli/cli_chat_turn_mixin.pyhermes_cli/cli_commands_mixin.pyhermes_cli/cli_info_mixin.pyhermes_cli/cli_loops_mixin.pyhermes_cli/cli_model_switch_mixin.pyhermes_cli/cli_status_bar_mixin.pyhermes_cli/cli_tui_mixin.pyhermes_cli/cli_voice_mixin.pyhermes_cli/free_tier_bootstrap.pyhermes_cli/local_runtime/bootstrap.pyhermes_cli/local_runtime/supervisor.pyhermes_cli/main.pyhermes_cli/mcp_startup.pyhermes_cli/model_switch_providers.pyhermes_cli/nous_auth_keepalive.pyhermes_cli/voice.pyhermes_cli/web_routers/actions.pyhermes_cli/web_routers/cron.pyhermes_cli/web_routers/dashboard_ui.pyhermes_cli/web_routers/local_models.pyhermes_cli/web_routers/ops.pyhermes_cli/web_routers/status.pyhermes_cli/web_server.pyhermes_cli/web_server_dashboard.pyhermes_cli/web_server_sessions.pyplugins/hermes-achievements/dashboard/plugin_api.pyplugins/kanban/dashboard/plugin_api.pytools/browser_tool_lifecycle.pytools/computer_use/cua_backend.pytools/mcp_tool_loop.pytools/tirith_security.pytools/transcription_tools.pytools/voice_mode.pytui_gateway/change_watcher.pytui_gateway/compute_host.pytui_gateway/host_supervisor.pytui_gateway/hosted_room_driver.pyprofileto server RPCs that bind scopetui_gateway/methods_voice.pytui_gateway/session_reaper.pyResidual (unscoped, not this PR — same class, follow-up):
_scope_to_profile-> start_loopback_flow_background -> bare Thread(_run) -> authorize_via_loopback -> resolve_endpoints() -> HonchoClientConfig.from_global_config()/resolve_config_path()/resolve_active_host() in EMPTY context — OAuth environment/base_url/token_url resolve from the LAUNCH profile's honcho.json (persist path/host are correct)lane4 row verdicts (
/tmp/mux_retro/lane4_high.json,lane4_mid.json)tui_gateway/server.py:243_SlashWorkertools/bot_relay.py:426delivery_env--run-deliverypasses the roster home)tools/browser_tool.py:44plugins/platforms/a2a/adapter.py:588agent/command_token_source.py:49agent/title_generator.py:491-497tui_gateway/session_lifecycle.py:209-245, 303-317(+5 spawn sites)methods_session.py:1902needs no wrap (goes through_teardown_popped_session)plugins/platforms/google_chat/adapter.py(Pub/Sub bridge + send chain)hermes_cli/kanban_db_dispatch.py:458bare-PID kill_pid_alive), FIXEDplugins/platforms/a2a/adapter.py:69_reply_timeoutos.getenva2a/adapter.py:125)a2a security.py/protocol.pyaudit/conversation paths under launch hometui_gateway/server.py:2177-2208_schedule_mcp_late_refreshserver.py:2208)tui_gateway/session_notifications.py:677pollertui_gateway/methods_session.pyrows (1081 oneshot, 1180 usage, 318 create, 1416 pet, 2110 spawn_tree),methods_session_control.py:234,hermes_cli/web_server*.py,web_routers/*tools/browser_tool_lifecycle.pycleanup_all_browsers,tools/process_registry.pyturn reaper,tools/terminal_tool.py:701cleanup threadInfographic