fix: stop internal-timeout hangs from freezing the agent turn or worker forever by swifmo · Pull Request #72500 · NousResearch/hermes-agent

swifmo · 2026-07-27T06:01:34Z

Summary

Four related bugs let a single terminal command, approval check, environment setup step, or session-resume ever hang the agent turn (or, in the fourth case, the entire worker process) forever, with no error and no log line. All four were root-caused from live production incidents (well over a dozen multi-hour freezes across many days of use) and are fixed the same way: wrap a call that's supposed to self-enforce a timeout with an outer daemon-thread watchdog, so a defeated internal timeout can no longer hang the caller indefinitely.

1. `tools/terminal_tool.py` — foreground `env.execute()` hang

The idle-environment reaper (_cleanup_inactive_envs) only refreshes keep-alive for background processes via process_registry.has_active_processes(). A plain foreground env.execute() call never registers there, so a command that legitimately runs long can have its environment reaped mid-execution. Added an _inflight_foreground counter so the reaper treats "someone is inside env.execute() right now" the same as a registered background process.
env.execute() is expected to self-enforce execute_kwargs["timeout"] via BaseEnvironment._wait_for_process's poll loop, but that deadline lives entirely inside the call being waited on — if it's ever defeated, nothing outside can notice. Added _execute_foreground_with_watchdog: runs env.execute() on a daemon thread with a hard ceiling (requested timeout + 45s grace). On timeout, returns a normal timeout result instead of hanging the caller forever.

2. `tools/approval.py` — smart-approval auxiliary LLM call hang

Every terminal command flagged for review goes through _smart_approve(), which calls the auxiliary LLM (agent.auxiliary_client.call_llm, task="approval") synchronously before the command is allowed to run at all. That call is expected to self-enforce a timeout (default 30s), but as with the env.execute() case above, an internal timeout with no outside observer can hang the entire turn before the command it's gating ever executes — upstream of any protection in terminal_tool.py.
Root-caused from a live incident: a resumed session's tool call never progressed past the point where the auxiliary-approval log line would have been emitted, while a concurrent session on the same gateway kept working normally — ruling out a global/process-wide stall and pointing squarely at this synchronous call.
Added _call_llm_with_watchdog(): runs call_llm() on a daemon thread with a 45s hard ceiling. On timeout it raises, which _smart_approve()'s existing broad except already treats as "escalate" — so this doesn't change the function's contract, it just guarantees that path is actually reached within a bounded time instead of never.

3. `tools/terminal_tool.py` — environment creation and its lock

After the first two fixes landed, a third hang still occurred: a session stuck 40+ minutes with zero log output between the model's tool-call request and any tool_executor completion/error — well past the point either watchdog above would already have resolved it, meaning the stall was upstream of both, in environment setup itself.
_create_environment() (sandbox/shell setup, run once per task_id) had no bound at all. Added _create_environment_with_watchdog(): same daemon-thread pattern, 90s hard ceiling (generous for the "local" backend, which should be sub-second, while leaving room for container/cloud backends to pull images or cold-start).
The per-task creation lock (task_lock) was acquired via a plain blocking with task_lock: — if the thread creating the sandbox ever wedges, every subsequent call for that task_id piles up waiting on the same lock forever, turning one stuck call into a permanently dead task_id for the rest of the session. Changed to a bounded task_lock.acquire(timeout=...) with the corresponding release() in a finally. A plain threading.Lock can be released by any thread, not just the one that acquired it, so releasing it after giving up is safe even if the abandoned creation call is technically still running in the background — its eventual result is simply discarded.

4. `tui_gateway/slash_worker.py` — `HermesCLI(resume=...)` construction hang

A fourth, structurally distinct case: the worker's very first step, constructing HermesCLI (which opens SessionDB/state.db and, when resuming, replays that session's history), had no timeout at all — and unlike the three fixes above, this happens before the worker can process a single command, so a hang here doesn't show up as a stuck tool call. It shows up as the UI displaying a stale snapshot of whatever the previous (often already-dead) worker was last doing, forever, because the new worker never gets far enough to send any update.
Root-caused live via py-spy dump on a worker stuck for 17+ minutes: the main thread was blocked in native code at exactly the HermesCLI(...) call site, with no deeper Python frames — consistent with a slow SQLite operation, not a Python-level infinite loop. The profile's state.db had grown to 150+MB. SessionDB's connection sets only a 1s lock-wait timeout (deliberately short, per its own comments); nothing bounds how long schema init/integrity work takes once a statement actually has the lock. This codebase already has a documented precedent of a large state.db causing multi-minute stalls elsewhere (a hermes update fix), just not one covering this path.
Added _build_cli_with_watchdog(): runs the HermesCLI(...) construction on a daemon thread with a 120s ceiling (env-overridable via HERMES_SLASH_CLI_INIT_WATCHDOG_S, matching this file's existing _env_float knob pattern). Unlike the other three watchdogs, there's no "return an error and keep going" option here — if HermesCLI can't be built, this worker cannot serve any command at all. On timeout it logs a clear diagnostic to stderr and calls os._exit(1), so the process dies loudly and immediately instead of sitting invisibly frozen for the rest of its life.

All four watchdogs deliberately use a plain daemon thread rather than concurrent.futures.ThreadPoolExecutor: the latter's non-daemon workers plus its atexit join-all-threads hook would make a later, unrelated clean shutdown of the agent process (or, for #4, the worker's own prompt exit) hang too if a call ever actually got abandoned.

Testing

All four fixes were verified in isolation (not via the full test suite, which I didn't have a way to run against this checkout):

Fast/normal calls: unaffected, ~0ms overhead.
Simulated hung env.execute() / call_llm() / _create_environment() / HermesCLI(...): each watchdog fires, the caller regains control (or, for Fix terminal interactivity #4, the process exits) within its configured ceiling, and the abandoned daemon thread does not block process exit.
Reaper: correctly protects an in-flight foreground task while still reaping a genuinely idle one.
Lock timeout: a task_lock held by another thread correctly times out via acquire(timeout=...) rather than blocking forever.
Fix terminal interactivity #4's timeout-and-exit path verified via subprocess: a simulated hung construction exits with code 1 at the configured ceiling instead of hanging.
All modified files compile cleanly and the new functions import correctly.

Happy to adjust the ceiling values, add config knobs, or split into separate PRs if preferred. #4 in particular may also be worth pairing with a look at why state.db grows this large for a long-lived profile in the first place (auto-vacuum/pruning cadence), since the timeout is a safety net, not a fix for the underlying size.

Two related bugs let a single terminal_tool foreground command wedge the whole agent turn indefinitely, with no error and no log line — observed as a dozen multi-hour freezes in production, always stuck on a foreground "terminal" tool call. 1. The idle-environment reaper (_cleanup_inactive_envs) only refreshes keep-alive for BACKGROUND processes via process_registry.has_active_processes(). A plain foreground env.execute() call never registers there, so a command that legitimately runs long (e.g. the model requested a timeout close to or past lifetime_seconds) can have its environment reaped while a tool call is still actively blocked on it. Added an _inflight_foreground counter so the reaper treats "someone is inside env.execute() right now" the same as a registered background process. 2. env.execute() is expected to self-enforce execute_kwargs["timeout"] via BaseEnvironment._wait_for_process's poll loop, but that deadline lives entirely inside the call being waited on — if it's ever defeated (lock contention before the loop is reached, a backend-level stall, etc.) nothing outside the call can notice or recover. Added _execute_foreground_with_watchdog: runs env.execute() on a daemon thread and bounds the wait with a hard ceiling (requested timeout + 45s grace). If exceeded, logs full diagnostics and returns a normal timeout result instead of hanging the caller forever. Deliberately uses a plain daemon thread rather than concurrent.futures.ThreadPoolExecutor: the latter's non-daemon workers plus its atexit join-all-threads hook would make a later, unrelated clean shutdown of the agent process hang too if a call ever actually got abandoned. Verified in isolation: fast calls are unaffected (~0ms overhead), a simulated hung env.execute() is bounded and returns control to the caller, the abandoned daemon thread does not block process exit, and the reaper correctly protects an in-flight foreground task while still reaping a genuinely idle one.

…chdog Every terminal command flagged for review goes through _smart_approve(), which calls the auxiliary LLM (agent.auxiliary_client.call_llm, task= "approval") synchronously before the command is allowed to run at all. That call is expected to self-enforce a timeout (default 30s via auxiliary.approval.timeout), but as with the terminal_tool env.execute() case fixed in the prior commit, an internal timeout living entirely inside the call being waited on has no outside observer if it's ever defeated — and this one gates every flagged command, so a wedge here hangs the entire agent turn *before* the command it's gating ever executes, upstream of any protection in terminal_tool.py. Root-caused from a live incident: a resumed session's tool call never progressed past the point where the auxiliary-approval log line would have been emitted, while a concurrent session on the same gateway kept working normally — ruling out a global/process-wide stall and pointing squarely at this synchronous call. _call_llm_with_watchdog() runs call_llm() on a daemon thread with a 45s hard ceiling (same daemon-thread-not-ThreadPoolExecutor rationale as the terminal_tool fix, for the same clean-shutdown reason). On timeout it raises, which _smart_approve()'s existing broad except already treats as "escalate" -- so this doesn't change the function's contract, it just guarantees that path is actually reached within a bounded time instead of never. Verified in isolation: a wedged call_llm resolves to 'escalate' in ~1s instead of hanging, and the process still exits cleanly afterward.

…d hang class A third distinct hang point in the same family as the previous two fixes: a live incident showed a session stuck for 40+ minutes with zero log output between the model's tool-call request and any tool_executor completion or error -- well past the point either the terminal-execute watchdog or the smart-approval watchdog would already have resolved it, meaning the stall happened upstream of both, in environment setup itself. Two related gaps in the environment-acquisition path: 1. `_create_environment()` (sandbox/shell setup, run once per task_id) had no bound at all. Added `_create_environment_with_watchdog()`: runs it on a daemon thread with a 90s hard ceiling (generous for the "local" backend, which should be sub-second, while leaving room for container/cloud backends to pull images or cold-start). 2. The per-task creation lock (`task_lock`) was acquired via a plain blocking `with task_lock:` -- if the thread creating the sandbox ever wedges, every subsequent call for that task_id piles up waiting on the same lock forever, turning one stuck call into a permanently dead task_id for the rest of the session. Changed to a bounded `task_lock.acquire(timeout=_ENV_CREATION_WATCHDOG_SECONDS)` with the corresponding `release()` in a `finally`. A plain `threading.Lock` can be released by any thread, not just the one that acquired it, so releasing it here after giving up is safe even if the abandoned creation call is technically still running in the background -- its eventual result is simply discarded. As with the previous two watchdogs in this file, a plain daemon thread is used rather than concurrent.futures.ThreadPoolExecutor, for the same reason: the latter's non-daemon workers plus its atexit join-all-threads hook would make a later, unrelated clean shutdown of the agent process hang too if a call ever actually got abandoned. Verified in isolation: fast environment creation is unaffected (~0ms overhead), a simulated hung _create_environment() is bounded and returns control to the caller, and a lock held by another thread correctly times out via acquire(timeout=...) rather than blocking forever.

teknium1

Thanks for tracing the hang points; current main still has the synchronous calls at tools/terminal_tool.py:2346, :2399, :2820, and tools/approval.py:2805.

Problems

tools/terminal_tool.py:2217 moves env.execute() off the agent thread. _wait_for_process() checks current-thread interruption at tools/environments/base.py:936-948, while tools/interrupt.py:62-70 scopes /stop to the interrupted thread. The new worker will not see the agent-thread interrupt. It also loses the thread-local activity callback (tools/environments/base.py:43-45), so long commands lose gateway liveness heartbeats.
tools/approval.py:2737 starts a raw thread for call_llm(). agent/auxiliary_client.py:3218-3225 resolves the active main runtime from ContextVars and deliberately does not use cross-session compatibility globals; this worker has an empty context.
On creation timeout, tools/terminal_tool.py:2519 releases the lock while the worker can still be creating resources (:2160), allowing duplicate environment creation and discarding a late-created environment.

Suggested changes

Preserve interrupt, activity, and ContextVar propagation across watchdog workers, with regression tests.
Keep late environment creation owned until it can be safely registered or cleaned up, and cover the shared _create_environment() callers in tools/code_execution_tool.py:800 and tools/file_tools.py:1075.

Automated hermes-sweeper review.

teknium1 · 2026-07-30T13:11:11Z

+
+    def _run():
+        try:
+            result_queue.put(("ok", env.execute(command, **execute_kwargs)))


env.execute() now runs on a different thread, but _wait_for_process() checks a current-thread interrupt to implement /stop (tools/interrupt.py:62-70, tools/environments/base.py:936-948). The agent thread receives the interrupt, not this worker, so a stopped command can continue until its timeout. This worker also has no thread-local activity callback; preserve both execution-thread contracts before moving the call.

teknium1 · 2026-07-30T13:11:11Z

+
+    def _run():
+        try:
+            result_queue.put(("ok", call_llm(**kwargs)))


A bare thread starts with an empty ContextVars context. call_llm() derives the active runtime from _RUNTIME_MAIN_CONTEXT (agent/auxiliary_client.py:3218-3225), so smart approvals in concurrent gateway sessions can lose the initiating session's provider, endpoint, or credentials. Capture and run the parent context, or pass an explicit captured main_runtime.

teknium1 · 2026-07-30T13:11:11Z

                        env = new_env
                    logger.info("%s environment ready for task %s", env_type, effective_task_id[:8])
+            finally:
+                task_lock.release()


After _create_environment_with_watchdog() times out, its daemon can still finish creation, but releasing this lock immediately lets a retry start another creation. The late result is neither registered nor cleaned up because registration is in the caller after the wrapper returns. Keep ownership until late completion has a deterministic cleanup/registration outcome.

…tchdog A fourth, structurally distinct hang class: the worker's very first step, constructing HermesCLI (which opens SessionDB / state.db and, when resuming, replays that session's history), had no timeout at all -- and unlike the three fixes already in this branch, this one happens BEFORE the worker can process a single command, so a hang here doesn't show up as a stuck tool call. It shows up as the UI displaying a stale snapshot of whatever the PREVIOUS (often already-dead) worker was last doing, forever, because the new worker never gets far enough to send any update at all. Root-caused from a live incident: py-spy dump on a worker stuck for 17+ minutes showed the main thread blocked in native code at exactly the HermesCLI(...) call site, with no deeper Python frames -- consistent with a slow SQLite operation, not a Python-level infinite loop. The profile's state.db had grown to 150+MB. SessionDB's connection sets only a 1s *lock- wait* timeout (deliberately short, per its own comments, to avoid sitting in SQLite's internal busy handler); nothing bounds how long schema init/ integrity work takes once a statement actually has the lock. This codebase already has a documented precedent of a large state.db causing multi-minute stalls elsewhere (a `hermes update` fix), just not one covering this path. _build_cli_with_watchdog() runs the HermesCLI(...) construction on a daemon thread with a 120s ceiling (env-overridable via HERMES_SLASH_CLI_INIT_WATCHDOG_S, matching this file's existing _env_float knob pattern). Unlike the other three watchdogs in this branch, there is no "return a normal error and keep going" option here -- if HermesCLI can't be built, this worker cannot serve any command at all. On timeout it logs a clear diagnostic to stderr and calls os._exit(1), so the process dies loudly and immediately instead of sitting invisibly frozen for the rest of its life. Same daemon-thread (not concurrent.futures.ThreadPoolExecutor) rationale as the other three: an abandoned worker must not block this process's own exit. Verified: fast construction is unaffected (~0ms overhead); a simulated hung construction times out at the configured ceiling and the process exits with code 1 (confirmed via subprocess) instead of hanging.

… watchdog A fifth distinct hang point, downstream of all three env.execute()-side watchdogs already in this branch -- and the reason they couldn't have caught it: this one fires AFTER the shell command has already fully finished (bash exited, output captured), in the plugin-hook post-processing that runs before the result is returned to the model. Root-caused from a live incident: the underlying command (a trader-service restart, spawning a correctly-detached long-running child) had completed -- confirmed via the process list: neither bash.exe nor the invoked script were still running, only the intentionally-detached background process -- yet the tool call sat showing "Running" for 18+ minutes with no completion or error ever logged. The only remaining unbounded work in the foreground path once env.execute() returns is invoke_hook("transform_terminal_output", ...), which is wrapped in a bare `except Exception: pass` ("fail-open" per its own comment) -- that catches a raised error but not a hang. Any third-party or user-installed plugin registering this hook can therefore freeze every foreground terminal call forever, with none of the earlier watchdogs able to help since the command they guard has already finished. _invoke_terminal_output_hook_with_watchdog() runs invoke_hook() on a daemon thread with a 20s ceiling -- short, since this is in-process output post-processing, not I/O like the other watchdogs' targets. On timeout it returns [] (the same shape invoke_hook returns when no plugin handles the event), which the call site already treats as "keep the untransformed output" -- preserving the existing fail-open contract, just bounding it. Same daemon-thread rationale as the other watchdogs in this file: an abandoned worker must not block a later clean process shutdown. Verified in isolation: a fast hook is unaffected; a simulated hung hook returns [] at the configured ceiling instead of hanging, and the process exits cleanly afterward.

…ll and log its outcome The smart-approval guardian (`_smart_approve`) gates every flagged terminal command with a synchronous auxiliary LLM call, but it never passes `timeout=` and logs nothing on the normal path. In production a stalled provider response silently froze the agent turn for 62 minutes with zero log output; the gateway kill-switch eventually fired, and only an unrelated error surfaced afterwards (NousResearch#82846; watchdog-style fix in NousResearch#72500). The call was invisible by design — nothing logs at the hang point. Changes in tools/approval.py: - Resolve the same configured timeout the client would use internally (`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and pass it explicitly to `call_llm`, so the deadline cannot be lost if the internal default resolution changes or is misconfigured. - Log the assessment call and its duration (DEBUG), and promote the failure branch from DEBUG to WARNING with elapsed time + exception class, so a wedged guardian call is visible in the logs instead of silent. - Failure still returns "escalate" (fail open to the human/pattern gate) — behavior unchanged, observability only. Complements NousResearch#72500 (watchdog hard ceiling) rather than duplicating it: explicit timeout is the root-cause hardening, logging closes the silence gap; the watchdog remains the safety net if the SDK-level timeout itself is defeated. Tests: explicit timeout forwarded to call_llm (revert-fails), failure logs WARNING + escalates. 49 approval-adjacent tests pass; one unrelated test_approval.py failure is pre-existing (fails on clean main too).

… site Bug-class extension of NousResearch#72500's F4: main grew a second _create_environment call site (the lazy per-task environment init helper) with its own per-task task_lock that predates the PR. Give it the same bounded lock acquire (timeout=_ENV_CREATION_WATCHDOG_SECONDS) and route creation through _create_environment_with_watchdog; its existing 'except Exception -> return None' best-effort contract already absorbs the watchdog's TimeoutError.

kshitijk4poor · 2026-08-24T20:11:11Z

@swifmo — salvaged into #94182 with your 6 commits cherry-picked, authorship preserved through ~700-line drift and hand-resolved conflicts. All four watchdog fixes + the F4 bug-class extension to the second _create_environment site. Closing in favor of #94182 — thanks for the thorough root-cause work.

kshitijk4poor · 2026-08-25T20:03:22Z

@swifmo — triage update: the salvage (#94182) was closed. CI caught a real behavioral regression: F2's watchdog wraps env.execute() in a daemon thread, which breaks Hermes' per-thread interrupt delivery (tools/interrupt.py uses threading.current_thread().ident). A pre-existing interrupt bit no longer reaches commands running on the watchdog worker — exit_code=0 instead of 130. Your approach is sound for bounding timeouts but conflicts with the per-thread interrupt invariant. A correct fix needs the watchdog to propagate the caller's thread identity to the worker, or use a process-wide signal. Re-opening your PR so this context is visible for a future attempt.

…ll and log its outcome The smart-approval guardian (`_smart_approve`) gates every flagged terminal command with a synchronous auxiliary LLM call, but it never passes `timeout=` and logs nothing on the normal path. In production a stalled provider response silently froze the agent turn for 62 minutes with zero log output; the gateway kill-switch eventually fired, and only an unrelated error surfaced afterwards (#82846; watchdog-style fix in #72500). The call was invisible by design — nothing logs at the hang point. Changes in tools/approval.py: - Resolve the same configured timeout the client would use internally (`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and pass it explicitly to `call_llm`, so the deadline cannot be lost if the internal default resolution changes or is misconfigured. - Log the assessment call and its duration (DEBUG), and promote the failure branch from DEBUG to WARNING with elapsed time + exception class, so a wedged guardian call is visible in the logs instead of silent. - Failure still returns "escalate" (fail open to the human/pattern gate) — behavior unchanged, observability only. Complements #72500 (watchdog hard ceiling) rather than duplicating it: explicit timeout is the root-cause hardening, logging closes the silence gap; the watchdog remains the safety net if the SDK-level timeout itself is defeated. Tests: explicit timeout forwarded to call_llm (revert-fails), failure logs WARNING + escalates. 49 approval-adjacent tests pass; one unrelated test_approval.py failure is pre-existing (fails on clean main too).

…ll and log its outcome The smart-approval guardian (`_smart_approve`) gates every flagged terminal command with a synchronous auxiliary LLM call, but it never passes `timeout=` and logs nothing on the normal path. In production a stalled provider response silently froze the agent turn for 62 minutes with zero log output; the gateway kill-switch eventually fired, and only an unrelated error surfaced afterwards (NousResearch#82846; watchdog-style fix in NousResearch#72500). The call was invisible by design — nothing logs at the hang point. Changes in tools/approval.py: - Resolve the same configured timeout the client would use internally (`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and pass it explicitly to `call_llm`, so the deadline cannot be lost if the internal default resolution changes or is misconfigured. - Log the assessment call and its duration (DEBUG), and promote the failure branch from DEBUG to WARNING with elapsed time + exception class, so a wedged guardian call is visible in the logs instead of silent. - Failure still returns "escalate" (fail open to the human/pattern gate) — behavior unchanged, observability only. Complements NousResearch#72500 (watchdog hard ceiling) rather than duplicating it: explicit timeout is the root-cause hardening, logging closes the silence gap; the watchdog remains the safety net if the SDK-level timeout itself is defeated. Tests: explicit timeout forwarded to call_llm (revert-fails), failure logs WARNING + escalates. 49 approval-adjacent tests pass; one unrelated test_approval.py failure is pre-existing (fails on clean main too).

) * feat(desktop): wire managed SSH update engine into main process Re-implements the #93042 main.ts wiring against current main (post-#94724 drift), making the extracted managed-ssh-update engine reachable: - ManagedConnectionUpdateGate instance + owner-only recovery journal at DESKTOP_MANAGED_SSH_RECOVERY_PATH (read/write/persist/mark/clear with strict record validation). - IPC: hermes:connections:update-managed (requestManagedSshUpdate with correlation-id claim + in-flight dedupe); update-all's ssh rows now route through the transactional drain/update/restore lifecycle instead of POSTing the remote backend updater. - Gate enforcement at every dial/mutate seam: bootstrapSshConnectionInner (pre-dial + publication fence with exact-serve rollback via rollbackSshBootstrapResult), resolveRemoteBackend, ensureRegistryBackend, saveRegistryConnection dial-field edits, connections:remove, and primary-routing mutations (set-primary, set-launch-mode, connection-config save/apply, profile:set) via assertCanMutateManagedPrimaryRouting. - Scope capture/drain/restore drivers: captureManagedSshScopes (pool + primary discovery, bootstrap fence join), drainManagedSshScope (exact identity-re-proof termination, no-kill forward recovery), ensureManagedSshBackend(AtKey)/restoreManagedPrimarySshBackend restores, openManagedSshUpdateTransport for serve-less connections. - Startup recovery (resumeManagedSshRecoveries before createWindow) and before-quit join of in-flight update/recovery operations BEFORE the SSH coordinator is sealed, so restore dials are not refused during quit. - Extended sshConnections state (spawnNonce/creationTime(Ns)/startedAt/ hermesPath/hermesHome/pythonPath/remoteProfile/registryConnectionId/ primaryRegistryScope) so drain can prove the exact serve it owns; bootstrap coordinator entries carry metadata for the update fence; persistSshConnectionToken mirrors tokens per managedSshTokenPersistencePlan. - preload/global.d.ts: connections.updateManaged + DesktopManagedConnectionUpdateResult/Receipt types. Renderer UI (fleet-updates store, about-settings, system.ts ProfileScope plumbing, i18n) intentionally NOT wired — it belongs to the deferred fleet rollout UI and follows separately. Wiring re-implemented against current main; design from #93042 by @andrexibiza tsc -p apps/desktop clean; electron project 1924/1924 passed (133 files, incl. 131/131 across the three engine suites); eslint clean on touched files. * fix(update): check and apply config migrations on current checkout / retry paths (#91360) When an update was interrupted or failed mid-install (e.g. dependency install timeout) after pulling new code, the subsequent update run takes the 'commit_count == 0' path and early-returned without checking or migrating the configuration. Fresh code requiring a newer config version would fail to boot on the next run. Extract _check_and_apply_config_migration and invoke it across all update completion paths (normal update, current checkout / node repair, and python dependency repair). * fix(update): run config migration on the 'Already up to date' repair path (#91360) A failed update attempt can pull fresh code onto disk and then die before the config-migration block (e.g. a PyPI timeout during the dependency sync). The desktop hand-off retries; the retry takes the commit_count == 0 branch, repairs deps, prints 'Already up to date!' and returns early - skipping _run_config_check_fresh / migrate_config entirely. The fresh code (requiring a newer _config_version) then refuses to start against the old config until 'hermes doctor --fix' is run. Fix: _maybe_migrate_config_on_current() mirrors the version_bump_only handling (silent, non-interactive) and is called on both repair-path completion points before claiming success. Also: scripts/desktop-update/posix.sh no longer retries when the update was deliberately SKIPPED (checkout parked on a non-target branch) -, the retry is deterministic and only wastes time. Uses a dedicated non- colliding exit code (8) and an honest message instead of 'Update failed'. New tests: tests/hermes_cli/test_update_config_migration_on_current.py (5 cases: migrate-when-behind, noop-current, noop-ahead, warning re- surface, silent check failure). * chore: map fred0m noreply email in contributors registry (#91360 salvage) * fix(mcp): register stdio MCP helper children in the spawn ledger and reap orphans (#61514) Stdio MCP helper subprocesses (npx/binary servers) never import Hermes code, so they could not self-register in the machine spawn ledger and an unclean parent exit left them running invisibly forever. - process_identity.register_child(pid, purpose): ledger mirror of register_self for spawned children — records the CHILD (pid, create_time) with this process as spawner. Refuses pid-only entries a PID reuse could forge. Writes go through the single _append_entry path under _LEDGER_LOCK (prune + atomic tmp/replace unchanged). - 'mcp-helper' added to REAPABLE_PURPOSES so the updater's _ledger_reapable_backend_pids rung flows helpers through its existing spawner_is_dead gate (live spawner => never reaped). - tools/mcp_tool.py: best-effort register_child(pid, 'mcp-helper') at the post-spawn PID capture; never breaks MCP startup. - reap_orphaned_mcp_helpers(): startup sweep mirroring _reap_orphaned_desktop_local_serves but ledger-driven — kills only helpers whose recorded spawner is PROVABLY dead, with a create_time re-check at kill time. Wired next to the desktop serve reap in web_server.py. * ci: give fork Python suite a bounded hour * fix(update): recover stalled Windows desktop handoffs * fix(update): quiesce stalled Windows updater trees * fix(update): assign Windows steps before execution * fix(update): count logs/update.log growth as watchdog progress The #95625 watchdog cancels a step after StepIdleTimeoutSeconds (300s) with no stdout/stderr. But a real `hermes update` is stdout-silent for 40+ minutes by design: the Electron/vite build streams to logs/update.log, not the child's pipes (hermes_cli/update_cmd.py's update-log tee). An output-only ceiling would therefore kill every healthy large update at 5 minutes and mark it exit 124. The drain now fingerprints logs/update.log (size + mtime) and, when the idle ceiling is otherwise reached, treats growth of that file as progress: reset the clock instead of terminating the tree. The stat runs only once the ceiling fires, so the hot drain path never touches the filesystem. HERMES_UPDATE_STEP_IDLE_SECONDS remains the override; HERMES_UPDATE_PROGRESS_LOG points the self-test at its own file. TDD proof: -SelfTestPipeDrain gains a fourth arm, logstall -- a step that is silent on its pipes but appends to the progress log every second and must reach its natural exit 3, never 124. Linux CI pins the same contract at source level (TestIdleWatchdogCountsUpdateLogGrowth); sabotage-verified: making the log-growth consult inert fails test_stall_branch_consults_log_growth_before_terminating. * fix(update): stop counting the Windows resume token as a fleet runtime Fixes #93406 (residual). _fleet_probe_expected_runtimes counted the _windows_gateway_resume pause/resume token (profiles/unmapped entries) as an 'expected fleet rows' signal. The token is pause/resume bookkeeping, not a runtime inventory, and its entries have no rows collect_fleet_versions() can return: unmapped Scheduled-Task gateways never publish gateway_state.json, and a resumed profile gateway relaunches detached and may not republish within the probe window. So every Windows update that paused a gateway set _fleet_rows_expected, the verification loop silently waited out its polling window (~14 min wall clock with the retry loop on user reports), printed 'Fleet version check returned no rows', and exited 1 for an update that succeeded. Expected-runtimes now keys only on row-capable signals: restart-phase bookkeeping, the pre-restart PID snapshot, and the pre-update plan inventory -- which already cover any genuinely live pre-update Windows gateway. Counterfactual proof: tests/hermes_cli/test_update_fleet_probe_resume_token.py fails on the pre-fix predicate (token-only => True) and passes with the fix; the row-capable signals are pinned unchanged. * fix(update): normalize windows.ps1 to LF and keep fixture here-string braces off column 0 The cherry-pick landed the file with CRLF endings and a fixture here-string whose col-0 brace prematurely terminated the handoff test's SelfTest-block strip, tripping the drive-python-not-the-shim guard on fixture code. LF restored (matching main), child-script loop inlined. * test: normalize CRLF in the windows.ps1 handoff guard reader .gitattributes forces eol=crlf for *.ps1, so CI checkouts hand the test CRLF content and the SelfTest-strip regex anchors never matched; it went unnoticed while the stripped fixture blocks contained no offenders. * test: invert the resume-token fleet-probe pin to the new contract Main's pin (added after this branch was cut) froze the #93406 bug as the contract: resume-token services demanded probe rows that SCM-paused services can never produce, stalling every healthy Windows desktop update (#95589). The pin now asserts the exclusion; restart-phase and pre-restart-pid signals keep failing closed. * fix(desktop): scope durable transcript-tail cache by owning profile/connection Stored session ids are only unique within one profile's state.db, and localStorage survives profile switches in the same window. The durable transcript-tail cache keyed entries by bare stored id, so a tail cached while working in profile A was painted against profile B's backend after a switch; that backend never held the session and retried it on every wake ('session not found'), matching the desktop.log pattern of repeated session.reclaimed (ws_orphan_reap) events for ids that live in another profile's state.db (#94828). Scope every entry by its owner — the same {connectionId, profile} shape the REST layer already threads through getLatestSessionMessages and the in-memory twin stores in TranscriptTailState: - key entries under v2:[connectionId, profile, storedId]; loads only return a tail saved under the SAME scope - thread the session's owning scope through all call sites (resumeSession's warm-path save + cold-paint load/rollback drop, final save, removeSession's delete drop) - sweep pre-scoping v1 entries once per window: a bare-id key cannot be attributed to an owner, so it must never paint again - dropTranscriptTail with a scope leaves other backends' same-id tails intact * fix: delete-path drops every scope of a removed id; legacy purge latches on success only Review follow-ups on the salvage (#94914): the delete path derives its scope from the removed row while saves derive from ownerRoute — a shape drift orphaned the entry until LRU eviction. Deletes now sweep every scope of the stored id (safe: deleted ids are never reused), while the failed-resume path keeps the exact-scope drop (a same-id twin in another profile must keep its tail). purgeLegacyV1 latches only after a completed sweep so a mid-sweep throw retries next touch. * fmt(js): `npm run fix` on merge (#95969) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * test: make updater checks portable on Windows * fix(update): raise the step-idle watchdog default to 10 minutes Desktop builds legitimately produce no output for minutes on Windows (the updater terminal is static until a step completes), so 300s risks cancelling healthy long steps. 600s keeps the watchdog meaningful for true stalls while clearing slow builds; HERMES_UPDATE_STEP_IDLE_SECONDS still overrides. Per Teknium's review. * refactor(terminal): honest schema, pager defaults in the env, unified notify arg (837 → 670 tok/call, −20%) (#95937) * fix(terminal): stop claiming a Linux environment — point at the env section; near-neutral tokens * feat(terminal): default GIT_PAGER/PAGER=cat in session env; drop schema lines the runtime already enforces; fix pty backend claim * refactor(terminal): unify notify_on_complete+watch_patterns into notify (bool|list); trim pipe-masking prose (runtime hint owns it) * fix(terminal): background param referenced the unadvertised legacy arg name * refactor(terminal): background-only modifiers (pty, notify) fail loud on foreground calls with corrected shape * fix(execute_code): block the new notify arg in the sandbox terminal stub (foreground-only) * feat(browser): consent-gated real default-Chromium profile for local browsing + local_browser arg * fix(browser): pass encoding to detector subprocesses and accept local_browser in test spies The two default-browser detectors call subprocess.run(text=True) without an explicit encoding, which the Windows-footgun linter (and its full-repo test, tests/scripts/test_footgun_subprocess_encoding.py) rejects. Pass encoding='utf-8', errors='replace' like the rest of the tree. Two existing tests replace browser_navigate / _navigation_session_key with positional-only lambdas; both callables now receive local_browser= from the registry handler and browser_navigate, so the spies raised TypeError. Accept the keyword with its default. Fixes the three CI failures on the PR head (Windows footguns lint, test_browser_extension_router_wiring x2, test_browser_open_timeout). * fix(browser): read the macOS https handler per entry and drop the installed-browser fallback _detect_default_darwin matched a Chromium bundle id and the literal 'https' anywhere in the whole LSHandlers dump, so a browser registered for ftp or a content type was reported as the https default, and map order decided ties. When nothing matched it fell back to the first installed Chromium app — with Safari or Firefox as the actual default that drove a browser the user never consented to, contradicting the docstring, the config comment and the desktop copy ('a non-Chromium default fails with a clear message'). Parse the dump entry by entry, take the LSHandlerRoleAll/Viewer of the entry whose LSHandlerURLScheme is https, and fail closed on anything else — an empty handler list is what macOS stores while Safari is still the implicit default. Tests feed real 'defaults read' output shapes instead of patching the detector (reviewer fixture from the PR discussion: Safari on https, Chrome on ftp). * fix(browser): resolve snap and Flatpak Chromium profiles on Linux real_profile_data_dir hard-wired Linux to $XDG_CONFIG_HOME/<name>, and the xdg fragment map only knew the native package names. Ubuntu's default snap Chromium (xdg reports chromium_chromium.desktop, profile under ~/snap/chromium/common/chromium) and Flatpak builds (~/.var/app/<id>/config/…) therefore ended in 'profile directory was not found' for a browser the user runs every day, and Flatpak Chrome (com.google.Chrome.desktop) was reported as 'not a supported Chromium browser'. Try the native, snap and Flatpak locations and return the first that exists; fall back to the native path so the error message still names a concrete directory. Map the Flatpak application ids in the xdg lookup. Tests cover the xdg names for all four browsers in native and Flatpak form, and the directory preference order with a temp HOME. * fix(browser): keep local_browser inside the private-URL policy and the existing local session _navigation_session_key returned the ::local sidecar key for local_browser before the cloud-provider and auto_local_for_private_urls checks. Two consequences: - Every private-URL gate in browser_navigate (credential-bearing query, _is_safe_url pre-nav, post-redirect) is keyed off the sidecar key, so a model-supplied browser_navigate(url, local_browser=True) opened LAN addresses in a host-side Chromium — with the real profile's cookies — even when the user had set browser.auto_local_for_private_urls: false. Consent to the profile is not consent to override the LAN routing opt-out; a private URL now follows auto_local_for_private_urls exactly as without the flag. - Without a cloud provider the bare session already is the local Chromium (and already carries the real profile when consented), so the flag created a second session for the same task on the same user-data-dir, which Chromium's process singleton refuses. The flag is now a no-op there. The existing consent/CDP-precedence tests keep their assertions; the sidecar test now states the cloud-provider precondition it silently relied on. * feat(browser): real-profile browsing via agent-browser copy + browser-use CDP Copy the user's default-Chromium profile (auth state only) into a managed snapshot, launch Hermes' packaged Chromium on it via agent-browser, and hand the CDP endpoint to the Browser Use CLI (and built-in tools) to drive. The snapshot is a non-default dir, so it sidesteps Chrome 136+'s default-profile remote-debugging block and never contends with the user's running browser; launched without mock-keychain switches so keyring-encrypted cookies decrypt. - consent-gated browser_exec 'local' arg (schema only appears with consent) - fail-closed on non-Chromium default / snapshot failure - stale-session guard: reuse only when the live session is on our copy dir - snapshot excludes extensions/service-workers (renderer wedge) + caches * fix(browser): real-profile snapshot is a first-class secret store + preserve channel identity Addresses two P1 review blockers (kshitij / @kxee) on the real-profile feature: Credential-store lifecycle for ~/.hermes/browser-profile/ (copied Cookies/ Login Data): - exclude the singular 'browser-profile' dir from backup AND import (_EXCLUDED_DIRS drives both) — was silently archiving cookies/logins - add a browser-profile/ directory-PREFIX read-deny to agent/file_safety.py, same class as auth.json / mcp-tokens - secure the snapshot dir through the canonical hermes_cli.config._secure_dir (honors managed/NixOS group-share + HERMES_UID/GID), not a bespoke chmod Channel identity (#95549 invariant — never normalize Beta/Dev/Canary to stable, which would drive a different account's profile): - detect recognized pre-release channels FIRST (Win ProgIds, macOS bundle ids, Linux .desktop) and return UNSUPPORTED_CHANNEL - macOS bundle match is now EXACT (was startswith); Linux/Win channel-before- stable ordering; real_profile_data_dir/chromium_executable reject the sentinel - _real_profile_cdp fails closed with a channel-specific message, never snapshots Tests: channel-not-normalized (linux/darwin/windows), wrong-principal fail-closed, backup exclusion, read-guard block/allow, snapshot dir secured. 187 browser + 222 backup/file_safety pass. Live re-verified: real Gmail inbox still loads. * fix(browser): real-profile review round 2 — last_used profile, sidecar isolation, macOS26 parser, perms, lightpanda Addresses the five findings from @kshitijk4poor + @GottZ on #95620: 1. macOS 26 LSHandlers parser returned a version number ('7559.97') from the nested LSHandlerPreferredVersions block instead of the bundle id — detection returned None on a machine whose default IS Chrome. Strip the nested block before the role regex. 2. Wrong profile launched (the LinkedIn/Gmail 'logged out' bug): Chrome opens Default, but the session lives in Local State profile.last_used (e.g. 'Profile 6'). Resolve last_used and mirror its auth files into the copy's Default on both fresh and refresh paths, so the launched browser is signed in. 3. Private-URL sidecar carried the real cookie jar to arbitrary LAN hosts: _create_local_session gains allow_real_profile (default True); the force_local sidecar passes False → always a throwaway profile, and a real-profile resolve failure no longer breaks private-URL routing. 4. Snapshot permissions were set once (fresh only): now secure the snapshot dir AND its browser-profile parent on every consented launch. 5. browser.engine=lightpanda + consent gave an unactionable error: guard with _using_lightpanda_engine() before detection, naming the setting and the fix. Tests: last_used mirroring (fresh+refresh+fallback), sidecar throwaway + error isolation, macOS26 parser + detect, perms-on-refresh, lightpanda guard. 198 browser tests pass. Live: Profile-6 cookie DB lands in copy Default (file-level); real Gmail (Default profile) still signed in. * refactor(computer_use): drop the cua_browser_* route — browser work goes through browser_exec Real-profile browsing routes all in-page browser work through the Browser Use CLI (browser_exec), which obsoletes the cua-driver typed-browser surface baked into computer_use. Remove it so computer_use is a pure DESKTOP-control tool (screenshots / mouse / keyboard / window management) and every call's schema drops ~24 browser-only params + 9 actions. - schema.py: 9 cua_browser_* actions and the typed-browser param block removed; 14 desktop actions + shared params kept; description drops the browser rung. - tool.py: cua_browser entries out of _SAFE/_DESTRUCTIVE_ACTIONS; the whole cua_browser dispatch block deleted; {"type","cua_browser_type"} → "type" (desktop typing untouched); _config_preauthorized (browser-prepare-only, a no-op for every desktop action) and the browser-page escalation hint removed. - browser_route.py deleted (no importers outside the package); cua_backend.py drops the import + typed_browser_* methods; backend.py drops the non-abstract defaults. - tests: browser-route/contract suites removed; browser assertions trimmed. Desktop control unchanged. 233 computer_use tests pass; the 1 remaining failure (test_gateway_session_key_yolo_maps_to_unrestricted_mode) is a pre-existing cross-test state leak — fails identically on origin/main, passes in isolation. * fix(browser): real-profile review round 3 — overlay-ordering race, torn-copy marker, consent cleanup, active-only copy Addresses the round-3 findings from @Adolanium + @kshitijk4poor on #95620: 1. Overlay-before-reuse race (blocker): _real_profile_cdp ran snapshot_real_profile BEFORE the session-reuse check, so a cold resolve that ends in reuse rewrote Cookies/Login Data under a live Chromium holding the user-data-dir open (torn DBs, locked txns, phantom logouts). Now: resolve copy dir as a PATH, probe reuse first, return early on a hit; snapshot/overlay only on the relaunch path when no live browser owns the dir. 2. Torn first copy poisoned freshness forever: freshness keyed on isdir(Default), so a half-written copy (disk full / Ctrl+C) was treated as populated and only ever got auth overlays. Now gated on a .hermes-snapshot-complete marker written only after a full copy succeeds; a torn copy is rebuilt from scratch. 3. Consent revocation left copied credentials on disk: turning use_real_profile off now deletes ~/.hermes/browser-profile/ on next browser use (cleanup_real_profile_snapshots), so cookies/logins don't outlive consent. 4. Stale non-active profile copies: only the ACTIVE profile (last_used) is copied into the copy's Default now — other Chrome profiles are never snapshotted (smaller copy, no stale credential dirs lingering). 5. Docs/config/desktop wording aligned to actual behavior (active-profile only, refresh on fresh session, consent-off cleanup). Tests: overlay-skipped-on-reuse + overlay-runs-on-relaunch, done-marker gating + torn-copy rebuild, active-only copy, consent-off cleanup (removes store + idempotent + triggered from _real_profile_cdp). 206 browser + 222 backup/file_safety pass. Live: reuse skips re-snapshot; direct launch on the active-only copy loads the real signed-in Gmail inbox. * fix(browser): copy real-profile auth DBs lock-aware (Windows 'file in use') On Windows a running Chrome holds Cookies / Login Data / Web Data with an exclusive lock, so the raw file copy the snapshot used raised WinError 32 ('being used by another process') and the best-effort skip left a signed-out copy — the reported profile-cloning failure. Fix: copy the SQLite auth DBs via SQLite's online-backup API (read-only connection + Connection.backup()), which reads a consistent COMMITTED snapshot while the writer holds the lock. Non-DB files (Preferences, Local State) stay a plain copy. If even the online-backup can't read a DB, snapshot_real_profile now FAILS CLOSED with an actionable 'close <browser> and retry' message instead of launching a silently signed-out session. - _copy_auth_file: sqlite-backup for Cookies/Login Data/Web Data, raw copy otherwise, raw-copy fallback if backup fails. - Drop -journal/-wal/-shm sidecars from the auth set + snapshot ignore: the backed-up DB is self-contained; a stale sidecar next to it corrupts it. - Fresh copytree excludes the auth DBs (raw copytree of a locked file raises on Windows); they're always mirrored lock-aware afterward. - _mirror_profile_auth returns the count of DBs it could not copy so the caller can fail closed. Tests: locked-DB copied-via-backup (open write txn = live-lock analog, 42 committed rows, uncommitted excluded, no journal sidecar), _copy_auth_file DB vs plain, fail-closed when unreadable. 192 browser tests pass. Live: 68 real cookies copied through the backup path and the session launches. * test(browser): PROOF — Windows live E2E for locked-DB real-profile copy [do-not-merge] One-shot windows-latest E2E: launches real Chrome on a user-data-dir so it holds the cookie DB with a Windows share lock, asserts a RAW copy fails (WinError 32 precondition — else skip, no vacuous green), then asserts _copy_auth_file copies it via SQLite online-backup and the result is a readable Cookies DB with the cookies table. This proves the Windows 'file in use' fix on a real runner — the coverage the Linux lanes cannot provide. PROOF branch evidence only: this workflow + test are reverted before merge and must never land on main. * test(browser): PROOF round 2 — Windows locked-profile fails CLOSED (live-corrected) The first live Windows run DISPROVED the sqlite-online-backup claim: Chrome's share lock on Windows is strong enough that even a read-only SQLite open is refused by the OS (raw-copy precondition fired, _copy_auth_file still returned False). Copy-while-Chrome-runs is impossible on Windows — the earlier fix was theatre that only passed on Linux (no mandatory locking). Corrected contract, now asserted live: with a running Chrome holding the cookie DB, snapshot_real_profile FAILS CLOSED with 'could not read ... login data (N locked). Close <browser> and retry' — never a silent signed-out/torn copy. Second test proves the supported path (Chrome closed) copies cleanly. So real-profile browsing on Windows requires the browser closed; Linux/macOS unaffected; live-drive-the-real-profile is tracked in #95669. PROOF branch evidence only — workflow + test reverted before merge. * test(browser): PROOF diag — probe which read strategy beats Chrome's Windows lock [do-not-merge] Adds a Windows-live diagnostic that, against a cookie DB held by a running Chrome, reports which read strategy succeeds: shutil, open-rb, sqlite mode=ro, sqlite immutable=1, sqlite ro+nolock, raw win32 CreateFile with full share flags. This tells us empirically whether any in-process read path exists (immutable=1 / share-all open) before reaching for VSS/admin. Fails-closed test marked xfail while the real behavior is derived from the diagnostic. * test(ci): proof workflow uses per-sha concurrency, no cancel-in-progress [do-not-merge] Previous runs were auto-cancelling each other (ref-scoped group + cancel-in-progress). Per-sha group lets each proof run finish so the diagnostic actually reports. * test(ci): run Windows diagnostic first + hard-bound the live test [do-not-merge] Prior run hung 24min in the product-path test (snapshot_real_profile against a locked profile blocks on Windows — itself a finding). Diagnostic now runs FIRST (each strategy internally bounded, reports fast), live test second under a faulthandler 150s dump-and-die so a hang can't burn the job. Job timeout 12min. * fix(browser): Windows real-profile fails fast when the browser is running Live windows-latest proof settled it: a running Chrome opens its cookie DB deny-all (even CreateFile with FILE_SHARE_READ|WRITE|DELETE fails; sqlite mode=ro/immutable/nolock all 'unable to open'), so copy-while-running is impossible on Windows without VSS/admin — and the prior code HUNG ~24min on the locked file. Fix: a fast up-front lock probe (_profile_is_locked: one open() of the active profile's cookie DB; PermissionError = locked) runs BEFORE any copy in snapshot_real_profile. If locked, bail immediately with 'fully quit the browser (incl. background/tray) and retry, or turn browser.use_real_profile off'. Never hangs, never a silent signed-out copy. POSIX has no mandatory locking so the probe never trips there — copy-while-running still works on macOS/Linux. Docs: admonition stating Windows needs the browser fully closed (background apps included); the live-drive-while-running path is #95669. Tests: lock-probe unit coverage (readable/no-db/PermissionError), snapshot fails-fast-no-copytree when locked. Windows live E2E asserts the fast-fail contract (returns <30s with the quit message) + the read-strategy diagnostic. * chore: remove Windows real-profile PROOF workflow + live tests The windows-latest proof E2E and its live/diagnostic tests were branch-only evidence (they proved the deny-all lock + fast-fail contract on a real runner). Per policy proof workflows never land on main. The product fix (fast lock probe + fail-fast message) and its portable unit tests remain in tests/tools/test_browser_real_profile.py. * feat(browser): consented auto-close of a running browser for Windows real-profile [proof workflow do-not-merge] Live Windows CI proved copy-while-running is impossible (Chrome opens the cookie DB deny-all). So to make Windows actually WORK — not just fail cleanly — add opt-in auto-close: browser.real_profile_autoclose (default false). When the profile is locked and consent is on, snapshot_real_profile terminates the browser process tree bound to THAT user-data-dir (psutil, identity+binding verified like the daemon reaper — browser binary AND this exact --user-data-dir in cmdline, fail-closed on ambiguity), waits for the lock to release, then snapshots. Destructive (loses unsaved tabs) so it's off by default and the agent asks first; the fail-fast message names the option. No effect on POSIX. - close_browser_holding_profile: graceful terminate → kill → poll until the cookie DB is openable again (bounded); reports relaunch/tray failure clearly. - _processes_holding_profile: identity+binding matcher (never kills an unrelated same-name process on a different dir). - Config key + docs admonition. Tests: autoclose closes-then-snapshots, autoclose-failure-reports, fail-fast names the option, process-matcher identity/binding. 74 real-profile tests pass. Windows live E2E (PROOF workflow, reverted before merge): autoclose-off fails fast <30s; autoclose-on terminates real Chrome, lock releases, valid cookie DB copied. * test(ci): assert cookie DB at either location + dump Default contents on miss [do-not-merge] Auto-close live test asserted the legacy Default/Cookies path, but modern Chrome writes Default/Network/Cookies. Accept either; on miss, print the copy's Default listing so a real copy gap (vs a path-assertion bug) is visible. * chore: remove Windows real-profile PROOF workflow + live test Branch-only evidence — proved on windows-latest that consented auto-close terminates a running Chrome, releases the lock, and produces a valid profile copy (and that autoclose-off fails fast, not hangs). Per policy proof workflows never land on main. Product fix + portable unit tests remain in hermes_cli/browser_connect.py and tests/tools/test_browser_real_profile.py. * feat(browser): close-with-approval flow for Windows real-profile (toggle arms, agent asks, blocked if still locked) [proof do-not-merge] Refines the Windows path per three requirements: 1. Only when the toggle is set — closing is offered only if browser.real_profile_autoclose is on. 2. Blocked when locked — snapshot_real_profile NEVER kills; a locked profile always returns the [profile-locked] signal and the copy is refused. A later attempt that is still locked blocks again (no loop, no auto-kill). 3. Ask approval to close — closing is an explicit, user-approved step: (new CLI subcommand) runs close_browser_holding_profile only when the agent has the user's OK. The locked error tells the agent to ask first, then run it, then retry. - browser_connect: snapshot blocks with _PROFILE_LOCKED_PREFIX (autoclose-armed message offers the close; off message says fully-quit); no in-snapshot kill. - main.py: subcommand (identity+binding-verified tree kill via close_browser_holding_profile); added to _BUILTIN_SUBCOMMANDS. - browser_tool: surfaces the locked signal + the exact approved-close command. - Docs/config: toggle arms + agent asks + blocked-if-still-locked. Tests: snapshot blocks-not-kills with autoclose on AND off; process matcher identity/binding. 73 real-profile tests pass. Windows live E2E (proof): locked blocks fast without killing → approved close terminates Chrome → snapshot then copies a valid DB; autoclose-off blocks with quit guidance. * chore: remove Windows real-profile PROOF workflow + live test Proved on windows-latest that a locked profile blocks (no kill/hang), the approved close terminates Chrome + releases the lock, and snapshot then copies a valid DB — and autoclose-off blocks with quit guidance. Per policy proof workflows never land on main. Product + portable unit tests remain. * fmt(js): `npm run fix` on merge (#96017) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix(windows): make upstream browser tests portable * fix(desktop): preserve bounded bot hydration budget * fix(tui): fail closed when a named profile's state.db won't open A deferred agent build for a named-profile session swallowed a failed profile-store open (except Exception -> session_db = None), so _make_agent silently bound the launch _get_db() handle and every turn bled into the wrong profile's state.db exactly when the profile store was briefly unopenable. Opening the named profile then looked blank. Route the open through _open_profile_session_db, which raises a clear 'profile session store unavailable' error instead; the deferred build's existing except path turns that into agent_error + an error event, so the user gets a clear failure and no agent turn against the wrong store. Salvaged from #90219 (hardening half), adapted to main's current deferred-build/_transfer_db_to_agent structure. Related to #87723 and #89789. #88532 covered SessionStore only. * fix(tui): _init_session cwd hydration must not fall back to the launch DB Sibling site of the same class fixed in the previous commit: a failed profile-store open during _init_session fell back to _get_db(), hydrating and persisting a named-profile session's cwd row against the launch state.db. Fail closed instead — skip the hydration (log a warning) so nothing ever reads or writes the wrong profile's store. The other profile-store open sites (_db_for_profile, _ensure_session_db_row, _session_db) already degrade to None/skip and were left as-is. * fix(desktop): bound getConnection()/resolveGatewayWsUrl() on every remaining route 20s withTimeout() on the boot() and soft-switch paths (use-gateway-boot.ts), since these are IPC round-trips into the main process with no timeout of their own — a wedged main-process round-trip hangs the awaiting caller forever instead of surfacing a failure. Every other production call site of the same IPC pair was still unbounded: - store/gateway.ts's openSecondary() and sharedPrimaryRoute() — the actual connection-establishment underneath requestGatewayForProfile/Agent, ensureGatewayForProfile/Agent, and every other exported routing entry point that opens a non-primary profile's socket. - use-gateway-request.ts's on-demand reconnect (the primary gateway's "not connected" retry path hit by every RPC). - voice-playback.ts's resolveSpeakStreamUrl(). - api/plugins.ts's activeConnection() (pluginSocket's connect()). Extracted RECONNECT_ATTEMPT_TIMEOUT_MS into the shared lib/with-timeout.ts (previously local to use-gateway-boot.ts) so every call site uses the same budget instead of duplicating the constant. Regression tests mirror the existing use-gateway-boot.test.tsx hang-repro pattern: wedge getConnection()/getConnectionFor() with a never-resolving promise, advance fake timers past the 20s bound, assert the caller settles instead of hanging. Mutation-verified: reverted the production fix (kept tests) and confirmed the 6 new tests fail — 4 by genuinely timing out at the vitest level, 2 by TypeError on the not-yet-exported activeConnection — restored the fix and confirmed all 70 tests across the gateway/voice/boot/ plugins suites pass, with tsc -p . --noEmit clean throughout. * fix: restore the default resolving dial mock in the salvaged #95343 probe test The #92434 mid-handshake pin (added on main after #95343 branched) latches gatewayMocks.connect on a never-resolving mockImplementation; vi.clearAllMocks() clears calls, not implementations, so the salvaged wedged-probe test inherited a dial that never completes and timed out. * fix(desktop): bound the onActiveConnectionInvalidated fallback getConnection() call The registry's active-connection-invalidated fallback re-dial was the one getConnection() await in this file the #93454 bound-every-IPC-round-trip sweep never reached. A wedged main-process round-trip during an eviction fallback (idle reap, connection removal, profile delete) left $connection latched on a promise that never settles instead of rejecting into the existing catch/publish(null) path. * fix(desktop): defer gateway liveness force-close while a turn is in flight (#95327) A wake-path liveness-probe timeout force-closed the primary renderer socket even when the backend was merely busy mid-tool-call; the gateway then saw its client vanish, ws_orphan_reap expired, and the running turn died as a bare "Operation interrupted." placeholder. While any session still reports working, the first inconclusive probe timeout now defers the teardown behind one bounded re-probe; only an exhausted consecutive-failure streak (or no in-flight work at all) rebuilds the transport. The streak resets on a successful ping, a healthy -32601 answer, a clean open, a gateway switch, and unmount. * fix(desktop): claim-guard every remaining ensureRegistryBackend()/ensureBackend() call in Electron main (connectionId, profile) scope, but it was only wired into hermes:connection, hermes:connection:for, and the power-resume pool rebuild. Five other real call sites still invoke ensureRegistryBackend()/ensureBackend() directly: the media-protocol connection resolver, the terminal-pane backend resolver, the ~5s roster-enumeration probe, the connections update-all dispatch, and dispatchRegistryApiRequest (every registry-scoped hermes:api REST call). ensureRegistryBackend() has a genuine await-then-check race (reuseMatchingPrimarySshBackend before the pool entry check-and-set), so any of these five racing a guarded dial for the same scope can each bootstrap their own SSH tunnel / remote dashboard for the same connection — exactly the symptom class #90812 was written to prevent, just reached through an unguarded path instead of two renderer windows. Left the ensureRegistryBackend() self-call inside its own dispatch-time health-probe reconnect branch (registryDispatchRevalidation) unguarded — that recursive path has its own coordination semantics and touching it without modeling reentrancy against the newly-guarded entry points here is a separate, riskier change better done on its own. * test: widen the update-all scan window for the salvaged #95606 wiring pin main's hermes:connections:update-all handler grew (renderer-side exclusions + the managed-SSH dispatch branch) after #95606 branched, pushing the claim- guarded dial past the 2,000-char slice. * fix(desktop): retain local startup profile across SSH switches * fix(desktop): hide unreachable same-name Bot Mode roster twins Group rooms persist source-qualified members. After Desktop switches to the built-in This-device source, a dead loopback row still listed next to the live profile and looked like a second agent. Collapse only the sidebar tiles; $lastRoster, group seats, and mentions keep every (connectionId, profile) identity. * test: flush the prune-lease dial by condition, not by hop count #95343's withTimeout wrapper added an await hop to openSecondary's dial, breaking the sibling test's exactly-two-Promise.resolve flush. Condition- bounded flushing survives future hop-count changes. * chore: map contributor email for kvnloo (#95173) * fix(gateway): isolate control routes from default executor * fix(tui): adopt live compression config on the next Desktop/TUI turn Co-authored-by: Cursor <cursoragent@cursor.com> * fix(compaction): exclude operational notifications from tail anchor and auto-focus (#92703) Kanban/background completion wakes persist as role=user rows typed with display_kind="internal_notification" (the synthetic-wake path in run.py). The model-payload builder already strips display_kind before the request and is_user_originated_turn already ignores it, but two compaction scans still treated those rows as real user turns: - _is_actionable_user_turn (tail anchor) only checked role/content, so a notification became the protected 'last user turn' the compressor keeps. - _derive_auto_focus_topic only skipped synthetic compression turns, so operational notices leaked into the compact focus hint. Both now exclude display_kind-typed rows, mirroring the existing is_user_originated_turn exclusion. No schema change; cache- and role-alternation-safe. Behavior-contract tests feed 1,000 operational notifications around one human turn and assert they never anchor the tail, become the auto-focus source, or count as actionable user turns. Fixes #92703 * fix(cli): repair interrupted update fleet restart An interrupted hermes update after git pull advanced HEAD never restarted running gateways, and the next update said "Already up to date" and skipped the fleet. Persist a HERMES_HOME fleet_restart_pending marker after HEAD moves, clear it only when restart completes (or nothing was running), and catch up on the next hermes update even when git is current — also when latest.json records a stale runtime SHA. Co-authored-by: GokayAI <gokay-ai@users.noreply.github.com> * fix(desktop): widen pool keepalive-fresh window to absorb WSL2 IPC stalls (#95189) The renderer pings each pool backend every 60s (`hermes:backend:touch` → `touchPoolBackend` → updates `lastActiveAt`). The LRU eviction cap used a keepalive-fresh window of 90s — only 1.5× the ping cadence — to decide whether a backend was "plausibly still alive". One missed or delayed ping pushed a live backend past the threshold and the cap-driven eviction killed the active profile's backend mid-session, restarting the gateway and re-minting runtime ids. On WSL2, where the renderer→Electron IPC roundtrips through 9p, brief 9p hiccups commonly stretch a ping to seconds of observed silence, producing the ~80–90s exit / ~2 min cycle reported in #95189 (122 gateway starts on 2026-08-26 alone, driving renderer OOM via reconnect churn at ~5GB/day). Widen POOL_KEEPALIVE_FRESH_MS to 4 minutes (3× ping cadence + IPC stall headroom, still bounded well below POOL_IDLE_MS=10min). Backends with one or even two missed pings are now spared; truly idle backends (multiple lapses, minutes idle) remain eligible for eviction by the cap and the idle reaper. The constant is also overridable via HERMES_DESKTOP_POOL_KEEPALIVE_FRESH_MS to make this tunable without a rebuild. * feat(desktop): Managed updates section drives per-connection SSH updates Slim renderer UI for the managed SSH remote update engine (#95942), adapted from #93042's renderer unit with the deferred canary/rollout scope stripped. Adds a per-connection store (idle/updating/terminal states, receipt, managed-update-in-progress busy envelope) and a 'Managed updates' section on the Gateways settings page with an Update button, progress line, and correlated receipt per registered Desktop-managed SSH connection. Fails closed when the Electron main lacks connections.updateManaged. * fix(desktop): Bot Mode model picker always settles and stops remount churn (#95279) The Bots model picker's catalog read rode the bot's own socket with no deadline: a wedged dial left the query pending forever, so the picker spun indefinitely. On top of that every fetch forced refresh:true, bypassing the staleTime cache, so each Bots view remount (tab re-front, dialog reopen, pane visibility flip) knocked the picker back into its loading state and wiped the staged provider/model pick mid-edit. Bound every attempt to 20s (rejection falls through to the picker's existing free-text fallback), drop the forced refresh so the read participates in the cache like every other surface's catalog, and pin the contract with a red->green regression suite. * fix(desktop): switch model after refresh when it leaves the catalog Refresh Models only updated the catalog cache, so the composer kept showing a model that was no longer in the new group list. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(desktop): Bots-mode picker routes guarded model switches through the shared confirm handler The Bots editor's model write (profiles.configure) was the one switch surface that bypassed the data-policy / expensive-model selection guard: a guarded pick (e.g. muse-spark contributor tier) was applied silently, with no confirm flow anywhere — the #95293 remainder after the core picker's confirm handshake landed in use-model-controls. Gateway: profiles.configure now answers confirm_required + confirm_message for a guarded model (same handshake as config.set model) and writes NOTHING until the client resends with confirm_expensive_model: true. Other sections still apply; the pending model section is not reported as failed. Desktop: the confirm flow is extracted out of use-model-controls into one shared applier (lib/guarded-model-switch.ts, exported through the plugin SDK) — warning toast, staleness-guarded Confirm, single confirmed resend, never a retry loop. The core picker and the Bots editor now consume the SAME handler; the Bots editor's Confirm resends only the model section with confirm_expensive_model: true. Fixes #95293 (Bots surface remainder). * fix(desktop): preserve group turn reason codes * 🐛 fix(desktop): include route scope in gateway dial errors * test(desktop): method-aware gateway mock for the refresh-reconcile confirm interaction test The reconcile-to-guarded-model interaction test's requestGateway mock must only answer config.set with the confirm handshake — the panel's model.options read rides the same dispatcher and was eating the first mocked response. * fmt(js): `npm run fix` on merge (#96076) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * ci: give fork test suite enough time * fix(relay): restore voice-note STT — wire media[] MIMEs, message_type "voice", and a User-Agent for CDN downloads (#95274) * fix(relay): map wire media[] → event.media_types; accept message_type voice A relayed voice note arrived as MessageType.AUDIO with media_types=[] — the STT gate (_event_media_is_stt_input) excludes AUDIO unconditionally and its per-attachment MIME rescue was unreachable, so STT never fired and the agent fell back to the "user sent an audio file attachment" context note (live-verified on staging 2026-08-26, Discord + Telegram). Two wire-boundary fixes, both additive within contract_version 1: - "voice" parses to MessageType.VOICE: the enum already had it — pinned by test so a future refactor can't collapse the two. - media[] is now mapped into event.media_types (positional alignment with media_urls; mime-less entries keep their slot as ""). This is what run.py's per-attachment classifiers key off, so EVERY relayed attachment — image vs document, audio vs voice — now routes like its native-adapter equivalent, not just voice notes. Behaviour pinned: new-connector voice → STT-eligible; legacy audio-typed events unchanged (no STT); music uploads never STT-eligible (direct _event_media_is_stt_input assertions on real wire-parsed events, not mocks). Pairs with the gateway-gateway PR that puts "voice" on the wire. * review: pin the STT gate by test; fail safe on media/media_urls mismatch Addresses independent review of #95274. 1. The PR's acceptance criterion is STT ROUTING, but no committed test called _event_media_is_stt_input — it was only asserted ad-hoc. Adds TestSttGate: voice→eligible, voice-without-media_types→eligible (the new-connector/old-gateway shape), legacy audio-typed voice note→not eligible, music→not eligible. Mutation-verified: removing the VOICE branch from the gate turns these RED. 2. media_urls and media[] are INDEPENDENT wire fields that consumers index by the same i. Mapping MIMEs positionally without checking agreement means a disagreeing producer misassociates a MIME with the wrong URL and mis-routes that attachment — strictly worse than no MIME, which degrades safely to message-level classification. _media_types_from_wire() now maps only when the lengths agree, warns and returns [] otherwise. Note for the record: MessageType.VOICE predates this PR and the gate's VOICE branch ignores media_types, so a NEW connector against an OLD gateway ALREADY fires STT. That is desirable, but it is not "unchanged" — the PR body's rollout matrix said otherwise and is corrected. * fix(relay): send a User-Agent on relay media requests (Discord CDN 403) Discord's CDN rejects urllib's default "Python-urllib/x.y" User-Agent with HTTP 403, and RelayMediaClient never set one. Every Discord CDN pass-through download therefore failed; _localize_inbound_media then kept the raw URL (its "a public URL still has value" branch), and the consumer tried to open a URL as a FILE PATH: WARNING gateway.relay.media: relay media download failed for https://cdn.discordapp.com/...voice-message.ogg: HTTP Error 403 INFO gateway.run: Voice transcription failed for https://cdn.discord... : Audio file not found: https://cdn.discordapp.com/... This killed ALL Discord relay media inbound — voice notes, images and documents alike — not just the voice lane. Telegram/WhatsApp were unaffected because their media is connector-re-hosted (/relay/media/{id}, fetched from our own host) and localizes to real /tmp paths. Reproduced from a clean shell against a live CDN URL: curl (own UA) -> 200 urllib, no UA -> 403 Forbidden urllib + descriptive UA -> 200, 14583 bytes, OggS magic Fix: a module-level _MEDIA_USER_AGENT sent on both download() and upload(). upload() only ever targets our own connector so it was not broken, but a single client should identify itself consistently. Validated on staging: hot-patched hermes-agent-stg-test-6698, restarted the gateway service, and Ben's Discord voice note transcribed successfully — zero new 403s and zero new transcription failures after the patch (last 403 predates it). Test is mutation-verified: removing the UA from download() turns it RED while the other five media tests stay green. * fix(relay): keep url↔mime pairing through media localization Addresses a blocking review finding on my own change: mapping media[] into media_types created a POSITIONAL contract that the rest of the inbound path then broke. 1. _localize_inbound_media (adapter.py) filtered media_urls without filtering media_types. Dropping a dead connector re-host is a NORMAL best-effort path, so every surviving attachment inherited its neighbour's mime. Reproduced through the real functions: before urls [.../relay/media/dead, .../kept.png] types [application/pdf, image/png] after urls [.../kept.png] types [application/pdf, image/png] <-- PNG reads as PDF _event_media_is_image(ev, 0) -> False The loop now carries (url, mime) as PAIRS, so a dropped URL drops its mime with it. 2. _media_types_from_wire compared LENGTHS only, which is not alignment: equal-length-but-reordered wire fields were accepted and paired wrongly, and an absent media_urls skipped the check entirely while still emitting types. Resolution is now BY URL (url -> mime lookup over media_urls); an unmatched URL degrades to "" and falls back to message-level classification. Tests: 4 new cases driving the real chain (wire parse -> localization -> run.py classifier), incl. the dropped-first-attachment case the existing localization test could not catch (it builds events without media_types). The obsolete length-mismatch test now asserts the stronger by-url guarantee. Both fixes mutation-verified: reinstating the URL-only filter fails 1 test, reverting to positional resolution fails 3. Relay suite 258 passed; media/voice/stt selection 685 passed; ruff clean; cross-repo integration payload re-verified. * fix(relay): media_types is always one slot per media_url Self-review after two review rounds flagged this bug class in adjacent seams: I checked the function I edited, not every consumer of the parallel arrays I created. Grepping ALL writers found a third instance. merge_pending_message_event (gateway/platforms/base.py:2725-2735) EXTENDS media_urls and media_types together when a second media message merges into a pending one. My mapping could emit a POPULATED media_urls with an EMPTY media_types (an older connector sends media_urls but no media[]), so extend() concatenated lists of different lengths: A urls [old1.png, old2.png] types [] B urls [new.pdf] types [application/pdf] merged urls [old1.png, old2.png, new.pdf] types [application/pdf] -> old1.png reads as application/pdf; the real PDF gets '' Fix: media_types is now ALWAYS len(media_urls), padded with '' — the url-keyed lookup runs even when media[] is absent, and the localizer rewrites the list unconditionally (no short-circuit that could leave a stale/short list behind). Tests: 4 new cases — padding with no media[], the merge shift above driven through the real merge_pending_message_event, localization preserving the invariant while dropping an entry, and normalization of a short/empty media_types arriving from a non-wire source. All mutation-verified: removing the padding fails 4; restoring the guard fails 1. Relay 262 passed; media/voice/stt selection 689 passed; ruff clean; cross-repo integration payload re-verified. * feat(slack): add link unfurl controls * test(slack): cover default and split unfurl behavior * fix(slack): honor unfurl controls for media captions * fix(slack): preserve unfurl controls during streaming * feat(relay): stamp slack unfurl_links/unfurl_media onto outbound frame metadata (gateway-directed) Relay-fronted Slack reads platforms.relay.extra.slack.unfurl_links/unfurl_media and stamps explicit booleans onto the frame metadata; the connector forwards them to chat.postMessage with no config of its own (mirrors reply_in_thread). Covers send, send_for_platform (cron/scheduled), and send_media lanes. * chore(contributors): map potatosaladx@gmail.com to potatosalad (attribution for salvaged #79436/#81128 commits) * fix(relay): coerce string unfurl knobs and disable Slack draft streaming Live staging (Coatue Slack): - hermes config set / Railway knobs persist "true" as a string; bots that omit unfurl_links do NOT inherit the human default, so dropping the string looked like suppression. - chat.startStream cannot carry unfurl_*. Native SlackAdapter already falls back to chat.postMessage; the relay now matches. * fix(slack): coerce string unfurl knobs on the native plane Relay-plane parity: hermes config set / Railway persist YAML booleans as strings, and _slack_unfurl_kwargs silently dropped them — so 'unfurl_links: "false"' was a no-op on native while working on relay. Coerce recognized string booleans exactly as _slack_unfurl_hints does; unrecognized values still drop so junk config keeps Slack's default instead of accidentally suppressing previews. Replaces test_send_ignores_non_boolean_unfurl_options (which froze the dropped-string behavior) with coercion + junk-drop tests. * fix(relay): fall back to descriptor platform for unfurl stamping The send and send_media lanes resolved the platform only from _platform_by_chat, which is empty until an inbound frame arrives (e.g. after a gateway restart). A proactive send to a Slack chat then missed the unfurl stamp. Mirror the streaming gate and delivery resolver: fall back to the negotiated descriptor's platform. * docs(slack): note caption ordering and streaming fallback for unfurl knobs When either unfurl key is set, media captions post as a separate message before the file (the upload API cannot carry unfurl controls) and native draft streaming falls back to edit-based delivery. Surface both side effects in the config reference table. * test(relay): cover media-lane unfurl stamping and descriptor fallback The send_media lane (08b95c3) had no committed regression test: cover explicit-bool stamping on media frames, the descriptor-platform fallback when _platform_by_chat is empty (post-restart proactive sends), and the omitted-key absence case. * test: isolate Windows updater fixtures * Revert "Merge pull request #94245 from kshitijk4poor/feat/gw-event-replay" This reverts commit df7d7f6e8d6af9a230f9a2b9e265b63062f003cb, reversing changes made to 1a66134404b891170e953f51662e6429f0b7b5a9. * fix(config): preserve lossy decimal values as strings * ci: allow change detection on deep histories * fix(desktop): stabilize cross-platform Windows tests * chore: normalize contributor mapping line ending * fix(compression): suppress duplicate completion notices * fix(compression): preserve terminal lifecycle for lock skips * test(compression): cover failed in-place split status * refactor(compression): fold review follow-ups on #71488 salvage - Reuse the existing _commit_status variable for the terminal-edge gate instead of the parallel _compaction_succeeded boolean (derived state). - Give the commit_fence_cancelled abort the same force_terminal=True terminal edge as the lock-contended abort, and reword the closure comment that overstated the lock contender as 'the one exception'. - Inline the codex app-server path's lifecycle closure: after gating on success it reduced to a single success-site emit, so the scaffolding (done-flag + closure + two no-op failure-path calls) was dead. * test(desktop): use Windows path semantics in platform fixture * test(tui): wait for status timer re-render * fix(hermes_cli): stop config set/unset from wiping user overrides on invalid YAML Fail closed when config.yaml is unparseable or non-mapping before set/unset writes, reuse the readable-config guard to return the parsed mapping, and cover refuse/empty-mapping paths with regression tests. * test(hermes_cli): align malformed YAML set/unset expectations with RuntimeError * fix(hermes_cli): surface fail-closed config write refusals cleanly Follow-ups to the salvaged #71385 guard (which raises RuntimeError from require_readable_config_before_write on unparseable / non-mapping YAML): - config_command: catch RuntimeError for set/unset and print a clean one-line error + exit(1) instead of a raw traceback on the primary 'hermes config set/unset' CLI path. - console_engine._capture_output: convert escaping RuntimeError into a ConsoleCommandError so 'hermes console' and the dashboard console report the refusal instead of crashing the REPL/websocket session. - _warn_config_parse_failure: add a dedicated 'refuse-write' wording branch — the old fallthrough claimed 'falling back to default config' even though the write was refused and the file preserved. - approval_mode: update the stale SystemExit-only comment. - Regression tests for the console path and both config_command paths. * test(hermes_cli): make test_default_path pass on native Windows TestGetHermesHome.test_default_path asserted ~/.hermes unconditionally, but the native Windows default is %LOCALAPPDATA%\hermes (see hermes_constants._get_platform_default_hermes_home). Branch the assertion by platform so the test passes everywhere. Salvaged from PR #96003 by @Aoshi-Dev (the parse-guard half of that PR was superseded by #96169); authorship preserved. * chore: map paulapsp157@gmail.com -> Aoshi-Dev (PR #96003 salvage) * fix(mcp): un-invert the stdio children liveness check (#94335) _stdio_children_dead returned True ('all children dead') on the first LIVE pid — the intended False was dead code right below it. Every spawn path that captures child PIDs (observed in hermes -z oneshots) then failed the #81995 pre-call fast-fail with 'TimeoutError: MCP stdio subprocess ... has exited' on every tools/call while the subprocess was demonstrably alive. Long-lived gateway/dashboard sessions were unaffected only when _stdio_child_pids was empty (the not-pids short-circuit). Return False on the first live pid and drop the unreachable line. * test(mcp): absorb watcher-consumer and fail-open liveness cases from #94521/#94661 Apply-ready delta distilled by @andrexibiza: deterministic watcher-consumer tests (watcher times out while a child is alive, resolves when all are dead), psutil-unavailable fail-open pin, and probe-failure fail-open handling in _stdio_children_dead (unknown is never proof that every child exited). Local: 8 passed on tests/tools/test_mcp_stdio_children_dead.py * fix(cron): tree-kill script timeout descendants via agent.deadline.kill_process_tree The script-timeout path used a site-local process-group kill, which cannot reach a grandchild that created its OWN session (start_new_session background jobs, watchdogs). Such descendants kept running after the job reported failure (#71148, #59549). Migrate the timeout handler to the unified deadline layer's kill_process_tree (#85147, d6a5cb9725): psutil snapshots the descendant set before signalling, so own-session grandchildren are reached too. Fallback to the site-local group kill if the import ever fails, so the path cannot re-wedge. The explicit script-timeout message stays the classification anchor (#85536's contract), keeping cron timeouts distinct from provider timeouts. Salvage additions on review (#85125 Phase 4a): - migrate the sibling kill site too — the cancel_event/"ownership was lost" path orphaned setsid grandchildren the same way (whole-bug-class rule); pinned by test_cancel_path_also_tree_kills - proc.poll() early-return in _terminate_cron_script_tree so a script that exits right at the deadline doesn't log a spurious "no signal" warning (mirrors _terminate_cron_script_process); pinned by test_already_exited_proc_is_left_alone - acceptance test's script timeout 1s -> 2s: interpreter startup under CI load could eat the whole 1s window before the spawner wrote its pid file - note: kill_process_tree hard-kills (SIGKILL) immediately, whereas the old path gave a 1s SIGTERM grace window; intended for a deadline- expiry hard stop (both docstrings say "hard stop") Based on #86791 by @ayushnangia; cherry-picked to preserve authorship. Co-authored-by: dante32683 <dante32683@users.noreply.github.com> Co-authored-by: supotato-ipj <supotato-ipj@users.noreply.github.com> * fix(approval): enforce explicit timeout on smart-approval guardian call and log its outcome The smart-approval guardian (`_smart_approve`) gates every flagged terminal command with a synchronous auxiliary LLM call, but it never passes `timeout=` and logs nothing on the normal path. In production a stalled provider response silently froze the agent turn for 62 minutes with zero log output; the gateway kill-switch eventually fired, and only an unrelated error surfaced afterwards (#82846; watchdog-style fix in #72500). The call was invisible by design — nothing logs at the hang point. Changes in tools/approval.py: - Resolve the same configured timeout the client would use internally (`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and pass it explicitly to `call_llm`, so the deadline cannot be lost if the internal default resolution changes or is misconfigured. - Log the assessment call and its duration (DEBUG), and promote the failure branch from DEBUG to WARNING with elapsed time + exception class, so a wedged guardian call is visible in the logs instead of silent. - Failure still returns "escalate" (fail open to the human/pattern gate) — behavior unchanged, observability only. Complements #72500 (watchdog hard ceiling) rather than duplicating it: explicit timeout is the root-cause hardening, logging closes the sil…

…ll and log its outcome The smart-approval guardian (`_smart_approve`) gates every flagged terminal command with a synchronous auxiliary LLM call, but it never passes `timeout=` and logs nothing on the normal path. In production a stalled provider response silently froze the agent turn for 62 minutes with zero log output; the gateway kill-switch eventually fired, and only an unrelated error surfaced afterwards (NousResearch#82846; watchdog-style fix in NousResearch#72500). The call was invisible by design — nothing logs at the hang point. Changes in tools/approval.py: - Resolve the same configured timeout the client would use internally (`auxiliary.approval.timeout` via `_get_task_timeout("approval")`) and pass it explicitly to `call_llm`, so the deadline cannot be lost if the internal default resolution changes or is misconfigured. - Log the assessment call and its duration (DEBUG), and promote the failure branch from DEBUG to WARNING with elapsed time + exception class, so a wedged guardian call is visible in the logs instead of silent. - Failure still returns "escalate" (fail open to the human/pattern gate) — behavior unchanged, observability only. Complements NousResearch#72500 (watchdog hard ceiling) rather than duplicating it: explicit timeout is the root-cause hardening, logging closes the silence gap; the watchdog remains the safety net if the SDK-level timeout itself is defeated. Tests: explicit timeout forwarded to call_llm (revert-fails), failure logs WARNING + escalates. 49 approval-adjacent tests pass; one unrelated test_approval.py failure is pre-existing (fails on clean main too).

swifmo added 2 commits July 27, 2026 01:25

alt-glitch added type/bug Something isn't working tool/terminal Terminal execution and process management area/auth Authentication, OAuth, credential pools P2 Medium — degraded but workaround exists labels Jul 27, 2026

teknium1 reviewed Jul 30, 2026

View reviewed changes

swifmo changed the title ~~fix(terminal,approval): stop internal-timeout hangs from freezing the agent turn forever~~ fix: stop internal-timeout hangs from freezing the agent turn or worker forever Jul 31, 2026

yubingz mentioned this pull request Aug 4, 2026

MCP tool calls with malformed parameters cause indefinite hang with no fallback or circuit breaker #78260

Open

alt-glitch mentioned this pull request Aug 10, 2026

Smart-approval auxiliary LLM call has no enforced timeout — a stalled provider response wedges the entire agent session indefinitely #82846

Closed

fmercurio mentioned this pull request Aug 10, 2026

fix(agent): recover silent streams and scoped cleanup #83355

Open

yflmq001 mentioned this pull request Aug 11, 2026

fix(approval): enforce explicit timeout on smart-approval guardian call and log its outcome #84125

Closed

kshitijk4poor mentioned this pull request Aug 14, 2026

Tracking: unified deadline layer — architectural fix for the timeout/hang backlog (4 phases) #85125

Closed

1 task

kshitijk4poor mentioned this pull request Aug 24, 2026

fix(terminal): bound internal-timeout hangs — salvage #72500 (#85125 2e triage) #94182

Closed

kshitijk4poor closed this Aug 24, 2026

kshitijk4poor reopened this Aug 25, 2026

This was referenced Sep 3, 2026

fix(approval): isolate long command classification #101965

Open

fix(terminal): break exact retries after timeouts #101966

Open

Navigation Menu

Uh oh!

fix: stop internal-timeout hangs from freezing the agent turn or worker forever - #72500

fix: stop internal-timeout hangs from freezing the agent turn or worker forever#72500
swifmo wants to merge 5 commits into
NousResearch:mainfrom
swifmo:pr-terminal-watchdog

swifmo commented Jul 27, 2026 •

edited

Loading

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 30, 2026

Uh oh!

teknium1 Jul 30, 2026

Uh oh!

teknium1 Jul 30, 2026

Uh oh!

kshitijk4poor commented Aug 24, 2026

Uh oh!

kshitijk4poor commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

swifmo commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. tools/terminal_tool.py — foreground env.execute() hang

2. tools/approval.py — smart-approval auxiliary LLM call hang

3. tools/terminal_tool.py — environment creation and its lock

4. tui_gateway/slash_worker.py — HermesCLI(resume=...) construction hang

Testing

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

kshitijk4poor commented Aug 24, 2026

Uh oh!

kshitijk4poor commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

swifmo commented Jul 27, 2026 •

edited

Loading

1. `tools/terminal_tool.py` — foreground `env.execute()` hang

2. `tools/approval.py` — smart-approval auxiliary LLM call hang

3. `tools/terminal_tool.py` — environment creation and its lock

4. `tui_gateway/slash_worker.py` — `HermesCLI(resume=...)` construction hang