fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup by halaprix · Pull Request #65422 · NousResearch/hermes-agent

halaprix · 2026-07-16T06:45:52Z

What

Adds a startup-time sweep that closes session rows orphaned by a dead gateway process, fixing the phantom "active" rows described in #65194.

hermes_state.py — SessionDB.sweep_orphaned_sessions(max_idle_seconds, sources=("tui","desktop","subagent"), exclude_ids=()). One BEGIN IMMEDIATE write: SELECT candidates, then UPDATE with the same dual-clock predicate so a sibling cannot sneak activity between the check and the close. Stamps end_reason='startup_orphan_reap'.
tui_gateway/server.py — once-per-process _schedule_startup_orphan_sweep(), delayed by the existing WS-orphan grace window so a client reconnecting after restart can session.resume first. Live in-memory sessions are excluded. Gated by sessions.orphan_reaper (default on) and HERMES_TUI_SESSION_TTL_S (0 disables).
tui_gateway/entry.py + tui_gateway/ws.py — both gateway entry points schedule the sweep. entry.main() covers the stdio TUI; handle_ws covers the desktop app and web dashboard WS sidecar (which never run entry.main()). The once-guard makes the second site a no-op.
hermes_cli/config_defaults.py / cli-config.yaml.example / configuration.md — documented sessions.orphan_reaper flag.

Why

The ws-orphan grace timer (_schedule_ws_orphan_reap) and the idle reaper are in-process threading.Timer/thread state. When the gateway restarts (update, crash, systemd) before they fire, disconnected TUI/desktop/subagent rows stay ended_at IS NULL forever and accumulate as phantom "active" sessions in /resume and dashboards. Nothing re-checked stale rows on the next boot — this adds that missing startup-time complement.

Design notes:

Both timestamps must be stale on purpose: message-recency alone would sweep a freshly created compression/branch child that carries old copied message timestamps; started_at alone would sweep a long-lived session that is still actively talking. Empty rows fall back to started_at via COALESCE.
Desktop is in the default allowlist. Desktop chat sessions persist as source="desktop" and use the same source-agnostic Timer path, so a desktop-only restart left the same orphan class.
Messaging-gateway sessions are never touched. Positive source IN ('tui','desktop','subagent') allowlist, so telegram/discord/… rows cannot be ended by the TUI stack (avoids the ws_orphan_reap kills gateway-originated sessions, causing Groundhog Day routing loop #60609 "Groundhog Day" routing loop).
Live in-memory sessions are excluded so a session.resume that lands during the startup grace window is not closed out from under the reconnecting client.
First end_reason wins — the ended_at IS NULL guard means real end reasons are never rewritten, and a second boot is a no-op.
Swept rows stay resumable — resolve_resume_session_id doesn't filter by end_reason.
The distinct end_reason='startup_orphan_reap' keeps the new sweep distinguishable from the runtime timer's ws_orphan_reap.

Thanks to @hansai-art — #65478 called out the WS-sidecar scheduling site and desktop source coverage; both are folded in here.

How to test

Reproduction from the issue: start a TUI/desktop/subagent session, kill the gateway before the ~20 s grace timer fires, then SELECT * FROM sessions WHERE ended_at IS NULL — the row previously stayed open forever. With this change, the next gateway boot (stdio or desktop/dashboard WS) closes it once it is TTL-stale.

Automated:

scripts/run_tests.sh tests/hermes_state/test_sweep_orphaned_sessions.py tests/tui_gateway/test_startup_orphan_sweep.py

Platforms tested

Linux (x86_64). The change is pure SQLite + stdlib threading — no platform-specific I/O.

Fixes #65194

teknium1

Thanks for adding a conservative startup-time complement to the in-process websocket reaper. The current-main premise is valid: tui_gateway/server.py:765-792 uses a process-local threading.Timer, while tui_gateway/entry.py:293-355 has no equivalent startup sweep.

Problems

The new default source allowlist omits desktop (hermes_state.py:2939 in this PR). Desktop chat sessions use this same gateway and are persisted as source="desktop" when HERMES_DESKTOP=1 (tui_gateway/server.py:2150-2167; session.create resolves the source at tui_gateway/server.py:5226). The websocket disconnect path schedules the existing orphan reaper without filtering source (tui_gateway/server.py:795-829), so a desktop restart before its Timer fires still produces an unswept open row.

Suggested changes

Include desktop gateway sessions in the startup eligibility policy without widening into messaging-owned sources.
Add a stale-desktop startup-sweep regression test.

Automated hermes-sweeper review.

hansai-art · 2026-07-19T13:42:43Z

I opened #65478 for the same bug about eighty minutes after this one, before seeing it. Yours is the better-formed PR: the configurable sources tuple, the both-clocks staleness predicate, and the cli-config.yaml.example / configuration.md entries are all things mine either lacks or had to be told to add. I am not going to argue for mine over yours.

One thing mine covers that this PR does not, offered here so it does not get lost whichever way the maintainers go: the sweep needs a second scheduling site in tui_gateway/ws.py.

This PR hooks tui_gateway/entry.py:main(). That is the stdio TUI path. The desktop app and the web dashboard reach the agent through the WS sidecar instead, which hermes_cli/web_server.py:17357-17359 serves by importing handle_ws from tui_gateway/ws.py and awaiting it directly. entry.main() never runs on that path, so a desktop-only or dashboard-only user would never trigger the startup sweep.

This is not a hypothetical. The repo already hit exactly this shape once, and the existing test comment at tests/test_tui_gateway_ws.py:12-13 records it:

sidecar, not through tui_gateway.entry.main() (which spawns the discovery thread for the stdio TUI). handle_ws must start discovery itself, otherwise [...]

MCP discovery needed the same dual hook for the same reason. The fix is the same shape here, roughly:

# in handle_ws, after the MCP discovery kickoff
try:
    server._schedule_startup_orphan_sweep()
except Exception:
    _log.warning("startup orphan sweep scheduling failed", exc_info=True)

with the once-per-process guard and the config gate living inside the scheduler so the second call site is a no-op when entry.main() already ran.

Related: if you take the sources tuple route, desktop is worth considering alongside tui and subagent for the same reason, since those rows are created on the sidecar path too.

Happy for this PR to be the one that lands. I have left #65478 open only because of the ws.py piece and have noted the duplication there; if you would rather fold it in here, take it, no attribution needed.

GottZ · 2026-08-03T18:14:29Z

This was generated by AI during triage.

Summary

Two PRs address #65194 with startup sweeps for session rows left open when the process-local websocket orphan timer dies. Both use start-time and latest-message staleness checks, while #65478 additionally covers desktop sessions, excludes live in-memory sessions, and schedules the sweep from both the stdio and WS-sidecar paths.

Related pull requests

fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup #65422 partial — (+381/-0) — n/a: Adds a configurable and documented dual-clock sweep for stale TUI/subagent rows, but its source allowlist omits desktop and its scheduling is limited to entry.main(), leaving the desktop/dashboard WS-sidecar path uncovered. Despite the contributor keep_open review on fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup #65422, close it as a duplicate of fix(tui): sweep orphaned TUI/subagent session rows at gateway startup #65478 because fix(tui): sweep orphaned TUI/subagent session rows at gateway startup #65478 contains the same core sweep plus those missing paths.
fix(tui): sweep orphaned TUI/subagent session rows at gateway startup #65478 best fix — (+377/-0) — n/a: The recorded best fix adds a transactionally rechecked dual-clock sweep for tui, desktop, and subagent rows, excludes live in-memory sessions, and schedules it from both entry.main() and the WS sidecar. Its updated diff addresses the contributor keep_open review by protecting fresh rows with copied old history and adding the requested regression test.

Duplicates

#65422 and #65478 implement substantially the same startup orphan-session sweep for #65194; #65422 is superseded by the broader #65478 implementation.

Suggested consolidation

Keep #65478 open with a salvage path: preserve its desktop source coverage, WS-sidecar scheduling, live-session exclusions, transactional staleness recheck, and fresh-row regression test, while allowing author action to split or rebase that focused implementation as needed. Close #65422 as a duplicate of #65478; this departs from its keep_open review because the visible diff still omits desktop and WS-sidecar coverage that #65478 supplies.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I65194(["issue #65194 (open)"])
    subgraph Dup65422 ["PRs duplicating each other"]
        P65422["PR #65422 (open)"]
        P65478["PR #65478 (open)"]
    end
    P65422 -.->|partial| I65194
    class I65194 open
    class P65422 open
    class P65478 open
    class P65478 best
    class P65422 target
    click I65194 "https://github.com/NousResearch/hermes-agent/issues/65194"
    click P65422 "https://github.com/NousResearch/hermes-agent/pull/65422"
    click P65478 "https://github.com/NousResearch/hermes-agent/pull/65478"

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 2 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 38 kB of PR diffs, 19 kB of issue/PR text, 8 kB of discussion (5 comments), 3 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

… startup Close session rows left ended_at IS NULL when the in-process websocket orphan timer dies with the process (NousResearch#65194). Dual-clock staleness (started_at AND newest message), desktop included, live in-memory sessions excluded, scheduled once from both entry.main and the WS sidecar so desktop/dashboard boots also run the sweep.

halaprix · 2026-08-19T07:45:46Z

Refreshing the PR head after rebasing onto current main and folding in desktop + WS-sidecar coverage. Reopening immediately.

halaprix · 2026-08-19T07:46:26Z

Thanks everyone for the careful reads — especially @GottZ for the triage write-up and @hansai-art for laying out the missing pieces so clearly.

The earlier snapshot of this PR was incomplete in the ways the triage called out: default sources omitted desktop, and the sweep only ran from entry.main(), so a desktop/dashboard-only process never triggered it. That assessment is now obsolete.

This branch is rebased onto current main and now includes:

desktop in the default allowlist alongside tui / subagent, with a dedicated stale-desktop regression test (the @teknium1 review note).
Both gateway entry points. entry.main() still schedules the sweep for the stdio TUI; handle_ws schedules it for the desktop app and web dashboard WS sidecar. Once-per-process + config-gated inside, so the second site is a no-op when the first already ran. @hansai-art — this is the hook you described; thank you for offering it so generously.
Live in-memory sessions excluded, so a session.resume that lands during the startup grace window is not closed out from under a reconnecting client.
Transactional dual-clock recheck in one BEGIN IMMEDIATE write (SELECT + UPDATE with the same started_at AND newest-message predicate), so a sibling cannot sneak activity between the check and the close. Fresh rows that carry copied old history stay spared.
Config/docs stay on the existing sessions.orphan_reaper key in DEFAULT_CONFIG (default on), rather than a new ad-hoc top-level flag.

@hansai-art, #65478 can close as a duplicate of this one whenever you like — the coverage you added is folded in here, and I'm grateful you left the notes instead of turning it into a contest.

Happy to take any further review.

…to dashboard.* Follow-up to the #65422 salvage: - startup_orphan_reap joins _RECOVERABLE_END_REASONS (kept distinct from ws_orphan_reap for forensics): every recovery fence (find_latest_gateway_session_for_peer, unarchive_recoverable_session, promote_to_session_reset) now treats a startup-swept row as an accidental end, so a sweep never makes a session unresumable. - Config key moves from sessions.orphan_reaper to dashboard.startup_orphan_sweep in DEFAULT_CONFIG, next to its siblings ws_ping_interval / ws_ping_timeout / ws_orphan_reap_grace_s; the raw loader in tui_gateway.server reads the new key (fail-open on missing). cli-config.yaml.example and website/docs/user-guide/configuration.md follow the dashboard.* documentation pattern. - New regression test: a stranded 'active' row (ended_at NULL, no live runtime) is swept AND still recoverable via peer-keyed lookup and fully revivable via reopen_session afterward.

teknium1 · 2026-08-24T01:59:11Z

Merged via ring-2 consolidated PR #93430 — your commit cherry-picked with authorship preserved (once-per-process startup sweep of orphaned session rows; fixes #65194). We added a follow-up on top: startup_orphan_reap joined the recoverable end reasons (so crash-orphaned sessions stay resumable, matching the recovery semantics that landed in #93361 after your PR was filed) and the config key moved to dashboard.startup_orphan_sweep. Live-verified: a backdated stranded row was swept AND resumed via a real RPC afterward. Thanks @halaprix!

* fix(browser): floor browser-use CLI subprocess PATH with sane system dirs Profile-spawned workers (kanban bots, cron jobs) can inherit a PATH of only version-manager dirs — observed in the wild as one nvm node dir repeated 7x. The uv-installed browser-use binary is a POSIX sh trampoline that resolves dirname/realpath through PATH, so it died with 'realpath: not found … exec: /python: not found' (exit 127) before its own Python ever started. _base_subprocess_env now floors the child PATH via browser_tool's _merge_browser_path (the agent-browser backend already guards the same hazard), degrading to appending FHS bin dirs if that import is ever unavailable. Windows is a no-op (.cmd shims don't trampoline). Verified: unit tests + real uvx browser-use --version under a nvm-only-PATH worker env, rc 127 -> rc 0. * fix(bots): protect new drafts from title sweep * test(bots): pin draft sweep age boundary * chore: add contributor email mappings for salvage * installer: check for a C++ compiler before building native Node modules npm install inside install_node_deps() builds native addons (e.g. node-pty) via node-gyp, which needs a C/C++ compiler. That was never checked, so a missing g++ only surfaced as a generic "npm install failed or timed out" deep inside npm's own output — and because install_node_deps failing short-circuits the rest of main() via `|| return`, users end up with no `hermes` command and no clue why. * fix(install): refresh Playwright upgrade for current main Reapply the Playwright dependency update from NousResearch/hermes-agent#77773 and regenerate package-lock.json against current main. This fixes the Chromium installation hang under Node 26. Fixes NousResearch/hermes-agent#76312 Supersedes NousResearch/hermes-agent#77773 * fix(install): actually invoke check_cxx_compiler in both install stages Salvage follow-up for #88993: the preflight was defined but never called from the prerequisites stage or the full-install path, so the Fedora node-gyp failure (#93063) would still occur. Wire it in after check_node in both sequences. * chore: add contributor email mappings for salvage * chore(tests): remove the never-executed kanban stress/chaos suite tests/stress/ was dead weight: its own conftest set collect_ignore_glob = ["*.py"], so pytest has never collected a single file from it, the advertised --run-stress flag was a permanent no-op, and no CI workflow ever invoked the scripts (#93135). Rather than wire a nightly lane for scripts that were never verified end-to-end, remove the suite. Kanban concurrency behavior remains covered by the regular tests under tests/hermes_cli/ and tests/gateway/. Closes #93135. * fix(desktop): authenticate remote liveness probes * fix(desktop): synchronize applied gateway registry * test(desktop): replace apply/liveness source-regex assertions with behavior tests The salvaged hardening tests matched main.ts source text to assert that fetchConnectionStatus reaches for a bearer and that Apply preflights before persisting. A rename breaks them while a real auth regression that keeps the substrings passes. Make the preflight a first-class option on applyConnectionConfigAtomically so its ordering is observable, and assert it through the seam: preflight runs before either write, and a rejected preflight leaves both stores and the activation untouched. * fix(desktop): boot-time source restore keeps the All-profiles preference (#93197) The showAllProfiles browse-mode flag is persisted to localStorage, but every restart it was force-collapsed anyway: initializeConnectionsRegistry restores the last-used source via selectConnection, and selectConnection's post-activation path unconditionally ran $showAllProfiles.set(false). That collapse is correct for a user click on the connection picker (a concrete-source action), but the silent boot restore is not a user action. Gate both reset sites on pendingTarget === null && activeConnectionId === null (the fresh-boot state) so the persisted preference survives restart, while any user-initiated switch still collapses browse mode. Regression tests cover both directions: boot restore preserves true, a user switch collapses it. Fixes #93197 * fix(desktop): heal v1/v2 connection drift instead of re-homing onto local migrateV1ToRegistry runs exactly once, only when connections.json is absent. A user who was local at that moment and pointed Settings -> Gateway at a remote afterwards gets a live remote the registry cannot name: the descriptor resolves to no connectionId, primary still says 'local', and the boot-time launch pick force-switches the window onto a fresh local backend seconds after the sessions list paints. That backend has no provider, so onboarding pops. Reconcile on read: when the v1 global route names a remote with no matching registry entry, register it and adopt it as primary/last-used, then persist so the repair happens once. Narrow on purpose — an already-registered route is left alone even when primary names something else, because that is the user's pick in the Connections panel, not drift. Replaces the hand-edit-connections.json workaround users have been trading. * fix(desktop): boot restore never overrides a live unnameable source Reconciliation repairs the drift at its source, but it can still fail to persist (read-only or full userData), which leaves a window live on a source the registry cannot name. $activeConnectionId is null there, the preferred-id guard misses, and the restore re-homes a working connection. Return early when a connection is live but unnameable. The registry has no claim on a source it does not know about. * fix(desktop): show the launch-source preference for a single connection The toggle was gated on having 2+ registered sources, which hid it in exactly the local-only state the drift produces — the state where a user most needs to change what launch restores. * fix(gemini): wrap schema-bearing tool results as opaque text Gemini 3 resolves JSON-Schema $ref/$defs pointers inside a functionResponse.response payload and rejects unknown references with HTTP 400 INVALID_ARGUMENT ('referenced name #/$defs/...' does not match a display_name; see vercel/ai#14369). tool_describe (and any tool whose result is itself a JSON Schema) returns schema text that previously went back as a structured response, tripping Gemini's pointer resolution. Detect such results with a $ref-pointer scan and wrap them as opaque text instead. Adds regression tests for the wrap path and the unchanged structured path. * test(gemini): cover nested/list/non-pointer ref cases; document false-positive tolerance Address review feedback: - Add tests for deeply-nested $ref (recursion), top-level JSON array (already wrapped, no 400 path), and $ref without '#/' prefix (stays structured). - Document the deliberate structural (false-positive-tolerant) detection and its O(n) cost in the helper docstring. * chore: map Aintworth contributor email * fix(compression): structural no-ops defer retries instead of striking the breaker Fixes #93022. A short session (protection window >= transcript) hits the "insufficient messages" / "no compressible window" branches twice and permanently trips the anti-thrash breaker, even though nothing was eligible to compress - compression was never attempted, so there is nothing "ineffective" to score. The session then rides past the threshold with no compaction possible (recovery probes only soften, not fix, the misclassification). Distinguish "nothing eligible right now" from "attempted and underperformed": - New transient _structural_no_op_backoff_until (in-memory, 300s) armed by _record_structural_no_op() at the three structural no-op sites: insufficient_messages, no_compressible_window, empty_post_handoff_window. No strikes accumulate; auto-compaction resumes on its own once the backoff lapses or the transcript outgrows the protection window. - The backoff gates should_compress via _automatic_compression_blocked_locally and surfaces in _compression_block_reason as "structural_backoff:<seconds>". - #40803's frozen-CLI guarantee is preserved: a transcript that can never shrink retries at most once per backoff window instead of every turn. - force=True (/compress) clears an active backoff before attempting; record_completed_compaction() lifts it - both prove the transcript is compressible/being worked. - Genuine attempted-but-underperformed verdicts still strike the durable ineffective counter unchanged. Tests: new tests/agent/test_context_compressor_structural_backoff.py; updated the two tests that asserted the old strike-on-noop behavior. * test(compression): align no-op strike tests with structural backoff (#93022) Two suites still encoded the pre-#93093 contract that the three structural no-op branches (insufficient_messages, no_compressible_window, empty_post_handoff_window) increment _ineffective_compression_count: - tests/agent/test_compaction_anti_thrash.py:: TestMinimumMessagesBranch::test_too_few_messages_records_an_ineffective_pass - tests/run_agent/test_infinite_compaction_loop.py:: TestCompressNoOpRegistersIneffective::{test_no_op_increments_counter, test_two_no_ops_block_should_compress} Structural no-ops are transcript-shape facts, not evidence of an incompressible floor, so they now arm _structural_no_op_backoff_until and leave the strike counter untouched. Update the tests to pin the new contract (count unchanged, backoff armed via time.monotonic(), should_compress blocked while it holds) and rename accordingly. The outcome contract of test_two_no_ops_block_should_compress is preserved: repeated no-ops still block further automatic compression. * chore: map contributor email aniruddhaadak80@gmail.com * test(agent): stop truncation-warning ContextVar leaking between test files Running `pytest tests/agent/test_prompt_builder.py tests/agent/test_system_prompt.py` failed test_build_system_prompt_records_stable_prefix with AttributeError: '...SimpleNamespace' object has no attribute '_emit_status' (#93018). A truncation warning recorded by test_prompt_builder.py stays in the shared thread context under plain pytest, so the later file's build_system_prompt call drains a warning and forwards it to agent._emit_status - which the test stub lacked. Harden both sides: - tests/agent/test_system_prompt.py: _make_agent() stub gains a no-op _emit_status, so draining a stray warning is harmless. - tests/agent/test_prompt_builder.py: autouse fixture drains pending truncation warnings after every test, leaving the ContextVar clean. The order-dependent failure no longer reproduces in either ordering. * test(agent): drain truncation warnings before and after each prompt-builder test Follow-up to the ContextVar-leak fix: the autouse fixture now drains on both sides (drain(); yield; drain()) so earlier files can't pollute this file's assertions either. * fix(cron): keep hermes console script on child PATH * test(cron): e2e regression — scrubbed child env resolves bare hermes under minimal parent PATH Exercises the real build_subprocess_env()/_resolve_hermes_bin_dir chain (no helper mocks) under a simulated systemd/cron minimal PATH, the exact call path cron/scheduler._run_job_script uses. Companion to #93082. * chore: map UniversePeak contributor email * test(cua): pin PATH-preservation contract, not byte equality The CUA spawn-env tests froze PATH == '/usr/bin:/bin' verbatim. _sanitize_subprocess_env now (intentionally) prepends the hermes console-script dir for all sanitized children (#92998), so these assertions flip to the contract: original entries preserved as suffix, hermes bin dir first when prepended. * fix(install.ps1): initialize LastResolver before the resolved-path report ConvertTo-LongPath short-circuits for ordinary long paths (no ~\d alias), so $script:LastResolver is only assigned when a short path actually needs expansion. The ResolvedPathReport block read it unconditionally, which is fatal under Set-StrictMode before any install stage runs (#93017: fresh installs died at line 367 through three different invocation styles). Initialize it to 'none' — the resolver's own value for "nothing ran" — at script scope before Set-LongProfileEnvVars can invoke a resolver. * fix(install.ps1): record 'skipped-long-path' when ConvertTo-LongPath short-circuits The ordinary-long-path early return now records why no resolver ran, so the ResolvedPathReport stays truthful instead of silently inheriting a stale value from an earlier call in the same session. Diagnostics hunk taken from PR #93100. Co-authored-by: aniruddhaadak80 <aniruddhaadak80@users.noreply.github.com> * fix(cron): normalize id-keyed jobs stores on load * fix(cron): self-heal id-keyed jobs.json to canonical list form on load Layer on the load-boundary flatten: when load_jobs() encounters an ID-keyed jobs map ({"jobs": {"<job_id>": {...}, ...}} — written by external tools or hand edits, never by save_jobs()), it now not only flattens to the list contract but persists the canonical {"jobs": [...]} form back to disk via the existing auto-repair path (save_jobs), so the store self-heals and subsequent reads are idempotent. Note: _peek_jobs_unlocked() intentionally does NOT tolerate the dict shape — it returns None so the save path never shrink-merges against an unrepaired baseline. The flatten + repair live only at the load_jobs() boundary. Regression tests cover the flatten, the reported list_jobs() traceback path, idempotent on-disk repair, and the empty-map edge case. Salvaged from PR #92994. Co-authored-by: a-yeyang <88581400+a-yeyang@users.noreply.github.com> * fix(cron): preserve map keys as ids and skip junk values when flattening id-keyed jobs.json Harden the id-keyed-map flatten with an id-preserving merge: {**value, "id": value.get("id") or key} — an inline "id" wins, otherwise the map key is adopted (external tools often key by id and omit the inline copy; plain list(values) would emit id-less records that collide or get dropped downstream). Non-dict junk values are skipped with a warning instead of crashing the load. The self-heal rewrite persists the id-merged, junk-free records. Tests: key adopted when no inline id (and inline id wins over a differing key), non-dict junk skipped with warning + list_jobs survives + self-heal persists only valid records, all-junk map flattens to []. * chore: map wingkwong contributor email * fix(security): see through wrapper prefixes in the gateway lifecycle guards `sudo`, `env`, `nohup`, `timeout` and friends exec their argument tail, so the command that actually runs sits further right. Three guards read only the first token of a segment, saw the wrapper, and never inspected what it runs: bash ~/restart.sh → blocked sudo bash ~/restart.sh → allowed launchctl submit -l com.x -- helper → blocked sudo launchctl submit -l com.x -- helper → allowed Same foot-gun, one word of prefix. That reaches both enforcement points — `cron.jobs.create_job` and `tools/terminal_tool.py` under `_HERMES_GATEWAY=1` — and defeats the label-independent submit block that #62891 added precisely because a persistent helper is the indirect route to a restart loop. `_peel_transparent_prefixes()` walks past a bounded chain of these wrappers, skipping their own options, their value-taking options (`sudo -u deploy`, `stdbuf -o0`), `VAR=value` assignments, a `--` end-of-options separator, and `timeout`'s duration operand, then returns the index of the real command. It is applied to the referenced-script walk, the `sh -c` payload walk, and the `launchctl submit`/`bootstrap` block. In the referenced-script walk the peel is ADDITIVE — the segment is read at the original token and again at the peeled one — because peeling must never remove a reference the un-peeled read would have found. A local script named `./timeout` is a script, not the coreutils wrapper, and consuming it as a prefix would have silently stopped scanning it. (The other two call sites need no such care: no wrapper name is also a shell name or `launchctl`, so peeling there can only add.) That split is why the per-index logic now lives in `_references_at()`. This is not a new reading of shell syntax for this module — `_PIPE_TO_INTERPRETER` already treats `sudo ` as transparent for the pipe case (`... | sudo sh`). This generalises the same reading to the command position. Deliberately NOT applied to the data-sink masking in `_mask_data_sink_arguments`: peeling there would widen an exemption, and the conservative reading is the safe one. No false positives: peeling only changes which token is treated as the command, so a wrapper around ordinary work resolves to a non-shell executable and yields nothing, exactly as before (`sudo apt-get update`, `timeout 60 curl ...`, `nice -n 10 make -j4`, a bare `env`). Tests: 40 new cases in tests/hermes_cli/test_gateway_restart_loop.py — every wrapper form against a script reference, a dot-source, a nested `sh -c` payload and `launchctl submit`, plus the benign wrapped commands, a wrapped clean script, and the `./timeout`-style lookalike names that pin the additive reading. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p * fix(cron): stop a relative path from disabling the data-sink exemption `_mask_data_sink_arguments` exempts lifecycle text living in the arguments of executables that cannot run them (`grep`, `rg`, `journalctl`, `sqlite3`, …), so hunting for a restart string in logs is diagnostics rather than a command. The exemption is dropped when an argument looks like an escape back into execution — including anything starting with a dot, because sqlite3 spells its escapes as dot-commands (`.shell`, `.system`). But `.`, `./x` and `../x` are ordinary path operands, and grep -r 'systemctl restart hermes-gateway' . is the most ordinary recursive search there is. The leading-dot test treated its `.` operand as a sqlite3 escape, disabled masking for the whole segment, and blocked the command outright — the exact false-positive class the exemption exists to prevent, on the shape most likely to hit it. Searching a relative subdirectory (`./logs`, `../archive`) fails the same way, as does a relative sqlite3 database path (`sqlite3 ./stats.db "SELECT ..."`). Require a dot followed by a NAME character (`^\.[A-Za-z]`) so a dot-command still defeats the exemption while a relative path stays a path. A dotfile operand (`.env`) still reads as a dot-command — conservative, and unchanged from today's behavior. This narrows a security guard in the permissive direction, so the escape hatches are pinned explicitly: with a relative-path operand present, `.shell`/`.system`, psql's `\!`, a pipe into `sh`/`bash`/`sudo sh`/`xargs`, command substitution, and a `;`/`&&` continuation all still block. Only the segment's own data arguments are masked, and only when nothing in it can reach execution. Tests: 18 new cases in tests/hermes_cli/test_gateway_restart_loop.py — the relative-path shapes that must now be allowed, plus the ten escape-hatch shapes that must still block. The allow cases fail on the unfixed tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p * fix(security): cover privilege wrappers and command-string options Review follow-up on #84203. Both points reproduce; neither was a regression from the first pass, but both are live bypasses of the same guard. **Privilege and namespace wrappers were missing.** The allowlist covered the coreutils-shaped wrappers but not the privilege ones, so each of these ran a lifecycle script straight past the walk: pkexec bash ~/restart.sh runuser -u root -- bash ~/restart.sh setpriv --reuid=0 -- bash ~/restart.sh systemd-run --scope bash ~/restart.sh nsenter --target 1 --mount bash ~/restart.sh unshare -r bash ~/restart.sh Added `pkexec`, `su`, `runuser`, `setpriv`, `systemd-run`, `nsenter` and `unshare`, each with the value-taking options that would otherwise be mistaken for the command (`nsenter -t 1`, `systemd-run -p X=1`, `runuser -u root`, …). **An option can carry a command STRING, not an argv tail.** `env -S` and `su`/`runuser` `-c` take shell source. The peel treated the operand as an opaque value and skipped it, so `env -S 'bash ~/restart.sh'` was never scanned — the string went unread rather than being recursed into. `_STRING_COMMAND_OPTIONS` now names those options and their values are re-scanned as shell source, the same treatment `sh -c` payloads already get. They are read at the ORIGINAL command token, before the transparent-prefix peel, because peeling past `su`/`env` would discard the very option carrying the command. `--opt value` and `--opt=value` are both handled. Scope, stated plainly: this is an enumerated allowlist, not a general solution to "wrapper that execs its tail". A wrapper outside the set, or a value-taking option outside these tables, still resolves to no reference — that fails open, exactly as it did before this PR, and it is a miss rather than a false block. The reviewer offered "extend the set with tests, or document that the list is heuristic"; this does the first and states the second. Tests: 23 new cases (220 in the file) — every added wrapper against a script reference including the value-operand option forms, both command-string option spellings for env/su/runuser, and the same wrappers around ordinary work (`pkexec systemctl status nginx`, `su -c 'ls -la'`, `env -S 'echo hi'`, `nsenter -t 1 -m ps aux`) which must stay allowed. 15 fail on the tree before this commit. False positives re-checked at scale: the 9,258 command lines from this repo's own scripts and docs give an identical verdict set before and after — 0 new false positives, 0 lost detections, 0 exceptions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cSddnhxiUmdGbgyKnpg8p * fix(desktop): distinguish provider quota exhaustion * fix(agent): honor structured quota reset signals * fmt(js): `npm run fix` on merge (#93429) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix(gateway): adopt stranded bot sessions from the default store on profile resume Pre-#93296, the desktop routed session RPCs by the focused tile, so a profile bot's turns executed on the default backend and its canonical session accumulated in the DEFAULT profile's state.db. Post-fix, the profile backend correctly receives the resume — but its store has never seen the session, so the same chat 4001s forever (unreachable instead of misrouted). Live repro: Teknium's Developer bot, session c93770. - hermes_state_portability: SessionDB.adopt_session_lineage_from() — composes the existing export_session_lineage()/import_sessions() primitives; donor rows are archived (never deleted) with end_reason=adopted_by_profile, which is deliberately NOT in RECOVERABLE_END_REASONS so canonical-lookup resurrection cannot undo an adoption. Idempotent (already-present ids skip). - tui_gateway/methods_session: profile-scoped session.resume falls back to adoption from the default store right before the 4007; ids unknown to BOTH stores still 4007 exactly as before, and launch-profile resumes never consult the fallback. - tests: 10 new (7 unit on the primitive incl. compression-lineage unit adoption + non-resurrectable archive; 3 handler-level through server.handle_request incl. the live repro shape); db-ownership leak test taught that the shared launch handle probe is by design. Follow-up to #93296/#93311; part of #93091. * harden(adoption): review findings — exact-id donors only, divergence guard, honest donor_retired Review batch (3 reviewers) on the final diff surfaced: - H1: title-based donor matching could adopt AND non-recoverably retire an UNRELATED default-store conversation (bot titles collide by design; get_session_by_title has no archived filter/ordering). Donor probe is now exact-id only — the stranded repro always has the id. - H2: re-adoption after a partial run could retire a donor that had accumulated NEWER messages than the profile copy (skip-based idempotency never merges). New divergence guard compares message counts and refuses retirement when the donor is ahead (still adopts). - M1: donor_retired reported True even when every retirement step failed under suppress. Now per-segment tracked + warn-logged; True only when all applied. - M3: adopted=False (e.g. import validation limits) was silent — now warn-logged with import errors. - M4: archived donors are never re-adopted (no cross-profile cloning). - Dead 'from pathlib import Path' dropped; contextlib no longer needed. 5 new red-first-verified regressions (title-collision immunity, archived-donor immunity, non-vacuous owns_db gating with a real donor seeded, divergent-donor retirement refusal, donor_retired truthfulness). tests/tui_gateway: 578 passed. ruff clean. * fix(bot-mode): fail closed on transient group-session resume failures ensureGroupChatSession's resume loop caught ANY session.resume error (stored sid, then title lookup) identically and fell through to session.create — the same bug findExistingCanonicalChat was fixed for hours earlier (87b645f52c) in the same file: a transient failure (the backend still warming up after a restart, a network blip on a cross-connection lookup, an oversized-resume refusal) read as "no session, mint a new one". That forks the member's real session AND silently overwrites room.sessions[key], making the original unreachable from the room. ensureGroupChatSession is actually more exposed than the 1:1 case: it runs every group turn (runGroupChatMemberTurn), with two independent swallow points. Distinguish "genuinely doesn't exist" from "transient failure" the same way the gateway itself does: session.resume's own handler (tui_gateway/methods_session.py) returns JSON-RPC code 4007 only when the target truly isn't found; every other failure (including 4130, "session too large to resume" — a session that DOES exist) now surfaces instead of being silently swallowed. The existing outer try/catch at the call site already treats a thrown error as "this member passes the round" (recordGroupActivity kind: 'failed'), so nothing new needs to catch it — a transient hiccup now costs one skipped round instead of a permanent fork. * fix(tui): log 4001 session-not-found rejections for diagnosability Messages sent into a session whose in-memory runtime was detached on WS disconnect and orphan-reaped vanished silently: _sess_nowait returned 4001 with no log line, so 'request arrived and was rejected' was indistinguishable from 'request never arrived' in a 'message vanished' report. Log a WARNING with the session id and request id on every session-scoped RPC rejected against an unknown runtime id. Adds a regression test asserting the 4001 response and the warning. Closes #90428 * fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup Close session rows left ended_at IS NULL when the in-process websocket orphan timer dies with the process (#65194). Dual-clock staleness (started_at AND newest message), desktop included, live in-memory sessions excluded, scheduled once from both entry.main and the WS sidecar so desktop/dashboard boots also run the sweep. * fix(tui): make startup_orphan_reap recoverable and move its config onto dashboard.* Follow-up to the #65422 salvage: - startup_orphan_reap joins _RECOVERABLE_END_REASONS (kept distinct from ws_orphan_reap for forensics): every recovery fence (find_latest_gateway_session_for_peer, unarchive_recoverable_session, promote_to_session_reset) now treats a startup-swept row as an accidental end, so a sweep never makes a session unresumable. - Config key moves from sessions.orphan_reaper to dashboard.startup_orphan_sweep in DEFAULT_CONFIG, next to its siblings ws_ping_interval / ws_ping_timeout / ws_orphan_reap_grace_s; the raw loader in tui_gateway.server reads the new key (fail-open on missing). cli-config.yaml.example and website/docs/user-guide/configuration.md follow the dashboard.* documentation pattern. - New regression test: a stranded 'active' row (ended_at NULL, no live runtime) is swept AND still recoverable via peer-keyed lookup and fully revivable via reopen_session afterward. * fix(tui-gateway): revalidate transport ownership before sentinel-parking on WS disconnect Reimplements the concept from #77129 on the current structure (viewer rebinding from #83716 and _client_gone_interrupt_requested clearing are preserved). _close_sessions_for_transport snapshots owned sessions under _sessions_lock, then wrote session['transport'] = _detached_ws_transport WITHOUT re-checking that the session still pointed at the disconnecting transport. A session.resume that rebinds the session to a new live transport between the snapshot and the stomp got knocked back onto the drop sentinel with an orphan-reap Timer armed against a client that is attached right now. The park now happens under _sessions_lock and first revalidates ownership: if the session already moved to a different live transport, the disconnect has nothing to tear down — skip the sentinel park AND the reap scheduling. Regression test simulates the rebind landing between snapshot and stomp. Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com> * fix(tui_gateway): enable TCP keepalive on websocket sockets (dead-peer detection) Without SO_KEEPALIVE a silently-dropped client (SSH tunnel reset, laptop sleep, NAT timeout) leaves the TCP leg half-open forever: receive_text() blocks indefinitely and the disconnect teardown (detach, orphan reap, resume replay) never runs. The server then leaks the session and never reclaims its orphans. _disable_nagle already reaches the raw socket, so enable keepalive there: SO_KEEPALIVE on, plus TCP_KEEPIDLE=30s / TCP_KEEPINTVL=10s / TCP_KEEPCNT=3 on Linux and TCP_KEEPALIVE=30s on macOS. A dead peer is now detected in ~60s instead of never. Best-effort like the Nagle tuning — any failure to reach the socket is logged at debug and skipped. Tests: new tests/tui_gateway/test_ws_keepalive.py fakes the socket and pins SO_KEEPALIVE + the platform-specific idle tuning, plus the no-transport no-raise path. tests/tui_gateway: 336 passed. * chore: map contributor emails for ring-2 salvage (A2chitect, c-pompa) * fix(state): serialize cross-process FTS rebuild with file lock When two Hermes processes (e.g. gateway + serve) detect FTS corruption simultaneously, both run rebuild_fts() on the same database file in parallel. rebuild_fts() only holds an in-instance threading lock, so the concurrent rebuilds collide on write and structurally corrupt the database ('file is not a database' / 'database disk image is malformed'). This happened twice in production (2026-08-15 and 2026-08-23), each time requiring a full page-level salvage of state.db: sessions b-tree clobbered, 5507 messages recovered row-by-row. Fix: acquire an exclusive fcntl.flock on <db_path>.fts_rebuild.lock before rebuilding, with a bounded 30s wait. The SQLite writer lock remains the final backstop. POSIX-only; no-op elsewhere. * fix(state): single fail-closed cross-process authority for all full FTS rebuilds Follow-up to the salvaged #93200 commit. Factors the portable _cross_process_repair_lock ownership pattern (msvcrt on Windows, flock on POSIX, bounded 120s wait) into a cycle-safe shared primitive, fts_rebuild_admission() in hermes_state_common, and routes EVERY full structural FTS rebuild entry point through it: - SessionSearchMixin.rebuild_fts() (replaces the POSIX-only, fail-open 30s flock from the original commit) - _init_schema's trigger-repair rebuilds (_rebuild_fts_indexes / _rebuild_legacy_fts_indexes) via _run_admitted_startup_rebuild - _recover_stale_fts() Fail closed: a caller that cannot acquire the authority DEFERS the rebuild (FTS detached + durable stale breadcrumb, retried at next startup) instead of proceeding into the exact concurrent-rebuild interleaving that structurally corrupted state.db in production. Chunked deferred backfill (fts_rebuild_step) intentionally stays outside the authority. Adds spawned-process regression tests (real child process holding the real lock file): holder blocks contender, deferral fails closed on both the runtime and schema paths, release/holder-death permits the next owner, and stale recovery completes after contention clears. Sabotage-verified: 4/6 tests fail with the admission forced open. * chore: add contributor email mapping for jackijianxa * fix(terminal): subagents no longer hijack the tty with an interactive sudo prompt delegate_task children run on worker threads of the parent process and inherit the process-wide HERMES_INTERACTIVE=1 the CLI sets at startup. _transform_sudo_command's interactive gate therefore fired inside children with no sudo callback registered, falling through to the raw /dev/tty password prompt: a password box printed mid-TUI from a background thread, parallel children racing for the tty, and each child blocked for the full 45s timeout. Gate the prompt (and the sibling 'you will be prompted again' message after an auth failure) on agent.delegation_context.is_delegated_child_context(), the ContextVar set around every child run and propagated through contextvars.copy_context onto the executor thread. Children now behave as headless for sudo: configured SUDO_PASSWORD, the session cache, and the NOPASSWD probe still work; otherwise the command fails gracefully with a subagent-specific tip. A/B verified: 3 regression tests fail on merge-base, 7/7 pass at head. * fix(codex): settle pending Responses tool calls when output_item.done is omitted Backends that omit per-item done events on a successful completion (anomalyco/opencode#37159) caused an announced function call to be silently dropped: the turn ended with output == [] and the tool never executed. Track calls announced via output_item.added, accumulate argument deltas, and settle still-pending calls from accumulated state at a successful terminal event. output_item.done stays authoritative. Mirrors anomalyco/opencode#43575. * fix(agent): harden pending Responses tool call settlement * fix: reuse first-observed sequence when announced items land via output_item.done Follow-up to salvaged PR #92767 (review round 2 P1): the .done path allocated a fresh tail sequence even for items announced earlier via output_item.added, so a mixed announced/pending stream without output_index values reordered the calls ([B, A] instead of [A, B]). First-observed ordering metadata is now recorded for every announced item and reused at .done; a fresh sequence is allocated only for genuinely unannounced items. The .done event's own output_index wins when present, with the announced index as fallback. Regressions: two announced calls without indices where the first later receives .done; an announced non-function item preceding a pending call. * chore: map contributor email for cxxCoolStar * feat: /review briefing carries the parent's loaded skills The reviewer subagent now inherits the primary agent's working skill context: collect_parent_loaded_skills() gathers launch-preloaded skills (from the activation notes in ephemeral_system_prompt) and mid-session skill_view loads (from assistant tool_calls in history), deduped and capped at 8, and the briefing instructs the reviewer to skill_view each and treat their conventions as binding for the assessment. Reference-file reads (file_path=...) don't count as loads; full-skill injection was rejected as too costly (a single dev skill can be 40KB+). Docs: delegation.md /review flow updated (en + zh-Hans). * feat: /review briefing embeds the workspace's project context files load_workspace_context() resolves the parent's workspace via the same _resolve_workspace_hint used for child prompts (explicit sources only — TERMINAL_CWD / agent cwd hints, never a bare getcwd fallback, so the #64590 install-tree-leak guard concern doesn't apply) and runs it through agent.prompt_builder.build_context_files_prompt — the exact discovery/priority/cap logic the main system prompt uses (.hermes.md > AGENTS.md chain > CLAUDE.md > .cursorrules; SOUL.md skipped). The result is embedded in the reviewer briefing as binding review standards. Subagents are built with skip_context_files=True, so without this the reviewer judged repo work without the repo's own conventions. 5 new tests incl. real-filesystem AGENTS.md discovery through the real loader. Docs updated (en + zh-Hans). * feat: every subagent's prompt embeds the workspace's project context files Widened from /review to the class: _build_child_system_prompt now runs the parent's resolved workspace_path through agent.prompt_builder.build_context_files_prompt (same discovery/ priority/caps as the main system prompt: .hermes.md > AGENTS.md chain > CLAUDE.md > .cursorrules; SOUL.md skipped) and embeds the result as binding conventions. All delegate_task children get it — reviewer included — since children are built with skip_context_files=True and previously worked in repos without the repo's own conventions. The review-engine-local load_workspace_context duplicate is removed; the reviewer inherits the block via the shared child prompt path. workspace_path comes only from explicit sources (_resolve_workspace_hint — TERMINAL_CWD / agent cwd hints, never bare getcwd), so the #64590 install-tree-fallback guard concern doesn't apply. Tests moved to pin the generalized path (real-filesystem AGENTS.md via _build_child_system_prompt, empty/no-workspace negatives, reviewer E2E through start_review). Docs: subagent-context section + /review flow (en + zh-Hans). * fix: managed-runtime guard no longer trips on sdist/build copies in the workspace The bare-which() scanner rglobs the repo root; a CI job that builds the wheel leaves an sdist extraction (hermes_agent-<version>/) in the workspace, and the scanner re-found every already-exempted call site under that versioned prefix — which can never match an _ALLOWED key — failing the guard on untouched code (flaked PR #93420's Python-tests job). _source_files now skips build/, dist/, *.egg-info, and any top-level dir carrying PKG-INFO. A/B: planted a fake hermes_agent-9.9.9/ sdist with a which('node') site — old scanner 1 failed, fixed scanner 7 passed, clean tree unchanged. * fix(telegram): watchdog silent long-poll death via last getUpdates progress (#92991) * fix(state): stop rebuilding the whole FTS index on every open when the trigram tokenizer is missing `_init_schema` decided whether the FTS triggers needed repair by comparing the live trigger count against `len(_FTS_TRIGGERS)`, the full six-name set. Three of those six are the `messages_fts_trigram_*` triggers, and they are declared only inside `FTS_TRIGRAM_SQL` / `LEGACY_FTS_TRIGRAM_SQL`, whose `CREATE VIRTUAL TABLE ... tokenize='trigram'` needs a tokenizer SQLite only gained in 3.34. On an older build `_ensure_fts_schema` soft-fails that DDL by design (via `_is_trigram_unavailable_error`) and returns False, so those three triggers can never be created. The count is therefore pinned at 3, `3 < 6` is permanently true, and the repair path ran on every single `SessionDB` open, forever, while holding the SQLite write lock. It never converged: every `hermes` command, gateway start, dashboard request and cron tick paid a full re-index of the message corpus. That is ordinary LTS territory — Ubuntu 20.04 ships 3.31, RHEL/CentOS 8 and Alibaba Cloud Linux ship 3.26, and Hermes has no minimum-SQLite gate precisely because it is supposed to degrade gracefully here. The v23 repair also ends by clearing `fts_rebuild_high_water` and `fts_rebuild_progress`, which is correct after a genuine full rebuild but means an interrupted `hermes sessions optimize-storage` silently lost its resume point on the next open, restarting the chunked backfill from zero every time. Fix: keep `_FTS_TRIGGERS` as the single source of truth and derive two subsets from it, then measure each half against the DDL that can actually create it. `_fts_trigger_count` takes an optional `names` sequence (defaulting to the full set, so no caller changes), and both branches gate on `base_triggers_missing or (trigram_enabled and trigram_triggers_missing)`. The counts are still taken before the DDL runs so they describe the pre-repair state, while `trigram_enabled` is only known afterwards — hence the combination at the `if` rather than at the assignment. Behaviour is unchanged wherever the tokenizer exists: a genuinely missing trigram trigger on a capable host still triggers the rebuild. Only the permanently unsatisfiable comparison changes. * fix(dashboard): secure loopback public URL proxy mode * fix(dashboard): name the exact gate trigger in fail-closed refusals When the bind is loopback and the only gate trigger is dashboard.public_url, the startup refusal now says so explicitly and gives both exits (configure a dashboard auth provider, or remove dashboard.public_url if the proxy no longer exists). Prevents the stale-public_url mystery-locked-dashboard upgrade trap. Adds a truth-table regression suite for should_require_auth and the fail-closed message shape. * chore: map e-macgregor contributor email * perf(bluebubbles): move attachment reads off the event loop * feat(bots): retry session policy — resume transient turns, compress-and-resume on context overflow (#93091 item 5) Maintainer ruling (2026-08-23): a retried bot turn never mints a fresh session. retry_action() maps the #93091 item-1 reason enum to one of resume / compress_then_resume / none: - transient classes (runtime_offline, delivery_timeout, rate limit, server error) re-run the same Bot Chat session once; - context_overflow also re-runs the same session — the retried turn goes through the pre-API compaction pass in conversation_loop.py, which compacts the over-threshold transcript first (the one sanctioned context mutation); no fresh-session escape hatch exists; - auth/quota/config/model classes never auto-retry. Wired at both delivery surfaces (fix the class, not one site): bot_relay.deliver (relay handler) and _run_delivery (local message_agent runner). Failed deliveries now carry the classified reason in the structured error payload (error.data.reason). Sabotage-verified: with the retry blocks removed, 3 consumer tests fail; with them present, 22/22 pass. * test(bots): turn-lock fake Proc gains stdout/stderr attrs _run_delivery now captures output to drive the retry policy; the turn-lock test's minimal _P fake predates that contract. Sibling-test blast radius fix, no behavior change. * fix(gateway): stop multiplex allowlist leak and bot-relay python -c injection _auth_env fell through to os.environ on a scoped miss, so one profile could inherit another profile's allowlists and allow-all flags. bot_relay.waiter_command put connection_id into python -c source. A quote in the id broke the waiter. A crafted id could run extra Python in the sender gateway. * fix(auth): normalize configured provider pool keys * fix(auth): preserve configured provider compatibility * fix(auth): canonicalize configured provider display names * fix(auth): rotate credentials for named custom providers after 401/429 Salvage of #93214 (5 commits squashed onto current main; agent_runtime_helpers.py diverged since the PR base and was 3-way reapplied). The credential-rotation guard in recover_with_credential_pool and both restore_primary_runtime paths only tolerated the custom-naming split when the agent carried the literal label 'custom', so a named custom provider (agent.provider='gemini-no-filter', pool 'custom:gemini-no-filter') tripped the mismatch guard and skipped rotation on every 401/429. Now all three guard sites use the canonical credential_pool_matches_provider boundary predicate + resolve_runtime_pool_key, which recognizes configured named-custom aliases and validates endpoints. Fixes #93188. * fix(state): recover FTS after orphan holder deferrals * fix(state): reap only proven database holders * fix(classifier): 429 quota walls route to billing across providers; reset signals stay rate-limited Consolidates the 429-quota-classifier cluster on top of the merged #93419 Anthropic core. Three independent contributor findings salvaged into one coherent change to the single 429 branch: - Broaden the 429 usage-limit check from the narrow 'usage limit' string to the full _USAGE_LIMIT_PATTERNS ('quota', 'limit exceeded', 'key limit exceeded') and add _BILLING_PATTERNS detection on 429 ('insufficient credits' wrapped in a 429 instead of 402), guarded by a _RATE_LIMIT_PATTERNS exclusion so an explicit 'Rate limit exceeded' never promotes to non-retryable billing. (credit @Pluviobyte, #39441 — earliest submitter) - Add 'resets in' to the transient signals: Codex's 'Weekly usage limit reached. Resets in 6hr 29min.' wrongly read as terminal billing because main only had 'reset in' (no substring match). (credit @LeonSGP43, #63021) - Add 'reset after' / 'available in' / 'per minute' / 'per second' transient signals. (credit @jtstothard, #74785) Supersedes #65633 (defective branch placement, no tests). The aux-client path already covers these shapes (_is_payment_error catches weekly/quota walls; _is_rate_limit_error treats 'resets in' as transient), so no change there. Tests: 6 new cases (generic quota wall, insufficient-credits 429, rate-limit guard, Codex resets-in, extra transient phrases). Guard sabotage-verified. Co-authored-by: Pluviobyte <Pluviobyte@users.noreply.github.com> Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com> Co-authored-by: jtstothard <jtstothard@users.noreply.github.com> * feat(bots): typed failure reasons reach the sending agent on A2A calls (#93091) message_agent callers previously got provider prose (a raw 401 paragraph, a missing-provider essay) and could not branch on the failure class. Now the #93091 item-1 reason enum rides the whole relay roundtrip: - Desktop relay drain forwards bot_relay.deliver's error.data.reason into bot_relay.reply (and prefers it for the attention badge over free-text re-parsing); - write_reply already persisted reason / classified fallbacks; - the sender-side waiter prints "[reason: <code>]" ahead of the free text, so the completion notification the sending agent receives is machine-branchable. Additive everywhere: healthy replies unchanged, reasonless errors classify to a code, old consumers keep working. * fix(desktop): resolve get-windows from the staged copy first window-below asked node_modules for get-windows, whose lib/windows.js locates its native binding through preGyp.find() — by HOST platform. When the tree was installed on one OS and Electron is running on another (a WSL-hosted dev run driving a win32 Electron), pre-gyp picks the host's slot, ignores the correct binding sitting beside it, and upstream's fail-soft path returns no-op stubs. Enumeration then reports 'unavailable' on a machine that answers perfectly well, which silently disables read_window_below. scripts/stage-native-deps.mjs already writes a staged lib/windows.js that requires its binding directly, so prefer it and keep the bare import as the fallback. * fix(desktop): Windows HUD paints opaque white setBackgroundMaterial on a transparent window permanently kills per-pixel alpha on Win11 — every transparent pixel composites as opaque white, so the HUD showed a white slab instead of the desktop behind it. Verified against a minimal repro on Electron 40.10.2: the break happens with ANY material value including 'none', which is exactly what the idle HUD asks for, and neither 'auto' nor a follow-up setBackgroundColor('#00000000') restores it. The DWM backdrop and window transparency are mutually exclusive, so the Windows HUD keeps the CSS tint its sheet already paints and skips the native frost. macOS is untouched: setVibrancy composites correctly. * feat(desktop): HUD game-overlay mode While a fullscreen app owns the screen, the HUD becomes an in-game chat frame: the idle bar steps back to a glanceable opacity, and the transcript is held open for as long as the game is there rather than fading on a timer — you look back at a chat log during a lull, not while the text happens to be fresh. Detection is a pure pass over the same front-to-back window enumeration read_window_below uses (electron/hud-game-overlay.ts); main polls it while the HUD is open and pushes changes to the renderer, which owns the treatment. Two details the enumeration forced: - Hysteresis. Entering needs the game to be what the user is actually looking at, so a windowed app on top vetoes it. Staying only needs the game to still exist: clicking the HUD to type de-foregrounds the game and floats every other open window above it, which otherwise dropped overlay mode at the moment the user engaged with it. - The last state is replayed on did-finish-load. The watch pushes only on change and its first tick fires at window creation, before the renderer has mounted its listener, so a HUD opened over an already-fullscreen game consumed its only message and sat at 'no game' forever. The band itself is reworked for living over someone else's window: - Light-on-dark unconditionally. The theme's near-black body ink is unreadable over a dark game, and every attempt to gate the light ink on some condition — focus, then the game flag — produced a state where it evaluated false and the words went black on black. The sheet is a dark scrim in every theme so white is always right; anything that paints its own light surface (a clarify question, an approval card, a code block, a form control) opts back into theme ink by re-pointing the ink variable, matched on the fill it paints rather than the feature it belongs to. - Your own lines are gold rather than bubbled. With no card the log otherwise reads as one voice; blue and purple are what most game UIs use for their own text, so they disappear into the background. - The scrollback ramps out at the top instead of being cut off, masked on the scroller (the band is a static box — its rows overflow the thread viewport nested inside it, so a mask on the band ramps over empty space). - The sheet is inset under the bar, so its square top corners no longer poke out past the bar's rounded ones. * fmt(js): `npm run fix` on merge (#93503) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix(curator): report pin failures instead of false success and surface pinned unmanaged skills `hermes curator pin <skill>` printed success even when the underlying write never landed. set_pinned() routes through _mutate() with require_curation_eligible=True, which silently returns None for skills that pass is_agent_created() but fail is_curation_eligible() — e.g. a user-created skill named "plan", which PROTECTED_BUILTIN_SKILLS blocks by name. The CLI then announced a pin that does not exist (#92993). Also, a pin that DID land on an eligible-but-unmanaged skill (no created_by marker) was invisible: curated_report() only iterated list_agent_created_skill_names(), which requires the management marker, so the skill showed up under 'unmanaged' with no trace of its pin. - set_pinned() now returns bool write success; _cmd_pin() checks it, exits nonzero and explains the refusal when the write did not land - curated_report() additionally includes curation-eligible skills whose usage record carries pinned=true, so their pins are visible in status Fixes #92993 * fix(curator): check unpin result, guard status ghost rows, tighten test Review feedback on #93149: - _cmd_unpin now checks set_pinned's return (same false-success defect existed symmetrically on the unpin path) - curated_report() pinned-visibility branch requires a local skill dir, so stale records for deleted dirs don't render as ghost rows - test 2 asserts rc==0 unconditionally instead of vacuous-passing - error message points to list-unmanaged (status doesn't render reasons) * fix(curator): say what pin actually does on an unmanaged skill `hermes curator pin` guarded on is_agent_created (a filesystem-shape check), but the flag only matters when the skill carries the curator-management marker: curated_report() walks marker-carrying skills only, so auto-transitions never consider an unmanaged (pre-marker) skill at all. Pinning one recorded the flag and then printed "will bypass auto-transitions" — an effect that does not exist. Keep the write (the flag becomes meaningful after `hermes curator adopt`) and branch the message on is_curator_managed: unmanaged pins now say the skill is unmanaged and point at adopt. Unpin gets the symmetric wording. * fixup(curator): align #93002 test stubs with #93149 set_pinned bool contract Combining both PRs for issue #92993: #93149 makes set_pinned() return a bool and _cmd_pin/_cmd_unpin exit 1 on a no-op write; #93002's tests stubbed set_pinned with a None-returning lambda, which the combined _cmd_pin now reads as failure. The stub reports True (write landed) so #93002's messaging assertions exercise the intended success path. * chore: map beplee contributor email for attribution gate * fix(desktop): harden Hermes API transport * fix(desktop): gate transport retries to idempotent or provably-unsent requests Follow-up hardening on #92977 (issue #92976). The cherry-picked retry wrapped every verb, so an ECONNRESET arriving after the backend had already processed a POST (prompt submitted, session created) would silently double-submit on retry. - Extract the transport policy into electron/api-transport.ts so it is unit-testable without Electron: keep-alive agent pools, transient error classification, and a verb-gated withRetry. - Retry rule: GET/HEAD/OPTIONS retry on any transient transport error; POST/PUT/PATCH/DELETE retry only when the request provably never reached the server (connect-phase failures like ECONNREFUSED / ENOTFOUND, or an error thrown before the body was flushed — requestState.bodySent === false). Ambiguous resets after the body went out surface to the caller; when in doubt, don't retry. - Separate keep-alive pools for JSON calls vs streaming downloads so long downloads can't starve latency-sensitive JSON calls. - Destroy pooled agents on app will-quit. - Tests: shouldRetryRequest truth table, withRetry behavior, plus LIVE transport tests against real misbehaving node HTTP servers: a GET burst where the server resets keep-alive sockets (bare attempt fails, retried succeeds) and a POST whose socket is RST after server-side processing (hit counter stays 1 — no double submit). * chore(release): map KHALIDagara contributor email * style: blank line between node and external import groups (perfectionist/sort-imports) * style: satisfy curly rule in api-transport (eslint --fix) * style: import order in main.ts (perfectionist/sort-imports) * fix(docker): sanitize the session-key task_id used as a sandbox path With terminal.backend: docker and container_persistent: true, every gateway session failed on its first tool call: docker run exited 125 with "invalid spec ... too many colons" and no command could execute. _resolve_container_task_id() returns "session:<key>" whenever a session key is present, and gateway session keys are colon-delimited (session:agent:main:telegram:dm:<chat_id>). DockerEnvironment joined that id into the persistent sandbox path verbatim, so the -v spec became ".../docker/session:agent:main:telegram:dm:<id>/home:/root" — docker splits a spec on ':', read the extra fields as extra mount options, and refused the run. The container label a few lines below already guards this exact value class via _sanitize_label_value(); the bind-mount source did not. Derive the directory name through _sandbox_dir_name() instead. Ids that are already bind-mountable are returned verbatim, so the shared "default" sandbox and RL/benchmark rollouts keep their existing directory and no installed package or /root state moves; only ids that could never have produced a working mount are rewritten. A rewrite carries a digest of the original id, because ':' -> '_' alone is not injective and would otherwise collapse two chats onto one persistent /root. * test(docker): cover session-key sandbox paths and their collision boundary Drives the real DockerEnvironment constructor with a Telegram DM session key and asserts every persistent -v spec is a two-field bind whose source holds no colon — the assertion that reproduces exit 125 on the unfixed path. The derivation's own contract is covered separately: ids that already work stay verbatim (no sandbox migration), docker's separator and the path separators never survive, ids differing only in rewritten characters keep distinct directories, the mapping is stable across calls so cross-process container reuse still resolves, pathological keys stay inside the per-component length limit, and "."/".."/empty cannot resolve to the docker sandbox root. * fix(tools): share the task-id path sanitizer across backends; cover singularity overlays Hoist the sandbox-directory sanitizer into tools/environments/base.py as sanitize_task_id_for_path() and route BOTH host-path consumers through it: the docker persistent sandbox (get_sandbox_dir()/docker/<id>) and the singularity persistent overlay (hermes-overlays/overlay-<id>). One helper, one mapping, whole bug class fixed in one place instead of per-backend copies (#92414, #92640, #93044). docker.py keeps _sandbox_dir_name as an alias of the shared helper so the sanitized mapping (safe ids verbatim, digest suffix on rewrite for collision safety) is unchanged for existing sandboxes. Co-authored-by: salch-cred <salch-cred@users.noreply.github.com> Co-authored-by: Parker Fawcett <259203091+Parker-Fawcett@users.noreply.github.com> * test(tools): shared sanitizer contract + singularity overlay coverage Behavior-contract tests for sanitize_task_id_for_path (colon/separator removal, verbatim pass-through for existing safe ids, determinism, collision-freedom incl. the a:b vs a_b digest case, traversal and oversized-id bounds) and for the singularity persistent overlay path (sanitized, verbatim for safe ids, distinct dirs for colon-vs-underscore ids). Co-authored-by: chelsealong <chelsealong@126.com> Co-authored-by: Parker Fawcett <259203091+Parker-Fawcett@users.noreply.github.com> * fix(desktop): route SSH media through active connection * test(desktop): cover registered file fallback routing * fix(desktop): clear stale group metadata on disband * fmt(js): `npm run fix` on merge (#93563) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fmt(js): `npm run fix` on merge (#93566) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix(agent): bypass response cache for empty retries * fix(desktop): route remote file requests by connection * fix(cron): due-scan must not dispatch a one-shot past its grace window create_job / update_job / resume_job all reject a one-shot whose run time is more than ONESHOT_GRACE_SECONDS in the past ("will never fire"), and _recoverable_oneshot_run_at never recovers such a schedule — but _get_due_jobs_locked dispatched ANY one-shot whose *persisted* next_run_at was in the past, even hours later (gateway down past the window, host asleep, hand-edited jobs.json). A wall-clock one-shot then ran hours late, violating the "will never fire" contract enforced everywhere else. - Grace gate: a once-kind job whose next_run_dt is more than ONESHOT_GRACE_SECONDS in the past is never appended to the due list. - If no run_claim/fire_claim exists (nothing was ever dispatched), retire the record with a diagnostic file so it stops being scanned and the miss is operator-visible. - If a (possibly stale) claim exists, a run may still be in flight in another process: skip this scan but KEEP the record so its mark_job_run can land (avoids re-introducing mid-flight record deletion). - Manual re-trigger still works: trigger_job sets next_run_at=now (inside grace) so an explicitly re-run stale one-shot fires. Tests (tests/cron/test_oneshot_grace_due_scan.py): stale-not-due+retired, within-grace-still-due, stale+claim-skipped-but-kept, retriggered-is-due, and recurring-jobs-unaffected. * fix(cron): misfire backstop honors the one-shot grace window (#93526) The hosted-provider misfire catch-up (fire_overdue_jobs) fired any runnable overdue job with no one-shot grace check, so a stored past-due one-shot bypassed ONESHOT_GRACE_SECONDS and executed arbitrarily late after downtime. Sibling site of the due-scan gate from #89571; pins both directions with tests. * fix(desktop): bots group chat sends message on IME composition Enter macOS Chinese pinyin IME: pressing Enter to confirm a candidate word in the group-chat composer submitted the draft as a message mid-composition. The GroupMentionInput onKeyDown checked only `event.key === 'Enter' && !event.shiftKey` with no IME guard, unlike the core composer which guards isComposing + keyCode 229 (#44135). Add the same guard to the three Enter handlers in the bots plugin: - GroupMentionInput (group composer + reply box) — the reported bug - GroupClarifyCard free-text answer input — same premature-submit - skill-hub search input — same premature-trigger Closes #93528 * docs(auth): correct the SameSite contract in the cookie source docs The SameSite=None change updated the website docs but left two source-level contracts asserting the opposite: - base.py: LoginStart.cookie_payload said cookies set there "MUST" be SameSite=Lax. - cookies.py: the module docstring said all three cookies are SameSite=Lax. Both now describe the actual behaviour: session cookies stay Lax, the short-lived PKCE cookie is SameSite=None; Secure over HTTPS and Lax over plain HTTP. A provider author following the old base.py contract would have had a documented reason to undo the fix. Also records the forwarded_allow_ips caveat in cookies.py: uvicorn only honours X-Forwarded-Proto from a peer inside forwarded_allow_ips (default 127.0.0.1), so a TLS terminator reaching the dashboard from a non-loopback address (a reverse proxy in its own container) leaves the request looking like HTTP and the cookies written in their HTTP shape. Docstrings only; no behaviour change. * fix(auth): thread use_https into the native password-login PKCE clear Merging main brought in the RFC 8252 native sign-in path for password providers (#75808), added while this PR was open. Its loopback-code branch calls clear_pkce_cookie() without use_https, which is now a required keyword-only argument — so /auth/native/password-login raised TypeError on the success path. This is the same call-site class the PR already fixed at the other three sites: the deletion must mirror the shape the setter emitted for the active origin, or the browser keeps the stale PKCE cookie. Caught by CI running the merge commit against main's newer test_dashboard_auth_native_flow.py suite, which does not exist on the branch. Three tests failed there and pass with this change. * fix(cron): refuse to run terminal jobs * feat(cron): add explicit one-shot re-arm * fix(agent): honor prompt_caching for custom providers Apply explicit per-model prompt_caching capabilities to custom chat-completions routes, rather than limiting them to recognized providers, hosts, or model families. Keep undeclared routes conservative, derive the marker layout from the wire transport, and leave Responses and Bedrock caching paths unchanged. * fix(agent): normalize custom provider route identity * fix: align cache-policy pre-gate identity with the capability matcher Follow-ups on top of the salvaged #92785 commit: - Pre-gate now matches base URLs via normalize_route_base_url and provider ids via custom_provider_aliases, mirroring the semantics of get_custom_provider_model_capability. The raw string comparison silently dropped declarations whose config spelling differed only by host case or trailing slash (proven empirically: …/v1/ vs …/v1 with a non-matching provider name returned (False, False) despite an explicit prompt_caching: true). - get_provider(..., allow_network=False) in the early-init/stub branch: the policy runs per request destination (MoA aggregator, auxiliary replans via blank_cache_policy_stub, early agent init) and a cold models.dev cache triggered a measured ~450 ms foreground registry fetch from the send path. A catalog miss degrades to the conservative side. - Debug-log the previously silent provider-lookup exception fallback. - Tests: _make_agent defaults _custom_providers=[] (post-init reality; keeps built-in-route tests off the catalog/config fallback), the two early-init tests delete the attr explicitly, and three regression tests pin the URL-drift, spaced-legacy-name, and no-network contracts (all three fail…

…to dashboard.* Follow-up to the NousResearch#65422 salvage: - startup_orphan_reap joins _RECOVERABLE_END_REASONS (kept distinct from ws_orphan_reap for forensics): every recovery fence (find_latest_gateway_session_for_peer, unarchive_recoverable_session, promote_to_session_reset) now treats a startup-swept row as an accidental end, so a sweep never makes a session unresumable. - Config key moves from sessions.orphan_reaper to dashboard.startup_orphan_sweep in DEFAULT_CONFIG, next to its siblings ws_ping_interval / ws_ping_timeout / ws_orphan_reap_grace_s; the raw loader in tui_gateway.server reads the new key (fail-open on missing). cli-config.yaml.example and website/docs/user-guide/configuration.md follow the dashboard.* documentation pattern. - New regression test: a stranded 'active' row (ended_at NULL, no live runtime) is swept AND still recoverable via peer-keyed lookup and fully revivable via reopen_session afterward.

teknium1 reviewed Jul 18, 2026

View reviewed changes

Comment thread hermes_state.py Outdated

teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 18, 2026

hansai-art mentioned this pull request Jul 19, 2026

fix(tui): sweep orphaned TUI/subagent session rows at gateway startup #65478

Open

19 tasks

teknium1 added the area/sessions Session lifecycle, resume, persistence, history label Jul 19, 2026

GottZ mentioned this pull request Jul 29, 2026

fix: no startup-time sweep for orphaned TUI/subagent sessions (timer dies with gateway restart) #65194

Closed

Nicolas-Formenton mentioned this pull request Aug 12, 2026

feat(sessions): heal non-chat orphan rows after 24h idle #84846

Open

18 tasks

halaprix marked this pull request as draft August 19, 2026 07:43

halaprix marked this pull request as ready for review August 19, 2026 07:43

chore: refresh PR head after rebase onto current main

fad9997

halaprix closed this Aug 19, 2026

halaprix reopened this Aug 19, 2026

halaprix changed the title ~~fix(tui): sweep orphaned session rows at gateway startup~~ fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup Aug 19, 2026

teknium1 mentioned this pull request Aug 24, 2026

fix(sessions): gateway restarts no longer strand rows, and closing sockets can't unbind live sessions (ring 2, 5 salvages) #93430

Merged

teknium1 closed this Aug 24, 2026

andrexibiza mentioned this pull request Aug 24, 2026

fix(sessions): finalize abandoned session rows #93803

Open

andrexibiza mentioned this pull request Aug 26, 2026

fix(tui-gateway): serialize WS session transport ownership #95709

Open

Navigation Menu

fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup - #65422

fix(tui): sweep orphaned tui/desktop/subagent session rows at gateway startup#65422
halaprix wants to merge 2 commits into
NousResearch:mainfrom
halaprix:fix/session-orphan-reaper

halaprix commented Jul 16, 2026 •

edited

Loading

Uh oh!

teknium1 left a comment

Uh oh!

Uh oh!

hansai-art commented Jul 19, 2026

Uh oh!

GottZ commented Aug 3, 2026

Uh oh!

halaprix commented Aug 19, 2026

Uh oh!

halaprix commented Aug 19, 2026

Uh oh!

teknium1 commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

halaprix commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How to test

Platforms tested

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

Uh oh!

hansai-art commented Jul 19, 2026

Uh oh!

GottZ commented Aug 3, 2026

Summary

Related pull requests

Duplicates

Suggested consolidation

Complex graph

Uh oh!

halaprix commented Aug 19, 2026

Uh oh!

halaprix commented Aug 19, 2026

Uh oh!

teknium1 commented Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

halaprix commented Jul 16, 2026 •

edited

Loading