Conversation
After updater.stop() times out, HTTPXRequest.initialize() is a no-op unless the client is already closed, so start_polling reused the wedged socket and the gateway stayed alive but deaf. Rebuild the polling client after a hung drain, watch getUpdates I/O independently of get_me(), and enable TCP keepalive on the fallback transport.
Contributor
fix(telegram): recover Windows CLOSE-WAIT getUpdates deadlock Solid, layered fix (keepalive + drain client swap + heartbeat liveness), with strong tests. Points:
|
Collaborator
|
Salvaged via PR #99691 — both of your commits ( Two salvage adjustments, for transparency:
The fix is proven live on |
19 tasks
teknium1
added a commit
that referenced
this pull request
Aug 31, 2026
…live probes (#87057) Follow-ups on top of the salvaged commits from PR #87111 (@HexLab98) and PR #87265 (@JoaoMarcos44): - keep main's #92991 stall watchdog (150s progress-based) as the single steady-state liveness probe instead of adding a second overlapping one - orphaned-client aclose() cleanup uses the wall-clock thread deadline and is tracked in _background_tasks so a wedged close can neither hang nor leak one task per reconnect attempt (from #87265's review findings) - merge #87265's no-keepalive getUpdates pool (max_keepalive_connections=0) with #87111's TCP-keepalive socket options on all transports - add tests/gateway/test_telegram_closewait_windows_live.py: live probes against a real half-closing HTTP server, skipif non-win32, wired into the on-demand windows-venv-e2e lane (wine2e/**)
EduardoSolanas
pushed a commit
to EduardoSolanas/hermes-agent
that referenced
this pull request
Sep 2, 2026
…live probes (NousResearch#87057) Follow-ups on top of the salvaged commits from PR NousResearch#87111 (@HexLab98) and PR NousResearch#87265 (@JoaoMarcos44): - keep main's NousResearch#92991 stall watchdog (150s progress-based) as the single steady-state liveness probe instead of adding a second overlapping one - orphaned-client aclose() cleanup uses the wall-clock thread deadline and is tracked in _background_tasks so a wedged close can neither hang nor leak one task per reconnect attempt (from NousResearch#87265's review findings) - merge NousResearch#87265's no-keepalive getUpdates pool (max_keepalive_connections=0) with NousResearch#87111's TCP-keepalive socket options on all transports - add tests/gateway/test_telegram_closewait_windows_live.py: live probes against a real half-closing HTTP server, skipif non-win32, wired into the on-demand windows-venv-e2e lane (wine2e/**)
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…live probes (NousResearch#87057) Follow-ups on top of the salvaged commits from PR NousResearch#87111 (@HexLab98) and PR NousResearch#87265 (@JoaoMarcos44): - keep main's NousResearch#92991 stall watchdog (150s progress-based) as the single steady-state liveness probe instead of adding a second overlapping one - orphaned-client aclose() cleanup uses the wall-clock thread deadline and is tracked in _background_tasks so a wedged close can neither hang nor leak one task per reconnect attempt (from NousResearch#87265's review findings) - merge NousResearch#87265's no-keepalive getUpdates pool (max_keepalive_connections=0) with NousResearch#87111's TCP-keepalive socket options on all transports - add tests/gateway/test_telegram_closewait_windows_live.py: live probes against a real half-closing HTTP server, skipif non-win32, wired into the on-demand windows-venv-e2e lane (wine2e/**)
joojalre
added a commit
to joojalre/hermes-agent-almorshednet
that referenced
this pull request
Sep 4, 2026
* fix(update): reject terminated import probes
* fix(update): authenticate import health markers
* fix(update): preserve unknown restore cleanup state
* fix(update): fail closed on incomplete restore checks
* fix(update): verify failed restore cleanup
* fix(update): resume deferred Windows desktop updates
* fix(scripts): clarify Windows update retry marker semantics
* fix(scripts): preserve update retry fallback
* fix(scripts): align retry recovery documentation
* fix(desktop): confirm before deleting a session in the Command Center
The Command Center -> Sessions delete button fired instantly on click,
hard-deleting the session (row + messages + request_dump files) with no
confirm and no undo. e6708af1f confirmed the sidebar rows, tab menus and
chat header, but missed the Command Center's independent entry point in
command-center/index.tsx.
Gate the row's delete button behind the same ConfirmDialog used by the
sidebar path, reusing t.sidebar.row copy and t.common.delete, so every
delete entry point is confirmed as e6708af1f intended.
* test(desktop): regression coverage for Command Center delete confirmation (#99410)
Renders the real CommandCenterView + ConfirmDialog: trash click alone must
not call onDeleteSession, delete fires only after explicit confirm, and
cancel closes without deleting. All three fail against the unguarded
pre-fix Command Center (verified by A/B against origin/main).
* fix(buzz): localize inbound relay media
* fix(buzz): preserve inbound media captions
* fix(buzz): gate authenticated inbound media on explicit authorization
Localizing inbound relay media spends the agent's own Buzz credentials on
a URL chosen by the sender, so it must not run on the strength of the
adapter's local allow-list alone. Require the gateway's authorization
callback to return an explicit True before any `buzz media get` runs; a
denial, a missing callback, or a raising callback fails closed and leaves
the message text exactly as it arrived.
`_is_sender_authorized` previously wrapped the callback result in
`bool()`, so a truthy non-boolean (a status string, a sentinel) would
satisfy an `is True` gate's intent while bypassing its guarantee. Only
the literal booleans now propagate; anything else is "unknown", which the
existing Slack and Discord callers already treat as trust-unknown.
Reviewers asked for this boundary on the sibling inbound-media PRs
(#77734, #78051); it applies equally to the retrieval path in #75614,
which this change builds on.
* fix(buzz): ingest verified native attachments
* fix(buzz): gate inbound attachment side effects
* fix(gateway): require boolean authorization decisions
* test(buzz): isolate authorization cases from CLI lookup
* fix(buzz): merge URL-localization and imeta attachment paths in dispatch
Reconciles #84113 (authenticated same-relay URL localization) with #78051
(native imeta ingestion): _dispatch_message now merges caller-provided
verified imeta attachments with text-localized relay media instead of
clobbering them, dedupes paths, and downgrades mixed-source media to
DOCUMENT semantics so audio members are not routed through STT.
* fix(buzz): deliver local images through native upload
* fix: deliver Buzz media as native attachments
* fix(buzz): reconcile probe-race contract with shared file-attachment sender
#95688's _send_file_attachment refactor re-probed file existence, which
#74999's tests prove can race into a false 'not found' when the file
disappears between the caller's check and the helper's. Callers that
already verified the file pass probe=False; unverified document/video/
voice callers keep the guard.
* fix(buzz): support media in standalone sends
* fix(buzz): verify live media delivery receipts
* fix(buzz): complete media-only delivery reporting
* fix(buzz): redact media paths before bounding errors
* fix(buzz): route shared attachment sender through redacted receipt errors
Follow-up reconciliation: _send_file_attachment (the merged #95688/#74999
helper) now uses #78046's strict _parse_send_receipt contract and
redact_path error bounding, so CLI failures never leak host filesystem
paths and zero-exit unverified receipts are rejected on every outbound
media path.
* chore: contributor email mappings for Buzz media salvage
* fix(buzz): reconcile media pipeline with landed dispatch + threading contracts
Post-rebase composition over #99431/#99429/#99427: file-attachment sends
route through _run_message_send so the mention-recovery ladder covers
media captions; _send_file_attachment/_send_local_file honor the
resolved thread-root anchor and reply_to_mode opt-out; send() records
event_meta on the verified receipt id (#75826); test fakes gain the
auth_tag kwarg and accepted-receipt shape.
* test(send_message): drop duplicate buzz UUID target tests
Dispatch cluster (#99431) landed equivalent coverage first; the media
branch's copies shadowed them and tripped
test_no_shadowed_test_definitions.
* fix(desktop): stop Settings autosave from clobbering out-of-band config edits
ConfigSettingsInner seeds its local draft once from the config record and
never re-seeds it while the page stays open, but every autosave PUT still
sent the entire draft. Since PUT /api/config deep-merges onto disk, that
degenerates into a full overwrite for every field the UI's schema knows
about: if `hermes config set` (or another profile/session) changes a
schema-known key like fallback_providers while Settings is open, the next
autosave — triggered by editing any unrelated field — writes the stale
seed-time value back over it.
Diff the draft against the seed-time baseline and send only the changed
branches, so an untouched key is never resent and the backend's deep-merge
actually protects it.
* fix(desktop): stop model_context_length edits from being dropped or wiped
_denormalize_config_from_web only wrote model_context_length into the
on-disk model dict inside the branch gated on `model` also being present
in the payload. That was harmless when the frontend always sent the full
config, but the prior commit switched Settings autosave to send only the
diff (diffConfig), so editing the Context Window control alone omits
`model` from the payload and the context-length edit is silently thrown
away. The mirror case regressed too: editing `model` alone now omits
model_context_length from the diff, and the old code treated that missing
key the same as an explicit 0, wiping an existing context_length override
that the user never touched.
Track whether model_context_length was actually present in the payload
and only mutate context_length when it was, independent of whether
`model` also changed.
* fix(desktop): advance the autosave baseline after each accepted save
Without this, diffConfig kept comparing against the page-load snapshot
forever, so reverting a field to its original value produced an empty
patch and left the earlier (now-stale) save on disk. Saves are now
queued so an older in-flight request can't resolve after a newer one
and re-advance the baseline with stale data.
* fix: align config-settings test mock with the settings-scope store on main
The salvaged tests mocked @/store/settings-scope from before
$settingsRequestProfile landed (c942cd9ea1); the page now reads it, so
the mock needs the export.
* fix(dashboard): don't gate Desktop-owned loopback backends on public_url
A non-loopback dashboard.public_url engaged the ticket-only auth gate for
EVERY hermes serve on the machine — including the private loopback
backends the Desktop app spawns for itself (HERMES_DESKTOP=1). Those
backends authenticate with the per-spawn session token, which the gated
WS path refuses outright, so Desktop failed to boot with:
Local Hermes backend is HTTP-reachable but the WebSocket (/api/ws)
rejected the session token.
The public_url describes a DIFFERENT deployment: the actual public
dashboard is a separate process on a non-loopback bind whose own startup
keeps its gate. Exempting Desktop-owned loopback backends therefore never
opens the public surface.
Exemption requires ALL of: loopback bind, HERMES_DESKTOP=1 (set by every
Desktop spawn path, local and SSH), and an operator-minted credential
(HERMES_DASHBOARD_SESSION_TOKEN, SSH session token, or owner nonce).
Non-Desktop serves and non-loopback binds keep the exact previous
behaviour — verified by regression tests on both sides of the boundary.
Fixes #96490
* fix(cli): launch-context-independent Linux desktop-entry Exec (salvaged from #94874)
Rewrites resolve_exec_command so the generated .desktop Exec no longer
depends on how the installer happened to be launched: fixes the bare
repo-script form whose shebang escapes the venv, and the symlinked-venv
form that .resolve() dereferenced into the base interpreter store.
Salvaged squashed from PR #94874 (24 commits) after the original branch
was found to carry stray __pycache__/.gitignore payload.
Co-authored-by: Gökhan <gkhn.yldrmlr@gmail.com>
* fix(desktop): pin --publish never in run-electron-builder.mjs (salvaged from #87937)
* chore: map contributor emails for gokhanyildirimlar and mottledMantis
* fix(curator): restore complete skill packages on ledger rollback (#96962)
Consolidation re-homes a skill's references/ / scripts/ out of the tree
before delete/archive, so the ledger captured only what was left
(files: 1 = SKILL.md) and `hermes curator rollback` restored a hollow
skill — the support files were only recoverable by hand out of the
pre-run .curator_backups tar.
The ledger's delete/archive/purge captures now complete themselves from
the newest curator skills.tar.gz: disk hashes win, the backup fills only
missing paths, tar members escaping the package prefix are rejected,
and every fill target stays under skills/ and HERMES_HOME. The same
fill runs at rollback time, so hollow entries recorded before this fix
still restore the complete package.
Wired at the four capture sites (skill_manage delete, archive_skill,
purge, record_mutation) and verified end-to-end: incident shape
(re-home -> delete -> entry has both files -> rollback restores both),
historical hollow entry repair, no-backup degradation, disk-hash
priority, and tar path-traversal rejection.
* fix(curator): remove terminal from the consolidation fork (issue #96962)
The curator LLM fork was steered by its own prompt to re-home skill
support files with terminal `mkdir -p ... && mv ...`. A terminal move
writes the same bytes with NO ledger entry, so the archive that follows
snapshots an already-stripped package (files: 1) and `hermes curator
rollback` restores a hollow skill — SKILL.md back, references/ gone.
Remove the capability rather than guard it: the fork's enabled_toolsets
drops "terminal", so terminal and process disappear together and there
is no shell to parse, no process stdin to feed, no remote-backend
divergence — a heuristic command guard over a Turing-complete input
space can guarantee none of that. Every mutation the pass needs has a
ledgered skill_manage action (write_file / remove_file / delete), and
the prompt now steers exactly those. Reading works through skill_view.
Tests pin both halves: the call-site kwarg (["skills"] only), the
resolved surface (no execution/write tools), and the prompt steering
(no mkdir -p / mv shapes).
* fix(config): warn when a platform_toolsets entry is an empty list
validate_platform_toolsets() accumulated a single valid_count across every
platform, so the "zero valid toolsets" safety net was suppressed as soon as any
one platform carried a valid toolset. A platform wiped to [] — the active one,
typically cli — therefore produced no warning at all.
resolve_enabled_toolsets() honours that empty list verbatim ([] is a list, so
the platform-default fallback is skipped), leaving the agent with zero tool
schemas. The model then has nothing to call and emits the tool call as
assistant text with finish_reason=stop: no error, no warning, no log entry.
That is the silent-failure mode this module was written to prevent (#38798).
Note the asymmetry this leaves intact: a malformed *string* value is not a list,
so it falls back to the platform default and fails open (#78103); an empty list
fails closed. The fail-closed resolution is deliberate (the explicit_empty_
selection contract in tools_config.py, and #82010 wants it persistable), so this
only adds the missing warning and does not change resolution semantics.
Fixes #89050
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(config): warn for empty platform toolsets
* chore: add contributor email mapping for humdrum00001010
* fix(cli): stop raw CSI bytes from Shift+Space leaking into buffer (#88071)
* test(cli): lock buffer-level Shift+letter coverage onto the KeyPress.data fix (#92343)
Follow-up to the salvaged #88097: the same normalization covers the
Shift+letter class reported in #92343 (xterm modifyOtherKeys and both
kitty CSI-u codepoint forms), plus a guard that plain ASCII typing
never triggers the ESC-prefix predicate.
* fix(xai): alias the reserved tool_search bridge on the wire (#95003)
xAI reserves the function name `tool_search` for Grok's native
server-side Tool Search and rejects the client declaration outright:
HTTP 400 {"code":"invalid-argument","error":"The function name
tool_search is reserved for the tool_search tool"}
Hermes's progressive-disclosure bridge registers exactly that literal
(`TOOL_SEARCH_NAME` in tools/tool_search.py) and assembly is not
provider gated, so with the default `tools.tool_search.enabled: auto`
every grok turn fails the moment the catalog crosses the threshold —
mid-session, which reads to the user as a session reset.
Same treatment as the two collisions already handled on this
transport (xAI `web_search` #48108, OpenCode reserved names #85589):
alias to `hermes_tool_search` on the wire in build_kwargs, map back in
normalize_response so Hermes dispatch and the bridge contract are
untouched. `tool_describe` / `tool_call` are not reserved by xAI and
are left alone.
Folds the per-provider rename helpers into one `_alias_reserved_tools`
owner parameterized by the reserved-name tuple, and extends the
existing `_RESERVED_ALIAS_TO_NAME` reverse map so the dispatch-side
un-aliasing needs no new branch.
Scope note: this covers the Responses transport, which is where every
api.x.ai route lands by default (`_fallback_api_mode` maps api.x.ai →
codex_responses, and the xai provider profile declares it). An xAI
model forced onto `api_mode: chat_completions` would still hit the
400; that path has no provider-specific tool rewriting today and would
need the symmetric hook in agent/transports/chat_completions.py. Happy
to add it here if you'd rather have both in one change.
Tests: new TestXaiReservedToolSearchAlias covering the wire alias,
non-xAI backends keeping the canonical name, composition with the
native web_search swap, and the normalize_response round trip.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vLaAmnsdii3Gm9jMDs5gw
* fix(xai): alias the reserved tool_search bridge name on chat completions
xAI's chat-completions API reserves the function name tool_search for
its native server-side tool and rejects the whole request when the
client Tool Search bridge declares it (HTTP 400 'The function name
tool_search is reserved for the tool_search tool', #95003) — Grok
providers were unusable whenever the bridge assembled into the payload
(default tools.tool_search: auto). Mirror the web_search treatment in
transports/codex.py: rename the bridge's wire declaration to
hermes_tool_search for xAI targets (deep-copied first, #27907 lesson)
and map the alias back to tool_search in normalize_response so dispatch
is unchanged. Alias matches the Codex-side fix for the same class
(#83122).
* fix(xai): request-local alias provenance + collision-safe wire aliasing
Hardens the two #95003 alias carriers per review feedback on #95019/#95011:
- _alias_reserved_tools / _rename_tool_search_bridge_for_xai now return the
alias map THIS request emitted; the transport stashes it
(_last_wire_aliases) and normalize_response reverses ONLY those aliases.
A real user/plugin/MCP tool named hermes_tool_search is never silently
dispatched as tool_search when no alias was sent.
- Collision safety: if a real tool already occupies the alias name, the
bridge takes hermes_tool_search_2/_3 — no duplicate wire declarations.
- Legacy static reverse map retained only for normalize-only call sites
that never built a request on the transport instance.
- chat_completion_helpers resets provenance per request so stale maps from
a prior request can't leak into the next response's dispatch.
Refs #95003
* fix(discord): gate relay-only thread rename kwargs
* fix(cli): answer clarify headless in single-query turns
hermes chat -q wired the interactive prompt_toolkit clarify callback
unconditionally, but a -q turn never builds the prompt_toolkit
application — the modal can never be painted or answered, so the turn
polls its response queue until agent.clarify_timeout expires (default
3600 s, 0 = unlimited). The gateway, cron jobs, the kanban dispatcher
and inter-agent wakeups all deliver work as -q turns. Route the
single-query case to a headless callback at the agent-construction site
that already knows _single_query_mode, mirroring _oneshot_clarify_callback
on the -z path (#94943; third member of the family after #86909 and
#88013).
* test(cron): pin _REDACT_ENABLED in incident redaction test
test_redaction_applied_to_incident_error asserted real redaction while
relying on the ambient HERMES_REDACT_SECRETS default. agent.redact
snapshots _REDACT_ENABLED at import time; when a co-collected module
(tests/cron/test_codex_execution_paths.py) imports the gateway chain at
COLLECTION time under a shell exporting HERMES_REDACT_SECRETS=false, the
snapshot freezes False before the conftest env scrub runs, and the test
fails only in full-directory runs. Pin the flag via monkeypatch like the
~30 other redaction tests do.
Bisect evidence: pytest tests/cron/test_codex_execution_paths.py
tests/cron/test_cron_incidents.py -k redaction_applied -> 1 failed on
main under HERMES_REDACT_SECRETS=false; passes with the pin.
* fix(prompt): skip bundled AGENTS.md for desktop launch cwd
* fix(prompt): preserve resumed workspace provenance
* fix(desktop): keep @tanstack/react-query in one runtime chunk (#95560)
The packaged app crashed at launch with 'No QueryClient set, use
QueryClientProvider to set one': useQuery in a lazy chunk (session-list-density)
read a second @tanstack/react-query runtime whose QueryClientContext was never
populated by the entry's QueryClientProvider. The source tree was correct — the
duplication happened at build time, because react-query was the one
context-bearing runtime not pinned to a shared vendor chunk, and rolldown's
merge heuristics inline the spare copy into a lazy chunk depending on toolchain
version.
- vite.config.ts: add @tanstack/react-query to the vendor-react
advancedChunks group + dev dedupe list, mirroring the react-router fix.
- assert-dist-built.mjs: fail the build when the 'No QueryClient set'
invariant appears in more than one JS asset (launch-smoke guard).
- assert-dist-built.test.mjs: unit tests for the new invariant check.
- launch-packaged-app.spec.ts: e2e smoke test asserting the packaged app
boots to real UI, not the QueryClient error boundary.
* fmt(js): `npm run fix` on merge (#99598)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(compression): truncated summaries no longer become compaction checkpoints (port of earendil-works/pi#7048)
A summarization response with finish_reason == "length" contains PARTIAL
text — the generation stopped on the output-token cap mid-summary.
Previously all compressor summarization sites accepted such responses as
complete: the cut-off text replaced the real middle turns AND was fed back
into every subsequent iterative-update prompt, compounding the loss across
compactions.
Guards added at all four summarization sites (whole bug class):
- _generate_summary: length stop raises, gets the existing one-shot
main-model fallback (a larger output budget may finish the summary), and
on terminal failure ABORTS compression preserving the session unchanged
(new _last_summary_truncated_failure flag, same class as empty-content).
- _micro_summarize_one: partial rolling-summary merge is discarded; the
exchange stays unabsorbed for a later pass.
- _build_chunk_digests: partial lean digest degrades to the
recover-via-session_search placeholder.
- trajectory_compressor (sync + async): length stop raises into the
existing retry/backoff loop.
_response_finish_reason() reads dict- and object-shaped responses and
returns "" when the provider omits the field, so proxies that never send
finish_reason are unaffected.
Ported from earendil-works/pi commit 97fa14e39 (pi#7048), adapted to
hermes' abort-preserving compression failure machinery.
Tests: tests/agent/test_compressor_truncated_summary_guard.py (12 tests;
sabotage-verified — disabling the guards fails 4).
* fix(hermes_cli): fail-closed PID-ownership guard before Windows taskkill
Guard every Windows `taskkill /PID` against stale/recycled PIDs
(#89614: 8x 0xEF blue screens; a rebooted PID can be svchost.exe).
Adopted the community patch by AlexMnrs (commit 0162465): shared
psutil-based (pid, create_time) guard reusing the repo's existing
get_process_start_time machinery:
- fail closed on invalid/unknown/recycled identities (0/-1/None/bool/non-int)
- capture identity at discovery, re-validate at kill time
- all three sites through pid_is_hermes; taskkill stays hidden
Sites: _subprocess_compat.kill_process_tree,
dashboard_procs._kill_stale_dashboard_processes (win32),
update_cmd._stop_process_trees.
Refs #90471, #89614
Co-authored-by: Alex Monrás <AlexMnrs@users.noreply.github.com>
* fix(windows): require process identity before taskkill
* fix(update): refuse gateway ancestor tree-kill on Windows
* fix(windows): compose the taskkill identity guards into one fail-closed class fix
Salvage hardening on top of the three cherry-picked contributor commits
(#91297 gebilaowang404 + AlexMnrs, #96741 burak33bb, #98826 ayushnangia),
closing the remaining unverified-PID kill sites as one class (#98814, #89614):
- pid_is_hermes: token-boundary 'hermes' match (no more loose substring
false-positives), and an explicit start-time expectation is now honored
on POSIX too (a mismatched fingerprint is a recycled PID on any platform).
- kill_process_tree: drop the guard on our OWN retained Popen child — a
retained handle pins the PID, so the check could only false-refuse.
- gateway.status.terminate_pid: POSIX force-kills also refuse when a
caller-provided expected_start_time no longer matches.
- kill_gateway_processes: re-verify the LIVE cmdline at kill time (the
scan-time match is a TOCTOU window).
- _reap_unsupervised_gateway_orphans: fingerprint orphans at scan time and
require a still-matching identity before the delayed SIGKILL escalation.
- whatsapp _kill_port_process: never kill a bare netstat/lsof-scanned PID
unless the live process is actually a node bridge (was a stranger-kill).
- browser daemon reap/close paths: pass the start-time fingerprint into
ProcessRegistry._terminate_host_pid (previously unverified), and the
session-close path now runs the same daemon identity verification as
the orphan reaper.
- tests/hermes_cli/test_taskkill_identity_windows_live.py: live Windows
probes (real spawned processes, real psutil ancestry) wired into the
on-demand windows-latest wine2e lane.
Fixes #98814
Fixes #89614
* fix(update): fingerprint orphan backends from the classification psutil handle
The orphan-backend classifier fingerprinted candidates via
gateway.status.get_process_start_time, which prefers /proc/<pid>/stat —
the HOST process table, in clock ticks. Under the fake-psutil test harness
(and any containerized run where the PID number happens to exist on the
host) that returns the WRONG process's fingerprint in the WRONG units,
while pid_is_hermes verifies via psutil centiseconds at kill time: the
guard would then refuse every legitimate reap. Read create_time() from the
same psutil handle used for classification, quantized exactly like
gateway.status does on Windows, so the fingerprint round-trips.
Also covers the Windows-lane sibling: test_uses_netstat_and_taskkill_on_windows
now pins the guarded call path, plus a new refusal test for a non-bridge
listener PID (#89614 class).
* fix(terminal): bound env.execute wait so a wedged poll cannot disable every timer
A hung terminal wait on the loop thread silently disabled asyncio deadlines
and let cron jobs idle thousands of seconds past HERMES_CRON_TIMEOUT. Drive
the wait from run_bounded_sync (sliced Event.wait, kill-on-timeout) and
move the cron inactivity monitor onto a daemon thread with the same kernel
timeout primitive. Copy the caller ContextVar scope and activity callback
onto the wait worker so profile secrets, session id, and heartbeats survive
the thread hop (#94285).
* test(terminal): cover hung-wait bound, parent-tid interrupt, and cron inactivity watchdog
Pin that execute() returns at the wall-clock deadline when the inner wait
never returns, that /stop on the tool-worker tid still kills the subprocess,
that the cron inactivity helper fires while the caller thread is blocked,
and that ContextVars plus the activity callback reach the deadline worker.
* fix: clamp invalid effective_timeout to the 120s wait default instead of unbounded (review follow-up for #94305)
* fix: restore _inactivity_watchdog_loop dropped in rebase conflict resolution
* fix(state): self-heal SessionDB writes after close() races an in-flight worker
Subagent/cron sessions died mid-run with "Session DB append_message
failed: 'NoneType' object has no attribute 'execute'": a teardown owner
(cron run_job finally, delegate timeout owner, agent close()) called
SessionDB.close() — nulling _conn — while a still-unwinding worker had
one more transcript flush to land. The flush then hit None.execute, the
turn force-ended as session_persistence_failed, and the session tail was
silently dropped while cron delivery reported last_status: ok.
Fix at the shared persistence boundary: _execute_write and the _read_ctx
writer-lock fallback detect the closed handle under self._lock and
reopen a connection to the same database file with a loud WARNING naming
the race. Read-only handles never reopen — they raise an explicit
'was closed' error. A failed reopen raises an OperationalError naming
the teardown race so classify_persistence_error gets a real cause.
Closes #94736
* test(agent): update enqueue-after-close contract to the #94736 self-heal
The old contract (write after close() raises AttributeError and drops
the token delta) is superseded: the persistence boundary now reopens
the connection, so the delta lands. Assert the new, stronger contract.
* fix(install): never adopt a pre-release Node.js build
install_node() picks the newest tarball out of
nodejs.org/dist/latest-v${NODE_VERSION}.x/ and installs it without ever asking
whether the binary inside is usable. That index currently serves
node-v26.8.0-<os>-<arch>.tar.xz -- a final-looking filename -- whose binary
reports v26.8.0-alpha.0.0.0. Node publishes the headers tarball named by
process.release.headersUrl only for final releases, so node-gyp cannot compile
against that build and every native module fails to install.
Probe the extracted tree before it replaces anything on disk, and fall back to
an older release line when the probe rejects it, instead of leaving the install
with an unbuildable runtime. Mirror the guard in node-bootstrap.sh, and let
_managed_node_tree_outdated() treat a pre-release tree as outdated so an
already-broken install heals itself -- the existing heal only fires below the
target major, and a pre-release sits above it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GrEaXSjvFoBXKxAHTjnUbS
* test(install): repin #87460 probe tests on the line-walk contract
The pre-release line-walk (#96601 salvage) moved the download/probe body
into install_node_line() and rejects an unstartable binary BEFORE
adoption via node_satisfies_build on the extracted tree. The sandboxed
driver now inlines all three functions, and the broken-node test pins
the stronger pre-adoption rejection instead of post-adoption cleanup.
* fix(feishu): gate approval/update-prompt card clicks on operator allowlist, not group policy
The synchronous card-action handlers and the update-prompt resolver
authorized clicks with _allow_group_message(), which answers "may this
sender chat in this group?" — with group_policy=open it returns True
for everyone. The approval resolver already used the correct operator
gate (_is_interactive_operator_authorized), so the three code paths
disagreed: with an open group policy an out-of-allowlist click on an
update-prompt card was fully executed, and approval clicks returned a
resolved-looking card before being rejected asynchronously.
Authorize all three paths with _is_interactive_operator_authorized(),
which checks membership of admins ∪ allowed_group_users (wildcard and
the empty pairing-mode allowlist keep their existing allow semantics,
matching _admit's DM pairing default). A missing operator identity now
fails closed on the update-prompt resolver instead of skipping the
check.
Fixes #96045
* test(feishu): cover DM paired-mode card clicks and fail-closed identity checks
Adapt five scenarios from @liuliu0223's regression suite in #99021:
- paired-mode (empty allowlist) positive paths for approval and
update-prompt cards, the DM breakage this fix resolves
- fail-closed rejection of clicks with an empty operator identity
- chat-mismatch rejection when an approval card is forwarded
* fix(cli): honour model_aliases api_key, stop cross-provider key leak (#83612)
Salvaged from PR #84199 by @RickyYii. DirectAlias gains api_key/key_env; the direct-alias override re-resolves credentials against the alias endpoint (host-gated, #28660) and reuses the pre-alias key only on an origin match; oneshot -m <alias> passes the alias key as explicit_api_key; direct-alias branch gains the OLLAMA_API_KEY host gate. Fixes #83612.
* fix(redact): keep lowercase assignment scans linear
* fix(context): fail closed when preflight compression stalls
* test(context): cover preflight timeout provider boundary
* fix(compression): count streamed reasoning details as progress
* fix(profiles): make_targz writes to a temp file and renames, not the destination directly
tarfile.open(archive_path, "w:gz") truncates the destination the instant
it opens. If tf.add() fails partway (disk full, permission loss,
interruption), whatever was previously at that path is gone — including
an existing profile or board export the caller chose to overwrite. This
is the same failure shape a7e7de6407 just fixed for the desktop gateway
file-save path, one commit earlier in the same window, but it was never
propagated to this shared archive-writing primitive even though board
export gained a new caller into it in that same window.
make_targz now writes into a sibling temp file (mkstemp, same directory
as the destination so the final step is a same-volume rename) and only
replaces the destination via os.replace() after the archive is fully
written and closed, mirroring the mkstemp+os.replace pattern already
used throughout this codebase (agent/secret_sources/_cache.py,
cron/jobs.py, gateway/status.py, etc). The temp file is unlinked on any
failure.
* fix(models): support OpenRouter preset references
* refactor(models): hoist preset suffix re-attachment into one helper
Follow-up to salvaged PR #89129: both auto-correct sites now call
_with_preset_suffix() so a future correction path can't forget to
re-attach the @preset/<slug> routing suffix.
* fix(state): bound state.db read connections per FILE, and stop opening two gateway handles
Issue #98573 reports a long-lived gateway holding ~20 `state.db` descriptors
that never shrink, walking into the 256 soft RLIMIT_NOFILE a launchd/systemd
service manager hands the process. The cause named there — a per-thread
`threading.local()` read connection — is already gone (87aedbe7b6 pooled the
read connections, 0472c31aa1 added the peak permit). Measured on main: one
SessionDB with 40 concurrent reader threads peaks at 9 live connections, not 40.
The symptom survives one layer up. `_READ_POOL_MAX` was enforced by a
BoundedSemaphore owned by each SessionDB, which bounds the wrong noun: the
descriptors are spent on a FILE, so every additional handle on one state.db got
its own allowance and peak scaled as `instances x (1 + _READ_POOL_MAX)`.
Two changes, both needed:
* The permits move to a per-path `_PathReadBudget`, shared by every SessionDB
in the process that points at that file. A permit miss first reclaims an IDLE
pooled connection from a peer handle before degrading to the writer lock —
without that, whichever handle warmed up first would pin the whole budget and
permanently demote every later one (a cron job's transient handle, a second
profile's store) to the locked writer connection.
* `GatewayRunner` borrows `SessionStore`'s handle instead of opening its own.
Both caches resolve the same `_default_db_path()`, so the process was holding
two writer connections and two read pools against one file for no reason, and
doubling again per profile on a multiplexed gateway. The store owns the
connection and sweeps it at shutdown; the runner's cache now holds only the
async wrapper and its sweep skips borrowed handles.
Measured peak live connections against one file, 40 reader threads, by handle
count 1/2/4/8:
before: 9 / 18 / 36 / 51 (51 not 72 only because the sample window ended
before every pool filled)
after: 9 / 10 / 12 / 16 (read connections capped at 8 in total; the
remainder is one writer per handle, and the
gateway's per-profile pair is now one)
Fixes #98573
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(state): cap read connections per PROCESS and yield when the fd table is tight
Follow-up to the per-file budget in the previous commit, which closed the
scaling axis it measured and left three others open.
* A per-file ceiling still lets the cost grow with the PROFILE count: a
multiplexed gateway serves N profiles from one process and each has its own
state.db, so `_READ_POOL_MAX` bounded each file while the process total went
unbounded — the per-instance bug one level out. `_READ_POOL_PROCESS_MAX`
(three files' worth) now bounds the process, and a miss reclaims an idle
connection from ANY path before degrading: a profile quiet for an hour must
not hold descriptors the profile being served right now needs.
* Hermes's SQLite descriptors are only ever a share of the fd table. In #98573
the ~20 state.db handles were not the whole 256 — they were the share that
pushed httpx sockets and terminal subprocess pipes over, and the EMFILE
surfaced in tools/terminal_tool.py rather than here. New read connections are
now refused when the process is within `_FD_HEADROOM_RESERVE` of its soft
RLIMIT_NOFILE, measured from /proc/self/fd or /dev/fd and cached briefly. The
guard fails OPEN where it cannot measure (Windows has neither the fd
directory nor RLIMIT_NOFILE, and a CRT limit in the thousands) and CLOSED on
evidence — including a probe that could not get a descriptor of its own.
`_read_open_denied_fd_headroom` makes it diagnosable from a running process.
* Writer connections cannot be rationed the way read connections can: a
SessionDB without one cannot write. Their only real bound is not opening
redundant handles, so a process that accumulates more than
`_HANDLES_PER_PATH_WARN` handles on one file now says so once, and the next
duplicate is visible before it is an incident instead of inferred from an
lsof after one.
`_READ_POOL_MAX` itself is deliberately unchanged at 8. Retuning that constant
is #98585's subject; with a process ceiling above it and the headroom guard
in front of it, the value is no longer the binding constraint.
Refs #98573
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(state): drop unused is_explicit_fork_child wrapper
teknium flagged it as scope creep on PR #98691 review: no callers in the diff.
_is_explicit_fork_child_row (the row-based helper actually used) is unchanged.
* fix(docker): keep forwarded secret values out of world-readable argv
docker run/exec argv previously carried -e KEY=VALUE pairs for every
forwarded/passthrough variable. On Linux /proc/<pid>/cmdline is
world-readable regardless of process owner, so every allowlisted secret
was visible to all local users via plain ps for the duration of every
terminal call.
Emit name-only -e KEY flags and supply values via the docker client
subprocess env instead: the docker CLI resolves valueless --env KEY from
its own environment (documented docker/podman behavior), moving secrets
from /proc/*/cmdline (0444) to /proc/*/environ (0400). Covers the docker
run container-start path, the recreation/recovery path, the init-seeding
exec path, and the per-command runtime exec path.
Reported by @sashalab. Fixes #96268
* fix(state): fail closed on unscoped corruption
* fix(gateway): require FTS provenance before transcript rebuild-and-retry
Widen #96038's fail-closed classifier to the gateway transcript retry
path: SessionStore._is_fts_corruption_error no longer treats a generic
'database disk image is malformed' as FTS-only damage. It now delegates
to SessionDB._is_fts_write_corruption_error (SQLITE_CORRUPT_VTAB result
code or explicit fts5 corrupt-structure text) and only keeps the
messages_fts-named cases. Structural corruption falls through to the
bounded retry/backoff path instead of rebuilding FTS and retrying writes
against a damaged database.
Sibling site spotted in PR #98090 by @fangliquanflq.
* fix(gateway): bound signal interrupt grace
* fix(desktop): recover incomplete transcript turns
* fix(compression): dead Codex summary streams fail over in 60s instead of stacking 5-minute waits
The Codex auxiliary Responses adapter enforced a single absolute
deadline (300s floor for compression). A dead stream held the entire
budget before fallback ran, and repeated compression attempts stacked
those waits into 20+ minute 'Summarizing thread' stalls (masoria debug
bundle, Aug 31 2026). Meanwhile a healthy-but-slow reasoning summary
was killed at the same absolute deadline even while producing tokens.
Replace the absolute kill with progress-aware deadlines:
- 60s no-progress window for the first substantive payload AND between
payloads; keepalive/lifecycle frames do not re-arm (mirrors the
commit-fence gating, #96707)
- a live stream re-arms per token and is bounded only by
_aux_stream_total_ceiling() (max(600s, 4x configured timeout)), the
same backstop the streamed chat.completions path already uses
- the compression critical-path retry gate now distinguishes failure
cost: a cheap first-token no-progress failure retries the same
provider once; mid-stream stalls and ceiling hits still skip straight
to provider fallback (#54465 semantics preserved)
Live A/B (real OpenAI SDK against a local SSE server, real adapter):
dead keepalive-only stream: main waits the full budget; fixed fails
over at the window. Slow-but-alive stream (tokens past the configured
timeout): main kills it mid-generation; fixed completes.
* fix(desktop): latch dead runtime recovery across remounts
* fix(desktop): retain session remount polling reset
* fix(desktop): satisfy import ordering
* chore: AUTHOR_MAP entries for fangliquanflq and sycamoregroupltd
Maps the contributor emails for the PR #99265 and #97779 salvages so
check-attribution passes on the salvage PRs.
* fix(redact): keep dotted config-key scans linear past the keyword pre-gate
The _CFG_SECRET_WORD_RE pre-gate only skips secret-FREE text. A compaction
payload containing one real secret assignment plus a long opaque dotted run
still reaches _CFG_DOTTED_RE's backtrackable '*' prefix, which re.sub retries
from every byte of the run — quadratic while holding the GIL (same class as
the _ENV_ASSIGN_LOWER_RE fix in this branch, #99255).
Anchor each attempt to the start of a key run with a negative lookbehind.
Match set is unchanged: any match starting mid-run implies a leftmost match
at the run start, verified 20/20 identical over a dotted-config corpus.
30k-char adversarial run: 102s -> 0.015s.
* fix(desktop): resolve the e2e Electron binary per platform and layout
findElectron() probed exactly one path, and got three things wrong at
once for anyone not on a hoisted POSIX install:
* It looked only under the REPO ROOT. This is an npm workspaces repo and
npm hoists a dependency only when nothing conflicts, so `electron`
installing into apps/desktop/node_modules is an ordinary outcome, not a
broken tree.
* It joined a bare `electron`. On Windows the dist file is
`electron.exe`, so the probe could never match there.
* Its PATH fallback spawned `which`, which is not a command on Windows,
so the fallback failed for a reason unrelated to whether electron is on
PATH.
The three combine into a misleading error: the suite refuses to start
with 'Run "npm install" from the repo root' on a tree that has electron
installed. Reproduced on Windows 11 against this repo, where
apps/desktop/node_modules/electron/dist/electron.exe exists and the old
body throws that message; the reporter on #88036 hit the same thing on
Linux and had to hand-symlink the package before the suite would run.
Resolution now asks the installed `electron` package for its own path
first (its main export IS the absolute executable, resolved from
path.txt and honouring ELECTRON_OVERRIDE_DIST_PATH), then falls back to
explicit dist probes for each root, then to PATH with the platform's
lookup command. The error message lists what was searched.
The rules live in e2e/electron-binary.ts so they can be unit-tested
without importing the Playwright runner, with the platform passed in
rather than read from process.platform: reading it would leave every
Windows rule untested on the Linux CI runner.
Wiring: the vitest `electron` project picks up e2e/**/*.unit.test.ts and
Playwright ignores the same pattern, so helper unit tests run in exactly
one runner and the specs are untouched.
Verified: 5 unit tests pass; mutation-checked one rule at a time
(hardcoding the binary name fails 2, reversing the probe order fails 1,
hardcoding `which` fails 1). tsc -p . and tsc -p tsconfig.e2e.json
clean.
This is the environment blocker called out in #88036, not its rendering
bug, so it is deliberately a subset.
Refs #88036
* test(cli): cover mixed-config ImportError recovery hint on chat startup (#96900)
* fix(cli): print partial-update hint when chat startup hits a first-party ImportError (#96900)
HermesCLI construction imports helpers from hermes_cli.config before the agent-setup mixin can run, so a mixed-version tree crashed with a raw traceback. Catch that ImportError on the chat entry path and tell the user to run hermes update.
* fix(desktop): keep primary SSH session resumes remote
Untagged session rows come from the ambient primary backend. Do not synthesize a local owner for them, and clear stale explicit hints before issuing an id-only resume.
* fix(desktop): prevent venv scan timeout on busy Windows hosts
* test(state): physical-corruption acceptance tests for the fail-closed classifier
Real byte-flip fixtures (no mocks) proving the #96038/#98090-class fix
end to end, closing the acceptance gate on issue #97940:
- test_canonical_btree_corruption_fails_closed: checkpoint the WAL,
clobber every messages-table B-tree leaf page header, then assert a
live append raises the genuine bare SQLITE_CORRUPT, the classifier
refuses the FTS route, no rebuild/detach/stale-marker side effects
occur, and the field incident's misdiagnosis log line ('canonical
message rows are preserved') never appears.
- test_fts_only_corruption_still_self_heals: contrast case — a real
messages_fts_data shadow-table stomp raises SQLITE_CORRUPT_VTAB (267),
is classified as FTS-scoped, and the write path still self-heals with
canonical rows intact.
Sabotage-verified: reverting the classifier fix (96739033c4) makes the
canonical-corruption test fail by entering the FTS self-heal route.
Credits @fangliquanflq (PR #98090) for the production timeline analysis
and @diatche (PR #96038) for the classifier fix these tests gate on.
Refs #97940, #98077.
* fix(gateway): run MCP shutdown off-loop with a bounded wait on the shutdown path
shutdown_mcp_servers() blocks on future.result(timeout=15) which, called
from the gateway event-loop thread during SIGTERM teardown, freezes the
loop for up to 15s when the MCP loop and its stdio children are torn down
concurrently. Supervisors with a shorter kill grace (s6-overlay: 3s)
SIGKILL the gateway before lifecycle_ledger.mark_exited() runs, producing
phantom 'exited UNCLEANLY' reports on every subsequent boot.
Run the sync shutdown on a daemon thread and poll via _await_thread_exit
with a 5s budget; proceed with teardown if it wedges. Fixes #82874;
completes the shutdown half of #64155.
* fix(gateway): isolate PID check and credentials per profile (#74872)
Add _pid_record_belongs_to_current_profile() helper that verifies a
PID record's persisted hermes_home matches the current process. Use
it in get_running_pid() and get_runtime_status_running_pid() so the
default-profile gateway never mistakes another profile's gateway PID
as its own.
In _apply_profile_override(), clear HERMES_HOME instead of returning
early when it points to a profile directory but no --profile flag was
given, letting the sticky active_profile logic resolve the right one.
In _guard_existing_gateway_process_conflict(), detect stale PID files
from other profiles and emit a warning.
* fix(cli): supervised gateway launches skip the sticky active_profile redirect
Generalize the HERMES_S6_SUPERVISED_CHILD supervisor-marker mechanism so
ANY supervised gateway launch (systemd, launchd, Windows Scheduled Task,
external supervisor) skips the active_profile redirect in
_apply_profile_override(). Previously only the s6 container marker was
honored, so a systemd-launched default-profile gateway with
HERMES_HOME=<root> followed the sticky active_profile file and silently
assumed another profile's identity — logging under that profile's tree
and connecting with its Telegram bot token (double-polling a token owned
by that profile's own live gateway).
- hermes_cli/main.py: honor HERMES_SUPERVISED_CHILD (new generalized
marker), HERMES_S6_SUPERVISED_CHILD (back-compat), INVOCATION_ID
(systemd; gateway commands only, since it leaks into every descendant
of systemd-launched processes), and HERMES_GATEWAY_EXTERNAL_SUPERVISOR.
- hermes_cli/gateway.py: export HERMES_SUPERVISED_CHILD=1 in generated
systemd units (user + system) and the launchd plist.
- hermes_cli/gateway_windows.py: export it from the Scheduled-Task cmd/vbs
launchers and the windowless respawn env overlay.
- hermes_cli/service_manager.py: export it alongside the s6 sentinel.
- tests: regression coverage for all markers + non-gateway INVOCATION_ID
neutrality + generated-unit marker presence.
Fixes #74872
* fix(estop): honor canonical ~/.hermes/ESTOP from profile gateways
Profile processes launch with HERMES_HOME=~/.hermes/profiles/<name>, so
`hermes pause` at the fleet root did not bind fleet-analyst dispatch
(t_7b65ff88). Check/resume both the process home and the fleet root.
* refactor(estop): drop redundant isinstance branch, fix stale docstring
The isinstance(primary, Path) branch in _candidate_sentinel_paths was dead
weight: the surrounding except Exception already covers non-Path test
doubles, and .resolve() failing on them falls through to the plain
inequality comparison. Verified the pre-existing fail-safe stat fixture
(test_is_engaged_fails_safe_on_stat_error) still passes without it.
Module docstring still claimed 'a single os.stat'; the fleet-root check
makes it one or two stats. Updated.
* fix(state): defer corrupt FTS rebuilds past live operations
* fix(desktop): bounded auto-restart for no-mux SSH tunnel flaps instead of instant connection death (#96266)
A no-mux tunnel is a single persistent `ssh -N -L` child. On main, ANY
death of that child after readiness immediately set tunnel.alive=false,
which poisons SshConnection.isAlive() forever; upstream lifecycle probes
then treat the whole SSH connection as dead, tear down the scope, and
SIGTERM a perfectly healthy backend (~10s after HERMES_BACKEND_READY in
the #96266 logs: '[ssh] connection closed (no-mux tunnels killed)' ->
'Ignoring stale Hermes backend exit (SIGTERM)' -> 90s port-announcement
timeout, with retry/repair looping the same failure).
Now a post-readiness child death is a tunnel FLAP: the child is
restarted with a bounded budget (5 attempts, 1s delay by default,
injectable for tests) and only an exhausted budget marks the tunnel —
and thus the connection — unhealthy. Deliberate teardown (cancelForward
/ close) sets tunnel.stopping, cancels any pending restart timer, and
never restarts. Pre-readiness deaths keep failing fast with classified
stderr (auth/bind errors unchanged).
Fixes the kill chain of #96266.
* fix(desktop): heal v1 SSH gateway routes into the v2 connections registry
reconcileRegistryDrift only healed remote/cloud v1 routes. A v1 global
mode:'ssh' route (host, no url) written by Settings after the one-shot
migration had no registry identity: resolvedConnectionId returned null,
primary stayed 'local', and every launch re-homed the window onto a
fresh local backend. Because the heal skipped SSH entirely, the two
config files re-drifted after every update relaunch instead of
converging once.
Normalize the v1 SSH descriptor into a v2 kind:'ssh' entry (via the
same validated normalizeConnectionInput path the editor uses) and align
primary/lastUsed, with the same narrow-heal rules as remote: already-
registered targets and deliberate primary picks are left alone, and
unusable hosts never touch the registry.
Diagnosis credit: mgallmur-glitch (root cause) and jakewvincent
(re-drift after update relaunch) on #93888.
* fix(gateway): relay compute-host clarify state
* fix(gateway): gate compute-host interrupt forward on hosted activity
Follow-up to the salvaged #98571: forward the interrupt to the compute
host whenever the parent 'running' mirror is stale, but only for
sessions that actually have hosted activity — HostSupervisor.interrupt()
calls start(), so an unconditional forward would spawn a compute-host
child just to deliver an interrupt for an idle lazy session.
Adds a regression test asserting the idle-lazy-session no-spawn path.
Refs #92916
* fix(cli): support literal dots in config set/unset key paths (#84064)
* fix(config): greedy literal-key matching + loud phantom-sibling refusal for dotted key names
Builds on webtecnica's escape-aware _split_key_path (#84152, cherry-picked
with authorship preserved; earliest fix in the family was RelaxJonh's #80253
greedy-match approach — both behaviors now ship together):
- _greedy_literal_match: when navigating an EXISTING mapping, prefer an
existing literal key equal to the dot-join of the next N path segments
(longest match wins). Dotted model IDs are the norm, so the common
unescaped command (config set providers.p.models.grok-4.6.supports_vision
true) now hits the real key across set/get/unset instead of creating a
phantom sibling. Plain dotted paths with no dotted-key collision split
exactly as before.
- _phantom_sibling + ValueError in _set_nested: refuse to CREATE a new
intermediate mapping that would shadow an existing dotted literal sibling
(Soju06's fail-loudly suggestion on #84064); set_config_value surfaces it
as a clean CLI error with the escaped spelling to use.
- utils.py::atomic_roundtrip_yaml_update (the second split site, #91607 —
/model + TUI persistence) now uses the same escape-aware split + greedy
literal matching.
- CFG-04 empty-segment guard now splits escape-aware so escaped keys are
not misclassified.
- Tests for every repro shape in the family: #84064 provider model keys,
#80006 Matrix room IDs, #91095 dotted models under custom_providers list
index (incl. escaped creation-when-absent), #91607 model_overrides via
atomic_roundtrip_yaml_update, #99124 dotted leaf keys; plus
backward-compat coverage. Also fixed the carrier's one stale assertion
(structured-value coercion landed on main after #84152 branched) and
removed its dead _MCP_SECRETS_CONFIG fixture flagged in review.
- Docs: 'Dots inside key names' section in website/docs/reference/cli-commands.md.
Fixes #84064, fixes #80006, fixes #91095, fixes #91607, fixes #99124
* fix(agent): cap compaction threshold floor at 85% of the context window
The MINIMUM_CONTEXT_LENGTH floor in _compute_threshold_tokens only
degraded to the 85% trigger when it met or exceeded the effective
window exactly (#14690). Near-minimum windows slipped through: at
context_length=65536 the threshold passed through at 64,000 — 97.7%
of the window, ~1.5K tokens of output room — so pre-API compaction
effectively could not fire.
Providers that silently truncate over-window prompts instead of
rejecting them (e.g. ollama's OpenAI-compatible /v1 endpoint) never
deliver the reactive context-overflow backstop either. Observed live
on a 65,536-token local model: the session rode into the window
ceiling and each length-continuation retry re-sent a window-filling
prompt (65,120 -> 65,273 prompt tokens, 263 output tokens of room)
until the turn died with "Response remained truncated after 4
continuation attempts" — every retry paying a full multi-minute
prefill.
Cap the floored threshold at _MIN_CTX_TRIGGER_RATIO (85%) of the
effective input budget whenever the floor is the binding term. An
explicit threshold_percent above 85% is user intent and stays
uncapped; windows where the floor lands at/below the cap are
unchanged.
* chore: map komzpa@gmail.com -> Komzpa in contributor email registry
* fix(model_metadata): parse Google's 'supports up to N' context-limit phrasing
Google Gemini/Gemma overflow errors read 'Unable to submit request because
the input token count is 32825 but model only supports up to 32768'.
parse_context_limit_from_error had no pattern for the 'supports up to N'
phrasing, so overflow recovery kept the wrong window and burned its retry
attempts instead of recalibrating to the provider-reported limit.
Add the anchored pattern (limit follows 'supports up to'; the larger input
count before it is never captured) plus regression tests covering the exact
message and the get_context_length_from_provider_error recalibration path.
Reported by @Artemonim in #57275 (residual claim 5).
* fix(gateway): keep long turns controllable without blocking Telegram
* fix(update): self-heal broken Git-for-Windows trampoline on Windows
A Git-for-Windows trampoline launcher (bin\git.exe / cmd\git.exe shim,
~46KB) that fails to re-exec the real git-core binary refuses every git
call with a "BUG (fork bomb)" guard instead of running it (#87876).
Detect the trampoline up front via `git --version`, locate a real git
binary (Git for Windows or Hermes-managed PortableGit locations), and
rebuild the git command with it so fetch/pull/checkout keep working with
a real git instead of degrading to the ZIP fallback. When no real binary
can be found, leave the command untouched so the existing fetch-failure
handler still falls back to the ZIP path on Windows (#88046).
* fix(update): locate PortableGit under the shared root, not profile home
Review feedback on #88136 (monerostar): a profile-scoped `hermes update`
sets HERMES_HOME to <root>/profiles/<name>, but the Hermes-managed
PortableGit tree lives under the SHARED root (<root>/git/...). The locator
checked get_hermes_home() only, so a broken trampoline during a
profile-scoped update was not swapped and fell through to ZIP.
Extract _portable_git_candidates() (shared root first, profile home as
fallback) and add a regression test for the profile layout.
* test(windows): live E2E for the git trampoline self-heal on the wine2e lane
Real windows-latest coverage for the #88136 salvage: probes drive the
actual _git_is_trampoline/_locate_real_git/_ensure_non_trampoline_git
helpers against the runner's genuine Git-for-Windows install plus a real
fork-bomb-guard trampoline stand-in. Wired into the on-demand
windows-venv-e2e lane (wine2e/** pushes only).
* fix(agent_init): clamp compressor window to Ollama num_ctx resolved after construction
model.ollama_num_ctx is resolved AFTER the context compressor is
constructed, so a config that sets only ollama_num_ctx (without
model.context_length) ran every request at the smaller served num_ctx
while the compressor still targeted the probed GGUF window (e.g. 256K
Gemma metadata). The compaction trigger then sat several times above the
window the server actually serves and never fired — reproducing the
original #57275 'blows past the limit' symptom on current main.
Live repro (real imports, temp HERMES_HOME, config = {model:
{ollama_num_ctx: 65536}}, probed window 262144):
before: _ollama_num_ctx=65536, compressor.context_length=262144,
threshold_tokens=196608 (300% of the served window)
after: compressor.context_length=65536, threshold below the window
The clamp is one-directional (a num_ctx larger than the resolved window
never inflates the compressor) and reuses update_model() so every
threshold-derived budget recalibrates. Overlaps #60103 (silent-clamp
dead zone) — this is the init-order half.
Reported by @Artemonim in #57275 (residual claim 3).
* fix(agent_init): reserve Gemini's default maxOutputTokens in the compressor when max_tokens is unset
The native generateContent adapter never runs uncapped: when
model.max_tokens is unset it sends maxOutputTokens=65,535
(GEMINI_DEFAULT_MAX_OUTPUT_TOKENS) because Gemini treats an omitted cap
as a low internal default. The context compressor's trigger is
pct×(window − max_tokens), and constructing it with max_tokens=None
reserved 0 — so on a 128K Gemma window the trigger landed at 98,304
while the real safe input budget was 65,537, and the provider 400'd
before compaction fired.
Live repro (real imports, temp HERMES_HOME, native Gemini base_url,
window=131072, max_tokens unset):
before: compressor.max_tokens=None, threshold_tokens=98304,
wire maxOutputTokens=65535 → trigger ABOVE the safe budget
after: compressor.max_tokens=65535, threshold_tokens=64000 → below it
Scoped to the native Gemini wiring (provider names + native base_url via
is_native_gemini_base_url; the /openai compat endpoint is excluded). The
generic provider-default reservation gap remains tracked in #63839.
Reported by @Artemonim in #57275 (residual claim 4).
* fix(telegram): recover Windows CLOSE-WAIT getUpdates deadlock
After updater.stop() times out, HTTPXRequest.initialize() is a no-op unless
the client is already closed, so start_polling reused the wedged socket and
the gateway stayed alive but deaf. Rebuild the polling client after a hung
drain, watch getUpdates I/O independently of get_me(), and enable TCP
keepalive on the fallback transport.
* test(telegram): cover CLOSE-WAIT drain rebuild and getUpdates liveness
* fix(telegram): prevent Windows long-poll socket reuse deadlock
Prevent the dedicated getUpdates pool from reusing server-closed connections and replace a polling HTTP client left open after a timed-out CLOSE-WAIT drain. Keep the general Bot API pool reusable so concurrent sends and edits are unaffected. Add regression coverage for both transport limits and stale-client replacement. Fixes #87057
* fix(telegram): bound stale-client cleanup and add Windows CLOSE-WAIT live probes (#87057)
Follow-ups on top of the salvaged commits from PR #87111 (@HexLab98) and
PR #87265 (@JoaoMarcos44):
- keep main's #92991 stall watchdog (150s progress-based) as the single
steady-state liveness probe instead of adding a second overlapping one
- orphaned-client aclose() cleanup uses the wall-clock thread deadline and
is tracked in _background_tasks so a wedged close can neither hang nor
leak one task per reconnect attempt (from #87265's review findings)
- merge #87265's no-keepalive getUpdates pool (max_keepalive_connections=0)
with #87111's TCP-keepalive socket options on all transports
- add tests/gateway/test_telegram_closewait_windows_live.py: live probes
against a real half-closing HTTP server, skipif non-win32, wired into
the on-demand windows-venv-e2e lane (wine2e/**)
* chore: release v0.21.0 (2026.8.31)
* test: stabilize Telegram deadline assertion on Windows
* fix: address hosted room review findings
* fix: close hosted room publication races
* fix(bot-mode): preserve UTF-8 local DM delivery on Windows
* fix(hosted-rooms): close remaining lifecycle races
* fix(bot-mode): use subprocess env factory for peer delivery
* test: accept asynchronous stop settlement
* test: replace fixed waits with lifecycle conditions
* fix(groups): close hosted room review races
* fix: close hosted room authority review gaps
* docs: record v0.21.0 reconciliation provenance
* chore(ci): probe PR21 storage reconciliation
* chore(ci): add temporary PR21 patch1 TDD runner
* chore(ci): run PR21 patch1 storage TDD
* fix(ci): validate PR21 patch1 workflow context
* chore(ci): add sequential PR21 storage reconciler
* fix(ci): preserve exact v0.21 storage APIs in sequential gate
* fix(ci): test sequential PR21 storage reconciliation
* fix(ci): use AST-guided PR21 storage reconciliation
* fix(ci): run AST-guided PR21 storage reconciliation
* fix(ci): port PR19 control reserve fixture with storage invariant
* fix(ci): verify PR21 storage invariant with final fixture
* fix(ci): scope PR21 format gate to new test
* fix(hosted-rooms): restore storage recovery invariants
* chore(ci): add PR21 patch2 driver compatibility probe
* chore(ci): run PR21 patch2 driver probe
* chore(ci): expand PR21 patch2 compatibility matrix
* chore(ci): add selective PR21 patch2 reconciler
* chore(ci): run PR21 patch2 driver TDD
* fix(ci): align Patch2 RED gates with root cause
* fix(ci): compare Patch2 runtime against baseline
* fix(hosted-rooms): restore atomic driver admission
* chore(ci): add selective PR21 patch3 reconciler
* chore(ci): run PR21 patch3 replica TDD
* fix(ci): align Patch3 RED capacity gate
* fix(ci): normalize Patch3 selective output
* fix(ci): include Patch3 demotion RPC contract
* fix(ci): keep Patch3 formatting scope minimal
* fix(hosted-rooms): restore replica correctness
* chore(ci): analyze Patch4 runtime delta
* fix(ci): preserve Patch4 RED baseline evidence
* chore(ci): inventory Patch4 runtime AST
* chore(ci): snapshot Patch4 runtime sources
* chore(ci): include Patch4 state contracts
* chore(ci): stage Patch4 runtime probe
* test(ci): probe Patch4 runtime candidate
* fix(ci): run Patch4 probe without unavailable retry plugin
* fix(ci): probe Patch4 without replacing current attempt loop
* chore(ci): expose exact Patch4 candidate AST delta
* fix(ci): probe compatible Patch4 runtime hybrid
* fix(ci): probe final Patch4 compatibility conditions
* chore(ci): add Patch4 compatibility transformer
* fix(ci): verify final Patch4 hybrid candidate
* fix(ci): apply verified Patch4 runtime liveness
* fix(ci): prove Patch4 adds no service regressions
* chore(ci): add Patch4 service compatibility transformer
* fix(ci): probe Patch4 service lifecycle integration
* fix(ci): fetch immutable PR19 service source
* fix(ci): make Patch4 completion win local Stop race
* fix(ci): stress Patch4 completion race before service probe
* test(ci): preserve Patch4 admission fences in hybrid probe
* test(ci): probe Patch4 admission-fenced runtime hybrid
* fix(ci): keep receipt-safe Patch4 cancel path
* fix(ci): preserve exact peer Stop acknowledgement
* fix(ci): rerun Patch4 admission-fenced probe
* test(ci): verify peer-safe Patch4 runtime hybrid
* test(ci): add selective Patch4 state contract builder
* fix(ci): include selective Service uuid dependency
* test(ci): probe selective Patch4 state and service contracts
* test(ci): add Patch4 service compatibility builder
* test(ci): run full Patch4 compatibility probe
* fix(ci): verify Patch4 hybrid contracts semantically
* fix(ci): preserve reachable Stop and peer terminal harvest
* test(ci): prove reachable Stop and peer terminal compatibility
* test(ci): add Patch4 approval and service compatibility v2
* test(ci): run Patch4 full compatibility probe v15
* test(ci): add Patch4 immutable approval and owner fixture v3
* test(ci): run Patch4 full compatibility probe v16
* test(ci): make replay-page byte fixture deterministic
* test(ci): con…
kvnloo
pushed a commit
to kvnloo/hermes-agent
that referenced
this pull request
Sep 15, 2026
…live probes (NousResearch#87057) Follow-ups on top of the salvaged commits from PR NousResearch#87111 (@HexLab98) and PR NousResearch#87265 (@JoaoMarcos44): - keep main's NousResearch#92991 stall watchdog (150s progress-based) as the single steady-state liveness probe instead of adding a second overlapping one - orphaned-client aclose() cleanup uses the wall-clock thread deadline and is tracked in _background_tasks so a wedged close can neither hang nor leak one task per reconnect attempt (from NousResearch#87265's review findings) - merge NousResearch#87265's no-keepalive getUpdates pool (max_keepalive_connections=0) with NousResearch#87111's TCP-keepalive socket options on all transports - add tests/gateway/test_telegram_closewait_windows_live.py: live probes against a real half-closing HTTP server, skipif non-win32, wired into the on-demand windows-venv-e2e lane (wine2e/**)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
updater.stop()can time out on a stale long-poll socket. PTB'sHTTPXRequest.initialize()only rebuilds the httpx client whenis_closedis true, so a hungaclose()leftstart_polling()on the same CLOSE-WAIT connection. The gateway process stayed alive (PID, cron, heartbeat) while inboundgetUpdatesblocked indefinitely (#87057).get_me()/pending_update_count), and enable TCP keepalive on the fallback transport used forapi.telegram.orgIPs such as149.154.166.110.Test plan
scripts/run_tests.sh tests/gateway/test_telegram_network_reconnect.py tests/gateway/test_telegram_network.py tests/gateway/test_telegram_polling_progress.py tests/gateway/test_telegram_closewait_limits_31599.py tests/test_telegram_polling_progress_ptb.py tests/gateway/test_telegram_init_deadline.py tests/gateway/test_telegram_pending_update_probe.py tests/gateway/test_telegram_conflict.py tests/gateway/test_telegram_start_polling_timeout.py tests/gateway/test_telegram_connect.pyTimedOut/Bad Gateway, confirm logs still showupdater.stop() timed outthen a client rebuild, and that new DMs arrive instead of a silent 48h hang.