fix(compression): persist anti-thrash state across process restarts - #69872
Merged
Conversation
Contributor
૮ >ﻌ< ა ci reviewran on 6eebbf1 ℹ️ InfoDesktop E2E visual evidence · View test artifacts · View job1 visual diff. inline evidence upload failed. Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso) |
alt-glitch
added
type/bug
The anti-thrash guard (_ineffective_compression_count) was in-memory only: a fresh compressor bound to a resumed, already-compacted session started with compression_count=0 and a disarmed guard, so a near-threshold session could legally re-compact once per process restart, forever. Persist the counter through the durable session-state channel, mirroring the failure-cooldown (#54465) and fallback-streak (af7dcea) pattern: - hermes_state.py: sessions.compression_ineffective_count column (declarative reconciliation adds it on existing DBs) + get/set_compression_ineffective_count accessors. - context_compressor.py: every strike/clear verdict routes through _record_ineffective_compression_verdict() which writes through to the session row (no-change verdicts skip the DB write); bind_session_state() loads the persisted value; the compression rotation boundary carries the counter onto the child row; update_model()'s reset also clears the durable copy; the ineffective-only fast path in _automatic_compression_blocked() is removed because the counter is now durable and another agent's clear must unblock a stale local snapshot. - conversation_compression.py: _refresh_persisted_compression_guards re-reads the counter alongside cooldown + fallback streak. Reset semantics are unchanged: any real provider reading below the threshold still clears the counter — and now clears it durably too. Resolves the residual gap identified in #54923 by @lanyusea (the second-threshold mechanism was superseded by persisting the existing guard state). Co-authored-by: lanyusea <lanyusea@gmail.com>
teknium1
force-pushed
the
salvage/54923-hysteresis
branch
from
July 23, 2026 14:40
f119afb to
6eebbf1
Compare
vashkartik
added a commit
to vashkartik/hermes-agent
that referenced
this pull request
Jul 24, 2026
* fix(agent): demote oversized tool results in protected compression tail
After multiple in-place compactions, short tool-heavy sessions can leave
nearly every remaining message inside protect_last_n while those messages
are huge completed file/tool outputs. The middle compress window then
makes no material token progress and the turn dies with
"Cannot compress further" (#61932).
Cap the prune message floor at the same bound as tail-cut, and under
pressure demote bulky protected-tail tool bodies (keeping a short recent
floor) so preflight can reclaim headroom without wiping the active ask.
* test(agent): cover protected-tail last resort
* test(agent): pin the #61932 all-oversized-tail dead-end shape as compressible
Regression test for the exact issue #61932 report: head + an 8-message
protected tail made exclusively of oversized tool pairs. Pre-fix,
compress_start >= compress_end made compress() a pure no-op and the
retry loop ended in 'Cannot compress further'; post-fix the Phase-1
pressure demotion reclaims the tail in one pass while preserving
tool_call/tool_result pairing.
* fix(compression): merge todo snapshot into trailing user msg to avoid consecutive user/user turns
After context compression, the preserved todo list was unconditionally
appended as a standalone user message. When the compressed transcript
already ends with a user message (common case), this creates consecutive
user/user turns — a role-alternation violation some providers reject.
Fix: fold the snapshot into the trailing user message (blank-line separated)
when one exists with plain-string content. Falls back to append when the
tail is non-user, empty, or has structured (list) content.
Rebased on current upstream/main.
Closes #53890
* fix(compression): preserve multimodal todo tails
* fix(compression): gate todo-snapshot merge on real-user tails, refresh stale snapshots
Follow-up hardening on the salvaged merge-into-trailing-turn fix:
- Merge only into REAL user tails (_is_real_user_message probe). Merging
into scaffolding tails (continuation marker, summary-as-user handoff)
would upgrade them to real-user evidence after SessionDB projection
strips the flags, breaking zero-user provenance (#69292 -
_is_synthetic_compression_user_turn keys on the TODO_INJECTION_HEADER
content marker, which merge-at-tail would bury mid-content).
- Strip a previously merged snapshot block before re-injection so
repeated boundaries refresh rather than accumulate todo state, and
refresh a bare stale snapshot row in place instead of stacking a
duplicate (empty/stale-skip semantics from #26981 by @YLChen-007).
- Scaffolding tails keep the flagged standalone append (pre-#53890
status quo; adjacent user rows are repaired downstream by
repair_message_sequence / _merge_consecutive_roles).
* test(compression): pin scaffolding-tail standalone append + stale-snapshot refresh
Covers the follow-up hardening: continuation-marker and summary-as-user
tails keep the flagged standalone snapshot (zero-user provenance #69292
verified via _transcript_has_real_user_turn on the projected rows),
stale snapshot rows are refreshed in place, a previously merged snapshot
is stripped before re-injection, and an all-completed todo store injects
nothing (#26981).
* fix(context-engine): honor quiet compaction status
* test: isolate quiet compaction status assertions
* fix(context-engine): adapt quiet compaction status to turn-context refactor
* fix(context-engine): route pre-API and idle compaction status through the quiet-engine resolver
Follow-up for the salvaged #35191: the mid-turn pre-API pressure emit in
conversation_loop.py and the idle-resume emit in turn_context.py were not
routed through automatic_compaction_status_message, so an engine with
emit_automatic_compaction_status=False still leaked those lines. Both now
resolve through the hook (phases "pre_api" and "idle") while keeping the
#69550 template constants as the default wording. Suppression also skips the
#69546 structured 'compacted' terminal edge for compress-phase events that
opened no visible phase; failure warnings (_emit_warning) remain never
suppressible, pinned by test.
* fix(gateway): bound hygiene compression failures
* fix(gateway): route hygiene-timeout warning via profile-aware adapter lookup + verify lock reacquire after fence cancel
- gateway/run.py: use _adapter_for_source(source) instead of the raw
adapters.get(source.platform) map so the compression-timeout warning
respects transport provenance, relay ingress, and multiplexed profiles
(matches every other user-facing send in the hygiene block).
- tests: add a lock-release verification regression — a fence-cancelled
hygiene compression must leave the per-session compression lock free so
the next attempt (manual /compress retry) acquires it and commits
normally.
* fix(cli): widen startup worktree pruning to all .worktrees/ trees and detect squash-merged work (#69831)
The startup pruner only considered directories named hermes-* (the
hermes -w scratch trees), so salvage/review/port lanes created with raw
'git worktree add' accumulated forever — a real checkout reached 117
directories / 26 GB with trees dating back months. Two further leaks:
squash-merged branches' local commits stay unreachable from
refs/remotes/* forever, so the unpushed-commits guard preserved fully
merged scratch trees indefinitely; and preserved trees rotted silently
with no visibility.
- Pruner now covers every directory under .worktrees/ except kanban
task trees (t_<hex>, owned by 'hermes kanban gc'). Named (non
hermes-*) trees get a 3x timeline (72h soft / 9d hard) since they
were created deliberately.
- New _worktree_commits_all_merged_upstream(): git-cherry
patch-equivalence check against origin/HEAD|main|master, bounded at
20 commits ahead, fails safe toward preserve. Lets the pruner reap
trees whose every local-only commit already landed upstream via
squash-merge/cherry-pick.
- Dirty guard now applies at every tier (previously the 24-72h tier
skipped it — it only survived because the unpushed check usually
caught the same trees).
- Trees preserved for unpushed/dirty reasons older than 7 days are
listed in a single WARNING so in-flight work can't rot silently.
- tips.py text updated; 13 new behavior-contract tests.
* fix(update): self-heal venv after failed lazy backend refresh
Upgrade pip before lazy refreshes, probe core imports when a lazy
install fails, force-reinstall corrupted packages with pyproject pins,
use package-only install (no shim quarantine) for repair, and keep the
.update-incomplete marker until refresh/repair succeeds (#57828).
* test(update): cover lazy refresh venv repair after failed installs
Add repair/probe/quarantine regression tests and update autostash mocks
for the new lazy-refresh signature.
* fix(update): import-based recovery under Windows hermes.exe self-lock
Keep .update-incomplete across normal hermes.exe launches, heal via
package-only import probes first, and only clear the marker after repair
succeeds (#57828 / #58004 review).
* fix(update): split core vs lazy markers; probes cannot false-clear
Keep .update-incomplete for full .[all] recovery only. Lazy refresh uses
.lazy-refresh-incomplete and clears only after confirmed import probes;
unavailable probes are indeterminate, not healthy (#58004 review).
* fix(update): stdlib-only early recovery before hermes_cli.main imports
The hermes console entry point is hermes_cli.main:main, and main.py imports
dotenv (via env_loader) and yaml (via config) at module level. In the #57828
failure state — a failed lazy backend refresh wiping a core package's import
files while metadata survives — a normal launch crashed while importing
main.py, before _recover_from_interrupted_install() and the recovery markers
from PR #58004 could act.
- hermes_cli/_early_recovery.py: stdlib-only bootstrap repair invoked at the
very top of main.py, before any third-party import. Probes the fragile
core packages via real imports, force-reinstalls broken ones using the
pyproject.toml pins, shares main.py's single-flight recovery lock, and
never clears markers (the confirmed lifecycle stays with the full recovery
path in main.py).
- Probe/repair tables now have one canonical home in _early_recovery, reused
by main.py so the two layers cannot drift.
- Manual --force-reinstall fallback commands now print pinned specs via
_lazy_refresh_repair_specs() instead of bare package names.
- tests: entry-point lifecycle coverage proving a broken dotenv import
crashes main.py without repair and imports cleanly with it, a stdlib-only
import guard for _early_recovery, and unit coverage for marker gating,
lock single-flight, pinned specs, and marker preservation.
* fix(credential-pool): attribute failures to the key that failed, not the shared current() pointer
recover_with_credential_pool identified "which credential failed" via
pool.current(), a shared mutable pointer that is advanced by every
select() (round-robin rotation, concurrent turns, and other processes
reloading the pool reset it to None). By the time recovery ran, it
routinely pointed at a different, healthy entry — mark_exhausted_and_rotate
then stamped the failing request's error message and reset time onto that
innocent entry. With round_robin and one hard-capped key this
deterministically exhausted the healthy key too and took the entire pool
offline ("no available entries") from a single rate-limited credential.
mark_exhausted_and_rotate already supports api_key_hint for exactly this
(the auxiliary-client path passes it); the main conversation-loop path
never did. Pass agent.api_key — kept in sync with the entry in use by
_swap_credential — as the hint on all four rotation call sites, and make
the "already exhausted → rotate immediately" pre-check look up the failing
entry by key with the same fallback to current().
Adds regression tests that fail on the old attribution logic: a fresh
pool (current() is None) failing on key B must mark entry B, never
entry A.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(credential-pool): refresh the failing entry, not current(), on auth recovery
Review follow-up: the auth path called pool.try_refresh_current() before
the hinted rotation, so a stale current() pointer could force-refresh a
different, healthy entry — consuming its single-use refresh token, or
(for non-OAuth entries, where a forced refresh marks the entry exhausted
outright) killing it entirely before api_key_hint was ever consulted.
Use try_refresh_matching(api_key_hint=...) to resolve and refresh the
entry that supplied the failing key under the pool lock, falling back to
the previous behavior when no key is known.
Adds a regression test with current() deliberately pointed at the
healthy entry: on the old code the healthy entry is exhausted by the
forced refresh and the pool ends up fully offline; with the fix the
failing entry is exhausted and recovery rotates to the healthy one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: make interrupt-pool double's entries callable
Follow-up to the #58738 salvage: the pre-exhausted check now enumerates
pool.entries() to find the failing key, so the MagicMock pool double must
expose entries as a callable, not a bare list.
* chore(contributors): add email mapping for schattenan
* fix(windows): hide console flashes in GUI-reachable exec paths and provider transports (#56747)
Six spawn sites reachable from the desktop GUI / TUI gateway lacked CREATE_NO_WINDOW, so a windowless parent (pythonw/Electron) flashed a conhost per spawn: cli.exec RPC, quick-commands exec dispatch, and shell.exec RPC in tui_gateway/server.py; the CLI REPL quick-commands exec in cli.py; and the per-session provider transports in agent/copilot_acp_client.py and agent/transports/codex_app_server.py (Popen, hide-only so PIPE stdio stays intact).
All use hermes_cli._subprocess_compat.windows_hide_flags() (no-op on POSIX), matching the pattern already used at three other sites in tui_gateway/server.py. Deliberately hide-only — no detach flags, no Electron changes (per the #54220 revert history).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(windows): regression coverage for the six #56747 hide-flag sites
Mocked-subprocess tests asserting creationflags == CREATE_NO_WINDOW for
each path salvaged from PR #56877: tui_gateway cli.exec / shell.exec /
quick-command dispatch, the CLI quick-command exec handler, and the
Copilot ACP + Codex app-server Popen transports (pipes asserted intact).
Verified: all 6 fail with the fix reverted, pass with it applied.
* fix(kanban): isolate delegated children from parent task
* fix(kanban): harden delegated-child mutation boundary
* chore: map trkim@vms-solutions.com to ddifa86
* test: pin repo root on PYTHONPATH for subprocess-boundary kanban isolation tests
The three subprocess tests spawn 'sys.executable -c' children that import
hermes_cli. From a worktree, the child resolved the MAIN checkout's editable
install instead of the tree under test, so the new DB/CLI guards appeared
missing and the tests failed with rc=0. Route the spawns through a helper
that pins the repo root under test on PYTHONPATH.
* fix(auxiliary): unwrap explicit provider:moa to its aggregator, not the literal name
_resolve_task_provider_model() returned an explicit provider="moa" override
(from a caller-passed arg, or auxiliary.<task>.provider: moa in config.yaml)
verbatim, with no MoA-preset unwrap. Only the *implicit* "main provider is
moa" path inside _resolve_auto() unwraps to the aggregator slot (#53827) —
this function never goes through _resolve_auto() at all, so the explicit
case was never covered.
MoA is a virtual provider with no real HTTP endpoint: resolve_provider_client()
looks "moa" up in PROVIDER_REGISTRY (no such entry), falls to the
unknown-provider dead end, and call_llm surfaces a nonsensical "Provider
'moa' is set in config.yaml but no API key was found. Set the MOA_API_KEY
environment variable..." error for a provider that was never meant to be
reached over the wire.
Fix mirrors #53827's aggregator-resolution approach exactly: when either the
explicit `provider` arg or the config-derived `cfg_provider` is "moa",
resolve the named (or default) MoA preset via resolve_moa_preset() and
continue with its aggregator's real provider+model, dropping any explicit
base_url/api_key (the moa:// virtual endpoint and placeholder key belong to
the facade, not the aggregator's real provider). If the preset can't be
resolved (renamed/deleted), degrades gracefully to the pre-fix behavior
instead of raising harder.
- agent/auxiliary_client.py: _unwrap_moa_provider() helper + call sites for
both the explicit-arg and config-derived provider="moa" cases in
_resolve_task_provider_model(). Also tightened base_url/api_key parameter
types to Optional[str] (matching their actual None-accepting behavior),
which incidentally resolved 5 pre-existing ty diagnostics at call sites.
- 5 new regression tests in tests/agent/test_auxiliary_client.py: explicit
arg unwrap, config-derived unwrap, default-preset fallback when no model
is configured, graceful degradation on preset-resolution failure, and a
non-moa regression guard.
* fix(auxiliary): route all MoA aux resolution through one shared aggregator helper
Follow-up to srojk34's explicit-provider unwrap (PR #56691):
- Extract _resolve_moa_aggregator() as the single preset->aggregator
resolver shared by _resolve_auto(), _resolve_task_provider_model(),
and resolve_provider_client() so preset lookup/validation can't drift.
- When the main provider is moa, the aggregator model is now the default
for every UNSET auxiliary model: _read_main_model_for_aux() substitutes
the preset's acting (aggregator) model wherever fallback chains
pre-filled from _read_main_model() (router prefill, custom-endpoint
fallback, named-custom default, external-process default,
_try_main_agent_model_fallback).
- Unwrap moa at the resolve_provider_client() chokepoint so direct
callers (vision auto-detect, plugin code) can't dead-end in the
unknown-provider branch, and unwrap the vision auto-detect main
provider before capability probes run against the preset name.
- Real-config tests: temp HERMES_HOME + actual config.yaml exercising
the genuine load_config()/resolve_moa_preset() boundary.
* fix(api_server): fail closed when API_SERVER_KEY strength can't be verified
`_api_key_passes_startup_guard` refuses to start the API server on a weak
`API_SERVER_KEY`, and its own log says why:
This endpoint dispatches terminal-capable agent work — a guessable key
is remote code execution.
But the check is wrapped so that a failure to import it starts the server
anyway:
try:
from hermes_cli.auth import has_usable_secret
if not has_usable_secret(self._api_key, min_length=16):
... return False
except ImportError:
pass
return True
`hermes_cli.auth` imports httpx at module scope and pulls in a large slice of
the CLI, so an import failure is not hypothetical — a trimmed image, a partial
install, or a circular import during gateway startup all produce one. When it
happens the strength check silently disappears and only the presence check
above it remains, so a placeholder key passes.
Reproduced against the real guard with the import blocked:
weak key, normal : False
weak key, ImportError : True <-- starts on a 4-char key
strong key, normal : True
Fail closed instead: an unverifiable key does not get to expose the endpoint,
and the log names the actual problem so the operator can repair the install.
This is the posture tools/credential_files.py already takes — it refuses a
mount when its deny-list cannot be consulted rather than risking it. The catch
also widens from ImportError to Exception, so an AttributeError or an error
raised inside the check cannot reopen the same hole.
Both happy paths are untouched: a strong key still starts, a weak or missing
key is still refused with the existing messages.
Unrelated to #38803, which fixes the retry behaviour after this guard rejects
and assumes the guard ran.
tests/gateway/test_api_server.py: new TestApiKeyStartupGuardFailsClosed — a
weak key is refused when the check is unavailable, a strong key is refused too
(fail-closed), plus three controls pinning the unchanged normal paths. The two
fail-open tests fail on main; the three controls pass there. 222 passed in the
api_server suites; 1475 passed across every suite touching api_server, with
the same 8 pre-existing failures on clean main.
* feat(skills): add tldraw-offline agent scripting skill
Optional skill for driving the tldraw offline desktop app via its local
HTTP control API (the same curl-based path the app's own agent skills use
for Codex/Claude Code/Cursor/Gemini) — read the canvas, make live edits,
and write embedded document scripts.
Grounded in the app's bundled script-context.d.ts and agent playbook, and
in the real tldraw SDK v5 shape schema:
- document-script contract: export default function ({ editor, helpers, signal })
- HTTP API: /api/search, /api/doc/:id/exec, /api/doc/:id/script-workspace,
/api/doc/:id/script-status (bearer token from server.json, re-read per call)
- shape schema table validated against @tldraw/tlschema (scripts/validate_shapes.mjs, 3/3)
- interactive-UI example (scripts/counter.js) + diagram-generation (scripts/main.js)
- honest verification boundary: click->state logic verified via /exec dispatch
(0->1->2->1->0), with documented host caveats (inotify watcher, Electron
background-click rejection)
Tests: tests/skills/test_tldraw_offline_skill.py (15 passing).
* fix(skills/tldraw-offline): correct the computer-use delivery note
The skill claimed Chromium/Electron 'reject synthetic clicks' so computer-use
can't drive the canvas. The Cua team disproved this on the exact v1.11.0
AppImage (Linux/X11): background delivery returns background_unavailable, but
that's the first rung, not a wall — cua-driver returns escalation:'foreground'
and its X11 XTest path (x11_xtest_fg) with delivery_mode:'foreground' clicks
through, dismissing the consent dialog and landing canvas clicks.
Corrected the note to say: climb to foreground on background_unavailable, don't
conclude Electron is unclickable. Ref: NousResearch/hermes-agent#67052.
* fix(compression): persist anti-thrash state across process restarts (#69872)
The anti-thrash guard (_ineffective_compression_count) was in-memory
only: a fresh compressor bound to a resumed, already-compacted session
started with compression_count=0 and a disarmed guard, so a
near-threshold session could legally re-compact once per process
restart, forever.
Persist the counter through the durable session-state channel,
mirroring the failure-cooldown (#54465) and fallback-streak (af7dceaf7)
pattern:
- hermes_state.py: sessions.compression_ineffective_count column
(declarative reconciliation adds it on existing DBs) +
get/set_compression_ineffective_count accessors.
- context_compressor.py: every strike/clear verdict routes through
_record_ineffective_compression_verdict() which writes through to the
session row (no-change verdicts skip the DB write);
bind_session_state() loads the persisted value; the compression
rotation boundary carries the counter onto the child row;
update_model()'s reset also clears the durable copy; the
ineffective-only fast path in _automatic_compression_blocked() is
removed because the counter is now durable and another agent's clear
must unblock a stale local snapshot.
- conversation_compression.py: _refresh_persisted_compression_guards
re-reads the counter alongside cooldown + fallback streak.
Reset semantics are unchanged: any real provider reading below the
threshold still clears the counter — and now clears it durably too.
Resolves the residual gap identified in #54923 by @lanyusea (the
second-threshold mechanism was superseded by persisting the existing
guard state).
Co-authored-by: lanyusea <lanyusea@gmail.com>
* fix(computer-use): handle Linux cua window metadata
Treat cua-driver's Linux `is_on_screen: null` as unknown instead of
off-screen, and skip GNOME Shell desktop/backdrop helper windows
(ding "Desktop Icons", @!x,y;BDHF) when selecting the default capture
target — they are targetable X11 windows but capture as empty.
Reconciled with the _NET_ACTIVE_WINDOW fallback from #58030: helper
windows are filtered out of the candidate pool first, then the tied
z-order active-window probe runs on the remaining real app windows.
Also falls back to the requested app name for _last_app when Linux
windows carry no app_name.
Salvaged from #54173 by @dnth.
* fix: signal lock-hold to callers when compression skips
* fix(cli): show lock-hold reason when /compress no-ops
* fix(gateway): show lock-hold reason when /compress no-ops
* fix(tui): show lock-hold reason when /compress no-ops
* fix: prevent stale lock-skip signal leaking between compress_context calls
Advisor review found a critical stale-signal leak: if auto-compress
sets _compression_skipped_due_to_lock during a lock-skip, a subsequent
successful manual /compress will see the stale signal, falsely report
'Compression already in progress', and discard the compression results.
Fix:
- compress_context clears _compression_skipped_due_to_lock = None at
entry so each call's outcome alone determines the signal.
- Unified gateway 'holder: unknown' drift to match CLI/TUI pattern
(omit holder clause when not a descriptive string).
- Added MagicMock opt-outs in 3 sibling test files broken by the new
signal check (test_compress_here, test_compress_focus,
test_compress_plugin_engine).
- Added stale-signal-leak invariant test proving the fix.
* fix(compress): classify unconfirmed lock-acquire failures and cover all manual-compress surfaces
Follow-up to the salvaged #57634 commits:
- agent/manual_compression_feedback.py: new describe_compression_lock_skip()
— single source of truth for lock-skip wording. A descriptive holder
string means another compressor CONFIRMED holds the lock ('already in
progress (holder: ...)'); True/None means acquisition failed without a
confirmed holder (hermes_state.try_acquire_compression_lock catches
sqlite3.Error internally and returns False), so the message says
'could not acquire ... the lock check failed' instead of falsely
claiming a concurrent compression is running.
- cli.py, gateway/slash_commands.py, tui_gateway/server.py (all three
in-process consumers: session.compress RPC, command.dispatch compress
branch, slash.exec mirror) now route through the shared helper.
- tui_gateway/server.py command.dispatch compress branch: catch
CompressionLockHeld explicitly — it previously fell into the generic
'compress failed' error handler.
- Deferred-notify contract (#69324): lock-skip discards the pending
context-engine notification (committed=False) in _compress_session_history
and the CLI path before returning.
- tests: lock-skip wording pins per surface, VISIBLE_COMPRESSION_MESSAGES
noise-filter carve-outs for both wordings, MagicMock signal opt-outs for
sibling tests added on main after the original PR.
* fix(compress): type-pin the lock-skip signal check at every consumer
The bare truthiness test on _compression_skipped_due_to_lock is fooled
by MagicMock auto-attributes on test-double agents (skill pitfall:
MagicMock defeats hasattr/truthiness duck-typing) — the type-ahead CLI
test's MagicMock agent took the lock-skip branch and skipped the
transcript commit. Real values are None/True/holder-string; pin the
check to 'is True or isinstance(str)' at all three consumer sites.
* fix(moa): route Copilot slots by target model
* fix(moa): route every Copilot credential path by target
* feat(image_routing): accept vision alias for custom provider models
Extend the existing candidate-name resolver in _supports_vision_override
to accept 'vision' as an alias for 'supports_vision' on per-model config,
for both the providers.<name>.models dict and the legacy list-style
custom_providers form.
Per review feedback on #31912: this extends the current resolver rather
than replacing its candidate-name logic. Named custom providers resolve
to the runtime value provider='custom' while the config keeps the
user-declared name under model.provider; that lookup path is preserved.
Adds regression tests covering model.provider=my-vllm with runtime
provider='custom' for both config shapes.
* fix(kanban): keep delegated results in worker turn
Dispatcher-spawned Kanban workers are finite one-shot processes, so detached delegation completions can outlive their only consumer. Mark that runtime as unable to deliver async completions and reuse the synchronous delegation fallback, returning required child results before the worker exits.\n\nAlso make unsupported-session notes runtime-generic and cover the delayed-child lifecycle regression.\n\nRefs #63169
* chore: map rmk799@outlook.com to MustafaK99
* fix(kanban): isolate worker-created child workspaces
Default kanban_create children now keep fresh scratch paths, while explicit dir sharing remains supported and project context resolves to a per-task worktree. Surface resolved workspace fields in create responses/events and cover scratch mutation, nesting, explicit sharing, and project inheritance.
Fixes #67567
* fix(kanban): preserve cross-profile project child routing
* chore: map team@williepeacock.com to peacockesq
* Surface warning when context exceeds compression threshold but compression is blocked
Previously, when a session crossed the compression threshold but compression
was skipped (summary-LLM cooldown, #11529, or anti-thrashing, #40803), the
model kept accumulating context until it hit the hard provider token limit and
silently stopped answering — with no signal to the user about why.
Changes:
- context_compressor.should_compress_info() returns a (should_compress, reason)
tuple. reason is 'cooldown:<seconds>' or 'ineffective' when compression is
needed but blocked. should_compress() keeps its bool contract so existing
callers (conversation_loop.py) and regression #29335 are unaffected.
- turn_context.build_turn_context() emits a deduped _emit_warning when the
context is over threshold but compression is blocked, advising /new or
/compress. Dedup keys on the block *kind* (cooldown/ineffective), not the
ticking countdown, so a cooldown doesn't re-fire the warning every turn.
- Adds tests/agent/test_turn_context_overflow_warning.py covering the tuple
shape, both block kinds, dedup, and re-fire-after-clear.
* Address sweeper review: safe should_compress_info + cover all guards
- ContextEngine.should_compress_info() default impl so plugin engines
(e.g. _StubEngine) don't raise AttributeError at the call site.
- Centralise warning/reset in AIAgent._warn_context_overflow_blocked /
_clear_context_overflow_warn so turn-context and conversation-loop guards
share identical dedup logic and reset on the real compression boundary.
- Cover conversation_loop.py pre-API (~L1007) and loop-compaction (~L4774)
guards, not just the turn-context preflight.
- _FakeAgent mirrors the two helpers; test suite green (219 passed).
Fixes #62708
* fix(compression): reset blocked-overflow dedup on every compression path + noise-filter survival pins
Follow-up fixes for the #62625 salvage:
- Dedup-reset gap (sweeper review): when the block clears while the
context is STILL over threshold, execution enters the compression
branch — the PR's 'else' reset never ran, so the warning stayed
suppressed forever after the first block. _clear_context_overflow_warn()
now fires on every automatic compression path: turn-context preflight,
conversation_loop pre-API gate, and the post-tool loop-compaction gate.
- should_compress_info on current main: main refactored should_compress
into _automatic_compression_blocked()/_locally(); the tuple variant now
derives its reason from the same in-memory state via
_compression_block_reason(), keeping cooldown:<s>/ineffective shapes.
- ContextEngine.should_compress_info ABC default now actually returns
(should_compress(tokens), None) — the PR's default had a docstring but
no return (returned None, would crash tuple-unpacking call sites).
- Below-threshold guard: the turn-context persisted-cooldown branch and
the conversation_loop pre-API cooldown branch no longer warn when the
estimate is under threshold (should_compress_info returns a None
reason; the preflight pre-check is not a threshold guarantee). The
pre-API guard also honors compression.max_attempts instead of a
hardcoded 3, and no longer fabricates a cooldown reason.
- Noise-filter survival (#69550 composition): warning text is now a
template constant (CONTEXT_OVERFLOW_BLOCKED_WARNING_TEMPLATE) marked
FAILURE-CLASS, pinned un-swallowed in VISIBLE_COMPRESSION_MESSAGES and
in new tests that execute the real _TELEGRAM_NOISY_STATUS_RE +
_prepare_gateway_status_message.
- Contributor mapping for stanislav@local -> sl4m3.
* fix(compression): guard overflow-warn dedup reset against minimal test doubles
The dedup-reset calls assumed a full AIAgent; gateway/loop test doubles
built via object.__new__ lack _clear_context_overflow_warn and crashed
in build_turn_context (caught by test_api_content_sidecar on CI slice 3).
getattr-guard all four call sites per the established test-double pitfall
pattern (AGENTS.md #17).
* fix(compression): compose the blocked-warning probe with engine preflight
Two composition fixes vs the merged #69865 engine-preflight arm:
1. should_compress_info probe getattr-guarded — minimal compressor
doubles (SimpleNamespace) and plugin engines lack it; absence means
no block reason, no warning.
2. Engine maintenance hook stays un-consulted when any skip-branch
fired (failure cooldown / deferred estimate / codex-native) —
restoring the #20316 contract the warn-chain restructure broke.
* fix(credential-pool): exhaust all entries sharing a failed API key on 402
A 402/429/401 is an API-key–level failure (account out of balance,
rate-limited, or key rejected), but the same key can back more than one
pool entry — e.g. an explicit pool entry plus a `model_config` entry
auto-seeded from `model.api_key`, both carrying the identical
`runtime_api_key`.
`mark_exhausted_and_rotate(api_key_hint=...)` only marked the *first*
matching entry, leaving the sibling OK. `_select_unlocked()` then kept
handing back the same depleted key, so the billing-recovery `continue`
loop in the conversation retry path never converged: the request hung
until the client disconnected (~2.5min observed against DeepSeek),
emitting only `response.created` with no 402 ever surfaced to the user.
Mark every entry sharing the failed key so the pool can reach the
"no available entries" state and let the error propagate immediately.
Adds a regression test covering two entries backed by the same key.
* perf(credential-pool): persist same-key sibling exhaustion once
Follow-up to the #68565 salvage: batch the sibling _mark_exhausted calls
behind a single _persist() instead of one auth.json write per sibling.
* chore(contributors): add email mapping for airclear
* fix(update): survive undeletable untracked files during autostash (#70161)
git stash push --include-untracked exits non-zero when it saved
everything but could not DELETE some swept untracked files from the
working tree (e.g. a root-owned packaging/ directory left behind by a
sudo'd build: 'warning: failed to remove ...: Permission denied').
The updater ran the push with check=True, so this benign partial
failure raised CalledProcessError and aborted the whole update before
it even fetched — reliably, on every run, for any user with an
undeletable untracked path in the checkout.
Fix, both ends of the class:
- _stash_local_changes_if_needed: probe refs/stash before/after the
push. Non-zero push + fresh stash entry = changes are saved; warn,
reset the tracked-side leftovers (they're in the stash), and
continue the update. Non-zero push + NO stash entry = real failure;
keep aborting.
- _restore_stashed_changes: on restore, those same undeletable files
still sit in the tree, so 'git stash apply' exits 1 with 'already
exists, no checkout' even though every tracked change applied and
nothing was lost. Classify that stderr shape (strictly — any other
error line still routes to the conflict path) as restored instead
of resetting the tree and telling the user the restore failed.
Repro'd both halves with real git; behavioral E2E test covers
stash -> checkout -> restore round-trip with an undeletable dir.
* fix(credential-pool): stop lost-update cooldown erasure and wrong-key quarantine
Two related races in credential-pool cooldown state:
1. Lost update across processes: write_credential_pool merged only
entries missing from the caller's snapshot; for entries present on
both sides the caller's in-memory copy won wholesale. A process
holding a snapshot taken before another process marked a key
exhausted would, on its next persist (e.g. a round-robin rotation),
write the key back as healthy — erasing the cooldown so every
process resumes hammering a rate-limited key. Merge status fields by
last_status_at recency: adopt the on-disk status only when it is
strictly newer AND still binding (DEAD, or EXHAUSTED with an
unexpired cooldown), and never onto re-authed (token-changed)
entries, so legitimate expiry-clears and fresh logins are preserved.
2. Wrong-key quarantine: when mark_exhausted_and_rotate received an
api_key_hint that matched no entry, it fell through to
current()/_select_unlocked() — on a freshly loaded pool that selects
the NEXT healthy key and benches it for the full cooldown TTL,
punishing an innocent credential. When a hint is provided but
unmatched, rotate without marking anything instead of guessing.
Includes regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: isolate unmatched-hint regression from live ~/.claude credentials
Follow-up to the #65844 salvage: the new anthropic pool test must stub
read_claude_code_credentials like the sibling tests, otherwise a dev
machine's live claude_code singleton seeds a third entry and the
no-benching assertion fails outside CI.
* chore(contributors): add email mapping for drleadflow
* fix(kanban): stop decompose siblings sharing one worktree checkout
Decompose children inherit the root's literal workspace_path (#37172),
so every sibling of a worktree-kind root points at the SAME checkout.
_resolve_worktree_workspace's existing-checkout shortcut then reuses
that directory on whatever branch is currently checked out, ignoring
the task's own branch_name. Net effect: sibling workers — which can be
promoted and dispatched concurrently — run in one directory on the
first sibling's branch, with no lock. Work lands on the wrong task's
branch (provenance corruption) and concurrent siblings trample each
other's index/tree.
Fix, two layers:
- decompose_triage_task: worktree-kind children no longer inherit the
root's literal path; each child materializes its own
<repo>/.worktrees/<child-id> at dispatch (dir/scratch inheritance
unchanged — children legitimately share those).
- _resolve_worktree_workspace: when the requested path is an existing
checkout of a DIFFERENT branch, fall back to a fresh
<repo>/.worktrees/<task-id> instead of silently reusing it (heals
rows that already carry a shared path). Same-branch reuse and the
no-repo/own-path degenerate cases keep the legacy behaviour.
Tests: tests/hermes_cli/test_kanban_worktree_isolation.py (5); full
test_kanban_db.py + test_kanban_decompose_db.py suites pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(credential_pool): acquire the pool lock in has_available/peek/current/entries
`has_available()`, `peek()`, `current()` and `entries()` read (and, via
`_available_entries()`, mutate and persist) `self._entries` without holding
`self._lock`, while every other entry point — `select()`,
`mark_exhausted_and_rotate()`, `acquire_lease()`, `try_refresh_current()` —
guards the exact same access with the lock.
`_available_entries()` is not read-only: it prunes aged-out DEAD manual
entries (rebinding `self._entries` at the prune step) and calls `_persist()`
(writes auth.json). The gateway runs platform adapters in threads and cron
runs jobs in a ThreadPoolExecutor, so a status probe via `has_available()`
or `peek()` can race a concurrent `select()`/rotation: torn iteration of
`self._entries`, interleaved auth.json writes, or a lost token rotation.
Fix: take `self._lock` in all four query methods. Because the lock is
non-reentrant and `peek()` composes `current()` + `_available_entries()`,
add a lock-free `_current_unlocked()` helper and route the already-locked
internal callers (`_select_unlocked`, `mark_exhausted_and_rotate`,
`_try_refresh_current_unlocked`) through it to avoid self-deadlock.
Added regression tests: a no-deadlock check (peek re-entrancy) and a
lock-held-blocks-the-call check for each of the four methods.
* fix(credential_pool): complete the locking boundary across the public pool surface
Follow-up to review feedback:
- Acquire self._lock in the remaining public pool-state methods:
has_credentials, reset_statuses, remove_index, resolve_target, and
add_entry. All of them read or rebind self._entries (and the mutating
ones persist auth.json), so they now hold the same lock as select()
and the query methods. None are called from within the lock, so no
unlocked helpers are needed.
- Make the blocking test deterministic: an instrumented lock records the
acquire attempt, and the test first waits for the worker to actually
reach self._lock before asserting it blocks. Previously an unlocked
method could pass if the worker thread was scheduled late.
- Extend the lock test matrix to all nine public methods; the five newly
locked ones fail the test without this fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: resolve current entry unlocked in try_refresh_matching no-hint branch
Follow-up to the #62614 salvage: try_refresh_matching (added by the
#69843 salvage after this PR's base) calls self.current() while already
holding the now-locking non-reentrant pool lock — a guaranteed deadlock
that git merges silently (no textual conflict). Use _current_unlocked()
and cover the method in the no-deadlock test.
* fix(windows): hidden-console daemons — extend the parent-console fix to every detached spawn path (#70205)
Extends the desktop backend's root-cause fix (aa2ae36c3f) to all remaining
console-less parent launch paths. The Windows console-flash class
(#54220/#56747) is governed by the PARENT's console: a DETACHED_PROCESS or
pythonw.exe daemon has no console, so every console-subsystem descendant
(git, gh, cmd, node, wmic, powershell) allocates its own visible conhost —
one flash per spawn, unreachable by any per-call-site CREATE_NO_WINDOW
sweep. Worse, MSDN specifies CREATE_NO_WINDOW is IGNORED when combined
with DETACHED_PROCESS, so the hide bit in the old detach bundle was dead.
Changes:
- _subprocess_compat: drop DETACHED_PROCESS from windows_detach_flags()
and windows_detach_flags_without_breakaway(); the daemon now owns a
single hidden console (CREATE_NO_WINDOW) that all descendants inherit.
- gateway_windows: _resolve_detached_python() returns the venv console
python.exe (no pythonw/base-interpreter detour — the uv-shim flash
premise only held while DETACHED_PROCESS was masking the hide bit);
UAC handoff launches console python under SW_HIDE; cmd/vbs launchers
render console python (vbs runs it window-style 0).
- gateway/run.py: restart watcher keeps sys.executable instead of
swapping in GUI-subsystem pythonw.
- web_server: dashboard actions spawn sys.executable (already carries
windows_detach_flags()).
Tests updated to pin the new invariants, including an explicit
DETACHED_PROCESS-must-stay-out regression guard.
* fix(auxiliary): treat explicit model:auto sentinel, not just cfg_model
'auto' is a sentinel meaning "inherit from main runtime / auto-detect",
not a literal model id -- already handled for cfg_model (config-derived)
in _resolve_task_provider_model, but not for the explicit `model` kwarg.
MoA reference/aggregator slots (agent/moa_loop.py's _slot_runtime) forward
a preset's `model:` field as this explicit argument rather than through
auxiliary.<task> config, so a MoA preset configured with `model: auto`
(a natural thing to try given the existing auxiliary.*.model: auto
convention) reached this function as the explicit `model` arg and took
the `model or cfg_model` branch, bypassing the cfg_model-only sentinel
check entirely -- sending the literal string "auto" to the wire as a
model id.
Normalize both the explicit `model` and `cfg_model` the same way, fixing
this at the single chokepoint every caller (MoA included) already goes
through, rather than patching moa_loop.py separately.
* fix(moa): pass custom extra_body to slots
* test(moa): cover the one-shot /moa aggregator path for slot extra_body
Follow-up to #60168's salvage: aggregate_moa_context() is the third
independent MoA call path; assert its aggregator call receives the
custom-provider request_overrides.extra_body via **agg_runtime.
* fix(moa): preserve custom provider context metadata
Preserve compatible custom provider metadata through MoA aggregator context resolution and cover the resolver and compressor paths.
* fix(gateway): honor explicit api_server enabled:false under env key
_apply_env_overrides() force-set ``api_server.enabled = True`` whenever
API_SERVER_KEY (or API_SERVER_ENABLED) was present in the environment.
In multiplex mode, a secondary profile pins
``platforms.api_server.enabled: false`` in its config.yaml so that it
shares the default profile's API-server listener instead of binding its
own port. That profile still inherits the process-level env, including
API_SERVER_KEY, so the unconditional re-enable flipped api_server back on
and tripped the MultiplexConfigError check.
Honor an explicit disable, flagged by ``_enabled_explicit`` in the
platform's extra. Use ``extra.pop("_enabled_explicit", False)``: the
api_server branch is terminal (unlike the migrated plugin platforms, no
later registry pass re-enables api_server), so popping consumes the flag
in a single read and avoids the double-read hazard, while the final
per-platform cleanup remains a no-op.
Adds a regression test asserting that with API_SERVER_KEY set, a config
with api_server explicitly enabled:false + _enabled_explicit:true survives
_apply_env_overrides() as enabled=False (fails without the fix), while the
key is still wired through for the shared listener.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(auth): count MoA preset slots as explicit provider configuration
A user who configured a provider only inside a MoA preset (advisor or
aggregator slot) has explicitly opted into that provider — the consent
gate (is_provider_explicitly_configured) now scans moa.reference_models,
moa.aggregator, and all moa.presets.* slots, so Claude Code OAuth pool
seeding and the auxiliary auto-fallback chain treat MoA-only Anthropic
users consistently with model.provider users.
Salvaged from PR #57778 (trimmed): the auxiliary_client fallback half of
the original PR was independently landed on main in ddd3a2d247 and is
dropped here; a secret-scrubber artifact in the gate test fixture is
restored to the real placeholder token.
* test(gateway): loosen hygiene-timeout wall-clock bound to flake policy minimum (#70202)
0.15s missed by 1-8ms on busy CI shards twice today (runs 30025889952,
30026770278 — branches not touching this file). The assertion pins
'handler did not block on the timeout path' (blocking = seconds); 2.0s
keeps the contract per the loose-bounds flake policy.
* docs(slack): document bot message handling
* docs(messaging): document the '!' prefix for Slack thread commands
Salvaged from #45765 by @navahc09 — kept the PR's callout structure
and placement, rewrote the content to match current behavior:
Slack blocks native slash commands in threads and never delivers them,
so Hermes recognises a leading '!' as an alternate command prefix.
Post-C3 command fixes the bang form also works behind a mention
(@Hermes !cmd) and with leading whitespace; unknown '!' tokens pass
through to the agent unchanged. Cross-linked the detailed
slack.md section.
* docs(slack): gap pass — mention-gating decision table, allow_bots deep dive, clarify buttons, ephemeral slash replies, cron/DM targeting
Documents user-facing wave-1+2 Slack behavior that had no docs coverage:
- decision table for require_mention / free_response_channels /
require_mention_channels / thread_require_mention / strict_mention /
ignore_other_user_mentions and how they compose
- 'Accepting messages from other bots' section with the post-#69483
semantics: allow_bots=mentions requires a CURRENT mention from
peer bots (text or Block Kit blocks); thread state never admits them
- clarify one-tap buttons (choice buttons + Other free-text mode,
in-place resolution, double-click guard, expiry message)
- slash replies are ephemeral: replace-ack, chunking, 5-post cap with
explicit truncation notice, postEphemeral fallback, never-public rule
- cron deliver targeting (slack -> home channel, slack:C... channel,
slack:U... resolved to DM) incl. standalone sender + MEDIA uploads
- send_message media + bare-user-ID DM resolution and caption behavior
Refs #26184.
* chore: map shubhambc09@gmail.com -> navahc09
haran2001's commit uses the numeric GitHub noreply
(56040092+haran2001@users.noreply.github.com) — no mapping file needed.
* fix(gateway): bridge top-level port/host into extra for webhook and api_server
WebhookAdapter and ApiServerAdapter read port/host from config.extra, but
PlatformConfig.from_dict only populates extra from the 'extra:' sub-key in
the YAML platform section. Top-level keys like port and host are silently
ignored, causing the adapter to fall back to DEFAULT_PORT (8644).
This causes silent port conflicts in multi-profile setups: a profile that
configures 'platforms.webhook.port: 8649' still binds 8644, colliding with
the default profile's webhook on the same port.
Fix: extend the shared-key bridging loop in load_gateway_config() to bridge
top-level port/host/secret into extra for WEBHOOK, MSGRAPH_WEBHOOK, and
API_SERVER platforms, following the same pattern already used for
dm_policy, allow_from, gateway_restart_notification, and other keys.
The extra dict takes precedence: if port is already under 'extra:', the
top-level value does not clobber it.
* test(gateway): cover msgraph_webhook port/host/secret bridging + contributor mapping for #57320
Follow-up to the salvaged PR #57320 commit: the PR bridged port/host/secret
for MSGRAPH_WEBHOOK but only tested webhook and api_server. Adds a test
covering the msgraph_webhook branch including extra-precedence, plus the
contributors/emails mapping for kjames2001.
* fix(gateway): make Slack platform note capability-aware when slack tools present
Ports #63234 forward onto current main per teknium1's review.
gateway/session.py hard-coded the stale-API disclaimer for every Slack
session regardless of whether Slack tools were actually loaded. This
contradicted the system prompt when MCP or native slack tools were
present, causing the agent to refuse Slack API actions it could
actually perform (issue #6536).
Per review, the original predicate only checked the native 'slack'
toolset, missing Slack MCP servers (registered under mcp-<server> in
tools/mcp_tool.py) entirely. _slack_tools_loaded() now checks two
independent paths:
1. Native 'slack' toolset + SLACK_BOT_TOKEN (as before, but now calls
_get_platform_tools() with include_default_mcp_servers=True instead
of False, so a default-enabled MCP server also counts).
2. A connected MCP server that has ACTUALLY registered tools into the
live registry (new tools.mcp_tool.get_registered_mcp_server_names()),
whose name suggests Slack. This is session-scoped in the sense that
matters here: MCP servers connect once per gateway process (not
per-session), so checking the live per-server tool-registration map
is the correct availability-filtered signal -- unlike the earlier
get_all_tool_names() approach this replaces, which conflated ALL
built-in tool names process-wide, this only inspects the small,
purpose-built MCP server-name map.
Added a real regression test that registers a tool via the actual
tools.mcp_tool._track_mcp_tool_server() tracking function (not a mock
of the capability check) to verify a genuine Slack MCP server is
detected, plus a negative case for an unrelated MCP server.
5/5 Slack-specific tests pass; 126/126 in the full
tests/gateway/test_session.py file.
* fix(gateway): key Slack capability gate into the prompt pin; defer positive note to tool schemas
Follow-up to the #68627 cherry-pick (cluster C15 — Slack platform
capability-note accuracy; earliest report/fix: #6545 by @daikeren):
1. Session/prompt stability: the pinned session-context render
(_pinned_session_context_prompt) is keyed by _ephemeral_change_key,
whose contract requires every rendered input to appear in the key.
The new _slack_tools_loaded() gate reads config + the live MCP
registration map, so its state is now hashed into the key exactly
like the existing Discord gate — a gate flip re-renders ONCE (a
legitimate bust); within a session the note stays byte-stable for
the life of the conversation (A/B: the new parity test fails with
this key change reverted, passes with it).
2. Derived, non-overpromising positive note: rather than hardcoding a
capability list that can drift stale again (the original bug class),
the tools-present note tells the agent to consult the actual loaded
Slack tool schemas for supported operations — the schemas ARE the
source of truth, so the note cannot overclaim ops a given Slack
toolset/MCP server doesn't expose (e.g. a read-only history server).
3. Tests: parity test proving a gate flip changes both render and key;
byte-stability test proving three consecutive turns in one Slack
session return the identical pinned object (sha256-equal); autouse
fixture pins the new gate so key<->render parity is env-independent.
* fix(slack): throttle channel directory warnings
* fix(slack): surface bot-event arrival and allow_bots interop diagnostics (#30091)
* fix(gateway): quiet Slack missing_scope channel directory fallback
Treat Slack users.conversations missing_scope as an expected limited-scope app condition and fall back to session history without recurring warnings.
Add tests for not-ok and SlackApiError-like missing_scope responses.
* fix(slack): add catch-all event handler to prevent Slack auto-disabling Event Subscriptions
Without a catch-all handler, slack-bolt returns HTTP 404 for every
unhandled bot event (user_change, user_huddle_changed, reaction_added,
etc.) and never sends the Socket Mode ack. On active Slack workspaces
where the app is subscribed to high-volume events, this produces a
near-100% un-acked failure rate that crosses Slack's >95%/60-min
threshold and triggers automatic disabling of the app's Event
Subscriptions — silently killing all inbound event delivery.
Place a catch-all re.compile(r".*") handler AFTER the specific event
handlers so bolt's router matches those first. Truly unhandled events
are silently acked (200) and logged at DEBUG. The failure rate stays
near 0% regardless of which events the Slack app manifest subscribes to.
Fixes #6572
* fix(slack): avoid logging block text previews
* fix(slack): quiet Slack display defaults — no heartbeat/busy-ack breadcrumbs in channels
Slack posts are durable workspace messages, not an ephemeral terminal
status area. Default long_running_notifications and busy_ack_detail to
off for Slack so long-running agent work does not leave permanent
operational breadcrumbs like 'Working — 9 min — iteration 12/90' in
channels. Both remain opt-in per platform via
display.platforms.slack.*.
Also covers the platform-generic shutdown-notification mute path with a
regression test (gateway_restart_notification=false must suppress both
the active-session interruption notice and the home-channel copy).
Salvaged from #69028 (quiet-defaults half only). The PR's other half —
the channel_session_scope_channels session-scoping feature — is a new
config feature outside this log-noise cluster and overlaps the session
scoping territory reworked by merged wave-1/2 Slack session work; it is
deliberately not taken here.
* test(slack): pin catch-all event matcher registration and non-shadowing
Regression test for the catch-all ack: a re.compile(r'.*') event matcher
must be registered (after every named handler, so it never shadows
message/app_mention/reaction/file routing) and must match unhandled
subscribed event types like member_joined_channel / channel_archive /
pin_added.
Salvaged from #64218 (test half only — its adapter-side catch-all is a
duplicate of #38847, which landed as the base commit of this cluster
with first-submitter credit). Fixes #6572.
Co-authored-by: shivasymbl <sdevinarayanan@asymbl.com>
* test(slack): behavioral log-noise/privacy suite + keep clarify choice text out of INFO logs
Follow-up hardening for the C13 log-noise cluster:
- plugins/platforms/slack/adapter.py: clarify button resolution logged
the full chosen option text at INFO (choice=%r). Choice text is user
content — log the choice INDEX at INFO and the (truncated, %.100r)
text at DEBUG only. Widens #58478's principle: no message content
above DEBUG level anywhere in the adapter.
- tests/gateway/test_slack_log_noise.py (new): behavioral suite pinning
the cluster's invariants:
* catch-all ack registered AFTER every named handler (registration
order is bolt's dispatch priority — no shadowing);
* catch-all fires for an unsubscribed event type (reaction_added),
logs only a DEBUG line naming the type, and never logs content;
* named handlers still dispatch (message → _handle_slack_message);
* end-to-end inbound message run leaves NO message text or block
content in any adapter log record (caplog at DEBUG);
* #30185's event-arrival diagnostic is metadata-only;
* clarify resolution: INFO carries index/user only, text is
DEBUG-only (fails with the adapter fix reverted — A/B verified).
Content-leak audit of all 139 logger call sites in the adapter found
two above-DEBUG leaks: the clarify choice line (fixed here) and none
else carrying message text; remaining sites log error strings, URLs
via safe_url_for_log, ids, and counts. The block-extraction DEBUG
preview was already removed by #58478 (chars= length only).
* chore(contributors): add email mappings for slack C13 log-noise salvage
- sdevinarayanan@asymbl.com -> shivasymbl (#38847)
- mycodeisbad@gmail.com -> peterw (#69028; commit author name 'wpeterr'
— PR opened by the peterw account, mapping follows the PR author)
LeonSGP43, ooiuuii, ygd58 mappings already present; nanckh and
haran2001 author via GitHub noreply addresses (no mapping needed).
* test(slack): mark block-privacy fixture message as human-authored (client_msg_id)
#58478's caplog test predates main's unlabeled-bot users.info probe
(#69xxx wave-2 gating): events without client_msg_id now hit
_resolve_user_is_bot, which the fixture's mock client doesn't wire up
(AttributeError on _user_is_bot_cache). Real human-authored Slack
messages carry client_msg_id — add it to the fixture so the test
exercises the intended block-extraction path.
* fix(slack): surface thread images/files as markers in fetched thread context
Images and files posted in a Slack thread before the bot joins were
invisible to the agent: _fetch_thread_context renders text only, and a
caption-less image post was dropped from context entirely (empty text →
skip). "@bot what do you think of the chart above?" read as a question
about nothing.
_render_message_text now appends a compact, sanitized marker per file
attachment — [image: chart.png], [video: demo.mp4], [audio: note.m4a],
[file: report.pdf (application/pdf)] — so the agent can SEE that prior
thread messages carried attachments and ask for a re-share when it needs
the bytes. Filenames are stripped of newlines/brackets so a hostile name
can't fake context structure. Because both thread-context formatting and
parent-text rendering go through _render_message_text, markers appear on
the cold-start hydrate, the explicit-mention delta refresh, restart
rehydration, and reply_to_text.
Reapplied from #32315 onto the current adapter (original patched the
pre-plugin gateway/platforms/slack.py, moved in the plugin migration;
annotation labels reworked to per-file typed markers, download side
handled separately).
* feat(slack): deliver thread-root images on the first mention turn
When the bot is mentioned mid-thread for the first time, the thread root
is very often the artifact the mention is about ("@bot what's in this
chart?" posted as a reply under an image) — but the root's image never
reached the agent, so it answered blind.
On the cold-start hydrate path (and only there), _collect_thread_root_images
reads the root message from the thread-context cache the immediately
preceding _fetch_thread_context call just populated (zero extra Slack API
calls in the normal case), downloads its image/* attachments through the
existing authenticated _download_slack_file helper, and delivers them as
media_urls/media_types on the same MessageEvent — upgrading the message
type to PHOTO so vision routing engages.
Scope and safety:
- One-time delivery by construction: the cold-start path is guarded by
_has_active_session_for_thread, so later turns in the same session can
never re-download or re-deliver. No new gateway/session plumbing needed.
- Bounded by _THREAD_ROOT_IMAGE_MAX (4); non-image root attachments stay
text-only markers.
- Slack Connect stubs (file_access=check_file_info) resolve via files.info.
- Best-effort: a failed download degrades to the [image: ...] marker from
the thread context — never an error turn.
- Also hardens the video mimetype fallback (mimetype can be empty) so
media_types entries are always non-None strings.
Adapted from #69185 by @KCAYAAI — the original plumbed MessageEvent media
through gateway/base.py, run.py and session.py with durable one-time
delivery markers (2,441 lines); this lands the user-visible behavior
adapter-locally by reusing the session guard already on the hydrate path.
* test(slack): regression coverage for thread image/file context visibility
Covers cluster C1-images (#69185, #32315, #66136):
- _slack_file_marker unit tests: typed markers per mimetype family,
hostile-filename sanitization (newlines/brackets can't fake context
structure).
- _render_message_text appends markers; a caption-less image post no
longer vanishes from thread context.
- Cold-start hydrate integration: prior-message images surface as
markers in channel_context; the thread root's image is downloaded,
delivered as media_urls, and upgrades message_type to PHOTO.
- Failure path: root-image download failure degrades to the marker,
never blocks the turn.
- Bounds: root delivery capped at _THREAD_ROOT_IMAGE_MAX; non-image
root attachments stay marker-only (no download).
- One-time delivery: active thread session skips the hydrate → no
re-download/re-delivery on later turns.
- Composition: the trigger's own event files still ride alongside a
delivered root image; Slack Connect stubs resolve via files.info;
the collector never issues its own conversations.replies call.
- Delta refresh (#23918 path): images in new replies past the watermark
surface as markers, with no root re-download.
A/B: 14 of 15 tests fail with the adapter fix reverted, all pass with
it applied.
* chore: map yemi@lagosinternationalmarket.com -> yemi-lagosinternationalmarket
Contributor-email mapping for the #32315 salvage (thread image/file
markers reapplied onto the plugin adapter).
* fix(slack): edit status bubbles in place instead of posting new ones
Progress/status callbacks (context-pressure, compression retries,
model fallback) route through _send_or_update_status_coro, which
edits the previous bubble for the same status_key when the adapter
implements send_or_update_status — but only Telegram did. On Slack
every status event posted a fresh thread message, so a compression
retry loop spammed a dozen out-of-order bubbles into the thread
('Context too large 1/3... 2/3... 3/3', fallback switches, etc.).
Implement send_or_update_status on the Slack adapter following the
Telegram pattern (#30045): first call posts and caches the message ts
per (channel, thread, status_key); subsequent calls edit that message
via chat.update. Edit failure drops the cached ts and falls back to a
fresh send. Cache is FIFO-bounded.
* fix(slack): preserve progress edits on network failures
* fix(slack): delete stale progress messages
* fix(slack): avoid assistant status on synthetic top-level threads
When reply_in_thread=false, top-level channel events carry their own
message ts as metadata.thread_id for session keying. Calling
assistant.threads.setStatus on that ts activated a Slack assistant
thread ('is thinking...') before the actual response was sent, and the
flat reply then never cleared it.
send_typing now routes through the same _resolve_thread_ts synthetic-
thread guard as message sending, and the gateway threads message_id
through progress/status metadata so the adapter can distinguish real
threads from synthetic top-level session keys.
Reapplied from #18859-sibling PR #17184 by @dorukardahan (both commits:
fix + progress-metadata test) onto current main via 3-way apply — the
original patched gateway/platforms/slack.py, moved to
plugins/platforms/slack/adapter.py in the plugin migration.
* fix(gateway): respect reply_in_thread=false for Slack progress messages
The Slack adapter honours platforms.slack.extra.reply_in_thread=false
in _resolve_thread_ts, but the Gateway's progress-message path forced
event_message_id as the thread_id for Slack regardless. The first
progress message ('terminal: …', 'Processing…') created a thread that
all subsequent edits and the final answer inherited, defeating the
user's reply_in_thread=false setting.
Check the live Slack adapter's reply_in_thread flag before applying the
event_message_id fallback, and treat a synthetic source.thread_id (==
the event's own message ts, used only for session keying) as 'no
thread' so progress messages stay at the channel/DM top level.
Folds both #18859 commits (reply_in_thread gate + synthetic thread_id
drop) into main's extracted _resolve_progress_thread_id helper — the
original patched the pre-refactor inline block; the gate now composes
as a keyword argument so Mattermost/oth…
teknium1
added a commit
that referenced
this pull request
Jul 24, 2026
When two consecutive compactions each failed to clear the threshold, the anti-thrashing breaker blocked automatic compaction PERMANENTLY for the life of the session: nothing decremented _ineffective_compression_count (or _fallback_compression_streak) while blocked, so a session whose middle region was briefly too small to compact never auto-compacted again — it grew unbounded until the provider's hard context limit, and only /new or /reset recovered it. Recovery is a probation probe, not amnesty: after _ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants exactly ONE attempt by dropping tripped counters to 1 strike (persisted, so sibling agents on the same session row — gateway hygiene — unblock too). An ineffective probe re-trips the guard on the next real-usage verdict and the next recovery waits a full fresh window, so the worst case in a truly incompressible session is one compaction attempt per window — bounded, not thrash. The recovery clock is armed lazily on the first BLOCKED evaluation and is deliberately not durable: a restart that loads a durable tripped counter (#69872) starts a full fresh window blocked, preserving the restart-must-never-disarm contract (#54923). Fixes #14694
teknium1
added a commit
that referenced
this pull request
Jul 24, 2026
When two consecutive compactions each failed to clear the threshold, the anti-thrashing breaker blocked automatic compaction PERMANENTLY for the life of the session: nothing decremented _ineffective_compression_count (or _fallback_compression_streak) while blocked, so a session whose middle region was briefly too small to compact never auto-compacted again — it grew unbounded until the provider's hard context limit, and only /new or /reset recovered it. Recovery is a probation probe, not amnesty: after _ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants exactly ONE attempt by dropping tripped counters to 1 strike (persisted, so sibling agents on the same session row — gateway hygiene — unblock too). An ineffective probe re-trips the guard on the next real-usage verdict and the next recovery waits a full fresh window, so the worst case in a truly incompressible session is one compaction attempt per window — bounded, not thrash. The recovery clock is armed lazily on the first BLOCKED evaluation and is deliberately not durable: a restart that loads a durable tripped counter (#69872) starts a full fresh window blocked, preserving the restart-must-never-disarm contract (#54923). Fixes #14694
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…ousResearch#69872) The anti-thrash guard (_ineffective_compression_count) was in-memory only: a fresh compressor bound to a resumed, already-compacted session started with compression_count=0 and a disarmed guard, so a near-threshold session could legally re-compact once per process restart, forever. Persist the counter through the durable session-state channel, mirroring the failure-cooldown (NousResearch#54465) and fallback-streak (87d4da0) pattern: - hermes_state.py: sessions.compression_ineffective_count column (declarative reconciliation adds it on existing DBs) + get/set_compression_ineffective_count accessors. - context_compressor.py: every strike/clear verdict routes through _record_ineffective_compression_verdict() which writes through to the session row (no-change verdicts skip the DB write); bind_session_state() loads the persisted value; the compression rotation boundary carries the counter onto the child row; update_model()'s reset also clears the durable copy; the ineffective-only fast path in _automatic_compression_blocked() is removed because the counter is now durable and another agent's clear must unblock a stale local snapshot. - conversation_compression.py: _refresh_persisted_compression_guards re-reads the counter alongside cooldown + fallback streak. Reset semantics are unchanged: any real provider reading below the threshold still clears the counter — and now clears it durably too. Resolves the residual gap identified in NousResearch#54923 by @lanyusea (the second-threshold mechanism was superseded by persisting the existing guard state). Co-authored-by: lanyusea <lanyusea@gmail.com>
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
When two consecutive compactions each failed to clear the threshold, the anti-thrashing breaker blocked automatic compaction PERMANENTLY for the life of the session: nothing decremented _ineffective_compression_count (or _fallback_compression_streak) while blocked, so a session whose middle region was briefly too small to compact never auto-compacted again — it grew unbounded until the provider's hard context limit, and only /new or /reset recovered it. Recovery is a probation probe, not amnesty: after _ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants exactly ONE attempt by dropping tripped counters to 1 strike (persisted, so sibling agents on the same session row — gateway hygiene — unblock too). An ineffective probe re-trips the guard on the next real-usage verdict and the next recovery waits a full fresh window, so the worst case in a truly incompressible session is one compaction attempt per window — bounded, not thrash. The recovery clock is armed lazily on the first BLOCKED evaluation and is deliberately not durable: a restart that loads a durable tripped counter (NousResearch#69872) starts a full fresh window blocked, preserving the restart-must-never-disarm contract (NousResearch#54923). Fixes NousResearch#14694
19 tasks
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
…ousResearch#69872) The anti-thrash guard (_ineffective_compression_count) was in-memory only: a fresh compressor bound to a resumed, already-compacted session started with compression_count=0 and a disarmed guard, so a near-threshold session could legally re-compact once per process restart, forever. Persist the counter through the durable session-state channel, mirroring the failure-cooldown (NousResearch#54465) and fallback-streak (c27ce1c) pattern: - hermes_state.py: sessions.compression_ineffective_count column (declarative reconciliation adds it on existing DBs) + get/set_compression_ineffective_count accessors. - context_compressor.py: every strike/clear verdict routes through _record_ineffective_compression_verdict() which writes through to the session row (no-change verdicts skip the DB write); bind_session_state() loads the persisted value; the compression rotation boundary carries the counter onto the child row; update_model()'s reset also clears the durable copy; the ineffective-only fast path in _automatic_compression_blocked() is removed because the counter is now durable and another agent's clear must unblock a stale local snapshot. - conversation_compression.py: _refresh_persisted_compression_guards re-reads the counter alongside cooldown + fallback streak. Reset semantics are unchanged: any real provider reading below the threshold still clears the counter — and now clears it durably too. Resolves the residual gap identified in NousResearch#54923 by @lanyusea (the second-threshold mechanism was superseded by persisting the existing guard state). Co-authored-by: lanyusea <lanyusea@gmail.com>
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
When two consecutive compactions each failed to clear the threshold, the anti-thrashing breaker blocked automatic compaction PERMANENTLY for the life of the session: nothing decremented _ineffective_compression_count (or _fallback_compression_streak) while blocked, so a session whose middle region was briefly too small to compact never auto-compacted again — it grew unbounded until the provider's hard context limit, and only /new or /reset recovered it. Recovery is a probation probe, not amnesty: after _ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants exactly ONE attempt by dropping tripped counters to 1 strike (persisted, so sibling agents on the same session row — gateway hygiene — unblock too). An ineffective probe re-trips the guard on the next real-usage verdict and the next recovery waits a full fresh window, so the worst case in a truly incompressible session is one compaction attempt per window — bounded, not thrash. The recovery clock is armed lazily on the first BLOCKED evaluation and is deliberately not durable: a restart that loads a durable tripped counter (NousResearch#69872) starts a full fresh window blocked, preserving the restart-must-never-disarm contract (NousResearch#54923). Fixes NousResearch#14694
teknium1
added a commit
that referenced
this pull request
Sep 2, 2026
…y agent rebuilds cannot block a session forever The #14694 recovery clock (`_anti_thrash_recovery_deadline`) was a process-local `time.monotonic()` value zeroed in `bind_session_state()`. The gateway rebuilds the AIAgent (and its ContextCompressor) on every cache eviction, so each fresh compressor bound to a durably tripped session row (#69872) re-armed a full 300s window and the half-open probe never fired — a long messaging conversation above the threshold stayed blocked permanently. Persist the deadline as a wall-clock epoch in a new `sessions.compression_recovery_deadline REAL` column (declarative column reconciliation; SCHEMA_VERSION 26 -> 27) with `SessionDB.get/set_compression_recovery_deadline`. The compressor loads it in `bind_session_state()` and writes it on change only via `_set_anti_thrash_recovery_deadline()`. A fresh compressor with no stored deadline still starts a full window blocked (#54923 restart contract); one that loads an armed deadline resumes that window. Backward clock jumps are bounded to one window. The 300s window is unchanged. Minimal salvage of #100185 (the probe-lease/fencing state machine and model_config-blob storage were not carried). Refs #100185 Co-authored-by: Komzpa <me@komzpa.net>
bottlerex
added a commit
to bottlerex/hermes-agent
that referenced
this pull request
Sep 3, 2026
* test(gateway): skip real-UNIX-socket witness cases on native Windows
All seven TestLoopTickWitness cases that need real UNIX-domain sockets
(socket.AF_UNIX socket nodes or asyncio.start_unix_server producers)
fail on native Windows, where neither primitive exists. Mark exactly
those cases with a shared skipif so a Windows run reports SKIPPED
instead of erroring, while the platform-independent witness-absent
contracts (mocked probes, file-only heartbeats) keep running there.
Split the legacy two-witness-contract test in two: its stale-file arm
is file-only and keeps running on Windows; its dead-listener-node arm
needs a real socket node and is skipped with the rest.
* fix(desktop): guard the whole build-critical dep set, before clean
Refs #86443
assert-root-install.mjs exists to turn an incomplete root install into one
actionable line instead of a failure deep inside the build. It only ever
checked that vite resolved, so an install covering part of the workspace
graph passed the guard and died later on something else. That is the shape
reported in #86443: the updater's npm install brought in 521 of the 769
packages a full install gives, root node_modules had vite but not katex, and
the build failed on an unresolved katex/dist/katex.min.css with nothing
pointing at the install as the cause. apps/desktop/src/styles.css imports
that stylesheet, so katex is as load-bearing for the renderer bundle as vite
is, and electron / electron-builder are the same for packaging.
Check all four and name every missing one, so a partial install is reported
once and completely rather than one package per build attempt.
Resolution walks node_modules upward the way Node's own lookup does, rather
than going through require.resolve: a package whose exports map does not
expose ./package.json is not resolvable by path even when correctly
installed, and that must not read as missing. It also keeps a dependency
that landed in the app workspace instead of the hoisted root passing.
The guard now runs from prebuild, ahead of npm run clean, so a tree that
cannot build is rejected before the build deletes its own outputs. On this
checkout clean removes build/electron-types and the tsbuildinfo files, not
release/, so this ordering is not by itself what saves a packaged app; it is
the narrow correctness point that a doomed build should not destroy anything
first. build keeps its own call for anyone invoking the build steps directly,
and the check is pure filesystem lookups, so running it twice costs nothing.
The check is extracted as a pure checkRootInstall() returning {ok, error},
matching assert-dist-built.mjs, so it is unit testable without spawning a
process.
* chore: export BUILD_CRITICAL_PACKAGES for the test, drop dead default export
Follow-up to the salvaged #87980: the test kept its own copy of the
build-critical package list (drift hazard) and the module's default
export had no consumer.
* fix(desktop): refuse the build when ANY declared non-optional dep is missing
Widen the salvaged guard from a hand-maintained four-package floor to the
class it stands for: every `dependencies` + `devDependencies` entry in the
desktop workspace manifest. Live probe on this box: a tree holding vite,
katex, electron and electron-builder but missing `@rolldown/plugin-babel`
still passed the floor-only guard, and `vite build` died loading
`vite.config.ts` after `prebuild` had already run. The floor stays as an
unconditional fallback for an unreadable manifest; optionalDependencies
are skipped because npm legitimately omits them (get-windows).
Five new vitest cases (12 total); the two class tests fail when the
manifest union is removed. Refs #86443.
* fix(gateway): live foreign token lock at startup exits 78 instead of retry-queueing forever
BasePlatformAdapter._acquire_platform_lock emits `{scope}_lock` with
retryable=True on purpose (#54167): a MID-RUN reconnect must be able to
recover once the live holder exits or a stale record is cleared. The
startup router keyed solely off that flag, so a live foreign holder of the
bot token at zero-connected startup landed in `_failed_platforms` with
gateway_state=running — alive, deaf, and retry-storming the token every
backoff — instead of the exit-78 (EX_CONFIG / startup_failed) contract
that #51228 established for single-writer conflicts.
Minimal class fix, salvaged from #83183 (@alexgunsberg) against current
main:
- gateway/restart.py: `is_global_startup_conflict(error_code)` — matches
the `*_lock` / `lock_conflict` code families every adapter emits for
scoped-lock and identity conflicts. Code only, never message text.
- gateway/run.py primary startup routing: a lock-conflict failure is
routed as non-retryable (parked `fatal`, not queued). Nothing else
connected → exit 78; alongside a transient peer → NS-609 mixed mode,
gateway stays alive and only the peer retries.
- gateway/run.py `_schedule_secondary_profile_startup_reconnect`: the same
contract for multiplex secondaries — park `<profile>:<platform>` fatal
like `duplicate_credential` instead of scheduling a reconnect storm.
- Mid-run behavior is untouched: `_handle_adapter_fatal_error_impl` and
the reconnect watcher still treat `*_lock` as retryable (#54167).
Not carried over from #83183 (superseded on main or out of scope): the
`degraded` lifecycle write only fires on the all-retryable path and the
runner immediately overwrites it with `running` (so busy/drain already
see `running`); the secondary retry bridge landed separately in
96489f3c1b (#92064); Buzz/IRC/LINE lock-tuple unpack and the reconnect
ownership registry are separate class fixes.
Live repro (real GatewayRunner.start(), isolated HERMES_HOME + lock dir,
live holder subprocess owning the lock via production
acquire_scoped_lock): before — exit_code=None, gateway_state=running,
telegram `retrying`, queued in _failed_platforms; after — exit_code=78,
gateway_state=startup_failed, telegram `fatal`, _failed_platforms={}.
Co-authored-by: alexgunsberg <alex@gunsberg.fi>
* fix: hard stop tool loops on non-interactive platforms
* fix(guardrails): preserve interactive platform defaults
* fix(agent): guard repeated skill reads
Treat skill_view and skills_list as idempotent read-only tools so the existing no-progress guardrail can warn or block repeated identical skill loads. This prevents large skill outputs from being re-added to the context in tool loops.
Add regression coverage for repeated skill_view results under hard-stop guardrails.
* fix(guardrails): identical-call streaks hard-stop any tool on unattended platforms
Widen the salvaged #49189 hard-stop default so it covers the loop shape in
the #100849 debug bundle and #89069: a model replaying the same SUCCESSFUL
call (terminal, skill_view, memory) with a byte-identical result. The
per-turn idempotent_no_progress block only tracks IDEMPOTENT_TOOL_NAMES, so
those loops ran until the iteration budget (600 calls, ~40 min) with only a
notice appended.
- agent/tool_guardrails.py: observe_call's tool-agnostic consecutive-identical
streak raises a halt (identical_call_streak_halt) at
hard_stop_after.idempotent_no_progress when hard stops are active. Pollers
stay exempt; a changed result resets the streak; warning-only sessions are
unchanged.
- run_agent.py: surface that halt from _append_guardrail_observation like
every other guardrail halt (appends guidance, ends the turn).
- hermes_cli/config_defaults.py: declare non_interactive_hard_stop_enabled.
- docs: configuration.md describes the streak hard-stop.
- tests: streak halts terminal under hard_stop; never under soft mode,
for pollers, or when results change.
Live A/B (real AIAgent platform=telegram, mocked client replaying one call):
identical failing read_file main: 602 API calls, budget exhausted
branch: 8 calls, repeated_exact_failure_block
identical successful terminal main: 602 API calls, budget exhausted
branch: 5 calls, identical_call_streak_halt
* fix(guardrails): hard stops catch replays, never legitimate iteration
Before turning hard stops on for unattended platforms, make sure they cannot
cut off normal work:
- Edit -> re-run is progress. A successful mutating call (write_file/patch,
a green terminal/execute_code, browser actions, job/message/cron/memory/
skill mutations) marks progress for every failing signature still being
counted this turn; the next identical retry restarts its streak instead
of accumulating toward exact_failure_block_after. A pure replay never
mutates anything between attempts, so it is still blocked at 5.
- Distinct red commands are diagnosis. For FAILURE_TOLERANT_TOOL_NAMES
(terminal, execute_code, process pollers, browser_navigate, web_extract)
same_tool_failure_halt_after warns but never halts.
- subagent and api_server keep the warn-only default: both are supervised
task loops with a live parent/client and do real edit -> re-run work.
Live A/B (real AIAgent platform=telegram, real patch+terminal, 8 rounds of
patch -> red check -> patch ...):
unmitigated branch: HALTED at round 6 (repeated_exact_failure_block)
this commit: COMPLETED all 8 rounds, final answer delivered
Loop shapes still stopped: identical failing read_file 8 calls,
identical successful terminal 5 calls (vs 602 on main).
Six new tests pin these flows; all fail on the unmitigated version.
* fix(gateway): rescue orphaned FIFO overflow when session goes idle (#99882)
When a follow-up is demoted to /queue during compression-in-flight,
it lands in SessionState.conversation.queued_events (overflow) with
the slot event in adapter._pending_messages. After the slot's turn
completes, _promote_queued_event should move the overflow head into
the slot for the recursive drain. When that drain never runs — the
#99882 shape: busy window ended through an exit that skipped the
promotion site — the overflow is silently orphaned: never dispatched,
never persisted, never logged. A 170-char Telegram follow-up vanished
without a trace; its re-send also vanished for the same reason.
Fix: _rescue_orphaned_overflow stages one orphan into the empty slot
on the next idle arrival, and the new message is enqueued behind it
so FIFO order (#28503) holds — oldest orphan runs as this turn, the
rest drain in order, the new message last. The helper is best-effort
(slot occupied or no overflow → no-op) and logs at WARNING when it
fires so a future drain regression is visible.
Tests (tests/gateway/test_fifo_overflow_rescue.py, 4 cases on the real
GatewayRunner FIFO):
- moves overflow head to empty slot
- no-op when slot occupied
- no-op when no overflow
- FIFO preserved: orphan-1, orphan-2, new-msg in exact arrival order
Existing queue suites pass unchanged (test_queue_consumption — 5 passed).
Fixes #99882
* refactor(gateway): drop constant conditional in rescue helper
Review note on #99912: rescued = 1 followed by if rescued: is a constant
conditional — the log block runs unconditionally now that staging is
single-orphan by design.
* fix(gateway): rescued FIFO orphan runs exactly once, chain stays in order (#99882)
Follow-up to the salvaged #99912 rescue. The original helper left the
rescued orphan IN the adapter slot while the caller also swapped it in as
the current turn, so the post-turn _dequeue_pending_event ran the same
follow-up a second time (live repro: TURNS=['Sent','C','C','D']). The
helper now pops the oldest orphan and returns it to run as this turn,
stages the NEXT orphan in the slot so the drain continues the chain in
arrival order, and the call site parks the incoming message behind the
chain via _enqueue_fifo (slot when free, overflow otherwise) instead of
always appending to overflow. The rescued event's own source drives the
turn so reply anchors point at the message actually being answered.
Tests: contract updated for the new return type; added the 2-orphan chain
case and the single-orphan-then-new-message slot case (both fail against
the original helper shape).
* fix(gateway): flush the FIFO overflow tail to disk at shutdown too (#99882)
Sibling site of the same loss class. The #72680 shutdown flush only
serialised the adapter slot (_pending_messages); the FIFO tail parked in
SessionState.conversation.queued_events was discarded with the process,
so every follow-up queued behind the head at restart time vanished the
same way the idle-orphan did. flush_overflow_to_file writes one payload
per overflow event in the slot-flush shape (plus seq for arrival order),
so the existing recover_pending_to_db startup replay inserts them with no
new reader. Wired into _stop_impl beside the slot flush.
* fix(desktop): route approval responses through the runtime event's exact owner
recordSessionEventScope already captures the exact (connectionId, profile) a
runtime's inbound events proved, but knownOwnerForSession never consulted it:
with no tile/hint/row binding for the runtime id, approval.respond failed
owner resolution (SessionOwnerResolutionError) even though the event source
itself named the owner.
Add a structured owner twin of the scope ledger, written and cleared with it,
consumed as the LAST rung of knownOwnerForSession so durable stored identity
still outranks it and untagged/unknown runtimes keep failing closed.
* test(desktop): pin sole-local registry approval routing through the event owner (#96394)
Regression for the single-connection/single-profile report: hasRegistryTopology()
is true on every modern Desktop, so the ambient escape hatch stays closed; the
approval.request event's own (connectionId, profile) stamp is what routes
approval.respond back to the primary socket.
* fix(update): stop crediting unmanaged serve runtimes with a gateway's restart
match_runtime_outcomes() treats any default-profile runtime as covered
once the bare "hermes-gateway" unit restarts, regardless of the
runtime's own kind. An sshd-spawned `serve --isolated` backend (no
systemd unit, supervisor "manual-serve") shares the default profile
and gets silently marked "restarted" even though its own PID was never
touched — so the #91277 Phase 2 unaccounted-runtime tripwire never
fires for it and `hermes update` reports success while it keeps
running pre-update code (#100479).
Restrict the "hermes-gateway" special case to kind == "gateway" so a
serve/dashboard runtime under the same profile falls through to
"unaccounted" instead of borrowing the gateway's outcome.
* fix(update): warn surviving pre-update serve and dashboard runtimes on success (#100479)
* fix(update): reconcile serve/dashboard runtimes in their own vocabulary and escalate survivors (#100479)
Widen the two salvaged fixes (#100490, #100493) to the whole class:
- match_runtime_outcomes: serve/dashboard rows never borrow gateway
bookkeeping at ANY site — not just the bare hermes-gateway unit name
(#100490) but also relaunched_profiles / externally_supervised_profiles
and the profile-substring unit match (hermes-gateway-work credited the
'work' serve). They reconcile against hermes-serve*/hermes-dashboard*
units (exact names, scope prefix tolerated) or, when the caller passes
the (pid, create_time) survivor probe result, by incarnation liveness.
- update_cmd success path: the survivor rows from #100493's new call now
feed the Phase-2 reconciliation, so a surviving unmanaged serve is
'unaccounted' -> exit 1 + 'partial' receipt, not warn-and-exit-0.
- report_unaccounted_runtimes: a serve/dashboard miss names the serve
remedy instead of 'hermes gateway restart', which cannot reach it.
Tests: 6 reconciliation cases (sibling sites, unit vocabulary, exact-name
guard, incarnation probe, remedy text) + an end-to-end cmd_update case
asserting warn + unaccounted + exit 1 + receipt runtime_outcomes.
* chore: map contributor email for salvaged #100493
* fix(update): Windows progress server hands out its URL only once it is serving
`Start-UiServer` printed the -SelfTestUi URL (and opened the browser window)
as soon as the TcpListener was bound, but the runspace that answers /progress
starts asynchronously — BeginInvoke returns before the pipeline is open and
the script block is JIT'd, which is seconds on a loaded runner. The kernel
accepted connections into the backlog during that gap and nobody answered
them. The self-test hit it three times (#90371 and two follow-ups each
widened a timeout instead of removing the race) and it just failed an
unrelated hermes_state.py PR (run 33591547099, two 5s stale-backlog
timeouts = red).
- windows.ps1: readiness handshake after BeginInvoke — one /progress
round-trip must succeed (≤15s) before the server is returned; on failure
tear the listener down and continue without UI. The URL now means
"serving", not "bound". Also fixes the browser opening to a page that never
loads on a slow machine.
- test: 1s per-attempt probe timeout so a single dead backlog socket cannot
consume half the readiness budget.
- CI: new `desktop_updater` classifier lane. tests/test_desktop_update_windows_*.py
spawn the real PowerShell script; the Windows-only job now runs them only
when scripts/desktop-update/**, the Electron updater launcher, conftest,
pyproject, or those tests change (push/dispatch fail open). A PR that
never touched that surface cannot be failed by its process timing.
* fix(cron): surface delivery_failed instead of last_status ok
A successful agent run whose delivery failed used to persist
last_status=ok and bury the failure in last_delivery_error. CLI list
painted that as green and the run looked identical to a quiet success.
Record last_status=delivery_failed instead, keep last_delivery_error,
do not increment failure_streak, and teach cron list/doctor not to
treat it as ok.
Fixes #83993
* fix(cron): stop manual-run notice from asserting delivery that never happened
The _execute_job_now completion notice unconditionally claimed
"(output was delivered there by the job itself)" for non-local
delivery targets, even when the job record's last_delivery_error
showed the delivery failed (#83993). Derive the note from the
refreshed job record so a failed delivery is reported honestly to
the calling agent.
* fix(cron): treat falsy deliver as local in manual-run notice
Review follow-up on the #83993 fix: a stored falsy deliver ("", JSON
null) fell through the local check and produced 'output was delivered
there by the job itself' for a target that does not exist — the exact
false-delivery-claim class the PR removes. Fire time already normalizes
falsy deliver to local (no delivery, output persisted in last_output,
no delivery error), so the summary now canonicalizes with the
scheduler's own _normalize_deliver_value and reads saved-locally.
Whitespace-only deliver is deliberately not folded in: fire time
records 'no delivery target resolved' for it, and the error-driven
FAILED wording must stay visible.
* fix(cron): adapt delivery-notice tests to the return_job claim API
Main grew claim_job_for_fire(job_id, return_job=True) — a claimed
snapshot dict instead of a bool — while this branch sat on an older
base. The merge-ref CI ran the hybrid: the wiring tests still mocked
return_value=True, which fails isinstance(claimed_job, dict) and fell
into the 'already being fired' branch, so every dispatch assert failed.
Mock the claim to return the job snapshot (the API's success shape),
read the summary's deliver from the claimed snapshot the run actually
executes, and keep the dispatch-result failure renderer. Rebased onto
current main; cron suite 710 passed.
* fix(cron): manual run reports delivery_failed as a failed run; docs for the distinct status
A manual cronjob(action='run') derived success from last_status == 'ok'
and read the error from last_error — so a run that now records
delivery_failed came back as success=False with error=None, an unexplained
failure. Surface last_delivery_error as the error in that case (the
#84006 direction, re-applied on the delivery_failed status), and pin the
manual-run completion summary to say 'Result: FAILED' over an undelivered
run. Document the status in the cron user guide.
Co-authored-by: webtecnica <webtecnica@gmail.com>
* fix(cron): every last_status consumer renders delivery_failed explicitly (dashboard badge, Desktop inspector, /cron list, docs)
Audit of every last_status reader outside the scheduler (rg last_status across
web/, apps/desktop/, hermes_cli/, tui_gateway/, tools/, scripts/, website/):
- web dashboard CronPage: last_status was never rendered at all — a
delivery_failed job showed a green 'scheduled' badge and only a small red
'delivery: ...' line. New pure cronLastResult() helper maps the closed
literal set to tones (ok=success, delivery_failed/blocked_config=warning,
error/unknown=destructive) and the card now shows an amber
'delivery_failed' badge (title = last_delivery_error).
- Desktop hermes-bots routine inspector: 'Last result' printed the raw
literal; routineLastResult() spells out each one ('Ran, but delivery
failed', 'Blocked by configuration (not run)', ...), unknown passes through.
- /cron list (cli_commands_mixin): 'Last run: <ts> (delivery_failed)' now
appends the delivery reason, since last_error is None for those runs.
- hermes cron list/doctor and the cronjob tool already handled the literal
on this branch; no consumer compared == 'ok' for success apart from the
cronjob manual-run path, which the branch already fixed.
- developer-guide/cron-internals.md: table of last_status literals + which
detail field carries the reason.
Live repro (real 'hermes dashboard' on a temp HERMES_HOME with a
delivery_failed job, CronPage rendered against the live /api/cron/jobs):
before — badges [scheduled, default, telegram:123]; after — badges
[scheduled, delivery_failed (warning tone, title 'telegram: 502 Bad
Gateway'), default, telegram:123].
* fix(agent): thinking-only length truncations no longer wedge continuations
GLM-5.3-flash on ollama-cloud with reasoning_effort=high can spend the ENTIRE
output cap on reasoning delivered in a separate field and return
finish_reason=length with no visible content (verified live: max_tokens=4096,
completion_tokens=4096, content empty).
The length-continuation path handled that shape badly:
1. the empty response was appended as an interim assistant fragment,
poisoning the transcript until the pre-call sanitizer healed it
(observed 3+ healings per turn on the reporting user's session);
2. every continuation re-ran with thinking ON, re-deriving the whole
thinking budget against a growing context, so 4 attempts still produced
nothing and the turn died with 'Response remains truncated after 4
continuation attempts'.
Now:
- interim assistant fragments with no visible content are never appended
(whichever way they got empty);
- a thinking-only truncation sets a one-shot reasoning-off override that
build_api_kwargs consumes for the next request, so the continuation
writes the answer instead of re-thinking it;
- the ceiling exit clears a pending override and, when every fragment was
empty, returns an actionable final_response instead of an invisible None.
* fix(agent): reasoning-off continuation reaches the wire on the legacy chat path; reset one-shot flag per turn
Follow-up to the #99622 salvage:
- agent/transports/chat_completions.py: the legacy (no provider profile)
chat_completions path always re-emitted extra_body.reasoning with
enabled=True, so both reasoning_effort: none and the one-shot
length-continuation override went out as {enabled: true, effort: none}.
Honor enabled=False / effort=none the way the profile path does.
- agent/conversation_loop.py: reset agent._ephemeral_reasoning_off at
turn start so a flag armed by an interrupted/errored turn can never
strip thinking from the next turn's first request.
- User-facing hints now name the real slash command (/reasoning); the
/thinkon//thinkoff commands do not exist.
- tests: wire-level regression (continuation request carries
reasoning.enabled=false) and a stale-flag turn-scope test.
* test(agent): pin the reasoning-off continuation to exactly one request; document its prompt-cache cost
The one-shot reasoning-off retry changes a request parameter that is part
of the provider cache key on config-sensitive providers (Anthropic renders
thinking/effort into the prompt; OpenAI lists reasoning.effort as
prefix-affecting), so that request is a deliberate single cache miss.
Pin the bound: the request AFTER it must carry the configured reasoning
again and the system prompt must be byte-identical across the whole retry
sequence. Sabotage-verified (sticky flag -> test fails on request 3).
Docstring on _consume_ephemeral_reasoning_off states the cost honestly.
* fix(cron): require positive evidence for live-adapter delivery confirmation
A cron job fired, the scheduler logged "delivered to telegram:<chat> via
live adapter", and nothing reached Telegram (#77763). The log line was not
evidence of a send:
* the silence-narration filter returns {"success": True, "delivered": False}
(a successful *drop*), and the dict-normalization branch read only
"success", so a filtered message counted as delivered;
* an empty payload (no text, no media) skipped the send entirely and still
fell into the "delivered" branch;
* the log line named the chat but not the lane, so a wrong-thread delivery
and a phantom one are indistinguishable after the fact.
_confirm_adapter_delivery now inspects both result shapes: an explicit
`delivered: False` is a rejection even with a truthy `success`, and a
success with no message_id and no raw_response is accepted but logged as
UNVERIFIED. The empty-payload case fails closed into the existing
standalone/warn handling, and the delivered log carries thread= and
message_id=.
Failing closed on the live lane is only half the fix on a native target:
the standalone fallback sent the same empty payload, and the Telegram
adapter returns SendResult(success=True) for empty content without an API
call — a phantom live delivery became a phantom standalone one. Both
_send_to_platform call sites now sit behind one skip guard, so "empty
payload fails closed" holds on every lane (#77763).
* fix(gateway): exempt cron artifacts from the silence-narration drop
The filter guards against bot-to-bot mirror loops of model chatter. Cron
output is an artifact: a job whose brief is legitimately terse ("...", a
single emoji from a script) has no loop partner, and dropping it while
returning {"success": True} is how a cron was logged as delivered with
nothing on the wire (#77763). Cron sends carry job_id in metadata; every
other caller keeps the filter unchanged.
* fix(cron): mark live deliveries as final notifications
* test(cron): pin notify=True on live cron text and media routes
The #58262 assertion lived in test_scheduler.py against a harness that has
since moved; re-home it in the delivery-confirmation suite alongside the
positive-evidence tests, and widen it to the forum-topic route and the media
route so the marker cannot drift out of any lane.
* fix(cron): make cron push-notify configurable (cron.delivery.notify) and surface UNVERIFIED live deliveries in cron list/doctor
De-risking for the notify=True UX change: the marker is now driven by
cron.delivery.notify (config.yaml, default true = current behaviour), read
once per delivery and applied to both the text and media routes; a missing or
malformed section keeps the default.
An evidence-free live-adapter ack (bare SendResult(success=True) from
Slack/Matrix/Mattermost) is still accepted, but the target is recorded on the
job as last_delivery_unverified (cleared by the next evidenced delivery) so
the state shows up in 'hermes cron list' (⚠ Delivery UNVERIFIED), 'hermes cron
doctor', and the cronjob tool listing — not only in a WARNING log line.
Live repro (real _deliver_result + real 'hermes cron list' against a temp
HERMES_HOME, Slack target, SendResult(success=True)): before — list showed
nothing beyond the Deliver line and route metadata always carried
notify=true; after — list prints the UNVERIFIED line, and
cron.delivery.notify: false yields notify=false in the route metadata.
* fix(auth): never fork single-use OAuth grants across profiles (#100339)
Anthropic / Codex / xAI OAuth refresh tokens are single-use: a grant copied
into a second auth.json is one credential with two owners, and the first
profile to refresh it revokes the pair for every sibling (invalid_grant /
refresh_token_reused). Two code paths forked grants that way:
1. `hermes profile create --clone-all` and the dashboard/TUI
`mirror_credentials` flow copied auth.json (+ .anthropic_oauth.json)
verbatim. Both now run `strip_cloned_single_use_oauth_grants()`, which
drops OAuth rows for SINGLE_USE_REFRESH_POOL_PROVIDERS, the matching
`providers.<id>` device-code blocks, and the PKCE singleton file; API
keys are still copied. The clone reads the root grant through the
existing credential-pool root fallback.
2. A named profile with no local rows BORROWS the root grant via
`read_credential_pool()`'s fallback, but every persist
(`CredentialPool._persist`, `load_pool` reseed, `remove_index`) wrote the
rows into the profile's own auth.json — materializing a fork on the first
rotation. `persist_pool_entries()` now routes borrowed single-use rows
back to the root store (update-only, under the root lock; never falls
back to a local copy). A borrowed `hermes_pkce` rotation commits its
singleton to the root `.anthropic_oauth.json`, the borrower never prunes
root-seeded rows it cannot see the backing file for, and
`hermes -p <profile> auth add` persists only the profile's own rows.
Live repro (real imports, temp root + profiles, fake single-use token
endpoint): before — first profile rotation RT0->RT1 in profile only; root
and sibling then hit `invalid_grant`, `resolve_anthropic_token()` -> None.
After — rotation lands in root; root and both siblings select AT1, no reuse.
Direction per Teknium: stop cloning OAuth into profiles (ONE grant at root,
children inherit via context) rather than making clones survive. Supersedes
the clone-strip/root-write-through half of #100389 and the init-refresh idea
in #100703 (an expired-but-refreshable row already refreshes on select()).
Closes #100339
Co-authored-by: HexLab98 <liruixinch@outlook.com>
* fix(auth): auto-heal single-use OAuth grants already forked across profiles (#100339)
The clone-strip and root-write-through in the previous commit stop NEW forks
but leave installs that forked before upgrading in the broken state: each
profile keeps its own copy of the root grant, whichever profile rotated last
holds the only live refresh token, and root plus every sibling still hit
invalid_grant on their next refresh. The PR body asked those users to
re-auth at root and hand-edit profiles/*/auth.json; this makes it automatic.
`heal_forked_single_use_oauth_grants(provider)` (hermes_cli/auth.py) runs at
the top of a profile's `load_pool()` for SINGLE_USE_REFRESH_POOL_PROVIDERS.
Under the profile lock then the root lock it matches each profile OAuth row
to its root counterpart by lineage — same pool id (preserved by both fork
paths), same JWT account identity, same token material, else same provider +
same client (Anthropic pkce grants carry no claims) — keeps the copy with the
freshest rotation (`expires_at_ms` / `last_refresh` / JWT exp), writes it into
ROOT when root's is older, and strips the profile copy (pool rows, the
`providers.<id>` device-code block for Codex/xAI, and a profile-local
`.anthropic_oauth.json`) so the profile borrows root from then on. Root's
singleton and its hermes_pkce row are kept in step so root's own re-seed
cannot resurrect the spent pair.
Guarantees: idempotent (mtime-keyed clean mark skips the locked scan on the
per-call hot path); one INFO line per healed profile; API-key rows untouched;
a row with no root counterpart (root lost its grant, or an independent
account whose claims differ) is never deleted; only the two auth.json files
the root fallback already reads are touched — no environ/secret-scope reads.
`hermes auth list` / `hermes auth status <provider>` print the heal note.
Live repro (real imports, temp root + forge/atlas each holding a pre-fix
verbatim copy, forge already rotated RT0->RT1 into its own file, fake
single-use token endpoint): before — atlas None, forge AT2 (only in forge),
root None; server log 4x REUSE of spent RT0. After — forge's load heals to
root and rotates there, atlas and root select AT2, profiles/*/auth.json hold
no anthropic rows, server log exactly one ROTATE and zero REUSE.
* fix(agent): isolate background review snapshots
* fix(agent): clone the /refine snapshot too, not just the automatic review
Widen #100802 to the two explicit review entry points. The CLI and gateway
/refine handlers built their own snapshot with a shallow list(), which
aliases the nested tool_calls/content containers of the live history. The
review fork sanitizes its transcript in place (sanitize_tool_call_arguments
rewrites function["arguments"]), so a /refine could rewrite the parent's
persisted transcript exactly like the automatic review could (#100795).
Both sites now use _clone_background_review_messages, the same structural
clone the automatic review uses. Regression tests drive the real handlers
and assert the snapshot shares no containers with the live transcript.
* refactor(agent): clone the review snapshot once at the spawn chokepoint
Move the structural clone from the four call sites (auto review, codex
runtime, CLI /refine, gateway /refine) into AIAgent._spawn_background_review,
which every review path — immediate, idle-queue deferred, requeued — passes
through. Callers can no longer forget it, and the private helper is no longer
imported across hermes_cli/ and gateway/ package boundaries.
Tests now bind the real chokepoint (capturing at _spawn_background_review_now)
so they still fail if the clone is removed.
* feat(delegate): tag every subagent progress line with its batch id
Concurrent or nested delegation batches (a parent's 9-way fan-out plus a
child's own 3-way fan-out) printed interleaved `✓ [3/3]` / `✓ [3/9]` lines
with nothing identifying which batch each belongs to.
- CLI: batch header `🔀 [6a66] delegating 9 tasks`; completion lines and
child tree-view lines become `[6a66 3/9]`; spinner remaining-count tagged.
- Relay: `delegation_id` rides on every `subagent.*` event (TUI gateway
payload, api_server SSE subagent.start/complete).
- TUI: `[6a66 3/9]` prefix on /agents rows; Desktop Agents pane groups
workers by exact delegation_id (heuristic shape/time grouping kept for
older backends) and shows the tag on the group header.
- Tag = last 4 hex of the deleg_xxxxxxxx id (format_batch_tag), same id
returned by the dispatch and used for cache/delegation/live/<id>/.
* fix(dashboard-auth): a non-JWT bearer is "not my token", not "provider unreachable" (#94558)
NousDashboardAuthProvider._verify_jwt (and the identical hunk in the
self-hosted OIDC provider) folded EVERY PyJWKClient failure into
ProviderError, which the gate translates to HTTP 503
{"detail":"Auth provider 'nous' unreachable"}. That branch fires for
jwt.DecodeError('Not enough segments') — i.e. the bearer is not a JWT at all
(an opaque peer key, a legacy token, garbage) — and for PyJWKSetError (JWKS
fetched fine, foreign kid). Neither involves reaching Portal, which is why
the hosted sjc agents in #94558 returned a fast, well-formed 503 that
survived token re-mint and instance restart while Portal was healthy.
Add one shared classifier, hermes_cli.dashboard_auth.classify_jwks_lookup_error:
only PyJWKClientConnectionError (transport) and an unexpected bare
PyJWKClientError stay ProviderError; DecodeError / PyJWKSetError /
InvalidTokenError become InvalidCodeError so verify_session() returns None
and the middleware proceeds to the next provider / refresh / 401 exactly as
the protocol documents. Both providers now use it.
Live repro (real NousDashboardAuthProvider against a local reachable JWKS
server; and the real gated web_server app): before — opaque bearer ->
ProviderError "JWKS lookup failed: DecodeError('Not enough segments')" ->
503 unreachable; after — verify_session() -> None, gated GET /api/auth/me
with the opaque bearer -> 401; a real JWT against an unreachable JWKS still
-> ProviderError (503).
This does not add /api/v1/message to the public-path allowlist (#94579):
that route has no verifier in this repo, so bypassing the gate would leave a
state-changing ingress fail-open. The correct fix is classification, which
also covers every other opaque-bearer surface.
Refs #94558
* fix(state): reap stale state-owned sessions safely
* fix(state): report closed stale-open count from auto-maintenance, document the sweep (#54189)
Follow-up on top of the salvaged #94095 commit:
- maybe_auto_prune_and_vacuum() now returns 'closed' (stale open state-owned
sessions marked ended) alongside 'pruned', so entrypoints can report the
reconciliation without parsing logs.
- Docstring explains the two-window lifecycle (close now, delete after a
further retention window).
- Regression test: cron/kanban/subagent rows with ended_at NULL are closed on
pass 1 and deleted on pass 2; a telegram row is never touched.
- website/docs sessions.md documents the automatic stale-open sweep.
* fix(tui-gateway): adopt late compute-host compress acks instead of a false 120s timeout (#97948)
Manual /compress on a compute-host (turn_isolation) session blocked its RPC
waiter for a hard-coded 120s, answered error 5019, and then DROPPED the
host's late `control.ack`: HostSupervisor.control() popped the pending
queue in `finally`, so `_handle_host_frame` had nothing to deliver to. The
host kept compressing, succeeded minutes later, rotated the session — and
the gateway session never mirrored the new session_key/history_version and
the desktop never refreshed its transcript.
- host_supervisor: `control(..., on_late_ack=)` leaves a one-shot handler
registered when the waiter times out; control.ack/control.error/error
frames for that request_id fire it (bounded: 30min TTL, cap 64). A host
crash fails outstanding handlers with a synthetic control.error.
- server: `_compute_host_compress_wait_seconds()` derives the wait from
`compression.context_total_ceiling_seconds` (+30s slack, floor 120s,
cap 630s) instead of the literal 120. `_adopt_late_compute_host_compress_ack`
applies the metadata mirror and emits the same `session.info` a normal
compress does plus the existing `status.update kind=compacted` edge; a
late error goes out through the existing `error` event.
- session.compress / slash.compress (methods_tools + _mirror_slash_side_effects):
on waiter timeout answer `status: pending` (not 5019) and register the
late-ack handler.
- desktop: SESSION_COMPRESS_TIMEOUT_MS 120s -> 660s (above the gateway cap);
`status: 'pending'` renders as an info notice, not `error:`; the
`compacted` status edge rehydrates an idle active session's transcript
(mid-turn compaction still defers to the turn settle path).
Minimal extraction of the design in #99630 by @vsd2807 (design trace by
@andrexibiza and @JoaoMarcos44 in the #97948 thread); no new DB tables,
modules, or polling protocol.
Refs #97948
Co-authored-by: VVV <vaibhavdahiya28@gmail.com>
* fmt(js): `npm run fix` on merge (#101102)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(gateway): honor explicit platforms.<x>.enabled: false over env credentials (#48820)
Twelve credential-presence branches in _apply_env_overrides (weixin,
whatsapp_cloud, homeassistant, email, sms, dingtalk, feishu, wecom,
wecom_callback, bluebubbles, qqbot, yuanbao) force-set enabled = True
unconditionally, so a user's explicit `platforms.<x>.enabled: false` in
config.yaml was silently overridden whenever the platform's token/secret
lived in .env. Telegram/Discord/Slack/Signal/Matrix already routed through
_enable_from_env, which honors the `_enabled_explicit` marker written by
load_gateway_config.
Route all twelve sites through the same helper. Credentials are still wired
into the (disabled) PlatformConfig so send-only tooling keeps working —
the same contract Slack and api_server already follow.
Live repro (real load_gateway_config against a temp HERMES_HOME, yaml
`enabled: false` + creds in env): 12/13 platforms flipped to enabled=True
on main; 0/13 after the fix (telegram control unchanged).
Bug 2 of #48820. Fix direction from @JoaoMarcos44 in #48852 (surgically
reapplied on current main — the June branch no longer applies).
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
* gateway: warn once when an explicit platforms.<x>.enabled: false overrides env credentials
De-risking for the #48820 behaviour change: before this branch, credentials
in the environment force-enabled twelve platforms regardless of an explicit
enabled: false in config.yaml. Now that the explicit disable wins, users who
relied on the old override would see the platform go dark with no trace.
_enable_from_env (and Slack's inline copy) now emit ONE WARNING per platform
per process when the platform is explicitly disabled AND its env credentials
are present, naming the platform, the winning key
(platforms.<x>.enabled: false), the env var(s) being ignored, and the remedy.
A plain disable with no credentials, an enabled platform, and the env-only
(no YAML opinion) path stay silent; repeated config reloads do not repeat it.
_ENV_ENABLE_CREDENTIALS maps every _enable_from_env platform to its
triggering env var(s); a test pins that the map covers every routed branch.
Docs: messaging/index.md gains a 'Disabling a platform whose credentials are
still in .env' section with the exact warning text.
Live repro (real load_gateway_config on a temp HERMES_HOME with
platforms.weixin/telegram.enabled: false + WEIXIN_TOKEN/TELEGRAM_BOT_TOKEN in
env): before — both stayed disabled with zero log output; after — one
WARNING each ('Platform 'weixin' is explicitly disabled by
platforms.weixin.enabled: false ... (WEIXIN_TOKEN, WEIXIN_ACCOUNT_ID) will
NOT start its adapter ...'), none for the enabled homeassistant, none on the
second load.
* fmt(js): `npm run fix` on merge (#101107)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* perf(desktop): group chat rooms answer in the time of one bot, not the sum of all
Bot Mode group rooms were slow by construction: the round engine ran every
member's turn one after another, and each turn found out its bot had finished
by re-reading session.resume on a fixed 2s timer. A 4-bot room paid
4 x (model latency + up to 2s) per round, serially.
- group-rounds: members of a round now take their turns concurrently
(Promise.all). Rounds stay serial so bots still build on each other's
replies. Each member's delta is computed at its own turn start and its
watermark advances only to the pre-turn log length, so sibling replies
that land while it thinks are delivered next round exactly once; a
member's own replies are excluded from its delta by author (they are
already in its session). Message cap enforced per round; stop path
interrupts every member mid-turn (room.turn -> room.turns map).
- group-turns: the poll wakes on the member session's terminal frame
(message.complete / error via host.onEvent), then re-checks at 250ms
until session.running clears. The timer poll stays as a 5s backstop for
hosts without the event tap. Feature-detected; node test harness unaffected.
- group-chat-view: "X is thinking..." lists every member mid-turn.
- docs: bot-mode.md describes concurrent rounds + push-woken replies.
Live A/B (real tui_gateway over WS, 4 members, one round, same model):
serial+2s poll 35.0s -> concurrent+push 8.6s; every turn woke on the event.
Refs #92760
* chore: map contact@danteschrauwen.be -> deinte (PR #101090 salvage)
* fix(cron): don't silently skip a due run after a timezone-offset migration
Upgrading from a UTC-scheduling build to one that honours the profile
timezone (Europe/Brussels) left daily cron jobs sitting in jobs.json with
pre-migration instants — e.g. next_run_at "2026-09-02T04:00:00+00:00" for
expr "0 4 * * *". _ensure_aware normalizes that to 06:00+02, which the
expression excludes, so the stale-expression guard (#93049) read it as a
direct jobs.json edit, logged exactly that, and re-anchored to tomorrow
without firing. The due occurrence disappeared with no error anywhere.
The guard only asked "is the stored instant an occurrence of the current
expr?", never "why not?" — and the two possible answers demand opposite
actions. Add _classify_stale_cron_next_run, which distinguishes them by
whether normalization itself moved the wall clock:
* expr_edit — wall clock unchanged (or the stored wall clock is
not an occurrence either): the instant is genuinely
excluded by the current expression. Re-anchor
without firing, exactly as before.
* timezone_migration — the stored value's own wall clock IS a legal
occurrence and it only left the lattice because
_ensure_aware converted it to a different offset.
Fall through and fire the overdue run once.
Because every value written by this build carries the configured offset, a
real expr edit leaves the wall clock untouched and can never be reclassified
as a migration, so the #93049 protection is intact. At-most-once is
unchanged: the fire flows through the normal due path and the usual
advance_next_run / mark_job_run re-anchor rewrites next_run_at in the
current offset, so the legacy instant is never read again. Future local
wall-clock occurrences are untouched — not-yet-due rows never reach the
guard, and the #28934 offset-repair branch still runs first for a
still-future stored wall clock.
The migration case is classified explicitly rather than retried broadly: it
logs cron.timezone_migration.catch_up with the stored and normalized
instants plus both offsets, and increments a probe-visible counter
(get_timezone_migration_catchup_stats, timezone_migration_catchups.jsonl)
kept separate from catch_up_occurrences so an operator can tell "the upgrade
backlog is draining" from "runs are missing their grace window".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(cron): share the fire-path telemetry recorder
_record_timezone_migration_catchup was a line-for-line clone of
_record_persisted_error_recovery (counter bump, bounded recent list,
best-effort jsonl append). Extract _append_telemetry_record and route
both through it; one shared history cap replaces the two per-counter
constants. Also correct the "distinct from catch_up_occurrences" comment:
a migrated row that is also past its grace window increments both.
No behavior change; both recorders write the same entries to the same
files.
* fix(desktop): group chat rooms are serial again; keep only the push-woken turn poll
#101112 made round members take their turns concurrently. That changed what
a group chat IS: later speakers in a round no longer saw earlier speakers'
replies, so bots answered the user independently instead of building on
each other. Group rooms are serial round-robin by design — this restores the
pre-#101112 round engine (group-rounds.ts, group-chat.ts, group-chat-view.tsx,
their tests, and the docs) byte-for-byte.
What stays from #101112: the per-turn poll wakes on the member session's
terminal frame (message.complete / error via host.onEvent) instead of
sleeping a fixed 2s between session.resume reads; 5s timer kept as backstop.
That is a pure latency fix with no change to room semantics.
Live A/B (real tui_gateway over WS, 4 members, one serial round):
2s poll 32.5s -> push-woken 22.5s. The remaining time is model latency.
Refs #92760
* feat(desktop): status bar can show live cache-hit rate and tokens/sec (off by default)
Two new right-click-toggleable status bar items, mirroring the CLI/TUI
Pantheon status bar upgrades: prompt-cache hit rate ("87%") and rolling
output throughput ("42 t/s"). Both are hidden by default and enabled from
the bar's existing 'Show in status bar' context menu, like the context meter.
Renderer-only: the tui_gateway already emits cache_hit_pct and avg_tps in
every session.usage tick and message.complete payload, so the items ride
the same UsageStats the context meter reads — no new RPC, no polling.
Labels show a placeholder until the backend has data, never self-hide.
* fmt(js): `npm run fix` on merge (#101150)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(desktop): a bot row click always lands on the Bot Chat the row previews
A plain roster click fronted whatever bots-workspace tab the user last had
active for that bot (#96649). A '+' side thread persists in Local Storage
across restarts, so it won every click forever while the row kept previewing
the canonical Bot Chat (profiles.list canonical_session) — sidebar and center
described two different conversations; a message typed there landed in the
side thread and the row never moved. Support thread "[Bots] - Sessions is not
in sync again" (bundle 7dfff039), reproduced live on origin/main.
- roster-actions: the open-tab shortcut may front only the canonical chat
(registry id or lineage tip, via a new onlyStoredIds allowlist on
focusWorkspaceOwnerSessionTile); anything else resolves the registry and
opens in place. Side tabs stay open beside it. "Open Bot Chat" in the row
menu is the same action; the `canonical` option goes away.
- roster-actions: when the FOCUSED Bot Chat's canonical session advances on
the gateway (cron bot-chat delivery, message_agent, group round, CLI turn —
none reach this window's stream), re-open it in place so the transcript
refreshes instead of waiting for an app restart (#99393 class).
Tests: the fronting-shortcut unit file and its e2e spec pinned the reversed
behavior; replaced by one unit file (5 tests) and one e2e spec that fails on
main and passes here. group-to-local-bot-handoff e2e still passes.
* perf(bot-mode): cold DM hops skip the live /models probe; relay replies land within 250ms
Every bot-to-bot DM is a fresh `hermes -p <bot> chat -Q` process, so it
pays agent startup on each hop. Profiling one hop showed the single
largest controllable cost was a live GET /models against the provider on
EVERY launch (0.3-0.6s normally, up to the 15s probe timeout on a slow
endpoint) — the in-memory endpoint-metadata cache is per process and the
Nous persistent context cache is bypassed by design so the portal stays
authoritative.
- model_metadata: memoize successful remote /models probes on disk
(cache/endpoint_model_metadata.json) with the SAME 300s TTL as the
in-memory cache, so authority semantics are unchanged (reconciliation
still lands within 5 minutes) but the answer is shared across
processes. Local endpoints are never memoized (LM Studio reloads).
- bot_relay: the cross-machine reply waiter polls the reply file every
250ms instead of every 2s — up to 2s of dead air on every relayed reply.
Nothing here changes turn ordering: DMs and group rounds stay serial.
Live (polis-hermes bot, spawn -> first API request, cold, 5-6 runs):
main median 1.23s (one 20.8s outlier = probe stall) -> 0.96s, no stalls.
* fix(state): single-flight shared database opens
* test(state): reset the single-flight _opening map in the registry fixture
The _clean_registry fixture clears _generations and _retired between
tests; the new _opening map needs the same reset so a test that aborts
mid-construction cannot leave a stale opening event that stalls the
next test's cold acquire.
* fix(agent): preserve busy steer during compression and avoid replaying historical user request
Compression with display.busy_input_mode: steer embeds the follow-up
as an out-of-band marker inside the latest role=tool result. The
post-compression user-turn preservation path only classified
non-scaffolding role=user rows as real intent, so a compressed
transcript that contained no role=user row would discard the steer
and clone an older historical role=user message as the new active
turn, re-activating a previously consumed request.
Fix _ensure_compressed_has_user_turn to (1) treat a compressed
transcript that already carries a steer marker as having user intent,
and (2) prioritize the latest steer payload from the original
transcript over historical user cloning, inserting it as a proper
role=user turn via _insert_real_user_anchor. This preserves the
actual current intent exactly once and never turns history into new
input.
Closes #100053
* fix(compression): anchor on the LAST intent row — newer user turn outranks older steer (#100053 follow-up)
Follow-up to the salvaged #100114 commit. Its two-pass anchor selection
scanned steers first and real user rows second, so a transcript shaped
[user A, tool(steer B), ..., user C] anchored the already-consumed steer B
over the newer real request C — the same replay class the PR set out to
fix. Replace it with one reversed positional scan that picks whichever
intent-bearing row is last (real role=user or steer-bearing role=tool),
and make the compressed-transcript steer check count only role=tool rows
(the only place the runtime delivers a steer), so a summary quoting the
marker cannot masquerade as live intent.
Adds S1/S2/S3 regression tests (steer dropped by compaction, steer
surviving in tail, newer user turn after steer) plus alternation and
use-exactly-once assertions.
* feat(gateway): one gateway.trust_env key controls aiohttp proxy-env honoring at every adapter site (#48820 bug 3)
Every gateway/plugin platform adapter hard-coded aiohttp.ClientSession(trust_env=True)
(~20 sites), so a gateway launched by a Windows Scheduled Task that inherits a stale
HTTP_PROXY (Clash/V2Ray on 127.0.0.1:7890) looped on 'Cannot connect to host' with no
way to opt out short of NO_PROXY hacks per vendor host.
- gateway/platforms/base.py: gateway_trust_env() reads gateway.trust_env (default true);
resolve_proxy_url() skips generic HTTP(S)_PROXY/ALL_PROXY + macOS system-proxy
auto-detect when false (explicit per-platform vars still win).
- All aiohttp ClientSession sites in weixin, qqbot, matrix, line, wecom, slack, sms,
teams, google_chat now pass trust_env=gateway_trust_env(); mattermost + homeassistant
bare sessions gain the same kwarg (intent of #70119 / #56229).
- DEFAULT_CONFIG + cli-config.yaml.example + messaging docs.
- tests/gateway/test_gateway_trust_env.py: config flip + no-bare-literal sweep.
Reported-by: @ranlingfeng (#48820), @frontnopipe-cloud (#76309)
Co-authored-by: rcarrata <rcarratalasanchez@gmail.com>
Co-authored-by: Backroads4Me <TEDLANHAM@GMAIL.COM>
* fix(compression): persist the anti-thrash recovery deadline so gateway agent rebuilds cannot block a session forever
The #14694 recovery clock (`_anti_thrash_recovery_deadline`) was a
process-local `time.monotonic()` value zeroed in `bind_session_state()`.
The gateway rebuilds the AIAgent (and its ContextCompressor) on every
cache eviction, so each fresh compressor bound to a durably tripped
session row (#69872) re-armed a full 300s window and the half-open probe
never fired — a long messaging conversation above the threshold stayed
blocked permanently.
Persist the deadline as a wall-clock epoch in a new
`sessions.compression_recovery_deadline REAL` column (declarative column
reconciliation; SCHEMA_VERSION 26 -> 27) with
`SessionDB.get/set_compression_recovery_deadline`. The compressor loads it
in `bind_session_state()` and writes it on change only via
`_set_anti_thrash_recovery_deadline()`. A fresh compressor with no stored
deadline still starts a full window blocked (#54923 restart contract); one
that loads an armed deadline resumes that window. Backward clock jumps are
bounded to one window. The 300s window is unchanged.
Minimal salvage of #100185 (the probe-lease/fencing state machine and
model_config-blob storage were not carried).
Refs #100185
Co-authored-by: Komzpa <me@komzpa.net>
* fix(state): fail fast on non-contention flock errors and retry deferred FTS rebuilds in-process (salvage #100130)
Two pieces of PR #100130 (@HexLab98) re-applied on top of the orphaned-flock
break (894fc35337) and fail-closed admission (#100895) that landed since:
* `is_advisory_lock_contention` (hermes_state_common): only EAGAIN /
EWOULDBLOCK / EACCES / EDEADLK mean "another process holds the lock".
ESTALE / ENOTSUP / ENOLCK / EIO from flock or msvcrt.locking are
environment failures that polling cannot fix — `_acquire_db_flock` and
both Windows msvcrt loops (FTS rebuild admission, state.db repair lock)
now defer immediately with the real errno instead of burning the full
120s / holder timeout and then logging a fake "held by another process".
* `retry_deferred_fts_recovery` (hermes_state_schema): a SessionDB whose
open-time `_recover_stale_fts` deferred (foreign holders or busy rebuild
lock) stayed `_fts_stale` — LIKE-only search — until the process
reopened state.db. Short-lived CLIs reopen every run; the gateway opens
once and stays up for days, so the deferral was effectively permanent
(#100108). The retry runs from the EXISTING gateway housekeeping tick
(`_start_gateway_housekeeping`, 60s) against the shared SessionDB
instances via `hermes_state_registry.live_shared_session_dbs()`:
non-blocking admission (`fts_rebuild_admission(timeout_seconds=0)`),
bounded backoff 60s -> 1h, no new thread, still fails closed on live
holders. `fts_rebuild_admission` gains the `timeout_seconds` kwarg.
* WAL-reset warning names `sys.executable` so a "linked SQLite 3.45.1"
line can be matched to the interpreter that actually linked it
(#100108 point 3).
Deliberately NOT carried from #100130: the "leftover lock file = holder"
premise (a 0-byte lock file never blocked flock; the real cause was the
fork-inherited fd, fixed in 894fc35337) and the `_rebuild_fts_once`
one-shot rework.
Co-authored-by: HexLab98 <liruixinch@outlook.com>
* test(state): cover deferred FTS retry, leftover lock files, and WAL interpreter identity
* test(state): non-contention errno table, repair-lock sibling, in-process deferred-FTS retry via housekeeping tick
Regression coverage for the #100130 salvage, all against real SessionDB
files and a real child process holding the flock:
* errno table for `is_advisory_lock_contention` (EAGAIN/EWOULDBLOCK/EACCES
contend; ESTALE/ENOTSUP/ENOLCK/EIO fail fast); no misleading "held by
another process" line on the fast-fail path; `_cross_process_repair_lock`
shares the filter (sibling site).
* `retry_deferred_fts_recovery`: open under a live holder -> stale; retry
returns in <2s with a 30s admission budget (timeout=0); rate limit +
60s->120s backoff engaged; holder dies -> same instance recovers, triggers
restored, breadcrumb cleared; no-op when not stale / read-only.
* `_start_gateway_housekeeping` tick (real loop, 50ms interval) recovers a
stale shared-registry SessionDB with no direct call and no extra thread.
Backoff floor: a monkeypatched 0s base interval must not zero the doubled
interval (min 1s), so the cap math is testable.
Sabotage run (source at origin/main, these tests): 16 failed / 35 passed,
including 30s timeouts on the fast-fail tests.
* chore: map contributor email for leocamilo@me.com
* fix(state): quarantine SessionDB handle after structural corruption
A bare SQLITE_CORRUPT/NOTADB on a live write (not FTS-scoped, not a
replaced file) now sets a sticky per-instance flag: later writes fail
fast with StateDbCorruptError, the handle never reopens after close(),
and close() skips its explicit PASSIVE WAL checkpoint. Gateway and agent
flush paths divert pending transcripts to JSONL/spool like the replaced
case instead of retrying forever.
Field evidence: a handle that kept writing for ~50 minutes after the
first structural error checkpointed 15 pages under the wrong page
numbers on shutdown (page 1 <- messages_fts_trigram_data leaf), turning
"malformed" into "file is not a database".
Refs #90837, #90950, #97940, #89332, #45383
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNX8rNYHqA5pT4tAGSzXtb
* fix(state): also disable SQLite's internal close-time checkpoint on quarantine (py3.12+)
Skipping the explicit PRAGMA wal_checkpoint(PASSIVE) in close() left
sqlite3.Connection.close() running SQLite's own last-connection PASSIVE
checkpoint, which still checkpoints the WAL and unlinks -wal/-shm on a
structurally corrupt file (E2E: the -wal vanished on close despite the
quarantine). Python 3.12+ exposes SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE via
Connection.setconfig(); arm it in _halt_db_corrupt so the WAL image
survives close() for forensics/recovery. On 3.11 the switch does not
exist; the docstring and docs now say so instead of claiming sqlite3
cannot reach it at all.
Follow-up to #101095; flagged by JoaoMarcos44 on #101093.
* fix(desktop): keep drafts editable while connecting
* fix(profiles): profile delete refuses to kill another profile's gateway (#89315)
`hermes profile delete` read the target profile's gateway.pid raw and
SIGTERMed it. When that pid file was poisoned by a sibling profile's gateway
(the #89315 shape), deleting profile A killed profile B's running gateway.
- gateway/status.py: `_pid_record_belongs_to_profile()` helper — a pid
record whose recorded home differs from the expected profile home is not
ours; legacy records without a home prove nothing and are left alone.
- hermes_cli/profiles.py: `_stop_gateway_process` refuses (and says so)
when the record belongs to another profile; still stops its own gateway.
The stop/restart paths in hermes_cli/gateway.py did not need a guard:
`get_running_pid()` already filters cross-profile records and unlinks the
poisoned pid file before any kill can happen — verified live; the test for
that path now pins the real contract (returns False, other process alive,
poisoned pid file gone).
Live repro (unpatched main): `_stop_gateway_process(tim_home)` -> "Gateway
stopped (PID ...)" and the OTHER profile's process exits -15. After: "Refusing
to stop PID ..." and the process stays alive. 8 tests; sabotage (guard
removed) fails 1.
* fix(kanban): judge unachievable goals as blocked, never done
* chore: map contributor email
* fix(loops): pause /loop --until on a blocked verdict; trim redundant gate condition and duplicate test
The goal judge now returns 'blocked' for unachievable goals, but the
/loop --until gate only checked == 'done', so an impossible stop
condition would re-fire every tick until loops.max_ticks. Pause the
loop with the judge's reason instead. Also collapse the kanban gate
callers' 'gate_verdict == "continue" or rejection is not None' to
'rejection is not None' (rejection is None iff verdict == done), drop
the duplicate blocked-verdict goal test, and document the verdict.
* test(kanban): pin that stale blocked-task notify subs are purged
Adapted from #101103: a task parked in blocked past the retention window
must have its notify subscriptions reaped like a stale done task.
* fix(kanban): reap notify subscriptions for stale blocked tasks too
purge_stale_done_notify_subs only matched status='done', so a task the
circuit breaker parked in 'blocked' kept its notify-sub rows forever on
boards that never archive. Widen the predicate to done OR blocked while
keeping the existing age clause; backlog/ready cards are idle, not
abandoned, and stay exempt (test_gc_spares_reopened_task_even_when_old).
Watcher comment/log and docs updated to say done/blocked.
Closes #100955
Co-authored-by: itsflownium <itsflownium@users.noreply.github.com>
* fix(providers): give alibaba-coding-plan-cn its own API key env var
ALIBABA_CODING_PLAN_CN_API_KEY is checked first for the China Coding Plan
endpoint (mirroring kimi-coding-cn), so the intl and CN rows no longer
light off the same key. Fixes #101122.
* fix(providers): hide phantom -cn picker rows lit only by shared intl keys; give alibaba-token-plan-cn its own key var
- alibaba-coding-plan-cn / alibaba-token-plan-cn keep the shared intl key vars
as ordered fallbacks after their dedicated *_CN_API_KEY, so users who set
ALIBABA_CODING_PLAN_API_KEY / ALIBABA_TOKEN_PLAN_API_KEY for the CN endpoint
keep working (the PR as filed dropped them).
- list_authenticated_providers hides a '-cn' row whose only lit key vars are
ones it shares with its non-CN sibling, unless that CN provider is the
configured model.provider. With only the shared key: one row, not two;
DASHSCOPE_API_KEY alone: 3 alibaba rows, not 4.
- Docs: environment-variables.md, providers.md.
* chore(contributors): map umit.ediz@hotmail.com -> Edizzier
* feat(email): configurable IMAP/SMTP transport security (tls/starttls/plain) and TLS verify toggle
Adds EMAIL_IMAP_SECURITY / EMAIL_SMTP_SECURITY and EMAIL_IMAP_TLS_VERIFY …
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…ousResearch#69872) The anti-thrash guard (_ineffective_compression_count) was in-memory only: a fresh compressor bound to a resumed, already-compacted session started with compression_count=0 and a disarmed guard, so a near-threshold session could legally re-compact once per process restart, forever. Persist the counter through the durable session-state channel, mirroring the failure-cooldown (NousResearch#54465) and fallback-streak (af7dcea) pattern: - hermes_state.py: sessions.compression_ineffective_count column (declarative reconciliation adds it on existing DBs) + get/set_compression_ineffective_count accessors. - context_compressor.py: every strike/clear verdict routes through _record_ineffective_compression_verdict() which writes through to the session row (no-change verdicts skip the DB write); bind_session_state() loads the persisted value; the compression rotation boundary carries the counter onto the child row; update_model()'s reset also clears the durable copy; the ineffective-only fast path in _automatic_compression_blocked() is removed because the counter is now durable and another agent's clear must unblock a stale local snapshot. - conversation_compression.py: _refresh_persisted_compression_guards re-reads the counter alongside cooldown + fallback streak. Reset semantics are unchanged: any real provider reading below the threshold still clears the counter — and now clears it durably too. Resolves the residual gap identified in NousResearch#54923 by @lanyusea (the second-threshold mechanism was superseded by persisting the existing guard state). Co-authored-by: lanyusea <lanyusea@gmail.com>
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
When two consecutive compactions each failed to clear the threshold, the anti-thrashing breaker blocked automatic compaction PERMANENTLY for the life of the session: nothing decremented _ineffective_compression_count (or _fallback_compression_streak) while blocked, so a session whose middle region was briefly too small to compact never auto-compacted again — it grew unbounded until the provider's hard context limit, and only /new or /reset recovered it. Recovery is a probation probe, not amnesty: after _ANTI_THRASH_RECOVERY_SECONDS (300s) of continuous block the gate grants exactly ONE attempt by dropping tripped counters to 1 strike (persisted, so sibling agents on the same session row — gateway hygiene — unblock too). An ineffective probe re-trips the guard on the next real-usage verdict and the next recovery waits a full fresh window, so the worst case in a truly incompressible session is one compaction attempt per window — bounded, not thrash. The recovery clock is armed lazily on the first BLOCKED evaluation and is deliberately not durable: a restart that loads a durable tripped counter (NousResearch#69872) starts a full fresh window blocked, preserving the restart-must-never-disarm contract (NousResearch#54923). Fixes NousResearch#14694
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…y agent rebuilds cannot block a session forever The NousResearch#14694 recovery clock (`_anti_thrash_recovery_deadline`) was a process-local `time.monotonic()` value zeroed in `bind_session_state()`. The gateway rebuilds the AIAgent (and its ContextCompressor) on every cache eviction, so each fresh compressor bound to a durably tripped session row (NousResearch#69872) re-armed a full 300s window and the half-open probe never fired — a long messaging conversation above the threshold stayed blocked permanently. Persist the deadline as a wall-clock epoch in a new `sessions.compression_recovery_deadline REAL` column (declarative column reconciliation; SCHEMA_VERSION 26 -> 27) with `SessionDB.get/set_compression_recovery_deadline`. The compressor loads it in `bind_session_state()` and writes it on change only via `_set_anti_thrash_recovery_deadline()`. A fresh compressor with no stored deadline still starts a full window blocked (NousResearch#54923 restart contract); one that loads an armed deadline resumes that window. Backward clock jumps are bounded to one window. The 300s window is unchanged. Minimal salvage of NousResearch#100185 (the probe-lease/fencing state machine and model_config-blob storage were not carried). Refs NousResearch#100185 Co-authored-by: Komzpa <me@komzpa.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A tripped or armed anti-thrash guard now survives process restarts:
_ineffective_compression_countis persisted through the durable session-state channel, so a fresh compressor bound to a resumed, already-compacted session inherits the guard instead of re-compacting a near-threshold session once per restart, forever. Root cause: the counter was in-memory only while every other compression breaker (failure cooldown #54465, fallback streak af7dcea) already round-tripsstate.db.Changes
hermes_state.py: newsessions.compression_ineffective_countcolumn (declarative reconciliation adds it to existing DBs on startup) +get/set_compression_ineffective_countaccessors mirroring the fallback-streak pair.agent/context_compressor.py: all five strike/clear verdict sites route through a new_record_ineffective_compression_verdict()write-through helper (no-change verdicts skip the DB write, so ordinary fitting responses stay free);bind_session_state()loads the persisted value; the compression rotation boundary carries the counter onto the child row;update_model()'s reset clears the durable copy; the "ineffective-only block skips DB refresh" fast path in_automatic_compression_blocked()is removed — the counter is durable now, so another agent's clear must unblock a stale local snapshot.agent/conversation_compression.py:_refresh_persisted_compression_guards()re-reads the counter alongside cooldown + fallback streak.tests/agent/test_compression_anti_thrash_persistence.py(restart inheritance, per-session isolation, durable reset semantics, rotation carry, persist-failure resilience); updated pins intest_compression_rotation_state.py/test_idle_compaction_lock_and_guards.py(the old "in-memory-only, skip refresh" contract inverted); round-trip pin intests/test_hermes_state.py.Validation
should_compress()stays blockedTargeted tests:
scripts/run_tests.sh tests/ -q -k 'ineffective or anti_thrash or rotation_state'→ 51 passed; direct suites (anti-thrash persistence, rotation state, idle guards, session-end clears, host contract, infinite-compaction loop, hermes_state) → 456 passed, 0 failed on the rebased tip.Credit
Resolves the residual gap identified in #54923 by @lanyusea (the second-threshold mechanism was superseded by persisting the existing guard state).
Co-authored-by: lanyusea lanyusea@gmail.com
Infographic