fix(discord): stop "is typing…" orphaned by typing-loop recreate race - #49924
Closed
Kyzcreig wants to merge 164 commits into
Closed
fix(discord): stop "is typing…" orphaned by typing-loop recreate race#49924Kyzcreig wants to merge 164 commits into
Kyzcreig wants to merge 164 commits into
Conversation
Some OpenAI-compatible auxiliary providers return a 200 OK with a
control-plane error string (e.g. "There is an issue with the selected
model, run --model to pick a different model") rather than raising an
HTTP error. Before this fix, ContextCompressor would accept that text as
the conversation summary, store it as _previous_summary, and drop the
real middle turns. The result was silent context corruption: the agent
saw a "summary" that was actually a model-selection error message, and
had no way to recover because no exception was raised and
_last_summary_error stayed None.
The most common trigger today is the auxiliary.<task>.model: auto
resolver bug being fixed in a sibling PR (literal "auto" goes over the
wire, bridge politely refuses, compressor swallows the refusal as the
summary). But the failure mode is more general — content-policy refusals,
rate-limit refusals worded as text, and any "polite 200" path from a
misconfigured aggregator can produce the same corruption.
Fix
Add a conservative substring match against the summary content before
storing it. If it matches a known provider-error / refusal pattern,
treat it like a transient failure on the summary model: log clearly,
set _last_summary_error / _last_aux_model_failure_error so downstream
consumers can surface a warning, and either retry on the main model
(when a distinct summary_model is configured) or enter the same
short cooldown the existing transient-error branch uses.
The substring set is deliberately small and bridge / aggregator
control-plane shaped — it would not match a real conversation summary
that happened to discuss "models" or "selection". Pure detection is
tested with positive and negative cases; integration is tested both
with and without a distinct summary model configured.
Verification (script at /tmp/verify-compressor-error-guard.py and
/tmp/verify-compressor-fallback.py — not part of the diff):
- Positive cases (5 known provider error / refusal strings) all match.
- Negative cases (real summary mentioning "model", empty string,
arbitrary summary, None, int) all reject.
- Integration: compressor patched to return a provider error string
correctly sets _last_summary_error, populates _last_aux_model_
failure_error with a preview, and produces output messages free of
the error string content.
- Fallback retry: with summary_model_override set to a broken model,
the guard fires, _fallback_to_main_for_compression is invoked, the
second call goes to main without the override, and the resulting
summary is used.
Two related fixes to delegate_task toolset scoping: 1. Remove code_execution from the default subagent block list. Subagents already inherit `terminal` (a strictly larger capability), so blocking only `execute_code` was asymmetric and prevented legitimate use cases — e.g. "compute SHA256 + sum of primes" would silently fail because the subagent had no execute_code tool despite requesting toolsets=['code_execution']. 2. Stop calling _strip_blocked_tools() on explicit caller-requested toolsets. When a user/parent agent explicitly passes `toolsets=[...]` to delegate_task(), their intent wins over the implicit safety block list. Inherited/default toolsets still get the strip applied. This prevents silent toolset deletion which was the root cause of the debugging session that surfaced this bug. Repro before fix: parent with toolsets=['hermes-cli'] delegating with toolsets=['code_execution'] → intersection preserves code_execution → _strip_blocked_tools deletes it → child gets [] toolsets → falls back to default tool surface, missing execute_code. After fix: child receives ['code_execution'] as requested. Tests updated: - TestStripBlockedTools.test_removes_blocked_toolsets: assert code_execution survives strip. - TestStripBlockedTools.test_code_execution_no_longer_blocked: new regression test pinning the explicit allow. Local patch on main; upstream PR deferred per Ace's call.
When a /stop command (or any other generation invalidation) interrupts
an agent run, the gateway took the early-return path at
_message_handler around line 7541. That path discarded the stale
result but did NOT call stop_typing on the platform adapter, so the
chat_action: typing indicator stayed sticky until the NEXT inbound
message cycled it.
Observed 2026-05-28 in gateway.log:
15:43:30 inbound message (image)
15:43:55 STOP invalidates generation 18
15:43:55 STOP response sent
15:43:56 Discarding stale agent result — generation 18 is no longer current
^-- early return at run.py:7555, no stop_typing
15:44:01 next inbound message (which cycles the indicator)
The happy path at 7533-7537 and the exception path at 7894-7898 both
already clear typing correctly. Adding the same guard to the
stale-result path closes the gap.
Defensive try/except matches the surrounding pattern: typing-clear is
never load-bearing — if it fails, the next message cycles it anyway.
…al model id
When auxiliary.<task>.model is set to "auto" in config.yaml,
_resolve_task_provider_model() was treating it as a truthy model id
and propagating the literal string "auto" to the wire. The provider
then returned a 200 OK with an error-text body (e.g. "the model auto
does not exist, run --model to pick a different model"), which
downstream consumers such as ContextCompressor accept as the
compressed summary -- silent corruption with no exception raised.
The provider-side auto-resolution path (_resolve_auto via main_runtime
fallback) is already wired up and does the right thing when cfg_model
is None. The fix is to normalize the auto sentinel at the resolver
layer: when cfg_model.lower() == "auto", drop it to None so the
resolver can fall through to main_runtime / auto-detect.
Reproduction (pre-fix):
>>> from agent.auxiliary_client import _resolve_task_provider_model
>>> _resolve_task_provider_model("compression") # with model: auto in config
("auto", "auto", None, None, None)
Post-fix:
>>> _resolve_task_provider_model("compression")
("auto", None, None, None, None)
Verified end-to-end: ContextCompressor.compress now produces a real
summary (~4KB of compaction text) instead of swallowing the bridge
error string. Aux compression on auto/auto config no longer silently
corrupts the conversation summary.
…odel on caller-passed model _get_cached_client returns 'model or default_model' as the second tuple element. When the caller passes a truthy model name — which is always, except when caller explicitly passes None — the raw caller input wins, bypassing the namespace stripping that resolve_provider_client just performed via _normalize_resolved_model. For providers in _DOT_TO_HYPHEN_PROVIDERS (bundled: 'anthropic'; plugin-provider plugins extending the set hit this too), namespaced model IDs like 'anthropic/claude-haiku-4-5' leak past normalization and arrive at the wire unstripped. api.anthropic.com 404s with 'model not found: anthropic/...'. The reason this hasn't been broadly noticed: most callers happen to pass already-normalized names. But any caller passing the namespaced form (which _PROVIDER_MODELS advertises for some providers, and which fallback chains routinely produce) silently fails — fatally for Anthropic-wire providers. Fix: re-apply _normalize_resolved_model on the raw caller-passed model before falling back to default_model. Preserves caller-wins semantics — caller's explicit override still wins over the provider's default — but no longer bypasses namespace stripping. The normalizer is idempotent so running it twice (once in the resolver, once here) produces the same output. Companion to NousResearch#24586 (resolver consults ProviderProfile.api_mode). Either can land first; together they fully enable third-party plugin providers that target Anthropic-Messages-API endpoints via call_llm without per-call workarounds. Reproduction (pre-fix): >>> _get_cached_client('anthropic', 'anthropic/claude-haiku-4-5') (<client>, 'anthropic/claude-haiku-4-5') # un-stripped Post-fix: >>> _get_cached_client('anthropic', 'anthropic/claude-haiku-4-5') (<client>, 'claude-haiku-4-5') # correctly stripped
…warning + fix provider_label shadowing Manual re-apply of a02a289 onto v0.15.1 (validate_requested_model moved to ~3343). Improves the Anthropic-Messages /v1/models fallback warning to name the active provider label + resolved endpoint, and fixes provider_label shadowing by renaming the two local rebinds (catalog branch -> provider_label_for_catalog; OAuth branch -> provider_label_oauth) so the module-level provider_label() function stays reachable. Upstream added a SECOND shadowing rebind (openai-codex/grok OAuth branch) not present in the original commit; renamed that one too. Ports regression test tests/hermes_cli/test_anthropic_messages_warning_clarity.py. Original-commit: a02a289
Manual re-apply of c1d5f9e onto v0.15.1 (_resolve_task_provider_model moved to ~4432). When an explicit provider is given with no task-config api_mode override, fall back to the provider profile's declared api_mode so plugin providers whose upstream speaks the Anthropic Messages API are wrapped with the correct transport regardless of base-URL shape. Task config still wins; wrapped in try/except. Adds a fresh regression test (original commit was code-only) pinning profile fallback, task-config-wins, and None passthrough. Original-commit: c1d5f9e
… path
resolve_provider_full() (the /model switch + --provider resolver) consulted
only the models.dev catalog, Hermes overlays, and config.yaml providers/
custom_providers. It never consulted the provider-module plugin registry
(plugins/model-providers/<name>/), even though hermes_cli.auth.PROVIDER_REGISTRY
auto-extends from that same source at import time.
Result: a provider declared only as a plugin profile (e.g. a local Anthropic
proxy registered via providers/) resolves fine during runtime startup — so it
works as the configured default model — but switching INTO it via /model fails
with "Unknown provider '<name>'. Check 'hermes model' ...". Two code paths,
two different provider registries.
Reproduction (pre-fix), with a plugin provider declaring api_mode=anthropic_messages:
from hermes_cli.model_switch import switch_model
r = switch_model(raw_input="my-proxy/some-model",
current_provider="openrouter", current_model="x",
current_base_url="", current_api_key="",
is_global=False, explicit_provider="my-proxy")
assert r.success # -> False, "Unknown provider 'my-proxy'"
Fix: add a resolution step (1b) in resolve_provider_full that consults the
providers/ plugin registry via get_provider_profile(), translating the
ProviderProfile into a ProviderDef. api_mode maps back to transport via the
inverse of TRANSPORT_TO_API_MODE (default openai_chat). This reunifies the
switch path with the startup path, so a provider that works as a default also
works as a /model target.
Scope: IN — make the switch resolver see plugin providers. OUT — refactoring
the two registries into one (larger change, own tradeoffs); this PR keeps both
but makes the switch path layer the plugin registry the same way auth.py does.
Verified: resolve_provider_full + switch_model now succeed for a plugin
provider with correct transport/base_url/api_mode; unknown providers still
return None. Pre-existing 3 failures in test_model_switch_custom_providers.py
(model-catalog 403 in sandbox) are unrelated and fail identically on origin/main.
…ult tolerance A transient local DNS outage (nodename nor servname) makes getUpdates fail repeatedly. The old hardcoded ladder (10 retries x 60s cap = ~7min) escalated to a retryable-fatal error too eagerly. Make MAX_NETWORK_RETRIES / BASE_DELAY / MAX_DELAY config-overridable via telegram.network_retry_* and raise defaults to 20 retries / 120s cap (~25min tolerance). Bridge the YAML keys into config.extra and add coverage.
Free-response channels are explicitly allowed to trigger Hermes without a direct mention. The prior multi-agent guard dropped any message that mentioned another bot but not Hermes, which incorrectly suppressed migration/context messages that merely quoted an old bot mention. This keeps the existing suppression outside free-response channels, and still yields inside free-response channels when the message begins with another bot mention, which is the direct-address case. Verification: - venv/bin/python -m pytest tests/gateway/test_discord_connect.py::test_free_response_channel_allows_inline_other_bot_mention tests/gateway/test_discord_connect.py::test_free_response_thread_inherits_parent_for_inline_other_bot_mention tests/gateway/test_discord_connect.py::test_free_response_channel_yields_when_message_starts_with_other_bot_mention tests/gateway/test_discord_connect.py::test_normal_channel_still_ignores_other_bot_mentions -q -o 'addopts=' - venv/bin/python -m pytest tests/gateway/test_discord_connect.py::test_free_response_inline_other_bot_mention_reaches_platform_event -q -o 'addopts=' - venv/bin/python -m pytest tests/gateway/test_discord*.py -q -o 'addopts='
…tach The /status session recap lists touched file paths via _shortened_path, which emits a bare absolute or ~-relative path for any file outside the gateway cwd. The gateway scans outbound message text for bare local file paths and auto-uploads matches as native attachments (gateway/platforms/base.py _extract_local_media_paths). As a result, /status could silently upload the contents of a touched file (e.g. ~/.hermes/config.yaml) into the chat -- a data-exposure footgun. Wrap each recap path in inline-code backticks so it lands inside an inline-code span, which the detector's _in_code filter explicitly skips. Files touched stays readable; nothing gets uploaded. Adds two regression tests pinning the backtick wrapping and the inline-code-span structural guarantee.
Store the most recent successful provider-call usage in agent.last_turn_usage. The session_* counters remain cumulative; this snapshot preserves the last turn's normalized input/output/cache-read/cache-write/reasoning split for future /context and /usage surfaces without parsing provider-specific payloads later. Reset the snapshot on new sessions so usage does not leak across session boundaries.
…#2) * feat(usage): persist last-turn token snapshot to sessions row The last turn's token split (input/output/cache/reasoning) lived only on the ephemeral AIAgent instance (agent.last_turn_usage) and was lost when the idle sweep evicted the agent between turns. The cumulative session_* counters already persisted; the per-turn snapshot did not. Add 5 nullable last_turn_* columns to the sessions table (declarative migration via _reconcile_columns), write the snapshot in update_token_counts (COALESCE so omitting them preserves the prior snapshot), and add get_last_turn_usage() to read it back. Snapshot survives a fresh SessionDB handle == survives agent eviction. Wire the conversation-loop persistence call to pass the snapshot each call; the per-call overwrite leaves the turn's final call as the snapshot, matching agent.last_turn_usage semantics. * feat(usage): surface persisted last-turn snapshot in /usage between turns When the agent has been evicted by the idle sweep (no running or cached agent), the gateway /usage command previously fell through to only a rough transcript token estimate. Now it reads the persisted last_turn_* snapshot from the sessions row (SessionDB.get_last_turn_usage) and renders the real last-turn token split — input/cache/output/reasoning — so users get accurate last-turn numbers even between turns. Defensive int-coercion guards against malformed snapshot values. Header is inline English (single between-turn diagnostic line) to avoid adding keys to all ~30 locale catalogs; existing label_* keys are reused and stay in i18n parity. No behavior change when no snapshot exists. --------- Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…redaction (#3) The ENV-assignment redaction pattern matches any VAR=value whose name contains a secret-like substring. GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL contain "AUTH" (inside "AUTHOR"), so they were being falsely redacted to VAR=*** mangling git commit-authoring commands in tool output (observed when re-authoring commits for the contributor-attribution CI check). Add a leading word-boundary + negative-lookahead allowlist (_GIT_IDENTITY_ALLOWLIST) that exempts the four git identity vars while still redacting every genuine secret. Verified: real keys (OPENAI_API_KEY, AWS_SECRET_ACCESS_KEY, *_AUTH_TOKEN, *_AUTHORIZATION_KEY) still redact; git identity vars (incl. with 'export ' prefix) pass through untouched. Tests: 3 new allowlist tests + 2 still-redact guards; full redact suite 79 passed, 0 regressions. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
The test asserted the fallback warning contains the provider's friendly label 'Claude API Proxy', but that label only resolves when the claude-api-proxy provider plugin is registered at runtime from $HERMES_HOME/plugins/model-providers/. It is not a built-in canonical provider, so on a clean CI checkout provider_label() falls through to the raw 'claude-api-proxy' slug and the assertion fails — an environment- dependent flake that only passed on machines with the user plugin present. Seed _PROVIDER_LABELS via monkeypatch so the message-building path is exercised deterministically regardless of which providers are registered. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
write_file reported success based only on the write command's exit code plus a wc -c byte count, with no content comparison. A silent persistence failure — the write exits 0 but the bytes on disk differ from what we sent (a third-party editor/sync client clobbering the file right after the write, a truncated stdin pipe, a backend FS oddity) — was reported as a successful write. Mirror the read-back verification patch_replace already performs: re-read the file after writing and compare (line-ending normalized) against the intended content, returning an error if they differ or the re-read fails. Adds TestWriteFilePostWriteVerification (clobber detected, happy path, verify-read-error). Full tests/tools/test_file_operations.py green (80). Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
Adds a capture-mode gate to the mem0 memory provider. Default 'auto' preserves existing behaviour (sync every completed turn). Setting MEM0_CAPTURE=off (or capture: off in mem0.json) keeps recall (prefetch + search) and explicit mem0_conclude writes, but skips per-turn auto-capture. Use case: latency-sensitive / high-traffic agents (voice agents Agora + Clanker) that want shared cross-agent recall without paying a Mem0 write on every turn. Exposed in the config schema for discoverability.
) * blackbox T1: per-turn usage accumulator + on_session_end turn_usage enrichment + subagent attribution + shared TurnRecord contract * blackbox: plugin (store/cost/card/routing/commands/__init__) + diff-review fixes Senior Opus diff-review BLOCK resolved: - B1: /cost crashed — card.render(dict) didn't exist (only render_card(TurnRecord)). Added card.render() dict-or-TurnRecord facade; added real-card integration tests (test_commands_real_card.py) that exercise the path with NO card mock. - B3: log inside on_session_end outer except (silent telemetry failures). - B5: pop _sessions entry on disabled/early-return path (leak guard). - RC6: Decimal(str(amount)) to avoid float fp drift in cost sum. - RC9: atomic sweep — deletes + sentinel in one commit. - RC11: routing prefers run_coroutine_threadsafe onto gateway loop (on_session_end runs in a worker thread); retain task refs to avoid GC. - RC16: seam test pins real post_tool_call kwarg (tool_name), drops masking fallback. 35 blackbox+core tests green; 19 adjacent usage tests green (no regression). * blackbox: re-review refinements (RC2/RC7) — config-aware /cost threshold, status-vocab pin tests Focused re-review (APPROVE WITH CHANGES) follow-ups: - card.render() threshold now reads blackbox.cost_alert_threshold_usd from config (falls back to turn cost) so /cost dig-in Threshold line is meaningful. - RC7: test_cost_status_vocabulary_pinned asserts every status agent.usage_pricing can emit (actual/estimated/included/unknown) is handled by cost._STATUS_RANK; test_cost_actual_maps_to_estimated pins the actual->estimated remap. - Verified reviewer false-positive #1 (latency_s 'missing'): it's a @Property on TurnRecord deriving ts_end-ts_start; real-card test renders it and passes. 37 blackbox+core tests green. * blackbox: capture tool args/result previews into side table The post_tool_call hook now records args/result previews (gated by store_text) alongside tool names, populating the turn_tool_calls side table that /cost <id> dig-in already reads. Closes the last spec gap: the dig-in now shows per-tool args/results, not just names. - _on_post_tool_call captures args/result via _preview (bounded, JSON-coerced) - _build_record threads state['tool_calls'] into TurnRecord.tool_calls - store scrubs+truncates previews before persist (already wired) - 2 real-store seam tests: dig-in round-trip + store_text:false privacy gate * blackbox: fix /cost registration + real-loader E2E + /cost debug CRITICAL FIX: register() only registered hooks, never delegated to commands.register() — so /cost would NOT exist in a live gateway despite every unit test passing (they called handle_cost directly). The new real-loader E2E test (test_loader_e2e.py) drives PluginManager.discover_and_load → invoke_hook → registered command handler and would have caught this. E2E coverage (no mocks): - registration: hooks + /cost wired through the real loader - opt-in gating: not loaded without plugins.enabled - full turn lifecycle: hooks fire → real SQLite persist → /cost renders card+dig-in - disabled gate: hooks are no-ops Debugging capability — /cost debug: - store.debug_stats(): DB path/size, turn/tool/alerted/subagent counts, ts range, last sweep date (read-only, never raises) - _handle_debug: config gate state + resolved channel + store health, so 'why no cards?' is self-diagnosable in-session - plugin.yaml: declare provides_commands: [cost] 48 blackbox+core green; 110 adjacent plugin-loader tests green (no regression). --------- Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
* usage_pricing: notional Anthropic pricing for subscription proxies/bridges The local Claude subscription proxies and tailnet failover bridges (claude-api-proxy, claude-api-proxy-f1, claude-bridge, claude-bridge-f1) were unrecognized by resolve_billing_route → fell through to billing_mode 'unknown' → cost None → status 'unknown'. This suppressed all blackbox /cost spending-alert cards on the fleet's primary stack (every Apollo/Aegis turn runs claude-opus-4-8 through claude-api-proxy). Marginal cash cost is $0 (flat Claude subscription), but for cost VISIBILITY we now price these at official Anthropic API rates ($5/$25 per M, $0.50 cache-read, $6.25 cache-write — the existing claude-opus-4-8 table entry) and label 'estimated'. A real 500k-in/559-out turn now prices at $2.77 and fires a card at the $1 threshold. - NOTIONAL_ANTHROPIC_PROVIDERS frozenset (fleet provider aliases) - resolve_billing_route remaps them to the anthropic billing route (reuses all pricing + dot-notation normalization, bare/ prefixed model names) - openai-codex intentionally unchanged (stays subscription_included/$0 — no authoritative gpt-5.x per-token pricing to use; documented as a follow-up) - 5 targeted tests incl. a codex no-regression guard; 16/16 pricing green * usage_pricing: notional OpenRouter pricing for openai-codex Codex (ChatGPT-subscription) turns previously short-circuited to billing_mode=subscription_included → $0/included, so /cost cards could never fire. The marginal cash cost is $0, but for fleet cost *visibility* we now resolve openai-codex to the underlying OpenAI model and price it from the live OpenRouter catalog (status estimated) — the same dynamic source that already powers provider: openrouter routes. - NOTIONAL_OPENROUTER_PROVIDERS frozenset (openai-codex) - resolve_billing_route: codex → openrouter/official_models_api route - _openrouter_pricing_entry: fall back to base model when a '-codex' variant is absent from the catalog (gpt-5.5-codex → gpt-5.5) - _MODEL_CACHE_TTL 1h → 6h (pricing/context rarely change; cheaper + resilient to OpenRouter blips, per request) - tests: replace 2 obsolete codex=included tests with 5 hermetic monkeypatched tests (route, pricing, -codex fallback, exact-id preference, unknown-stays-unknown) Verified empirically against live catalog + blackbox compute path: gpt-5.5 heavy turn → ~$1.44 estimated, card fires @ $1. --------- Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…#8) The post-write-verification tests stubbed env.execute by matching command.startswith('cat >') to detect the write. Upstream commit 39f6b6e ("make write_file/patch atomic (temp-file + rename)") rewrote write_file to stream content through _atomic_write, which emits a wrapped script (`set -e; ...; cat > "$tmp"; mv -f "$tmp" "$t"`) rather than a bare `cat >`. The old prefix match no longer fired, so the mock never stored the written content and the post-write verify read returned 0 chars -> 'write did not persist' on main (fleet CI red). Fix: discriminate the write by `stdin_data is not None` (the write is the only exec that streams content over stdin; reads/mkdir/wc never do). Backend-agnostic — survives future changes to the write command string. Verified: reverting the mock to the old 'cat >' shape reproduces the exact CI failure locally; the stdin_data discriminator passes. 80/80 in test_file_operations.py green. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…che (#9) * blackbox: fix card chat fields + cache %, add alerts_enabled toggle, 3d price cache Four fixes to the Blackbox turn-telemetry card and pricing cache: 1. Chat fields on the card. The on_session_end emitter (conversation_loop) never passed chat_id/chat_name, so every gateway turn recorded empty chat fields and the card rendered 'Session: Discord <#>'. Pass agent._chat_id/_chat_name through the hook (mirrors platform/provider). 2. Cache % denominator. _cache_line divided cache_read by bare input_tokens, so a cache-heavy turn (12 fresh input, 669.2k cached) reported 669.2k/12 = 5,576,942%. Divide by the TOTAL prompt (input + cache_read + cache_write) → a real 0-100% hit rate. 3. Session line readability. _session_line now shows the channel NAME alongside the Discord mention ('Discord #ops (<#id>)') and degrades gracefully when id or name is missing (never an empty 'Discord <#>'). 4. alerts_enabled config flag (default True). When False, turns are still recorded (visible to /cost and /context) but no alert card is pushed — lets an operator keep telemetry without the channel noise. Plus: _MODEL_CACHE_TTL 6h → 3 days (OpenRouter notional-pricing cache; pricing/context rarely change, cheaper + more blip-resilient). Tests: updated 2 expectations that codified the cache/session bugs; added cache-denominator, discord session-line, alerts-disabled (×2), and chat-field-passthrough tests. 67 blackbox+pricing green, ruff clean. * fix(blackbox): show total prompt tokens in card 'Tokens in', not the uncached remainder The card's 'Tokens: N in' line rendered record.input_tokens raw, but that field is only the FRESH (uncached) input remainder. Under prompt caching nearly all input arrives as cache_read/cache_write, so the bare field reads as a tiny leftover (e.g. 12 / 16) while the real input is hundreds of thousands. Same root cause already fixed the Cached% denominator; this extends the _prompt_total helper to the Tokens-in display so both agree. Update test_hooks_alert expected bullet 500k->999k to match the corrected total (fresh 500k + cache_read 499k). --------- Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…OOT) The reboot/shutdown family is on the unconditional hardline blocklist — the correct default for most agents, but a hard blocker for fleet/ops agents that legitimately need to restart the Linux hosts they manage. Add an opt-in escape hatch: when HERMES_ALLOW_REBOOT is set truthy, the reboot/shutdown family downgrades from HARDLINE to the DANGEROUS layer — still approval-gated, and yolo / approvals.mode=off can pass it through. Every other catastrophic pattern (root recursive delete, filesystem format, raw-device overwrite, fork bomb, kill -1) stays unconditionally hardline. Default (flag unset) is byte-identical to the historical always-block behavior. Verified: 129 tests pass including new downgrade and isolation tests.
…sweep (#10) Two follow-ups to the turn-telemetry plugin: 1. Subagent cost rollup. delegate_tool records each subagent as its own turn stamped with the PARENT's channel (platform/chat_id) and is_subagent=1. They were silently folded into the session total with no visibility. - store.subagent_rollup(platform, chat_id): sums cost/tokens of a channel's subagent turns, counts unpriced (cost_usd IS NULL) separately so the display can show an honest '+N unpriced' instead of undercounting, and lists the models seen. Keyed by channel (not parent_turn_id, which holds the parent SESSION KEY and isn't linkable to a parent turn row). - store.session_rollup now also returns subagent_count/subagent_usd as a subset of the (already subagent-inclusive) total. - /cost session appends: '↳ Subagents: $X across N turn(s)[, +U unpriced][models]' only when the channel has subagent turns. 2. Per-turn retention sweep. _on_session_end calls store.sweep(retention_days) after recording. sweep() is self-throttling (a last_sweep_date sentinel makes it a no-op after the first call each UTC day), so it costs one indexed SELECT/day and keeps the store bounded. Guarded so a sweep failure never blocks recording/alerting. Verified empirically against the live store: a real telegram session renders '↳ Subagents: $0 across 1 turn(s), +1 unpriced [claude-opus-4-8]' (the one unpriced row predates the pricing fix). 73 blackbox+pricing tests green (added: subagent_rollup channel/unpriced/isolation, session split, sweep called with configured days, sweep-failure-doesn't-block, /cost session subagent display), ruff clean. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
* docs: remote desktop connect uses username/password, not --insecure + session token (NousResearch#38926) The documented path for connecting Hermes Desktop to a remote backend was `--insecure` + a pinned HERMES_DASHBOARD_SESSION_TOKEN — an unauthenticated bind plus a copy-pasted token. Replace it everywhere with the bundled username/password dashboard-auth provider: set HERMES_DASHBOARD_BASIC_AUTH_*, run `hermes dashboard --host 0.0.0.0` (the non-loopback bind engages the auth gate), and Sign in from the app. - desktop.md: rewrite 'Connecting to a remote backend' for the user/pass + Sign in flow - web-dashboard.md: rewrite both remote-backend sections (overview + dedicated); reframe the auth-gate section so --insecure is a discouraged escape hatch, not a co-equal use case; drop the removed --tui flag from the systemd example - environment-variables.md: lead with HERMES_DASHBOARD_BASIC_AUTH_*; drop the session-token / HERMES_DESKTOP_REMOTE_TOKEN remote-connect entries - docker.md: mention the username/password provider as the simplest gate provider * fix(cli): clear screen on exit so live chrome isn't stranded in scrollback (NousResearch#38928) The classic CLI left its live bottom chrome — the status bar, input box, and separator rules — frozen in terminal scrollback after exit, on every exit path (/exit, /quit, Ctrl+C, EOF) and on both Linux and Windows. The prior erase_when_done=True fix (bf82a7f) routes prompt_toolkit's teardown through renderer.erase(), but that walks back by the renderer's internal cursor model and does not reliably wipe the chrome in practice — users still saw a dead status bar + the rest of the session sitting above the resume summary. Clear the screen + scrollback directly at the single exit funnel instead. All exit paths converge on _print_exit_summary() (called from the run-loop finally block after app.run() returns and prompt_toolkit has restored terminal modes), so a new _clear_terminal_on_exit() helper runs there before the summary prints. It writes ESC[3J ESC[2J ESC[H (erase scrollback, erase screen, home cursor) on a real TTY, no-ops silently when stdout is not a terminal (pipes/redirects), and falls back to the platform clear command if the escape write fails. Works on Linux, macOS, and modern Windows terminals (Terminal/conhost with VT processing, already enabled by prompt_toolkit). The resume/goodbye summary now prints at a clean top-left with nothing stranded above it. Fixes NousResearch#38252. * feat(approval): env-gated reboot/shutdown downgrade (HERMES_ALLOW_REBOOT) The reboot/shutdown family is on the unconditional hardline blocklist — the correct default for most agents, but a hard blocker for fleet/ops agents that legitimately need to restart the Linux hosts they manage. Add an opt-in escape hatch: when HERMES_ALLOW_REBOOT is set truthy, the reboot/shutdown family downgrades from HARDLINE to the DANGEROUS layer — still approval-gated, and yolo / approvals.mode=off can pass it through. Every other catastrophic pattern (root recursive delete, filesystem format, raw-device overwrite, fork bomb, kill -1) stays unconditionally hardline. Default (flag unset) is byte-identical to the historical always-block behavior. Verified: 129 tests pass including new downgrade and isolation tests. * feat(log): tag Turn-ended diag line with cron task id (task=) The cron scheduler already threads the job id into the agent as task_id (cron/scheduler.py -> run_conversation(task_id=...) -> effective_task_id). Surface it in the per-turn 'Turn ended' diagnostic log as task=<id> so cron turns can be attributed to their job. Enables model-fallback-check.py to detect when a model-pinned cron silently ran on a fallback model, replacing an unreliable time-window heuristic with exact job-id attribution. Additive logging only — no control flow change. Both logger.info and the 'pending tool result' logger.warning branch share _diag_msg/_diag_args, so both gain the field. Format-specifier/arg count kept in lockstep (guarded by test). tests/agent/test_turn_ended_task_id_log.py: 3 contract tests (task= present, specifiers==args, last arg is effective_task_id). --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com> Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com> Co-authored-by: Aegis <aegis@hermes.local>
* docs: remote desktop connect uses username/password, not --insecure + session token (NousResearch#38926) The documented path for connecting Hermes Desktop to a remote backend was `--insecure` + a pinned HERMES_DASHBOARD_SESSION_TOKEN — an unauthenticated bind plus a copy-pasted token. Replace it everywhere with the bundled username/password dashboard-auth provider: set HERMES_DASHBOARD_BASIC_AUTH_*, run `hermes dashboard --host 0.0.0.0` (the non-loopback bind engages the auth gate), and Sign in from the app. - desktop.md: rewrite 'Connecting to a remote backend' for the user/pass + Sign in flow - web-dashboard.md: rewrite both remote-backend sections (overview + dedicated); reframe the auth-gate section so --insecure is a discouraged escape hatch, not a co-equal use case; drop the removed --tui flag from the systemd example - environment-variables.md: lead with HERMES_DASHBOARD_BASIC_AUTH_*; drop the session-token / HERMES_DESKTOP_REMOTE_TOKEN remote-connect entries - docker.md: mention the username/password provider as the simplest gate provider * fix(cli): clear screen on exit so live chrome isn't stranded in scrollback (NousResearch#38928) The classic CLI left its live bottom chrome — the status bar, input box, and separator rules — frozen in terminal scrollback after exit, on every exit path (/exit, /quit, Ctrl+C, EOF) and on both Linux and Windows. The prior erase_when_done=True fix (bf82a7f) routes prompt_toolkit's teardown through renderer.erase(), but that walks back by the renderer's internal cursor model and does not reliably wipe the chrome in practice — users still saw a dead status bar + the rest of the session sitting above the resume summary. Clear the screen + scrollback directly at the single exit funnel instead. All exit paths converge on _print_exit_summary() (called from the run-loop finally block after app.run() returns and prompt_toolkit has restored terminal modes), so a new _clear_terminal_on_exit() helper runs there before the summary prints. It writes ESC[3J ESC[2J ESC[H (erase scrollback, erase screen, home cursor) on a real TTY, no-ops silently when stdout is not a terminal (pipes/redirects), and falls back to the platform clear command if the escape write fails. Works on Linux, macOS, and modern Windows terminals (Terminal/conhost with VT processing, already enabled by prompt_toolkit). The resume/goodbye summary now prints at a clean top-left with nothing stranded above it. Fixes NousResearch#38252. * feat(approval): env-gated reboot/shutdown downgrade (HERMES_ALLOW_REBOOT) The reboot/shutdown family is on the unconditional hardline blocklist — the correct default for most agents, but a hard blocker for fleet/ops agents that legitimately need to restart the Linux hosts they manage. Add an opt-in escape hatch: when HERMES_ALLOW_REBOOT is set truthy, the reboot/shutdown family downgrades from HARDLINE to the DANGEROUS layer — still approval-gated, and yolo / approvals.mode=off can pass it through. Every other catastrophic pattern (root recursive delete, filesystem format, raw-device overwrite, fork bomb, kill -1) stays unconditionally hardline. Default (flag unset) is byte-identical to the historical always-block behavior. Verified: 129 tests pass including new downgrade and isolation tests. * fix(blackbox): background_review fork inherits parent chat context Memory/skill background-review forks constructed AIAgent with platform= but no chat_id/chat_name/chat_type, so their blackbox telemetry rows landed with empty chat fields and were un-attributable in /cost session. Pass through parent's _chat_id/_chat_name/_chat_type (coerced to '') so review-fork turns are attributed to the originating channel. Regression test asserts the fork inherits all three chat fields. --------- Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com> Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
The Codex non-streaming path already has a fast-reconnect no-byte TTFB watchdog, but its default cutoff (120s) was >= the wall-clock stale timeout (default 90s), so the blunt stale detector always won first. A wedged chatgpt.com/backend-api/codex socket (ReadError/Broken pipe, zero stream events) therefore burned the full 90s stale timeout on every retry (90s x 3 ~= 4.5min) before the fallback model kicked in, instead of reconnecting in ~2s. Couple the TTFB cutoff to fire before the stale timer for the Codex backend: target a fast-reconnect cutoff (~40s, tunable via HERMES_CODEX_TTFB_FAST_RECONNECT_SECONDS) clamped strictly below the stale timeout. The large-request (>=25k token) TTFB disable still applies, so legitimate long prefills are untouched. Opt out via HERMES_CODEX_TTFB_BELOW_STALE=0. Adds two regression tests: one pinning that a no-byte hang is killed via the fast-reconnect TTFB path well under the 90s stale timer at the real default config (fails without the fix: waits the full stale timeout), and one verifying the coupling can be disabled via env.
Lets a single cron job raise its API retry budget without changing the global agent.api_max_retries default for every other agent and job. The morning-digest job pins gpt-5.5 on the flaky openai-codex subscription backend; combined with the fast-reconnect Codex TTFB watchdog, a higher per-job retry count means transient backend hangs are retried several more times (~40s each) on the requested model before the fallback chain swaps to another model — keeping the job on Codex in almost all cases. Read from job['api_max_retries'] and applied to agent._api_max_retries after construction; only ever raises the floor (clamped >= 1). Absent the field, the agent keeps its configured default (no behavior change). Tests: override reaches the agent (fails without the scheduler change); absence inherits the agent default (3).
…ro loop ROOT CAUSE of the K=2 rerun infinite loop: the identifier-fidelity instruction (Prong A) was hardcoded into the live summarizer prompt with no off-switch. The baseline-repro arm (--no-escalation, meant to PROVE the merge bug still reproduces) therefore ran the FIXED summarizer -> clean nodes -> CW=0 -> AC-5 abort -> net cron re-fired -> loop, burning ~$40/3h going nowhere. FIX (root cause, not gate-patch): - escalation.py: extract L1/L2 identifier-fidelity blocks behind _identifier_fidelity_enabled(), gated by internal env LCM_IDENTIFIER_FIDELITY. Default (unset) = ON = production behaviour; fail-safe (garbage/empty -> ON). ONLY the baseline-repro arm sets it 0 to reproduce pre-fix summarization on IDENTICAL code (true A/B). - harness _hermes(): exports LCM_IDENTIFIER_FIDELITY=0 when escalation OFF (baseline), =1 when ON (fix arm). Verified E2E that hermes chat -Q summarizes in-process and reads the subprocess env (same path as LCM_CONTEXT_THRESHOLD). - 6 new tests (default-on, off-reproduces-prefix, failsafe-to-on, harness wiring). 159 context_engine tests green.
Two campaign bugs that compounded the loop: - per-stage reset_store(): only S0 reset before; baseline-repro's MERGED nodes then polluted the fix stages (sentinel text collisions in the node-match harness). Now every stage (S0/S2/S3/S4) starts from messages=0/nodes=0. - abort-safe terminal marker: EXIT trap appends /tmp/lcm-k2-campaign.done on ANY exit (success OR deliberate abort), so the 15m net cron treats an AC-5 abort as a FINDING, not a crash to recover from (was the re-fire trigger). The marker is cleared once by the launcher at fire time, never by the campaign (race-free).
On databases created before the messages.content -> search_content rename, the msg_fts_* triggers still referenced the dropped content column. They exist by name, so the name-only _fts_missing_triggers check reported healthy while every message INSERT aborted at runtime with "table messages_fts has no column named content" -- LCM could not persist a single message, and doctor/repair both treated the store as fine. Add body-aware drift detection (_fts_stale_triggers) wired into external_content_fts_needs_repair, and make repair_external_content_fts drop-then-recreate when triggers are missing OR stale (a bare CREATE TRIGGER IF NOT EXISTS no-ops on a stale-but-named trigger). Detection normalizes IF NOT EXISTS / whitespace / terminator before comparing so healthy DBs are never falsely flagged. Adds tests/context_engine/test_lcm_fts_stale_trigger_repair.py: 5 regression tests incl. the name-only-blindness gap, the real insert-abort symptom, full repair-and-persist, a discriminating healthy-DB guard, and the sibling missing-trigger case. Verified: 164 context_engine tests green; teeth-tested (fix disabled -> 2 red, restored -> green).
The messages_fts stale-trigger bug slipped through because every prior LCM test created a FRESH store on the current schema, where triggers are always correct and writes always work. The bug only existed on databases created before the content -> search_content rename and carried forward -- a state a fresh-store test cannot reproduce. Add tests that build the two real on-disk drift shapes seen on the fleet and drive the genuine MessageStore.__init__ startup path, then assert a write persists AND is FTS-searchable: - half-migrated (Apollo): search_content column + FTS present, but msg_fts_* triggers left on the dropped content column. - never-migrated (daedalus): only a content column; FTS + triggers all on content -- must migrate to search_content on open without crashing. Plus fresh-DB baseline and reopen-idempotency. Teeth-proven: with the stale-trigger fix disabled, the 3 legacy-shape tests go RED and the fresh baseline stays green. 168 context_engine tests green.
…back (Phase B) A mid-session fallback (e.g. claude-opus-4-8 1M -> gpt-5.5 272K on the Codex route) collapsed the usable window, but ContextCompressor.update_model re-derived the trigger from the OLD model's stored threshold_percent instead of re-resolving the DESTINATION model's configured compression.per_model_threshold. Result: configured gpt-5.5: 0.9 was never applied after the switch, contributing to the 2026-06-19 compaction-thrash incident (#claude-bridge-fix). - Add resolve_compression_threshold() in auxiliary_client as the single source of truth for the precedence chain (per_model_threshold -> built-in family default -> global), shared by agent_init (init) and update_model (switch). - Thread per_model_threshold + global_threshold + codex autoraise flag into ContextCompressor; update_model now re-resolves for the new model. Init behavior unchanged (threshold_percent still the init-resolved value). - Legacy/direct callers that don't thread the config keep prior behavior. Invariant I3: after any update_model, threshold_tokens reflects the destination model's resolved threshold, not the source model's stored percent on the new window. Tests: tests/agent/test_compaction_threshold_reresolve.py (RED w/o fix).
… (Phase C) The load-bearing fix for the 2026-06-19 compaction-thrash incident. Root cause (proven against the code, correcting the spec's original session-split premise): compress() decided the anti-thrash effectiveness verdict by comparing a REQUEST-level pre value (display_tokens / current_tokens, ~205K incl. tool schemas) against a MESSAGES-ONLY post estimate (estimate_messages_tokens_rough, excludes the ~30K of tool schemas + system prompt). The post side was understated by the schema overhead, so a compaction that did NOT drop the real request below the trigger still read as a >=10% "saving" -> _ineffective_compression_count reset to 0 -> the anti-thrash guard never reached 2 -> the loop re-fired forever (the 205,072 -> 297,723 "tokens went UP" signature). Verified the spec's other premises against the live code and found them already handled, so this fix is deliberately small: - The compressor instance SURVIVES the compaction-driven session split (same cached agent keyed by session_key); the split calls on_session_start( boundary_reason="compression") which the built-in compressor ignores. So the counter already persists across splits and already resets on real /new via on_session_reset(). No agent-anchoring needed (dropped RC#1's added complexity). - The protected tail is ALREADY token-capped (_find_tail_cut_by_tokens, floor capped at _MAX_TAIL_MESSAGE_FLOOR=8, never cuts a tool_call/result pair) since aec3885. No new tail logic needed. The fix: - Add ContextCompressor.record_compaction_effectiveness(pre, post) as the SINGLE owner of the counter for the normal path: ineffective iff the post-compaction REQUEST estimate stayed >= threshold AND shed < 10%. Increments by exactly 1 per call (no double-wire). - compress() no longer mutates the counter from its messages-only verdict (kept only for human-readable logging); the structural no-window increment stays. - The done-site (conversation_compression) calls record_compaction_effectiveness with request-level pre (estimate_request_tokens_rough of the original messages) and post (_compressed_est), so a failed/placeholder summary that leaves the request over threshold is correctly counted as ineffective (closes D-6). Invariant I2: after <=2 consecutive request-level-ineffective passes, should_compress backs off. Tests: tests/agent/test_compaction_antithrash.py (RED without fix). Existing infinite-compaction-loop + compressor suites stay green.
Integration proof on the real estimate_request_tokens_rough: a placeholder/ partial summary that shrinks MESSAGES >=10% but leaves the full REQUEST (tool schemas + system + bulky tail dominate) over threshold with <10% saved is correctly counted ineffective. Proven discriminating (RED when the fix is neutered).
…uped (Phase A) When the primary model fails and a fallback activates SUCCESSFULLY, the user now sees a single status line naming the destination model + provider + the context- window change. Previously the only fallback signal was _buffer_status, which is suppressed on successful recovery (flushed only on a TERMINAL turn failure) — so the 2026-06-19 opus->gpt-5.5 fallback that collapsed the window 1M->272K and triggered the compaction thrash was completely invisible to the user. - Add _emit_fallback_announce() routed through _emit_status, which reaches the gateway status_callback (Discord/Telegram) AND the CLI every time — verified reachable once the gateway wires the callback (gateway/run.py:14683). - Deduped on the (old_model, new_model) pair via agent._last_fallback_announced so a re-entrant fallback chain that bounces to the same destination in one turn announces once (I5). Cleared on primary recovery (restore_primary_runtime) so a later fallback episode re-announces. A no-op transition (old==new) is silent. - Window delta (e.g. "context window 1M→272K") is the load-bearing diagnostic fact that explains a thrash, included when it changes. Tests: tests/agent/test_fallback_announce.py (RED without the emit) — once-only, re-entrant dedupe, and no-announce-when-same-model.
…h (Phase D) Negative/adversarial gate: with abort_on_summary_failure=false, a summary failure inserts a placeholder and compaction 'completes'. Drives the REAL compress_context done-site with a real ContextCompressor whose _generate_summary fails and a transcript whose protected 8-msg tail IS the request, so the post stays over threshold and sheds ~0% (reproduces the exact incident signature: tokens 86,858 -> 89,975 UP). Asserts the agent-visible _ineffective_compression_count increments on the real path, so a failed-summary loop can't thrash invisibly. The matched-pair #61 fix is verified separately (test_auxiliary_main_first.py + live runtime probe at deploy).
Per Ace's request, the fallback announce now shows the provider explicitly on BOTH sides as provider/model: 🔄 Model fallback: claude-app/claude-opus-4-8 → openai-codex/gpt-5.5 · context window 1M→272K The same model slug can be served by different providers, and the provider is the thing that explains a window/behavior change — so naming it on both sides makes the route unambiguous. Captures old_provider at the fallback site (alongside old_model, before agent.provider is reassigned) and threads it into _emit_fallback_announce as a keyword-only arg. Source side degrades to the bare model slug when old_provider is unknown. Test updated to assert the provider/model format on both sides.
A single-process multi-file pytest run (e.g. `pytest tests/agent/`) shares
one interpreter, so three module-level runtime caches in
`agent.auxiliary_client` leaked state across files:
* the "recently 402'd" unhealthy-provider cache — a real AIAgent
construction in an earlier compressor/agent test marks nous/openrouter
unhealthy (600s TTL), then `_resolve_auto` skips a *mocked* provider in
a later test_auxiliary_main_first test and returns None → 5 spurious
failures.
* the runtime-main override (set_runtime_main / clear_runtime_main).
* the resolved-client cache keyed by provider config.
The canonical runner (scripts/run_tests_parallel.py) spawns a fresh
subprocess per test FILE, so CI never saw this — but any single-process
run did. Wire each cache's existing reset entrypoint (all authored "for
tests") into the hermetic autouse fixture, mirroring the _plugin_manager
singleton reset already there. Adds a regression test proving each cache
is clean at the start of every test.
No production code touched — test-harness hermeticity only.
…t tests
Two more single-process / dev-Mac hermeticity leaks surfaced by running
`pytest tests/agent/` on a logged-in Mac (green in CI, which is Linux with
no Keychain and no exported HERMES_REAL_HOME):
* test_anthropic_adapter (14 failures): resolve_anthropic_token →
read_claude_code_credentials reads the macOS Keychain entry
'Claude Code-credentials' BEFORE the ~/.claude/.credentials.json file.
The adapter tests stub Path.home for the *file* source but nothing
intercepts the Keychain, so a dev's live OAuth token leaks past the
stub and fails ~14 'no creds' assertions. New tests/agent/conftest.py
autouse fixture defaults agent.anthropic_adapter.platform.system to a
non-Darwin value so the real `security find-generic-password` call can't
fire. Keychain-behaviour tests re-patch platform.system to 'Darwin'
inside the test body (and mock subprocess.run), so their patch wins and
they still exercise the real reader — verified test_anthropic_keychain
stays green.
* test_copilot_acp_client (1 failure): get_real_home() prefers
HERMES_REAL_HOME over HOME, so a developer shell that exports
HERMES_REAL_HOME defeats the test's monkeypatch.setenv('HOME', tmp) and
the real account home leaks into the asserted child-process env. Added
HERMES_REAL_HOME to the hermetic fixture's behavioral-var strip list,
next to HERMES_HOME_MODE.
Regression coverage added to test_aux_cache_isolation.py. No production
code touched — test-harness hermeticity only.
Fixes the last 6 single-process `pytest tests/agent/` failures (vision×4,
display_todo×1, bedrock×1) — all green in isolation and under the canonical
per-file-subprocess runner, so CI never saw them; they only bit a
single-process local run. Three more accumulating-state leaks, each reset in
the hermetic fixture (same pattern as the aux caches):
* models.dev capability cache (agent.models_dev._models_dev_cache):
test_models_dev.py assigns a tiny 6-provider SAMPLE_REGISTRY to the
module global with no teardown; test_vision_routing_31179._fresh_modules()
reimports auxiliary_client/image_routing but NOT agent.models_dev, so the
stale cache (missing real capability metadata) makes _lookup_supports_vision
return wrong answers and flips the vision routing assertions. Reset to
empty+expired.
* active-skin singleton (hermes_cli.skin_engine._active_skin /
_active_skin_name): once any test switches away from "default", the cached
skin persists and get_cute_tool_message reads the wrong tool prefix
(ares→"╎", daylight/poseidon→"│" vs default "┊"). Reset to lazy-init state.
* bare *_KEY credential env vars: hermes_cli.env_loader seeds os.environ from
the real ~/.hermes/.env at import time (before a test's HERMES_HOME redirect
applies), leaking CLAUDE_API_PROXY_KEY / CLAUDE_API_PROXY_F{N}_KEY. They end
in _KEY (not _API_KEY) so the credential filter missed them; they register as
claude-api-proxy providers and hijack resolve_provider('auto'), returning
claude-api-proxy before the Bedrock branch and breaking the AWS auto-detect
test. Added _KEY to _CREDENTIAL_SUFFIXES (only non-credential _KEY var,
HERMES_SESSION_KEY, is already stripped via _HERMES_BEHAVIORAL_VARS).
Verified no regression across credential-pool / api-key-provider /
auth-command suites (303 passed).
tests/agent/ now passes 4390/0 in a single process. No production code touched.
…atter + deduped emitter Adds _extract_compaction_summary_snippet, _format_compaction_announce (allow-list gating, token-reduction conditional tier, engine-correct recovery reference) and _emit_compaction_announce (set-after-emit dedupe, swallowed emit failure). 30 tests RED->GREEN. Tasks 1-3 of the compaction-announce plan.
…allback linkage _emit_fallback_announce now stamps agent._last_fallback_event (incl. turn_id + monotonic_time). _compaction_after_fallback links a compaction to a fallback ONLY when same-turn AND fallback-before-compaction (the §0 fallback-after- compaction case is NOT labeled); tight 75s wall-clock fallback only when no turn id. 37 tests green; existing fallback-announce tests unaffected. Tasks 4-5.
…ne-site Wires _emit_compaction_announce into compress_context after the done-log, inside the lock hold, before _release_lock(). Engine-discriminated (name=='lcm' vs built-in), dedupe key namespaced by engine, post-fallback linkage computed via _compaction_after_fallback. Real-agent done-site integration tests prove the built-in path emits one announce with the session pointer, and an aborted summary emits nothing. 39 tests green; existing compaction done-path tests unaffected. Task 6.
Drives the REAL LCMEngine through the real compress_context done-site (offline: summarizer stubbed, no LLM) and proves: (1) a real compaction emits exactly one engine-correct announce (engine: lcm, lossless guidance, no session pointer); (2) ingest-write-ahead — raw rows exist in lcm.db at announce time, backing the 'preserved in lcm.db' claim; (3) a no-op LCM pass stays silent (I8). This is the P1 reachability proof made executable. Task 7.
…-site guards (Task 8) Confirms (no prod change needed): the completion announce + recovery line are NOT matched by _TELEGRAM_NOISY_STATUS_RE (reach Telegram) while the transient start line stays suppressed; and all compaction callers (reactive conversation_loop + proactive turn_context) route through the one compress_context done-site, so the announce is structurally deduped with no second emit site.
WHAT: gateway/run.py now resolves _resolved_provider = getattr(_agent,
"provider", None) and includes "provider" in both _run_agent result
dicts (success path + the early/failure return). gateway/runtime_footer.py
guards _split_provider_model so a model string already carrying a
"provider/" prefix wins over a redundant passed provider (no triple
"a/b/model"). Adds the triple-collapse test case.
WHY: the runtime footer reads agent_result.get("provider"), but the
result dict only ever set "model" (bare, prefix already stripped) and
never "provider" — so provider_model silently degraded to a bare model
name on every live gateway turn (observed in Discord:
"claude-opus-4-8 · …" with no "claude-app/" prefix). Unit tests passed
provider explicitly, so the green suite never caught the missing dict key
— a classic green-in-tests / dead-on-the-gateway gap.
EFFECT: live footer now renders "claude-app/claude-opus-4-8 · …".
Confirmed live on both Apollo and Aegis gateways. 50/50 footer tests green.
This is the same change shipped to fork PR NousResearch#47600 (commit ddfb92dbb,
on a fresh-origin/main base); committing it to live fork/main so the
running gateway's behavior is reproducible and reset-proof rather than a
floating uncommitted working-tree edit.
No new config surface, no env vars, presentation-layer only.
… codex (#63) A multi-byte emoji whose UTF-16 surrogate pair lands across two SSE text_delta events produces a lone surrogate in the accumulated string. When that string is later UTF-8 encoded for the gateway/CLI wire, it raises UnicodeEncodeError. _is_provider_stream_parse_error classified that error as non-retryable, so a partial stream with deltas already sent fell straight through to the codex fallback (observed live: the 23:31 "Fallback activated: claude-opus-4-8 -> gpt-5.5"). Fix, three layers: - message_sanitization: active surrogate recombiner (_SurrogateSplicer) that rejoins a high/low pair split across deltas via a surrogatepass round-trip, plus _sanitize_surrogates floor that replaces any remaining lone surrogate so the wire write can never crash. - run_agent._fire_stream_delta / reasoning callback: apply the encode floor at the wire chokepoint, before any callback sees the delta. - run_agent._is_provider_stream_parse_error: treat a partial-stream UnicodeEncodeError caused by a lone surrogate as retryable, so a fresh stream re-chunks the boundary instead of stubbing to codex. Adds tests/agent/test_stream_surrogate_splicer.py (recombiner, floor, classification flip, both wire paths, idempotency, trailing-high flush). Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
Restores the gated pin_user_id feature + provenance to the live plugin file on disk. A sibling's fast-forward of fork/main had cleaned the working tree back to a HEAD without this code, leaving the feature only on the branch — the next gateway restart would have silently stopped pinning to 'ace' and re-fragmented memory by platform sender-id. Merging makes the live file carry the feature permanently. No conflict: main never touched plugins/memory/mem0/__init__.py since merge-base fdee6d9. Feature stays default-off (pin_user_id:false); the fleet enables it per-profile via config + MEM0_USER_ID=ace.
…olation (#64) Two follow-up fixes from Greptile review of merged PR #43. - plugins/memory/mem0/__init__.py: api_key was marked required:True even though is_available() bypasses it when host is set (self-hosted gates on admin_api_key). The required flag forced any setup/validation UI to demand a cloud key for a self-hosted server that never uses one. Mark it optional and document the conditional requirement; is_available() remains the real gate. - tests/test_request_composition.py: test_fixed_divisor_default_and_out_of_range_fallback mutated os.environ directly without try/finally, so a mid-loop assertion failure would leak HERMES_COMPOSITION_CHARS_PER_TOKEN_FIXED into later tests. Switch to monkeypatch + try/finally, matching the adjacent override tests. Adds behavior-contract tests: api_key not required in schema, self-hosted is_available without an api_key, and cloud mode still requires one. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…o_count bump, clamp /redo (#65) Four follow-up fixes from Greptile review of merged PR #49 (half-turn /undo+/redo). 1. hermes_undo.redo: a multi-op /redo that restored rows in an earlier op then hit the transcript-rewrite path (restore_ids->0) in a later op discarded the whole stack and returned reactivated_count:0 — so the caller printed 'nothing to redo' and SKIPPED its history reload (screen/DB desync), with redo_count never bumped despite committed work. Now stops at the rewrite, keeps + reports the earlier ops' progress, and bumps redo_count only when real work committed. 2. hermes_state.SessionDB.bump_redo_count: new public helper; hermes_undo no longer reaches into the private _execute_write. 3. hermes_undo._states: was an unbounded module-global dict keyed by session_id (memory leak in a long-running gateway). Now an LRU OrderedDict capped at _STATE_CAP=2048; eviction drops an in-memory redo branch, identical to the existing 'redo doesn't survive a restart' contract. 4. cli.py /redo handler: add the non-positive-count clamp /undo already has, so /redo 0 and /redo -N clamp to 1 instead of a misleading 'nothing to redo'. Tests: 8 new cases (partial-progress preservation, helper usage, LRU eviction, clamp parametrization) + full undo/redo suite (49) green. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
…imeout (#66) * test(run_agent): split monolithic test_run_agent.py to fix CI shard timeout tests/run_agent/test_run_agent.py had grown to 6,619 lines / 378 tests and exceeded the 140s per-file wall-clock cap in scripts/run_tests_parallel.py (the runner spawns one subprocess per file and cannot split within a file, so the slowest single file pins the whole sharded matrix). The file was being SIGKILL'd at ~10% complete, turning a green 4269-pass shard RED on a timeout — red on main and on every PR, firing the Fleet CI-fail alert. Split the monolith into 8 theme-focused modules (reasoning, streaming, api_kwargs, tool_exec, conversation, providers, init_memory, misc), moving the shared fixtures (agent, agent_with_memory_tool) into conftest.py and the mock-builder helpers into _run_agent_helpers.py. Pure move — zero test-body or production edits. Repointed the one sibling (test_partial_stream_finish_reason.py) that imported helpers from the old module. Faithful-split proof: - collection count for tests/run_agent/ unchanged: 1671 before and after. - the 8 new files collect exactly the original 378 (51+45+18+28+59+90+61+26). - full dir green via the CI runner: 116 files, 1668 passed, 0 failed. - slowest single file now 50s (was >140s -> killed); max well under the cap. Matches the repo's existing per-theme test layout in tests/run_agent/ and the 'refactor god-files into clean modules' guidance in AGENTS.md. * chore(secret-scan): allowlist fake test-fixture API keys in gitleaks Splitting test_run_agent.py re-introduced its long-standing placeholder api_key fixtures as 'added' lines, so the diff-scoped gitleaks scan flagged them as new (they were never real — the OpenAI/Anthropic clients are mocked, no network call). Add a .gitleaks.toml that extends the default ruleset and allowlists ONLY the distinctive tokens of those known fakes via stopwords: - 1234567890 (test-key-1234567890 constructor stub) - 2mno (gsk_ab...2mno / sk-ant...2mno redaction fixtures) - mnop (sk-or-...mnop key-masking fixture) Verified against the CI-pinned gitleaks 8.18.4: the PR range scans clean, and a freshly-injected high-entropy api_key in the same test file is still CAUGHT — the allowlist exempts only the known fakes, not the rule. --------- Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
… A (explicit when-to-save/when-not + proactive system-prompt block). Validated by holdout under-save eval: 24/25 recall LB 0.805, 0/25 false-save.
The 3-tier Anthropic lookup (exact -> dot-normalized -> date-stripped base) was undocumented while both helpers it delegates to (_strip_anthropic_release_date, _normalize_codex_model_name) carry precedence docstrings. Add a concise docstring stating the most-specific-first order and the None contract, matching the module convention. Comment/docstring only -- no behavior change. Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
A Discord "is typing…" bubble could stay lit indefinitely after a turn was fully delivered, with NO rate-limiting involved (distinct from the 429 stuck- bubble fixed in #34). Field report 2026-06-20: a bubble stayed lit ~26 min on a channel with zero 429s and zero turns since the last response. Root cause: a check-then-act race across teardown. Under concurrent same- channel turns, a late send_typing() — from BasePlatformAdapter._keep_typing() refresh tick, or the gateway follow-up restart (gateway/run.py:16006) — can RECREATE the per-channel _typing_loop AFTER the owning turn's stop_typing() already popped+cancelled it. The duplicate-guard (chat_id in _typing_tasks) is empty at that moment, so a new orphaned loop arms and re-POSTs the typing indicator (~every 12s) forever with no live owner to cancel it. base.py's own _keep_typing finally already documented this recreate path. Fix: a monotonic per-chat typing OWNERSHIP token, issued by the base adapter at turn start (_issue_typing_token) and threaded through _keep_typing -> send_typing -> _typing_loop. A late/stale send_typing carries its own now-superseded token and refuses to arm; the loop re-checks ownership each iteration and self-terminates if superseded. stop_typing never mutates the owner dict (single incrementer = _issue_typing_token; single popper = _stop_typing_refresh on owner-match), so the owner-match teardown compare can't be raced. The Discord adapter co-locates task+stop_event+token in three dicts popped together and only clears shared state when the loop is still the registered owner (no clobber of a newer turn's loop). token=None preserves the legacy unconditional behavior for all existing call sites and other adapters, so this is opt-in and behavior-preserving everywhere else. The base-level primitive covers the whole bug class; the Discord adapter is the per-adapter consumer in this PR. Other adapters with the same _typing_tasks/stop_typing shape (signal/yuanbao/slack/bluebubbles/weixin) can opt in with a one-line loop check as a follow-up. tests/gateway/test_discord_typing_recreate_race.py: - test_orphaned_loop_outlives_stop drives the REAL unmodified adapter through the field interleaving and was captured RED on main (orphan survived + kept POSTing) before this fix; GREEN after. - 3 structural guards for the token primitive (late-arm refusal, no-false-stop of a superseding turn, bounded owner dict). All existing typing tests (test_discord_typing_stop.py #34 429 cases, test_keep_typing_timeout.py) stay green.
Contributor
Author
|
Wrong base — this fix targets the deployed fork (which carries the #34 429-fix prerequisite that upstream main lacks). Re-opening as a fork-targeted PR with a clean 4-file diff. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A Discord "is typing…" indicator can stay lit indefinitely after a turn is fully delivered — with no rate-limiting involved. This is distinct from the 429-driven stuck-bubble fixed in #34 (commit
d40157738): that fix is live and working; this is a second, separate stuck-bubble path.Field report (2026-06-20): an "is typing…" bubble stayed lit ~26 minutes on a busy channel that had zero typing-429s and zero turns since the last response — i.e. something was still re-POSTing the typing indicator with no live owner.
Root cause — a check-then-act recreate race across teardown
Two stacked typing layers per chat:
BasePlatformAdapter._keep_typing()refreshes every ~2s while a turn runs.DiscordAdapter.send_typing()spawns a per-chat_idbackground_typing_loop(tracked in_typing_tasks[chat_id]) that POSTs/channels/{id}/typingevery ~12s;stop_typing()pops + cancels it.send_typingdedups withif chat_id in self._typing_tasks: return. The bug is a check-then-act race across teardown, whichbase.py's own_keep_typingfinallyalready documents:Under concurrent same-channel turns, a late
send_typing()— from a_keep_typingrefresh tick already in flight, or the follow-up restart atgateway/run.py:16006— lands after the owning turn'sstop_typing()popped the task dict. The duplicate-guard is empty at that instant, so a fresh_typing_looparms with no live owner scheduled to cancel it. No 429 is involved (so the #34 consecutive-failure cap never trips), and successful typing POSTs aren't logged, so the channel looks silent while the bubble stays lit forever.Fix — monotonic per-chat typing ownership token
A turn that drives typing claims a monotonic ownership token at turn start (
_issue_typing_token, in the base adapter), threaded through_keep_typing → send_typing → _typing_loop:send_typingcarries its own, now-superseded token and refuses to arm (_typing_token_is_current). The token is captured at turn start, not read at send time, so a tick belonging to an already-stopped turn can't re-arm._issue_typing_tokenis the only incrementer (new turn);_stop_typing_refreshis the only popper (owner-matched teardown).stop_typingnever mutates the owner dict, so the owner-match compare can't be raced.task+stop_event+token(three dicts popped together) and clears shared state in the loop'sfinallyonly when it's still the registered owner (is asyncio.current_task()), so a stale loop's unwind can't clobber a newer turn's loop.token=Nonepreserves the legacy unconditional behavior for every existing call site (error/interrupt stop paths) and every other adapter — this is opt-in and behavior-preserving everywhere except the Discord turn path.Growth is bounded to one int per distinct
chat_idever seen (issuance doesget()+1on the same key) — same bound as the existing per-chat typing dicts.Whole-bug-class scoping
The token primitive (issue / supersede / owner-matched teardown) lives in
BasePlatformAdapter, shared by all adapters. This PR wires the per-adapter consumer for Discord (the reproduced surface). Other adapters with the same_typing_tasks/stop_typingshape (signal,yuanbao,slack,bluebubbles,weixin) can opt in later with a one-line loop check — they keep unchanged (token=None) behavior until then.Tests
tests/gateway/test_discord_typing_recreate_race.py:test_orphaned_loop_outlives_stop— the authoritative repro. It drives the real, unmodifiedDiscordAdaptersend_typing/stop_typing/_typing_loopthrough the field interleaving (no 429s) and asserts no typing task survives the owning turn's stop. Captured RED onmain(orphan survived + kept POSTing) before the fix; GREEN after.Existing typing tests stay green:
tests/gateway/test_discord_typing_stop.py(the #34 429 cases) andtests/gateway/test_keep_typing_timeout.py. The full discord + base-platform gateway suite (-k "discord or test_base") passes (489 passed, 1 skipped).