CI evidence for upstream #99451 @ ce59b609 (do not merge) - #3
Open
KostaGorod wants to merge 700 commits into
Open
KostaGorod wants to merge 700 commits into
KostaGorod wants to merge 700 commits into
Conversation
…cket The CLI-based presence loop (`buzz-cli users set-presence`) can never keep Hermes agents visible in Buzz: each invocation runs the CLI as a fresh process, publishes a kind-20001 event, and closes its socket — and the relay drops the presence lease with the socket. A client that subscribes after that sees no presence records at all (verified live against the relay: read-only REQ for all five profile pubkeys returned ABSENT across three rounds 70s apart, while a connected subscriber saw only the single connect-time publish). Presence now rides the adapter's own long-lived authenticated WebSocket: a signed kind-20001 `online` event published inline right after the NIP-42 handshake — before the connection is reported ready — then refreshed by a heartbeat task at the expiry-aware cadence (every publish lands at least the configured margin before the relay's 180s TTL lapses). `offline` is published only on graceful shutdown, on the still-open socket, before the WebSocket task is cancelled; a transient disconnect never flaps offline because the relay clears the record itself and the reconnect's first publish restores it. Signing runs in the default executor (pure-Python schnorr is CPU-bound, ~50ms/event) and every frame shares a send lock so the inbound pump is never stalled. Two shutdown hazards surfaced while hardening this and are fixed here as well: - Cancelling during the initial publish (or inside `wait_for` on the NIP-42 handshake) can be silently consumed on Python 3.11 and the loop resumes with a pending cancel that is never observed as terminal — shutdown then wedges with orphan heartbeat/discovery tasks. The loop now re-arms an observed-but-consumed cancellation before spawning any child task, and the reconnect loop's generic `except Exception` re-raises when the task is cancelling. - The read-loop `finally` retired the heartbeat but left a window where a duplicate heartbeat task could run per connection; the reconnect test now pins exactly one heartbeat per connection and zero after the loop exits (the duplicate-heartbeat red test at the PR base is the first failing test). Kind-20001 events are rejected by the relay's HTTP bridge, so presence publishing cannot regress to the CLI path; the adapter tests pin that presence never spawns a CLI process. TDD: all new behavior covered by failing tests first (8 websocket + 4 adapter tests red at 96dd89a, green after the fix). Live verification: read-only relay probes recorded on kanban card t_5eabc763; post-deploy observation pending user confirmation.
GitHub answers anonymous fetches with HTTP 401 during outages (and for renamed/private repos). git then prompts `Username for 'https://github.com':` on the inherited terminal and `hermes update` sits there — users read it as Hermes demanding a GitHub login. Every network git call in the updater (fetch/pull/push, apply + --check + fork sync) now runs with GIT_TERMINAL_PROMPT=0 / stdin=DEVNULL, so the 401 fails fast into the fetch-failure classifier, which now reports it as a GitHub-side rejection (likely outage) rather than blaming the user's credentials. Credential helpers/askpass are left configured so private-fork origins still authenticate. Live repro: PTY-attached update --check against a 401 origin hung 15s+ on the prompt before; exits rc=1 in 0.2s with the diagnosis after. Same class as NousResearch#73751 (@Frowtek, pre-main.py decomposition); passive banner half salvaged from NousResearch#101421 (@RobbertC5).
Needed by the check-attribution gate before salvaging NousResearch#95964 and NousResearch#85806.
Issue NousResearch#54465 established that a same-provider retry after a full-budget timeout costs a second whole `timeout` window before the fallback chain is reached, doubling the user-visible stall, and that compression must not pay it because it sits on a critical path. The guard added for that is spelled `task == "compression"`, so vision — which sits on the interactive path — still retries. The cost is the same and the stall is more visible: the turn holding the image cannot answer, and because turns are serialised the following user messages queue behind it. Two sequential full-budget timeouts on an unhealthy vision provider is a long stall for something the fallback chain could have served immediately. Replaces the string comparison at both retry sites (sync `call_llm` and `async_call_llm`) with `_TIMEOUT_NO_RETRY_TASKS = {"compression", "vision"}`, so the two paths cannot drift again. Behaviour is unchanged for every other task: fast blips (a streaming-close or a 5xx) still retry, and only full-budget timeouts on those two tasks skip straight to fallback. Tests: vision now falls straight through to fallback with the primary tried exactly once, and a non-critical task still gets its one same-provider retry, so the change stays scoped. Reverting the source change fails the vision test and leaves the scoping test green. Not the same as NousResearch#51513, which fixes five separate defects in the vision fallback chain (capability detection, sync/async client misuse, geo-block and RemoteProtocolError classification, and chain iteration). This is about what happens before that chain is reached. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… skip f50b5bb taught the sync retry site to keep the cheap same-provider retry when a Codex stream dies inside the 60s no-progress window (zero output), skipping straight to fallback only on a stall or hard-ceiling timeout. The async site never got that carve-out, so after widening the skip to vision (NousResearch#97572) an async vision call on a stillborn stream would have jumped to fallback where the sync path retries. Both sites now apply the same rule. Adds the async twin of the vision-skip test and a no-progress-still-retries guard for the async site.
The sync and async retry sites each re-derived the same three-clause decision (critical task + full-budget timeout + not a no-progress fail) with their own copy of the rationale — which is exactly how the async site drifted in the first place. _should_skip_same_provider_retry() now owns the rule and its carve-out next to _TIMEOUT_NO_RETRY_TASKS; both sites call it. Behavior-preserving: same clauses, same exception object, same outer guard.
iter_skills_files() walked the skills tree with a bare rglob("*"), so the
.hub download cache, .archive, curator backups, and any node_modules/.git
under a skill package were uploaded to the sandbox on every sync. The
sandbox never reads them: skill content is resolved host-side.
EXCLUDED_SKILL_DIRS is already the canonical exclusion set, honoured by
discovery and backup. Apply it to the sync path too, across all three
roots iter_skills_files() walks (local, external, project-local), and add
.curator_backups to the set.
Measured on a local install: 900 files / 67.3 MB -> 771 files / 8.4 MB.
This is not just wasted bandwidth on the SSH backend, where the oversized
payload can exceed the 120s _ssh_bulk_upload deadline and surface as the
agent hanging on every tool call.
The filter intentionally does not reuse is_excluded_skill_path(), which
also prunes references/, templates/, assets/ and scripts/ -- those hold
support files and bundled scripts the sandbox does read and execute.
…the sync walk Replaces the three hand-copied rglob loops + post-hoc parts check with one os.walk generator that drops EXCLUDED_SKILL_DIRS from dirnames before recursing. Same file set as the cherry-picked fix (the test binds it), but the walk no longer stats every file under .hub/.curator_backups/node_modules on each 5s FileSyncManager tick. Bench (synthetic skills tree: 20 skills + 400 .hub files + 5x8MB curator tarballs + 50 archived files): iter_skills_files() 35ms -> 2.4ms.
…mount copy _safe_skills_path() is the sibling of iter_skills_files(): when a symlink in skills/ forces a sanitized copy for mount-based backends (Docker/Singularity), it rglob-copied the whole tree — .hub, .curator_backups, node_modules and all. Prune EXCLUDED_SKILL_DIRS before descending, same rule as the sync generator, so the mounted copy never carries (or walks) the bookkeeping trees either.
… the .git exclusion Folds the incident explanation from NousResearch#91458 (@liuhao1024) into the comment on _EXCLUDE_TOP_LEVEL so the reason .git is excluded — compounding snapshot growth, not just rollback safety — survives next to the set. Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Excluding nested .git from snapshots has a side effect on rollback: the staging move takes the whole live skill dir (including its .git) into .rollback-staging-*, the extract restores the snapshot without it, and the staging dir is then deleted — so a skill that is itself a git checkout lost its .git on any rollback. Reproduced: main preserves it, the exclusion-only branch did not. After a successful extract, move excluded subtrees from the staged copy back under their restored skill dir (mirroring how a top-level .git survives by never being staged). Regression test included.
…kill drop Submodule and worktree checkouts store .git as a file (gitdir: pointer); the carry-over only looked at directory names, so that form was still lost on rollback. Handle files with the same guard. Docstring now states the deliberate limit: an excluded entry whose skill dir the target snapshot lacks is dropped with staging (no orphan .git) and is not undoable via the safety snapshot, which excludes these paths as well.
`create_app` registers `handle_proxy` as an `async def`, and it called `adapter.get_credential()` directly on the aiohttp event loop. `UpstreamAdapter` is a synchronous contract (`adapters/base.py` — every method is a plain `def`), and the shipped adapters implement it with blocking I/O. `NousPortalAdapter.get_credential` takes `_auth_store_lock()` — a cross-process advisory lock with `AUTH_LOCK_TIMEOUT_SECONDS = 15.0` (`hermes_cli/auth.py:110`) — reads `auth.json` off disk, and may issue a token-refresh POST; on a terminal `AuthError` it takes that lock a second time to persist the quarantined state. `XAIGrokAdapter.get_credential` reads its key pool off disk. The proxy is one process with one loop, and `handle_proxy` streams with `sock_read=300`, so long-lived completions are the normal case. Blocking inside credential resolution therefore freezes *every* concurrent in-flight stream mid-token for the duration — a concurrent `hermes auth` command holding the auth-store lock is enough to do it. Every proxied request goes through this path. Dispatch through `asyncio.to_thread` instead. This is a pure scheduling change: `to_thread` re-raises the worker's exception in the awaiting frame, so the existing `except Exception` -> 401 `upstream_auth_failed` mapping is unchanged, and the adapters' own `self._lock` still serialises concurrent resolutions exactly as before. Fixing it at the handler also leaves the synchronous `UpstreamAdapter` ABC untouched, so it covers every adapter without conflicting with in-flight work that subclasses it.
Adds `tests/hermes_cli/test_proxy_off_loop.py`, mirroring the harness in
`test_proxy.py`: the proxy and a fake upstream run as real aiohttp
servers on ephemeral ports under a single `asyncio.run`, guarded by
`pytest.importorskip("aiohttp")` — no pytest-aiohttp dependency.
The primary assertion is thread identity, not latency. A latency
assertion measured with an HTTP client on the blocked loop is vacuous:
the client's own timer cannot advance until the block ends, so it reports
a fast response on code that was provably frozen.
- `test_get_credential_runs_off_the_event_loop` records
`threading.get_ident()` inside the adapter and compares it to the loop
thread. Before the fix both are the same ident.
- `test_event_loop_keeps_running_while_credentials_resolve` runs a
heartbeat task on the loop and has the adapter sample its counter on
entry and exit, so the reading is taken from the loop rather than
through a client that shares it. Before the fix exactly 0 iterations
run across a 0.5s stall; after it, ~50.
- `test_credential_failure_still_maps_to_401` pins the error contract
across the change of call form. It is deliberately not in the
red-before set — it guards behaviour the fix must leave alone.
`handle_health` called `adapter.is_authenticated()` inline from an `async def`. `UpstreamAdapter.is_authenticated` is documented as "Should be cheap — no network calls. Used by `proxy start` for a clear up-front error before binding a port." (`adapters/base.py`), and that is true of the `proxy start` preflight, which runs in a plain synchronous CLI function. It is not true on the event loop: `NousPortalAdapter.is_authenticated` goes through `_read_state()`, which takes `_auth_store_lock()` — the same cross-process lock with a 15s timeout as credential resolution — and `XAIGrokAdapter` reads its key pool off disk. `/health` is precisely what a supervisor, systemd unit, container healthcheck or load balancer polls, on a fixed interval, so it is the endpoint least able to afford a lock wait; and a wait here freezes every concurrent proxied stream, not just the healthcheck. Offload it with `asyncio.to_thread`. The response body is byte-identical; only the scheduling changes.
Extends `test_proxy_off_loop.py` with the `/health` half, using the same two-assertion shape as the credential tests: - `test_is_authenticated_runs_off_the_event_loop` compares the thread the adapter's `is_authenticated` ran on against the loop thread. Before the fix they are the same ident. - `test_event_loop_keeps_running_while_health_resolves_auth_state` reads a loop-side heartbeat counter sampled by the adapter across its own stall. Before the fix exactly 0 iterations run across 0.5s. Both also assert the response is unchanged (`200`, `authenticated: true`), so the offload cannot quietly alter what `/health` reports.
`handle_proxy` already offloads the two credential reads on the happy path,
but the rotation inside the `upstream_resp.status in {401, 429}` branch still
called `adapter.get_retry_credential` inline on the event loop.
That is the most expensive of the three blocking methods on the
`UpstreamAdapter` contract, not the cheapest:
* `NousPortalAdapter.get_retry_credential` routes into
`_get_credential(force_refresh=True)`, so the token-refresh POST that
`get_credential` performs only near expiry is unconditional here — and it
runs under the same `_auth_store_lock()`, a cross-process advisory lock
with a 15s timeout.
* `XAIGrokAdapter.get_retry_credential` loads the key pool off disk and
calls `try_refresh_current` / `mark_exhausted_and_rotate` under its lock.
So every upstream 401 or 429 froze the proxy's single event loop — and with
it every other in-flight streaming completion — for the whole rotation. A 429
is exactly when the proxy is busiest, which is the worst moment to stall.
Wrap it in `asyncio.to_thread`, matching the two sites above. The error
contract is unchanged: `to_thread` re-raises the worker's exception in the
awaiting frame, so the existing `except Exception -> retry_cred = None` still
swallows a failed rotation and streams the upstream's own rejection back.
Extends the off-loop suite to the third and last blocking method on the
`UpstreamAdapter` contract, the 401/429 rotation.
As with the two existing pairs, the primary assertion is **thread identity**,
not latency: a latency assertion measured by an HTTP client on the blocked
loop is vacuous, because the client's own timer cannot advance until the
block ends and it therefore reports a fast response on provably frozen code.
* `test_get_retry_credential_runs_off_the_event_loop` records
`threading.get_ident()` inside the fake adapter and compares it to the
loop thread, and checks the rotation still works end to end (rejected
bearer forwarded first, rotated bearer second).
* `test_event_loop_keeps_running_while_the_retry_credential_resolves`
samples a loop-side heartbeat counter from inside the stalled adapter. On
the unfixed handler it records exactly 0 loop iterations across a 0.5s
rotation.
* `test_retry_credential_failure_still_returns_the_upstream_rejection`
guards the error contract the change must leave alone: a raising rotation
is still swallowed and the upstream's own 401 is streamed back, with no
second forward.
A new `_build_rejecting_upstream` harness drives the `status in {401, 429}`
branch by rejecting every bearer except the rotated one.
DELETE /api/profiles/{name} called profiles.delete_profile() inline on the
ASGI event loop. When the target profile has a gateway running, that call
stops it via _stop_gateway_process(), which polls the PID every 500 ms for
up to 10 s before escalating to a force kill, and then removes the profile
tree.
For the whole of that window the dashboard process serves nothing else.
web_server.py's own notes record what that costs: a stall of this length
"caus[ed] the Desktop's 10-second WebSocket ready-probe to time out
(NousResearchGH-73083)", and both the desktop app and the dashboard's Chat tab drive the
agent over those WebSockets. Deleting a profile whose gateway is up is a
routine action that reliably reaches the full ten seconds — the handler's
own output announces "Gateway is running - it will be stopped".
Move the call into the default executor via run_in_executor, matching
list_profiles_endpoint, export_profile_endpoint and import_profile_endpoint
in this same module. The exception-to-status mapping is unchanged:
FileNotFoundError/ValueError are raised inside the worker and re-raised by
the await, so they still map to 404/400.
POST /api/profiles/{name}/describe-auto called
profile_describer.describe_profile() inline. That function is a plain def;
it reaches agent.auxiliary_client.call_llm(), also a plain def, which makes
a synchronous provider request with a 60-second ceiling.
Held on the ASGI event loop that is six times the 10-second WebSocket
ready-probe threshold web_server.py records as the point where the desktop
app gives up (NousResearchGH-73083). A single describe-auto on a slow or unreachable
auxiliary provider therefore takes the whole dashboard offline for up to a
minute, including the /api/ws and /api/pty sockets the desktop app and the
Chat tab run on.
Move the import and call into the default executor. _resolve_profile_dir()
deliberately stays on the loop ahead of the hop: it is a name validation
plus a single stat, and it owns the 400/404 responses that the handler's
`except Exception` would otherwise turn into a 500.
Three more handlers in this router did filesystem work inline on the ASGI
event loop:
- PATCH /api/profiles/{name} calls rename_profile(), which stops a running
gateway through the same 10-second _stop_gateway_process() poll that
delete uses, then renames the profile directory, rewrites the Honcho
host blocks and regenerates the wrapper script.
- GET /api/profiles/active reads the active_profile state file and
resolves HERMES_HOME against the profiles root. The sidebar polls it.
- POST /api/profiles/active stats the target profile, creates the state
directory and writes active_profile via a temp file plus replace.
Rename carries the same worst case as delete and belongs off the loop for
the same reason. The two active-profile handlers are individually cheap,
but they are the routes the dashboard polls, so they are the ones most
likely to be queued behind something slower — and leaving them inline is
what made the router inconsistent with list_profiles_endpoint, which
already offloads a plain directory listing eight lines above.
The two reads in GET share one executor hop rather than taking one each.
The remaining in-scope handlers in this router read and write profile
documents inline on the ASGI event loop:
- GET /api/profiles/{name}/soul reads SOUL.md
- PUT /api/profiles/{name}/soul atomic_write_text(SOUL.md)
- PUT /api/profiles/{name}/description write_profile_meta(profile.yaml)
- GET /api/profiles/{name}/desktop-overlay reads desktop.json
The persona save is the sharpest of the four: atomic_write_text() writes a
temp file, fsyncs it and replaces the original, so the loop is parked for
however long the filesystem takes to durably commit — unbounded on a slow
or contended disk, and paid on every Save in the editor.
Each handler keeps its existing status-code mapping. The reads probe and
load in a single executor hop rather than two, which also avoids widening
the gap between the existence check and the read.
Both readers return a _MISSING sentinel rather than None for an absent
file. desktop.json may legitimately contain the document `null`; collapsing
that onto None would newly report an existing-but-empty overlay as absent.
The same distinction is what the SOUL.md durability tests rely on, where
"file missing" and "file empty" must not both read as never-set.
_resolve_profile_dir() stays on the loop in all four, as it does in the
rest of this sweep: it is a name check plus one stat, and it owns the
400/404 responses.
Two assertions per offloaded site: - a loop probe, where the stubbed callee records whether an event loop is running in its own thread — the idiom already used by tests/hermes_cli/test_cron_dashboard_off_loop.py; and - a concurrency proof, where the stubbed callee blocks on a threading.Event while an unrelated request is timed. On the unfixed handlers that request waits out the whole block; served off the loop it returns in milliseconds. The concurrency proof needs a single event loop across requests, so the client fixture enters the TestClient context manager: that pins one blocking portal for the whole fixture, where a bare TestClient(app) would spin up a fresh loop per request and pass even unfixed. Also covers the status-code mapping through the executor hop (404 on a missing profile, 400 on a rename collision, 404 from the resolve that stays on the loop ahead of describe-auto) and the _MISSING sentinel cases: a desktop.json holding `null` still reports exists=true, an absent one reports exists=false, and an empty SOUL.md is still distinguishable from a missing one. The client fixtures read web_server._SESSION_TOKEN from the module rather than pinning a literal. web_server resolves that token once at import, so whichever test file imports it first fixes the value for the session and a later monkeypatch.setenv is silently ignored — two files hardcoding different tokens would 401 depending on collection order.
…ile model write The module already binds run_in_threadpool (used by list_profiles_endpoint) and every sibling router uses the same starlette helper; the nine new loop.run_in_executor(None, _run) sites now go through that alias so the file has one offload idiom. Behaviour-identical (both hand the callable to a worker thread). Also sweeps the one endpoint the PR left synchronous: update_profile_model_endpoint's _write_profile_model reads and rewrites the profile's config.yaml on the event loop.
Network off (unplugged) froze the backend 17 minutes: the async /api/credentials/pool endpoints (GET/POST/DELETE) called load_pool() synchronously on the event-loop thread -> Copilot token exchange -> blocking urlopen -> getaddrinfo stuck in C for 1016s, immune to urlopen(timeout=10). WS dropped (1006), sessions detached, even log writes stalled. Fix 1 (web_server.py): move the three endpoint bodies into asyncio.to_thread, matching the file's existing _run pattern. Fix 2 (copilot_auth.py): _urlopen_bounded() runs the request in a daemon thread with a wall-clock hard cap (timeout+5s) so DNS hangs can no longer block any caller indefinitely. Measured: simulated DNS hang now raises TimeoutError after 6.0s instead of freezing the loop.
…sponses Follow-up to the off-loop move: once the credential-pool handlers run on worker threads, the dashboard's periodic /api/credentials/pool polls can overlap, and during a DNS outage each poll would have started its own exchange and abandoned its own hung resolver thread. - Per-fingerprint threading.Lock around the exchange: concurrent callers wait on the one in-flight attempt, then hit the positive or negative cache (bounded worker count, no duplicate network calls). - _urlopen_bounded: when the hard cap fires and the abandoned worker later succeeds, close the HTTPResponse instead of leaking the socket. - Tests (none shipped with the original PR): hard cap + late-close, single-flight success and failure paths, and the pool endpoint running off-loop / keeping the loop responsive under a 200 ms blocking read.
…ootgun false positive The naive line scanner's open() regex matched the human-readable phrase "next picker open (or refresh)." inside the warning string, tripping the Windows footgun gate. Reword to "next picker open or refresh."
… media gates A ProviderProfile that declares supports_vision_tool_messages=False accepts images in user messages but rejects list-type tool-result content with 400 (xiaomi/MiMo "text is not set"). supports_vision=True alone used to flip _supports_media_in_tool_results to True, and a vision-capable capability lookup could re-open _should_use_native_vision_fast_path — so the native multimodal envelope landed in a role:tool message and 400'd every turn. Both gates now go through one _profile_rejects_tool_media() veto. Refs NousResearch#89981 (cherry picked from commit daed88f, trimmed)
…k 400s on image tool results Muse Spark accepts images on user turns but returns HTTP 400 invalid_request_error 'messages[N].content did not match any supported type' when the vision_analyze multimodal envelope lands in a role:tool message. With the profile veto now honored by the vision fast-path gates, declaring the limitation routes tool-result images through the aux-LLM text path while user-message vision stays enabled. Fixes NousResearch#101668 Refs NousResearch#47742
Add meta/muse-spark-1.3 and meta/muse-spark-1.3-contributor to the OpenRouter curated list, the meta-ai provider fallback, the opencode-zen / opencode-free / opencode-go floors, the setup-wizard shortlist, and regenerate the hosted model catalog.
…rk-1.3 - model_data_policy_guard: name the triggering -contributor model instead of hardcoded 1.2; per-version verified price tables (1.3 standard $1.25/$4.25 via OpenRouter live metadata; cached figures 1.2-only) - model_metadata: muse-spark-1.3 + muse-spark family at 1048576 (OpenRouter verified 2026-09-02) with pre-catalog stale-cache keys so 256K-fallback sessions self-heal - docs: contributor-tier notes cover 1.2 + 1.3 - tests: 1.3 guard regression, muse stale-cache guard, live-catalog mirror gains 1.3-contributor-free (confirmed on live relay) 143 tests pass (guard, selection guards, opencode catalog, model_metadata); ruff clean.
Muse Spark 1.2 family (api.meta.ai) ships 1M context (models.dev opencode/muse-spark-1.2 = 1048576, meta/muse-spark-1.2 = 1048576). Zen/GO SG /v1/models only returns id (no limit.context), and models.dev lookup via opencode was missing a hardcoded fallback, so get_model_context_length fell back to DEFAULT_FALLBACK_CONTEXT=256k. Banner showed Context: 256,000 for both zen and router-sg lanes. Add longest-prefix entries 'muse-spark' and 'muse' = 1_048_576 so all variants (1.1, 1.2, contributor, contributor-free) resolve to 1M without network.
commandcode (api.commandcode.ai) exposes authoritative context_length via /models (muse-spark 1M, etc.) but as a known provider it skipped the custom-endpoint probe at step 2 and has no models.dev entry, so every model fell through to the 256K DEFAULT_FALLBACK. Add a provider-aware branch mirroring gmi/nous to resolve via _resolve_endpoint_context_length. Fixes GOAT docs vs status-bar mismatch: muse-spark 1M was shown as 256K.
…in Muse Spark 1M invariant opencode-free had no PROVIDER_TO_MODELS_DEV entry, so every models.dev lookup on the free tier missed and Muse Spark fell to the 256K default. The free tier is served by the Zen relay (hermes_cli/models.py: "opencode-free is Zen-hosted"), and models.dev's "opencode" provider is the catalog that lists muse-spark-1.2 / -1.2-contributor-free / -1.3-contributor-free at 1,048,576 — so the alias is "opencode", not "opencode-go" (Go's catalog carries only the paid -contributor SKUs). Missing alias identified by @Steve-prog001 in NousResearch#101905. Tests: one parametrized offline invariant (models.dev + live /models mocked away) asserting 1,048,576 on opencode-free / opencode-go / meta-ai / commandcode — fails on main, passes here — plus the alias pin.
…e on defaults (NousResearch#96550) On agent.tool_use_enforcement/execution_guidance "auto", muse-spark-* was in neither model tuple, so it received only the universal finish-the-job block, answered in prose with 0 tool calls, and the turn closed on finish_reason=stop. Add "muse" to both tuples; Claude and every other family are unchanged. Co-authored-by: Edder Talmor <talmoredder@gmail.com>
The notification action (`NotificationItem`) rendered as `variant="textStrong" size="xs"` — an 11px underlined muted-grey text link with a ~44x20px hit target. On the data-training confirm toast raised by `surfaceModelSwitchConfirm` / `confirmModelWarning` (e.g. picking `muse-spark-1.2-contributor`) it read as a footnote, not the one action the toast exists for, and users reported not being able to "press to accept". Promote it to the SDK's `default` variant at `size="sm"`: a filled primary button, larger hit target, obvious affordance. No new styles. Salvaged from NousResearch#96562 (toast half only). Refs NousResearch#96563.
…an already-open tab A roster click on a bot whose canonical Bot Chat is already open only fronted the tile: the pane kept whatever transcript it last painted, which can predate rows the bot wrote while the user was elsewhere (a cron delivery, a teammate's message_agent, another bot's turn). The stale snapshot persisted until the next user turn — NousResearch#95600's forceResume only covered the not-yet-open registry path. Reuse refreshOpenBotChat (the NousResearch#99393 reclaim mechanism) on the fronted branch so forceResume re-pulls the latest transcript. Regression test pins the behavior: fronting an open Bot Chat now requests the canonical registry open.
…its read readPersistedPoolLimits() runs at module evaluation and logs through rememberLog() on every branch, but hermesLog / desktopLogBuffer / desktopLogFlushTimer / desktopLogFlushPromise were declared ~110 lines later. esbuild lowers const/let to var, so the packaged desktop died on every launch with "Cannot read properties of undefined (reading 'push')" (NousResearch#101941, NousResearch#101960). Moving the four declarations above the read fixes the crash and keeps the early [pool-limits] line in desktop.log. Salvaged from NousResearch#101945 (test dropped: Desktop E2E lane is disabled in CI).
…x clients per N children x W worktrees)
…ter-snapshot rows in --compare
…eFolders Multi-root servers (pyright) are keyed by server_id; a file whose resolved root is new for a running client is attached with workspace/didChangeWorkspaceFolders instead of spawning another server. Single-root servers keep the (server_id, workspace_root) key and behavior. A profiled fan-out across ~30 worktrees ran 30-60 pyright processes (~8.7 GB); the same fan-out now runs one.
A fan-out of 30 delegated children built 183 httpx.HTTPTransport objects (each with its own httpcore pool + parsed SSL context): 3 per agent x (primary + aux clients). A profiled session with ~130 children held 107 TLS sockets to one provider. Peak RSS for the 30-child bench drops 286 -> 195 MB; live HTTPTransports 183 -> 2, ConnectionPools 183 -> 7. What is shared: the sync `HTTPTransport` (pool + SSL context) per (scheme, verify, proxy, happy-eyeballs) identity, in a bounded module dict. What is NOT shared: the per-agent `httpx.Client` wrapper. Each client mounts a `_SharedTransport` view whose `close()` marks only that view closed and never touches the pool, so the NousResearch#10933 contract (close client A, build client B, B works) holds unchanged — the pinning tests in test_create_openai_client_reuse.py / test_sequential_chats_live.py pass as-is. Safety for cross-thread aborts: `_SharedTransport.handle_request` stamps its id into `request.extensions`; `_iter_pool_sockets` now only shuts down a shared pool's in-flight requests carrying the calling client's stamp and never its idle connections, so interrupting child A cannot sever child B's stream (NousResearch#29507 / NousResearch#72975 walker semantics preserved for unshared pools). Also: - `resolve_httpx_verify` caches one SSLContext per CA-bundle path. With SSL_CERT_FILE/HERMES_CA_BUNDLE set, every agent used to parse the bundle again and — because the share key is context identity — get a private pool. - The client no longer builds a third, unused default transport; its default transport is the https view. - Mounted transports now actually receive pool limits (Client-level `limits=` never reached them, so mounts ran on httpx defaults with a 5 s keepalive_expiry). The shared pool uses 50 keepalive / 1000 max so one pool covers a whole concurrent fan-out. - `close_shared_transports()` really closes the pools (tests / shutdown). Async clients (`async_mode=True`) stay unshared: an httpcore async pool is bound to the event loop that first uses it. Proxy-backed clients keep httpx's per-client proxy transport.
…scripts in the parent heap A parent that fanned out 1,320 subagents over 13h reached 2.6 GB RSS (1.9 GB anonymous heap). Every closed child AIAgent stayed reachable and still owned a copy of its full message history. gc.get_referrers on a finished child (30-child fan-out bench, evals/fanout_resource_bench.py) showed two retainers: 1. bind_subagent_parent() stored the agent strongly in the `hermes_subagent_lifecycle_parent` ContextVar. Each child binds ITSELF for its own turn, and every asyncio Handle/Future scheduled during that turn (LSP reader loops, kernel pipe transports) snapshots the Context — 56 live Contexts held 14 finished children after the bench. The ContextVar now holds a weakref (non-weakrefable doubles fall back to a closure); get_active_subagent_parent() dereferences it. 2. AIAgent.close() cleared _session_messages but not the _db_flush_scan_prefix snapshot (a `messages[:]` shallow copy taken on every successful DB flush) nor _streamed_assistant_text_parts, so the agent — kept alive by (1) — retained every message dict. close() now drops both. The delegate_task result entry never carried `messages`; a pin test confirms the per-child result JSON is unchanged. Bench (30 children / 10 worktrees, ~100 KB final replies so retention is visible): post-fan-out live child AIAgents 14 -> 0; RSS after fan-out 636 MB -> 556 MB. With the harness' tiny default replies both runs sit at ~192-194 MB (the children's transcripts were never the dominant cost there; the leaked objects were).
…ndex (schema v30)
On a fan-out-heavy install state.db reached 3.4 GB; 70% of message bytes
belonged to subagent sessions, and every one of those rows was also
indexed into messages_fts_trigram, whose shadow tables are ~2.6x the
text they cover (1,029 MB trigram vs 350 MB standard FTS on that DB).
session_search already hides source='subagent' sessions, so the
substring/CJK index bought nothing for them.
Extend the v29 cron exclusion: the messages_fts_trigram_src view, the
three sync triggers, and both deferred-backfill INSERT...SELECTs now use
one shared predicate (FTS_TRIGRAM_SESSION_SQL / fts_trigram_session_sql)
that skips sessions with source IN ('cron','subagent') or the
$._delegate_from creation marker (children spawned under a gateway turn
inherit the gateway's source). Compression/branch continuations carry
parent_session_id without the marker and stay indexed. Child rows remain
canonical in `messages` and fully indexed in the standard messages_fts
word index; explicit source_filter=['subagent'] CJK searches route to
LIKE like cron already did.
The v29 migration gate becomes `< 30` and reuses the same view-swap +
admitted rebuild, so existing installs purge historical child postings
once on open. Fresh DB with 2,000 x 2 KB child messages: 22.4 MB ->
12.5 MB (trigram shadow 10.09 MB -> 0.02 MB).
A fan-out of N in-process subagents used to add one sleeping daemon thread per delegated child (delegate heartbeat, 30s) and one or two per active turn (durable turn-lease refresher; turn-liveness watchdog). A profiled session with ~130 children was carrying ~1000 threads. All of these timers now run on a single process-wide daemon thread. - agent/periodic_scheduler.py (new): heap-ordered periodic scheduler on one Condition-driven daemon thread. schedule(fn, interval) -> handle; handle.cancel(wait=) blocks for an in-flight run like the old join. A callback returning False stops itself; a raising callback is logged at debug and rescheduled, so one bad timer cannot kill the rest. - tools/delegate_tool.py: _heartbeat_loop body -> _heartbeat_tick, scheduled at _HEARTBEAT_INTERVAL; stale-cycle closure state and idle/in-tool thresholds unchanged; cancel(wait=5) in finally where the stop-event + join(5) lived. - run_agent.py: _refresh_durable_turn_lease body scheduled at _lease_refresh_interval; lease-lost / refresh-error interrupt paths and the stop-event fencing are unchanged; the join(timeout=1.0) is now cancel(wait=1.0) so the interrupt clear still runs after any in-flight tick. - agent/turn_liveness.py: TurnLivenessWatchdog.make_thread/start -> schedule(); the poll body is _tick(), same sampling state machine. Bench (evals/fanout_resource_bench.py, 30 children / 10 worktrees, ok=30/30 both): peak threads 168 -> 132. At peak the old tree held 30 "Thread-N (_heartbeat_loop)" threads; the new one holds zero plus one "hermes-periodic-scheduler".
Drop the loop-side '(empty)' rewrite (the turn-completion explainer already owns that at delivery, and gateway/desktop match on the sentinel) and the extra token-count persistence. Keeps: usage-absent empty streaks arm the deterministic fast-fail after two attempts with no content or reasoning, and every completed API call logs even when the provider omits usage (NousResearch#101898).
…istant prose (NousResearch#101899) GLM-style models serialize tool calls as XML in the text channel; when the stream drops mid-serialization with finish_reason=stop, the orphan <arg_key>/<arg_value> fragment (or a bare unclosed <tool_call> opener) matched neither the complete-block stripper nor the partial-stream guard and was stored and displayed as ordinary assistant content. strip_think_blocks (storage boundary) and the CLI display copy now strip an unterminated block-boundary tool-call opener, or any line carrying stray argument markup, to end of text. The response then reads as empty and flows through the existing empty-retry path. Complete blocks and inline prose mentions are unchanged.
…ces/corrections) to skills; memory is the every-session exception The memory guidance led with 'Save proactively' and the memory tool schema ranked 'user preferences & corrections' as top priority, while the skills nudge was a conditional 'offer to save'. In practice that asymmetry made the agent end sessions writing memory entries (fighting a 2,200-char budget) and skip updating the skill it had just used, even though the procedure was the reusable artifact. Both surfaces now state the same rule with skills first: what you learn doing a task, including the user's preferences and corrections for that kind of work, goes in the task's skill; memory is only for facts that apply to every session.
…ng is disabled
Reasoning-mandatory routes answer reasoning: {enabled: false} with HTTP 400
"Reasoning is mandatory for this endpoint and cannot be disabled". Hermes
sends that disable for /reasoning none, agent.reasoning_effort: none, and the
one-shot thinking-exhaustion continuation override (which GLM-5.3-flash
triggers on its own). The Nous profile's catalog guard swallows the disable
only when its per-process capability cache already says mandatory; a gateway
that warmed the cache before the route flipped kept sending it, and the 400
was classified as a non-retryable format_error that aborted the turn.
- error_classifier: new reasoning_mandatory reason (retryable, no fallback,
no compression), matched before the request-validation branch.
- conversation_loop: one-shot recovery — set agent._reasoning_disable_rejected,
queue a catalog refresh for the provider, retry.
- chat_completion_helpers: _reasoning_config_for_wire drops every disable
(configured or ephemeral) once the route has rejected one.
- hermes_cli/models: refresh_reasoning_caps_async(provider) forces a
background re-fetch of the Nous/OpenRouter catalog.
- openrouter profile: omit a disable when the catalog marks the route
mandatory (parity with the Nous profile).
Live: z-ai/glm-5.3-flash on the Portal with a poisoned mandatory:false cache.
Before: turn aborted with the 400. After: one retry, thinking stays on, turn
completes.
…r's own effort
The retry must land on the same provider cache key as every prior request
in the session. Discard only the one-shot continuation disable and send
agent.reasoning_config verbatim; a config that is itself a disable is
omitted (that session never sent anything else, so nothing warm is lost).
Live: user effort=high, ephemeral disable → 400 → retry carries
{enabled: true, effort: high}.
Resolve adapter.py import conflict: keep our ProcessingOutcome (reaction lifecycle) alongside upstream's cache_media_bytes_async from the media-cache offload sweep (568b161) — its call sites already migrated to the async seam, so the sync cache_media_bytes import is dropped. Presence fix (ce59b60) invariants preserved: same-socket kind-20001 publishing via _send_ws send-lock, executor-signed events, single heartbeat-per-connection with reconnect replace, offline-before-close.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork-internal CI vehicle. Upstream PR NousResearch#99451 has zero check-suites registering (Actions ingestion anomaly) and its workflows sit action_required (first-time-contributor approval). This PR reproduces the identical diff (head ce59b60 vs base 375ce8e = upstream PR base) so the same CI lanes run at the exact head SHA. Do not merge.