fix(ci): make tests, workflows, and attribution reliable under load - #66373
Conversation
…s/emails/ directory The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet: every concurrent salvage PR appended entries to the same lines of the same file, so parallel PRs re-conflicted on every merge to main. New system: one file per email under contributors/emails/ — filename is the commit-author email, first non-comment line is the GitHub login. File additions never conflict, so any number of PRs can add mappings concurrently. - scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen) merged with the directory at import time (directory wins). All existing consumers (resolve_author, contributor_audit.py) unchanged. - scripts/add_contributor.py: idempotent CLI to add a mapping; refuses conflicting reassignments (incl. against the legacy map), validates email/login shapes. - contributor-check.yml: attribution gate now accepts a mapping file OR a legacy entry; failure message prints the exact add_contributor command. Also auto-resolves bare <login>@users.noreply.github.com emails is intentionally NOT added (kept id+login form only, matching previous behavior). - contributor_audit.py: guidance now points at add_contributor.py. - tests/scripts/test_contributor_map.py: 12 tests covering loader, merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry counts as green but is loudly reported in a '⚠ FLAKY' summary section (with both attempts' output preserved) so the flake gets fixed instead of eating a full-run rerun. Deterministic failures fail both attempts — regressions cannot be laundered green. - --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables) - E2E verified: simulated first-run-fail flake goes green with banner; deterministic failure still exits 1; retries=0 restores old behavior. This converts the dominant CI failure mode (one timing-sensitive test flaking a 4600-test shard, requiring a manual 10-minute rerun and an agent triage loop) into a self-healing retry that costs one file's runtime.
These guard against catastrophic regex backtracking (seconds-to-minutes class), but 0.15s is within scheduler-stall noise on loaded shared CI runners — test_max_accepted_separator_free_input_is_fast failed a CI shard this week on runner load alone. 2.0s still catches the regression class with zero flake surface.
Reliability pass over every workflow: - timeout-minutes on all 21 jobs that lacked one (a hung job previously burned the 6-hour default runner budget) - ./.github/actions/retry wrapped around every network-fetching install that lacked it: pip installs (deploy-site, skills-index), npm ci (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker test deps). Deterministic build steps (npm run build) deliberately NOT retried — split into separate steps so a real build failure fails fast instead of retrying 3x.
…Dockerfile From the workflow reliability audit: - tests.yml: duration-cache restore had NO restore-keys while saves use run_id-suffixed keys — the cache never matched once, so LPT slicing always ran blind and unbalanced slices pushed heavy files toward the per-file timeout. One-line restore-keys fixes slice balancing. - Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed): 'gh pr view || true' turned an API blip into 'label absent' → false BLOCKING failure. Now 3x retry, and API failure is reported as an API failure instead of a missing label. - detect-changes action: compare API retried before failing open (was silently running all lanes on any blip). - uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried so registry blips don't read as 'lockfile stale'. - docker.yml merge job: imagetools create retried (Docker Hub eventual consistency on just-pushed digests). - Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to curl --retry 3 (ADD cannot retry; checksums still enforced); npm --fetch-retries=5; playwright chromium fetch retried 3x. - Advisory artifact uploads (per-slice durations, ci-timings report) get continue-on-error so an artifact-service blip can't fail a green test slice.
… env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
Root-cause fixes from the flake audit (session-DB mining + repo sweep): Event-based sync instead of sleep-sync: - title_generator: mock sets threading.Event, wait(10) replaces sleep(0.3) hoping the daemon thread got scheduled - docker zombie_reaping / profile_gateway: poll-for-state helpers replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async) - process_registry tree test: select()-bounded readline replaces an unbounded blocking read (parent wedge now fails THIS test with a clear message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s (the 1s partition window mid-interpreter-startup is how a child PID escaped the live-system guard in CI) Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors; all of these complete in ms-to-1s when healthy so the raises cost nothing on green runs): - subprocess/thread waits <= 2s raised to 10-15s across mcp_tool, mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe, mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt, voice_cli_integration, docker_environment, session_store_lock_io, planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output (joins now also assert not is_alive() so stragglers fail loudly) - wall-clock discrimination ceilings loosened where the guarded hang is 10x larger: local_background_child_hang 4s->10s, interrupt_cleanup setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup 5s->15s, protocol/gil-starvation fast-handler 0.5s->2s, iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s - narrow assertion windows widened: honcho first-turn wait 0.4..0.65 -> 0.25..2.0 (property is bounded-not-hung, not an exact wall-clock); compression fork-lock TTL 1s->3s (12 refresh chances per lease); compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0) - telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
The full 42k-test run and complete npm check surfaced three more classes: - Environment isolation: local ~/.honcho defaultHost and SSH_* variables leaked into Python/TUI tests. Pin the default Honcho host in the hermetic fixture, isolate the one fallback test from ~/.honcho, and blank SSH_* around terminalSetup tests. This flipped 20 false failures back to deterministic behavior on developer machines. - Background-thread sleep-sync: Honcho async writer tests patched time.sleep globally, then busy-polled with that same mocked sleep. Under full-suite load the poller could starve the writer. Each test now waits on an Event emitted by the exact flush/retry transition; 30/30 passed under 15-way contention. - Desktop streaming: the test slept 80ms and assumed a 500ms timer could not fire before its assertion. A loaded runner descheduled the test for >500ms and both chunks arrived. Producer controls now gate second-chunk and completion transitions explicitly. Also make file-retry observability complete: a self-healed flaky file now prints BOTH attempts' full output in the FLAKY summary. Two behavioral runner tests prove pass-on-retry is green+loud+traceback-preserving, while a deterministic failure remains red.
Code Review: retry patterns vs
|
| File | Step | Command |
|---|---|---|
docker.yml:129 |
Install Python deps | uv sync --locked --python 3.11 --extra dev |
upload_to_pypi.yml:58 |
Build web dashboard | npm ci (web) |
upload_to_pypi.yml:61 |
Build TUI bundle | npm ci (ui-tui) |
deploy-site.yml:67 |
Install PyYAML | pip install pyyaml==6.0.2 httpx==0.28.1 |
deploy-site.yml:154 |
Install deps | npm ci (website) |
skills-index.yml:31 |
Install deps | pip install httpx==0.28.1 pyyaml==6.0.2 |
These are the textbook fit: single command, retry on non-zero exit, fail hard. The PR correctly split the combined cd web && npm ci && npm run build into a retried npm ci step + a separate npm run build step — nice touch, build shouldn't be retried.
✅ Correct to keep as curl --retry flags (not composite-action candidates)
deploy-site.yml:47—curl -fsS --retry 3 --retry-delay 10(Vercel webhook POST)skills-index-freshness.yml:31—curl -fsSL --retry 3 --retry-delay 10(index probe)- Dockerfile —
curl -fsSL --retry 3(s6-overlay tarballs)
curl's native --retry is the right tool for single curl invocations — it retries on transient HTTP errors and connection failures without wrapping a whole step.
✅ Correct to keep as inline loops (cannot use composite action)
Each of these has a structural reason the composite action can't handle:
1. detect-changes/action.yml:65 — gh api compare call
- Captures output into
$CHANGEDshell variable consumed later in the same step - Failure mode is fail-open (warn + empty = all lanes run), not hard-fail. The composite action always exits 1 on final failure.
2. docker.yml:221 — docker buildx imagetools create
- Depends on
${tags[@]}and${args[@]}shell arrays built conditionally earlier in the samerun:script - Can't be a separate step — the arrays don't survive step boundaries
3. lint.yml:191 — gh pr view label fetch
- Captures output into
$LABELS, consumed inline bygrep -Fxqin the same step
4. supply-chain-audit.yml:250 — gh pr view label fetch
- Identical pattern to lint.yml
5. uv-lockfile-check.yml:76 — uv lock --check
- On final failure, writes a detailed remediation guide to
$GITHUB_STEP_SUMMARYthen exits 1 — the composite action can't do custom failure paths
6. upload_to_pypi.yml:135 — gh release view poll loop
- This is polling for existence (30 attempts, 10s apart), not retry-on-failure
- On timeout, sets
skip_sign=truein$GITHUB_ENVand continues — not a hard failure
⚠️ Minor observations
-
lint.ymlandsupply-chain-audit.ymllabel-fetch loops are copy-pasted. The retry logic is identical. If we ever want to DRY this, a small composite action like.github/actions/gh-pr-labels/that outputs the label list would work — but that's a follow-up, not a blocker. The current duplication is only ~12 lines × 2. -
Dockerfile playwright loop (line 136) uses a compact one-liner
for i in 1 2 3; do ... && break || { ... }; done— slightly harder to read than the multi-line pattern used elsewhere, but functionally equivalent and correct.
Verdict
No changes needed. The PR already converted every convertible retry to the composite action, and correctly left the rest as inline loops or curl flags where they belong. The retry action's design (run command → retry on non-zero → hard-fail) simply can't express output-capture, fail-open, custom-failure-path, or polling semantics, and all the remaining inline loops need one of those.
Reviewed by Hermes Agent (ethie)
✅ CI-sensitive file review passedThe |
refactor(ci): use retry action for PR label fetch the retry action now captures stdout as a step output, so it can serve double duty: retry + output capture for commands like 'gh pr view' whose result must be consumed by later steps. Retry action gains: - 'stdout' output (heredoc-delimited to preserve newlines) - tee to temp file so stdout still streams to the job log - step id 'retry' for output reference Both lint.yml and supply-chain-audit.yml now use the retry action directly with 'command: gh pr view ...' and read steps.<id>.outputs.stdout. ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth Replace secrets.GITHUB_TOKEN and github.token with secrets.AUTOFIX_BOT_PAT across all workflows and composite actions that use the gh CLI or GitHub API. The PAT has consistent permissions across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate limit sharing with the default token, and is already used by js-autofix.yml for the same reasons. 19 sites swapped across 9 files: - lint.yml (3): label fetch, comment post/edit, comment update - supply-chain-audit.yml (5): scan, critical comment, unbounded dep comment, label fetch, mcp-catalog comment - lockfile-diff.yml (1): PR comment post/update - skills-index-freshness.yml (1): issue creation on degraded probe - skills-index.yml (2): index build, trigger deploy workflow - upload_to_pypi.yml (2): release view poll, release upload - ci.yml (1): timings report - deploy-site.yml (2): skills index crawl - detect-changes/action.yml (1): compare API call
4789691 to
39a63f5
Compare
* upstream/main: (807 commits) fmt(js): `npm run fix` on merge (NousResearch#66527) fix(ci): make tests, workflows, and attribution reliable under load (NousResearch#66373) fix(mem0): migrate legacy OSS base URL aliases fix(moa): surface stale presets without retries fix(codex): harden final cache-key boundaries test(codex): cover overlength cache-scope headers fix: cap cache-scope headers at 64 chars to avoid Codex 400 error (NousResearch#66045) docs(codex): document live app-server display; AUTHOR_MAP entries feat(codex): stream live app-server events to TUI/desktop tool cards fmt(js): `npm run fix` on merge (NousResearch#66505) fix(honcho): delegate the config.yaml timeout read to load_config_readonly fix(honcho): resolve the timeout staleness check from honcho.json like the build path feat(dev-sandbox): add --from DIR to seed sandbox HERMES_HOME (NousResearch#66486) fix(desktop): stop button sends interrupt to wrong session + stale events re-arm busy (NousResearch#66485) fmt(js): `npm run fix` on merge (NousResearch#66465) fmt(js): `npm run fix` on merge (NousResearch#66460) fmt(js): `npm run fix` on merge (NousResearch#66457) refactor(desktop): derive working/attention session sets from $sessionStates fix(desktop): session-scope fast mode, surface profile ownership + pinned model override fix(streaming): make the single-writer fence best-effort so a missing guard can't crash a turn (NousResearch#66448) ...
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That PAT is empty on fork PRs (forks get no repo secrets), which broke every fork PR two ways: 1. detect-changes classified with the empty PAT -> the compare API failed all 3 retries -> the classifier failed open and force-enabled the ci_review lane on EVERY fork PR. 2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with the same empty PAT via a hard-failing retry step -> the job failed with no recovery a fork contributor could perform (they can't self-add the label; re-running can't fix it). Restores the pre-#66373 fork-safe behavior without reverting the commit's real improvements (job timeouts, per-file flake retry, network-install retries): - detect-changes + ci.yml: token falls back to the built-in read-only github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT (authoritative); on forks it uses github.token, which can read the public compare endpoint. (An input `default:` only applies on omission, not on an empty passed value — hence the explicit `|| github.token`.) - lint ci-review + supply-chain mcp-catalog gates: restore the inline `gh pr view ... || true` label read with the github.token fallback, dropping the hard-failing retry "Fetch PR labels" step. Graceful degrade to "label absent" on an API blip, same as before #66373. Same-repo enforcement is unchanged (byte-identical logic; the PAT is still used there). Fork PRs classify correctly and the gates read labels via the read-only token exactly as they did before the regression.
#66577) #66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That PAT is empty on fork PRs (forks get no repo secrets), which broke every fork PR two ways: 1. detect-changes classified with the empty PAT -> the compare API failed all 3 retries -> the classifier failed open and force-enabled the ci_review lane on EVERY fork PR. 2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with the same empty PAT via a hard-failing retry step -> the job failed with no recovery a fork contributor could perform (they can't self-add the label; re-running can't fix it). Restores the pre-#66373 fork-safe behavior without reverting the commit's real improvements (job timeouts, per-file flake retry, network-install retries): - detect-changes + ci.yml: token falls back to the built-in read-only github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT (authoritative); on forks it uses github.token, which can read the public compare endpoint. (An input `default:` only applies on omission, not on an empty passed value — hence the explicit `|| github.token`.) - lint ci-review + supply-chain mcp-catalog gates: restore the inline `gh pr view ... || true` label read with the github.token fallback, dropping the hard-failing retry "Fetch PR labels" step. Graceful degrade to "label absent" on an API blip, same as before #66373. Same-repo enforcement is unchanged (byte-identical logic; the PAT is still used there). Fork PRs classify correctly and the gates read labels via the read-only token exactly as they did before the regression.
NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and NousResearch#66577 restored the `|| github.token` fork fallback for detect-changes and the label gates -- but it missed the ci-timings "Collect timings and generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork PRs that PAT is empty, so timings_report.py hard-fails at expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must never redden the PR" soft-fail path. Every fork PR gets a red run from this advisory job (e.g. NousResearch#66573). - ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback to the timings step. github.token has `actions: read`, enough to read the run's job/step durations on forks. - timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run (TimingsUnavailable) instead of a hard ValueError, so this whole class of failure can never redden a PR again even if a future workflow drops the token. Still writes no JSON, so no empty baseline is ever cached.
…ousResearch#66373) * feat(attribution): conflict-free contributor mappings via contributors/emails/ directory The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet: every concurrent salvage PR appended entries to the same lines of the same file, so parallel PRs re-conflicted on every merge to main. New system: one file per email under contributors/emails/ — filename is the commit-author email, first non-comment line is the GitHub login. File additions never conflict, so any number of PRs can add mappings concurrently. - scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen) merged with the directory at import time (directory wins). All existing consumers (resolve_author, contributor_audit.py) unchanged. - scripts/add_contributor.py: idempotent CLI to add a mapping; refuses conflicting reassignments (incl. against the legacy map), validates email/login shapes. - contributor-check.yml: attribution gate now accepts a mapping file OR a legacy entry; failure message prints the exact add_contributor command. Also auto-resolves bare <login>@users.noreply.github.com emails is intentionally NOT added (kept id+login form only, matching previous behavior). - contributor_audit.py: guidance now points at add_contributor.py. - tests/scripts/test_contributor_map.py: 12 tests covering loader, merge precedence, CLI idempotency/conflict/validation, subprocess E2E. * feat(ci): one-shot per-file flake retry in the parallel test runner A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry counts as green but is loudly reported in a '⚠ FLAKY' summary section (with both attempts' output preserved) so the flake gets fixed instead of eating a full-run rerun. Deterministic failures fail both attempts — regressions cannot be laundered green. - --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables) - E2E verified: simulated first-run-fail flake goes green with banner; deterministic failure still exits 1; retries=0 restores old behavior. This converts the dominant CI failure mode (one timing-sensitive test flaking a 4600-test shard, requiring a manual 10-minute rerun and an agent triage loop) into a self-healing retry that costs one file's runtime. * test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s These guard against catastrophic regex backtracking (seconds-to-minutes class), but 0.15s is within scheduler-stall noise on loaded shared CI runners — test_max_accepted_separator_free_input_is_fast failed a CI shard this week on runner load alone. 2.0s still catches the regression class with zero flake surface. * fix(ci): job timeouts everywhere + retries on all network installs Reliability pass over every workflow: - timeout-minutes on all 21 jobs that lacked one (a hung job previously burned the 6-hour default runner budget) - ./.github/actions/retry wrapped around every network-fetching install that lacked it: pip installs (deploy-site, skills-index), npm ci (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker test deps). Deterministic build steps (npm run build) deliberately NOT retried — split into separate steps so a real build failure fails fast instead of retrying 3x. * docs(agents): document the file-retry flake policy * fix(ci): curl retries on deploy hook + skills-index probe * fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile From the workflow reliability audit: - tests.yml: duration-cache restore had NO restore-keys while saves use run_id-suffixed keys — the cache never matched once, so LPT slicing always ran blind and unbalanced slices pushed heavy files toward the per-file timeout. One-line restore-keys fixes slice balancing. - Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed): 'gh pr view || true' turned an API blip into 'label absent' → false BLOCKING failure. Now 3x retry, and API failure is reported as an API failure instead of a missing label. - detect-changes action: compare API retried before failing open (was silently running all lanes on any blip). - uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried so registry blips don't read as 'lockfile stale'. - docker.yml merge job: imagetools create retried (Docker Hub eventual consistency on just-pushed digests). - Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to curl --retry 3 (ADD cannot retry; checksums still enforced); npm --fetch-retries=5; playwright chromium fetch retried 3x. - Advisory artifact uploads (per-slice durations, ci-timings report) get continue-on-error so an artifact-service blip can't fail a green test slice. * fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list - test_tui_gateway_server.py: session.create / non-eager session.resume arm a 50ms threading.Timer (_schedule_agent_build) that outlives its test and fires into the NEXT test's _make_agent mock, racily corrupting captured state (the recurring session_resume shard failures). Replaced the per-test whack-a-mole stub with a module-wide autouse fixture; the 3 worker-lifecycle tests that genuinely need the deferred build opt back in via @pytest.mark.real_agent_prewarm (new marker in pyproject). - test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the live PROVIDER_REGISTRY instead of a hand-list that had drifted (missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto') tests failed on any machine with HF_TOKEN exported. E2E-verified with HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass. * test: de-flake 30 timing-sensitive test files for loaded CI runners Root-cause fixes from the flake audit (session-DB mining + repo sweep): Event-based sync instead of sleep-sync: - title_generator: mock sets threading.Event, wait(10) replaces sleep(0.3) hoping the daemon thread got scheduled - docker zombie_reaping / profile_gateway: poll-for-state helpers replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async) - process_registry tree test: select()-bounded readline replaces an unbounded blocking read (parent wedge now fails THIS test with a clear message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s (the 1s partition window mid-interpreter-startup is how a child PID escaped the live-system guard in CI) Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors; all of these complete in ms-to-1s when healthy so the raises cost nothing on green runs): - subprocess/thread waits <= 2s raised to 10-15s across mcp_tool, mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe, mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt, voice_cli_integration, docker_environment, session_store_lock_io, planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output (joins now also assert not is_alive() so stragglers fail loudly) - wall-clock discrimination ceilings loosened where the guarded hang is 10x larger: local_background_child_hang 4s->10s, interrupt_cleanup setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup 5s->15s, protocol/gil-starvation fast-handler 0.5s->2s, iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s - narrow assertion windows widened: honcho first-turn wait 0.4..0.65 -> 0.25..2.0 (property is bounded-not-hung, not an exact wall-clock); compression fork-lock TTL 1s->3s (12 refresh chances per lease); compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0) - telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under) * fix(tests): repair indentation from de-flake batch edit * fix(tests): harden env isolation and replace remaining sleep-sync races The full 42k-test run and complete npm check surfaced three more classes: - Environment isolation: local ~/.honcho defaultHost and SSH_* variables leaked into Python/TUI tests. Pin the default Honcho host in the hermetic fixture, isolate the one fallback test from ~/.honcho, and blank SSH_* around terminalSetup tests. This flipped 20 false failures back to deterministic behavior on developer machines. - Background-thread sleep-sync: Honcho async writer tests patched time.sleep globally, then busy-polled with that same mocked sleep. Under full-suite load the poller could starve the writer. Each test now waits on an Event emitted by the exact flush/retry transition; 30/30 passed under 15-way contention. - Desktop streaming: the test slept 80ms and assumed a 500ms timer could not fire before its assertion. A loaded runner descheduled the test for >500ms and both chunks arrived. Producer controls now gate second-chunk and completion transitions explicitly. Also make file-retry observability complete: a self-healed flaky file now prints BOTH attempts' full output in the FLAKY summary. Two behavioral runner tests prove pass-on-retry is green+loud+traceback-preserving, while a deterministic failure remains red. * refactor(ci): use gh bot pat, better retries refactor(ci): use retry action for PR label fetch the retry action now captures stdout as a step output, so it can serve double duty: retry + output capture for commands like 'gh pr view' whose result must be consumed by later steps. Retry action gains: - 'stdout' output (heredoc-delimited to preserve newlines) - tee to temp file so stdout still streams to the job log - step id 'retry' for output reference Both lint.yml and supply-chain-audit.yml now use the retry action directly with 'command: gh pr view ...' and read steps.<id>.outputs.stdout. ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth Replace secrets.GITHUB_TOKEN and github.token with secrets.AUTOFIX_BOT_PAT across all workflows and composite actions that use the gh CLI or GitHub API. The PAT has consistent permissions across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate limit sharing with the default token, and is already used by js-autofix.yml for the same reasons. 19 sites swapped across 9 files: - lint.yml (3): label fetch, comment post/edit, comment update - supply-chain-audit.yml (5): scan, critical comment, unbounded dep comment, label fetch, mcp-catalog comment - lockfile-diff.yml (1): PR comment post/update - skills-index-freshness.yml (1): issue creation on degraded probe - skills-index.yml (2): index build, trigger deploy workflow - upload_to_pypi.yml (2): release view poll, release upload - ci.yml (1): timings report - deploy-site.yml (2): skills index crawl - detect-changes/action.yml (1): compare API call --------- Co-authored-by: ethernet <arilotter@gmail.com>
…esearch#66373 (NousResearch#66577) NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That PAT is empty on fork PRs (forks get no repo secrets), which broke every fork PR two ways: 1. detect-changes classified with the empty PAT -> the compare API failed all 3 retries -> the classifier failed open and force-enabled the ci_review lane on EVERY fork PR. 2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with the same empty PAT via a hard-failing retry step -> the job failed with no recovery a fork contributor could perform (they can't self-add the label; re-running can't fix it). Restores the pre-NousResearch#66373 fork-safe behavior without reverting the commit's real improvements (job timeouts, per-file flake retry, network-install retries): - detect-changes + ci.yml: token falls back to the built-in read-only github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT (authoritative); on forks it uses github.token, which can read the public compare endpoint. (An input `default:` only applies on omission, not on an empty passed value — hence the explicit `|| github.token`.) - lint ci-review + supply-chain mcp-catalog gates: restore the inline `gh pr view ... || true` label read with the github.token fallback, dropping the hard-failing retry "Fetch PR labels" step. Graceful degrade to "label absent" on an API blip, same as before NousResearch#66373. Same-repo enforcement is unchanged (byte-identical logic; the PAT is still used there). Fork PRs classify correctly and the gates read labels via the read-only token exactly as they did before the regression.
NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and NousResearch#66577 restored the `|| github.token` fork fallback for detect-changes and the label gates -- but it missed the ci-timings "Collect timings and generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork PRs that PAT is empty, so timings_report.py hard-fails at expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must never redden the PR" soft-fail path. Every fork PR gets a red run from this advisory job (e.g. NousResearch#66573). - ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback to the timings step. github.token has `actions: read`, enough to read the run's job/step durations on forks. - timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run (TimingsUnavailable) instead of a hard ValueError, so this whole class of failure can never redden a PR again even if a future workflow drops the token. Still writes no JSON, so no empty baseline is ever cached.
* feat(cron): add truthful execution ledger
* fix(cron): harden execution attempt ledger
* test(file-safety): unbreak session-snapshot suite; de-flake fixture to env-var resolution (#66293)
Two changes to tests/agent/test_file_safety_session_state.py:
1. Drop the stale monkeypatch on tools.file_tools._get_live_tracking_cwd
— the helper was deleted in the cwd-tracking refactor (c80b244b5),
and monkeypatch.setattr on a missing attribute raises AttributeError,
breaking CI slice 4/8 on main for every PR. The patch was redundant:
the test writes an absolute path, so cwd resolution never engages.
2. Make the fixture stale-proof: instead of monkeypatching the private
_hermes_home_path/_hermes_root_path helpers (same failure class if
they're ever renamed), set HERMES_HOME to <root>/profiles/work and
let the real resolution chain (get_hermes_home /
get_default_hermes_root's profiles-parent rule) derive both paths.
The fixture now references zero private symbols and exercises the
production resolution path.
* fix(gateway): retry transcript appends
Queue failed session DB appends so disk order cannot silently lag memory.\nRebuild corrupt FTS indexes once and surface repeated failures as warnings.
* fix: harden transcript append retry — lock, matcher, encapsulation, cap
Follow-up fixes for salvaged PR #65637:
1. Clear _dirty_transcripts in rewrite_transcript + rewind_session —
stale pending messages were re-inserted after /retry, /undo, /compress.
2. Narrow _is_fts_corruption_error to specific SQLite error strings —
bare 'fts' substring matched 'shifts', 'gifts', etc.
3. Move DB write outside _transcript_retry_lock — holding the lock
during writes serialized all sessions' transcript appends and blocked
during FTS rebuild. Now the lock guards only the pending queue.
4. Push rebuild_fts() into SessionDB — SessionStore was reaching into
_conn/_lock private attrs. SessionDB.rebuild_fts() follows the same
pattern as optimize_fts().
5. Cap pending per session at 200 — prevents unbounded memory growth
when DB is persistently broken. Oldest messages dropped with warning.
Added 4 new tests: dirty-clear on rewrite/rewind, FTS matcher false
positives, pending cap enforcement.
* fix(agent): execute valid tool calls in mixed batches with invalid names (#66317)
Degrading models (observed with gpt-5.6 past ~350K input) emit tool-call
batches like 6 valid named calls + 1 blank-name call. Previously the
whole turn was voided — every valid call got 'Skipped: another tool call
in this turn used an invalid name' — and three such batches tripped the
3-strike stop, killing sessions that were still making progress.
Now a mixed batch error-results ONLY the invalid call(s) (terse
anti-priming error for blank names per #47967, catalog dump for typos)
and dispatches the valid subset for execution. The assistant message
keeps every emitted call so provider-side tool_call/result pairing stays
intact. The 3-strike counter only advances when a turn contains NO valid
call, so a fully-degenerate model still stops while a mostly-coherent
one keeps working. Broken JSON args on a never-executing invalid call no
longer trigger the whole-turn JSON retry loop.
Field evidence: July 2026 debug bundle showed gpt-5.6-sol emitting
6-call batches with one blank-name rider at 559K/384K-token context in
two separate sessions; 13 valid tool calls were discarded before the
session stopped as partial.
* fix(cli): arm exit watchdog on shutdown signal, not at chat startup (#66278)
A hermes --tui session whose main thread wedges before app.run() returns
never executes the finally that calls _run_cleanup — the only place the
exit watchdog was armed — so a dead CLI lingered indefinitely (observed
~47 min at 4% CPU, the #65998 class).
Arm the backstop from the SIGTERM/SIGHUP handlers instead (both the
interactive and single-query paths), the earliest moment shutdown intent
is unambiguous. The signal-armed leash is 2x HERMES_EXIT_WATCHDOG_S so a
slow-but-progressing _run_cleanup (which still arms its own tighter timer)
is never cut short; the outer timer only wins when cleanup was never
reached. Idempotent across repeated signals; never raises from a handler.
Deliberately NOT armed at startup: the watchdog thread calls os._exit(0)
unconditionally after its sleep, so a startup-armed timer (the #65998
approach) would hard-kill every session that outlives the timeout.
Supersedes #65998; thanks @JeffStone69 for the report and root-cause gap
analysis.
* fix(streaming): fence superseded streams out of the delta sink (single-writer)
When the stale-stream detector reconnects past a stream whose socket abort
raced (the close never actually stopped the old worker), the superseded stream
and the retry's stream both write deltas into the same turn. The persisted
transcript is then two coherent responses interleaved token-by-token —
de-interleaving the stored text by alternation yields two complete, independent
answers to the same prompt, which is a dual-writer race in the harness, not a
model/context failure (#65991).
The interrupt path already positively cancels before force-closing (#6600), but
the stale-kill path relies only on the socket abort, and nothing fenced late
chunks from a superseded stream out of the shared delta sink.
Enforce a single-writer invariant on the sink itself, guarded by attempt id
rather than only socket state: every streaming attempt (chat_completions,
anthropic_messages, and bedrock paths) claims a monotonic writer token before
it begins consuming its stream. A newer claim supersedes any older one, so the
consume loop bails the instant it is superseded and _fire_stream_delta /
_fire_reasoning_delta / _record_streamed_assistant_text drop chunks from a
stale writer. The token is stored per-thread, so a thread that never claimed
(a non-streaming delta caller) is never fenced — the guard can only ever drop a
superseded stream, never the single legitimate writer. Discards are counted and
logged sparsely so a real provider problem stays visible instead of being
silently swallowed.
* test(streaming): cover the single-writer invariant for superseded streams
Assert that a superseded stream (older writer token, other thread) is fenced
from the delta sink, the active writer is never fenced, a non-claiming thread
is never treated as a writer, and the real consume loop stops the instant it is
superseded — so two streams can never interleave into one turn (#65991).
* fix(codex): claim the stream-writer token on the codex_responses path too
Widen the #65991 single-writer fence to run_codex_stream: each codex
attempt claims the delta sink before consuming events, and the consume
loop's interrupt_check now also stops the instant a newer attempt
supersedes this one. Parity with the chat_completions / anthropic /
bedrock paths from the salvaged fix.
Two regression tests: superseded codex stream is fenced mid-stream;
sole-writer codex stream delivers unchanged.
* fix(compression): affirm tool use stays active in the compaction handoff prefix (#66291)
The REFERENCE ONLY framing ('treat as background reference, NOT as active
instructions... Do NOT answer questions or fulfill requests') was observed
bleeding into general tool-use suppression: a production session went
narration-only for 7 consecutive turns immediately after a compression
event, describing edits instead of calling tools (#65848 report).
Fix is additive: one clause stating the note does not restrict HOW the
agent works — tools remain fully active for the active task. Every
anti-resumption protection stays intact; the previous prefix generation
is frozen into _HISTORICAL_SUMMARY_PREFIXES per the module contract so
persisted summaries still get the directive-strip on re-compaction.
The #65848 rewrite was not taken: dropping the 'Do NOT answer questions'
line and the four-heading discard directive risks re-opening the
stale-task-resumption class those clauses exist to prevent (the carveout
era regressions #41607/#38364/#42812 documented in this file).
Report and root-cause analysis: @yasserbousrih (#65848).
* fix(codex): forward drained notifications to on_event during approval roundtrips
The approval-drain loop in CodexAppServerSession.run_turn drains up to 8
pending notifications to keep per-turn state current before answering a
server-initiated approval request — but never forwarded them to the
on_event display hook. Tool bubbles for items drained alongside an
approval (e.g. the item/started for the very command awaiting approval)
silently disappeared.
Mirror the main notification path's on_event invocation in the drain
loop. Regression test demonstrates RED→GREEN.
Grafted from PR #26541 by @simpolism — the earliest submission of the
codex app-server display-bridge family (May 15). Confirmed independently
by #64698 and #65412.
* feat(codex): webSearch bubbles + bare hermes-tools names in app-server bridge
Two more display gaps from #26541 grafted onto the merged bridge:
- webSearch: codex's built-in web search now produces a tool.started/
tool.completed bubble pair (query as preview + args). Previously the
item type wasn't in _CODEX_TOOL_ITEM_TYPES, so built-in searches
showed nothing.
- mcp.hermes-tools.* stripping: tools codex invokes through Hermes' own
hermes-tools MCP server display as their bare names (web_search,
browser_navigate) instead of mcp.hermes-tools.web_search. The inner
dispatch subprocess can't fire native progress events, so the
codex-level event is the display event — name it the way users know
the tool.
Credit: both behaviors designed and first implemented by @simpolism in
PR #26541 (May 15, earliest of the app-server display-bridge family).
* fix(state): heal alternation at the ACP / CLI-resume / TUI-resume restore sites too
Follow-up to the restore-boundary alternation heal (#65492): get_messages_
as_conversation grew a repair_alternation flag, wired into gateway
load_transcript and the CLI startup resume. Three other LIVE-REPLAY
restore sites still loaded the transcript verbatim, so a durable
'user;user' violation there re-fires the pre-request defensive repair on
every request for the rest of the session (it only ever mutates the
per-request list, never the restored working conversation):
- acp_adapter/session.py::SessionManager._restore — the loaded history
becomes the resumed ACP (Zed) agent's SessionState.history.
- hermes_cli/cli_commands_mixin.py — the /resume slash command sets
self.conversation_history from the load (the startup resume was fixed,
this mid-session one was missed).
- tui_gateway/server.py — the resume handler feeds the load into the
deferred session record's working conversation.
Pass repair_alternation=True at all three so the wedge is healed once at
restore. Inspection/export consumers (trace upload, context guard,
api_server history, display_history) keep the verbatim default.
Adds an end-to-end regression test driving the ACP _restore path: a
seeded user;user session restores to an alternation-clean live history
with no user input lost.
* test(tui_gateway): accept repair_alternation in resume-path DB doubles
The lazy session.resume path now calls
db.get_messages_as_conversation(target, repair_alternation=True), but the
fake _DB stubs in test_protocol.py still declared the pre-change signature,
so the resume raised "unexpected keyword argument 'repair_alternation'"
and the three session_resume_lazy tests failed.
Mirror the real get_messages_as_conversation signature in the stubs by
accepting (and ignoring) repair_alternation.
* test(delegate): assert copilot probe with assert_any_call to de-flake under slicing
test_build_child_agent_ignores_acp_command_when_binary_missing patches
shutil.which globally and asserted the LAST call was which("copilot").
That is order-dependent: an unrelated which("uv") reached later in the same
process (which happens under some CI test-slice orderings) becomes the last
call, so assert_called_with("copilot") fails even though the copilot binary
was probed exactly as intended. Switch to assert_any_call("copilot"), which
verifies the actual intent and is robust to unrelated which() calls. The
behavioural assertions (provider, acp_command, acp_args) are unchanged.
* fix(tui): heal alternation at the remaining live-replay resume sites
Sibling-site audit on top of #65672: the interactive TUI resume, the
profile-scoped resume, and the /undo history reload also feed LIVE
REPLAY (raw_history -> sanitize_replay_history -> working conversation;
session['history'] after rewind). Pass repair_alternation=True on the
model-fed copies; display_history stays verbatim so inspection/export
show what is actually stored. Display-only consumers (session.history
RPC, formatted transcript output) intentionally unchanged.
* test(tui): accept repair_alternation in the top-level server test doubles too
The widened resume sites pass repair_alternation=True; the DB doubles in
tests/test_tui_gateway_server.py (separate from tests/tui_gateway/) needed
the same signature update as Frowtek's originals.
* fix(terminal): fall back when the configured cwd is unenterable, not just missing (#66306)
A root-launched CLI session can leak /root into the terminal cwd state a
non-root gateway/cron process later resolves (#65583). os.path.isdir('/root')
is True for a non-root user — stat only needs search permission on / — so
_resolve_safe_cwd returned it and subprocess.Popen(cwd='/root') died with
PermissionError: [Errno 13], failing EVERY cron job's terminal/file/search
tool on every command until restart.
_resolve_safe_cwd now requires X_OK (new _cwd_usable helper) and climbs to
the nearest enterable ancestor, logging a WARNING that names the leak class
when an existing-but-denied cwd is skipped. Missing-cwd recovery (#17558)
behavior unchanged.
E2E-verified: LocalEnvironment constructed with an unenterable cwd now runs
commands from the fallback directory instead of raising.
* perf(desktop): pre-warm profile pool backends on hover intent
A cold profile switch pays the full pool-backend spawn — Python boot,
port announcement, readiness probe, token adoption — before the
profile's gateway can even open. Measured with the new CDP harness
(scripts/measure-profile-switch.mjs, same family as
profile-session-switch.mjs): click → WS open is ~2.5-2.9s on a cold
profile, ~3-3.6s to a settled sidebar; a warm profile settles in
~0.5-0.8s. The pointer entering a profile square telegraphs the switch
hundreds of ms before the click lands, so start the spawn then.
- store/profile: prewarmProfileBackend(name) — fires the existing
hermesDesktop.getConnection IPC, which is idempotent (ensureBackend
returns the pooled connectionPromise), so the real switch joins the
in-flight spawn instead of starting it. Skips the active gateway
profile, throttles per profile (60s) so drive-by hovers can't spam
spawn attempts, and swallows failures — error UX belongs to the real
switch. No new IPC surface; the pool's existing LRU cap + idle reaper
still bound resource use, and the LRU guard never evicts a
keepalive-fresh backend for a hover spawn.
- sidebar/use-profile-prewarm: pointerenter/pointerleave handlers with
a 120ms dwell so sweeping the pointer across the rail or a
mixed-profile session list doesn't spawn a backend per element
crossed.
- Wired at the three switch surfaces: rail ProfileSquare, the condensed
ProfileDropdown items (extracted ProfileDropdownItem so each row owns
its dwell timer), and SidebarSessionRow (covers cross-profile resumes
from the all-profiles view; same-profile rows no-op inside the guard).
Measured E2E over CDP: synthetic hover on a cold profile square spawns
its backend in the background; the subsequent click settles in ~519ms
vs ~3.0-3.6s unhovered — and any hover shorter than the spawn still
shaves its dwell off the click's wait.
Verification: apps/desktop `npx tsc --noEmit` clean; full
`npx vitest run` 212 files / 1777 passed (new prewarm guard/throttle
tests in store/profile.test.ts); eslint + prettier clean.
* feat(kanban): modal create-task dialog, editable board project directory, comment workflow hint (#66333)
Community feedback (@LSanapalli on X): the inline task-creation form is
cramped inside a ~280px column with no way to resize; board-level
workspace defaults can't be changed after board creation; and users
believe they must block a task, comment, then unblock just to talk to
a worker.
- Create-task dialog: replace the inline column form with a centered
modal (reuses hermes-kanban-dialog chrome, 36rem wide) with labeled
fields for title, assignee, priority, skills, workspace kind/path,
goal mode, and parent task. Same request shape; Enter/Escape behavior
preserved; submit disabled until a title is present.
- Board settings dialog: new Settings button in the board switcher opens
a modal to edit display name, description, and the board-level default
project directory (default_workdir). PATCH /boards/:slug now accepts
default_workdir (validated absolute existing dir; empty string clears;
omitted leaves unchanged) and returns the recomputed
default_workspace_kind so task-creation defaults follow immediately.
- Comment workflow hint: the task drawer's comment box now explains that
comments land on the thread immediately and reach the worker on its
next run/kanban_show() — no block/unblock dance needed — with a fuller
tooltip for when blocking IS the right tool.
- i18n: new keys optional in the kanban namespace with English fallbacks
in the bundle (established pattern; avoids churning 17 locale files).
- Docs: dashboard section updated for the dialog + Settings button.
* fmt(js): `npm run fix` on merge (#66348)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(state): self-heal FTS corruption on the SessionDB write path (#66296)
Complements the #65637 salvage (53d358838 + a9cc17fd8): the gateway
session store now retries transcript appends through its own queue, but
cron and CLI writers call SessionDB directly — a corrupt FTS index still
hard-failed their appends until the next process restart triggered the
offline repair.
_execute_write now detects the FTS-corruption error class (both the
generic 'database disk image is malformed' and newer SQLite's
'fts5: corrupt structure record' variant), performs a one-shot in-place
rebuild by delegating to the existing rebuild_fts(), and retries the
failed write. One-shot per instance so an unrecoverable database cannot
loop; lock/busy jitter-retry path untouched.
E2E-verified: corrupted messages_fts_data rejects appends; with this fix
the same append self-heals, persists, and FTS search works again.
* fix(auxiliary): isolate runtime cache by live context
* fix(auxiliary): sync runtime after fallback restoration
* fix(compression): reset failure cooldown on runtime switch
* fix(tui): route images with the live switched model
* chore(release): map auxiliary runtime contributors
* fix(auxiliary): scope runtime state to each turn
* test(compression): expect complete runtime tuple
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm (contributes to #62698) (#66338)
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm
Credential selection runs on a hot path (every model call plus auxiliary
tasks), so an empty/exhausted pool logged "no available entries" at INFO on
*every* selection. On Windows, where multiple Hermes processes share one
rotating log guarded by concurrent-log-handler's cross-process lock, that
per-selection volume storms the lock (RuntimeError: Cannot acquire lock after
20 attempts), pegs a core, and stalls the asyncio event loop long enough that
the Desktop backend readiness probe times out ("Timed out connecting to Hermes
backend after 15000ms") even though the backend already announced
HERMES_BACKEND_READY.
Log the condition at most once per 60s window, re-arming on a successful
selection so recovery->re-exhaustion still surfaces promptly. Same fix class as
the warn-once dedup in #58265.
* test(credential-pool): cover no-available-entries log throttle
Assert the empty-pool INFO line logs at most once per throttle window, logs
again after the window elapses, and re-arms on a successful selection so a
recover->re-exhaust transition surfaces promptly. Uses a deterministic fake
monotonic clock (no sleeps, no network).
* perf(desktop): pre-warm opens the gateway socket too, not just the spawn
Answering the review question on the PR table — why a hovered-cold
switch still showed ~440ms click → WS open: getConnection-only
pre-warming left the WS connect chain to the click, and its microtask
continuation can only run after the click's fresh-draft React flush
(unmounting a large open transcript costs ~300-400ms of render work),
so the socket didn't even START connecting until the flush finished.
Add openGatewayForProfile: the same spawn + connect chain as a real
switch, minus activation — so the hover leaves the profile's socket
fully OPEN and the click's ensureGatewayForProfile just activates it
(no ws:new after the click at all; measured ws open at hover+136ms on
a warm backend). No scheduleReconnect on failure: a hover is
speculative, so a dead backend must not start a background retry loop
— the real switch owns retry and error UX. Pruning semantics are
unchanged: a hover-opened socket for an idle profile is dropped by the
next pruneSecondaryGateways recompute, which just returns the click to
the previous behavior.
Tests updated: pre-warm asserts openGatewayForProfile is called and
that activation (ensureGatewayForProfile) is NOT.
* feat(desktop): promote Fireworks AI to #2 in onboarding provider picker (#66432)
Mirror CANONICAL_PROVIDERS so Fireworks sits directly under Nous Portal
(always visible) ahead of OpenRouter across onboarding, Settings → Providers,
and the API-key catalog.
* fmt(js): `npm run fix` on merge (#66445)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* ci: add 2 minute timeout to osv scan (#66410)
this one ran for 5 hours lol
https://github.com/NousResearch/hermes-agent/actions/runs/29578577080/job/87878711479
* fix(streaming): make the single-writer fence best-effort so a missing guard can't crash a turn (#66448)
A cron job ("Daily Buzz Report") died with 'AIAgent' object has no
attribute '_claim_stream_writer'. The #65991 single-writer fence lives on
AIAgent (run_agent.py), but the streaming paths that use it live in other
modules — chat_completion_helpers (chat / anthropic / bedrock) and
codex_runtime (codex responses) — and called it directly as
agent._claim_stream_writer() / agent._stream_writer_is_current(). That makes
those modules hard-depend on the method being present on whatever object is
passed as agent.
The fence is an *additive* safety net that may only ever drop a provably
superseded stream, never the sole legitimate writer. But the direct calls
turned any agent that doesn't expose it — a version-skewed checkout (the
streaming helper module newer than run_agent), a hot-reloaded gateway mid
git-pull, a duck-typed agent, or a test double — into a fatal AttributeError
that aborts the whole turn (and, on cron, fails the job).
Route every cross-module claim/check through agent/stream_single_writer.py.
claim_stream_writer(agent) returns 0 when the fence is unavailable (or
raises), and stream_writer_is_current(agent, token) treats a 0 token or an
absent guard as "current" — so a guard-less agent degrades to "no fence"
instead of crashing, while a real AIAgent keeps the full single-writer
protection. Internal self.* uses inside run_agent are unchanged (self is
always a full AIAgent there).
* fix(desktop): session-scope fast mode, surface profile ownership + pinned model override
Model-picker audit follow-through — closes the remaining pieces of the
"switch one session, switches everywhere / can't tell whose session this
is" report class:
- tui_gateway: `config.set key=fast` with a session no longer writes the
global agent.service_tier to config.yaml (sibling of the earlier
`reasoning` scoping fix). It pins create_service_tier_override
("priority" / "" for explicit normal) so lazy builds and rebuilds keep
the choice; the desktop's per-model presets were rewriting the global
tier on every model pick. Fast-support validation now checks a draft's
picked model, and `config.get key=fast` reads the pre-build pin.
- desktop: owning-profile tag (initial chip + tooltip/aria label) on
pinned rows and search results in the All-profiles sidebar, and on the
chat header once a second profile exists (#66003).
- desktop: composer model pill shows a pin dot + tooltip when a manual
sticky pick is overriding the Settings default for new chats (#62055).
Closes #66003. Addresses #62055.
* refactor(desktop): derive working/attention session sets from $sessionStates
$workingSessionIds and $attentionSessionIds were independently maintained
atoms that updateSessionState had to manually keep in sync with the session
cache (paired setSessionWorking/setSessionAttention calls, plus a rotation
special-case in ensureSessionState). Make them computed() projections of
$sessionStates instead, so the data flow is one-directional:
gateway event → cache → $sessionStates → computed views.
Transition side-effects (watchdog arm/disarm, settle grace, unread marker,
compression id rotation signal) move into handleTransition, fired from
publishSessionState by diffing previous vs next — one choke point instead
of per-callsite bookkeeping. The watchdog's force-clear reaches the cache
through setWatchdogClearFn rather than a listener set.
Also:
- clearAllSessionStates disarms all watchdog timers and drops settle-grace
entries so a gateway switch can't leak stale timers or keep-set rows
- dropSessionState disarms the dropped runtime's watchdog timer
- watchdog tests now exercise the real timer→callback wiring instead of
manually simulating the clear
* fmt(js): `npm run fix` on merge (#66457)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fmt(js): `npm run fix` on merge (#66460)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fmt(js): `npm run fix` on merge (#66465)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* perf: fast model picker + dialogs — config-load hot path, model.options off the reader thread, off-screen turns skip rendering
Third profiling round (after #66033 / #66347), targeting the composer
model picker and dialog opens (worktree dialog etc.), measured over CDP
on real 1000+-message sessions.
Backend — model.options took 4.8s cold / 1.8s warm per call, and the
desktop model pill/picker blocks on it every open:
- agent/credential_pool: _load_config_safe uses load_config_readonly().
Every consumer only reads, and the per-call deepcopy was the dominant
cost — list_authenticated_providers calls load_pool() per provider
row, and each load_pool loaded (and deep-copied) the full config
again via get_pool_strategy.
- hermes_cli/config: memoize ensure_hermes_home() per home path. It
runs inside the config lock on EVERY load_config(), paying ~14
mkdir/chmod syscalls per call. The fast path still re-checks that the
home dir exists, so a deleted home is recreated as before; profile
switches hit the new path and re-run. Tests cover both.
- tui_gateway/server: add model.options to _LONG_HANDLERS. It measured
seconds inline on the WS reader thread — while it ran, prompt.submit
and session.interrupt sat unread (same class as #21123).
Together: model.options RPC 4825/1842ms → 426/230ms (measured on the
live desktop backend); build_models_payload in isolation 6.2s → 0.97s
cold, 0.27s warm.
Desktop — every Radix dialog/popover open forced a whole-document style
recalc (Presence reads getComputedStyle on mount), which on a
1300-message transcript cost ~650-730ms per open (CPU profile:
getAnimationName 483ms self). The worktree dialog (⌘⇧B) paid it on
every single open:
- thread/list: content-visibility:auto + contain-intrinsic-size on the
per-turn group wrappers. Off-screen turns now skip style recalc,
layout, and paint entirely; never-rendered turns hold a placeholder
height (auto: remembered real size once rendered) so scrollbar and
anchoring stay stable. Verified over CDP: worktree dialog open 656-
730ms → ~200ms on the same session; stick-to-bottom pin, scroll-to-
top rendering, and sticky human bubbles all intact.
Also: profile-session-switch harness accepts CDP_HTTP (Chrome tends to
squat on 9222).
Verification:
- scripts/run_tests.sh: config, credential-pool, inventory,
model-switch routing, tui_gateway protocol, profiles suites green
(test_profiles has one pre-existing failure on main, unrelated);
new tests for the ensure_hermes_home memo.
- apps/desktop: tsc clean, eslint/prettier clean, thread + session
suites green (326 tests).
- E2E over CDP on the live app: numbers above, plus scroll/pin sanity.
* fix(desktop): stop button sends interrupt to wrong session + stale events re-arm busy (#66485)
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* feat(dev-sandbox): add --from DIR to seed sandbox HERMES_HOME (#66486)
Adds a --from DIR flag to scripts/dev-sandbox.sh that copies an existing
HERMES_HOME directory into the sandbox as the starting point before the
command runs. Lets you spin up a sandbox pre-populated with your real
config, sessions, skills, etc.
scripts/dev-sandbox.sh --from ~/.hermes hermes desktop
Design:
- cp -a dir/. dest/ — preserves perms, symlinks, hidden files
- Clobber guard: only seeds when sandbox HERMES_HOME is empty, so
re-running --persistent doesn't blow away existing sandbox state
- Validates: errors on nonexistent dir, missing arg, flag-like arg,
empty --from=
- Supports both --from DIR and --from=DIR forms
- Backwards compatible: no --from = unchanged behavior
* fix(honcho): resolve the timeout staleness check from honcho.json like the build path
The staleness check added in #66052 resolved the timeout from env,
config.yaml, and the default only, while the build path also reads the
honcho.json host block (timeout/requestTimeout). With a timeout
configured in honcho.json, the two permanently disagreed: every
no-config get_honcho_client() call — i.e. every HonchoSessionManager
.honcho property access — interpreted the mismatch as a config change
and tore down and rebuilt the client, defeating the singleton on the
hot path it was meant to protect.
Teach the check to read honcho.json through the same host-aware chain
as from_global_config, memoized on the file's mtime_ns so the per-call
cost stays one stat(). A genuine honcho.json timeout change is now also
detected, extending #57437 to that config surface.
* fix(honcho): delegate the config.yaml timeout read to load_config_readonly
The staleness check's bespoke mtime memo keyed only on the user
config.yaml, but load_config() merges the managed-scope config
(HERMES_MANAGED_DIR/config.yaml, /etc/hermes) whose leaf keys win. A
managed honcho.timeout with no user config.yaml made the memo cache
'no timeout' while _build resolved the managed value — the same
perpetual-rebuild mismatch this PR fixes for honcho.json. A managed
timeout edit was likewise invisible while the user file's mtime stayed
put.
load_config_readonly() is already cached on both files' signatures plus
the env-ref snapshot, so use it instead of duplicating that
invalidation logic; the defensive deepcopy the old memo existed to
avoid is skipped by the readonly variant. Drive the rebuild test
through a real config.yaml and add a HERMES_MANAGED_DIR regression
test covering stable reuse and managed-timeout edits.
* fmt(js): `npm run fix` on merge (#66505)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* feat(codex): stream live app-server events to TUI/desktop tool cards
Extends the app-server event bridge (make_codex_app_server_event_bridge)
to fire the authoritative stable-ID tool_start_callback /
tool_complete_callback alongside the existing tool_progress_callback,
and route item/reasoning/summaryDelta through the reasoning channel.
Surfaces that render structured tool cards (TUI, desktop) — not just
progress bubbles — now correlate live cards with the projected history
entry after a resume, because the call ids mirror CodexEventProjector's
_deterministic_call_id. Guarded per-callback so a broken display
consumer can't tear down the codex turn loop.
Grafted from PR #65412 by @HaiderSultanArc onto the merged bridge (the
PR's parallel _codex_live_event implementation was reconciled into the
bridge's existing _fire_tool_started/_fire_tool_completed helpers).
* docs(codex): document live app-server display; AUTHOR_MAP entries
- codex-app-server-runtime.md: add a Live display section covering the
stream/reasoning/tool-card bridge and show_commentary gating.
- release.py: AUTHOR_MAP entries for HaiderSultanArc, jjadeo-oss, juanfradb
(the latter two for forthcoming follow-up salvages of #62396 / #18050).
* fix: cap cache-scope headers at 64 chars to avoid Codex 400 error (#66045)
* test(codex): cover overlength cache-scope headers
Exercise the real transport path for long session ids, including stable hashing and bounded body/header cache keys.
* fix(codex): harden final cache-key boundaries
Fold #62349's broader provider-boundary handling into the header fix: bound top-level and xAI override keys again at preflight after middleware, preserve unrelated headers, and cover boundaries and collisions.
Co-authored-by: Nick Taylor <nicktaylor@TheWorldofNick-Lappy.local>
* fix(moa): surface stale presets without retries
Keep invalid persisted preset names fail-closed, list the valid configured choices, and classify the local lookup failure as deterministic so it reaches Desktop immediately.
* fix(mem0): migrate legacy OSS base URL aliases
Normalize stale api_base keys to each mem0 provider's accepted URL field before Memory.from_config, without mutating the saved config.
* fix(ci): make tests, workflows, and attribution reliable under load (#66373)
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
* fmt(js): `npm run fix` on merge (#66527)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(cron): avoid Windows Python launcher popups
* fix(cron): preserve POSIX script decoding defaults
* fix(model-switch): override stale api_mode with host-mandated mode on OpenAI-direct switch
Switching to a GPT-5.x model on api.openai.com while the session carried a
stale chat_completions api_mode (e.g. from a prior openrouter default) left
the request on /v1/chat/completions, which 400s with "Function tools with
reasoning_effort are not supported" once the switched model's reasoning is
applied. switch_model() only re-derived api_mode inside the
provider-changed branch, so a same-provider/carryover switch kept the wrong
wire protocol.
Add host_mandated_api_mode(base_url): the endpoints that accept exactly one
protocol (api.openai.com -> codex_responses, api.anthropic.com / *…/anthropic*
-> anthropic_messages, api.kimi.com /coding -> anthropic_messages,
bedrock-runtime -> bedrock_converse), matched by EXACT hostname so lookalike
hosts and path-segment spoofs are rejected (#32243). switch_model() now uses
it to override a stale carried api_mode, not merely fill an empty one;
determine_api_mode() shares the same helper.
Credit sjiangtao2024 (#15880) for the recompute-before-validation approach;
this strengthens it from fill-if-empty to a host-mandated override.
Co-Authored-By: sjiangtao2024 <siage@139.com>
* fix(cli,gateway): sync base_url/api_mode on global model switch persist
Same bug family as #47828, at the config-persistence layer instead of
the in-memory agent layer:
- cli.py (#25106): the --global /model handlers (both the typed-name
path in _handle_model_switch and the picker path in
_apply_model_switch_result) wrote model.default/model.provider to
config.yaml but never touched base_url/api_mode at all. A provider
switch left the OLD endpoint on disk; the next launch reconnected to
the previous provider's host under the new model name.
- gateway/slash_commands.py (#25107): both persist-global blocks (the
picker-tap callback and the typed /model --global path) guarded the
write with two INDEPENDENT ifs — `if result.base_url: ...` and
`if target_provider != "custom": clear_model_endpoint_credentials(...)`.
For named providers the second if always cleared stale values, masking
the bug. For a custom provider with an empty resolved base_url, neither
branch fired, so the previous custom endpoint's base_url/api_key/
api_mode survived untouched in config.yaml.
Fix: explicit set-if-truthy/clear-if-falsy for base_url and api_mode at
all four call sites, matching the already-correct pattern in
tui_gateway/server.py:_persist_model_switch (fixed for #48305).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* test(cli): cover picker-path persist_global=True in #25106 regression tests
hermes-sweeper review on #60970 flagged that _apply_model_switch_result
(the interactive-picker sibling of _handle_model_switch) was only ever
tested with persist_global=False elsewhere, so the picker's global-switch
base_url/api_mode persistence branch had no coverage.
* fix: stop infinite loop when assistant content is a block list
strip_think_blocks() ran re.sub() directly on content that could be a
list of blocks (Anthropic via OpenRouter returns assistant content as
[{type:text,...},{type:thinking,...}]). A list reaching re.sub raised
'TypeError: expected string or bytes-like object, got list', which the
outer conversation loop swallowed and retried forever — the observed
infinite 'preparing terminal...' loop that re-emitted the same
assistant text every iteration.
The live-turn path normalized list content to a string, but
_interim_assistant_visible_text reads a *stored* history message whose
content was persisted as a list and passes it straight into the shared
strip_think_blocks helper. Fix at the shared choke point: coerce
list/dict content to visible text (dropping reasoning blocks, which is
the function's job) before any regex runs, so every caller is safe.
* fix(tools/kanban): sync kanban_unblock response status with DB state
* docs(kanban): clarify unblock status routing
* docs(kanban): explain why an unblocked task can later land in triage
The reported confusion was an unblocked task 'unpredictably' ending up in
triage. unblock itself only ever routes to ready/todo; a subsequent same-cause
re-block hitting BLOCK_RECURRENCE_LIMIT is what escalates to triage. Document
this deterministic loop-breaker at the human-facing lifecycle level so users
stop reading it as an LLM decision.
* fix(ci): restore fork-safe token fallback on PR gates broken by #66373 (#66577)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That
PAT is empty on fork PRs (forks get no repo secrets), which broke every
fork PR two ways:
1. detect-changes classified with the empty PAT -> the compare API failed
all 3 retries -> the classifier failed open and force-enabled the
ci_review lane on EVERY fork PR.
2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with
the same empty PAT via a hard-failing retry step -> the job failed with
no recovery a fork contributor could perform (they can't self-add the
label; re-running can't fix it).
Restores the pre-#66373 fork-safe behavior without reverting the commit's
real improvements (job timeouts, per-file flake retry, network-install
retries):
- detect-changes + ci.yml: token falls back to the built-in read-only
github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT
(authoritative); on forks it uses github.token, which can read the
public compare endpoint. (An input `default:` only applies on omission,
not on an empty passed value — hence the explicit `|| github.token`.)
- lint ci-review + supply-chain mcp-catalog gates: restore the inline
`gh pr view ... || true` label read with the github.token fallback,
dropping the hard-failing retry "Fetch PR labels" step. Graceful
degrade to "label absent" on an API blip, same as before #66373.
Same-repo enforcement is unchanged (byte-identical logic; the PAT is still
used there). Fork PRs classify correctly and the gates read labels via the
read-only token exactly as they did before the regression.
* fix(desktop): trust Windows system CAs for remote gateways (#66304)
* fix(desktop): trust Windows system CAs for remote gateways
Load Windows-trusted roots into Node's default TLS context before Desktop probes remote backends, while preserving bundled and extra CAs.
* test(desktop): cover Windows system CA installation
Verify existing trust roots survive the merge and that unsupported or unavailable stores fail open without changing TLS defaults.
* docs(delegation): align guidance with current contract
* docs(delegation): fix stale internal batch-lifecycle comments
Two internal comments in delegate_tool.py still described the superseded
"N independent handles, no combined wait" model, contradicting the
authoritative batch contract (one async unit, one consolidated result
when all children finish). Aligns the comments with the runtime path in
_execute_and_aggregate / dispatch_async_delegation_batch.
* fix(desktop): expose Local / custom endpoint in Providers API-keys tab (#62818)
The onboarding overlay already contains a 'Local / custom endpoint' card
that writes model.provider:custom + base_url + api_key, but no reachable
Desktop GUI path opens it for a fresh add. The composer model pill falls
back to the gateway menu panel (Edit Models…), and Settings → Providers →
API keys is env-var-driven and never lists a custom endpoint — so users
following their instincts cannot add an OpenAI-compatible endpoint (Zyphra,
vLLM, Ollama, …) from the GUI.
Add a 'Local / custom endpoint' row to the API-keys tab that calls
startManualLocalEndpoint(), landing the overlay directly on the existing
custom-endpoint form. Reuses the tested onboarding flow; no new UI surface.
Regression test in providers-settings.test.tsx asserts the row renders and
opens the custom-endpoint flow.
Fixes #62817
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
* fix(desktop): preserve numeric and display LaTeX (#66173)
* fix(cli): pass TUI Python env from dashboard chat (salvage #44797) (#66581)
* fix: pass TUI Python env from dashboard chat
* fix: share TUI Python env setup
* fix: preserve TUI Python path semantics
* chore: map contributor email for releases
---------
Co-authored-by: AI on behalf of Álvaro Sánchez-Mariscal <alvaro.sanchez-mariscal@oracle.com>
* fix(model-picker): show exhausted-pool providers in interactive /model picker (#66584)
Salvages #66257 by @oppih (CI attribution check blocked the external
branch from merging).
When a provider's credential pool has entries but all are temporarily
rate-limited (exhausted), list_authenticated_providers() excluded the
provider from the interactive /model picker. Rate limits are per-model
for many providers (e.g. Google Gemini), so an exhausted key for
model-A may still work for model-B — the user should still be able to
select a different model under the same provider.
Adds a for_picker flag to list_authenticated_providers() that relaxes
the credential-pool availability check for the picker path only, falling
back to pool.has_credentials() when the pool has entries but none are
currently available. The runtime resolution path
(get_authenticated_provider_slugs) is unchanged, preserving the #45759
invariant that exhausted pools do not count as authenticated.
Co-authored-by: oppih <oppih@users.noreply.github.com>
* fix(dashboard): keep custom themes visible after embedded chat starts (#60601)
* fix(dashboard): resolve dashboard-owned assets from the process launch home
Profile-scoped chat / ?profile= requests install a context-local
HERMES_HOME override, which made custom dashboard themes AND user
dashboard-plugin extensions disappear once the embedded /chat started
under a different profile than the dashboard process.
Add get_process_hermes_home() (sharing _hermes_home_from_env() with
get_hermes_home() so the two can't drift, and splitting the profile
fallback warning into _warn_profile_fallback_once()) and use it for both
the theme YAML scan and the user dashboard-plugin scan — machine-level
assets that belong to the server's launch home and must not follow a
transient per-request override.
Genuinely profile-scoped callers (memories/backups/checkpoints/provider
config) and the paired _merged_plugins_hub classification are left
untouched so they keep following the override.
* test(dashboard): cover process-home asset discovery under profile override
- get_process_hermes_home(): env set returns that path, unset falls back
to the platform default, and an active context-local override is ignored.
- _discover_user_themes() and _discover_dashboard_plugins() keep returning
launch-home assets while a profile override scopes the request elsewhere.
* fix(dashboard): only open the chat PTY once the chat tab is active (#59551)
* fix(dashboard): only open the chat PTY once the chat tab is active
The dashboard mounts ChatPage persistently (hidden with CSS) on every route
so the embedded chat PTY survives tab switches. But the PTY-connect effect
never checked whether the chat tab was active, so it opened `/api/pty` on
mount for ANY dashboard page. On a source/RPi install that spawns the whole
TUI + agent bootstrap (`Installing TUI dependencies…` → `npm install`) merely
by loading /sessions, /system, etc. — work the user never asked for, and the
trigger behind "dashboard loses custom themes on /chat load".
Gate the connect effect on a sticky activation latch: the PTY is not spawned
until the chat tab has been active at least once, and stays connected across
later tab switches so the persistence UX is preserved.
* test(dashboard): cover chat PTY activation latch
Asserts the invariant behind the fix: activation is sticky. It stays false
while the chat tab has never been active (so the persistently-mounted,
hidden ChatPage never opens /api/pty), flips true when the tab activates,
and stays true after the user navigates away (PTY persistence).
* fmt(js): `npm run fix` on merge (#66731)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(desktop): restore exec bit on node-pty spawn-helper for dev terminals
node-pty's published npm tarball ships the POSIX `spawn-helper` with mode
0644 (no exec bit). node-pty `posix_spawnp`s that helper on macOS/Linux, so a
non-executable copy fails every embedded-terminal spawn with
`Error: posix_spawnp failed.`. Packaged builds are unaffected because
stage-native-deps.mjs chmods the staged copy, but the dev flow
(`npm run dev` -> `electron .`) resolves node-pty straight from node_modules,
which nothing chmods -- so the first terminal in dev always dies.
Restore the exec bit once, lazily, right before the first spawn, via a small
DI-testable helper. Idempotent: already-executable copies (packaged builds)
are left untouched, and stat/chmod failures are collected and logged rather
than thrown so terminal startup never breaks.
* fix(desktop): stop tooltips re-opening when a menu/dialog restores focus to its trigger
Picking a model from the composer model pill left the pill's tooltip
stuck open over the fresh selection: Radix Tooltip opens on ANY trigger
focus (its isPointerDownRef guard only covers a pointerdown on the
trigger itself), and Radix menus/dialogs restore focus to their trigger
on close — so every mouse-driven pick ended with a phantom tip. Same
pattern on every Tip-wrapped trigger that opens an overlay.
Gate the focus-open to KEYBOARD focus: the trigger's own onFocus runs
before Radix's composed handler and calls preventDefault() unless the
trigger matches :focus-visible — composeEventHandlers skips onOpen for
defaultPrevented events. Chromium keeps focus-visible modality across
the menu round-trip, so a mouse pick's focus restore no longer opens
the tip, while Tab-focus still shows it (a11y unchanged). Fails open if
:focus-visible is unsupported.
Tests cover the three branches (suppress on non-keyboard focus, keep on
keyboard focus, fail open on selector error); chat/shell suites green.
* fix(ci): make timings report fork-safe (missed by #66577)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and
#66577 restored the `|| github.token` fork fallback for detect-changes and
the label gates -- but it missed the ci-timings "Collect timings and
generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork
PRs that PAT is empty, so timings_report.py hard-fails at
expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must
never redden the PR" soft-fail path. Every fork PR gets a red run from this
advisory job (e.g. #66573).
- ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback
to the timings step. github.token has `actions: read`, enough to read the
run's job/step durations on forks.
- timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run
(TimingsUnavailable) instead of a hard ValueError, so this whole class of
failure can never redden a PR again even if a future workflow drops the
token. Still writes no JSON, so no empty baseline is ever cached.
* fmt(js): `npm run fix` on merge (#66741)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* perf(desktop): cut startup serialization and per-turn REST amplification
Hot-path pass following the switch-latency work. Four independent costs,
one theme — work that runs on every boot or every turn but only needed
to run on actual change:
- electron: start the Python backend in parallel with the renderer load
instead of on did-finish-load. The backend cold boot is the dominant
startup cost and was serialized behind Chromium's load; the connection
promise is shared, so the renderer's getConnection() joins the
in-flight boot, and its getBootProgress() pull on mount recovers any
progress events emitted before the renderer was listening.
- boot/soft-switch: after the socket connects, run the independent
post-connect fetches (cwd seed, config, session lists) concurrently
instead of serially — profile adoption still lands first because the
session fetch scopes by it.
- session.info: config refetch is now gated to the foreground context
and coalesced (one trailing fetch per event burst) — it used to fire
two REST calls per event, including background sessions' heartbeats.
model-options invalidation now requires a VALUE change vs the
session's cached runtime state; the backend stamps model/provider on
every event, so the presence-typed flags refetched the provider
catalog once or twice per turn for a model that never changed.
- turn complete: sidebar refreshes (recents + cron + messaging fan-out,
each scanning profile state.dbs server-side) coalesce across
near-simultaneous completions; $sessions and profile totals keep
their identity when a refresh returns content-identical rows (same
signature gate cron/messaging already use), and the loading flag no
longer flickers over a populated list.
* fix(delegate): declare stateless channel in one-shot and cron so delegate_task returns results
run_agent._dispatch_delegate_task forces background=True for every top-level
delegation, and async_delivery_supported() returns True for any session that
never binds the capability. On runners that cannot receive a completion after
their turn ends, that combination silently discards every subagent result: the
model gets a dispatch handle, ends its turn, and reports 'waiting for results'.
Two such runners never bind the capability:
* hermes -z (one-shot) prints one final response and exits. It bypasses cli.py,
so nothing drains process_registry.completion_queue (only the interactive
process_loop and the gateway watchers do).
* cron run_job clears the HERMES_SESSION_* routing keys, so a completion event
carries session_key="" — _enrich_async_delegation_routing cannot resolve it
and _inject_watch_notification drops it ("no routing metadata"). By then
run_job has already shipped the job's final response via _deliver_result;
there is no turn left to re-enter. Worse, get_current_session_key() can fall
back to the ambient os.environ HERMES_SESSION_KEY, so a cron subagent's output
can be routed into an unrelated user chat rather than merely dropped.
Add declare_stateless_channel() and bind it in both runners, routing
delegate_task to its existing inline/synchronous path — the same fallback the
stateless HTTP adapter already relies on, and the fix suggested in #63142. The
helper binds only the capability: set_session_vars() would also latch
_session_context_engaged, which a pure single-process one-shot must not trigger.
Also correct two agent-facing strings that hardcoded 'stateless HTTP API' as the
only channel without async delivery (delegate_tool, terminal_tool); they now name
the actual condition.
Repro (before): hermes -z 'Use delegate_task to spawn a subagent that replies
BANANA. Report its reply.' -> "Waiting for the subagent's response...", exit 0,
no BANANA. After: BANANA is returned in-turn.
Fixes #53027
Fixes #63142
* fix(docker): strip tini -g flags in legacy entrypoint shim
A plain /usr/bin/tini → /init symlink forwarded tini's -g into
s6-overlay's rc.init as the container CMD, causing boot loops after
image updates that preserve old entrypoints (#66679).
* test(docker): cover tini -g legacy entrypoint boot path
Unit-test flag stripping without Docker, and assert the image shim
rejects the rc.init '-g: not found' restart loop from #66679.
* fix(gateway): route inbound-image decision off the event loop
`_prepare_inbound_message_text` (async) called `_decide_image_input_mode`
inline for every inbound image. That decision is synchronous and does
blocking network I/O on the way to a capability answer:
- `agent.models_dev.fetch_models_dev` — an HTTP GET to models.dev (15s
timeout) whenever the 1-hour in-memory cache is cold or models.dev is slow.
- `agent.model_metadata.query_ollama_supports_vision` — HTTP probes
(`d…
* fix(tui): route images with the live switched model
* chore(release): map auxiliary runtime contributors
* fix(auxiliary): scope runtime state to each turn
* test(compression): expect complete runtime tuple
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm (contributes to #62698) (#66338)
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm
Credential selection runs on a hot path (every model call plus auxiliary
tasks), so an empty/exhausted pool logged "no available entries" at INFO on
*every* selection. On Windows, where multiple Hermes processes share one
rotating log guarded by concurrent-log-handler's cross-process lock, that
per-selection volume storms the lock (RuntimeError: Cannot acquire lock after
20 attempts), pegs a core, and stalls the asyncio event loop long enough that
the Desktop backend readiness probe times out ("Timed out connecting to Hermes
backend after 15000ms") even though the backend already announced
HERMES_BACKEND_READY.
Log the condition at most once per 60s window, re-arming on a successful
selection so recovery->re-exhaustion still surfaces promptly. Same fix class as
the warn-once dedup in #58265.
* test(credential-pool): cover no-available-entries log throttle
Assert the empty-pool INFO line logs at most once per throttle window, logs
again after the window elapses, and re-arms on a successful selection so a
recover->re-exhaust transition surfaces promptly. Uses a deterministic fake
monotonic clock (no sleeps, no network).
* perf(desktop): pre-warm opens the gateway socket too, not just the spawn
Answering the review question on the PR table — why a hovered-cold
switch still showed ~440ms click → WS open: getConnection-only
pre-warming left the WS connect chain to the click, and its microtask
continuation can only run after the click's fresh-draft React flush
(unmounting a large open transcript costs ~300-400ms of render work),
so the socket didn't even START connecting until the flush finished.
Add openGatewayForProfile: the same spawn + connect chain as a real
switch, minus activation — so the hover leaves the profile's socket
fully OPEN and the click's ensureGatewayForProfile just activates it
(no ws:new after the click at all; measured ws open at hover+136ms on
a warm backend). No scheduleReconnect on failure: a hover is
speculative, so a dead backend must not start a background retry loop
— the real switch owns retry and error UX. Pruning semantics are
unchanged: a hover-opened socket for an idle profile is dropped by the
next pruneSecondaryGateways recompute, which just returns the click to
the previous behavior.
Tests updated: pre-warm asserts openGatewayForProfile is called and
that activation (ensureGatewayForProfile) is NOT.
* feat(desktop): promote Fireworks AI to #2 in onboarding provider picker (#66432)
Mirror CANONICAL_PROVIDERS so Fireworks sits directly under Nous Portal
(always visible) ahead of OpenRouter across onboarding, Settings → Providers,
and the API-key catalog.
* fmt(js): `npm run fix` on merge (#66445)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* ci: add 2 minute timeout to osv scan (#66410)
this one ran for 5 hours lol
https://github.com/NousResearch/hermes-agent/actions/runs/29578577080/job/87878711479
* fix(streaming): make the single-writer fence best-effort so a missing guard can't crash a turn (#66448)
A cron job ("Daily Buzz Report") died with 'AIAgent' object has no
attribute '_claim_stream_writer'. The #65991 single-writer fence lives on
AIAgent (run_agent.py), but the streaming paths that use it live in other
modules — chat_completion_helpers (chat / anthropic / bedrock) and
codex_runtime (codex responses) — and called it directly as
agent._claim_stream_writer() / agent._stream_writer_is_current(). That makes
those modules hard-depend on the method being present on whatever object is
passed as agent.
The fence is an *additive* safety net that may only ever drop a provably
superseded stream, never the sole legitimate writer. But the direct calls
turned any agent that doesn't expose it — a version-skewed checkout (the
streaming helper module newer than run_agent), a hot-reloaded gateway mid
git-pull, a duck-typed agent, or a test double — into a fatal AttributeError
that aborts the whole turn (and, on cron, fails the job).
Route every cross-module claim/check through agent/stream_single_writer.py.
claim_stream_writer(agent) returns 0 when the fence is unavailable (or
raises), and stream_writer_is_current(agent, token) treats a 0 token or an
absent guard as "current" — so a guard-less agent degrades to "no fence"
instead of crashing, while a real AIAgent keeps the full single-writer
protection. Internal self.* uses inside run_agent are unchanged (self is
always a full AIAgent there).
* fix(desktop): session-scope fast mode, surface profile ownership + pinned model override
Model-picker audit follow-through — closes the remaining pieces of the
"switch one session, switches everywhere / can't tell whose session this
is" report class:
- tui_gateway: `config.set key=fast` with a session no longer writes the
global agent.service_tier to config.yaml (sibling of the earlier
`reasoning` scoping fix). It pins create_service_tier_override
("priority" / "" for explicit normal) so lazy builds and rebuilds keep
the choice; the desktop's per-model presets were rewriting the global
tier on every model pick. Fast-support validation now checks a draft's
picked model, and `config.get key=fast` reads the pre-build pin.
- desktop: owning-profile tag (initial chip + tooltip/aria label) on
pinned rows and search results in the All-profiles sidebar, and on the
chat header once a second profile exists (#66003).
- desktop: composer model pill shows a pin dot + tooltip when a manual
sticky pick is overriding the Settings default for new chats (#62055).
Closes #66003. Addresses #62055.
* refactor(desktop): derive working/attention session sets from $sessionStates
$workingSessionIds and $attentionSessionIds were independently maintained
atoms that updateSessionState had to manually keep in sync with the session
cache (paired setSessionWorking/setSessionAttention calls, plus a rotation
special-case in ensureSessionState). Make them computed() projections of
$sessionStates instead, so the data flow is one-directional:
gateway event → cache → $sessionStates → computed views.
Transition side-effects (watchdog arm/disarm, settle grace, unread marker,
compression id rotation signal) move into handleTransition, fired from
publishSessionState by diffing previous vs next — one choke point instead
of per-callsite bookkeeping. The watchdog's force-clear reaches the cache
through setWatchdogClearFn rather than a listener set.
Also:
- clearAllSessionStates disarms all watchdog timers and drops settle-grace
entries so a gateway switch can't leak stale timers or keep-set rows
- dropSessionState disarms the dropped runtime's watchdog timer
- watchdog tests now exercise the real timer→callback wiring instead of
manually simulating the clear
* fmt(js): `npm run fix` on merge (#66457)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fmt(js): `npm run fix` on merge (#66460)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fmt(js): `npm run fix` on merge (#66465)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* perf: fast model picker + dialogs — config-load hot path, model.options off the reader thread, off-screen turns skip rendering
Third profiling round (after #66033 / #66347), targeting the composer
model picker and dialog opens (worktree dialog etc.), measured over CDP
on real 1000+-message sessions.
Backend — model.options took 4.8s cold / 1.8s warm per call, and the
desktop model pill/picker blocks on it every open:
- agent/credential_pool: _load_config_safe uses load_config_readonly().
Every consumer only reads, and the per-call deepcopy was the dominant
cost — list_authenticated_providers calls load_pool() per provider
row, and each load_pool loaded (and deep-copied) the full config
again via get_pool_strategy.
- hermes_cli/config: memoize ensure_hermes_home() per home path. It
runs inside the config lock on EVERY load_config(), paying ~14
mkdir/chmod syscalls per call. The fast path still re-checks that the
home dir exists, so a deleted home is recreated as before; profile
switches hit the new path and re-run. Tests cover both.
- tui_gateway/server: add model.options to _LONG_HANDLERS. It measured
seconds inline on the WS reader thread — while it ran, prompt.submit
and session.interrupt sat unread (same class as #21123).
Together: model.options RPC 4825/1842ms → 426/230ms (measured on the
live desktop backend); build_models_payload in isolation 6.2s → 0.97s
cold, 0.27s warm.
Desktop — every Radix dialog/popover open forced a whole-document style
recalc (Presence reads getComputedStyle on mount), which on a
1300-message transcript cost ~650-730ms per open (CPU profile:
getAnimationName 483ms self). The worktree dialog (⌘⇧B) paid it on
every single open:
- thread/list: content-visibility:auto + contain-intrinsic-size on the
per-turn group wrappers. Off-screen turns now skip style recalc,
layout, and paint entirely; never-rendered turns hold a placeholder
height (auto: remembered real size once rendered) so scrollbar and
anchoring stay stable. Verified over CDP: worktree dialog open 656-
730ms → ~200ms on the same session; stick-to-bottom pin, scroll-to-
top rendering, and sticky human bubbles all intact.
Also: profile-session-switch harness accepts CDP_HTTP (Chrome tends to
squat on 9222).
Verification:
- scripts/run_tests.sh: config, credential-pool, inventory,
model-switch routing, tui_gateway protocol, profiles suites green
(test_profiles has one pre-existing failure on main, unrelated);
new tests for the ensure_hermes_home memo.
- apps/desktop: tsc clean, eslint/prettier clean, thread + session
suites green (326 tests).
- E2E over CDP on the live app: numbers above, plus scroll/pin sanity.
* fix(desktop): stop button sends interrupt to wrong session + stale events re-arm busy (#66485)
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
* feat(dev-sandbox): add --from DIR to seed sandbox HERMES_HOME (#66486)
Adds a --from DIR flag to scripts/dev-sandbox.sh that copies an existing
HERMES_HOME directory into the sandbox as the starting point before the
command runs. Lets you spin up a sandbox pre-populated with your real
config, sessions, skills, etc.
scripts/dev-sandbox.sh --from ~/.hermes hermes desktop
Design:
- cp -a dir/. dest/ — preserves perms, symlinks, hidden files
- Clobber guard: only seeds when sandbox HERMES_HOME is empty, so
re-running --persistent doesn't blow away existing sandbox state
- Validates: errors on nonexistent dir, missing arg, flag-like arg,
empty --from=
- Supports both --from DIR and --from=DIR forms
- Backwards compatible: no --from = unchanged behavior
* fix(honcho): resolve the timeout staleness check from honcho.json like the build path
The staleness check added in #66052 resolved the timeout from env,
config.yaml, and the default only, while the build path also reads the
honcho.json host block (timeout/requestTimeout). With a timeout
configured in honcho.json, the two permanently disagreed: every
no-config get_honcho_client() call — i.e. every HonchoSessionManager
.honcho property access — interpreted the mismatch as a config change
and tore down and rebuilt the client, defeating the singleton on the
hot path it was meant to protect.
Teach the check to read honcho.json through the same host-aware chain
as from_global_config, memoized on the file's mtime_ns so the per-call
cost stays one stat(). A genuine honcho.json timeout change is now also
detected, extending #57437 to that config surface.
* fix(honcho): delegate the config.yaml timeout read to load_config_readonly
The staleness check's bespoke mtime memo keyed only on the user
config.yaml, but load_config() merges the managed-scope config
(HERMES_MANAGED_DIR/config.yaml, /etc/hermes) whose leaf keys win. A
managed honcho.timeout with no user config.yaml made the memo cache
'no timeout' while _build resolved the managed value — the same
perpetual-rebuild mismatch this PR fixes for honcho.json. A managed
timeout edit was likewise invisible while the user file's mtime stayed
put.
load_config_readonly() is already cached on both files' signatures plus
the env-ref snapshot, so use it instead of duplicating that
invalidation logic; the defensive deepcopy the old memo existed to
avoid is skipped by the readonly variant. Drive the rebuild test
through a real config.yaml and add a HERMES_MANAGED_DIR regression
test covering stable reuse and managed-timeout edits.
* fmt(js): `npm run fix` on merge (#66505)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* feat(codex): stream live app-server events to TUI/desktop tool cards
Extends the app-server event bridge (make_codex_app_server_event_bridge)
to fire the authoritative stable-ID tool_start_callback /
tool_complete_callback alongside the existing tool_progress_callback,
and route item/reasoning/summaryDelta through the reasoning channel.
Surfaces that render structured tool cards (TUI, desktop) — not just
progress bubbles — now correlate live cards with the projected history
entry after a resume, because the call ids mirror CodexEventProjector's
_deterministic_call_id. Guarded per-callback so a broken display
consumer can't tear down the codex turn loop.
Grafted from PR #65412 by @HaiderSultanArc onto the merged bridge (the
PR's parallel _codex_live_event implementation was reconciled into the
bridge's existing _fire_tool_started/_fire_tool_completed helpers).
* docs(codex): document live app-server display; AUTHOR_MAP entries
- codex-app-server-runtime.md: add a Live display section covering the
stream/reasoning/tool-card bridge and show_commentary gating.
- release.py: AUTHOR_MAP entries for HaiderSultanArc, jjadeo-oss, juanfradb
(the latter two for forthcoming follow-up salvages of #62396 / #18050).
* fix: cap cache-scope headers at 64 chars to avoid Codex 400 error (#66045)
* test(codex): cover overlength cache-scope headers
Exercise the real transport path for long session ids, including stable hashing and bounded body/header cache keys.
* fix(codex): harden final cache-key boundaries
Fold #62349's broader provider-boundary handling into the header fix: bound top-level and xAI override keys again at preflight after middleware, preserve unrelated headers, and cover boundaries and collisions.
Co-authored-by: Nick Taylor <nicktaylor@TheWorldofNick-Lappy.local>
* fix(moa): surface stale presets without retries
Keep invalid persisted preset names fail-closed, list the valid configured choices, and classify the local lookup failure as deterministic so it reaches Desktop immediately.
* fix(mem0): migrate legacy OSS base URL aliases
Normalize stale api_base keys to each mem0 provider's accepted URL field before Memory.from_config, without mutating the saved config.
* fix(ci): make tests, workflows, and attribution reliable under load (#66373)
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
* fmt(js): `npm run fix` on merge (#66527)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(cron): avoid Windows Python launcher popups
* fix(cron): preserve POSIX script decoding defaults
* fix(model-switch): override stale api_mode with host-mandated mode on OpenAI-direct switch
Switching to a GPT-5.x model on api.openai.com while the session carried a
stale chat_completions api_mode (e.g. from a prior openrouter default) left
the request on /v1/chat/completions, which 400s with "Function tools with
reasoning_effort are not supported" once the switched model's reasoning is
applied. switch_model() only re-derived api_mode inside the
provider-changed branch, so a same-provider/carryover switch kept the wrong
wire protocol.
Add host_mandated_api_mode(base_url): the endpoints that accept exactly one
protocol (api.openai.com -> codex_responses, api.anthropic.com / *…/anthropic*
-> anthropic_messages, api.kimi.com /coding -> anthropic_messages,
bedrock-runtime -> bedrock_converse), matched by EXACT hostname so lookalike
hosts and path-segment spoofs are rejected (#32243). switch_model() now uses
it to override a stale carried api_mode, not merely fill an empty one;
determine_api_mode() shares the same helper.
Credit sjiangtao2024 (#15880) for the recompute-before-validation approach;
this strengthens it from fill-if-empty to a host-mandated override.
Co-Authored-By: sjiangtao2024 <siage@139.com>
* fix(cli,gateway): sync base_url/api_mode on global model switch persist
Same bug family as #47828, at the config-persistence layer instead of
the in-memory agent layer:
- cli.py (#25106): the --global /model handlers (both the typed-name
path in _handle_model_switch and the picker path in
_apply_model_switch_result) wrote model.default/model.provider to
config.yaml but never touched base_url/api_mode at all. A provider
switch left the OLD endpoint on disk; the next launch reconnected to
the previous provider's host under the new model name.
- gateway/slash_commands.py (#25107): both persist-global blocks (the
picker-tap callback and the typed /model --global path) guarded the
write with two INDEPENDENT ifs — `if result.base_url: ...` and
`if target_provider != "custom": clear_model_endpoint_credentials(...)`.
For named providers the second if always cleared stale values, masking
the bug. For a custom provider with an empty resolved base_url, neither
branch fired, so the previous custom endpoint's base_url/api_key/
api_mode survived untouched in config.yaml.
Fix: explicit set-if-truthy/clear-if-falsy for base_url and api_mode at
all four call sites, matching the already-correct pattern in
tui_gateway/server.py:_persist_model_switch (fixed for #48305).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* test(cli): cover picker-path persist_global=True in #25106 regression tests
hermes-sweeper review on #60970 flagged that _apply_model_switch_result
(the interactive-picker sibling of _handle_model_switch) was only ever
tested with persist_global=False elsewhere, so the picker's global-switch
base_url/api_mode persistence branch had no coverage.
* fix: stop infinite loop when assistant content is a block list
strip_think_blocks() ran re.sub() directly on content that could be a
list of blocks (Anthropic via OpenRouter returns assistant content as
[{type:text,...},{type:thinking,...}]). A list reaching re.sub raised
'TypeError: expected string or bytes-like object, got list', which the
outer conversation loop swallowed and retried forever — the observed
infinite 'preparing terminal...' loop that re-emitted the same
assistant text every iteration.
The live-turn path normalized list content to a string, but
_interim_assistant_visible_text reads a *stored* history message whose
content was persisted as a list and passes it straight into the shared
strip_think_blocks helper. Fix at the shared choke point: coerce
list/dict content to visible text (dropping reasoning blocks, which is
the function's job) before any regex runs, so every caller is safe.
* fix(tools/kanban): sync kanban_unblock response status with DB state
* docs(kanban): clarify unblock status routing
* docs(kanban): explain why an unblocked task can later land in triage
The reported confusion was an unblocked task 'unpredictably' ending up in
triage. unblock itself only ever routes to ready/todo; a subsequent same-cause
re-block hitting BLOCK_RECURRENCE_LIMIT is what escalates to triage. Document
this deterministic loop-breaker at the human-facing lifecycle level so users
stop reading it as an LLM decision.
* fix(ci): restore fork-safe token fallback on PR gates broken by #66373 (#66577)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That
PAT is empty on fork PRs (forks get no repo secrets), which broke every
fork PR two ways:
1. detect-changes classified with the empty PAT -> the compare API failed
all 3 retries -> the classifier failed open and force-enabled the
ci_review lane on EVERY fork PR.
2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with
the same empty PAT via a hard-failing retry step -> the job failed with
no recovery a fork contributor could perform (they can't self-add the
label; re-running can't fix it).
Restores the pre-#66373 fork-safe behavior without reverting the commit's
real improvements (job timeouts, per-file flake retry, network-install
retries):
- detect-changes + ci.yml: token falls back to the built-in read-only
github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT
(authoritative); on forks it uses github.token, which can read the
public compare endpoint. (An input `default:` only applies on omission,
not on an empty passed value — hence the explicit `|| github.token`.)
- lint ci-review + supply-chain mcp-catalog gates: restore the inline
`gh pr view ... || true` label read with the github.token fallback,
dropping the hard-failing retry "Fetch PR labels" step. Graceful
degrade to "label absent" on an API blip, same as before #66373.
Same-repo enforcement is unchanged (byte-identical logic; the PAT is still
used there). Fork PRs classify correctly and the gates read labels via the
read-only token exactly as they did before the regression.
* fix(desktop): trust Windows system CAs for remote gateways (#66304)
* fix(desktop): trust Windows system CAs for remote gateways
Load Windows-trusted roots into Node's default TLS context before Desktop probes remote backends, while preserving bundled and extra CAs.
* test(desktop): cover Windows system CA installation
Verify existing trust roots survive the merge and that unsupported or unavailable stores fail open without changing TLS defaults.
* docs(delegation): align guidance with current contract
* docs(delegation): fix stale internal batch-lifecycle comments
Two internal comments in delegate_tool.py still described the superseded
"N independent handles, no combined wait" model, contradicting the
authoritative batch contract (one async unit, one consolidated result
when all children finish). Aligns the comments with the runtime path in
_execute_and_aggregate / dispatch_async_delegation_batch.
* fix(desktop): expose Local / custom endpoint in Providers API-keys tab (#62818)
The onboarding overlay already contains a 'Local / custom endpoint' card
that writes model.provider:custom + base_url + api_key, but no reachable
Desktop GUI path opens it for a fresh add. The composer model pill falls
back to the gateway menu panel (Edit Models…), and Settings → Providers →
API keys is env-var-driven and never lists a custom endpoint — so users
following their instincts cannot add an OpenAI-compatible endpoint (Zyphra,
vLLM, Ollama, …) from the GUI.
Add a 'Local / custom endpoint' row to the API-keys tab that calls
startManualLocalEndpoint(), landing the overlay directly on the existing
custom-endpoint form. Reuses the tested onboarding flow; no new UI surface.
Regression test in providers-settings.test.tsx asserts the row renders and
opens the custom-endpoint flow.
Fixes #62817
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
* fix(desktop): preserve numeric and display LaTeX (#66173)
* fix(cli): pass TUI Python env from dashboard chat (salvage #44797) (#66581)
* fix: pass TUI Python env from dashboard chat
* fix: share TUI Python env setup
* fix: preserve TUI Python path semantics
* chore: map contributor email for releases
---------
Co-authored-by: AI on behalf of Álvaro Sánchez-Mariscal <alvaro.sanchez-mariscal@oracle.com>
* fix(model-picker): show exhausted-pool providers in interactive /model picker (#66584)
Salvages #66257 by @oppih (CI attribution check blocked the external
branch from merging).
When a provider's credential pool has entries but all are temporarily
rate-limited (exhausted), list_authenticated_providers() excluded the
provider from the interactive /model picker. Rate limits are per-model
for many providers (e.g. Google Gemini), so an exhausted key for
model-A may still work for model-B — the user should still be able to
select a different model under the same provider.
Adds a for_picker flag to list_authenticated_providers() that relaxes
the credential-pool availability check for the picker path only, falling
back to pool.has_credentials() when the pool has entries but none are
currently available. The runtime resolution path
(get_authenticated_provider_slugs) is unchanged, preserving the #45759
invariant that exhausted pools do not count as authenticated.
Co-authored-by: oppih <oppih@users.noreply.github.com>
* fix(dashboard): keep custom themes visible after embedded chat starts (#60601)
* fix(dashboard): resolve dashboard-owned assets from the process launch home
Profile-scoped chat / ?profile= requests install a context-local
HERMES_HOME override, which made custom dashboard themes AND user
dashboard-plugin extensions disappear once the embedded /chat started
under a different profile than the dashboard process.
Add get_process_hermes_home() (sharing _hermes_home_from_env() with
get_hermes_home() so the two can't drift, and splitting the profile
fallback warning into _warn_profile_fallback_once()) and use it for both
the theme YAML scan and the user dashboard-plugin scan — machine-level
assets that belong to the server's launch home and must not follow a
transient per-request override.
Genuinely profile-scoped callers (memories/backups/checkpoints/provider
config) and the paired _merged_plugins_hub classification are left
untouched so they keep following the override.
* test(dashboard): cover process-home asset discovery under profile override
- get_process_hermes_home(): env set returns that path, unset falls back
to the platform default, and an active context-local override is ignored.
- _discover_user_themes() and _discover_dashboard_plugins() keep returning
launch-home assets while a profile override scopes the request elsewhere.
* fix(dashboard): only open the chat PTY once the chat tab is active (#59551)
* fix(dashboard): only open the chat PTY once the chat tab is active
The dashboard mounts ChatPage persistently (hidden with CSS) on every route
so the embedded chat PTY survives tab switches. But the PTY-connect effect
never checked whether the chat tab was active, so it opened `/api/pty` on
mount for ANY dashboard page. On a source/RPi install that spawns the whole
TUI + agent bootstrap (`Installing TUI dependencies…` → `npm install`) merely
by loading /sessions, /system, etc. — work the user never asked for, and the
trigger behind "dashboard loses custom themes on /chat load".
Gate the connect effect on a sticky activation latch: the PTY is not spawned
until the chat tab has been active at least once, and stays connected across
later tab switches so the persistence UX is preserved.
* test(dashboard): cover chat PTY activation latch
Asserts the invariant behind the fix: activation is sticky. It stays false
while the chat tab has never been active (so the persistently-mounted,
hidden ChatPage never opens /api/pty), flips true when the tab activates,
and stays true after the user navigates away (PTY persistence).
* fmt(js): `npm run fix` on merge (#66731)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix(desktop): restore exec bit on node-pty spawn-helper for dev terminals
node-pty's published npm tarball ships the POSIX `spawn-helper` with mode
0644 (no exec bit). node-pty `posix_spawnp`s that helper on macOS/Linux, so a
non-executable copy fails every embedded-terminal spawn with
`Error: posix_spawnp failed.`. Packaged builds are unaffected because
stage-native-deps.mjs chmods the staged copy, but the dev flow
(`npm run dev` -> `electron .`) resolves node-pty straight from node_modules,
which nothing chmods -- so the first terminal in dev always dies.
Restore the exec bit once, lazily, right before the first spawn, via a small
DI-testable helper. Idempotent: already-executable copies (packaged builds)
are left untouched, and stat/chmod failures are collected and logged rather
than thrown so terminal startup never breaks.
* fix(desktop): stop tooltips re-opening when a menu/dialog restores focus to its trigger
Picking a model from the composer model pill left the pill's tooltip
stuck open over the fresh selection: Radix Tooltip opens on ANY trigger
focus (its isPointerDownRef guard only covers a pointerdown on the
trigger itself), and Radix menus/dialogs restore focus to their trigger
on close — so every mouse-driven pick ended with a phantom tip. Same
pattern on every Tip-wrapped trigger that opens an overlay.
Gate the focus-open to KEYBOARD focus: the trigger's own onFocus runs
before Radix's composed handler and calls preventDefault() unless the
trigger matches :focus-visible — composeEventHandlers skips onOpen for
defaultPrevented events. Chromium keeps focus-visible modality across
the menu round-trip, so a mouse pick's focus restore no longer opens
the tip, while Tab-focus still shows it (a11y unchanged). Fails open if
:focus-visible is unsupported.
Tests cover the three branches (suppress on non-keyboard focus, keep on
keyboard focus, fail open on selector error); chat/shell suites green.
* fix(ci): make timings report fork-safe (missed by #66577)
#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and
#66577 restored the `|| github.token` fork fallback for detect-changes and
the label gates -- but it missed the ci-timings "Collect timings and
generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork
PRs that PAT is empty, so timings_report.py hard-fails at
expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must
never redden the PR" soft-fail path. Every fork PR gets a red run from this
advisory job (e.g. #66573).
- ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback
to the timings step. github.token has `actions: read`, enough to read the
run's job/step durations on forks.
- timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run
(TimingsUnavailable) instead of a hard ValueError, so this whole class of
failure can never redden a PR again even if a future workflow drops the
token. Still writes no JSON, so no empty baseline is ever cached.
* fmt(js): `npm run fix` on merge (#66741)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* perf(desktop): cut startup serialization and per-turn REST amplification
Hot-path pass following the switch-latency work. Four independent costs,
one theme — work that runs on every boot or every turn but only needed
to run on actual change:
- electron: start the Python backend in parallel with the renderer load
instead of on did-finish-load. The backend cold boot is the dominant
startup cost and was serialized behind Chromium's load; the connection
promise is shared, so the renderer's getConnection() joins the
in-flight boot, and its getBootProgress() pull on mount recovers any
progress events emitted before the renderer was listening.
- boot/soft-switch: after the socket connects, run the independent
post-connect fetches (cwd seed, config, session lists) concurrently
instead of serially — profile adoption still lands first because the
session fetch scopes by it.
- session.info: config refetch is now gated to the foreground context
and coalesced (one trailing fetch per event burst) — it used to fire
two REST calls per event, including background sessions' heartbeats.
model-options invalidation now requires a VALUE change vs the
session's cached runtime state; the backend stamps model/provider on
every event, so the presence-typed flags refetched the provider
catalog once or twice per turn for a model that never changed.
- turn complete: sidebar refreshes (recents + cron + messaging fan-out,
each scanning profile state.dbs server-side) coalesce across
near-simultaneous completions; $sessions and profile totals keep
their identity when a refresh returns content-identical rows (same
signature gate cron/messaging already use), and the loading flag no
longer flickers over a populated list.
* fix(delegate): declare stateless channel in one-shot and cron so delegate_task returns results
run_agent._dispatch_delegate_task forces background=True for every top-level
delegation, and async_delivery_supported() returns True for any session that
never binds the capability. On runners that cannot receive a completion after
their turn ends, that combination silently discards every subagent result: the
model gets a dispatch handle, ends its turn, and reports 'waiting for results'.
Two such runners never bind the capability:
* hermes -z (one-shot) prints one final response and exits. It bypasses cli.py,
so nothing drains process_registry.completion_queue (only the interactive
process_loop and the gateway watchers do).
* cron run_job clears the HERMES_SESSION_* routing keys, so a completion event
carries session_key="" — _enrich_async_delegation_routing cannot resolve it
and _inject_watch_notification drops it ("no routing metadata"). By then
run_job has already shipped the job's final response via _deliver_result;
there is no turn left to re-enter. Worse, get_current_session_key() can fall
back to the ambient os.environ HERMES_SESSION_KEY, so a cron subagent's output
can be routed into an unrelated user chat rather than merely dropped.
Add declare_stateless_channel() and bind it in both runners, routing
delegate_task to its existing inline/synchronous path — the same fallback the
stateless HTTP adapter already relies on, and the fix suggested in #63142. The
helper binds only the capability: set_session_vars() would also latch
_session_context_engaged, which a pure single-process one-shot must not trigger.
Also correct two agent-facing strings that hardcoded 'stateless HTTP API' as the
only channel without async delivery (delegate_tool, terminal_tool); they now name
the actual condition.
Repro (before): hermes -z 'Use delegate_task to spawn a subagent that replies
BANANA. Report its reply.' -> "Waiting for the subagent's response...", exit 0,
no BANANA. After: BANANA is returned in-turn.
Fixes #53027
Fixes #63142
* fix(docker): strip tini -g flags in legacy entrypoint shim
A plain /usr/bin/tini → /init symlink forwarded tini's -g into
s6-overlay's rc.init as the container CMD, causing boot loops after
image updates that preserve old entrypoints (#66679).
* test(docker): cover tini -g legacy entrypoint boot path
Unit-test flag stripping without Docker, and assert the image shim
rejects the rc.init '-g: not found' restart loop from #66679.
* fix(gateway): route inbound-image decision off the event loop
`_prepare_inbound_message_text` (async) called `_decide_image_input_mode`
inline for every inbound image. That decision is synchronous and does
blocking network I/O on the way to a capability answer:
- `agent.models_dev.fetch_models_dev` — an HTTP GET to models.dev (15s
timeout) whenever the 1-hour in-memory cache is cold or models.dev is slow.
- `agent.model_metadata.query_ollama_supports_vision` — HTTP probes
(`detect_local_server_type` + `/api/show`) against a local Ollama server
when the active provider fronts one.
Running that inline blocks the gateway event loop for up to the request
timeout — so a single user attaching an image freezes EVERY session on that
gateway (no other messages processed, no heartbeats) until the fetch/probe
returns or times out. This is the same off-the-loop class as the cron-fire
verifier and the async_is_safe_url work.
Wrap the call in `asyncio.to_thread` so the blocking capability lookup runs
on a worker thread and the loop stays responsive. The decision result and
routing are unchanged.
Test: a gateway image-routing runtime test asserts the capability lookup runs
off the main (event-loop) thread; it runs on the main thread before the fix.
* fix(delegation): stop mixed platform bundles from re-exposing blocked tools to leaf children
A leaf subagent is meant to be denied delegate_task, execute_code, memory,
clarify, cronjob, and send_message. _strip_blocked_tools() only drops a
toolset when EVERY tool in it is blocked, so mixed platform bundles
(hermes-cli, hermes-telegram, and every other gateway bundle) survived
stripping and re-exposed the blocked tools after composite expansion. A
leaf child spawned from any gateway platform could recursively delegate,
run code, and write memory.
Pass exact one-tool deny toolsets into the child's disabled_toolsets so
model_tools subtracts the blocked names AFTER composite expansion, and the
restriction survives later registry/MCP refreshes. Orchestrators regain
only delegate_task.
Salvaged from #66036 by Mason Tanguay (@DictatorBacon); scoped to the
authority fix + its regressions (docs/interrupt changes dropped).
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
* chore(contributors): map mason@masontanguay.com -> DictatorBacon
* feat(tui+cli): change your Nous plan from the terminal (/subscription, /topup, terminal-billing UX) (#51639)
* feat(tui): rename /billing slash command to /topup
Behavior-preserving rename of the /billing command surface to /topup.
Changes: billing.ts → topup.ts (export topupCommands, name 'topup', new
help string), registry.ts import+spread updated, billingOverlay.tsx
overview header 'Usage credits' → 'Top up credits', billingCommand.test.ts
→ topupCommand.test.ts with import/lookup/call updated. RPC method names
(billing.state, billing.charge, etc.) and component/symbol names unchanged.
* refactor(tui): extract overlay primitives to shared module
Lift MenuRow, ActionRow, footer, and barCells() out of billingOverlay.tsx
into overlayPrimitives.tsx so the upcoming subscriptionOverlay.tsx can
import them instead of duplicating. spendBar now calls barCells() —
output is byte-identical. Pure behavior-preserving refactor.
* feat(tui): add /subscription + /topup CTAs to /usage output
Every /usage render now ends with 'Run /subscription to change plan
· /topup to add credits' — both the healthy (with-calls) and depleted
(no-calls) paths. Strings-only change, no WS1 dependency.
* feat(tui): add subscription wire types
Add SubscriptionTierOption, SubscriptionStateResponse, and
SubscriptionManageLinkResponse to gatewayTypes.ts. Type-only — no
usages yet. Mirrors the BillingStateResponse conventions (snake_case,
Decimals as strings) and reuses BillingErrorPayload for error mapping.
* feat(gateway): add subscription.state + subscription.manage_link RPCs
- agent/subscription_view.py: SubscriptionState dataclass + fail-open
build_subscription_state() (mirrors billing_view pattern) +
get_subscription_manage_link() for the Stripe deep-link.
- hermes_cli/nous_billing.py: get_subscription_state() +
post_subscription_manage_link() HTTP helpers for the two NAS endpoints
(WS1 Phase A/C). The manage-link endpoint raises BillingScopeRequired
when Remote-Spending is missing (Phase 4 step-up trigger).
- tui_gateway/server.py: _serialize_subscription_state() +
subscription.state RPC (fail-open) + subscription.manage_link RPC
(returns {ok,kind,url} or typed error envelope via
_serialize_billing_error). NOT added to _LONG_HANDLERS — synchronous
HTTP round-trip, not a device flow.
* feat(tui): add subscription overlay state types + store slot
Add SubscriptionScreen, SubscriptionOverlayCtx, SubscriptionOverlayState
to interfaces.ts and a 'subscription' slot to OverlayState. Wire it into
overlayStore.ts (buildOverlayState + $isBlocked). NOT added to
resetFlowOverlays preserve list — flow-scoped like billing, drops on
turn end.
* feat(tui): build SubscriptionOverlay — overview + confirm + handoff
Pure-render Ink component mirroring billingOverlay.tsx's structure.
Overview screen covers all 5 states (free-upgradeable, mid-tier,
top-tier, not-admin, downgrade-pending) + dunning. Confirm screen is
y/n deep-link to Stripe (NO in-terminal charge). Handoff is the
transient 'Opening Stripe' screen. Imports shared primitives from
overlayPrimitives.tsx. 8 render tests via renderSync covering every
state.
* feat(tui): add /subscription command + overlay wiring
- subscription.ts: SubscriptionOverlayCtx closure (openManageLink,
refreshState, requestRemoteSpending) + run handler that fetches
subscription.state and opens the overlay. Alias /upgrade.
- registry.ts: spread subscriptionCommands into SLASH_COMMANDS.
- appOverlays.tsx: render SubscriptionOverlay when overlay.subscription set.
- useInputHandlers.ts: Esc closes subscription overlay; promptOverlay OR
includes subscription so input is intercepted while open.
- subscriptionCommand.test.ts: 4 tests (fetch+open, logged-out sys line,
/upgrade alias, /subscription resolves).
* fix(tui/subscription): stop saying Stripe in deep-link copy + fix manage link kind type
Replace all user-facing 'Stripe' mentions in the /subscription overlay and
sys messages with 'your subscription page' — the deep-link target is NAS's
own /manage-subscription page, not the Stripe hosted portal. Stripe only
legitimately appears later at actual Checkout. Also add 'manage' to the
SubscriptionManageLinkResponse.kind union (NAS emits kind:'manage'; was
previously missing from the TypeScript type causing silent narrowing errors).
* feat(tui/subscription): render cancellation-scheduled note with headline precedence
Parse cancelAtPeriodEnd + cancellationEffectiveAt from the NAS contract
(camelCase) in the agent parser (_parse_current), emit cancel_at_period_end
+ cancellation_effective_at from the gateway serializer, extend the
SubscriptionStateResponse type, and render a warn note in OverviewScreen:
'Cancels on {date} — your plan stays active until then.'
Headline precedence when multiple flags co-occur:
past-due > cancel-scheduled > downgrade-pending > active
The downgradeNote guard is tightened to suppress when cancel is scheduled,
so at most one status line renders at a time.
* feat(tui/subscription): team-context screen — redirect to /topup for team orgs
Parse the NAS context:'personal'|'team' field (defaults to 'personal' for
unknown/missing values), emit it on the gateway wire, add it to
SubscriptionStateResponse. When context is 'team', SubscriptionOverlay
renders a dedicated read-only screen instead of the tier picker:
'This terminal is connected to {org_name}. Teams run on shared
credits — use /topup to add funds. Personal subscriptions live
on your personal account.'
The screen closes on Enter or Esc. The personal/tier-picker path is
unchanged.
* fix(subscription): drop manage-link gateway RPC, build URL locally
The NAS POST /api/billing/subscription/manage-link endpoint was dropped
(it added no server work — the target is the static /manage-subscription
page, not a Stripe-minted secret). Build the URL client-side instead:
{portal_base}/manage-subscription?org_id=<org.id>.
- Remove subscription.manage_link gateway RPC (server.py)
- Remove get_subscription_manage_link helper (subscription_view.py)
- Remove post_subscription_manage_link (nous_billing.py)
- Remove SubscriptionManageLinkResponse type (gatewayTypes.ts)
- Add org_id to SubscriptionState + wire through serializer + TS type
- openManageLink() builds the URL locally via buildManageUrl(), opens
it with the existing openExternalUrl(), no gateway round-trip
- Drop targetTierId param from openManageLink (v1 sends everyone to
/manage-subscription; no tier deep-link needed)
- Fix stale test expectations (Stripe copy → subscription page copy)
* chore(subscription): drop unused format_money import
* feat(cli): /subscription + /upgrade, /billing→/topup rename, /usage CTAs
Add the classic-CLI half of the terminal billing surface to match the TUI:
- /subscription (alias /upgrade) command + /topup (renamed /billing, keeps
'billing' as a back-compat alias) in the command registry.
- Drop the stale 'billing' entry from _SLACK_VIA_HERMES_ONLY (now cli_only).
* feat(subscription): CLI /subscription handler, drop dunning, current:null no-plan
- CLI _show_subscription mirrors the TUI overlay (plan read + tier list + usage
bar + browser deep-link via subscription_manage_url); credits render as counts.
- Adapt to the updated NAS read contract: remove is_past_due/dunning everywhere
(a card-failing subscriber returns as a normal plan now), and treat no-plan as
current:null (parser returns None) rather than an all-null object.
- HERMES_DEV_SUBSCRIPTION_FIXTURE env-driven fixtures + ui-tui fixture harness
drive every state (CLI + live TUI) with no portal.
Verified against handoff 2026-06-24_subscription-tui-handoff.md.
* feat(billing): CF-4 Remote-Spending revoked-terminal UX (NAS PR #481)
Wire the Remote-Spending gate denial contract end to end:
- nous_billing: BillingRemoteSpendingRevoked (403 remote_spending_revoked →
reconnect) + BillingSessionRevoked (401 session_revoked → re-login), distinct
from insufficient_scope; capture actor/code/recovery; 503 stays transient.
- gateway _serialize_billing_error threads the new typed kinds + actor/code/
recovery to the TUI.
- TUI renderBillingError: actor-aware revoke copy, kills the spend overlay
immediately (no 15-min zombie button), handles session_revoked, the dual-
emitted cli_billing_disabled/remote_spending_disabled, role_required,
idempotency_conflict; poll treats a mid-poll revoke as ambiguous (check
balance before retry), not a failure.
- CLI _billing_render_charge_error: same denial matrix, actor-aware copy.
Tests: gate-contract mapping + envelope (py) and revoke/session/disabled (TUI).
Per handoff 2026-06-24_remote-spending-TUI-contract-handoff.md.
* refactor(subscription): remove dead step-up scaffolding from /subscription
/subscription only opens a browser deep-link to manage-subscription — that needs
no billing scope, so it can never hit insufficient_scope. Drop the never-fired
'stepup' screen type, requestRemoteSpending ctx fn, and resumeScreen bookkeeping
(leftovers from a superseded plan). The resumable step-up lives on /topup, where
the charge actually gets gated.
* feat(tui/topup): resumable 'Allow Remote Spending' step-up on the charge path
Phase 4: when a charge returns insufficient_scope, the /topup modal no longer
tears down with a 'run /billing again' ConfirmReq. Instead it stays MOUNTED and
switches to a step-up screen:
- charge() is now awaitable, returning a discriminated outcome (submitted |
needs_remote_spending | error) so the overlay can route without closing.
- StepUpScreen: 'Allow Remote Spending' → await the device-flow grant (browser
opens via the existing out-of-band billing.step_up.verification event) →
replay the held charge (pendingCharge.amount) and settle, with no command
re-run. Never surfaces the raw billing:manage scope.
- armStepUp's fire-and-forget ConfirmReq replaced by requestRemoteSpending();
the leaky 'billing:manage' / 'Re-authorize' / 'run /billing again' copy is gone.
Tests: charge-outcome routing, step-up grant/deny, and a render test asserting
the step-up copy holds the amount and never leaks billing:manage.
Per handoff 2026-06-24_remote-spending-TUI-contract-handoff.md §2 (Grady #6).
* feat(billing): shared dollar usage model + two-bar view (drop "credits")
Single source of truth for the /usage and /subscription usage bars across
TUI + CLI. Reads the NAS account-info dollar fields (subscription/top-up/total
remaining, monthly allowance, renewal) and produces a surface-agnostic model:
two full-resolution bars (plan allowance + purchased top-up), a status
classification (free | healthy | low | depleted), and a human renewal date.
- agent/billing_usage.py: UsageModel/UsageBar, usage_model_from_account
(fail-open), build_usage_model (HERMES_DEV_CREDITS_FIXTURE-aware),
format_renews (ISO -> "Jul 24, 2026", Windows-safe), $5 low-balance threshold.
- tui_gateway/server.py: _serialize_usage_model/_serialize_usage_bar, a
usage.bars RPC, and the model embedded into subscription.state so the overlay
renders the same bars from its single fetch.
- Dollars only, never "credits"; two separate bars (not a crammed
three-segment one) for legibility at terminal widths.
- tests/agent/test_billing_usage.py: status classification, bar math
(clamp/over-cap), NaN/Inf rejection, fail-open invariants.
* feat(tui): dollar usage bars on /usage + /subscription, drop tier picker
Render the shared two-bar dollar model in both overlays; strip "credits" and
the in-terminal tier selection per UX feedback.
- overlayPrimitives.tsx: UsageBars (themed plan/top-up bars — gold allowance,
green top-up) + usageBarsText for the /usage panel. Plan name labels the
bar; "$X left of $Y · N% used" (disambiguated so the % matches); top-up
"never expires".
- subscriptionOverlay.tsx: status line dedupes ($X left once; bar carries the
breakdown), human renewal date, state-matched nudges (free upsell / <$5
low alert) with box-safe ASCII markers (! / >) instead of the width-unstable
emoji that broke the border. Tier picker removed — overview shows usage +
plan, then "Manage on portal" / "Close" (free users get "Start a
subscription"). No "credits" anywhere.
- session.ts: /usage renders the dollar bars + balance summary, falling back
to the legacy credits lines only when the model is unavailable; CTA reworded.
- gatewayTypes.ts: UsageModelData/UsageBarData wire types + usage on
SessionUsageResponse/SubscriptionStateResponse.
- Tests updated to the new contract (no "credits", "left of", dedup, markers).
* feat(cli): mirror dollar usage bars on /usage + /subscription
CLI parity with the TUI billing rework, from the same shared usage model.
- _print_nous_credits_block (/usage) and _subscription_overview render the
two-bar dollar view (plan name on the bar, "$X left of $Y · N% used",
top-up "never expires", total spendable) instead of the credits-worded block.
- Dollars only — dropped the tier catalog (no more "$N/mo (… credits)") and
every user-facing "credits"; team copy says "shared balance".
- Human renewal date via the shared format_renews; status line dedupes the
"$X left"; free upsell + <$5 low alert with ASCII markers.
- /subscription manage modal no longer dumps the raw manage-subscription URL
in its detail — the [1] Open / [2] Copy link / [3] Cancel options carry it.
Title is "Manage your subscription" (no in-terminal plan change). The raw URL
stays only in the non-interactive / not-admin fallbacks, which have no menu.
- /usage token-usage panel (model, tokens, cost, context) left untouched.
* feat(billing): embed dollar usage model into billing.state for /topup
The /topup overview renders the same two-bar dollar usage (plan + top-up) as
/usage and /subscription. Embed the shared usage model into the billing.state
RPC payload (mirrors subscription.state) so the overlay gets the bars from its
single fetch, and add the `usage` field to BillingStateResponse.
* feat(tui/topup): reorder overview + in-flight reauth with press-Enter resume
Reworks the /topup overlay per the Jun 19 review and the no-preflight decision.
Overview:
- Balance leads in the title ("Top up · balance $X"); the shared two-bar dollar
usage (plan + top-up) renders below. Dropped the old monthly-cap spend bar.
- "Add funds" is the first action (was "Buy credits"); auto-reload / monthly
limit / manage-on-portal follow. Dollars only — no "credits" anywhere.
- No "Enable terminal billing" menu item and NO scope preflight: whether the
terminal can charge is discovered reactively at pay time. (We deliberately do
not read/refresh the OAuth token to gate UI.)
Step-up (reached only on a charge's insufficient_scope 403):
- New 4-phase flow that keeps the modal mounted: prompt (one-time-setup
heads-up) → waiting (browser authorize) → granted (explicit "Press Enter to
resume") → replay the held charge → settle. The press-Enter beat is the
reassuring "you're back, finish your purchase" moment.
- Renamed user copy "Allow Remote Spending" → "Enable terminal billing"; never
leaks the raw billing:manage scope (guarded by the render test).
- topup.ts error copy de-crufted to terminal-billing wording, emoji removed.
Tests: step-up prompt copy, the no-raw-scope invariant, and new overview tests
(balance-in-title, Add-funds-first, two-bar usage, no "credits").
* feat(cli/topup): mirror overview reorder + in-flight reauth resume
CLI parity with the TUI /topup rehaul, from the same shared usage model.
- _billing_overview: balance in the title, the two-bar dollar usage (plan name
on the plan bar, top-up "never expires") in place of the old cap spend bar,
"Add funds" first, dollars throughout — no "credits", no scope preflight.
- _billing_handle_scope_required: now takes the held amount + idempotency key
and runs the in-flight flow — "Enable terminal billing" → browser device-flow
→ re-check the org kill-switch → press-Enter to resume → replay the held
charge (reusing the key so a double-submit collapses to one). Stops leaking
the raw billing:manage scope.
- Charge-error + buy/auto-reload copy de-crufted to terminal-billing/dollars.
- Tests updated to the new overview + buy copy.
* fix(billing): guard non-JSON 2xx responses in the billing HTTP client
A 2xx response with a non-JSON body — e.g. a reverse-proxy / SPA fallback HTML
page served when a billing route isn't actually mounted on a deployment — hit
json.loads() on the success path of _request() and raised a raw
json.JSONDecodeError. That escaped the typed-BillingError contract, so callers'
`except BillingError` missed it and fell through to a generic fail-open that
rendered as a misleading "not logged in" (observed when /api/billing/subscription
was briefly unshipped on staging: 200 text/html, x-matched-path /[...notFound]).
Now a non-JSON 2xx body raises a typed BillingError(error="endpoint_unavailable")
so surfaces degrade gracefully ("could not load …") instead of crashing or
mislabeling a valid session as logged-out. The 4xx/5xx path already guarded its
.j…
…ousResearch#66373) * feat(attribution): conflict-free contributor mappings via contributors/emails/ directory The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet: every concurrent salvage PR appended entries to the same lines of the same file, so parallel PRs re-conflicted on every merge to main. New system: one file per email under contributors/emails/ — filename is the commit-author email, first non-comment line is the GitHub login. File additions never conflict, so any number of PRs can add mappings concurrently. - scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen) merged with the directory at import time (directory wins). All existing consumers (resolve_author, contributor_audit.py) unchanged. - scripts/add_contributor.py: idempotent CLI to add a mapping; refuses conflicting reassignments (incl. against the legacy map), validates email/login shapes. - contributor-check.yml: attribution gate now accepts a mapping file OR a legacy entry; failure message prints the exact add_contributor command. Also auto-resolves bare <login>@users.noreply.github.com emails is intentionally NOT added (kept id+login form only, matching previous behavior). - contributor_audit.py: guidance now points at add_contributor.py. - tests/scripts/test_contributor_map.py: 12 tests covering loader, merge precedence, CLI idempotency/conflict/validation, subprocess E2E. * feat(ci): one-shot per-file flake retry in the parallel test runner A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry counts as green but is loudly reported in a '⚠ FLAKY' summary section (with both attempts' output preserved) so the flake gets fixed instead of eating a full-run rerun. Deterministic failures fail both attempts — regressions cannot be laundered green. - --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables) - E2E verified: simulated first-run-fail flake goes green with banner; deterministic failure still exits 1; retries=0 restores old behavior. This converts the dominant CI failure mode (one timing-sensitive test flaking a 4600-test shard, requiring a manual 10-minute rerun and an agent triage loop) into a self-healing retry that costs one file's runtime. * test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s These guard against catastrophic regex backtracking (seconds-to-minutes class), but 0.15s is within scheduler-stall noise on loaded shared CI runners — test_max_accepted_separator_free_input_is_fast failed a CI shard this week on runner load alone. 2.0s still catches the regression class with zero flake surface. * fix(ci): job timeouts everywhere + retries on all network installs Reliability pass over every workflow: - timeout-minutes on all 21 jobs that lacked one (a hung job previously burned the 6-hour default runner budget) - ./.github/actions/retry wrapped around every network-fetching install that lacked it: pip installs (deploy-site, skills-index), npm ci (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker test deps). Deterministic build steps (npm run build) deliberately NOT retried — split into separate steps so a real build failure fails fast instead of retrying 3x. * docs(agents): document the file-retry flake policy * fix(ci): curl retries on deploy hook + skills-index probe * fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile From the workflow reliability audit: - tests.yml: duration-cache restore had NO restore-keys while saves use run_id-suffixed keys — the cache never matched once, so LPT slicing always ran blind and unbalanced slices pushed heavy files toward the per-file timeout. One-line restore-keys fixes slice balancing. - Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed): 'gh pr view || true' turned an API blip into 'label absent' → false BLOCKING failure. Now 3x retry, and API failure is reported as an API failure instead of a missing label. - detect-changes action: compare API retried before failing open (was silently running all lanes on any blip). - uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried so registry blips don't read as 'lockfile stale'. - docker.yml merge job: imagetools create retried (Docker Hub eventual consistency on just-pushed digests). - Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to curl --retry 3 (ADD cannot retry; checksums still enforced); npm --fetch-retries=5; playwright chromium fetch retried 3x. - Advisory artifact uploads (per-slice durations, ci-timings report) get continue-on-error so an artifact-service blip can't fail a green test slice. * fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list - test_tui_gateway_server.py: session.create / non-eager session.resume arm a 50ms threading.Timer (_schedule_agent_build) that outlives its test and fires into the NEXT test's _make_agent mock, racily corrupting captured state (the recurring session_resume shard failures). Replaced the per-test whack-a-mole stub with a module-wide autouse fixture; the 3 worker-lifecycle tests that genuinely need the deferred build opt back in via @pytest.mark.real_agent_prewarm (new marker in pyproject). - test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the live PROVIDER_REGISTRY instead of a hand-list that had drifted (missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto') tests failed on any machine with HF_TOKEN exported. E2E-verified with HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass. * test: de-flake 30 timing-sensitive test files for loaded CI runners Root-cause fixes from the flake audit (session-DB mining + repo sweep): Event-based sync instead of sleep-sync: - title_generator: mock sets threading.Event, wait(10) replaces sleep(0.3) hoping the daemon thread got scheduled - docker zombie_reaping / profile_gateway: poll-for-state helpers replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async) - process_registry tree test: select()-bounded readline replaces an unbounded blocking read (parent wedge now fails THIS test with a clear message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s (the 1s partition window mid-interpreter-startup is how a child PID escaped the live-system guard in CI) Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors; all of these complete in ms-to-1s when healthy so the raises cost nothing on green runs): - subprocess/thread waits <= 2s raised to 10-15s across mcp_tool, mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe, mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt, voice_cli_integration, docker_environment, session_store_lock_io, planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output (joins now also assert not is_alive() so stragglers fail loudly) - wall-clock discrimination ceilings loosened where the guarded hang is 10x larger: local_background_child_hang 4s->10s, interrupt_cleanup setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup 5s->15s, protocol/gil-starvation fast-handler 0.5s->2s, iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s - narrow assertion windows widened: honcho first-turn wait 0.4..0.65 -> 0.25..2.0 (property is bounded-not-hung, not an exact wall-clock); compression fork-lock TTL 1s->3s (12 refresh chances per lease); compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0) - telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under) * fix(tests): repair indentation from de-flake batch edit * fix(tests): harden env isolation and replace remaining sleep-sync races The full 42k-test run and complete npm check surfaced three more classes: - Environment isolation: local ~/.honcho defaultHost and SSH_* variables leaked into Python/TUI tests. Pin the default Honcho host in the hermetic fixture, isolate the one fallback test from ~/.honcho, and blank SSH_* around terminalSetup tests. This flipped 20 false failures back to deterministic behavior on developer machines. - Background-thread sleep-sync: Honcho async writer tests patched time.sleep globally, then busy-polled with that same mocked sleep. Under full-suite load the poller could starve the writer. Each test now waits on an Event emitted by the exact flush/retry transition; 30/30 passed under 15-way contention. - Desktop streaming: the test slept 80ms and assumed a 500ms timer could not fire before its assertion. A loaded runner descheduled the test for >500ms and both chunks arrived. Producer controls now gate second-chunk and completion transitions explicitly. Also make file-retry observability complete: a self-healed flaky file now prints BOTH attempts' full output in the FLAKY summary. Two behavioral runner tests prove pass-on-retry is green+loud+traceback-preserving, while a deterministic failure remains red. * refactor(ci): use gh bot pat, better retries refactor(ci): use retry action for PR label fetch the retry action now captures stdout as a step output, so it can serve double duty: retry + output capture for commands like 'gh pr view' whose result must be consumed by later steps. Retry action gains: - 'stdout' output (heredoc-delimited to preserve newlines) - tee to temp file so stdout still streams to the job log - step id 'retry' for output reference Both lint.yml and supply-chain-audit.yml now use the retry action directly with 'command: gh pr view ...' and read steps.<id>.outputs.stdout. ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth Replace secrets.GITHUB_TOKEN and github.token with secrets.AUTOFIX_BOT_PAT across all workflows and composite actions that use the gh CLI or GitHub API. The PAT has consistent permissions across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate limit sharing with the default token, and is already used by js-autofix.yml for the same reasons. 19 sites swapped across 9 files: - lint.yml (3): label fetch, comment post/edit, comment update - supply-chain-audit.yml (5): scan, critical comment, unbounded dep comment, label fetch, mcp-catalog comment - lockfile-diff.yml (1): PR comment post/update - skills-index-freshness.yml (1): issue creation on degraded probe - skills-index.yml (2): index build, trigger deploy workflow - upload_to_pypi.yml (2): release view poll, release upload - ci.yml (1): timings report - deploy-site.yml (2): skills index crawl - detect-changes/action.yml (1): compare API call --------- Co-authored-by: ethernet <arilotter@gmail.com>
…esearch#66373 (NousResearch#66577) NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That PAT is empty on fork PRs (forks get no repo secrets), which broke every fork PR two ways: 1. detect-changes classified with the empty PAT -> the compare API failed all 3 retries -> the classifier failed open and force-enabled the ci_review lane on EVERY fork PR. 2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with the same empty PAT via a hard-failing retry step -> the job failed with no recovery a fork contributor could perform (they can't self-add the label; re-running can't fix it). Restores the pre-NousResearch#66373 fork-safe behavior without reverting the commit's real improvements (job timeouts, per-file flake retry, network-install retries): - detect-changes + ci.yml: token falls back to the built-in read-only github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT (authoritative); on forks it uses github.token, which can read the public compare endpoint. (An input `default:` only applies on omission, not on an empty passed value — hence the explicit `|| github.token`.) - lint ci-review + supply-chain mcp-catalog gates: restore the inline `gh pr view ... || true` label read with the github.token fallback, dropping the hard-failing retry "Fetch PR labels" step. Graceful degrade to "label absent" on an API blip, same as before NousResearch#66373. Same-repo enforcement is unchanged (byte-identical logic; the PAT is still used there). Fork PRs classify correctly and the gates read labels via the read-only token exactly as they did before the regression.
NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows and NousResearch#66577 restored the `|| github.token` fork fallback for detect-changes and the label gates -- but it missed the ci-timings "Collect timings and generate report" step, which still passes a bare AUTOFIX_BOT_PAT. On fork PRs that PAT is empty, so timings_report.py hard-fails at expect_env("GITHUB_TOKEN") before it can reach its own "degraded run must never redden the PR" soft-fail path. Every fork PR gets a red run from this advisory job (e.g. NousResearch#66573). - ci.yml: apply the same `secrets.AUTOFIX_BOT_PAT || github.token` fallback to the timings step. github.token has `actions: read`, enough to read the run's job/step durations on forks. - timings_report.py: treat a missing/empty GITHUB_TOKEN as a degraded run (TimingsUnavailable) instead of a hard ValueError, so this whole class of failure can never redden a PR again even if a future workflow drops the token. Still writes no JSON, so no empty baseline is ever cached.
…ousResearch#66373) * feat(attribution): conflict-free contributor mappings via contributors/emails/ directory The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet: every concurrent salvage PR appended entries to the same lines of the same file, so parallel PRs re-conflicted on every merge to main. New system: one file per email under contributors/emails/ — filename is the commit-author email, first non-comment line is the GitHub login. File additions never conflict, so any number of PRs can add mappings concurrently. - scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen) merged with the directory at import time (directory wins). All existing consumers (resolve_author, contributor_audit.py) unchanged. - scripts/add_contributor.py: idempotent CLI to add a mapping; refuses conflicting reassignments (incl. against the legacy map), validates email/login shapes. - contributor-check.yml: attribution gate now accepts a mapping file OR a legacy entry; failure message prints the exact add_contributor command. Also auto-resolves bare <login>@users.noreply.github.com emails is intentionally NOT added (kept id+login form only, matching previous behavior). - contributor_audit.py: guidance now points at add_contributor.py. - tests/scripts/test_contributor_map.py: 12 tests covering loader, merge precedence, CLI idempotency/conflict/validation, subprocess E2E. * feat(ci): one-shot per-file flake retry in the parallel test runner A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry counts as green but is loudly reported in a '⚠ FLAKY' summary section (with both attempts' output preserved) so the flake gets fixed instead of eating a full-run rerun. Deterministic failures fail both attempts — regressions cannot be laundered green. - --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables) - E2E verified: simulated first-run-fail flake goes green with banner; deterministic failure still exits 1; retries=0 restores old behavior. This converts the dominant CI failure mode (one timing-sensitive test flaking a 4600-test shard, requiring a manual 10-minute rerun and an agent triage loop) into a self-healing retry that costs one file's runtime. * test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s These guard against catastrophic regex backtracking (seconds-to-minutes class), but 0.15s is within scheduler-stall noise on loaded shared CI runners — test_max_accepted_separator_free_input_is_fast failed a CI shard this week on runner load alone. 2.0s still catches the regression class with zero flake surface. * fix(ci): job timeouts everywhere + retries on all network installs Reliability pass over every workflow: - timeout-minutes on all 21 jobs that lacked one (a hung job previously burned the 6-hour default runner budget) - ./.github/actions/retry wrapped around every network-fetching install that lacked it: pip installs (deploy-site, skills-index), npm ci (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker test deps). Deterministic build steps (npm run build) deliberately NOT retried — split into separate steps so a real build failure fails fast instead of retrying 3x. * docs(agents): document the file-retry flake policy * fix(ci): curl retries on deploy hook + skills-index probe * fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile From the workflow reliability audit: - tests.yml: duration-cache restore had NO restore-keys while saves use run_id-suffixed keys — the cache never matched once, so LPT slicing always ran blind and unbalanced slices pushed heavy files toward the per-file timeout. One-line restore-keys fixes slice balancing. - Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed): 'gh pr view || true' turned an API blip into 'label absent' → false BLOCKING failure. Now 3x retry, and API failure is reported as an API failure instead of a missing label. - detect-changes action: compare API retried before failing open (was silently running all lanes on any blip). - uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried so registry blips don't read as 'lockfile stale'. - docker.yml merge job: imagetools create retried (Docker Hub eventual consistency on just-pushed digests). - Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to curl --retry 3 (ADD cannot retry; checksums still enforced); npm --fetch-retries=5; playwright chromium fetch retried 3x. - Advisory artifact uploads (per-slice durations, ci-timings report) get continue-on-error so an artifact-service blip can't fail a green test slice. * fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list - test_tui_gateway_server.py: session.create / non-eager session.resume arm a 50ms threading.Timer (_schedule_agent_build) that outlives its test and fires into the NEXT test's _make_agent mock, racily corrupting captured state (the recurring session_resume shard failures). Replaced the per-test whack-a-mole stub with a module-wide autouse fixture; the 3 worker-lifecycle tests that genuinely need the deferred build opt back in via @pytest.mark.real_agent_prewarm (new marker in pyproject). - test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the live PROVIDER_REGISTRY instead of a hand-list that had drifted (missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto') tests failed on any machine with HF_TOKEN exported. E2E-verified with HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass. * test: de-flake 30 timing-sensitive test files for loaded CI runners Root-cause fixes from the flake audit (session-DB mining + repo sweep): Event-based sync instead of sleep-sync: - title_generator: mock sets threading.Event, wait(10) replaces sleep(0.3) hoping the daemon thread got scheduled - docker zombie_reaping / profile_gateway: poll-for-state helpers replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async) - process_registry tree test: select()-bounded readline replaces an unbounded blocking read (parent wedge now fails THIS test with a clear message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s (the 1s partition window mid-interpreter-startup is how a child PID escaped the live-system guard in CI) Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors; all of these complete in ms-to-1s when healthy so the raises cost nothing on green runs): - subprocess/thread waits <= 2s raised to 10-15s across mcp_tool, mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe, mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt, voice_cli_integration, docker_environment, session_store_lock_io, planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output (joins now also assert not is_alive() so stragglers fail loudly) - wall-clock discrimination ceilings loosened where the guarded hang is 10x larger: local_background_child_hang 4s->10s, interrupt_cleanup setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup 5s->15s, protocol/gil-starvation fast-handler 0.5s->2s, iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s - narrow assertion windows widened: honcho first-turn wait 0.4..0.65 -> 0.25..2.0 (property is bounded-not-hung, not an exact wall-clock); compression fork-lock TTL 1s->3s (12 refresh chances per lease); compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0) - telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under) * fix(tests): repair indentation from de-flake batch edit * fix(tests): harden env isolation and replace remaining sleep-sync races The full 42k-test run and complete npm check surfaced three more classes: - Environment isolation: local ~/.honcho defaultHost and SSH_* variables leaked into Python/TUI tests. Pin the default Honcho host in the hermetic fixture, isolate the one fallback test from ~/.honcho, and blank SSH_* around terminalSetup tests. This flipped 20 false failures back to deterministic behavior on developer machines. - Background-thread sleep-sync: Honcho async writer tests patched time.sleep globally, then busy-polled with that same mocked sleep. Under full-suite load the poller could starve the writer. Each test now waits on an Event emitted by the exact flush/retry transition; 30/30 passed under 15-way contention. - Desktop streaming: the test slept 80ms and assumed a 500ms timer could not fire before its assertion. A loaded runner descheduled the test for >500ms and both chunks arrived. Producer controls now gate second-chunk and completion transitions explicitly. Also make file-retry observability complete: a self-healed flaky file now prints BOTH attempts' full output in the FLAKY summary. Two behavioral runner tests prove pass-on-retry is green+loud+traceback-preserving, while a deterministic failure remains red. * refactor(ci): use gh bot pat, better retries refactor(ci): use retry action for PR label fetch the retry action now captures stdout as a step output, so it can serve double duty: retry + output capture for commands like 'gh pr view' whose result must be consumed by later steps. Retry action gains: - 'stdout' output (heredoc-delimited to preserve newlines) - tee to temp file so stdout still streams to the job log - step id 'retry' for output reference Both lint.yml and supply-chain-audit.yml now use the retry action directly with 'command: gh pr view ...' and read steps.<id>.outputs.stdout. ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth Replace secrets.GITHUB_TOKEN and github.token with secrets.AUTOFIX_BOT_PAT across all workflows and composite actions that use the gh CLI or GitHub API. The PAT has consistent permissions across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate limit sharing with the default token, and is already used by js-autofix.yml for the same reasons. 19 sites swapped across 9 files: - lint.yml (3): label fetch, comment post/edit, comment update - supply-chain-audit.yml (5): scan, critical comment, unbounded dep comment, label fetch, mcp-catalog comment - lockfile-diff.yml (1): PR comment post/update - skills-index-freshness.yml (1): issue creation on degraded probe - skills-index.yml (2): index build, trigger deploy workflow - upload_to_pypi.yml (2): release view poll, release upload - ci.yml (1): timings report - deploy-site.yml (2): skills index crawl - detect-changes/action.yml (1): compare API call --------- Co-authored-by: ethernet <arilotter@gmail.com>
…esearch#66373 (NousResearch#66577) NousResearch#66373 swapped GITHUB_TOKEN -> AUTOFIX_BOT_PAT across the workflows. That PAT is empty on fork PRs (forks get no repo secrets), which broke every fork PR two ways: 1. detect-changes classified with the empty PAT -> the compare API failed all 3 retries -> the classifier failed open and force-enabled the ci_review lane on EVERY fork PR. 2. The ci-reviewed / mcp-catalog-reviewed label gates then read labels with the same empty PAT via a hard-failing retry step -> the job failed with no recovery a fork contributor could perform (they can't self-add the label; re-running can't fix it). Restores the pre-NousResearch#66373 fork-safe behavior without reverting the commit's real improvements (job timeouts, per-file flake retry, network-install retries): - detect-changes + ci.yml: token falls back to the built-in read-only github.token when AUTOFIX_BOT_PAT is empty. On main it uses the PAT (authoritative); on forks it uses github.token, which can read the public compare endpoint. (An input `default:` only applies on omission, not on an empty passed value — hence the explicit `|| github.token`.) - lint ci-review + supply-chain mcp-catalog gates: restore the inline `gh pr view ... || true` label read with the github.token fallback, dropping the hard-failing retry "Fetch PR labels" step. Graceful degrade to "label absent" on an API blip, same as before NousResearch#66373. Same-repo enforcement is unchanged (byte-identical logic; the PAT is still used there). Fork PRs classify correctly and the gates read labels via the read-only token exactly as they did before the regression.
Summary
CI now self-heals one-off test-file flakes, eliminates the known timing/env races, retries transient external I/O, bounds every workflow job, restores duration-balanced shards, and maps contributor emails without a shared-file merge-conflict hotspot.
Root causes were tight loaded-runner timing assumptions, sleep-based synchronization, environment/config leakage, un-retried registries/APIs, a duration cache that never restored, and every salvage PR appending to the same 1,900-line
AUTHOR_MAPdict.Changes
⚠ FLAKYwith both complete attempts. Deterministic failures remain red. New behavior tests cover both paths.timeout-minutesto every job, bounded retries around network installs/API label gates/Docker manifest publication, working duration-cache restore keys, and non-blocking advisory artifact uploads.contributors/emails/<email>mapping files plusscripts/add_contributor.py; CI accepts either source and gives the exact remediation command.Validation
Infographic