feat(agent): opt-in recovery hook for wedged local inference backends

[OmarB97]

What

When a local backend trips the first-chunk or non-stream stale watchdog (see #37168, which this PR stacks on), Hermes can now invoke an operator-configured recovery command — useful for self-hosted servers that wedge in ways only the operator knows how to repair (slot cancellation, child-process kill, service restart).

How

• New agent/local_backend_recovery.py: maybe_recover_local_backend() runs HERMES_LOCAL_BACKEND_RECOVERY_COMMAND (opt-in, default off) with sanitized metadata passed via HERMES_RECOVERY_* env vars, rate-limited by HERMES_LOCAL_BACKEND_RECOVERY_COOLDOWN (default 60s) so retries cannot flap a shared GPU service.
• Hook points in the non-stream stale-kill and first-chunk timeout paths; failures in the hook are logged and never affect the turn.
• scripts/recover_local_llama_server.py: a reference recovery script for llama-server-style backends.
• New transport default: HERMES_LOCAL_DEFAULT_MAX_TOKENS caps local completions (default 8192) so a wedged generation cannot run unbounded.

Tests

tests/agent/test_local_backend_recovery.py (env contract, cooldown, opt-in gating), tests/test_recover_local_llama_server.py, transport tests for the max-tokens default. 300 tests pass locally on the stack.

Stacked on #37168 — until that merges, this diff shows the union of both; the recovery-specific changes are agent/local_backend_recovery.py, the two hook sites, scripts/recover_local_llama_server.py, and their tests. Part 5 of the former #35642 split.

[OmarB97]

From: Reviewer (ko-mac.codex#handoff-pr35642-codex-20260602) — on ko-mac

Needs changes — hermes-pr35642-split-snowball-20260602-pr37176, attempt 1 · by ko-mac.codex#handoff-pr35642-codex-20260602

Blocking: the bundled recovery script appends /slots directly to HERMES_RECOVERY_BASE_URL. Hermes passes OpenAI-compatible base URLs such as http://127.0.0.1:9090/v1 (the new hook test asserts that exact env value), so this probes http://127.0.0.1:9090/v1/slots instead of llama.cpp's management endpoint at /slots. That means the default shipped recovery command misses slot cancellation and falls through to no-op/kill behavior. Please normalize a trailing /v1 before building the /slots and /slots/{id}?action=cancel URLs, and add a script-level regression test for a /v1 base URL.

Merge gate: Evidence-backed blocking review feedback must be resolved in a new attempt.

[OmarB97]

From: Reviewer (ko-mac.codex#handoff-pr35642-codex-20260602) — on ko-mac

Needs changes — hermes-pr35642-split-snowball-20260602-pr37176, attempt 1 · by ko-mac.codex#handoff-pr35642-codex-20260602

Request changes: recover_local_llama_server.py must normalize OpenAI-compatible /v1 base URLs before probing/canceling llama.cpp /slots.

Merge gate: Evidence-backed blocking review feedback must be resolved in a new attempt.

[OmarB97]

Addressed the review blocker in 2a3903558.

• scripts/recover_local_llama_server.py now normalizes OpenAI-compatible .../v1 base URLs before building llama.cpp management URLs, so recovery probes/cancels hit /slots and /slots/{id}?action=cancel instead of /v1/slots.
• Added script-level regression coverage for both the /v1 active-slot probe and slot-cancel URLs.

Verification:

• scripts/run_tests.sh tests/test_recover_local_llama_server.py tests/agent/test_local_backend_recovery.py tests/agent/transports/test_chat_completions.py tests/agent/test_local_stream_timeout.py tests/hermes_cli/test_timeouts.py -> 148 passed
• python -m py_compile scripts/recover_local_llama_server.py tests/test_recover_local_llama_server.py
• git diff --cached --check

[OmarB97]

Refreshed onto upstream/main (rebase, live merge-base d33becd877, 349 commits behind). Two real conflicts:

1. agent/transports/chat_completion s.py import block — upstream dropped the now-unused import copy; kept this branch's import os (needed by the local-default-max-tokens env lookups this PR adds to the file), dropped copy.
2. agent/chat_completion_helpers.py (3 spots) — same add/add class as [codex] Add progress outcome canary #35702: upstream independently grew its own cross-turn stale-call circuit breaker (_stale_streak/_check_stale_giveup, Sessions permanently stuck in Stream stale loop - never recovers (OpenAI-compatible provider) #58962) at the same insertion points as this PR's local-provider timeout resolvers/local-backend-recovery hook. No logic overlap — kept both side by side (circuit-breaker block ahead of the timeout helpers; _bump_stale_streak alongside the recovery call in the non-streaming stale-kill path; _check_stale_giveup folded into the same result-dict block that carries local_first_chunk_timeout).

A third apparent conflict (tests/gateway/test_telegram_topic_mode.py) resolved to zero diff: upstream already carries an equivalent fix for the same "parallel work" tip-collision flake in the same test, so this PR's now-redundant duplicate was dropped in favor of upstream's.

Original head 025569e957 -> refreshed 3b930f3ce1. Content unchanged vs original diff (same 15-file set as the original merge-base..head diff). Local verification: scripts/run_tests.sh across all 8 touched-module test files — 821 passed, 0 failed. Checks re-running.

[teknium1]

Thanks for the careful local-backend recovery work, including the follow-up that normalizes OpenAI-compatible /v1 URLs before llama.cpp slot operations.

Automated hermes-sweeper review found that this PR’s user-facing interface is new non-secret HERMES_* configuration:

• agent/local_backend_recovery.py:39-58 configures the command, cooldown, and command timeout through HERMES_LOCAL_BACKEND_RECOVERY_* variables.
• agent/chat_completion_helpers.py and agent/transports/chat_completions.py add further local timeout and output-cap environment knobs.
• Hermes policy requires behavioral settings such as timeouts, thresholds, feature flags, and commands to be configured in config.yaml, not introduced as new .env variables. Existing provider stale-timeout configuration already resolves from providers.<id>.stale_timeout_seconds and model overrides (hermes_cli/timeouts.py:43-69).

Please re-scope any follow-up around a profile-safe config.yaml mechanism (with setup/docs integration where appropriate) rather than new user-facing environment variables.

This is an automated hermes-sweeper review.

---

Closed as not-planned per standing maintainer policy (env-var-for-config). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.