feat(agent): per-reasoning-model stale-timeout floor (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) by DavidMetcalfe · Pull Request #52238 · NousResearch/hermes-agent

DavidMetcalfe · 2026-06-25T01:36:15Z

Summary

Hermes's default stale-stream detector (HERMES_STREAM_STALE_TIMEOUT = 180s, agent/chat_completion_helpers.py:2544) and non-stream stale detector (HERMES_API_CALL_STALE_TIMEOUT = 90s, run_agent.py:1140) are calibrated for fast-responding chat models. Reasoning models routinely exceed those windows during their thinking phase — NVIDIA Nemotron 3 Ultra on hosted NIM has a documented ~120s upstream idle kill (first-party reproduction at NVIDIA/NemoClaw#4846: TTFB ~31s, stream dies at 120s), and the same failure mode exists on OpenAI o1/o3, Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning. The stale detector kills the connection mid-think, surfacing as API call failed after 3 retries: [Errno 32] Broken pipe.

Fix: a per-reasoning-model stale-timeout floor that lives between the chat-model defaults and explicit user configuration. Auto-mitigates the broken-pipe failure for every reasoning-model user without requiring per-user config edits. Never overrides explicit providers.<id>.models.<model>.stale_timeout_seconds config — that always wins.

Test Plan

python3 -m pytest tests/agent/test_reasoning_stale_timeout_floor.py tests/agent/test_non_stream_stale_timeout.py tests/agent/test_stream_read_timeout_floor.py tests/agent/test_local_stream_timeout.py tests/agent/test_error_classifier.py tests/agent/test_model_metadata.py tests/hermes_cli/test_timeouts.py tests/run_agent/test_primary_runtime_restore.py tests/run_agent/test_streaming.py -q — 480 passed, zero failures, zero regressions.
python3 -m pytest --doctest-modules agent/reasoning_timeouts.py — 9 doctests, 0 failed.
ruff check agent/reasoning_timeouts.py agent/chat_completion_helpers.py run_agent.py tests/agent/test_reasoning_stale_timeout_floor.py — clean.
Negative test: temporarily reverted the source fixes in agent/chat_completion_helpers.py and run_agent.py and confirmed the 2 end-to-end tests in test_reasoning_stale_timeout_floor.py fail with the exact symptom from the issue (_resolved_api_call_stale_timeout_base returns 90s chat-model default instead of 600s/240s reasoning floor). 49 of 51 tests still pass (the pure-function tests for the new agent.reasoning_timeouts module are unaffected by stashing source-file changes because the module exists regardless).
Cross-vendor dual review via agy -p:
- Gemini 3.5 Flash (Medium) — passed: true, zero blockers/should-fix/nits. Verified: regex matches start-of-slug boundary correctly, descending sort by length ensures longest slug wins, aggregator strip safely handles empty and slash edge cases, integration priority is correct, local NIM endpoints are protected, local streams are correctly bypassed to allow infinite timeouts, non-reasoning models are completely unaffected, test placement is clean and modular, doctests serve as valuable inline documentation, httpx read timeouts automatically scale with stale timeouts (so no other call sites are missed), explicit user config correctly overrides the floor.
- GPT-OSS 120B (Medium) — passed: true, zero blockers/should-fix/nits. Same 10 questions, same verdicts.

Notes

Symmetric with PR fix(agent): rebuild connection pool for socket-layer transport errors (BrokenPipe, ConnectionReset, etc.) #52226 (sister issue). That PR fixed a recovery-gate inconsistency where BrokenPipeError was classified as retryable by the error classifier but skipped pool-rebuild by the recovery gate. This PR addresses the upstream cause: the chat-model stale-timeout defaults are too tight for reasoning models, so the connection dies before the recovery gate is even reached. The two fixes are independent and can merge in either order.
Read-timeout auto-scaling works for free. The existing logic at agent/chat_completion_helpers.py:1815-1834 reads the new (raised) _stream_stale_timeout and bumps the httpx socket read timeout to match. So reasoning models get raised BOTH timeouts (stale detector AND socket read timeout) in one fix, with no additional integration point.
Local-endpoint short-circuit preserved. The new floor lives inside the else branch of the local-endpoint check at line 2548 — local providers (Ollama, llama.cpp, vLLM, local NIM) still get float("inf") (stale detection intentionally disabled because local providers don't have upstream idle timeouts). The floor only applies to cloud reasoning models.
Local NIM endpoint case handled. The non-stream integration returns uses_implicit_default=False when the floor fires, so _compute_non_stream_stale_timeout's local-endpoint short-circuit at run_agent.py:1152-1153 does not accidentally disable stale detection for users running reasoning models on a local NIM endpoint.
No new user-facing config knobs. The floor is auto-applied; users who want different behavior set providers.<id>.models.<model>.stale_timeout_seconds explicitly. No new .env variables (project policy: .env is for secrets only; behavioral settings live in config.yaml).
Behavioral regression risk for non-reasoning models is zero. gpt-4o, claude-3-5-sonnet, llama-3.3-70b-instruct, gemini-2.5-pro, qwen2-72b-instruct all return None from get_reasoning_stale_timeout_floor. They get the unchanged 90s / 180s default. 12 negative test cases pin this.
Allowlist maintenance burden. The table has 18 entries (3 Nemotron + 2 DeepSeek + 2 Qwen + 8 OpenAI o-series + 3 Claude + 3 Grok). Adding a new reasoning model means one tuple entry + 1-2 doctest lines + 1-2 test cases. The word-boundary regex shape (^slug(?:$|[\-._])) is robust against community derivatives and fork naming.
Why enumerate o1/o3 variants explicitly instead of using a generic ^o[1-9] regex. Bare o1 over-matches olmo-1 and hypothetical llama-4-70b-o1-preview-style forks; the start-of-slug anchor handles those, but o3-mini and o3 need different floors (300s vs 600s) so they can't share a single entry. Enumeration is unambiguous.
Trade-off in the qwen3 family entry. qwen3 matches all qwen3 models including non-thinking instruct variants, which get a slightly longer wait on a hung provider. The alternative (qwen3-.*-thinking) would break the moment Alibaba or NVIDIA ships a slightly different naming shape. The chosen trade-off is conservative-on-failures, conservative-on-over-match.

What this PR does NOT change

Issue [Bug]: BrokenPipeError skips connection-pool rebuild in transport-recovery gate #52216 (BrokenPipeError skips connection-pool rebuild) — that's a separate bug in the retry-recovery gate. Already merged as PR fix(agent): rebuild connection pool for socket-layer transport errors (BrokenPipe, ConnectionReset, etc.) #52226.
Path A2 (add "broken pipe" / "errno 32" to _SERVER_DISCONNECT_PATTERNS for user-facing guidance) — Gemini Pro flagged this as BLOCKER because the disconnect-with-large-session branch at error_classifier.py:712-732 triggers should_compress=True (silently deletes conversation history). Better solved with a thinking-specific guidance block in a follow-up.
Path C (adaptive TTFB scaling) — deferred per cross-vendor review until the allowlist pattern has soaked.
Path D (NVIDIA plugin defaults) — pending ProviderProfile schema check.

Fixes #52217.

…OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) Hermes's default stale-stream detector (180s) and non-stream stale detector (90s) are calibrated for fast-responding chat models. Reasoning models routinely exceed those windows during their thinking phase — NVIDIA Nemotron 3 Ultra on hosted NIM has a documented ~120s upstream idle kill (first-party reproduction at NVIDIA/NemoClaw#4846: TTFB ~31s, stream dies at 120s), and the same failure mode exists on OpenAI o1/o3, Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning. The stale detector kills the connection mid-think, surfacing as `BrokenPipeError` on the next read. Fix: a per-reasoning-model stale-timeout floor that lives between the chat-model defaults and explicit user configuration. Auto-mitigates the broken-pipe failure for every reasoning-model user without requiring per-user config edits. Never overrides explicit `providers.<id>.models.<model>.stale_timeout_seconds` config — that always wins. Implementation: - New `agent/reasoning_timeouts.py` — single pure function `get_reasoning_stale_timeout_floor(model)` returning the floor in seconds or None. Uses word-boundary-anchored regex (start-of-slug anchor after aggregator-prefix strip) so `openai/o3-mini` matches `o3-mini` but `llama-4-70b-o1-preview` does NOT match `o1-preview` (embedded, not at start of slug) and `olmo-1` does NOT match `o1`. - `agent/chat_completion_helpers.py:2572-2574` — stream-side integration: applies the floor after the existing context-size scaling block (lines 2557-2563) and inside the cloud-provider branch (not the local-endpoint branch where stale detection is intentionally disabled). The existing read-timeout auto-scaling at lines 1815-1834 picks up the new stale value and raises the httpx socket read timeout to match, so reasoning models get raised BOTH timeouts in one fix. - `run_agent.py:1145-1154` — non-stream integration: priority 4 in `_resolved_api_call_stale_timeout_base` (after explicit user config, provider config, env var; before the 90s default). Returns `uses_implicit_default=False` so the local-endpoint short-circuit at lines 1152-1153 does not disable detection for users running reasoning models on a local NIM endpoint. Key design decisions (each cross-vendor reviewed in advance): 1. **Floor is `max(default, floor)`, never overrides explicit user config.** Explicit `providers.<id>.models.<model>.stale_timeout_seconds` always wins (handled by `get_provider_stale_timeout` at the higher priority slot). 2. **Word-boundary-anchored regex matching, not bare substring.** The previous substring-with-trailing-hyphen design would have either false-negatived on bare slugs like `o3-mini` (no trailing hyphen in the slug itself) or false-positived on community derivatives like `llama-4-70b-o1-preview`. The start-of-slug anchor handles both. 3. **Each OpenAI o-series variant enumerated explicitly.** `o1`, `o1-mini`, `o1-pro`, `o1-preview`, `o3`, `o3-pro`, `o3-mini` (smaller floor for the lightweight variant), `o4-mini`. Future `o5` can be added without risk of over-matching. 4. **Qwen3 family entry matches all variants.** `qwen3` covers qwen3-235b, qwen3-32b, etc. without over-specifying — the alternative (`qwen3-.*-thinking`) breaks the moment Alibaba or NVIDIA ships a slightly different naming shape. Verified: - 480 tests passing across 9 directly-relevant test files (4 directly- affected + 5 nearest neighbors including the existing `test_non_stream_stale_timeout.py`, `test_stream_read_timeout_floor.py`, `test_error_classifier.py`, `test_model_metadata.py`, `test_timeouts.py`, `test_primary_runtime_restore.py`, `test_streaming.py`). - 51 new test cases in `tests/agent/test_reasoning_stale_timeout_floor.py`: 28 parametrized positive matches (every allowlist entry), 12 parametrized negative traps (olmo-1, llama-4-70b-o1-preview, qwen2, grok-3/4, etc.), 1 longest-prefix test, 4 end-to-end non-stream resolver tests, 3 stream-mirror tests. - Negative test (temporarily reverted source fixes): the 2 end-to-end tests fail with the exact symptom from the issue (`_resolved_api_call_stale_timeout_base` returns 90s chat-model default instead of 600s/240s reasoning floor). 49 of 51 tests still pass (the pure-function tests for the new module are unaffected by stashing source-file changes). - Lint clean (`ruff check` on all 4 modified/new files). - Doctests in `agent/reasoning_timeouts.py`: 9 passed, 0 failed. - Cross-vendor dual review via `agy -p`: - Gemini 3.5 Flash (Medium) — `passed: true`, zero blockers/should-fix/nits. - GPT-OSS 120B (Medium) — `passed: true`, zero blockers/should-fix/nits. - Both reviewers verified all 10 review questions (regex correctness, longest-match ordering, aggregator-strip edge cases, integration priority, local-endpoint handling, non-reasoning unchanged behavior, test placement, doctest value, no missed call sites, explicit user override priority). Fixes NousResearch#52217.

alt-glitch · 2026-06-25T01:47:33Z

This was generated by AI during triage.

Competing fix for #52217 alongside #52231 — both raise the stale-stream timeout floor for reasoning models in the same two detectors (agent/chat_completion_helpers.py + run_agent.py). This PR factors the floor into a dedicated agent/reasoning_timeouts.py with per-model entries; #52231 does inline prefix detection with a flat 300s floor. Same goal, different mechanism — cross-linking so a maintainer can consolidate. Also related to #52226 (socket-layer transport-recovery for the same broken-pipe family).

…actionable guidance instead of misleading file-write advice Two-part fix: Part 1 (classifier override at agent/error_classifier.py:720-738): A transport disconnect on a reasoning model — even on a large session — now routes to FailoverReason.timeout instead of context_overflow. Without this, large-session reasoning-model disconnects route to the compression branch and silently delete conversation history on a phantom context-length error. The override is strictly targeted: non-reasoning models (gpt-4o, claude-3-5-sonnet, llama-3.3-70b, etc.) still route to context_overflow on large sessions — the existing intentional behavior for chat models whose proxy doesn't idle-kill during prefill/generation. Part 2 (new agent/thinking_timeout_guidance.py + integration at agent/conversation_loop.py:3488-3567): New is_thinking_timeout() and build_thinking_timeout_guidance() helpers. When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3, Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) hits a transport-kill on a small session (classifier says timeout directly) or after Part 1 routes correctly (large session), the user now sees reasoning-specific guidance with three actionable workarounds in priority order: 1. Set providers.<provider>.models.<model>.stale_timeout_seconds: 900 in ~/.hermes/config.yaml (Hermes's built-in floor is already 600s for known reasoning models; raise further if upstream is even tighter). 2. Lower reasoning_budget or set reasoning_effort: medium on this model if the provider supports it. 3. Use a smaller / faster reasoning model if the task doesn't require deep thinking. The new guidance takes precedence via if/elif over the existing _is_stream_drop block, so a reasoning-model user with a transport-kill message sees actionable advice instead of the misleading "try execute_code with Python's open() for large files" advice (which is correct for the unrelated large-file-write stream-drop case but actively wrong for the thinking-timeout case). Verified: - 478 tests passing across 9 directly-relevant files (49 new + 429 existing, zero regressions). - Ruff lint clean on all 4 modified/new files. - Negative test: 6 parametrized regression guards confirm non-reasoning models still route to context_overflow on large sessions; 4 parametrized gates confirm non-timeout classifier reasons never trigger the guidance; 5 parametrized cases confirm non-transport messages never trigger it. - Regression guard: new guidance message does NOT contain "execute_code" or "open()" — the misleading advice is fully replaced, not appended alongside. - Cross-vendor dual review via agy -p: - Gemini 3.5 Flash (Medium) — passed: true, zero blockers, one SHOULD-FIX (vprint block duplication — fixed by extracting detection into a helper module). - GPT-OSS 120B (Medium) — passed: true, zero blockers, two nits (test placement — adopted at tests/agent/test_thinking_timeout_guidance.py; primary-model capture — accepted as non-issue per Flash's nit). Dependency note for maintainers: This PR includes agent/reasoning_timeouts.py (the reasoning-model allowlist module from PR #52238) because the Layer 1 override is load-bearing on get_reasoning_stale_timeout_floor(). After PR #52238 lands on main, this PR's duplicate agent/reasoning_timeouts.py should be rebased away. Either PR can land first; the other rebase is mechanical. Fixes #52271.

…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via #52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from #52238. Fixes #52217.

teknium1 · 2026-06-26T05:12:23Z

Salvaged and merged via #52845 (rebase-merge, your authorship preserved on main as DavidMetcalfe). Carried only the real delta — the stream + non-stream stale-detector wiring + the 51-test suite. The duplicate agent/reasoning_timeouts.py was dropped (it already landed on main via #52795), as were the stale-base MoA/file-mutation reverts your branch had picked up from predating those changes. Verified live: nemotron→600s floor, gpt-4o→90s default unchanged, explicit user config still wins over the floor, floor never lowers an existing threshold. Thanks!

…actionable guidance instead of misleading file-write advice Two-part fix: Part 1 (classifier override at agent/error_classifier.py:720-738): A transport disconnect on a reasoning model — even on a large session — now routes to FailoverReason.timeout instead of context_overflow. Without this, large-session reasoning-model disconnects route to the compression branch and silently delete conversation history on a phantom context-length error. The override is strictly targeted: non-reasoning models (gpt-4o, claude-3-5-sonnet, llama-3.3-70b, etc.) still route to context_overflow on large sessions — the existing intentional behavior for chat models whose proxy doesn't idle-kill during prefill/generation. Part 2 (new agent/thinking_timeout_guidance.py + integration at agent/conversation_loop.py:3488-3567): New is_thinking_timeout() and build_thinking_timeout_guidance() helpers. When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3, Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) hits a transport-kill on a small session (classifier says timeout directly) or after Part 1 routes correctly (large session), the user now sees reasoning-specific guidance with three actionable workarounds in priority order: 1. Set providers.<provider>.models.<model>.stale_timeout_seconds: 900 in ~/.hermes/config.yaml (Hermes's built-in floor is already 600s for known reasoning models; raise further if upstream is even tighter). 2. Lower reasoning_budget or set reasoning_effort: medium on this model if the provider supports it. 3. Use a smaller / faster reasoning model if the task doesn't require deep thinking. The new guidance takes precedence via if/elif over the existing _is_stream_drop block, so a reasoning-model user with a transport-kill message sees actionable advice instead of the misleading "try execute_code with Python's open() for large files" advice (which is correct for the unrelated large-file-write stream-drop case but actively wrong for the thinking-timeout case). Verified: - 478 tests passing across 9 directly-relevant files (49 new + 429 existing, zero regressions). - Ruff lint clean on all 4 modified/new files. - Negative test: 6 parametrized regression guards confirm non-reasoning models still route to context_overflow on large sessions; 4 parametrized gates confirm non-timeout classifier reasons never trigger the guidance; 5 parametrized cases confirm non-transport messages never trigger it. - Regression guard: new guidance message does NOT contain "execute_code" or "open()" — the misleading advice is fully replaced, not appended alongside. - Cross-vendor dual review via agy -p: - Gemini 3.5 Flash (Medium) — passed: true, zero blockers, one SHOULD-FIX (vprint block duplication — fixed by extracting detection into a helper module). - GPT-OSS 120B (Medium) — passed: true, zero blockers, two nits (test placement — adopted at tests/agent/test_thinking_timeout_guidance.py; primary-model capture — accepted as non-issue per Flash's nit). Dependency note for maintainers: This PR includes agent/reasoning_timeouts.py (the reasoning-model allowlist module from PR NousResearch#52238) because the Layer 1 override is load-bearing on get_reasoning_stale_timeout_floor(). After PR NousResearch#52238 lands on main, this PR's duplicate agent/reasoning_timeouts.py should be rebased away. Either PR can land first; the other rebase is mechanical. Fixes NousResearch#52271.

…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.