fix(agent): route reasoning-model thinking-timeouts to timeout, not phantom context_overflow (#52271) - #52795
Merged
Merged
Conversation
…actionable guidance instead of misleading file-write advice
Two-part fix:
Part 1 (classifier override at agent/error_classifier.py:720-738):
A transport disconnect on a reasoning model — even on a large session —
now routes to FailoverReason.timeout instead of context_overflow. Without
this, large-session reasoning-model disconnects route to the compression
branch and silently delete conversation history on a phantom
context-length error. The override is strictly targeted: non-reasoning
models (gpt-4o, claude-3-5-sonnet, llama-3.3-70b, etc.) still route to
context_overflow on large sessions — the existing intentional behavior
for chat models whose proxy doesn't idle-kill during prefill/generation.
Part 2 (new agent/thinking_timeout_guidance.py + integration at
agent/conversation_loop.py:3488-3567):
New is_thinking_timeout() and build_thinking_timeout_guidance() helpers.
When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3,
Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning)
hits a transport-kill on a small session (classifier says timeout
directly) or after Part 1 routes correctly (large session), the user
now sees reasoning-specific guidance with three actionable workarounds
in priority order:
1. Set providers.<provider>.models.<model>.stale_timeout_seconds: 900
in ~/.hermes/config.yaml (Hermes's built-in floor is already 600s
for known reasoning models; raise further if upstream is even
tighter).
2. Lower reasoning_budget or set reasoning_effort: medium on this
model if the provider supports it.
3. Use a smaller / faster reasoning model if the task doesn't
require deep thinking.
The new guidance takes precedence via if/elif over the existing
_is_stream_drop block, so a reasoning-model user with a transport-kill
message sees actionable advice instead of the misleading "try
execute_code with Python's open() for large files" advice (which is
correct for the unrelated large-file-write stream-drop case but
actively wrong for the thinking-timeout case).
Verified:
- 478 tests passing across 9 directly-relevant files (49 new + 429
existing, zero regressions).
- Ruff lint clean on all 4 modified/new files.
- Negative test: 6 parametrized regression guards confirm non-reasoning
models still route to context_overflow on large sessions; 4
parametrized gates confirm non-timeout classifier reasons never
trigger the guidance; 5 parametrized cases confirm non-transport
messages never trigger it.
- Regression guard: new guidance message does NOT contain
"execute_code" or "open()" — the misleading advice is fully
replaced, not appended alongside.
- Cross-vendor dual review via agy -p:
- Gemini 3.5 Flash (Medium) — passed: true, zero blockers, one
SHOULD-FIX (vprint block duplication — fixed by extracting
detection into a helper module).
- GPT-OSS 120B (Medium) — passed: true, zero blockers, two nits
(test placement — adopted at tests/agent/test_thinking_timeout_guidance.py;
primary-model capture — accepted as non-issue per Flash's nit).
Dependency note for maintainers:
This PR includes agent/reasoning_timeouts.py (the reasoning-model
allowlist module from PR #52238) because the Layer 1 override is
load-bearing on get_reasoning_stale_timeout_floor(). After PR #52238
lands on main, this PR's duplicate agent/reasoning_timeouts.py should
be rebased away. Either PR can land first; the other rebase is
mechanical.
Fixes #52271.
teknium1
force-pushed
the
hermes/hermes-897dfb86
branch
from
June 26, 2026 01:53
90b1c02 to
f8d4c2c
Compare
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-attribute |
1 |
invalid-argument-type |
1 |
unresolved-import |
1 |
First entries
tests/agent/test_thinking_timeout_guidance.py:183: [unresolved-attribute] unresolved-attribute: Unresolved attribute `status_code` on type `Exception`
tests/agent/test_thinking_timeout_guidance.py:291: [invalid-argument-type] invalid-argument-type: Argument to function `is_thinking_timeout` is incorrect: Expected `str`, found `None`
tests/agent/test_thinking_timeout_guidance.py:31: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
✅ Fixed issues: none
Unchanged: 5986 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
alt-glitch
added
type/bug
6 tasks
teknium1
pushed a commit
that referenced
this pull request
Jun 26, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via #52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from #52238. Fixes #52217.
teknium1
pushed a commit
that referenced
this pull request
Jun 26, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via #52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from #52238. Fixes #52217.
5 tasks
pai-scaffolde
pushed a commit
to pai-scaffolde/hermes-agent
that referenced
this pull request
Jun 28, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
habarmc1223-sudo
pushed a commit
to habarmc1223-sudo/hermes-agent-fluxmem
that referenced
this pull request
Jul 8, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…+ non-stream detectors Wire get_reasoning_stale_timeout_floor() into both stale detectors so known reasoning models (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) tolerate multi-minute thinking phases instead of the upstream gateway idle-killing the socket (BrokenPipeError) before first token. Applied as max(default, floor) — never overrides explicit user config, never lowers an existing threshold. The reasoning_timeouts.py allowlist module already landed on main via NousResearch#52795, so this salvage carries only the wiring + tests (the duplicate module and the stale-base MoA reverts from the original PR branch are dropped). Salvaged from NousResearch#52238. Fixes NousResearch#52217.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reasoning-model thinking-timeouts no longer trigger phantom context-compression that silently deletes conversation history.
Root cause: on a long session, a transport disconnect mid-thinking (NVIDIA NIM ~120s idle kill, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, Grok reasoning) was classified as
context_overflow+should_compress=True. That routed the user into_compress_context()3× → "Context length exceeded: max compression attempts (3) reached" — on a session that never actually exceeded context. The classifier's disconnect+large-session heuristic is correct for chat models but inverts for reasoning models, whose multi-minute thinking phases routinely outlast the upstream gateway's idle window.Salvage of #52272 (closes #52271), cherry-picked onto current
mainwith @DavidMetcalfe's authorship preserved.Changes
agent/error_classifier.py(Layer 1): reasoning-model override on the no-status disconnect branch — if the model is in the reasoning allowlist, reclassify asFailoverReason.timeout(retryable) instead ofcontext_overflow+compress. Fires before theis_largecheck; does not touch chat models or genuine HTTP-status overflows.agent/thinking_timeout_guidance.py(Layer 2, new):is_thinking_timeout()+build_thinking_timeout_guidance(). Detection requires reasoning-model +timeoutreason + transport-kill substring. Guidance lists three actionable workarounds (raisestale_timeout_seconds, lowerreasoning_budget, smaller model) — replacing the misleading "use execute_code/open() for large files" advice that's wrong for this case.agent/conversation_loop.py(Layer 2 wiring): new_is_thinking_timeoutbranch takes precedence (if/elif) over_is_stream_dropso reasoning users see the right guidance.agent/reasoning_timeouts.py(new): the per-reasoning-model stale-timeout floor allowlist (get_reasoning_stale_timeout_floor). Bundled here because it's the load-bearing dependency for both layers. If feat(agent): per-reasoning-model stale-timeout floor (Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning) #52238 (issue [Bug]: Reasoning models inherit chat-model stale-timeout defaults — broken-pipe on long-thinking models like Nemotron 3 Ultra, OpenAI o1/o3, Opus 4.x thinking #52217) lands first, this duplicate file should be dropped in a mechanical rebase.scripts/release.py: AUTHOR_MAP entry for @DavidMetcalfe.Validation
Live E2E against the real
classify_api_error/ detection helpers (real exception objects, no mocks):main)context_overflow+ compress (phantom)timeout, no compresscontext_overflow+ compresscontext_overflow+ compress (unchanged)context_overflow+ compresscontext_overflow+ compress (unchanged — override doesn't swallow real overflow)Allowlist anchoring verified:
olmo-1andllama-4-70b-o1-previewdo NOT matcho1;o3-mini(300s) beatso3(600s).Bug reproduced on pristine
origin/mainbefore fixing. Tests:tests/agent/test_error_classifier.py(161) +tests/agent/test_thinking_timeout_guidance.py(49) = 210/210 passing.Test plan
Infographic