Skip to content

fix(agent): eager fallback on persistent transport failures (#22277) - #53911

Merged
teknium1 merged 3 commits into
mainfrom
hermes/hermes-37d63a82
Jun 28, 2026
Merged

teknium1 merged 3 commits into
mainfrom
hermes/hermes-37d63a82

Conversation

@teknium1

Copy link
Copy Markdown
Collaborator

Summary

A degraded primary whose stream hangs (heartbeats keep the socket alive) now fails over to the configured fallback chain after one retry, instead of silently re-hitting the dead primary for 15+ minutes until the retry budget exhausts.

Root cause: the eager-fallback gate in conversation_loop.py fired only on rate_limit/billing. A stale-detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and fell through to retry-with-backoff against the same broken provider — 3 × ~180–300s stale kills + exponential backoff ≈ the reported 15+ min hang, with the fallback chain idle the whole time.

Changes

  • agent/conversation_loop.py: extend the existing eager-fallback gate to also cover FailoverReason.timeout and FailoverReason.overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same credential-pool-recovery guard and state reset as the rate-limit branch. No new config flag.
  • scripts/release.py: AUTHOR_MAP entry for @linyubin.

Why this shape (not a new config flag, not first-stale failover)

The original eager-fallback feature (#1730) was deliberately scoped to rate limits — a 429 can't clear within the retry window, so retrying is pure waste. Timeout was left to retry-with-backoff on purpose, because a transport blip usually does recover on retry. This PR preserves that intent: it only fails over once a transport error has proven persistent (second attempt), so the common transient case keeps retrying the primary. Users with no fallback chain are unaffected.

Validation

Scenario Behavior
timeout, attempt 1 fails retry primary (transient-recovery intent preserved)
timeout, attempt 2 fails activate fallback (caps bleed at ~2 stale cycles)
rate_limit fires immediately (unchanged)
timeout, no fallback chain configured keeps retrying primary (unchanged)

E2E verified with real imports (classify_api_error + gate logic). Targeted suites green: test_credential_pool_routing (10), test_error_classifier + test_failover_identity + test_gemini_fast_fallback (182). Total 192 tests.

Salvaged from PR #50228 by @linyubin, with their authorship preserved. Dropped #50228's error_classifier.py hunk (set should_fallback=True on the timeout result but the gate keys off the reason enum, never the flag — dead code).

Closes #22277.

Infographic

eager-fallback-transport

@github-actions

github-actions Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-37d63a82 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11522 on HEAD, 11520 on base (🆕 +2)

🆕 New issues (2):

Rule Count
unresolved-attribute 2
First entries
run_agent.py:3014: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

✅ Fixed issues (1):

Rule Count
invalid-assignment 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

Unchanged: 6065 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

linyubin and others added 3 commits June 27, 2026 19:11
Eager fallback previously fired only on rate_limit/billing. A stale-
detector-killed hung stream classifies as FailoverReason.timeout
(retryable=True) and the retry loop re-hit the same dead primary until
the budget exhausted -- 3 x ~180-300s stale kills compounding into a
15+ min silent hang while the configured fallback chain sat idle.

Extend the existing eager-fallback gate to also cover timeout and
overloaded, but only after one real retry (retry_count >= 2) so genuine
transient hiccups still recover on the primary. Reuses the same
pool-recovery guard and state-reset as the rate_limit branch -- no new
config flag, no change to the rate-limit intent.

Salvaged from PR #50228 by @linyubin. Closes #22277.

Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
@teknium1
teknium1 force-pushed the hermes/hermes-37d63a82 branch from 06cc9e4 to df3418f Compare June 28, 2026 02:12
@teknium1
teknium1 merged commit 4133cd9 into main Jun 28, 2026
2 checks passed
@teknium1
teknium1 deleted the hermes/hermes-37d63a82 branch June 28, 2026 02:12
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fallback chain not activated on stream-stall timeouts (15+ min silent hang on degraded primary)

3 participants