fix: trigger model fallback on transport errors (connection/timeout/overloaded) by linyubin · Pull Request #50228 · NousResearch/hermes-agent

linyubin · 2026-06-21T15:46:39Z

Summary

Previously, the fallback chain only activated for rate-limit (429) and billing (402) errors. When the primary model's API was unreachable (connection error, timeout, provider overloaded), the agent retried the same failing provider until max_iterations was exhausted, with no attempt to switch to a configured fallback provider.

This was observed in production: a cron job using mimo-v2.5 failed repeatedly with RuntimeError: Connection error. even though fallback_providers: [deepseek] was configured. The fallback never triggered because connection errors weren't in the fallback activation set.

Changes

`agent/error_classifier.py`

Mark transport errors (ConnectionError, ConnectError, APIConnectionError, timeout, overloaded) with should_fallback=True so the conversation loop can use this signal.

`agent/conversation_loop.py`

Expand the fallback trigger condition to include FailoverReason.timeout and FailoverReason.overloaded:

Rate-limit/billing: Keep existing eager fallback behavior (immediate switch)
Transport errors: Allow 1 retry first (transient hiccups can recover), then fall back if the provider is truly unreachable

Testing

test_error_classifier.py: 161/161 ✅
test_stream_drop_logging.py: 10/10 ✅
test_auxiliary_client.py: 222/222 ✅
test_scheduler.py: 135/136 (1 pre-existing failure unrelated to this change)

Impact

This fix benefits both cron jobs and interactive sessions. When the primary model provider is temporarily unreachable, the agent automatically recovers by switching to the configured fallback instead of failing after exhausting retries.

…verloaded) Previously, the fallback chain only activated for rate-limit (429) and billing (402) errors. When the primary model's API was unreachable (connection error, timeout, provider overloaded), the agent retried the same failing provider until max_iterations was exhausted, with no attempt to switch to a configured fallback provider. Changes: - error_classifier.py: Mark transport errors (ConnectionError, ConnectError, APIConnectionError, timeout, overloaded) with should_fallback=True so the conversation loop can use this signal. - conversation_loop.py: Expand the fallback trigger condition to include FailoverReason.timeout and FailoverReason.overloaded. Transport errors allow 1 retry first (for transient hiccups) before falling back, while rate-limit/billing keep their eager behavior. This ensures cron jobs and interactive sessions automatically recover when the primary model provider is temporarily unreachable, switching to the configured fallback (e.g. deepseek) instead of failing.

@linyubin

Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR #50228 by @linyubin. Closes #22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

@linyubin

Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR #50228 by @linyubin. Closes #22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com> (cherry picked from commit c946e67)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

teknium1 · 2026-07-14T22:32:12Z

Thanks for the transport-fallback fix. This is an automated hermes-sweeper review: the behavioral change is already implemented on current main.

Commit c946e6709fa68b8a85f8e71709dd6642016b46a8 (fix(agent): activate fallback on persistent transport failures (#22277)) explicitly records that it was salvaged from PR fix: trigger model fallback on transport errors (connection/timeout/overloaded) #50228 by @linyubin.
Current agent/conversation_loop.py:3173-3193 treats FailoverReason.timeout and FailoverReason.overloaded as transport failures and switches to a configured fallback after one retry.
Current agent/conversation_loop.py:3222-3234 reports the unreachable provider and activates the fallback through the existing failover path.

Closing as already implemented on main, with contributor attribution preserved in the salvage commit.

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

@linyubin

…esearch#22277) Eager fallback previously fired only on rate_limit/billing. A stale- detector-killed hung stream classifies as FailoverReason.timeout (retryable=True) and the retry loop re-hit the same dead primary until the budget exhausted -- 3 x ~180-300s stale kills compounding into a 15+ min silent hang while the configured fallback chain sat idle. Extend the existing eager-fallback gate to also cover timeout and overloaded, but only after one real retry (retry_count >= 2) so genuine transient hiccups still recover on the primary. Reuses the same pool-recovery guard and state-reset as the rate_limit branch -- no new config flag, no change to the rate-limit intent. Salvaged from PR NousResearch#50228 by @linyubin. Closes NousResearch#22277. Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>

…salvage)

alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 21, 2026

iaji mentioned this pull request Jun 21, 2026

fix: keep transport fallback independent of rate-limit pool gate #50251

Open

alt-glitch mentioned this pull request Jun 24, 2026

feat(core): retry primary on transient 429 before falling back #51677

Open

teknium1 mentioned this pull request Jun 28, 2026

fix(agent): eager fallback on persistent transport failures (#22277) #53911

Merged

teknium1 added a commit that referenced this pull request Jun 28, 2026

chore(release): add AUTHOR_MAP entry for linyubin (#50228 salvage)

fbb59e1

teknium1 added a commit that referenced this pull request Jun 28, 2026

chore(release): add AUTHOR_MAP entry for linyubin (#50228 salvage)

6514be5

pai-scaffolde pushed a commit to pai-scaffolde/hermes-agent that referenced this pull request Jun 28, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

da01061

…salvage)

waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

ca7eae8

…salvage)

Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

cb03b57

…salvage)

This was referenced Jul 8, 2026

fix(fallback): trigger immediate failover on overloaded (529) errors #60790

Open

fix(agent): fix two fallback chain bugs (overloaded immediate fallback + TTL-based suppression) #60785

Open

habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

5bbf1dd

…salvage)

santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

53b8165

…salvage)

teknium1 closed this Jul 14, 2026

teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026

Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

20261fc

…salvage)

leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026

chore(release): add AUTHOR_MAP entry for linyubin (NousResearch#50228 …

dab08f9

…salvage)

Navigation Menu

Uh oh!

fix: trigger model fallback on transport errors (connection/timeout/overloaded) - #50228

fix: trigger model fallback on transport errors (connection/timeout/overloaded)#50228
linyubin wants to merge 1 commit into
NousResearch:mainfrom
linyubin:fix/transport-error-fallback

linyubin commented Jun 21, 2026

Uh oh!

teknium1 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

linyubin commented Jun 21, 2026

Summary

Changes

agent/error_classifier.py

agent/conversation_loop.py

Testing

Impact

Uh oh!

teknium1 commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`agent/error_classifier.py`

`agent/conversation_loop.py`