Skip to content

fix(agent): reopen fallback chain after primary transport recovery - #54054

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-9f7df70b
Jun 28, 2026
Merged

fix(agent): reopen fallback chain after primary transport recovery#54054
teknium1 merged 1 commit into
mainfrom
hermes/hermes-9f7df70b

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

After _try_recover_primary_transport() succeeds, a follow-on HTTP 429 now correctly activates the configured fallback_providers instead of burning the retry budget on the rebuilt primary client.

Salvaged from #33846 by @sweetcornna. Closes #32646.

Root cause: try_activate_fallback() does _fallback_index += 1 before the fallback client resolves. A failed or skipped resolution leaves _fallback_index == len(chain) with _fallback_activated still False, which permanently shuts the eager-fallback gate (_fallback_index < len(_fallback_chain)) for the rest of the turn. In the reported timeout → primary-recovery → 429 sequence, the gate was already shut by the time the 429 arrived, so failover never fired. The recently merged #53911 extended that gate to transport failures but did not reset the burned index — this fills exactly that gap.

Changes

  • agent/conversation_loop.py: in the retry_count >= max_retries recovery branch, when primary transport recovery succeeds, also reset has_retried_429 = False, _fallback_index = 0, _fallback_activated = False (alongside the existing retry_count = 0). This is gated by primary_recovery_attempted, so it runs at most once per turn.
  • tests/run_agent/test_32646_fallback_429_after_timeout.py: full-loop regression + direct state-reset unit tests.

Validation

Before After
test_run_conversation_fallbacks_on_429_after_timeout_recovery FAIL — 4th call stays on primary PASS — fails over to backup
fallback / credential-isolation / primary-restore suites (66 tests) 66 passed
ruff check clean

Infographic

reopen-fallback-chain-after-primary-recovery

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 28, 2026
@teknium1
teknium1 merged commit 2701ea2 into main Jun 28, 2026
30 checks passed
@teknium1
teknium1 deleted the hermes/hermes-9f7df70b branch June 28, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fallback_providers not activated when 429 follows prior timeout recovery

3 participants