fix(oneshot): consult fallback_providers at resolution time - #81517
Open
Enough1122 wants to merge 3 commits into
Open
fix(oneshot): consult fallback_providers at resolution time#81517Enough1122 wants to merge 3 commits into
Enough1122 wants to merge 3 commits into
Conversation
…arch#81209) The gateway already had `_try_resolve_fallback_provider` that runs at provider-resolution time, before `AIAgent` is constructed — so a gateway turn survives a primary-provider quota window with a healthy `fallback_providers` chain. The CLI/oneshot lane was bare: a `hermes -z` invocation during a 429 window raised `resolve_runtime_provider(...)` outright, never reaching the `fallback_model=_fb` wiring in `AIAgent(...)` that handles mid-session failures. The docs claim "Where Fallback Works: CLI sessions ✔" but in practice the CLI was only covered for failures after the session started. Fix: extract the fallback loop into `hermes_cli.oneshot._resolve_runtime_with_fallback` and call it in place of the bare `resolve_runtime_provider` call in `_run_agent`. On total failure (primary + every fallback), the primary error is re-raised — it names the operator's configured primary provider, which is what the operator needs to fix first. Adds 5 regression tests covering: primary success short-circuit, primary 429 → fallback success, primary failure with no chain, primary failure with every entry also failing, and first-fallback failure → second-fallback success.
Enough1122
force-pushed
the
fix/81209-oneshot-fallback
branch
from
August 8, 2026 05:52
ae15ccf to
83c6e59
Compare
…in AIAgent (NousResearch#81209) The fallback resolver noted in its docstring that the fallback entry's model would be carried into the runtime dict (same as the gateway does at run.py:2540), but never actually did so. The returned runtime was the raw provider dict, so AIAgent still received the *primary* model — meaning a oneshot invocation that fell back to e.g. Anthropic credentials would still be asked for the primary provider's model (likely failing or mis-routing). This follow-up: - Passes `target_model=entry.get('model')` to `resolve_runtime_provider` for each fallback entry so the resolver's api_mode is derived correctly. - Stamps `runtime['model'] = entry['model']` on the returned dict so AIAgent sees the fallback's model. - Honors `runtime.get('model')` in `_run_agent` (falling back to `effective_model` on primary success). - Updates the two existing assertions to compare by content rather than identity (the helper now returns a fresh dict with the model field). - Adds a regression test asserting the fallback model is injected.
Contributor
Author
|
Fix follow-up: the original PR's helper docstring promised to stamp the fallback entry's This follow-up:
6/6 tests pass locally. |
…search#81209) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #81209
Root cause
The gateway has
_try_resolve_fallback_provider(gateway/run.py:2502) which resolves a fallback provider at resolution time — beforeAIAgentis constructed. A gateway turn therefore survives a primary-provider quota window whenfallback_providersis configured and healthy.The CLI/oneshot lane had no equivalent.
hermes_cli/oneshot.pycalledresolve_runtime_provider(...)bare at line 385. When the credential pool is exhausted, that call raises beforeAIAgentis constructed, so thefallback_model=_fbwiring atoneshot.py:436— which handles mid-session failures — never gets a chance. The result matches the issue's reproduction:The docs ("Where Fallback Works: CLI sessions ✔") read as full coverage, but in practice the CLI was only covered for failures that occur after the session starts. Headless deployments are hit hardest: cron jobs and queue workers that shell out to
hermes -zdied for the entire quota window even with a healthy fallback configured.Fix
New helper
_resolve_runtime_with_fallbackinhermes_cli/oneshot.py, called from_run_agentin place of the bareresolve_runtime_providercall:get_fallback_chain(cfg)in order — the same single source of truth the gateway uses — resolving each entry with the sameresolve_entry_api_keysemantics.The existing
fallback_model=_fbwiring is untouched and still handles mid-session failures; this change only adds the resolution-time lane the gateway already had.Regression tests
5 tests in
tests/hermes_cli/test_oneshot_fallback.py:test_primary_success_short_circuits— primary success must not touch fallback entries.test_primary_quota_failure_invokes_fallback— the issue's scenario: 429 → fallback entry resolves → agent proceeds.test_primary_failure_no_fallback_chain_returns_primary_error— no chain configured: primary error surfaces unchanged.test_all_fallbacks_exhausted_returns_primary_error— primary + every entry fails: primary error wins.test_second_fallback_succeeds_when_first_also_fails— chain walk continues past a failed entry.All 5 are RED on pre-fix code (helper does not exist → AttributeError) and GREEN after. Existing
test_oneshot_usage_file.py(3 tests) continues to pass.Scope
One new helper + one call-site change in
oneshot.py; no refactor, no API surface change. The gateway's_try_resolve_fallback_provideris left as-is (the oneshot path needs the cfg-driven semantics this helper has; reusing the gateway function would import the whole gateway config loader).