Skip to content

fix(oneshot): consult fallback_providers at resolution time - #81517

Open
Enough1122 wants to merge 3 commits into
NousResearch:mainfrom
Enough1122:fix/81209-oneshot-fallback
Open

fix(oneshot): consult fallback_providers at resolution time#81517
Enough1122 wants to merge 3 commits into
NousResearch:mainfrom
Enough1122:fix/81209-oneshot-fallback

Conversation

@Enough1122

Copy link
Copy Markdown
Contributor

Fixes #81209

Root cause

The gateway has _try_resolve_fallback_provider (gateway/run.py:2502) which resolves a fallback provider at resolution time — before AIAgent is constructed. A gateway turn therefore survives a primary-provider quota window when fallback_providers is configured and healthy.

The CLI/oneshot lane had no equivalent. hermes_cli/oneshot.py called resolve_runtime_provider(...) bare at line 385. When the credential pool is exhausted, that call raises before AIAgent is constructed, so the fallback_model=_fb wiring at oneshot.py:436 — which handles mid-session failures — never gets a chance. The result matches the issue's reproduction:

$ hermes -z "Health check: reply pong."
hermes -z: agent failed: Codex provider quota exhausted (429); retry after 39750s.

The docs ("Where Fallback Works: CLI sessions ✔") read as full coverage, but in practice the CLI was only covered for failures that occur after the session starts. Headless deployments are hit hardest: cron jobs and queue workers that shell out to hermes -z died for the entire quota window even with a healthy fallback configured.

Fix

New helper _resolve_runtime_with_fallback in hermes_cli/oneshot.py, called from _run_agent in place of the bare resolve_runtime_provider call:

  1. Try the primary provider as before.
  2. On failure, walk get_fallback_chain(cfg) in order — the same single source of truth the gateway uses — resolving each entry with the same resolve_entry_api_key semantics.
  3. On total failure, re-raise the primary error: it names the operator's configured primary provider, which is what needs fixing first (a fallback entry's failure is not the operator's configured-first provider).

The existing fallback_model=_fb wiring is untouched and still handles mid-session failures; this change only adds the resolution-time lane the gateway already had.

Regression tests

5 tests in tests/hermes_cli/test_oneshot_fallback.py:

  1. test_primary_success_short_circuits — primary success must not touch fallback entries.
  2. test_primary_quota_failure_invokes_fallback — the issue's scenario: 429 → fallback entry resolves → agent proceeds.
  3. test_primary_failure_no_fallback_chain_returns_primary_error — no chain configured: primary error surfaces unchanged.
  4. test_all_fallbacks_exhausted_returns_primary_error — primary + every entry fails: primary error wins.
  5. test_second_fallback_succeeds_when_first_also_fails — chain walk continues past a failed entry.

All 5 are RED on pre-fix code (helper does not exist → AttributeError) and GREEN after. Existing test_oneshot_usage_file.py (3 tests) continues to pass.

Scope

One new helper + one call-site change in oneshot.py; no refactor, no API surface change. The gateway's _try_resolve_fallback_provider is left as-is (the oneshot path needs the cfg-driven semantics this helper has; reusing the gateway function would import the whole gateway config loader).

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard area/auth Authentication, OAuth, credential pools area/usage-cost Token accounting, usage reporting, billing, cost tracking P2 Medium — degraded but workaround exists needs-decision Awaiting maintainer decision before any implementation sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 8, 2026
…arch#81209)

The gateway already had `_try_resolve_fallback_provider` that runs at
provider-resolution time, before `AIAgent` is constructed — so a
gateway turn survives a primary-provider quota window with a healthy
`fallback_providers` chain. The CLI/oneshot lane was bare: a
`hermes -z` invocation during a 429 window raised
`resolve_runtime_provider(...)` outright, never reaching the
`fallback_model=_fb` wiring in `AIAgent(...)` that handles
mid-session failures. The docs claim "Where Fallback Works: CLI
sessions ✔" but in practice the CLI was only covered for failures
after the session started.

Fix: extract the fallback loop into
`hermes_cli.oneshot._resolve_runtime_with_fallback` and call it in
place of the bare `resolve_runtime_provider` call in `_run_agent`.
On total failure (primary + every fallback), the primary error is
re-raised — it names the operator's configured primary provider,
which is what the operator needs to fix first.

Adds 5 regression tests covering: primary success short-circuit,
primary 429 → fallback success, primary failure with no chain,
primary failure with every entry also failing, and first-fallback
failure → second-fallback success.
@Enough1122
Enough1122 force-pushed the fix/81209-oneshot-fallback branch from ae15ccf to 83c6e59 Compare August 8, 2026 05:52
…in AIAgent (NousResearch#81209)

The fallback resolver noted in its docstring that the fallback entry's
model would be carried into the runtime dict (same as the gateway does at
run.py:2540), but never actually did so. The returned runtime was the raw
provider dict, so AIAgent still received the *primary* model — meaning a
oneshot invocation that fell back to e.g. Anthropic credentials would
still be asked for the primary provider's model (likely failing or
mis-routing).

This follow-up:
- Passes `target_model=entry.get('model')` to `resolve_runtime_provider`
  for each fallback entry so the resolver's api_mode is derived correctly.
- Stamps `runtime['model'] = entry['model']` on the returned dict so
  AIAgent sees the fallback's model.
- Honors `runtime.get('model')` in `_run_agent` (falling back to
  `effective_model` on primary success).
- Updates the two existing assertions to compare by content rather than
  identity (the helper now returns a fresh dict with the model field).
- Adds a regression test asserting the fallback model is injected.
@Enough1122

Copy link
Copy Markdown
Contributor Author

Fix follow-up: the original PR's helper docstring promised to stamp the fallback entry's model onto the returned runtime dict (same as the gateway does at run.py:2540), but never actually did. AIAgent received the primary model, so a oneshot that fell back to e.g. Anthropic credentials would still be asked for the primary provider's model.

This follow-up:

  • Passes target_model=entry.get('model') to resolve_runtime_provider for each fallback entry.
  • Stamps runtime['model'] = entry['model'] on the returned dict so AIAgent sees the fallback's model.
  • Honors runtime.get('model') in _run_agent (falls back to effective_model on primary success).
  • Adds a regression test asserting the fallback model is injected.

6/6 tests pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/auth Authentication, OAuth, credential pools area/usage-cost Token accounting, usage reporting, billing, cost tracking comp/cli CLI entry point, hermes_cli/, setup wizard needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: CLI/oneshot gets no resolution-time fallback — hermes -z dies for the whole quota window even with fallback_providers configured

2 participants