Add auxiliary same-provider model fallbacks

[getyolkboy]

What does this PR do?

Adds ordered same-provider fallback models for auxiliary LLM calls.

Today, auxiliary tasks like vision/compression/session search can be configured with an explicit provider/model, but if that selected model hits a retryable provider-side failure such as quota exhaustion, rate limiting, or temporary unavailability, the auxiliary call has no task-local ordered model chain to walk.

This PR adds auxiliary.<task>.fallback_models, allowing users to configure a provider-local ordered fallback list, for example:

yaml
auxiliary:
  vision:
    provider: google
    model: gemini-2.5-flash
    fallback_models:
      - gemini-2.5-flash-lite
      - gemini-3-flash-preview


The fallback stays within the same configured provider/client. That keeps behavior predictable, avoids silently switching credentials/providers, and lets users choose the exact model order they trust for auxiliary tasks.

Related Issue

No linked issue.

Fixes #

Type of Change

- [ ] 🐛 Bug fix (non-breaking change that fixes an issue)
- [x] ✨ New feature (non-breaking change that adds functionality)
- [ ] 🔒 Security fix
- [x] 📝 Documentation update
- [x] ✅ Tests (adding or improving test coverage)
- [ ] ♻️ Refactor (no behavior change)
- [ ] 🎯 New skill (bundled or hub)

Changes Made

- agent/auxiliary_client.py
  - Added support for ordered fallback_models on auxiliary task configuration.
  - Retries retryable auxiliary model failures against the configured fallback model chain.
  - Keeps fallbacks on the same provider/client instead of switching providers implicitly.
  - Preserves existing behavior when fallback_models is unset.

- cli-config.yaml.example
  - Documented the new fallback_models config key with examples for auxiliary task configuration.

- tests/agent/test_auxiliary_client.py
  - Added test coverage proving the full ordered fallback chain is walked.
  - Added async coverage for the same fallback behavior.

How to Test

1. Run the targeted auxiliary client test suite:

bash
python -m pytest tests/agent/test_auxiliary_client.py -q -o 'addopts='


2. Run related auxiliary/session-search/vision regression tests:

bash
python -m pytest \
  tests/agent/test_auxiliary_main_first.py \
  tests/agent/test_auxiliary_config_bridge.py \
  tests/agent/test_vision_resolved_args.py \
  tests/tools/test_session_search.py \
  -q -o 'addopts='


3. Validate compilation and config compatibility:

bash
python -m compileall -q agent/auxiliary_client.py tests/agent/test_auxiliary_client.py
hermes config check


Tested locally:

text
tests/agent/test_auxiliary_client.py: 101 passed
related auxiliary/session-search/vision tests: 71 passed
compileall: passed
hermes config check: passed


Checklist

Code

- [x] I've read the Contributing Guide
- [x] My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
- [x] I searched for existing PRs to make sure this isn't a duplicate
- [x] My PR contains only changes related to this fix/feature (no unrelated commits)
- [ ] I've run pytest tests/ -q and all tests pass
- [x] I've added tests for my changes (required for bug fixes, strongly encouraged for features)
- [x] I've tested on my platform: Linux Mint 22.3, Linux 6.14.0-37-generic x86_64

Documentation & Housekeeping

- [x] I've updated relevant documentation (README, docs/, docstrings) — or N/A
- [x] I've updated cli-config.yaml.example if I added/changed config keys — or N/A
- [x] I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
- [x] I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
- [x] I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

N/A — this PR does not add a skill.

Screenshots / Logs

text
$ python -m pytest tests/agent/test_auxiliary_client.py -q -o 'addopts='
101 passed

$ python -m pytest tests/agent/test_auxiliary_main_first.py tests/agent/test_auxiliary_config_bridge.py tests/agent/test_vision_resolved_args.py tests/tools/test_session_search.py -q -o 'addopts='
71 passed

$ python -m compileall -q agent/auxiliary_client.py tests/agent/test_auxiliary_client.py
passed

$ hermes config check
passed

[teknium1]

Thanks for the focused sync/async fallback coverage. Current main now has a shared per-task auxiliary fallback architecture, but it intentionally excludes a fallback whose provider matches the failed provider (agent/auxiliary_client.py:3911-3920). That means the same-provider/different-model use case in this PR is still not present, but it should be integrated as a policy decision in the existing chain rather than added as a parallel fallback path.

Problems

• Current fallback selection and dispatch live in agent/auxiliary_client.py:3889-3957, :6908-6928, and :7413-7433; the April patch predates those paths and cannot be salvaged mechanically.
• The current documented contract also says matching-provider entries are skipped (website/docs/user-guide/configuration.md:1095-1109). The change needs to define when a different model on the same provider is safe to treat as a distinct target.

Suggested changes

• Extend the shared fallback_chain selector to distinguish a different model/endpoint from the exact failed target, with sync and async regression coverage for ordered same-provider traversal and exact-target skipping.

Automated hermes-sweeper review.

[teknium1]

Current main now centralizes per-task fallback selection in _try_configured_fallback_chain, which deliberately skips candidates matching the failed provider (agent/auxiliary_client.py:3911-3920). Please integrate same-provider/different-model semantics into that shared policy rather than add a parallel fallback_models path, and preserve an explicit guard against retrying the identical failed target.

[GottZ]

This was generated by AI during triage.

Summary

Two PRs address same-provider auxiliary model fallback behavior. #17235 adds a separate fallback_models retry path for selected transient errors, while #59561 fixes the shared fallback_chain so sibling models on the failed provider remain eligible for model-specific failures without weakening provider-wide auth/payment handling.

Related pull requests

• #17235 related — (+276/-0) — close as duplicate of #59561: the patch adds a parallel fallback_models selector and sync/async dispatch path, but the later #59561 change implements the same-provider/different-model behavior inside the shared fallback architecture. Despite the keep_open review on #17235, its requested salvage path—integrating the policy into fallback_chain rather than retaining a parallel path—was subsequently implemented through #59561 and carried onto main via #72468.
• #59561 [closed] duplicate — (+199/-7) — relevant merged reference implementation via #72468: it narrows skipping to the exact failed (provider, model) for model-specific failures while retaining provider-wide skipping for auth and payment failures, with sync/async regression coverage. The automated keep_open verdict was superseded by the contributor's later confirmation that #59561's two commits were cherry-picked into #72468 with authorship preserved, so the closed PR remains the implementation reference.

Duplicates

#17235 and #59561 target substantially the same same-provider sibling-model fallback outcome, but through competing mechanisms; use the chain #17235 → duplicate of #59561 → incorporated into #72468.

Suggested consolidation

Close #17235 as duplicate of #59561, with the explicit chain that #59561 was incorporated into #72468. The later shared-chain implementation addresses the policy concern raised in the keep_open review on #17235 and preserves the important auth/payment provider-wide carve-out, whereas retaining #17235 would introduce a second fallback configuration and dispatch path; keep #59561 closed as the superseded implementation record.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    subgraph Dup17235 ["PRs duplicating each other"]
        P17235["PR #17235 (open)"]
        P59561["PR #59561 (closed)"]
    end
    class P17235 open
    class P59561 closed
    class P17235 target
    click P17235 "https://github.com/NousResearch/hermes-agent/pull/17235"
    click P59561 "https://github.com/NousResearch/hermes-agent/pull/59561"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 2 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 33 kB of PR diffs, 9 kB of issue/PR text, 3 kB of discussion (3 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.