feat(web): multi-source fallback chain and search_engine parameter

[Icather]

What does this PR do?

When a search provider fails, Hermes previously returned an error immediately — no retry with another backend. This adds a configurable fallback chain and a search_engine parameter so the model picks engines and Hermes retries when the configured backend is unavailable. Works with all currently-registered backends and auto-extends to new providers.

Rebased onto current main and updated for the keyless-rescue mechanism landed in the meantime.

Related Issue

Follows the intent of #35690 (closed without review) and extends it with dynamic engine names from the registry.

Type of Change

• ✨ New feature (non-breaking change that adds functionality)

Changes Made

• web_search_tool(query, limit, search_engine="auto") — new search_engine parameter; "auto" walks the fallback chain, explicit names run a single backend with no fallback
• _get_fallback_chain() — seeds from _get_search_backend() (per-capability web.search_backend override), then web.fallback_backends from config (user-ordered), then remaining registered providers
• _search_with_fallback() — skips backends that are unregistered or unavailable (missing key/package); a live backend that returns a failure result or raises stays authoritative (main's single-backend contract) with the one-shot keyless rescue integrated on both the explicit and auto paths
• Error envelope: when every backend is unavailable, the call raises through the top-level handler → {"error": "Error searching web: ..."} with a setup hint (e.g. FIRECRAWL_API_KEY). Per-backend diagnostics never leak into the public error string — they stay in the log and the model-facing _fallback_trace
• WEB_SEARCH_SCHEMA — search_engine property with enum built dynamically from _get_valid_engine_names() (currently-available providers + "auto"); registry handler forwards it only when supplied
• _get_registered_backend_names() / _get_valid_engine_names() — dynamically sourced from the plugin registry, no hardcoded lists
• tools/code_execution_tool.py — web_search sandbox stub synced with the new search_engine parameter

Tests

• tests/tools/test_web_fallback_chain.py — behavior tests for chain ordering, first-success return, skip-unavailable, authoritative-failure semantics, engine-name sets (deterministic rescue-off fixture)
• All previously-red CI checks (error envelope, limit clamp, handler wiring, schema drift, keyless rescue) now pass locally: 195 tests green

How to Test

1. Set BRAVE_SEARCH_API_KEY
2. Configure web.fallback_backends: [brave-free, ddgs] in config.yaml
3. Call web_search → should use brave-free
4. Unset BRAVE_SEARCH_API_KEY and pip install ddgs → should fall back to ddgs
5. Call web_search with search_engine: "ddgs" → pinned to ddgs only
6. With every backend unconfigured, the call returns {"error": "Error searching web: No web search backend available. Set a provider API key (e.g. FIRECRAWL_API_KEY) or run \hermes tools`."}`

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs (fix: restore search fallback routing #35690)
• My PR contains only changes related to this feature
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (tests/tools/test_web_fallback_chain.py)
• I've tested on my platform: Windows 11

Documentation & Housekeeping

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Icather]

New: fallback path transparency note (2026-06-28)

Added a _fallback_path field to _search_with_fallback results. When the auto chain steps through multiple backends, the response now includes a human-readable summary of the journey:

{
  "_fallback_path": "Search started with \"baidu\" (returned 0 results or failed) and ultimately succeeded via \"serper\".",
  "_fallback_trace": ["baidu: returned 0 results", "serpapi: not available (missing key)"],
  ...
}

The model can present this path to the user for transparency — no more silent "baidu returned 0, silently fell back to another engine."

Implementation is minimal: 18 lines in tools/web_tools.py:_search_with_fallback, building the note from the existing chain list and errors trace. Commit: de304fa10

[teknium1]

Thanks for the fallback work; current main still returns the selected provider’s response directly, so the underlying reliability gap is real (tools/web_tools.py:686-723).

Problems

• tools/web_tools.py:306 on this PR uses only web.backend as the auto-chain primary. Current main gives web.search_backend per-capability precedence (hermes_cli/config.py:1260-1264), so a separate search override would be ignored.
• tools/web_tools.py:317-324 assumes list_providers() returns registration order, but the registry sorts by provider name (agent/web_search_registry.py:78-82). The implicit chain is therefore alphabetical rather than the documented order.
• tools/web_tools.py:1484-1488 describes configured engine “choices,” but the schema is a free-form string and _get_valid_engine_names() is not used to create an enum.
• The source-text test at tests/tools/test_web_fallback_chain.py:177-188 is an implementation/change detector rather than a behavioral regression test.

Suggested changes

• Rework this atop current registry resolution, beginning the search chain from _get_search_backend() and preserving the disabled-plugin diagnostic covered in tests/tools/test_web_providers.py:599-623.
• Add behavioral coverage for divergent web.search_backend/web.backend values and deterministic fallback ordering.

This is an automated hermes-sweeper review.

[Icather]

Rebased onto current main. All four review items addressed:

• _get_fallback_chain() primary uses _get_search_backend() (respects per-capability web.search_backend override, works with divergent web.backend/web.search_backend configs)
• Dropped misleading "registration order" assumption — registry sorts alphabetically, chain is explicit
• _get_valid_engine_names() exposed via search_engine schema parameter
• Disabled-plugin diagnostic preserved in fallback exhaustion path
• Tests: de-hardcoding test removed, replaced with behavioral coverage for divergent backends, deterministic ordering, and provider availability gating

[GottZ]

This was generated by AI during triage.

Summary

Five PRs address or reference the provider-fallback problem. The supplied diffs verify that #53151 and #53158 add search-provider selection and runtime fallback, while #56916, #73631, and #74410 report broader search/extract-chain designs whose current diffs were not supplied and therefore cannot be compared at implementation level here.

Related pull requests

• #53151 [closed] duplicate — (+807/-29) — closed, still relevant as an earlier broad implementation: the supplied diff adds an automatic search fallback chain, explicit search_engine dispatch, and nine provider plugins, but its closure reason and any supersession relationship are not established by the evidence.
• #53158 related — (+471/-31) — keep open with a salvage path: the supplied diff contains the focused search fallback dispatcher and behavioral tests. Consistent with the contributor keep_open review on #53158, salvage the search-only chain after rebasing and resolving the reviewed configuration-precedence, deterministic-ordering, schema-exposure, and test-quality requirements against current main.
• #56916 [closed] related — (+455/-89) — closed, still relevant as a competing reported configuration design: its body describes capability-specific search and extraction chains with legacy scalar compatibility, but no diff or closure rationale is supplied, so neither implementation details nor supersession by another PR are verified.
• #73631 related — (+1861/-117) — keep open review noted, but current implementation is unverified here: the body reports explicit search/extract chains and per-URL extraction fallback. Despite the keep_open review on #73631 and the author's report that commit 80ea1efbc fixes schema exposure, no supplied diff or Verify verdict confirms that fix, so this PR cannot yet be selected as the consolidation base.
• #74410 related — (+695/-11) — keep open review noted, but current implementation is unverified here: the body reports primary-plus-fallback search/extract lists. The contributor keep_open review identifies that ordinary extraction error rows do not trigger fallback; without a supplied updated diff, that blocking contract issue remains unverified.

Duplicates

#53151 and #53158 substantially overlap on automatic search fallback and explicit engine selection, but the evidence does not establish a formal duplicate or supersession relationship. #56916, #73631, and #74410 overlap on configurable search/extract fallback chains, while their differing reported configuration contracts and extraction semantics prevent a verified duplicate designation without their diffs.

Suggested consolidation

Author action: rebase #53158 onto main, or split out its supplied, reviewable search-only fallback dispatcher and behavioral tests as the salvageable part. Do not select #73631 from the author-reported 80ea1efbc fix alone, and do not close #73631 or #74410 over their contributor keep_open reviews until updated diffs verify the schema-exposure fix and returned-error-row extraction fallback respectively; no additional duplicate closures are supported by the present evidence.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    subgraph Dup53151 ["PRs duplicating each other"]
        P53151["PR #53151 (closed)"]
        P53158["PR #53158 (open)"]
    end
    class P53151 closed
    class P53158 open
    class P53158 target
    click P53151 "https://github.com/NousResearch/hermes-agent/pull/53151"
    click P53158 "https://github.com/NousResearch/hermes-agent/pull/53158"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 5 pull requests and 0 issues in this complex. Diffs were read for 2 of 5 PRs (rest unavailable); Assessment working set: 64 kB of PR diffs, 13 kB of issue/PR text, 13 kB of discussion (14 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[GottZ]

This was generated by AI during triage.

Delta since our previous triage comment

@richkapp explicitly corrected the review chronology for #73631: the fix was pushed before the independent review, but a subsequent read-only review against exact head 80ea1efbc96fea9b80623bf89572bbbd14b9957b returned PUBLISH. That new report says the contributor-raised schema-exposure gap is resolved for both explicit chains and that GitHub CI is 38/38, strengthening—but not replacing—the required maintainer review because the updated diff is not supplied here.

Changed pull requests

• #73631 related — (+1861/-117) — consolidation candidate pending maintainer review: despite the visible keep_open review on #73631, @richkapp reports that exact-head review verified the requested check_web_api_key() schema exposure for both search_backends and extract_backends, with 38/38 CI checks passing; this addresses the reported blocker, though the fix is not independently diff-verifiable from this record.

Suggested consolidation

The consolidation direction is unchanged: retain #73631 as the preferred candidate, now with stronger exact-head evidence, and require normal maintainer review before merge.

Complex graph unchanged since our previous triage comment.

Cross-PR triage: Reviewed 5 pull requests and 0 issues in this complex. Diffs were read for 2 of 5 PRs (rest unavailable); Assessment working set: 64 kB of PR diffs, 13 kB of issue/PR text, 14 kB of discussion (15 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[Icather]

Thanks for the thorough review — rebased onto current main (f43eabee) and addressed everything. Also fixed the CI failures that were blocking the run.

Review items:

1. web.search_backend override bypassed — fixed. _get_fallback_chain() now seeds from _get_search_backend(), so a per-capability override is always first in the chain.
2. list_providers() order — the chain no longer relies on list_providers() ordering. Priority is now explicit: web.search_backend → web.fallback_backends (user-ordered) → remaining registered providers. Paid-first ordering belongs to fallback_backends, not to provider-name sorting.
3. Schema enum — search_engine now carries "enum": sorted(_get_valid_engine_names()), built dynamically at registration from currently-available providers (plus "auto").
4. Source-text assertion — replaced. The fallback-chain tests are now pure behavior tests (mocked registry/backends), including a rescue-off fixture so they're deterministic.

CI failures also fixed (all previously-red checks now green locally):

• Error envelope now matches main's contract: when every backend is unavailable, the call raises through the top-level handler → {"error": "Error searching web: ... <setup hint, e.g. FIRECRAWL_API_KEY>"}. Per-backend diagnostics never leak into the public error string — they stay in the log and the model-facing _fallback_trace.
• A live backend that returns a failure result or raises stays authoritative (single-backend contract) with the one-shot keyless rescue integrated on both the explicit and auto paths — no silent fall-through, no double-walk of the keyless ring.
• limit clamp and handler wiring tests pass; the web_search sandbox stub in code_execution_tool.py was synced so the schema-drift test is green.

Local run: 195 passed (web_fallback_chain, keyless rescue/fallback, web_tools_config, web_providers, registry, schema-drift). The only local skips are TestParallelClientConfig (needs parallel-web package, present in CI).