Skip to content

fix(web): make plugin-provided web backends selectable - #52057

Closed
ZOLAtheCodeX wants to merge 1 commit into
NousResearch:mainfrom
ZOLAtheCodeX:fix/plugin-web-backend-selectable
Closed

fix(web): make plugin-provided web backends selectable#52057
ZOLAtheCodeX wants to merge 1 commit into
NousResearch:mainfrom
ZOLAtheCodeX:fix/plugin-web-backend-selectable

Conversation

@ZOLAtheCodeX

Copy link
Copy Markdown

Problem

agent/web_search_provider.py documents a user / third-party web-provider
plugin surface: a provider subclasses WebSearchProvider, registers via
PluginContext.register_web_search_provider(), and is selected through
web.search_backend / web.extract_backend / web.backend.

In practice that surface is only half-wired. tools/web_tools.py::_is_backend_available()
gates selection on a hardcoded allowlist:

if backend == "exa":   ...
if backend == "parallel": ...
# firecrawl / tavily / searxng / brave-free / ddgs / xai
return False

Any backend name not in that list returns False. So a plugin provider is
discovered and dispatchable (the registry knows it), but can never be
selected: _get_capability_backend() sees _is_backend_available(name) is
False and falls through to the auto-detect cascade, picking some other
backend. For an extract-only plugin this surfaces as a misleading
"search-only backend" or "no extract provider configured" error even though
the provider is installed, enabled, and reports is_available() == True.

Fix

Add a generic fallback at the end of _is_backend_available(): when the name
isn't one of the built-ins, defer to the web registry's own provider
is_available(). Plugin discovery is triggered first because this gate can
run before the dispatcher's own _ensure_web_plugins_loaded() (in the
extract path, _get_extract_backend() is called before discovery).

    try:
        _ensure_web_plugins_loaded()
        from agent.web_search_registry import get_provider
        prov = get_provider(backend)
        if prov is not None:
            return prov.is_available()
    except Exception:
        pass
    return False

No per-provider hardcoding — it works for any backend registered through the
documented plugin path. Built-in backends are unaffected: they return from
their existing if backend == ... branches before reaching the new code.

Testing

Verified with a local Crawl4AI extract-only provider installed at
~/.hermes/plugins/web/crawl4ai/ and enabled via plugins.enabled:

  • Before: web.extract_backend: crawl4ai_get_extract_backend() returned
    searxng (cascade fallback); web_extract failed.
  • After: _get_extract_backend() returns crawl4ai; a live extract returns
    clean markdown. Built-in backends (firecrawl with a real key, searxng,
    ddgs) continue to resolve and run unchanged.

Scope

One file, +13 lines, no behavior change for existing backends.


🤖 Generated with Claude Code

`_is_backend_available()` gated backend selection on a hardcoded allowlist
(exa/parallel/firecrawl/tavily/searxng/brave-free/ddgs/xai) and returned
False for anything else. This silently broke the user / third-party
web-provider plugin surface documented in agent/web_search_provider.py:
a provider registered via `register_web_search_provider()` is discovered
and dispatchable, but could never be *selected* through
`web.search_backend` / `web.extract_backend` because this gate rejected
its name, so selection fell through to the auto-detect cascade instead.

Add a generic fallback: when the name isn't a built-in, defer to the web
registry's own provider `is_available()`. Plugin discovery is triggered
first because this gate can run before the dispatcher's
`_ensure_web_plugins_loaded()`. No per-provider hardcoding — works for any
registered plugin backend (verified with a Crawl4AI extract provider).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets tool/web Web search and extraction duplicate This issue or pull request already exists labels Jun 24, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #36987 — that earlier-open PR adds the same web_search_registry.get_provider(...).is_available() fallback at the end of _is_backend_available() so plugin-registered web backends become selectable. Also overlaps #35327 (same allowlist gate). Flagging the cluster so a maintainer can pick one.

@ZOLAtheCodeX

Copy link
Copy Markdown
Author

Thanks for flagging — agreed, this duplicates #36987 and is covered more fully by #35327 (which also adds tests and fixes the related search-only dead-end). Closing in favor of those.

One small thing that might be worth folding into whichever lands: this version calls _ensure_web_plugins_loaded() before the registry lookup, since _get_extract_backend() can run before discovery in subagent/subprocess paths — without it, the registry can still be empty at gate-check time. Either way, glad to see this getting fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have tool/web Web search and extraction type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants