fix(web): add crawl4ai to _is_backend_available + _get_backend chain

[reneil1337]

Summary

web.extract_backend: crawl4ai now actually routes to the bundled Crawl4AI plugin instead of being silently ignored. Before this, the setting fell through to the searxng default and surfaced a misleading "SearXNG is a search-only
backend" error — even with CRAWL4AI_URL + CRAWL4AI_API_TOKEN configured correctly.

Root cause: the dispatcher in tools/web_tools.py enumerates every backend it knows about in two places, and crawl4ai was in neither. _is_backend_available() (the explicit-config path) has a chain of if backend == "..."
checks for exa, parallel, firecrawl, tavily, searxng, brave-free, ddgs, xai — but no crawl4ai case, so it returned False and the explicit config was discarded. _get_backend()'s backend_candidates tuple (the
legacy auto-detect/fallback path) likewise had no crawl4ai entry, so even a fully-credentialed setup could never auto-select it. The bundled plugin at plugins/web/crawl4ai/ was therefore dead code — present in the tree but unreachable
from the dispatcher. Confirmed by reading both functions on main; grep -n crawl4ai tools/web_tools.py returned zero matches before the fix.

Changes

• tools/web_tools.py — _is_backend_available(): add a crawl4ai case (after ddgs, before xai) gating on _has_env("CRAWL4AI_URL") and _has_env("CRAWL4AI_API_TOKEN"). Crawl4AI 0.9.0+ requires Bearer auth by default, so both the
  URL and the token must be present before the dispatcher will route /crawl calls — a URL alone is not enough.
• tools/web_tools.py — _get_backend(): append ("crawl4ai", _has_env("CRAWL4AI_URL") and _has_env("CRAWL4AI_API_TOKEN")) to the backend_candidates chain, placed last so it never shadows an existing backend and only activates when
  nothing earlier matched. Same dual-credential gate as above.

Two hunks, 5 lines added. No behavior change for any existing setup — the new branches are dead unless both crawl4ai env vars are set, so installs without crawl4ai credentials resolve exactly as before (the legacy fallback still lands
on firecrawl).

Validation

grep -n crawl4ai tools/web_tools.py → two matches (one per function):

171:        ("crawl4ai", _has_env("CRAWL4AI_URL") and _has_env("CRAWL4AI_API_TOKEN")),
234:    if backend == "crawl4ai":

Behavior of both helpers under each credential state:

Scenario	_is_backend_available("crawl4ai")	_get_backend()
No crawl4ai env	False	firecrawl (legacy default, unchanged)
CRAWL4AI_URL only	False (token missing)	firecrawl (unchanged)
CRAWL4AI_URL + CRAWL4AI_API_TOKEN	True	crawl4ai

Targeted suites pass with no regressions: tests/tools/test_web_tools_config.py, tests/tools/test_web_providers.py, tests/tools/test_web_providers_ddgs.py → 84 passed. No existing tests cover _is_backend_available/_get_backend
directly, and per the mechanical, well-isolated nature of the change none were added.

Migration

Backward-compatible and dormant by default. Users upgrading get crawl4ai support automatically once they set web.extract_backend: crawl4ai in config.yaml and provide CRAWL4AI_URL + CRAWL4AI_API_TOKEN. No config changes are
required for existing setups.

[teknium1]

Thanks for tracing the old hardcoded backend gate.

This is an automated hermes-sweeper review. The requested behavior is now implemented on main by the generic provider-registry fix merged as a9cd0e07cbe6f411c875d6cfd3c8c356ca90b121 (PR #57779):

• tools/web_tools.py:230-232 accepts a configured registered web provider.
• tools/web_tools.py:305-307 resolves an explicit web.extract_backend through the availability gate.
• tools/web_tools.py:323-327 delegates non-legacy backend availability to the registered provider's is_available().
• tests/tools/test_web_tools_config.py:776-813 covers custom-provider selection, availability, extract routing, and tool registration.

That generalized implementation covers a registered Crawl4AI provider without adding a vendor-specific branch, so this additive patch is redundant.