fix(web): add runtime fallback between configured web backends

[mbac]

What does this PR do?

This fix was created by DeepSeek V4 Pro and revised by GPT-5.5. I (@mbac) do not have the necessary programming skills to thoroughly verify this bugfix, so I kindly ask reviewers to be extra careful when examining the PR.

Web tools could previously select one configured backend for a request and return an error if that backend hit a runtime failure, even when another configured provider could satisfy the same operation. This made transient failures such as rate limits, timeouts, network errors, and 5xx responses unnecessarily visible to the agent.

The change adds runtime fallback for web backend dispatch. Search, extract, and crawl now build a prioritized candidate list, try the configured provider first when it is available, and move to the next usable provider when the current backend raises a runtime error. Backend-specific helpers keep response normalization in one place, and Firecrawl extraction now lets backend scrape failures reach the fallback dispatcher while still preserving policy-blocked URL rows so blocked URLs are not fetched through another provider.

Related Issue

N/A

Related work: #11331 covers a similar runtime failover idea for search and extract. This PR applies the fallback structure to the current backend-candidate implementation, includes crawl dispatch, and includes a Firecrawl extraction regression fix.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Security fix
• Documentation update
• Tests (adding or improving test coverage)
• Refactor (no behavior change)
• New skill (bundled or hub)

Changes Made

• Add prioritized backend candidate discovery for Firecrawl, Parallel, Tavily, and Exa.
• Add sync and async fallback dispatch helpers for web backend operations.
• Extract provider-specific search, extract, and crawl helpers so fallback dispatch can share result normalization.
• Preserve configured-backend availability checks while allowing runtime fallback across other configured providers.
• Propagate Firecrawl scrape timeouts and SDK/HTTP failures so extraction can fall back to the next provider.
• Strengthen fallback tests to assert actual Exa results and add coverage for Firecrawl extract failure fallback.

How to Test

1. Configure multiple web providers, for example Firecrawl plus Exa or Tavily plus Exa.
2. Simulate or encounter a runtime failure in the preferred provider, such as a 5xx, timeout, or rate limit.
3. Confirm web_search or web_extract returns results from the next configured backend instead of immediately returning the first backend error.
4. Run the focused regression suite:

scripts/run_tests.sh tests/tools/test_web_backend_fallback.py tests/tools/test_web_tools_config.py tests/tools/test_web_tools_tavily.py tests/tools/test_website_policy.py

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run the full pytest tests/ -q suite and all tests pass
• I've added tests for my changes
• I've tested on my platform: macOS / Darwin with Python 3.11

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) - N/A
• I've updated cli-config.yaml.example if I added/changed config keys - N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows - N/A
• I've considered cross-platform impact - no file I/O, terminal, or process-management behavior changed
• I've updated tool descriptions/schemas if I changed tool behavior - N/A

Screenshots / Logs

Focused verification passed locally:

102 passed in 3.06s