fix(web): add runtime backend failover for search and extract

[sangjunseo613-bot]

Summary

• add runtime backend failover for web_search across Firecrawl, Exa, Tavily, and Parallel
• add runtime backend failover for web_extract with retryable provider-error classification
• preserve explicit web.backend pins; cross-provider failover only happens in auto-detect mode
• block policy-denied extract URLs before any backend dispatch
• document runtime failover behavior and add regression tests

Why

web_search/web_extract already supported multiple backends, but selected a single provider and returned errors immediately when that provider hit retryable runtime failures (for example Firecrawl 402 Insufficient credits). This patch makes the web tools materially more resilient while preserving user/provider-intent boundaries.

Behavior

• Retryable provider failures (402, quota/rate-limit, transient upstream/network failures) can fail over to another available backend.
• Non-retryable request/user errors (invalid parameters, malformed requests, auth/permission style errors) do not silently fail over.
• If web.backend is explicitly configured, Hermes honors that provider pin and does not switch providers.
• For extract, URLs blocked by website policy are filtered before backend dispatch so they never reach fallback providers.

Validation

• python -m pytest tests/tools/test_web_tools_config.py tests/tools/test_web_tools_tavily.py tests/tools/test_web_tools_failover.py -q
• python -m py_compile tools/web_tools.py tests/tools/test_web_tools_failover.py
• independent fresh-context review: passed after tightening explicit backend semantics, policy prefiltering, and docs
• live local verification with Firecrawl credits exhausted and Exa available:
  • web_search logged firecrawl failed ... retrying with exa and returned successful results
  • web_extract logged firecrawl failed ... retrying with exa and returned successful extraction

Notes

• Added regression coverage for retryable vs non-retryable failover, explicit backend stickiness, structured all-error extract failover, and policy-blocked URLs never reaching backend dispatch.

[teknium1]

Thanks for the resilience work. The underlying runtime-failover gap remains on current main: tools/web_tools.py:667-704 invokes one selected search provider, and tools/web_tools.py:821-907 invokes one selected extract provider; their outer handlers return provider exceptions rather than trying another provider.

Problems

• The implementation targets the retired inline dispatch model. Current main was cut over by b05253ceed5f9d139f4a7d8705f5c97fcf644a2c to agent.web_search_registry; the active dispatch paths use registered WebSearchProvider instances, including plugin providers and capability checks (tools/web_tools.py:656-704, tools/web_tools.py:821-907).
• The proposed fallback loop is hard-coded to four vendor names (tools/web_tools.py:136 in the PR), so salvaging it directly would bypass registered providers and their capability contract.

Suggested changes

• Rework failover around a registry-native, capability-aware candidate resolver, then exercise it from the existing search and extract dispatch paths.
• Add provider-level regression coverage for retryable failures, explicit/per-capability pins, sync/async extraction, and policy-blocked URLs.

Automated hermes-sweeper review.

[teknium1]

Current main dispatches through agent.web_search_registry and supports registered plugin providers plus capability checks. Please do not retain a hard-coded vendor list here when salvaging; build candidates from the registry filtered by search/extract capability and availability.