feat(web): add 9 search providers — serper, baidu, bocha, qiniu-baidu, serpapi, jina, google-cse, sogou, 360-search by Icather · Pull Request #53149 · NousResearch/hermes-agent

Icather · 2026-06-26T16:09:23Z

What does this PR do?

v0.17 removed the Parallel free MCP fallback (#46350), sharply reducing the free search options available to Hermes users. This adds 9 new web search providers as bundled plugins, registered after existing backends in composite quality-score order. Two prior attempts (#41015 by ViezeVingertjes, #35690 by jonathanwxh-cell) were self-closed without review — this salages their approach with a complete provider surface.

Related Issue

Fixes the gap left by #46350. See #41015 and #35690 for prior art.

Type of Change

✨ New feature (non-breaking change that adds functionality)

Changes Made

New functional providers (search-only)

plugins/web/serper/ — Google SERP, 2,500 free queries, no credit card
plugins/web/baidu/ — Baidu AI Search, 100/day free, native Chinese content
plugins/web/bocha/ — Chinese market leader, 1,000 free starter queries
plugins/web/qiniu-baidu/ — Qiniu Cloud Baidu, 3M tokens for new users
plugins/web/serpapi/ — Multi-engine (Google/Bing/Baidu), 100/month free
plugins/web/jina/ — Full-page extraction, 10M tokens free (blocked in China)
plugins/web/google-cse/ — Google Custom Search, 100/day (blocked in China)

Registry-only entries (no public API, listed for completeness)

plugins/web/sogou/ — Sogou (搜狗)
plugins/web/_360_search/ — 360 Search (好搜)

Tests added

tests/plugins/web/test_new_providers.py — 10 tests: provider discovery, availability, HTTP errors, search success, registry stubs, contract checks, de-hardcoding verification

Modified

tools/web_tools.py — +9 entries in backend_candidates tuple, +_check_provider_available() helper, +_get_registered_backend_names() (dynamic from plugin registry — no hardcoded list), generic plugin probe in _is_backend_available()

How to Test

Set SERPER_API_KEY (free signup at https://serper.dev)
hermes tools → serper should appear in web search backends
Start a session and call web_search — results should come from Serper

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits
I searched for existing PRs (feat(web): add Serper search provider plugin #41015, fix: restore search fallback routing #35690)
My PR contains only changes related to this feature
I've run pytest tests/ -q and all tests pass
I've added tests for my changes (tests/plugins/web/test_new_providers.py)
I've tested on my platform: Windows 10

Documentation & Housekeeping

I've updated relevant documentation — or N/A (no docs keys changed)
I've updated cli-config.yaml.example — or N/A
I've considered cross-platform impact (Windows, macOS) — or N/A (search-only, platform-agnostic HTTP)

Copilot

Pull request overview

This PR aims to restore/expand Hermes’ “free-ish” web search options after the removal of the keyless Parallel fallback by adding 9 new web search provider plugins and wiring them into tools/web_tools.py backend selection.

Changes:

Added 9 new bundled kind: backend web provider plugins (7 functional + 2 registry-only “unavailable” entries).
Updated tools/web_tools.py to recognize these backends in configured values and in the auto-detect fallback order.
Added a generic “plugin provider availability” probe path in _is_backend_available().

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
tools/web_tools.py	Adds the new backend names to config validation, fallback candidate ordering, and plugin-based availability probing.
plugins/web/serper/provider.py	Implements Serper search provider logic and result normalization.
plugins/web/serper/plugin.yaml	Declares bundled backend manifest for Serper.
plugins/web/serper/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/baidu/provider.py	Implements Baidu search provider logic and result normalization.
plugins/web/baidu/plugin.yaml	Declares bundled backend manifest for Baidu.
plugins/web/baidu/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/bocha/provider.py	Implements Bocha search provider logic and result normalization.
plugins/web/bocha/plugin.yaml	Declares bundled backend manifest for Bocha.
plugins/web/bocha/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/qiniu-baidu/provider.py	Implements Qiniu-wrapped Baidu search provider logic and result normalization.
plugins/web/qiniu-baidu/plugin.yaml	Declares bundled backend manifest for Qiniu Baidu.
plugins/web/qiniu-baidu/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/serpapi/provider.py	Implements SerpAPI search provider logic and result normalization.
plugins/web/serpapi/plugin.yaml	Declares bundled backend manifest for SerpAPI.
plugins/web/serpapi/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/jina/provider.py	Implements Jina search provider logic and result normalization.
plugins/web/jina/plugin.yaml	Declares bundled backend manifest for Jina.
plugins/web/jina/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/google-cse/provider.py	Implements Google CSE search provider logic and result normalization.
plugins/web/google-cse/plugin.yaml	Declares bundled backend manifest for Google CSE.
plugins/web/google-cse/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/sogou/provider.py	Adds a registry-only “unavailable” Sogou provider that always returns an error.
plugins/web/sogou/plugin.yaml	Declares bundled backend manifest for Sogou.
plugins/web/sogou/init.py	Plugin module entrypoint (currently missing provider registration).
plugins/web/_360_search/provider.py	Adds a registry-only “unavailable” 360 Search provider that always returns an error.
plugins/web/_360_search/plugin.yaml	Declares bundled backend manifest for 360 Search.
plugins/web/_360_search/init.py	Plugin module entrypoint (currently missing provider registration).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ider priority New subcommand: hermes tools web reorder Shows the current web.fallback_backends list with numbered indices. User enters a space-separated reorder (e.g. '3 1 2 4 5'). Validated for completeness (must include each number exactly once). Writes result back to config.yaml as web.fallback_backends. This gives users a simple way to prioritize search engines after adding new providers via feat/add-free-web-search-providers (PR NousResearch#53149).

…u-baidu, serpapi, jina, google-cse, sogou, 360-search) v0.17 removed the Parallel free MCP fallback (NousResearch#46350), sharply reducing free search options. This adds 9 new providers as plugins following the existing brave-free/ddgs pattern, registered in backend_candidates in composite quality-score order after the existing backends. New functional providers (search-only): - serper — Google SERP, 2,500 free queries, no credit card - baidu — Baidu AI Search, 100/day free, native Chinese content - bocha — Chinese market leader, 1,000 free starter queries - qiniu-baidu — Qiniu Cloud Baidu, 3M tokens for new users - serpapi — Multi-engine, 100/month free - jina — Full-page extraction, 10M tokens free (blocked in China) - google-cse — Google Custom Search, 100/day (blocked in China) Registry-only entries (no public API, listed for completeness): - sogou, 360-search Two prior attempts (NousResearch#41015 Serper by ViezeVingertjes, NousResearch#35690 fallback by jonathanwxh-cell) were self-closed without review. This salages their approach with a complete provider surface covering both Western and Chinese search engines. Changes: - plugins/web/*/ — 27 new files (provider.py, plugin.yaml, __init__.py per backend) - tools/web_tools.py — +9 entries in _KNOWN_WEB_BACKENDS set, +9 in backend_candidates tuple, +_check_provider_available() helper, generic plugin probe in _is_backend_available()

Covers: provider availability, missing key, HTTP errors, search success, registry-only stubs (sogou/360), contract checks, and de-hardcoding verification.

…httpx, fix Google CSE dual-credential gap Addresses all 13 Copilot review comments on PR NousResearch#53149: P0 — Missing register(ctx) entrypoints: All 9 provider __init__.py files now define register(ctx) that calls ctx.register_web_search_provider(ProviderClass()), matching the brave-free/ddgs pattern. Without this, PluginManager skips discovery. Medium — Google CSE dual credential: is_available() now requires both GOOGLE_CSE_API_KEY and GOOGLE_CSE_CX. get_setup_schema() advertises both env vars for hermes tools setup. Style — httpx consistency: All 7 functional providers switched from requests to httpx, matching the existing bundled provider pattern (brave-free, tavily, etc.). Test — Plugin registration validation: Added TestPluginRegistration class verifying all __init__.py files have register(ctx) and register_web_search_provider calls. Updated Serper tests to mock httpx instead of requests.

…u for Python import compatibility Python cannot import from directories with hyphens. google-cse/ and qiniu-baidu/ were renamed to google_cse/ and qiniu_baidu/ to match the module paths in their __init__.py imports.

The Baidu search API is accessible via qianfan.baidubce.com/v2/ai_search/chat/completions using the OpenAI-compatible chat format. The old websearch endpoint returned 400. Also added proxy=None since qianfan is a domestic endpoint that should bypass the system HTTPS_PROXY.

Icather · 2026-06-27T15:18:24Z

补充修复（2026-06-27 排查结论）

本地分支：fix/web-search-provider-discovery-cache（已推送到 Icather/hermes-agent）

问题发现

9 个新 provider 目录结构、plugin.yaml、register() 入口全部正确——跟 8 个老 provider 一模一样。但在实际运行中，provider 注册时有时无，导致 web_search 在 TUI 交互模式下不可用（-z 单次模式正常）。

根因 1：PluginManager `_discovered` 标志导致扫描被跳过

model_tools.py 和 gateway/run.py 调用 discover_plugins() 时不带 force=True。插件管理器首次扫描后设 _discovered = True，后续调用直接 return。在 TUI 路径下（不传 -t，enabled_toolsets=None），_ensure_web_plugins_loaded() → _ensure_plugins_discovered(force=False) → 命中 _discovered=True → 不扫描 → 新 provider 永不被注册。

修复（commit f9a2df0a3）：

model_tools.py:202 → discover_plugins(force=True)
gateway/run.py:5882 → discover_plugins(force=True)

根因 2：`_LEGACY_PREFERENCE` 不含新 provider

web_search_registry.py 的 _LEGACY_PREFERENCE 元组只列了 7 个老 provider（firecrawl→parallel→tavily→exa→searxng→brave-free→ddgs）。当 web.backend 未显式配置时，自动检测回退走这个顺序，新 provider 永远不被考虑。

修复（commit 81917cf3c）：

agent/web_search_registry.py:_LEGACY_PREFERENCE 追加 9 个新 provider
tools/web_tools.py:_get_backend() → backend_candidates 追加 9 个新 provider

根因 3：`_is_backend_available` 硬编码列表缺项

已在原 PR 中修复（generic get_provider().is_available() probe）。无需追加。

完整的修改列表

model_tools.py          — discover_plugins(force=True)
gateway/run.py          — discover_plugins(force=True)
tools/web_tools.py      — _is_backend_available generic probe + _get_backend candidates 扩展
agent/web_search_registry.py — _LEGACY_PREFERENCE 扩展

Icather · 2026-06-27T16:42:51Z

✅ 验证通过（2026-06-28）

TUI 交互模式中 web_search 现已正常工作。最终确认的完整修复清单：

代码层（4 处）

文件	修复	解决
`tools/web_tools.py`	`_is_backend_available` 加 generic `get_provider().is_available()` probe	新 provider 名不在 8 个硬编码列表中
`model_tools.py:202`	`discover_plugins(force=True)`	import 时插件管理器 `_discovered` 标志导致扫描被跳过
`gateway/run.py:5882`	`discover_plugins(force=True)`	同上，网关路径
`agent/web_search_registry.py` + `tools/web_tools.py`	`_LEGACY_PREFERENCE` 和 `_get_backend` 候选列表追加 9 个新 provider	自动回退时新 provider 不可见

环境层（1 处）

用户 PATH 中 Python 3.13 的 hermes.exe（旧的工作区可编辑安装）排在生产 venv 前面 → 实际运行的代码不含修复。已将生产 venv 路径提到 system PATH 最前面解决。

该 PR 代码层修复已验证，可 review。

teknium1 · 2026-07-15T05:54:16Z

Thanks for the substantial provider implementations and for tracing the registry-discovery behavior.

This automated hermes-sweeper review is closing this under the standing third-party integration policy:

The PR adds vendor-specific search integrations under plugins/web/ (for example plugins/web/serper/provider.py and plugins/web/baidu/provider.py). Third-party product integrations are maintained as standalone plugins rather than bundled into this repository.
Current tools/web_tools.py:176-209 and tools/web_tools.py:255-269 already support registry-backed custom web providers, so these providers can use the existing extension surface without adding core-tree vendor modules.
Please publish the providers as a standalone plugin repository installable under ~/.hermes/plugins/ or through a pip entry point; it can then be promoted in #plugins-skills-and-skins.

Closed as not-planned per standing maintainer policy (in-tree-provider-integration). This is a design-direction decision, not a code-quality judgment — see the Contribution Rubric in AGENTS.md for what the project is looking for. If you believe this policy was misapplied to your change, comment here and a maintainer will take a look.

Copilot AI review requested due to automatic review settings June 26, 2026 16:09

Copilot started reviewing on behalf of Icather June 26, 2026 16:09 View session

This was referenced Jun 26, 2026

feat(web): multi-source fallback chain and search_engine parameter #53151

Closed

feat(tools): hermes tools web reorder — interactive search provider priority #53152

Closed

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Icather marked this pull request as draft June 26, 2026 16:16

alt-glitch added type/feature New feature or request comp/tools Tool registry, model_tools, toolsets tool/web Web search and extraction P3 Low — cosmetic, nice to have labels Jun 26, 2026

Icather force-pushed the feat/add-free-web-search-providers branch from 551de72 to 6ef06e9 Compare June 26, 2026 16:25

Icather added 2 commits June 27, 2026 00:43

test(web): add unit tests for 9 new web search providers

edb5151

Covers: provider availability, missing key, HTTP errors, search success, registry-only stubs (sogou/360), contract checks, and de-hardcoding verification.

Icather marked this pull request as ready for review June 26, 2026 17:02

Icather added 2 commits June 27, 2026 01:18

Pauliehedron mentioned this pull request Jul 3, 2026

[Bug]: check_web_api_key() hardcodes built-in backends — third-party web search plugins silently disabled #31873

Closed

1 task

teknium1 closed this Jul 15, 2026

mijiu1807-ops mentioned this pull request Aug 19, 2026

[Feature] Smart web search routing with result cleaning/ranking + link click → external browser by default #89626

Open

Navigation Menu

Uh oh!

feat(web): add 9 search providers — serper, baidu, bocha, qiniu-baidu, serpapi, jina, google-cse, sogou, 360-search - #53149

feat(web): add 9 search providers — serper, baidu, bocha, qiniu-baidu, serpapi, jina, google-cse, sogou, 360-search#53149
Icather wants to merge 5 commits into
NousResearch:mainfrom
Icather:feat/add-free-web-search-providers

Icather commented Jun 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Icather commented Jun 27, 2026

Uh oh!

Icather commented Jun 27, 2026

Uh oh!

teknium1 commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Icather commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related Issue

Type of Change

Changes Made

New functional providers (search-only)

Registry-only entries (no public API, listed for completeness)

Tests added

Modified

How to Test

Checklist

Code

Documentation & Housekeeping

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Icather commented Jun 27, 2026

补充修复（2026-06-27 排查结论）

问题发现

根因 1：PluginManager _discovered 标志导致扫描被跳过

根因 2：_LEGACY_PREFERENCE 不含新 provider

根因 3：_is_backend_available 硬编码列表缺项

完整的修改列表

Uh oh!

Icather commented Jun 27, 2026

✅ 验证通过（2026-06-28）

代码层（4 处）

环境层（1 处）

Uh oh!

teknium1 commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Icather commented Jun 26, 2026 •

edited

Loading

根因 1：PluginManager `_discovered` 标志导致扫描被跳过

根因 2：`_LEGACY_PREFERENCE` 不含新 provider

根因 3：`_is_backend_available` 硬编码列表缺项