feat(anthropic): add native web search and fetch - #68337
Conversation
Co-authored-by: Adam Durham <amdnative@gmail.com>
alt-glitch
added
type/feature
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the focused native-transport implementation.
Problems
website/docs/user-guide/features/web-search.md:40says compatible third-party Anthropic endpoints “keep the normal client-side web tools.” The new adapter instead omits server-only tools on those endpoints, andtools/web_tools.pyreturns a local-execution error when the Anthropic backend is selected. The documentation should describe the actual unavailable/choose-another-backend behavior.
Suggested changes
- Correct that third-party-endpoint documentation sentence.
- Add a loop-level
pause_turntest.tests/agent/test_anthropic_web_server_tools.pycurrently verifies only normalization/mapping, while the new behavior inagent/conversation_loop.pyappends and persists messages, retries, and can activate fallback.
Automated hermes-sweeper review.
| `web_fetch` tools. The tools run inside the Messages API request, use the same | ||
| Anthropic credential as the model, and return source citations in Claude's | ||
| response. Compatible third-party Anthropic endpoints are not assumed to host | ||
| these tools; on those endpoints Hermes keeps the normal client-side web tools. |
There was a problem hiding this comment.
This contradicts the implementation: the new adapter omits server-only bindings on third-party Anthropic endpoints, and selecting web.backend: anthropic makes the local handlers return an error rather than fall back to client-side web tools. Please document that users must select another backend on those endpoints.
teknium1
added
sweeper:risk-session-state
The page claimed compatible third-party Anthropic endpoints keep the client-side web tools. They do not: the server-only binding is omitted from those requests, and the local handlers return an error because these tools only ever execute inside Anthropic's API. Document the actual behavior and point at the fix (select another backend).
The existing coverage stops at the transport: pause_turn survives map_finish_reason and normalization. The behavior that matters is in conversation_loop — it appends and persists the paused turn, replays the native blocks without inventing a user/tool message, bounds consecutive pauses, and hands the turn to the fallback chain before failing. These drive the real run_conversation against an in-process mock Messages endpoint, so the assertions are on observable turn behavior rather than internals. Only /v1/messages draws from the scripted response queue: the agent also POSTs probe endpoints, and letting those consume responses would shift each test's script by one and make results depend on probe timing rather than loop behavior. The fixture relies on the runner's per-file process isolation and on conftest's autouse HERMES_HOME redirect rather than purging sys.modules or managing its own tempdir: re-importing agent.transports mid-session builds a second, empty transport registry while modules imported earlier still hold the first, which makes get_transport return None for them. Each test was verified to fail against a deliberately broken loop: disabling the pause branch reds four of the five, removing the persist guard reds the persistence test, and removing the per-run counter reset reds the budget test.
Selecting web.backend anthropic while the model is served by anything else left the toolset registered and visible — in the startup banner, in `hermes tools`, in valid_tool_names and over the hermes-tools MCP export — while the transport stripped both tools from every request. The agent had no web capability and nothing said so; the handlers' own errors were unreachable, because a tool that is never advertised is never called. Availability now accounts for the endpoint, not just the credential: the native tools execute inside Anthropic's Messages API, so a key alone does not make them usable. The backend reports unavailable when the configured model is reached any other way, which makes the banner, the picker, valid_tool_names and the MCP export correct at once rather than papering over each in turn. Resolution stays a cheap, network-free read of the persisted model config — this runs while schemas are assembled and on every `hermes tools` repaint — and is deliberately permissive: a config it cannot classify keeps web access, since request-time projection remains the authority on what reaches the wire. Both places that silently withheld a server-only tool now say so once per process, naming the mismatch and the remedy: the transport projection and the third-party-endpoint path in convert_tools_to_anthropic. Documentation follows the code: it no longer claims the tools "report that they are unavailable" on foreign endpoints, which they never did.
The local web_extract path is bounded twice before a result reaches the model — the auxiliary summariser, then max_result_size_chars on the registry entry. The native fetch executes inside the Messages API request, so neither guard ever sees it: one large page was injected whole and, being preserved for replay, resent on every later turn of the session. Set max_content_tokens on the tool definition, tied to the local cap (100_000 chars, ~4 chars/token) rather than picked freely, so choosing this backend does not silently change how much of a page can land in the context. The test asserts that relationship instead of freezing the number, so raising one bound points a reader at the other. Anthropic applies the limit to page text only; binary content such as a large PDF remains bounded solely by the context window. Domain allow/deny lists are a separate hardening step and are not addressed here.
|
Both review points are addressed, and a closer pass over the branch turned up two more problems I fixed here. Four commits on top of the original three. Review feedback
Each test was checked against a deliberately broken loop: disabling the pause branch fails four of the five, removing the persistence guard fails the persistence test, removing the counter reset fails the budget test. Two earlier drafts passed under mutation and were rewritten — worth knowing, since a test that cannot fail proves nothing. Two problems found while re-reviewing
Availability now accounts for the endpoint, not just the credential, which corrects all four surfaces at once. It stays a cheap, network-free read of the persisted model config, and is deliberately permissive — a config it cannot classify keeps web access. Both paths that silently withheld a server-only tool now log it once per process. One real behaviour change: a session that starts on a non-Anthropic model and
Test runsmacOS, Python 3.11.15,
No new failures — the failing set is a strict subset of the branch point's (the one difference is That is why the "ran Two defects I did not fix — your callBoth reproduced. Left alone rather than expand a PR you have already reviewed, but happy to fold either in.
Separate PRs or folded in, whichever you prefer. Note that 1 wants a live check against a real third-party endpoint, which I have not run. |
3 participants
What does this PR do?
Adds support for Anthropic's native server-side web search and web
fetch tools while preserving Hermes' existing provider-neutral
web_searchandweb_extractcapabilities.When
web.backend: anthropicis selected and Hermes is talking directly to theAnthropic Messages API, the adapter maps:
web_searchtoweb_search_20250305web_extracttoweb_fetch_20250910with citations enabledThe implementation deliberately does not add a local
WebSearchProvider:Anthropic executes these tools inside the Messages API request, so the correct
integration boundary is the Anthropic adapter/transport rather than the local
provider registry. The logical server-only binding is projected per transport:
direct Anthropic receives the native definition, while third-party Anthropic
endpoints and non-Anthropic fallbacks omit the unexecutable tool entirely.
The change also preserves native server-tool blocks across turns, exposes
citations to CLI/gateway clients as a compact source list, and handles
Anthropic's
pause_turncontinuation without inserting a synthetic user ortool message.
Related Issue
Related to #25234, which explored native Anthropic web search but was closed
without merging. This PR is a focused implementation against current
main;the original author is credited in the commit trailer.
Type of Change
Changes Made
when the Anthropic backend is explicitly selected.
endpoint; omit them from compatible third-party and fallback transports.
reaches Chat Completions, Bedrock, or Responses API wire payloads.
substring matching.
server_tool_use,web_search_tool_result, andweb_fetch_tool_resultblocks in their original order.pause_turnas a distinct stop reason and continue it up to abounded limit before falling back or returning a partial error.
Hermes' provider-neutral response text.
hermes toolswithout requesting asecond API key.
explicit backend selection, per-capability selection, and CLI configuration.
Design and compatibility notes
ANTHROPIC_API_KEYdoes not replacean existing web backend. This avoids selecting an unavailable server-side
backend when the active model uses OpenRouter or another transport.
provider paths are unchanged.
web_searchandweb_fetcheach usemax_uses: 5per model request.protections apply instead of Hermes' local fetch guard.
How to Test
Configure a direct Anthropic model and
ANTHROPIC_API_KEY.Run
hermes config set web.backend anthropic.Ask Hermes to search for current information and then fetch one of the
returned URLs; verify that the answer contains source URLs.
Switch to a third-party Anthropic-compatible endpoint or Chat Completions
fallback and verify the server-only tools are omitted from the request.
Run:
Validation completed:
Anthropic adapter/replay tests passed locally.
native-tool serialization smoke test.
ruff check,py_compile, andgit diff --checkpassed for the changedfiles.
scripts/check-windows-footguns.py --diff origin/mainreported no findings.claude-sonnet-4-6: oneweb_searchand twoweb_fetchexecutions. Theresponse contained the expected server-use/result blocks, Anthropic usage
reported exactly one server-tool request per call, block order survived
replay, output-only fields stayed off the request path, and source URLs were
exposed in normalized CLI/gateway text. The first fetch run revealed a
missing source-URL fallback for document-relative citations; this was fixed,
covered by a regression assertion, and confirmed by the second live fetch.
live
web_searchandweb_fetch: Anthropic reported exactly one matchingserver-tool request per call, search/fetch exposed 2/1 normalized source URLs,
and native block order survived replay. The same run confirmed that a
third-party Anthropic endpoint and Chat Completions fallback omit the
server-only binding entirely.
CI evidence:
https://github.com/akinfold/hermes-agent/actions/runs/29794982779
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/A (no new config key)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AScreenshots / Logs
Not applicable; this is an adapter/transport integration. The linked GitHub
Actions run contains the focused test, image-build, and container smoke-test
logs.