feat(vertex): route Claude models through the AnthropicVertex SDK - #66522
Open
nickkpoon wants to merge 26 commits into
Open
feat(vertex): route Claude models through the AnthropicVertex SDK#66522nickkpoon wants to merge 26 commits into
nickkpoon wants to merge 26 commits into
Conversation
alt-glitch
added
type/feature
Collaborator
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
2 times, most recently
from
July 18, 2026 12:34
d465c2a to
b47cc57
Compare
teknium1
reviewed
Jul 18, 2026
teknium1
left a comment
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for carrying the Claude-on-Vertex path through runtime setup, the picker, and auxiliary clients. Current main still routes all Vertex requests through OpenAI chat completions (hermes_cli/runtime_provider.py:1589-1610), so the feature remains needed.
Problems
- Recovery is incomplete:
agent/agent_runtime_helpers.py:1093-1100and:1265-1274rebuild everyanthropic_messagesprimary withbuild_anthropic_client(). After a transient recovery or fallback restoration, a Vertex Claude session would no longer useAnthropicVertex; the PR only updates initialization andrun_agent.py::_rebuild_anthropic_client. - The new classifier accepts
anthropic/claude-*, but primary normalization retains that prefix for providervertex(hermes_cli/model_normalize.py:412-417,453-467). The PR strips it only in the auxiliary path, so main and auxiliary requests disagree on the model ID. - Vertex docs remain Gemini-only (
website/docs/guides/google-vertex.md:9;website/docs/integrations/providers.md:385-406).
Suggested changes
- Centralize Vertex-Anthropic client construction and use it in recovery/restore/switch paths; add regression tests for those paths.
- Normalize the supported Anthropic alias to the bare Vertex publisher ID before primary requests, with a main-agent test.
- Document the Claude protocol path, Model Garden enablement, and regional behavior.
Automated hermes-sweeper review.
| @@ -808,6 +813,36 @@ def init_agent( | |||
| agent._client_kwargs = {} | |||
Contributor
There was a problem hiding this comment.
This branch can be reached by anthropic/claude-*, because the new classifier accepts that alias, but the primary model normalizer does not strip an anthropic/ prefix for provider vertex (hermes_cli/model_normalize.py:412-417,453-467). Normalize the model to the bare Vertex publisher ID before constructing the client, and add a main-agent alias regression test; the auxiliary branch already performs that strip.
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
from
July 18, 2026 23:22
b47cc57 to
b728bf1
Compare
teknium1
added
sweeper:risk-session-state
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
from
July 20, 2026 05:43
eb185ed to
8231298
Compare
Author
|
Addressed all three review findings in 656820d4ac (plus 82312987a4 for the picker and a1b285d9ca for the update flow, also filed standalone as #67884):
While fixing (1) we caught a fourth issue in the same family, live: |
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
from
July 22, 2026 23:52
fcdce4e to
8c5bb15
Compare
|
This is great — thanks for carrying the full Claude-on-Vertex path through
This is a small, non-competing follow-on rather than anything that should block
I have this working against a corporate EU Vertex gateway and am happy to open |
alt-glitch
added
comp/gateway
Collaborator
Author
|
Rebased onto current Conflict resolution notes:
Full vertex/bedrock/model_switch/model_normalize/runtime_provider/auxiliary test subset (1410 passed) now shows the same 16–17 pre-existing failures as a clean |
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
from
August 10, 2026 09:30
f6929e3 to
3d1a705
Compare
15 tasks
Vertex Model Garden serves Claude over the Anthropic Messages protocol (rawPredict / streamRawPredict), not the OpenAI-compatible endpoint that Gemini and partner MaaS models use. Until now the `vertex` provider only spoke OpenAI-compat, so selecting a Claude model on Vertex was unreachable — the one way to bill heavy Claude usage to Google Cloud credits (Bedrock bills AWS only). This mirrors the existing salvaged Bedrock dual-path (NousResearch#8427 added the Gemini Vertex provider; this completes it for Claude): - agent/vertex_adapter.py: add `is_anthropic_vertex_model()` (detects `claude-*@YYYYMMDD` and the `anthropic/` alias) and `get_vertex_anthropic_config()`, which returns the google-auth Credentials object rather than a frozen token so the Anthropic SDK self-refreshes the OAuth2 access token on expiry — no per-turn 401 refresh hook needed for long-lived gateway sessions. - agent/anthropic_adapter.py: add `build_anthropic_vertex_client()` (AnthropicVertex, max_retries=0 so hermes owns retry, common betas without the 1M-context beta which Vertex Claude does not honor). - hermes_cli/runtime_provider.py: dual-path the vertex branch — Claude → api_mode=anthropic_messages; Gemini/partner → chat_completions. - agent/agent_init.py + run_agent.py: build/rebuild the AnthropicVertex client when provider=vertex and api_mode=anthropic_messages. - hermes_cli/{setup,models,model_setup_flows}.py: picker + setup wizard support (Claude model suggestions, model-aware endpoint preview, Model-Garden enablement guidance). Gemini/partner routing is unchanged. Adds 17 tests covering model detection, credential-object resolution, dual-path routing, client shape, and regional base URLs.
User ADC (authorized_user credentials) requires the x-goog-user-project header on aiplatform requests — without it Vertex returns 403 'requires a quota project'. google-auth's own transports attach it automatically, but the Anthropic SDK uses its own httpx client, so set it explicitly as a default header. Service accounts tolerate the header harmlessly. Found during live verification against a real GCP project.
Vertex shared-capacity preview MaaS endpoints (e.g. deepseek-v3.2-maas) return 429 RESOURCE_EXHAUSTED 'too many concurrent requests' on cold-start bursts, then serve normally within seconds. _is_payment_error() matched the 'resource exhausted' substring and classified this as permanent quota exhaustion -> no retry -> the advisor was silently dropped from every MoA reference fan-out on cold turns (where provider fallback is meaningless). - New _is_transient_concurrency_throttle(): fires on transient 'too many concurrent'/'try again later' 429s, never on daily/weekly/quota/billing wording (those still fall through to the payment/fallback path unchanged). - Wired into all 3 transient-retry gates (sync + both async) so the cold 429 now retries with exponential backoff via auxiliary.transient_retries (default 2 -> up to 3 attempts) before any fallback. - Async path upgraded from single-retry to the same bounded backoff loop. Reproduced with a raw curl to Vertex (429 -> 200 on retry); 6-case detector test isolates concurrency bounce from genuine quota exhaustion.
…x models Vertex has no /models discovery endpoint and fetch_models() returns None by design, but _PROVIDER_MODELS had no vertex entry — so the picker's vertex row always enumerated zero models and the configured model only appeared via unrelated cache paths (intermittently).
…y/MoA path resolve_provider_client() previously had no 'vertex' entry in PROVIDER_REGISTRY (plugin auto-extend only picks up api_key providers), so every auxiliary.<task> and MoA slot with provider: vertex returned (None, None) -> 'no API key was found' (NousResearch#61852), and Claude-on-Vertex slots that did resolve a base_url 404'd on /v1/messages. Mirrors the bedrock aws_sdk dual-path pattern: - hermes_cli/auth.py: register vertex ProviderConfig (auth_type=vertex) - agent/auxiliary_client.py: vertex branch now splits Claude -> AnthropicVertex SDK (AnthropicAuxiliaryClient wrapper, credentials object, self-refreshing tokens) vs Gemini/MaaS -> OpenAI-compat. - hermes_cli/providers.py: vertex overlay so get_provider('vertex') resolves and base_url forwarding preserves provider identity instead of collapsing to 'custom'. Live-verified: title_generation on vertex config, MoA slot vertex/claude-fable-5 (1.7s), vertex/gpt-oss-120b-maas regression OK. Pre-existing test failures (14) reproduce identically on clean tree.
…opicCompletionsAdapter The adapter's synthesized usage renamed input_tokens->prompt_tokens and dropped cache_read/cache_creation entirely, so normalize_usage( api_mode=anthropic_messages) — which reads the native field names — returned all-zero CanonicalUsage for every MoA advisor and auxiliary call routed through an Anthropic-SDK-backed client (native, custom anthropic_messages, Vertex, Bedrock). The whole reference fan-out was invisible to cost tracking. Expose both shapes: native names for the anthropic_messages branch, OpenAI prompt_tokens (cache-inclusive per OpenAI convention) for the fallback branch. Also: vertex Gemini aux branch now uses _create_openai_client so SDK max_retries defaults to 0 (Hermes owns retry policy, NousResearch#54465).
…redentials exist Vertex has auth_type="vertex" and env_vars=() — no API key to detect — so list_authenticated_providers never marked it authenticated and the picker omitted the provider row whenever vertex wasn't the configured model.provider. Regression surfaced when model.provider switched to moa: "Google Vertex AI" (and claude-fable-5 with it) vanished from the desktop picker despite working ADC. - model_switch: add _has_vertex_creds_for_listing() mirroring bedrock's aws_sdk special case, in both the overlay and canonical-provider loops - auth: treat the non-secret vertex: config section (project_id) or a credentials-path env var as explicit configuration, so explicit-only desktop pickers keep the row (vertex has no key for check 3 to find) - providers: add "Google Vertex AI" display label - tests: picker visibility with/without credentials + explicit-config signals Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ode; docs Addresses the three PR-review findings, plus a live failure the review predicted (a fresh desktop session on vertex/claude-fable-5 built an OpenAI chat_completions client against the Anthropic base_url → HTTP 404 on every request, then took the same 404 on the vertex fallback model). 1. Recovery/restore/switch completeness: new build_anthropic_client_for_provider() in anthropic_adapter is the single provider-aware chokepoint (bedrock → AnthropicBedrock, vertex → AnthropicVertex re-resolving credentials with the agent's cached _vertex_* attrs as fallback, else plain client). Used by try_recover_primary_transport, restore_primary_runtime, switch_model, and AIAgent._rebuild_anthropic_client (now a thin delegation). switch_model also pins the SDK placeholder keys (aws-sdk / vertex-oauth) so a stale previous-provider key can't leak in. 2. Alias normalization agreement: normalize_model_for_provider(vertex) now strips the anthropic/ prefix (and only that prefix — Gemini and partner MaaS IDs keep their required publisher/ form), so primary and auxiliary requests agree on the wire ID. 3. Model-aware api_mode: determine_api_mode() takes an optional model and returns anthropic_messages for Claude-on-vertex; the two model_switch.py call sites and agent-side switch_model pass the model. tui_gateway additionally stops persisted session api_mode/base_url/key overrides from clobbering an authoritative vertex_anthropic runtime — the exact clobber behind the live 404. 4. Docs: google-vertex.md gains Claude model IDs, the protocol split, Model Garden enablement (enable + data-sharing 403 + zero-quota 429), regional behavior, and troubleshooting entries; providers.md blurb updated. Tests: 19 new in tests/agent/test_vertex_client_recovery.py (chokepoint dispatch, recovery/restore/switch routing, api_mode determination incl. the empty-api_mode switch regression) + 9 normalization cases in test_vertex_provider.py. Suites touched all green (174+51+87). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the provider chokepoint
The live failure the previous commit missed: _create_request_anthropic_client
— the per-request streaming client the stale/interrupt watchdog ownership
contract requires — special-cased bedrock but not vertex, so EVERY streamed
request on a Claude-on-Vertex session built a plain Anthropic client and
POSTed {aiplatform_host}/v1/messages (Google's HTML 404), starting with the
first call of a brand-new session. The shared client built at init was
correct, which is why direct construction tested fine while real desktop
sessions failed.
Route it through build_anthropic_client_for_provider, along with the three
remaining direct build_anthropic_client sites on the primary path:
- chat_completion_helpers fallback activation (a vertex fallback target died
the same way the primary just had — observed live: fable-5 404 → fallback
claude-sonnet-5 404), with SDK placeholder keys pinned
- credential-refresh rebuild and credential-pool swap in run_agent
(defense-in-depth; vertex/bedrock have no bearer to refresh)
Also harden the gateway session-override path: session model overrides
persisted by older builds (or echoed from cached desktop client state) can
carry api_mode chat_completions / a stale bearer key for a Claude-on-Vertex
model. _sanitize_vertex_claude_runtime repairs them at both consumption
points (fast path + _apply_session_model_override).
Tests: per-request client routing for vertex + plain-provider preservation
(21 total in test_vertex_client_recovery.py); verified live end-to-end
through _create_request_anthropic_client on the real project.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ode path
try_activate_fallback has its own api_mode detection block, independent of
determine_api_mode (fixed for the primary path in 656820d4ac). It had cases
for openai-codex/anthropic/azure/openai/bedrock but none for Claude-on-
Vertex, so a vertex fallback model defaulted to chat_completions, skipped
the anthropic_messages client build, and 404'd on {host}/v1/messages.
Observed live: the primary claude-fable-5 (now correctly on AnthropicVertex
after da03dca51b) hit a transient Anthropic 'Overloaded' (529), which
triggered fallback to claude-sonnet-5 — and the fallback 404'd because of
this gap, so the whole turn still failed.
Add a vertex-alias + is_anthropic_vertex_model() branch mirroring
determine_api_mode's split, so the fallback reaches the (already provider-
aware) anthropic_messages client build. Verified live: sonnet-5 fallback
resolves anthropic_messages → AnthropicVertex → serves the request.
Test: integration test drives try_activate_fallback for a vertex Claude
fallback and asserts api_mode=anthropic_messages + chokepoint build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…o claude-fable-5
MODEL_ALIASES had sonnet/opus/haiku/claude but not fable. Typing
'/model fable' (e.g. from the Discord gateway) fell through alias
resolution and sent the literal string 'fable' to the wire. On Vertex,
is_anthropic_vertex_model('fable') is False (no 'claude' prefix), so
the request skipped the AnthropicVertex SDK path and hit the
OpenAI-compatible /openapi endpoint, which 400s with:
Malformed publisher model (model: 'fable'); expected '<publisher>/<model>'
Adds the alias plus a regression test asserting the short Claude
aliases present in Vertex's curated catalog resolve to bare IDs the
AnthropicVertex classifier recognizes.
…token expiry (401 ACCESS_TOKEN_TYPE_UNSUPPORTED) The Vertex Gemini/openapi auxiliary path bakes a frozen OAuth2 bearer token into the OpenAI client at build time. After the ~1h token lifetime every auxiliary call (compression, title_generation, background_review) 401s until process restart — observed live as hours of context_compressor failures in long-lived desktop sessions. - vertex_adapter.refresh_vertex_credentials(): forced token re-mint with thread-safe single-flight cooldown; clears the creds cache and warns when the re-mint returns the same (wedged) token. - _refresh_provider_credentials(): new vertex branch (all 5 accepted spellings) that re-mints then evicts stale clients. - _evict_cached_vertex_clients(): host-based sweep so clients cached under 'auto'/task labels are evicted too, not just 'vertex'-keyed ones. - _auth_refresh_provider_for_route(): maps aiplatform.googleapis.com hosts (global + regional prefix form) to 'vertex' so auto-routed clients recover. - tests: adapter re-mint semantics + recovery wiring + cache sweep.
Verified live against the Vertex publisher endpoint (global region): claude-opus-4-8 serves; opus-4-7 / opus-4-5 404 (not offered on this surface). With the catalog entry, the existing 'opus' short alias now resolves on vertex (alias machinery already handled it — the catalog entry was the missing piece). Extends the short-alias regression test to cover 'opus'.
…h messages on resume 1) anthropic_adapter: _ensure_leading_user_turn prepended a single-space text block when history starts with an assistant turn (branch spine). Anthropic rejects whitespace-only text blocks -> HTTP 400 on the branch's first query. Placeholder is now non-whitespace '(continued)'. 2) hermes_state: _session_lineage_root_to_tip now stops at a _branched_from boundary. Branches persist a full seeded copy of the pre-branch transcript, so walking into the parent concatenated the parent's rows on top of the branch's own copy — every pre-branch message rendered twice in resume/display projections. Compression continuations (no copy) still walk to their ancestors. Regression tests: adapter placeholder contract + branch-boundary lineage.
The merge in the previous commit (8ccd000c8e) auto-merged cleanly at the text level but left two real bugs behind, both invisible to git because they didn't produce conflict markers: 1. hermes_cli/models.py: two independent commits each added their own `"vertex": [...]` entry to the same _PROVIDER_MODELS dict literal (this branch's Claude+Gemini list, upstream's Gemini-only list from a parallel PR). Python silently keeps the last literal — upstream's Gemini-only list clobbered the Claude entries, breaking the /model picker and all four 'opus'/'sonnet'/'fable'/'claude' short-alias resolution tests on vertex. Merged into one entry carrying both. 2. agent/auxiliary_client.py::_refresh_provider_credentials(): the merge left two competing vertex branches — this branch's `_VERTEX_PROVIDER_NAMES` branch (calls refresh_vertex_credentials(), evicts by base_url host so aliased/auto-routed clients are caught too) shadowed upstream's simpler, permanently-dead `== "vertex"` branch (get_vertex_config()-based, provider-name-only eviction). Removed the dead duplicate; its behavior is already fully covered by the dedicated tests/agent/test_auxiliary_client_vertex_recovery.py suite (40/40 passing), so removed its two now-obsolete inline tests in test_auxiliary_client.py as well. Also fixed tests/hermes_cli/test_vertex_model_picker.py's test_vertex_has_curated_model_list, an upstream-only test written before this PR existed that asserted every vertex model carries the 'google/' publisher prefix — no longer true once Claude's bare publisher IDs (AnthropicVertex SDK path) are curated alongside Gemini/partner-MaaS vendor-prefixed IDs. Verified: tests/hermes_cli/ + tests/agent/ vertex/bedrock/model_switch/ model_normalize/runtime_provider/auxiliary subset now shows the same 17 failures as a clean upstream/main checkout (pre-existing flaky/ broken tests unrelated to vertex, reproduced with zero vertex changes applied) and zero vertex-specific failures.
claude-opus-5 was absent from DEFAULT_CONTEXT_LENGTHS and BEDROCK_CONTEXT_LENGTHS, so it fell through to the generic "claude": 200000 catch-all. The agent then compressed context at ~200K on a model that serves 1M, silently discarding context the user already paid to load. Verified against a live Vertex endpoint, which reports: prompt is too long: 8460056 tokens > 1000000 maximum Both tables are matched longest-key-first, so the new entries do not shadow the existing opus-4-* (200K) or haiku-4-5 (200K) entries.
…alog Adding claude-opus-5 alongside claude-opus-4-8 made the bare `/model opus` shorthand raise AmbiguousAliasError: upstream's resolve_alias() refuses to guess among multiple family matches rather than silently version-sorting. Drop opus-4-8 (superseded; still reachable via an explicit model_aliases entry) so each Claude family resolves to exactly one catalog id. Split the bare-"claude" case out of the resolve-happy-path parametrize into its own test asserting the raise, pinning the no-silent-selection contract.
…ontract switch_model() now calls _ensure_lmstudio_runtime_loaded(context_intent), _lmstudio_load_was_unverified(runtime_len) and _effective_lmstudio_context_length(intent, runtime_len). The fork's stub predates all three, so the vertex chokepoint tests died on a TypeError before reaching their assertions.
…rtex_config The vertex branch of _refresh_provider_credentials now delegates to vertex_adapter.refresh_vertex_credentials(); patching get_vertex_config left the real refresh running, so the unminted case returned True and the success case never asserted its dependency.
…e, curated vertex catalog, suppressed-key pool guard
…t, intro pricing, fast-model family, default aux)
nickkpoon
force-pushed
the
feat/claude-on-vertex
branch
from
August 16, 2026 00:41
3d1a705 to
cfb4c9e
Compare
Collaborator
Related to #3569 and the other open Claude-on-Vertex implementations. This PR has concrete broader recovery, picker, and runtime-integration deltas, so it is a competing implementation rather than a duplicate. |
…ing entries
Grok on Vertex Model Garden was falling into the provider='google' branch of
resolve_billing_route, so every call looked up a ('google','grok-*') key that
does not exist and was silently recorded as $0.00 / cost_status=unknown -- the
same bug class the Claude-on-Vertex branch already fixes. Adds the xai branch
plus pricing for the 4.1-fast and 4.20/4.3 tiers, and lists
grok-4.1-fast-non-reasoning in the curated vertex catalog.
Author
|
Thanks @lczupryn-tibco for the suggestion regarding corporate proxies and VPC-SC egress gateways. I have pushed an update that implements the The central chokepoint ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Vertex Model Garden serves Claude over the Anthropic Messages protocol (
rawPredict/streamRawPredict), not the OpenAI-compatible endpoint that Gemini and partner MaaS models use. Until now thevertexprovider only spoke OpenAI-compat, so selecting a Claude model on Vertex was unreachable.This completes the Vertex provider for Claude, mirroring the existing dual-path pattern used for AWS Bedrock (Claude →
AnthropicBedrockSDK; other models → Converse). PR #8427 added the Gemini Vertex provider; this is the Claude half.Motivation: it is the only way to bill heavy Claude usage against Google Cloud credits — Bedrock bills AWS only.
How
agent/vertex_adapter.py—is_anthropic_vertex_model()(detectsclaude-*@YYYYMMDDand theanthropic/alias) andget_vertex_anthropic_config(), which returns the google-auth Credentials object rather than a frozen token so the Anthropic SDK self-refreshes the OAuth2 access token on expiry. No per-turn 401 refresh hook is needed for long-lived gateway sessions (unlike the OpenAI-compat Gemini path, which mints a static bearer).agent/anthropic_adapter.py—build_anthropic_vertex_client()(AnthropicVertex,max_retries=0so the outer loop owns retry / Retry-After, common betas without the 1M-context beta which Vertex Claude does not honor).hermes_cli/runtime_provider.py— dual-path thevertexbranch: Claude →api_mode=anthropic_messages; Gemini/partner →chat_completions(unchanged).agent/agent_init.py+run_agent.py— build/rebuild theAnthropicVertexclient whenprovider=vertexandapi_mode=anthropic_messages.hermes_cli/{setup,models,model_setup_flows}.py— model picker + setup-wizard support: Claude model suggestions, model-aware endpoint preview, and Model-Garden enablement guidance.Notes
Tests
Adds 17 tests (all green) covering model detection, credential-object resolution, dual-path routing, client shape (
max_retries=0, no 1M beta), and regional vs global base URLs. Existing Vertex + Bedrock + runtime-provider suites remain green.Live end-to-end call against a real GCP project was not run in this environment (no GCP credentials available); everything up to the network boundary is verified against the real Anthropic SDK.