feat(gemini): add Vertex AI Express Mode + tool-call translation fixes by andhikapraa · Pull Request #29611 · NousResearch/hermes-agent

andhikapraa · 2026-05-21T03:29:59Z

What

Adds Google Vertex AI (Express Mode) as a first-class provider, plus the bug fixes that surfaced while dogfooding it on a real account.

Express Mode is Google's API-key path to Vertex (no service account, no token refresh) — sign up at https://console.cloud.google.com/expressmode for 90 days free. The URL shape is {base}/models/{model}:generateContent which GeminiNativeClient already builds, so the inference path needed no changes.

Commits

refactor(gemini): extract is_gemini_native_provider helper — collapse the scattered provider in {"gemini", "google-gemini-cli"} checks into one helper, so adding a third Gemini variant doesn't require hunting them all down.
feat(gemini): add Vertex AI Express Mode as a first-class provider — registers gemini-vertex (aliases: vertex, vertex-ai, google-vertex, vertex-express) in the plugin registry, the auth registry, and the curated catalog. VERTEX_API_KEY / GOOGLE_VERTEX_API_KEY / GOOGLE_CLOUD_API_KEY are recognised env vars.
test(gemini): cover Vertex provider registry, aliases, native routing — pins all four canonical aliases, the API-key auth shape, and the Express Mode base URL.
fix(providers): fall back to plugin registry in resolve_provider_full — --provider gemini-vertex returned "Unknown provider" because resolve_provider_full() only consulted HERMES_OVERLAYS, models.dev, and user config. Plugin-only profiles were invisible to the flag handler even though the /model picker (which uses the plugin registry directly) listed them. Adds a final fallback in get_provider() mapping ProviderProfile.api_mode → ProviderDef.transport via _API_MODE_TO_TRANSPORT, so any future plugin-only profile wires up automatically. +6 regression tests.
fix(gemini): coalesce parallel tool responses + drop orphans — switching to gemini-vertex mid-session after a model that batched tool calls produced HTTP 400 INVALID_ARGUMENT: "number of function response parts is equal to the number of function call parts of the function call turn". Root cause: _build_gemini_contents emitted one user turn per tool response, but Gemini requires N functionCall parts in a model turn → N functionResponse parts in ONE user turn, not N separate turns.
- Coalesce consecutive tool responses into the most recent user turn already holding functionResponse parts.
- Drop tool responses with no matching upstream tool_call_id. Orphan assistant tool_calls are preserved — that's the legitimate "tool replay" pattern and thoughtSignature round-trip relies on it.
- +3 regression tests (parallel coalescing, orphan-response drop, orphan-call preservation).
- Also adds gemini-3.5-flash to the curated catalogs for gemini, gemini-vertex, and google-gemini-cli (Google shipped it May 19, 2026).

Test

scripts/run_tests.sh tests/agent/test_gemini_native_adapter.py tests/hermes_cli/test_gemini_provider.py — green locally (14 + provider tests).

End-to-end on Express Mode account:

VERTEX_API_KEY=… ; /model gemini-3.5-flash --provider gemini-vertex → switch confirmed
"hello" → real response ("text": "ok" in raw curl, full response in CLI)
Mid-session switch from a parallel-tool-call provider → previously 400'd, now translates cleanly

Notes

Vertex's own /models endpoint requires OAuth2 (express API keys get 401 with CREDENTIALS_MISSING), so live model discovery isn't viable. AI Studio's /models is also blocked for vertex-scoped keys (API_KEY_SERVICE_BLOCKED). Curated catalog with manual updates is the practical path until Google opens express auth on the listing endpoint. models.dev's google-vertex provider does carry the catalog if a follow-up wants to wire it up via _MODELS_DEV_PREFERRED.

Replace 4 hardcoded `provider == "gemini"` checks across the agent runtime with a single helper in `agent/gemini_native_adapter.py`: agent/agent_runtime_helpers.py — create_openai_client agent/auxiliary_client.py — _resolve_api_key_provider (pool) agent/auxiliary_client.py — _resolve_api_key_provider (fallback) agent/auxiliary_client.py — resolve_provider_client agent/transports/chat_completions.py — thinking_config translation The new `NATIVE_GEMINI_PROVIDERS` frozenset is the single source of truth for "which providers route through GeminiNativeClient instead of the default OpenAI client". Adding a new Gemini-family provider (e.g. Vertex express mode) now requires extending one set in the plugin instead of widening four call sites in core. No behavior change — `NATIVE_GEMINI_PROVIDERS` currently contains only `"gemini"`. All 134 gemini provider tests still pass. Motivated by AGENTS.md guidance: 'never hardcode plugin-specific logic into core' — this prepares the routing surface for the gemini-vertex provider profile added in the follow-up commit.

Vertex AI rolled out Express Mode in 2025 — a plain API-key auth path that speaks the same native REST shape as AI Studio (`{base}/models/{model}:generateContent` with `x-goog-api-key` header). This adds it as a peer ProviderProfile next to the existing `gemini` (AI Studio) and `google-gemini-cli` (Cloud Code OAuth) profiles. What's new ---------- - `GeminiProfile(name="gemini-vertex")` in `plugins/model-providers/gemini/__init__.py` with aliases vertex, vertex-ai, google-vertex, vertex-express - `PROVIDER_REGISTRY["gemini-vertex"]` in `hermes_cli/auth.py` with env vars VERTEX_API_KEY (primary), GOOGLE_VERTEX_API_KEY, GOOGLE_CLOUD_API_KEY (fallbacks) - Model catalog entry in `hermes_cli/models.py`: gemini-3.1-pro-preview, gemini-3-pro-preview, gemini-3-flash-preview, gemini-3.1-flash-lite-preview, gemini-2.5-pro, gemini-2.5-flash - Alias map entries (vertex / vertex-ai / google-vertex / vertex-express → gemini-vertex) - `is_native_gemini_base_url()` now recognizes aiplatform.googleapis.com — same native REST shape, no /openai subpath - `NATIVE_GEMINI_PROVIDERS` now includes `"gemini-vertex"` so the helper from the prior refactor commit routes Vertex through GeminiNativeClient automatically (no client changes needed) - .env.example documents VERTEX_API_KEY + fallback env vars + VERTEX_BASE_URL override What's intentionally out of scope --------------------------------- - Full Vertex AI with service-account / ADC auth (stalled in NousResearch#3569, NousResearch#6491, NousResearch#8427, NousResearch#12674, NousResearch#21480 — each was too large). Express mode covers the 90 % case without pulling in google-auth. - Project / location URL builder. Express mode has a single canonical endpoint, so no GCP project ID or region flag is required. - MaaS / partner models on Vertex (Claude, GLM, Kimi). Those use a different URL shape — straightforward follow-up PR if demand exists. Addresses NousResearch#12639, NousResearch#13484, NousResearch#4983. Tested locally with VERTEX_API_KEY against gemini-3.5-flash and gemini-3.1-pro-preview through Hermes end-to-end. All round-trips, tool calls, and thinking-config flows verified. Local dokploy MCP exposed an unrelated schema-strictness gap in the dokploy-mcp upstream which I patched separately.

Adds 6 test classes mirroring the existing AI-Studio coverage in `tests/hermes_cli/test_gemini_provider.py`: - TestVertexProviderRegistry — PROVIDER_REGISTRY entry, env vars, base URL shape - TestVertexAliases — explicit name + 4 aliases via resolve_provider, normalize_provider, _PROVIDER_ALIASES; asserts vertex is a peer of (not an alias to) plain gemini - TestVertexAutoDetection — VERTEX_API_KEY auto-detection (robust to precedence tweaks via membership check) - TestVertexCredentials — VERTEX_API_KEY primary, fallback env vars, first-wins ordering - TestVertexCanonicalAndModelCatalog — CANONICAL_PROVIDERS entry, display label, model catalog - TestVertexNativeRouting — is_gemini_native_provider helper accepts vertex, rejects others (case-insensitive, None/'' safe); is_native_gemini_base_url recognizes both aiplatform and generativelanguage hosts; resolve_provider_client routes vertex to GeminiNativeClient - TestVertexAgentInit — AIAgent(provider='gemini-vertex') instantiates GeminiNativeClient (not OpenAI), reports api_mode='chat_completions' 30 new test cases, all green. Full file passes 76 / 76.

alt-glitch · 2026-05-21T03:38:48Z

Supersedes #4375 (Vertex Express header support, needs rebase) and takes a lighter approach than #16010 / #8427 (full Vertex AI with service account/ADC auth). Express Mode avoids the google-auth dependency entirely. Addresses #12639 and #13484.

slawt · 2026-05-22T20:34:23Z

I think this Express Mode is right for some users but not for some of the power users. #8427 has been open for almost 6 weeks and handles the auth path this PR defers.

The 2 aren't mutually exclusive, IMO, and I'd really appreciate if I can get some eyes on it.

andhikapraa · 2026-05-23T12:44:53Z

Hey @slawt — totally agree the two aren't mutually exclusive. Express Mode is deliberately the 90% path (solo devs, indie shops, anyone with GCP credits who just wants Gemini without the auth dance) and shouldn't block your SA/ADC work in #8427. Production GCP setups with IAM policies, audit logging, and key rotation absolutely need the full path you're working on.

Looking at #8427 quickly: 795 LOC, google-auth dep, ADC + SA JSON + token refresh — that's the right scope for what it covers. My PR is intentionally the smaller, complementary entry point. Hopefully having both lands faster than either alone would.

Vertex was reachable via the /model picker (which uses the plugin registry in providers/__init__.py) but not via /model --provider gemini-vertex (which uses resolve_provider_full from hermes_cli/ providers.py). The flag handler only consulted HERMES_OVERLAYS, models.dev, and user config — plugin-only profiles were invisible. Add a final fallback in get_provider() that consults the plugin registry, mapping ProviderProfile.api_mode → ProviderDef.transport via _API_MODE_TO_TRANSPORT so future plugin-only profiles wire up without further changes. Adds 6 regression tests pinning the four canonical aliases (gemini-vertex, vertex, vertex-express, vertex-ai), the API-key auth shape, and the Express Mode base URL. Reported in WhatsApp gateway: typing /model --provider gemini-vertex returned "Unknown provider 'gemini-vertex'" because the resolver gave up before checking the registry that owns the profile.

utajum · 2026-05-24T14:40:28Z

Cant add anything useful to the discussion, except adding my plead to the maintainers that we really need Vertex support. Please consider merging this

Gemini's native API requires that a model turn with N functionCall parts be followed by exactly one user turn with N functionResponse parts. The previous translation in _build_gemini_contents emitted one user turn per tool response, so any assistant turn with parallel tool calls produced N separate user turns — Gemini rejected it with HTTP 400 INVALID_ARGUMENT ('number of function response parts is equal to the number of function call parts of the function call turn'). Common trigger: switching to a Gemini provider mid-session after a model that batched tool calls. Two fixes in _build_gemini_contents: 1. Coalesce consecutive tool responses into the most recent user turn that already holds functionResponse parts (matches Gemini's one-turn-per-batch shape). 2. Drop tool responses whose tool_call_id has no matching upstream assistant tool_call. Assistant tool_calls without matching responses are PRESERVED — that's the legitimate 'tool replay' pattern, and thoughtSignature round-trip relies on it. Adds 3 regression tests covering parallel coalescing, orphan-response dropping, and orphan-call preservation. Also adds gemini-3.5-flash to the curated catalogs for gemini, gemini-vertex, and google-gemini-cli (released by Google in May 2026). Co-Authored-By: Althea (Hermes agent)

teknium1

Thanks for the focused Express Mode implementation and the regression coverage. The API-key route remains distinct from the OAuth2 Vertex provider now on main, but it needs integration changes before it can coexist safely.

Problems

agent/gemini_native_adapter.py:364 permits a tool/function message with no tool_call_id through as a functionResponse; it has no matching function call. Current main drops that malformed shape in agent/agent_runtime_helpers.py:505-516.
hermes_cli/models.py:1011-1013 reassigns vertex, vertex-ai, and google-vertex. Current main reserves those aliases for OAuth2 Vertex at hermes_cli/models.py:1233-1236 and plugins/model-providers/vertex/__init__.py:65-75.
.env.example:46-47 exposes VERTEX_BASE_URL as behavioral configuration, while current Vertex routing keeps non-secret settings in config.yaml (agent/vertex_adapter.py:50-87).

Suggested changes

Drop missing-ID tool responses and add a regression test.
Give Express Mode a separate, non-conflicting provider/alias surface and adapt it to the current provider-runtime architecture.
Keep endpoint overrides in config.yaml.
Leave the tool-response coalescing portion out of salvage: 936af2f4f already implements and tests the stronger general merge.

Automated hermes-sweeper review.

teknium1 · 2026-07-13T09:22:54Z

-                    ],
-                }
+            tcid = str(msg.get("tool_call_id") or "")
+            if tcid and tcid not in matched_ids:


tcid == "" bypasses this guard and emits a functionResponse with no possible matching functionCall. Reject missing IDs too (if not tcid or tcid not in matched_ids) and cover that malformed-history case.

teknium1 · 2026-07-13T09:22:54Z

    "google": "gemini",
    "google-gemini": "gemini",
    "google-ai-studio": "gemini",
+    "vertex": "gemini-vertex",


Current main now uses vertex for the OAuth2 Vertex provider and maps google-vertex/vertex-ai to it (hermes_cli/models.py:1233-1236). Reassigning these aliases would change existing users' credential and transport path; reserve them and use an Express-specific name.

teknium1 · 2026-07-13T09:22:54Z

+# GOOGLE_VERTEX_API_KEY=...
+# GOOGLE_CLOUD_API_KEY=...
+# Optional base URL override (default: Vertex express-mode endpoint)
+# VERTEX_BASE_URL=https://aiplatform.googleapis.com/v1beta1/publishers/google


VERTEX_BASE_URL is a non-secret behavioral setting. Keep user-facing routing configuration in config.yaml, as the current Vertex provider does for vertex.project_id and vertex.region.

andhikapraa added 3 commits May 21, 2026 10:25

andhikapraa changed the title ~~feat(gemini): add Vertex AI Express Mode as a first-class provider~~ feat(gemini): add Vertex AI Express Mode + tool-call translation fixes May 25, 2026

alt-glitch mentioned this pull request Jun 1, 2026

feat(vertex): add Google Vertex AI provider for Gemini models #36253

Closed

12 tasks

Harshavardhan-Yaddalapuri mentioned this pull request Jun 1, 2026

Feature: native Google Cloud Vertex AI provider support #13484

Closed

alt-glitch mentioned this pull request Jul 1, 2026

feat(vertex): add Google Vertex AI provider for Gemini (OAuth2) #56363

Merged

alt-glitch mentioned this pull request Jul 9, 2026

feat(plugins): add Google Vertex AI image and video generation providers #61579

Closed

13 tasks

teknium1 reviewed Jul 13, 2026

View reviewed changes

alt-glitch added comp/cli CLI entry point, hermes_cli/, setup wizard sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Jul 13, 2026

This was referenced Jul 24, 2026

feat(vertex): add API key (Express Mode) auth with region-specific model discovery #70562

Closed

feat(vertex): add API key (Express Mode) auth + native API endpoint support #70663

Open

Navigation Menu

Uh oh!

feat(gemini): add Vertex AI Express Mode + tool-call translation fixes - #29611

feat(gemini): add Vertex AI Express Mode + tool-call translation fixes#29611
andhikapraa wants to merge 5 commits into
NousResearch:mainfrom
andhikapraa:feat/gemini-vertex-express

andhikapraa commented May 21, 2026 •

edited

Loading

Uh oh!

alt-glitch commented May 21, 2026

Uh oh!

slawt commented May 22, 2026

Uh oh!

andhikapraa commented May 23, 2026

Uh oh!

utajum commented May 24, 2026

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 13, 2026

Uh oh!

teknium1 Jul 13, 2026

Uh oh!

teknium1 Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

andhikapraa commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Commits

Test

Notes

Uh oh!

alt-glitch commented May 21, 2026

Uh oh!

slawt commented May 22, 2026

Uh oh!

andhikapraa commented May 23, 2026

Uh oh!

utajum commented May 24, 2026

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andhikapraa commented May 21, 2026 •

edited

Loading