feat: add google-vertex provider support - #1070
Conversation
WalkthroughAdds a new provider alias Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Client
participant Gateway
participant Models as ProviderConfig
participant VertexAPI as GoogleVertexAPI
Client->>Gateway: Send chat request (provider="google-vertex", streaming?)
Gateway->>Models: getProviderEndpoint("google-vertex", streaming?, token?)
Models-->>Gateway: URL (aiplatform.googleapis.com/...generateContent or streamGenerateContent)
Gateway->>VertexAPI: POST / SSE (token/query params if provided)
VertexAPI-->>Gateway: Streamed events or final response
Gateway->>Gateway: parseProviderResponse (uses google-ai-studio logic for google-vertex)
Gateway->>Gateway: extract content/images/reasoning/usage/tool-calls
Gateway->>Gateway: transform to OpenAI-compatible payload
Gateway-->>Client: Emit normalized events/chunks, ensure single final "[DONE]"
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
❌ Preview Environment deleted from BunnyshellAvailable commands (reply to this comment):
|
4fa70dc to
4367fd2
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
apps/gateway/src/chat/chat.ts (3)
1329-1337: Vertex token not forwarded to endpoint buildergetProviderEndpoint receives the token only for google-ai-studio. Vertex’s API key typically needs to be appended as a query param (or handled consistently with headers), and the builder also adds alt=sse. Forward the token for google-vertex too to avoid auth/streaming issues.
Apply:
- url = getProviderEndpoint( + url = getProviderEndpoint( usedProvider, providerKey?.baseUrl || undefined, usedModel, - usedProvider === "google-ai-studio" ? usedToken : undefined, + (usedProvider === "google-ai-studio" || usedProvider === "google-vertex") ? usedToken : undefined, stream, supportsReasoning, hasExistingToolCalls, providerKey?.options || undefined, );
2331-2354: Include Vertex in streaming usage injectionUsage is injected into transformedData only for google-ai-studio. Add google-vertex to keep parity and avoid missing per-chunk usage for Vertex streams.
- if (usedProvider === "google-ai-studio") { + if (usedProvider === "google-ai-studio" || usedProvider === "google-vertex") { const usage = extractTokenUsage( data, usedProvider, fullContent, );
2411-2448: Pass raw Vertex chunks to extractorsFor streaming, raw data is only routed to extractContent/extractReasoning for google-ai-studio/anthropic. Vertex should also use raw chunks; otherwise the google-vertex paths in extractors won’t trigger, degrading content/reasoning capture.
- const contentChunk = extractContent( - usedProvider === "google-ai-studio" || - usedProvider === "anthropic" - ? data - : transformedData, - usedProvider, - ); + const contentChunk = extractContent( + (usedProvider === "google-ai-studio" || + usedProvider === "google-vertex" || + usedProvider === "anthropic") + ? data + : transformedData, + usedProvider, + );- const reasoningContentChunk = extractReasoning( - usedProvider === "google-ai-studio" || - usedProvider === "anthropic" - ? data - : transformedData, - usedProvider, - ); + const reasoningContentChunk = extractReasoning( + (usedProvider === "google-ai-studio" || + usedProvider === "google-vertex" || + usedProvider === "anthropic") + ? data + : transformedData, + usedProvider, + );
🧹 Nitpick comments (4)
apps/gateway/src/chat/tools/extract-tool-calls.ts (1)
42-57: Vertex aliasing for tool calls — OK; consider stable IDsLogic mirrors Studio and is fine. Optional: use a stable identifier (e.g., candidate/part index tuple or crypto.randomUUID) instead of Date.now() to avoid rare collisions across parallel streams.
- id: part.functionCall.name + "_" + Date.now() + "_" + index, + id: `${part.functionCall.name}_${index}`,apps/gateway/src/chat/tools/extract-reasoning.ts (1)
20-25: Vertex reasoning extraction — add type guard on textLooks good. Minor hardening: guard text type to avoid concatenating non-strings.
- const reasoningParts = parts.filter((part: any) => part.thought); - return reasoningParts.map((part: any) => part.text).join("") || ""; + const reasoningParts = parts.filter((part: any) => part.thought); + return ( + reasoningParts + .map((part: any) => (typeof part.text === "string" ? part.text : "")) + .join("") || "" + );apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (1)
247-251: Pass usedProvider to extractImages for consistencyHardcoding google-ai-studio here works today but is brittle. Pass usedProvider so Vertex follows future logic changes automatically.
- delta.images = extractImages(data, "google-ai-studio"); + delta.images = extractImages(data, usedProvider);apps/gateway/src/chat/tools/extract-token-usage.ts (1)
21-45: Alias looks good; consider de-duplicationHandling google-vertex identically to google-ai-studio is correct. Consider extracting a small helper or a provider set (e.g., const GOOGLE_PROVIDERS = new Set(["google-ai-studio","google-vertex"])) to avoid repeating switch cases in multiple files.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (20)
.env.unified.example(1 hunks).github/workflows/e2e.yml(1 hunks)apps/gateway/src/chat/chat.ts(1 hunks)apps/gateway/src/chat/tools/extract-content.ts(1 hunks)apps/gateway/src/chat/tools/extract-images.ts(1 hunks)apps/gateway/src/chat/tools/extract-reasoning.ts(1 hunks)apps/gateway/src/chat/tools/extract-token-usage.ts(1 hunks)apps/gateway/src/chat/tools/extract-tool-calls.ts(1 hunks)apps/gateway/src/chat/tools/parse-provider-response.ts(1 hunks)apps/gateway/src/chat/tools/transform-response-to-openai.ts(1 hunks)apps/gateway/src/chat/tools/transform-streaming-to-openai.ts(1 hunks)apps/playground/src/components/ui/providers-icons.tsx(1 hunks)apps/ui/src/lib/components/providers-icons.tsx(1 hunks)packages/models/src/get-provider-endpoint.ts(2 hunks)packages/models/src/get-provider-headers.ts(1 hunks)packages/models/src/models/google.ts(1 hunks)packages/models/src/prepare-request-body.ts(1 hunks)packages/models/src/provider.ts(1 hunks)packages/models/src/providers.ts(1 hunks)packages/models/src/validate-provider-key.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/extract-token-usage.tspackages/models/src/providers.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/get-provider-headers.tspackages/models/src/models/google.tsapps/gateway/src/chat/tools/extract-images.tsapps/gateway/src/chat/tools/extract-tool-calls.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/extract-content.tspackages/models/src/validate-provider-key.tspackages/models/src/get-provider-endpoint.tsapps/ui/src/lib/components/providers-icons.tsxapps/gateway/src/chat/tools/extract-reasoning.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/provider.tsapps/playground/src/components/ui/providers-icons.tsx
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/extract-token-usage.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/extract-images.tsapps/gateway/src/chat/tools/extract-tool-calls.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/extract-content.tsapps/gateway/src/chat/tools/extract-reasoning.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/extract-token-usage.tspackages/models/src/providers.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/get-provider-headers.tspackages/models/src/models/google.tsapps/gateway/src/chat/tools/extract-images.tsapps/gateway/src/chat/tools/extract-tool-calls.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/extract-content.tspackages/models/src/validate-provider-key.tspackages/models/src/get-provider-endpoint.tsapps/ui/src/lib/components/providers-icons.tsxapps/gateway/src/chat/tools/extract-reasoning.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/provider.tsapps/playground/src/components/ui/providers-icons.tsx
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/extract-token-usage.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/extract-images.tsapps/gateway/src/chat/tools/extract-tool-calls.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/extract-content.tsapps/gateway/src/chat/tools/extract-reasoning.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
apps/{ui,playground}/**/*.{ts,tsx}: In Next.js apps, use next/link for links and next/navigation’s router for programmatic navigation (avoid tags, window.location, or legacy next/router)
Use cookies for user settings that are not persisted in the database to ensure SSR works
apps/{ui,playground}/**/*.{ts,tsx}: Usenext/linkfor links andnext/navigation's router for programmatic navigation
Use cookies for user settings not saved in the database to ensure SSR works
Files:
apps/ui/src/lib/components/providers-icons.tsxapps/playground/src/components/ui/providers-icons.tsx
🧬 Code graph analysis (2)
apps/ui/src/lib/components/providers-icons.tsx (2)
apps/playground/src/components/ui/providers-icons.tsx (1)
GoogleStudioAIIcon(69-119)apps/playground/src/components/provider-icons.tsx (1)
GoogleStudioAIIcon(105-155)
apps/playground/src/components/ui/providers-icons.tsx (2)
apps/ui/src/lib/components/providers-icons.tsx (1)
GoogleStudioAIIcon(105-155)apps/playground/src/components/provider-icons.tsx (1)
GoogleStudioAIIcon(105-155)
🔇 Additional comments (16)
.github/workflows/e2e.yml (1)
58-58: LGTM!The addition of
LLM_GOOGLE_VERTEX_API_KEYto the e2e workflow is consistent with the existing provider API key pattern and correctly positioned in the environment variable list.packages/models/src/get-provider-headers.ts (1)
17-19: LGTM!Correctly extends the header handling to include
google-vertexalongsidegoogle-ai-studio, returning an empty headers object. This is appropriate since Google providers handle authentication via query parameters rather than headers.packages/models/src/providers.ts (1)
70-80: LGTM!The new provider definition for
google-vertexis complete and well-structured. The shared color withgoogle-ai-studio(#4285f4) appropriately reflects their relationship as related Google services, while the description clearly differentiates Vertex AI from AI Studio.packages/models/src/provider.ts (1)
8-8: LGTM!The environment variable mapping for
google-vertexis correctly added and follows the established naming pattern.packages/models/src/prepare-request-body.ts (1)
524-525: LGTM!Correctly extends the Google provider handling to include
google-vertex, allowing it to share the same request body transformation logic asgoogle-ai-studio. This is appropriate since both providers use the same API format.packages/models/src/models/google.ts (1)
208-220: LGTM!The
google-vertexprovider entry forgemini-2.5-flash-liteis correctly configured with pricing and capabilities that match the existinggoogle-ai-studioentry, ensuring consistent behavior across both providers.packages/models/src/validate-provider-key.ts (1)
84-86: LGTM!Correctly extends the token propagation logic to include
google-vertex, ensuring both Google providers pass the API key as a query parameter to the endpoint, which is consistent with Google's authentication mechanism.apps/gateway/src/chat/tools/extract-images.ts (1)
9-11: Alias looks goodTreating google-vertex the same as google-ai-studio here is correct. No changes needed.
apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (1)
212-388: Vertex streaming transform — overall goodProvider aliasing and usage mapping look correct; mirrors Studio behavior.
apps/gateway/src/chat/chat.ts (1)
2491-2513: Finish reason mapping for Vertex — LGTMMapping Vertex finish reasons alongside Studio is correct. The switch case at lines 2491-2513 properly groups both providers together with identical transformation logic, ensuring parity in finish reason handling (STOP → stop/tool_calls, MAX_TOKENS → length, SAFETY → content_filter).
Broader codebase scan confirms consistent Studio/Vertex grouping across response processing functions (extract-images, extract-token-usage, extract-reasoning, extract-content, transform-response-to-openai, etc.). Studio-only conditionals at lines 1332 and 2331 are specific to particular operations and do not affect finish reason mapping.
apps/gateway/src/chat/tools/extract-content.ts (1)
8-13: LGTM for provider aliasingSharing the Google content extraction path with google-vertex is appropriate.
apps/gateway/src/chat/tools/parse-provider-response.ts (1)
126-210: Google Vertex aliasing matches existing Studio handlingThe added case mirrors Google Studio parsing (content, reasoning, images, tools, finish reasons, and token totals). Looks consistent.
apps/ui/src/lib/components/providers-icons.tsx (1)
962-963: Icon alias added correctlyMapping "google-vertex" to GoogleStudioAIIcon keeps UI consistent.
apps/gateway/src/chat/tools/transform-response-to-openai.ts (1)
28-77: OpenAI-format transform for Vertex matches StudioThe new branch correctly mirrors Studio behavior, including usage and metadata.
apps/playground/src/components/ui/providers-icons.tsx (1)
887-888: Provider icon alias is correct"google-vertex" now resolves to GoogleStudioAIIcon in the playground too.
packages/models/src/get-provider-endpoint.ts (1)
54-56: Implementation is correct — uses express mode which does not require project/locationVertex AI's express mode REST endpoints authenticate with an API key passed as the query parameter
key(e.g.,?key=YOUR_API_KEY), and use the global express endpoint form without project/location, such ashttps://aiplatform.googleapis.com/v1/publishers/google/models/{model}:generateContent?key=YOUR_API_KEY. The current implementation correctly follows this pattern and requires no changes.Likely an incorrect or invalid review comment.
Add Google Vertex AI as a new provider with gemini-2.5-flash-lite model support. Implementation reuses Google AI Studio logic with Vertex-specific API endpoint configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
4367fd2 to
b2d4de2
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (6)
packages/models/src/get-provider-headers.ts (1)
18-19: Prefer header-based auth for Vertex to avoid leaking API keys in URLs.Using the x-goog-api-key header reduces accidental exposure via logs/proxies. Suggest splitting the cases so Studio remains unchanged while Vertex uses the header.
Please confirm the endpoint builder isn’t already appending ?key= for Vertex before applying.
- case "google-ai-studio": - case "google-vertex": - return {}; + case "google-ai-studio": + return {}; + case "google-vertex": + return { + "x-goog-api-key": token, + };apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (1)
212-214: Pass usedProvider to extractImages for correctness and future-proofing.Hardcoding "google-ai-studio" can bite if providers diverge. Use the actual usedProvider.
- if (hasImages) { - delta.images = extractImages(data, "google-ai-studio"); - } + if (hasImages) { + delta.images = extractImages(data, usedProvider); + }Also applies to: 248-250
apps/gateway/src/chat/tools/parse-provider-response.ts (2)
126-136: Filter only string text parts to avoid accidental empty/undefined joins.Align with the streaming path and make parsing more robust.
- const contentParts = parts.filter((part: any) => !part.thought); - const reasoningParts = parts.filter((part: any) => part.thought); + const contentParts = parts.filter( + (part: any) => !part.thought && typeof part.text === "string", + ); + const reasoningParts = parts.filter( + (part: any) => part.thought && typeof part.text === "string", + );
170-181: Optional: extend finishReason mapping for Google edge cases.If Google adds more reasons (e.g., provider-side content filters), consider mapping unknown non-STOP reasons to content_filter instead of stop to better reflect outcomes. No change required now.
If you’ve seen additional Google finish reasons in logs, reply with examples and I’ll propose exact mappings.
apps/gateway/src/chat/tools/extract-content.ts (1)
8-13: Tighten filter to only include string text.Prevents accidental inclusion of non-text parts in concatenation.
- const contentParts = parts.filter((part: any) => !part.thought); - return contentParts.map((part: any) => part.text).join("") || ""; + const contentParts = parts.filter( + (part: any) => !part.thought && typeof part.text === "string", + ); + return contentParts.map((part: any) => part.text).join("") || "";packages/models/src/prepare-request-body.ts (1)
524-597: Honor JSON output requests for Google by setting responseMimeType.When response_format.type === "json_object", set generationConfig.responseMimeType to application/json so Google models bias output to JSON without extra tokens.
requestBody.generationConfig = {}; // Add optional parameters if they are provided if (temperature !== undefined) { requestBody.generationConfig.temperature = temperature; } @@ if (top_p !== undefined) { requestBody.generationConfig.topP = top_p; } + // Bias output toward JSON when requested via OpenAI-style response_format + if (response_format?.type === "json_object") { + requestBody.generationConfig.responseMimeType = "application/json"; + } + // Enable thinking/reasoning content exposure for Google models that support reasoning // Note: google-vertex has stricter validation and doesn't support thinkingConfig for all models if (supportsReasoning && usedProvider === "google-ai-studio") { requestBody.generationConfig.thinkingConfig = { includeThoughts: true, };
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (20)
.env.unified.example(1 hunks).github/workflows/e2e.yml(1 hunks)apps/gateway/src/chat/chat.ts(2 hunks)apps/gateway/src/chat/tools/extract-content.ts(1 hunks)apps/gateway/src/chat/tools/extract-images.ts(1 hunks)apps/gateway/src/chat/tools/extract-reasoning.ts(1 hunks)apps/gateway/src/chat/tools/extract-token-usage.ts(1 hunks)apps/gateway/src/chat/tools/extract-tool-calls.ts(1 hunks)apps/gateway/src/chat/tools/parse-provider-response.ts(1 hunks)apps/gateway/src/chat/tools/transform-response-to-openai.ts(1 hunks)apps/gateway/src/chat/tools/transform-streaming-to-openai.ts(1 hunks)apps/playground/src/components/ui/providers-icons.tsx(1 hunks)apps/ui/src/lib/components/providers-icons.tsx(1 hunks)packages/models/src/get-provider-endpoint.ts(2 hunks)packages/models/src/get-provider-headers.ts(1 hunks)packages/models/src/models/google.ts(1 hunks)packages/models/src/prepare-request-body.ts(2 hunks)packages/models/src/provider.ts(1 hunks)packages/models/src/providers.ts(1 hunks)packages/models/src/validate-provider-key.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (13)
- apps/gateway/src/chat/tools/extract-tool-calls.ts
- apps/gateway/src/chat/tools/extract-images.ts
- apps/ui/src/lib/components/providers-icons.tsx
- packages/models/src/provider.ts
- apps/gateway/src/chat/chat.ts
- packages/models/src/get-provider-endpoint.ts
- apps/gateway/src/chat/tools/extract-token-usage.ts
- apps/playground/src/components/ui/providers-icons.tsx
- apps/gateway/src/chat/tools/extract-reasoning.ts
- apps/gateway/src/chat/tools/transform-response-to-openai.ts
- .env.unified.example
- packages/models/src/validate-provider-key.ts
- packages/models/src/models/google.ts
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
packages/models/src/get-provider-headers.tsapps/gateway/src/chat/tools/extract-content.tspackages/models/src/providers.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/prepare-request-body.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
packages/models/src/get-provider-headers.tsapps/gateway/src/chat/tools/extract-content.tspackages/models/src/providers.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/prepare-request-body.ts
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat/tools/extract-content.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/tools/extract-content.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
- GitHub Check: build-split (ui, linux/amd64)
- GitHub Check: build-split (docs, linux/amd64)
- GitHub Check: build-split (playground, linux/amd64)
- GitHub Check: build-split (gateway, linux/amd64)
- GitHub Check: build-split (api, linux/amd64)
- GitHub Check: build-unified (linux/amd64)
- GitHub Check: build / run
- GitHub Check: test / run
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (1)
.github/workflows/e2e.yml (1)
58-58: Vertex key wired for e2e — approved. No security issues found.Verification confirmed:
- Secret
GOOGLE_VERTEX_API_KEYproperly configured in e2e.yml (line 58) alongside other provider secrets- Endpoint URL with API key is not logged; logger.debug only captures provider metadata, not the constructed endpoint
- E2E tests log response bodies, not request URLs; no
?key=...exposure in logs- Implementation follows Google Vertex API standards
The API token wasn't being passed to getProviderEndpoint for google-vertex, causing 400 errors. Also skip thinkingConfig for google-vertex as it has stricter validation than google-ai-studio. Non-streaming requests now work correctly. Streaming still needs implementation as google-vertex uses JSON array format instead of SSE. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
6565cb1 to
b740580
Compare
The [DONE] event wasn't being sent for providers like Google Vertex that provide usage tokens in their stream. This was because the [DONE] event was only sent when needsUsageChunk was true. Additionally, providers like OpenAI that send [DONE] in their stream were getting a duplicate [DONE] event. This fix: - Adds a doneSent flag to track if [DONE] has been sent - Sets doneSent=true when [DONE] is sent from the upstream stream - Sends [DONE] at the end only if not already sent This ensures exactly one [DONE] event for all providers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added google-vertex provider configuration to all Google AI Studio models with identical parameters: - All Gemini 2.5 models (pro, flash variants, lite) - All Gemini 1.5 models (pro, flash, flash-8b) - All Gemini 2.0 models (flash, flash-lite) - All Gemma models (3n, 3-1b, 3-4b, 3-12b variants) This enables users to use the same models through Google Vertex AI with the same pricing, capabilities, and configurations as Google AI Studio. Total: 21 models now support google-vertex provider. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
packages/models/src/models/google.ts (1)
25-809: Consider reducing duplication betweengoogle-ai-studioandgoogle-vertexprovider entries.Each model definition duplicates 10-12 identical fields for both
google-ai-studioandgoogle-vertexproviders (~200+ lines total). This creates maintenance burden when updating pricing or capabilities.Consider introducing a shared configuration pattern, such as:
- A helper function that generates provider entries with common base config
- Extracting shared pricing/capability objects that both providers reference
- Using spread operators to inherit from a base Google provider config
This refactor can be deferred if provider-specific divergence is expected in the future.
apps/gateway/src/chat/chat.ts (2)
2346-2350: Compute Google usage after adding this chunk’s content to improve fallback estimates.Usage extraction for Google runs before
fullContentincludes the current chunk. IfextractTokenUsagefalls back to completion length, the estimate skews low. Use the chunk’s content when present:- if ( - usedProvider === "google-ai-studio" || - usedProvider === "google-vertex" - ) { - const usage = extractTokenUsage(data, usedProvider, fullContent); + if ( + usedProvider === "google-ai-studio" || + usedProvider === "google-vertex" + ) { + // Peek current chunk content for more accurate fallback + const contentForUsage = + extractContent( + usedProvider === "google-ai-studio" || usedProvider === "google-vertex" + ? data + : transformedData, + usedProvider, + ) || ""; + const completionSoFar = fullContent + contentForUsage; + const usage = extractTokenUsage(data, usedProvider, completionSoFar);Keeps behavior identical when provider returns explicit usage; only improves fallback accuracy.
Also applies to: 2432-2439, 2453-2460
2513-2533: Widen Google finishReason mapping; avoid defaulting unknowns to “stop”.Unknown or new Google reasons shouldn’t map to “stop”. Prefer explicit mapping with a neutral fallback and optionally log/preserve the raw provider reason.
Example:
- finishReason = - googleFinishReason === "STOP" - ? hasFunctionCalls ? "tool_calls" : "stop" - : googleFinishReason === "MAX_TOKENS" - ? "length" - : googleFinishReason === "SAFETY" - ? "content_filter" - : "stop"; + const map: Record<string, string> = { + STOP: hasFunctionCalls ? "tool_calls" : "stop", + MAX_TOKENS: "length", + SAFETY: "content_filter", + BLOCKLIST: "content_filter", + }; + finishReason = map[googleFinishReason] ?? "other"; + // Optionally: attach raw reason for observability + // transformedData.choices[0].finish_reason_detail = googleFinishReason;This avoids mislabeling terminations and aids troubleshooting when Google expands enums.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/gateway/src/chat/chat.ts(10 hunks)packages/models/src/models/google.ts(21 hunks)packages/models/src/prepare-request-body.ts(2 hunks)packages/models/src/providers.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
packages/models/src/providers.tspackages/models/src/models/google.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/chat.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
packages/models/src/providers.tspackages/models/src/models/google.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/chat.ts
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat/chat.ts
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (1)
apps/gateway/src/chat/chat.ts (1)
packages/logger/src/index.ts (2)
error(153-160)logger(181-181)
🔇 Additional comments (7)
packages/models/src/prepare-request-body.ts (2)
524-525: LGTM! Clean fallthrough pattern for shared Google provider logic.The case fallthrough allows
google-vertexto share the same request body transformation asgoogle-ai-studio, which is appropriate given their API similarities.
580-581: Clarify whether google-vertex models should declarereasoning: trueifthinkingConfigexposure is unavailable.The gating appears intentional based on the comment explaining API limitations, but it creates an ambiguity:
gemini-2.5-proandgemini-2.5-flash(google-vertex) declarereasoning: true, yet have no mechanism to expose thinking content sincethinkingConfigis restricted togoogle-ai-studio.This design is valid only if one of the following is true:
- google-vertex reasoning works internally without needing
thinkingConfigexposure- Clients shouldn't request thinking for google-vertex models, even if
supportsReasoningindicates they support reasoningIf either case is true, add a clarifying comment in the model definitions explaining why reasoning is enabled but thinking exposure is not, to prevent future confusion.
packages/models/src/models/google.ts (1)
25-38: LGTM! Google Vertex provider entries are consistent and complete.The
google-vertexprovider entries across all models correctly mirror theirgoogle-ai-studiocounterparts with identical pricing, capabilities, and feature flags. The pattern is applied uniformly and includes appropriate fields likecachedInputPrice,reasoning,vision,tools, andjsonOutput.Also applies to: 260-274, 315-328, 478-490, 513-525, 548-560, 583-595, 618-630, 653-665, 688-700, 723-735, 759-772, 796-809
apps/gateway/src/chat/chat.ts (4)
1993-1994: Single [DONE] guarantee in SSE looks solid.Introduction of
doneSentand checks prevents duplicate finals. Good defensive pattern. Please validate against providers that also emit terminal markers.Also applies to: 2263-2264, 2632-2633, 2751-2752, 2829-2842
2433-2434: Parity for google-vertex in content/reasoning extraction.Including
google-vertexalongsidegoogle-ai-studiois correct for chunk parsing.Also applies to: 2454-2455
2821-2827: Defensive logging around final usage and [DONE] send is good.Graceful error handling here improves resilience without breaking the stream contract.
Also applies to: 2839-2841
1344-1347: Critical security issue confirmed. Google Vertex AI should NOT use API key in URL query parameter; revise the fix to address proper authentication.The review comment correctly identifies a security risk but proposes an incomplete fix. API keys for Google AI (Gemini) should be passed using the HTTP header
x-goog-api-keyinstead of as a query parameter, which is a security vulnerability. However, Vertex AI Gemini API requires Bearer token authentication in the Authorization header, not API key parameters.The actual issue: Both
google-ai-studio(generativelanguage API) andgoogle-vertex(Vertex AI API) currently append the token as?key=...in the URL, but they have different authentication requirements:
- google-ai-studio: Should use
x-goog-api-keyheader (not URL param)- google-vertex: Should use
Authorization: Bearer <token>header (OAuth token from service account, not API key)Required changes:
- Remove token from URL for both providers (keep it out of query params entirely)
- Modify
getProviderHeadersto add proper header-based auth for each:
- google-ai-studio: Add
x-goog-api-keyheader- google-vertex: Add
Authorization: Bearerheader (requires token generation from service account)The review comment's suggested diff removes only google-vertex from URL params but doesn't address header-based auth—this would break google-vertex authentication. A complete fix requires updating both endpoint construction and header logic.
c49bb65 to
80f8683
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
apps/gateway/src/chat-toolcalls-result.e2e.ts (1)
26-26: LGTM! Test name abbreviation is clear.The shortened test name is sufficiently descriptive and aligns with test-name refactors across the PR. The abbreviation "res" is commonly understood.
Minor observation: The console log message on line 104 still uses the full phrase "tool calls with result response" – consider keeping naming consistent across test names and log messages if you're updating them broadly.
apps/playground/src/components/provider-icons.tsx (1)
157-337: Consider aligning props spreading pattern with GoogleStudioAIIcon for consistency.The GoogleVertexIcon implementation is functional and well-structured. However, for consistency with the related
GoogleStudioAIIcon(lines 105-155), consider matching its pattern:export const GoogleVertexIcon: React.FC<React.SVGProps<SVGSVGElement>> = ( props, ) => ( <svg {...props} xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24px" height="24px" className={props.className} >This approach:
- Spreads props first to apply all incoming props
- Then explicitly sets fixed attributes (xmlns, viewBox, width, height) to prevent accidental overrides
- Explicitly sets className for defensive consistency
Your current implementation spreads props last, which allows props to override the fixed attributes (unlikely to cause issues in practice, but less defensive).
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
apps/gateway/src/chat-basic.e2e.ts(1 hunks)apps/gateway/src/chat-toolcalls-result.e2e.ts(1 hunks)apps/gateway/src/chat/chat.ts(10 hunks)apps/playground/src/components/provider-icons.tsx(2 hunks)apps/ui/src/lib/components/providers-icons.tsx(2 hunks)packages/models/src/models/google.ts(15 hunks)
✅ Files skipped from review due to trivial changes (1)
- apps/gateway/src/chat-basic.e2e.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- apps/ui/src/lib/components/providers-icons.tsx
- packages/models/src/models/google.ts
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
apps/gateway/src/chat-toolcalls-result.e2e.tsapps/gateway/src/chat/chat.tsapps/playground/src/components/provider-icons.tsx
**/*.e2e.ts
📄 CodeRabbit inference engine (AGENTS.md)
End-to-end tests should be written in *.e2e.ts files
End-to-end test files should be named with the
.e2e.tssuffix
Files:
apps/gateway/src/chat-toolcalls-result.e2e.ts
apps/{gateway,api}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes
Files:
apps/gateway/src/chat-toolcalls-result.e2e.tsapps/gateway/src/chat/chat.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
apps/gateway/src/chat-toolcalls-result.e2e.tsapps/gateway/src/chat/chat.tsapps/playground/src/components/provider-icons.tsx
{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}
📄 CodeRabbit inference engine (CLAUDE.md)
For read operations, use
db().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat-toolcalls-result.e2e.tsapps/gateway/src/chat/chat.ts
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
apps/{ui,playground}/**/*.{ts,tsx}: In Next.js apps, use next/link for links and next/navigation’s router for programmatic navigation (avoid tags, window.location, or legacy next/router)
Use cookies for user settings that are not persisted in the database to ensure SSR works
apps/{ui,playground}/**/*.{ts,tsx}: Usenext/linkfor links andnext/navigation's router for programmatic navigation
Use cookies for user settings not saved in the database to ensure SSR works
Files:
apps/playground/src/components/provider-icons.tsx
🧬 Code graph analysis (1)
apps/gateway/src/chat/chat.ts (1)
packages/logger/src/index.ts (2)
error(153-160)logger(181-181)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (1)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (2)
- GitHub Check: autofix
🔇 Additional comments (8)
apps/playground/src/components/provider-icons.tsx (1)
1144-1144: LGTM!The GoogleVertexIcon is properly registered in the ProviderIcons map, following the established pattern for other providers.
apps/gateway/src/chat/chat.ts (7)
1363-1365: LGTM! Google Vertex AI token passing is correctly implemented.The condition properly extends token passing to include
google-vertexalongsidegoogle-ai-studio, which is correct since both Google providers require the API key as a URL query parameter.
2012-2012: Excellent defensive improvement to prevent duplicate [DONE] events.The
doneSentflag ensures a single final[DONE]event is emitted across all streaming completion paths (normal, error, and upstream error). This is properly set in all emission paths and checked before the final emission at lines 2849-2862.
2530-2570: LGTM! Google Vertex AI finish reason handling is correctly implemented.The switch case properly adds
google-vertexalongsidegoogle-ai-studio, sharing the same finish reason mapping logic. The implementation correctly:
- Maps Google's
STOP→ OpenAI'sstoportool_calls(based on function call presence)- Maps Google's
MAX_TOKENS→length- Maps Google's
SAFETY→content_filter- Provides a safe fallback for unknown reasons
2840-2862: Excellent error handling improvements for streaming completion.The refactored logic properly:
- Attempts to send the final usage chunk with error handling (lines 2840-2846)
- Logs any errors but continues execution to ensure [DONE] is sent
- Always sends [DONE] if not already sent, with its own error handling (lines 2848-2862)
This defensive approach ensures clients always receive a proper stream termination even if intermediate operations fail.
2450-2457: Verification complete—no issues found.The
extractContentfunction inapps/gateway/src/chat/tools/extract-content.tscorrectly handles thegoogle-vertexprovider. Bothgoogle-ai-studioandgoogle-vertexare implemented in the same case block with identical extraction logic (lines 8-12), which aligns with the calling code at lines 2450-2457 inchat.ts.
2366-2391: All verification checks passed—no issues found.The
extractTokenUsagefunction inapps/gateway/src/chat/tools/extract-token-usage.tscorrectly handles thegoogle-vertexprovider. The function uses a fall-through case pattern where bothgoogle-ai-studioandgoogle-vertexshare identical handling logic, extracting token usage fromdata.usageMetadataand applying the same estimation fallback. The implementation is consistent and complete.
2471-2487: No issues found. TheextractReasoningfunction correctly handles thegoogle-vertexprovider.The verification confirms that
extract-reasoning.tsincludes thegoogle-vertexcase (lines 20-24) with proper reasoning extraction logic that aligns with thegoogle-ai-studiopattern. The implementation inchat.ts(lines 2471-2487) correctly passes raw data toextractReasoningfor thegoogle-vertexprovider, consistent with the handler implementation.
80f8683 to
331627d
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
packages/models/src/models/google.ts (1)
23-36: Consider refactoring to reduce configuration duplication.Each
google-vertexprovider entry duplicates the correspondinggoogle-ai-studioconfiguration (~180 lines total). This creates a maintenance burden where pricing, capabilities, or other field updates must be made in multiple places, increasing the risk of configuration drift.Consider one of these approaches:
Option 1: Shared configuration objects
const gemini25ProConfig = { modelName: "gemini-2.5-pro", inputPrice: 1.25 / 1e6, outputPrice: 10.0 / 1e6, requestPrice: 0, contextSize: 1000000, maxOutput: undefined, streaming: true, vision: true, tools: true, reasoning: true, jsonOutput: true, }; { id: "gemini-2.5-pro", name: "Gemini 2.5 Pro", family: "google", providers: [ { providerId: "google-ai-studio", ...gemini25ProConfig }, { providerId: "google-vertex", ...gemini25ProConfig }, // ... other providers ], }Option 2: Provider generation helper
function createGoogleProviders(config: ProviderConfig, providerIds: string[]) { return providerIds.map(providerId => ({ providerId, ...config })); }This would ensure consistency and reduce the diff size for future model additions.
Also applies to: 75-89, 112-126, 148-161, 183-196, 240-254, 293-306, 367-380, 402-415, 437-450, 472-485, 507-520, 541-553, 574-586
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
apps/gateway/src/chat-basic.e2e.ts(1 hunks)apps/gateway/src/chat-toolcalls-result.e2e.ts(1 hunks)packages/models/src/models/google.ts(14 hunks)
✅ Files skipped from review due to trivial changes (1)
- apps/gateway/src/chat-basic.e2e.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/gateway/src/chat-toolcalls-result.e2e.ts
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level import; never use require() or dynamic import()
Files:
packages/models/src/models/google.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyin this TypeScript project unless absolutely necessary
Always use top-levelimport; do not userequireor dynamicimport()
Files:
packages/models/src/models/google.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: build / run
- GitHub Check: lint / run
- GitHub Check: test / run
- GitHub Check: generate / run
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (1)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (3)
- GitHub Check: autofix
🔇 Additional comments (2)
packages/models/src/models/google.ts (2)
569-586: LGTM: Consistent configuration between providers.The
gemini-2.0-flashconfiguration correctly maintains consistency betweengoogle-ai-studioandgoogle-vertexproviders, with both having identical capabilities (streaming, vision, tools, jsonOutput).
199-219: ****The premise of this review comment is incorrect. The script output definitively shows that NO preview models—including
gemini-2.5-pro-preview-05-06,gemini-2.5-flash-preview-04-17, and all others—containgoogle-vertexprovider entries. All preview models currently have onlygoogle-ai-studioproviders. There is no inconsistency to address in the new models at lines 199-219 and 325-345.Likely an incorrect or invalid review comment.
Previously, thinkingConfig was only enabled for google-ai-studio, which meant google-vertex models didn't return reasoning content with the thought flag set. This caused reasoning tests to fail. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 participant
Summary
Add Google Vertex AI as a new provider with
gemini-2.5-flash-litemodel support. Implementation reuses Google AI Studio logic with Vertex-specific API endpoint configuration.Changes
Provider Configuration
google-vertexprovider definitionLLM_GOOGLE_VERTEX_API_KEYenvironment variable mapping.env.unified.exampleModel Configuration
google-vertexprovider entry forgemini-2.5-flash-litemodelGateway Implementation
https://aiplatform.googleapis.com/v1/publishers/google/models/alt=sseparameterUI Integration
Testing
Manual testing with curl confirms the API endpoint works correctly:
Build Status
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Chores
Documentation