feat(web-search): native web search with billing - #1390
Conversation
❌ Some deployments failedPreview URLs
Deployed with Ploy |
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds native web search tooling: new web_search tool type, provider capability flags/pricing, routing and request-body mapping, provider response parsing for web citations, cost calculation and DB logging for web searches, streaming and non‑streaming annotation propagation, and UI/docs updates. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Gateway
participant Provider
participant WebSearch as WebSearchEngine
participant DB
Client->>Gateway: POST /chat (includes tools[], web_search tool)
activate Gateway
Gateway->>Gateway: Extract web_search tool\nvalidate provider supports webSearch
Gateway->>Gateway: prepareRequestBody(..., webSearchTool)
Gateway->>Provider: Forward provider-specific request\n(include web_search payload, headers with webSearchEnabled)
deactivate Gateway
activate Provider
Provider->>WebSearch: Perform web search(s)
WebSearch-->>Provider: Return search results + citations
Provider-->>Gateway: Response with annotations / web_search_count
deactivate Provider
activate Gateway
Gateway->>Gateway: parseProviderResponse -> annotations, webSearchCount
Gateway->>Gateway: transformResponseToOpenai(..., annotations)
Gateway->>Gateway: calculateCosts(..., webSearchCount)
Gateway->>DB: Log request/usage (include web_search_cost, webSearchCount)
Gateway-->>Client: Return transformed response (includes annotations, costs)
deactivate Gateway
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Meet Ploy
Inactive
github-actions
Bot
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
apps/gateway/src/chat/tools/transform-response-to-openai.ts (1)
192-231: Missing annotations propagation for inference.net/together.ai/groq when building new response.When
!transformedResponse.idis true (lines 196-230), the message object is constructed without annotations. Other providers include annotations in similar code paths. Consider adding annotations here for consistency.🔎 Proposed fix
message: { role: "assistant", content: content, ...(reasoningContent !== null && { reasoning: reasoningContent, }), + ...(annotations && + annotations.length > 0 && { annotations: annotations }), },apps/gateway/src/chat/chat.ts (1)
3640-3656: Streaming path may be missingwebSearchCountin cost calculation.The non-streaming path passes
webSearchCounttocalculateCosts(line 4267), but the streaming path'scalculateCostscall (lines 3640-3655) does not includewebSearchCount. This could result in web search costs not being calculated for streaming responses.Suggested fix
You'll need to:
- Track
webSearchCountduring streaming (similar to howoutputImageCountis tracked)- Pass it to
calculateCostsin the streaming finally blockThe streaming transformer should accumulate web search count from streaming chunks, then pass it to the cost calculation.
🧹 Nitpick comments (5)
apps/gateway/src/lib/costs.ts (1)
301-307: Consider addingwebSearchPriceto the provider type definition.Using
(providerInfo as any).webSearchPriceworks but bypasses type safety. Consider extending the provider type definition in@llmgateway/modelsto includewebSearchPrice?: numberto avoid theas anycast. This aligns with coding guidelines to avoidanyunless absolutely necessary.Based on coding guidelines: "Never use
anyoras anyunless absolutely necessary in TypeScript code".apps/gateway/src/chat/tools/parse-provider-response.ts (1)
309-338: Consider ifwebSearchCount = 1is appropriate as default for Google grounding.Setting
webSearchCount = 1as a default whengroundingMetadataexists (line 313) before potentially overwriting withwebSearchQueries.length(line 336) works, but ifwebSearchQueriesis empty or missing, it defaults to 1 even when no actual search occurred. Consider whether this should be0initially and only set to1if there are actual grounding chunks.🔎 Proposed refinement
// Extract web search citations from Google grounding metadata const groundingMetadata = json.candidates?.[0]?.groundingMetadata; if (groundingMetadata) { - webSearchCount = 1; // Google doesn't report individual search counts // Extract from groundingChunks (sources) if ( groundingMetadata.groundingChunks && Array.isArray(groundingMetadata.groundingChunks) ) { + if (groundingMetadata.groundingChunks.length > 0) { + webSearchCount = 1; // Google doesn't report individual search counts + } for (const chunk of groundingMetadata.groundingChunks) {packages/models/src/prepare-request-body.ts (2)
237-251: Consider using a typed object instead ofany.The
webSearchobject is typed asanyon line 243. Per coding guidelines, avoidanyunless absolutely necessary. Consider defining a local type or interface for the web search tool configuration.Suggested improvement
- const webSearch: any = { type: "web_search" }; + const webSearch: { type: "web_search"; user_location?: WebSearchTool["user_location"]; search_context_size?: WebSearchTool["search_context_size"] } = { type: "web_search" };
352-365: ZAI web search ignores WebSearchTool configuration options.The ZAI implementation only uses the presence of
webSearchToolto enable search but ignoresuser_location,search_context_size, andmax_usesproperties. If this is intentional (ZAI doesn't support these options), consider adding a brief comment for clarity.apps/gateway/src/chat/chat.ts (1)
471-490: Mutating thetoolsarray may have unintended side effects.The
tools.splice(webSearchToolIndex, 1)on line 488 mutates the original array fromvalidationResult.data. While this works for the current flow, it modifies the validated input data in place. Consider creating a filtered copy instead to avoid potential issues iftoolsis referenced elsewhere.Suggested approach
// Extract web_search tool from tools array if present // The web_search tool is a special tool that enables native web search for providers that support it let webSearchTool: WebSearchTool | undefined; + let filteredTools = tools; if (tools && Array.isArray(tools)) { const webSearchToolIndex = tools.findIndex( (tool: any) => tool.type === "web_search", ); if (webSearchToolIndex !== -1) { // Cast to any to access properties since the schema allows both function and web_search tools const foundTool = tools[webSearchToolIndex] as any; webSearchTool = { type: "web_search", user_location: foundTool.user_location, search_context_size: foundTool.search_context_size, max_uses: foundTool.max_uses, }; // Remove the web_search tool from the tools array so it's not sent as a regular tool - tools.splice(webSearchToolIndex, 1); + filteredTools = tools.filter((_, i) => i !== webSearchToolIndex); } }Then use
filteredToolsinstead oftoolswhen passing toprepareRequestBodyand other downstream functions.
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (15)
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/lib/costs.tspackages/db/src/schema.tspackages/models/src/get-provider-headers.tspackages/models/src/models.tspackages/models/src/models/anthropic.tspackages/models/src/models/google.tspackages/models/src/models/openai.tspackages/models/src/models/zai.tspackages/models/src/prepare-request-body.tspackages/models/src/types.ts
🧰 Additional context used
📓 Path-based instructions (8)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/models/src/models.tsapps/gateway/src/lib/costs.tspackages/models/src/get-provider-headers.tspackages/models/src/models/zai.tspackages/models/src/types.tspackages/models/src/models/anthropic.tspackages/db/src/schema.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tspackages/models/src/models/google.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
packages/models/src/models.tsapps/gateway/src/lib/costs.tspackages/models/src/get-provider-headers.tspackages/models/src/models/zai.tspackages/models/src/types.tspackages/models/src/models/anthropic.tspackages/db/src/schema.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tspackages/models/src/models/google.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
packages/models/src/models.tsapps/gateway/src/lib/costs.tspackages/models/src/get-provider-headers.tspackages/models/src/models/zai.tspackages/models/src/types.tspackages/models/src/models/anthropic.tspackages/db/src/schema.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tspackages/models/src/models/google.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/models/openai.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
packages/models/src/models.tsapps/gateway/src/lib/costs.tspackages/models/src/get-provider-headers.tspackages/models/src/models/zai.tspackages/models/src/types.tspackages/models/src/models/anthropic.tspackages/db/src/schema.tsapps/gateway/src/chat/tools/parse-provider-response.tspackages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tspackages/models/src/models/google.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/models/openai.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/lib/costs.tspackages/db/src/schema.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.ts
packages/db/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Drizzle ORM with latest object syntax for database operations
Files:
packages/db/src/schema.ts
🧬 Code graph analysis (7)
packages/models/src/get-provider-headers.ts (1)
packages/models/src/providers.ts (1)
ProviderId(446-446)
apps/gateway/src/chat/tools/parse-provider-response.ts (1)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)
packages/models/src/prepare-request-body.ts (1)
packages/models/src/types.ts (1)
WebSearchTool(373-395)
apps/gateway/src/chat/tools/types.ts (1)
packages/models/src/types.ts (1)
ToolCall(54-61)
apps/gateway/src/chat/tools/transform-response-to-openai.ts (2)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)packages/db/src/schema.ts (1)
message(589-611)
apps/gateway/src/chat/chat.ts (3)
packages/models/src/types.ts (1)
WebSearchTool(373-395)packages/db/src/types.ts (2)
tools(44-44)tool(18-21)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (1)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Agent
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (1)
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (5)
- GitHub Check: build / run
- GitHub Check: test / run
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: autofix
🔇 Additional comments (29)
packages/db/src/schema.ts (1)
431-431: LGTM! Web search cost field properly integrated.The
webSearchCostfield is correctly typed asreal()and nullable, matching the pattern of other optional cost fields (inputCost,outputCost,cachedInputCost,requestCost). Its placement among cost-related fields maintains logical schema organization.packages/models/src/models.ts (1)
139-146: LGTM! Web search fields well-documented and properly typed.The optional
webSearchandwebSearchPricefields are correctly typed and documented. Making them optional is appropriate since web search is an opt-in feature, ensuring backward compatibility.apps/gateway/src/chat/tools/types.ts (2)
29-40: LGTM! Annotation types well-structured for extensibility.The
UrlCitationAnnotationinterface uses a discriminated union pattern (type: "url_citation") that allows for future annotation types. The nestedurl_citationobject clearly separates metadata from the type discriminator.
49-49: LGTM! Optional annotations field maintains backward compatibility.Making
annotationsoptional inStreamingDeltacorrectly reflects the opt-in nature of web search features.packages/models/src/models/google.ts (1)
530-531: Gemini 3 web search pricing is accurate; remove misleading comparison.The code correctly specifies $14 per 1,000 queries ($0.014 per query) for Gemini 3 web search, which became effective January 5, 2026. However, the comparison to Gemini 2.5 pricing is misleading: Gemini 2.5 uses per-grounded-prompt billing ($35 per 1,000 prompts) while Gemini 3 switched to per-individual-query billing, making direct price comparison invalid since a single prompt may execute multiple queries.
packages/models/src/models/zai.ts (1)
26-27: Web search enablement is correctly restricted to reasoning models without vision.Web search (
webSearch: true) is enabled only for glm-4.5, glm-4.7, and glm-4.6 (all reasoning models without vision capabilities), and is intentionally excluded from:
- Vision models: glm-4.5v, glm-4.6v, glm-4.6v-flashx, glm-4.6v-flash
- Unstable reasoning model: glm-4.5-x
This selective pattern is consistent throughout the configuration.
packages/models/src/get-provider-headers.ts (1)
3-8: LGTM! Clean implementation for conditionally including web search beta header.The use of a block scope for the
anthropiccase and dynamicbetaFeaturesarray construction is well-structured.Also applies to: 19-29
apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (2)
119-153: LGTM! Proper handling of Anthropic web search tool result in streaming.The implementation correctly:
- Filters for
web_search_resulttype items- Builds annotations array with url_citation format
- Only includes annotations in delta when non-empty
456-474: LGTM! Google grounding metadata citation extraction is well-implemented.The code correctly extracts web citations from
groundingChunksand transforms them into the unifiedurl_citationannotation format.packages/models/src/types.ts (1)
373-395: Verify intentional difference inuser_locationstructure between interfaces.
WebSearchTool.user_locationhas a flat structure withcity,region,countrydirectly undertype: "approximate", whileOpenAIWebSearchOptions.user_locationnests these under an additionalapproximateobject. Ensure this aligns with the respective provider API specifications (Anthropic vs OpenAI).Also applies to: 423-436
apps/gateway/src/lib/costs.ts (1)
89-89: LGTM! Web search cost calculation is correctly integrated.The implementation properly:
- Handles null/zero webSearchCount gracefully
- Applies the discount multiplier consistently
- Includes webSearchCost in all return paths
- Adds webSearchCost to the totalCost calculation
Also applies to: 106-106, 191-191, 218-218, 309-313, 320-320
apps/gateway/src/chat/tools/transform-response-to-openai.ts (1)
75-75: LGTM! Annotations are properly propagated across most provider transformations.The implementation consistently uses conditional spread to include annotations when present and non-empty.
Also applies to: 98-99, 160-161, 296-297, 342-343, 380-383, 431-434
apps/gateway/src/chat/tools/parse-provider-response.ts (4)
103-143: Verify if inline citations should contribute towebSearchCountfor Anthropic.Currently,
webSearchCountis set towebSearchBlocks.length(line 109), which counts web search tool result blocks. However, inline citations extracted from text blocks (lines 129-143) do not incrementwebSearchCount. If inline citations represent additional web search results, this might undercount the actual searches performed.
471-504: LGTM! OpenAI Responses API citation extraction handles multiple sources correctly.The use of
Math.max(webSearchCount, webSearchCalls.length)ensures accurate counting when both annotation-based and web_search_call indicators are present.
584-601: LGTM! ZAI-specific web search result extraction is properly handled.The code correctly checks for the ZAI provider and extracts web search results from the provider-specific
web_searchfield in the message.
617-618: LGTM! Return values properly expose annotations and webSearchCount.The conditional null returns (
annotations.length > 0 ? annotations : null) ensure clean API responses when no web search data is present.packages/models/src/prepare-request-body.ts (4)
18-18: LGTM! WebSearchTool import and function signature extension.The import of
WebSearchTooltype and the addition of the optionalwebSearchToolparameter toprepareRequestBodyfollows the existing pattern for optional parameters in this function.Also applies to: 147-147
521-535: LGTM! Anthropic web search implementation correctly uses versioned tool type.The implementation uses
web_search_20250305which aligns with Anthropic's API versioning pattern, and correctly applies onlymax_useswhich is the Anthropic-specific option per theWebSearchToolinterface documentation.
830-836: LGTM! Google grounding implementation is correct.The
google_search: {}configuration enables Google's grounding feature. The empty object is appropriate as Google's grounding API typically doesn't require additional configuration options.
276-313: Theweb_search_optionsstructure is correct and matches OpenAI's API specification.The nested
user_locationstructure withtype: "approximate"and nestedapproximateobject containingcity,region, andcountrymatches OpenAI's official documentation forgpt-4o-search-previewandgpt-4o-mini-search-previewmodels. No changes are needed.packages/models/src/models/openai.ts (1)
70-95: LGTM!gpt-4o-mini-search-previewmodel definition.The configuration follows the same pattern as
gpt-4o-search-previewwith appropriately lower pricing for the mini variant. Thetools: falsesetting correctly reflects that search models don't support additional tools.apps/gateway/src/chat/chat.ts (8)
46-46: LGTM! WebSearchTool import added.The import of
WebSearchToolfrom@llmgateway/modelsaligns with the type usage in the web search tool extraction logic.
2203-2225: LGTM!webSearchToolcorrectly passed toprepareRequestBody.The
webSearchToolparameter is properly passed as the last argument, matching the updated function signature. The optional nature handles the case where no web search tool was extracted.
2317-2319: LGTM! Streaming path correctly setswebSearchEnabledheader option.The
!!webSearchToolcorrectly converts to boolean for the header option, enabling Anthropic's web-search beta feature when applicable.
2832-2834: LGTM! Non-streaming path correctly setswebSearchEnabledheader option.Consistent with the streaming path implementation.
4191-4204: LGTM! NewannotationsandwebSearchCountfields correctly extracted from parsed response.The destructuring properly captures the new web search metadata for use in cost calculation and response transformation.
4274-4304: LGTM!annotationscorrectly passed totransformResponseToOpenai.The annotations parameter enables web search citations to be included in the OpenAI-formatted response, maintaining consistency with the provider's web search output.
4404-4404: ThewebSearchCostfield exists in the log schema (packages/db/src/schema.ts:431aswebSearchCost: real()). The insertion at line 4404 is correct and aligns with the schema definition.
4252-4268: ThecalculateCostsfunction signature has been correctly updated with thewebSearchCountparameter.The function in
apps/gateway/src/lib/costs.tsincludeswebSearchCount: number | null = nullas the final parameter, matching the usage in the code snippet provided.
| webSearch: true, | ||
| webSearchPrice: 0.035, // $35 per 1000 prompts |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's check the file to see the current state
wc -l packages/models/src/models/google.tsRepository: theopenco/llmgateway
Length of output: 105
🏁 Script executed:
# Read the relevant sections of the file
head -n 75 packages/models/src/models/google.ts | tail -n 40Repository: theopenco/llmgateway
Length of output: 970
🌐 Web query:
Google Gemini 2.5 Pro web search pricing per prompt query
💡 Result:
Gemini 2.5 Pro web (Search) grounding: $35 per 1,000 grounded prompts (≈ $0.035 per grounded prompt). Free daily allotments differ by product page — Google AI developer page shows 1,500 free grounded prompts/day [1], while the Vertex AI (Google Cloud) pricing page lists 10,000 free grounded prompts/day for Gemini 2.5 Pro; overage is $35/1,000 grounded prompts. [1][2]
Sources:
[1] Google AI developer pricing — Gemini grounding details.
[2] Google Cloud Vertex AI pricing — Gemini 2.5 Pro grounding and overage rate.
🌐 Web query:
Google Gemini 3 Pro web search pricing billing unit
💡 Result:
Google’s Vertex AI pricing states Grounding with Google Search / Web Grounding: you get 5,000 search queries per month free (aggregated across Gemini 3 models). Excess queries are billed at $14 per 1,000 search queries (billing counted per individual search query). Billing for this starts January 5, 2026. [1]
Source: Google Cloud Vertex AI pricing — Gemini 3 (Grounding / Web Search section). [1]
🏁 Script executed:
# Check lines 72-73
sed -n '70,75p' packages/models/src/models/google.tsRepository: theopenco/llmgateway
Length of output: 220
🏁 Script executed:
# Check lines 381-382, 400-401
sed -n '380,410p' packages/models/src/models/google.tsRepository: theopenco/llmgateway
Length of output: 759
🏁 Script executed:
# Check lines 531, 565, 610, 644 (Gemini 3 models)
sed -n '530,535p; 564,569p; 609,614p; 643,648p' packages/models/src/models/google.tsRepository: theopenco/llmgateway
Length of output: 647
Clarify web search pricing terminology to match official Google documentation.
Gemini 2.5 comment should say "$35 per 1,000 grounded prompts" to match official pricing terminology. Gemini 3 correctly uses "queries". While both refer to the same billing unit, align Gemini 2.5 comments at lines 40-41, 72-73, 381-382, and 400-401 to include "grounded" for consistency with Google's official terminology.
🤖 Prompt for AI Agents
In packages/models/src/models/google.ts around lines 40-41 (and also update same
phrasing at lines 72-73, 381-382, and 400-401), the Gemini 2.5 comment currently
reads "$35 per 1000 prompts"; change those comments to "$35 per 1,000 grounded
prompts" to match Google's official terminology. Replace the existing comment
text exactly at those locations so Gemini 2.5 uses "grounded" and format the
numeric separator as shown (1,000) for consistency.
There was a problem hiding this comment.
Pull request overview
This pull request adds comprehensive native web search support across multiple LLM providers (OpenAI, Anthropic, Google, ZAI), enabling models to retrieve up-to-date information from the web. The implementation introduces a unified web_search tool interface, provider-specific integrations, cost tracking with per-search billing, and citation/annotation support.
Key Changes:
- Unified web search API through a
WebSearchToolinterface that abstracts provider-specific implementations - Per-search cost tracking with provider-specific pricing added to model definitions and persisted in database logs
- Citation extraction and annotation system that surfaces web sources to clients in OpenAI-compatible format
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 21 comments.
Show a summary per file
| File | Description |
|---|---|
packages/models/src/types.ts |
Defines unified web search types: WebSearchTool, WebSearchCitation, and OpenAIWebSearchOptions interfaces |
packages/models/src/prepare-request-body.ts |
Adds web search tool preparation for each provider with provider-specific formatting (Responses API, Chat Completions, Anthropic, Google grounding, ZAI) |
packages/models/src/models/openai.ts |
Introduces two new search-preview models (gpt-4o-search-preview, gpt-4o-mini-search-preview) with native web search capabilities and pricing |
packages/models/src/models/google.ts |
Adds webSearch capability flag and webSearchPrice to Google models supporting grounding |
packages/models/src/models/anthropic.ts |
Adds webSearch capability flag and webSearchPrice ($0.01 per search) to Anthropic models |
packages/models/src/models/zai.ts |
Adds webSearch capability flag and webSearchPrice ($0.01 per search) to ZAI models |
packages/models/src/models.ts |
Extends ProviderModelMapping interface with optional webSearch and webSearchPrice properties |
packages/models/src/get-provider-headers.ts |
Adds conditional Anthropic beta header for web search feature when enabled |
packages/db/src/schema.ts |
Adds webSearchCost column to log table for persisting per-query web search charges |
apps/gateway/src/lib/costs.ts |
Implements web search cost calculation multiplying search count by provider price, included in total cost |
apps/gateway/src/chat/tools/types.ts |
Defines UrlCitationAnnotation and Annotation types for web search citations, extends StreamingDelta to include annotations |
apps/gateway/src/chat/tools/transform-streaming-to-openai.ts |
Parses web search tool results and grounding metadata from streaming responses, transforms to annotation format |
apps/gateway/src/chat/tools/transform-response-to-openai.ts |
Propagates annotations through response transformation so OpenAI-compatible clients receive web search citations |
apps/gateway/src/chat/tools/parse-provider-response.ts |
Extracts web search citations from provider-specific response formats (Anthropic tool results, Google grounding chunks, OpenAI annotations, ZAI web_search) and counts search queries |
apps/gateway/src/chat/chat.ts |
Extracts web_search tool from tools array, passes to request preparation, integrates web search count into cost calculation, persists web search cost in logs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| vision: true, | ||
| tools: true, | ||
| webSearch: true, | ||
| webSearchPrice: 0.035, // $35 per 1000 prompts |
There was a problem hiding this comment.
The pricing comment "$35 per 1000 prompts" is ambiguous and appears in multiple locations. Does this mean $35 per 1000 prompts that use web search, or $35 per 1000 search queries? Based on the implementation in calculateCosts, it appears this should be per search query (multiplied by webSearchCount), not per prompt. The comment should clarify this is "$35 per 1000 searches" or "$0.035 per search" to avoid confusion.
| (item: any) => item.type === "web_search_call", | ||
| ); | ||
| if (webSearchCalls.length > 0) { | ||
| webSearchCount = Math.max(webSearchCount, webSearchCalls.length); |
There was a problem hiding this comment.
The webSearchCount is being incremented for each annotation (line 482), but then it's compared with webSearchCalls.length using Math.max (line 503). This logic is inconsistent - if there are multiple citations from a single search call, the count would be inflated. The search count should represent the number of search queries made, not the number of citations. Consider tracking search calls separately from citations, or using only webSearchCalls.length as the authoritative count when available.
| webSearchCount = Math.max(webSearchCount, webSearchCalls.length); | |
| // Use the number of web_search_call items as the authoritative search count | |
| webSearchCount = webSearchCalls.length; |
| // Remove the web_search tool from the tools array so it's not sent as a regular tool | ||
| tools.splice(webSearchToolIndex, 1); |
There was a problem hiding this comment.
The tools.splice(webSearchToolIndex, 1) operation mutates the original tools array from the request, which could cause issues if the tools array is used elsewhere or if there are multiple attempts to process the same request. This side effect may be unexpected. Consider creating a new filtered array instead of mutating the input.
| // Remove the web_search tool from the tools array so it's not sent as a regular tool | |
| tools.splice(webSearchToolIndex, 1); | |
| // Remove the web_search tool from the tools array so it's not sent as a regular tool. | |
| // Use a non-mutating operation to avoid side effects on the original array. | |
| tools = tools.filter((_, index) => index !== webSearchToolIndex); |
| vision: true, | ||
| tools: true, | ||
| webSearch: true, | ||
| webSearchPrice: 0.035, // $35 per 1000 prompts |
There was a problem hiding this comment.
The pricing comment "$35 per 1000 prompts" is ambiguous. Does this mean $35 per 1000 prompts that use web search, or $35 per 1000 search queries? Based on the implementation in calculateCosts, it appears this should be per search query (multiplied by webSearchCount), not per prompt. The comment should clarify this is "$35 per 1000 searches" or "$0.035 per search" to avoid confusion.
| ...(toolResults && { tool_calls: toolResults }), | ||
| ...(images && images.length > 0 && { images }), | ||
| ...(annotations && | ||
| annotations.length > 0 && { annotations: annotations }), |
There was a problem hiding this comment.
The spread operator { annotations: annotations } is redundant. When the property name and variable name are the same, you can use the shorthand { annotations } instead.
| annotations.length > 0 && { annotations: annotations }), | |
| annotations.length > 0 && { annotations }), |
| }), | ||
| ...(toolResults && { tool_calls: toolResults }), | ||
| ...(annotations && | ||
| annotations.length > 0 && { annotations: annotations }), |
There was a problem hiding this comment.
The spread operator { annotations: annotations } is redundant. When the property name and variable name are the same, you can use the shorthand { annotations } instead.
| annotations.length > 0 && { annotations: annotations }), | |
| annotations.length > 0 && { annotations }), |
| if (annotation.type === "url_citation") { | ||
| webSearchCount++; | ||
| annotations.push({ | ||
| type: "url_citation", | ||
| url_citation: { | ||
| url: annotation.url_citation?.url || annotation.url || "", | ||
| title: annotation.url_citation?.title || annotation.title, | ||
| start_index: | ||
| annotation.url_citation?.start_index ?? | ||
| annotation.start_index, | ||
| end_index: | ||
| annotation.url_citation?.end_index ?? annotation.end_index, | ||
| }, | ||
| }); | ||
| } | ||
| } | ||
|
|
||
| // For ZAI, extract web search info if present | ||
| // ZAI includes web_search content in the response | ||
| if (usedProvider === "zai") { | ||
| const webSearchResults = | ||
| json.choices?.[0]?.message?.web_search || null; | ||
| if (webSearchResults && Array.isArray(webSearchResults)) { | ||
| webSearchCount = webSearchResults.length; |
There was a problem hiding this comment.
Similar to the issue in the OpenAI Responses API parsing, webSearchCount is being incremented for each annotation (line 568), which conflates the number of search queries with the number of citations returned. Then at line 590, for ZAI, webSearchCount is overwritten with webSearchResults.length. This inconsistent logic means the count will be unreliable for billing. The count should consistently represent the number of search API calls made, not the number of citations. Consider refactoring to track search calls distinctly from citation results.
| maxOutput: 16384, | ||
| streaming: true, | ||
| vision: true, | ||
| tools: false, // Search models don't support additional tools |
There was a problem hiding this comment.
The comment indicates that search models don't support additional tools, but the implementation in prepareRequestBody (lines 280-312) has logic to handle both cases - search models with web_search_options and regular models with web_search as a tool. If search preview models truly don't support additional tools beyond web search, then any user-provided tools should be rejected when using these models, not silently accepted. Consider adding validation to prevent tools from being passed to search-only models, or updating the documentation if tools are actually supported.
| tools: false, // Search models don't support additional tools | |
| tools: false, // Disable additional tools via this model definition; web search is handled natively |
| if (!responsesBody.tools) { | ||
| responsesBody.tools = []; | ||
| } | ||
| const webSearch: any = { type: "web_search" }; |
There was a problem hiding this comment.
Using any type for webSearch object bypasses type safety. While the web search tool format may vary by provider, consider defining provider-specific types or at least using a more specific type annotation to catch potential errors at compile time.
| const webSearch: any = { type: "web_search" }; | |
| const webSearch: WebSearchTool = { type: "web_search" }; |
| }), | ||
| ...(toolResults && { tool_calls: toolResults }), | ||
| ...(annotations && | ||
| annotations.length > 0 && { annotations: annotations }), |
There was a problem hiding this comment.
The spread operator { annotations: annotations } is redundant in multiple places. When the property name and variable name are the same, you can use the shorthand { annotations } instead.
| annotations.length > 0 && { annotations: annotations }), | |
| annotations.length > 0 && { annotations }), |
steebchen
changed the title
Meet Ploy
Failure
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/chat/chat.ts (1)
4284-4300: Web search cost not included in streaming path.The
webSearchCountparameter is passed tocalculateCostsfor non-streaming responses (line 4299), but the streaming path (around line 3672-3687) does not includewebSearchCount. This may result in incomplete cost calculations for streaming requests that use web search.🔎 Verify streaming cost calculation
#!/bin/bash # Check if webSearchCount is passed in streaming calculateCosts calls rg -n "calculateCosts" apps/gateway/src/chat/chat.ts -A 15 | head -100
♻️ Duplicate comments (7)
packages/models/src/prepare-request-body.ts (5)
262-262: Use typed interface instead ofany.Per coding guidelines, avoid
anyunless absolutely necessary. Consider using a more specific type for the web search object.Based on past review comment on line 262.
301-301: Use typed interface instead ofanyfor web search options.Per coding guidelines, avoid
anyunless absolutely necessary. ThewebSearchOptionsobject has a known structure that could be typed.Based on past review comment on lines 301-323.
550-550: Use typed interface for Anthropic web search tool.Per coding guidelines, avoid
anyunless absolutely necessary.Based on past review comment on line 550.
371-384: ZAI web search ignores user-provided options.The ZAI implementation hardcodes
search_engine: "search-prime"and ignoresuser_location,search_context_size, andmax_usesfrom thewebSearchToolparameter. Consider either passing through relevant options or documenting the limitation.Based on past review comment on lines 377-382.
861-867: Google web search tool structure may need verification.The code adds
{ google_search: {} }to the tools array. When function declarations exist, this creates a mixed tools array structure. Verify this is the correct approach per Google's API documentation.Based on past review comment on lines 861-866.
apps/gateway/src/chat/chat.ts (1)
486-505: Array mutation viasplicemodifies the original tools array.The
tools.splice(webSearchToolIndex, 1)operation mutates the originaltoolsarray from the request. Consider using a non-mutating approach to avoid potential side effects.Based on past review comment on lines 502-503.
🔎 Suggested non-mutating approach
- // Remove the web_search tool from the tools array so it's not sent as a regular tool - tools.splice(webSearchToolIndex, 1); + // Remove the web_search tool from the tools array so it's not sent as a regular tool + // Use a non-mutating operation to avoid side effects on the original array + tools = tools.filter((_, index) => index !== webSearchToolIndex);Note: This requires
toolsto be declared withletinstead of destructured fromvalidationResult.data.packages/models/src/types.ts (1)
441-454: Document the nested structure for OpenAI web search options.The
OpenAIWebSearchOptions.user_locationhas a nestedapproximateproperty, which differs from bothWebSearchToolandOpenAIWebSearchToolInput. This is the correct format for OpenAI's API, but the structural difference should be documented to clarify why the transformation inprepare-request-body.ts(lines 303-310) is necessary.Based on past review comment on lines 444-454.
🧹 Nitpick comments (2)
apps/docs/content/features/web-search.mdx (1)
347-360: Error message mismatch with code.The error message example mentions "does not support tool calls" but based on the code in
chat.ts(lines 816-820), the actual error message is slightly different. Consider aligning the documentation with the actual implementation to avoid confusion.packages/models/src/types.ts (1)
391-401: Inconsistency betweenWebSearchTool.user_locationandOpenAIWebSearchToolInput.user_location.
WebSearchTool.user_locationincludes atype: "approximate"field, butOpenAIWebSearchToolInput.user_location(lines 127-132) does not. This creates a type mismatch when extracting web search tools from the request inchat.ts. The extraction at lines 496-501 inchat.tscopiesuser_locationdirectly, which would not include thetypefield.Consider aligning the structures or documenting the transformation that occurs between the API input format and the internal format.
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (3)
apps/admin/src/lib/api/v1.d.tsis excluded by!**/v1.d.tsapps/playground/src/lib/api/v1.d.tsis excluded by!**/v1.d.tsapps/ui/src/lib/api/v1.d.tsis excluded by!**/v1.d.ts
📒 Files selected for processing (5)
apps/docs/content/features/web-search.mdxapps/gateway/src/chat/chat.tspackages/db/src/types.tspackages/models/src/prepare-request-body.tspackages/models/src/types.ts
🧰 Additional context used
📓 Path-based instructions (8)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/models/src/types.tspackages/db/src/types.tsapps/gateway/src/chat/chat.tspackages/models/src/prepare-request-body.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
packages/models/src/types.tspackages/db/src/types.tsapps/gateway/src/chat/chat.tspackages/models/src/prepare-request-body.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
packages/models/src/types.tspackages/db/src/types.tsapps/gateway/src/chat/chat.tspackages/models/src/prepare-request-body.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
packages/models/src/types.tspackages/db/src/types.tsapps/gateway/src/chat/chat.tspackages/models/src/prepare-request-body.ts
packages/db/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Drizzle ORM with latest object syntax for database operations
Files:
packages/db/src/types.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/db/src/types.tsapps/gateway/src/chat/chat.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/chat.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/chat.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:14.219Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-03T12:42:14.219Z
Learning: Applies to **/*.{ts,tsx} : Never use `any` or `as any` unless absolutely necessary in TypeScript code
Applied to files:
packages/models/src/prepare-request-body.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: lint / run
- GitHub Check: generate / run
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (14)
packages/models/src/prepare-request-body.ts (2)
22-29: LGTM!The
isFunctionTooltype guard is well-implemented with proper typing and clear logic.
176-183: LGTM!Clean separation of concerns - filtering function tools from web_search tools before building the request body. The comment clearly explains the reasoning.
apps/gateway/src/chat/chat.ts (7)
196-218: LGTM!The Zod schema correctly defines the union type for function and web_search tools with appropriate optional fields for web search configuration.
799-821: LGTM!The validation logic correctly handles the web search capability check. The
isWebSearchOnlydetermination and the conditional validation allowing web-search-only requests for models that support web search is well-implemented.
2349-2351: LGTM!Provider headers correctly include
webSearchEnabledflag for streaming requests.
3864-3866: LGTM!Provider headers correctly include
webSearchEnabledflag for non-streaming requests.
4234-4236: LGTM!The
parseProviderResponsecall correctly extractsannotationsandwebSearchCountfrom the provider response.
4335-4336: LGTM!Annotations are correctly passed to
transformResponseToOpenaifor inclusion in the final response.
4436-4436: LGTM!Web search cost is correctly persisted in the log entry.
packages/db/src/types.ts (2)
18-37: LGTM!Clean refactoring of the tool schema to support both function and web_search tool types. The union type approach with
functionToolandwebSearchToolis well-structured and type-safe.
60-60: LGTM!The
toolsarray correctly uses the updated union type.packages/models/src/types.ts (3)
114-140: LGTM!The function and web search tool input types are well-defined. The union type
OpenAIToolInputprovides proper type safety for API requests accepting both tool types.
385-413: LGTM!The
WebSearchToolinterface is well-documented with JSDoc comments explaining each field's purpose and provider-specific applicability. The type structure correctly represents the unified format.
415-439: LGTM!The
WebSearchCitationinterface properly defines the citation structure with all necessary fields for source attribution.
| | Provider | Models | Pricing | | ||
| | --------- | -------------------------------------------------------- | ---------------- | | ||
| | OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.03 per search | | ||
| | Anthropic | `claude-sonnet-4-20250514` | $0.01 per search | | ||
| | Google | `gemini-2.0-flash`, `gemini-2.5-pro`, `gemini-2.5-flash` | Free (grounding) | | ||
| | Z.AI | `zai-search` | Varies | |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Search for Anthropic model references in provider configuration
rg -n "claude-sonnet" --type=ts -C3Repository: theopenco/llmgateway
Length of output: 5439
🏁 Script executed:
# Search for web_search tool/model configuration
rg -n "web_search" --type=ts -C2Repository: theopenco/llmgateway
Length of output: 15262
🏁 Script executed:
# Find provider configuration files
fd -t f "provider\|anthropic" --type=f | grep -E "\.(ts|js|json)$" | head -20Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Look for model name definitions in Anthropic-related files
fd -n "*anthropic*" -t fRepository: theopenco/llmgateway
Length of output: 294
🏁 Script executed:
# Check OpenAI search models
rg -n "gpt-4o.*search" --type=ts -C2Repository: theopenco/llmgateway
Length of output: 1802
🏁 Script executed:
# Check Google Gemini models
rg -n "gemini" --type=ts -C2Repository: theopenco/llmgateway
Length of output: 30669
🏁 Script executed:
# Check Z.AI model
rg -n "zai-search" --type=ts -C2Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Search for Z.AI or ZAI in the codebase
rg -i "zai" --type=ts -C2Repository: theopenco/llmgateway
Length of output: 13493
🏁 Script executed:
# Look for Z.AI in model definitions
rg -i "z\.ai\|zai" --type=ts --type=jsonRepository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
cat apps/docs/content/features/web-search.mdxRepository: theopenco/llmgateway
Length of output: 9119
🏁 Script executed:
# Final verification: Search for any "zai-search" string in codebase
rg "zai-search" --type=ts --type=json --type=mdxRepository: theopenco/llmgateway
Length of output: 91
🏁 Script executed:
# Search for zai-search in all files
rg "zai-search"Repository: theopenco/llmgateway
Length of output: 279
Correct the Z.AI model name in the provider table.
The Z.AI entry lists zai-search, but this model does not exist in the codebase. The available Z.AI models are GLM variants (e.g., glm-4.5, glm-4.7, glm-4.6). Update the table to reference an actual supported Z.AI model, or clarify if zai-search is a planned feature.
(The Anthropic model name claude-sonnet-4-20250514 is correct and does not require changes.)
🤖 Prompt for AI Agents
In apps/docs/content/features/web-search.mdx around lines 27 to 32, the Z.AI
model listed as `zai-search` is incorrect; replace it with an actual supported
GLM variant (for example `glm-4.7`) or add a clarifying note if it’s a planned
feature. Update the table cell so the Z.AI row shows a real model name (e.g.,
`glm-4.7`) and adjust the Pricing column text if needed (keep "Varies" if
unknown); do not change the Anthropic entry.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
packages/models/src/models/openai.ts (2)
44-69: Fix incorrect web search pricing and update release dates.The
webSearchPriceis set to0.03($30 per 1,000 searches), but OpenAI's official pricing is $10.00 per 1,000 web-search calls (equals0.01per search). Additionally, thereleasedAtandpublishedAtdates of 2024-10-01 are outdated; the current model snapshot is gpt-4o-search-preview-2025-03-11.🔎 Proposed fix
{ id: "gpt-4o-search-preview", name: "GPT-4o Search Preview", description: "GPT-4o with native web search capabilities for up-to-date information.", family: "openai", - releasedAt: new Date("2024-10-01"), - publishedAt: new Date("2024-10-01"), + releasedAt: new Date("2025-03-11"), + publishedAt: new Date("2025-03-11"), providers: [ { providerId: "openai", modelName: "gpt-4o-search-preview", inputPrice: 2.5 / 1e6, outputPrice: 10.0 / 1e6, requestPrice: 0, webSearch: true, - webSearchPrice: 0.03, // $30 per 1000 searches + webSearchPrice: 0.01, // $10 per 1000 searches contextSize: 128000, maxOutput: 16384, streaming: true, vision: true, tools: false, // Search models don't support additional tools jsonOutput: false, }, ], },
167-169: Add missingwebSearchPricefor gpt-4o.The model has
webSearch: trueenabled but lacks awebSearchPricedefinition. This creates an inconsistency where the cost calculation defaults to 0, making web search free for this model, while all other web-search-enabled models explicitly define pricing.🔎 Proposed fix
streaming: true, vision: true, tools: true, webSearch: true, // Supports web_search tool via Responses API + webSearchPrice: 0.01, // $10 per 1000 searches jsonOutputSchema: true, jsonOutput: true,apps/docs/content/features/web-search.mdx (2)
33-33: Correct the Z.AI model name in the provider table.The Z.AI entry lists
zai-search, but this model does not exist in the codebase. The available Z.AI models are GLM variants (e.g.,glm-4.5,glm-4.7,glm-4.6). Update the table to reference an actual supported Z.AI model.🔎 Proposed fix
| Google | `gemini-2.0-flash`, `gemini-2.5-pro`, `gemini-2.5-flash` | Free (grounding) | -| Z.AI | `zai-search` | Varies | +| Z.AI | `glm-4.7` (and other GLM variants) | Varies |
385-388: Correct Z.AI model reference in provider notes.The Z.AI section references
zai-search, which doesn't exist. Update this to reference actual GLM model variants available in the codebase.🔎 Proposed fix
### Z.AI - - Dedicated search model (`zai-search`) - - Optimized for search queries + - GLM model variants (e.g., `glm-4.7`) support web search + - Pricing varies by model
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/docs/content/features/web-search.mdxpackages/models/src/models/openai.ts
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
packages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
packages/models/src/models/openai.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
packages/models/src/models/openai.ts
🔇 Additional comments (2)
apps/docs/content/features/web-search.mdx (1)
44-348: LGTM! Well-structured documentation with comprehensive examples.The documentation provides clear, accurate examples covering basic usage, SDKs (Python/TypeScript), streaming, citations, cost tracking, and combining with function tools. The code snippets correctly demonstrate the web search API and match the implementation in the codebase.
packages/models/src/models/openai.ts (1)
685-687: GPT-5 is now officially released; web search pricing requires documentation review.GPT-5 was released by OpenAI in 2025, so these models are no longer speculative future releases. However, the specific web search pricing values (0.02–0.05 per search) could not be verified against official OpenAI documentation through public sources. Confirm these values align with current OpenAI pricing documentation for the web search capability in the Responses API.
Also applies to: 752-754, 818-820, 916-918, 982-984, 1012-1014, 1062-1064, 1110-1112, 1175-1177, 1225-1227
| | OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.1`, `gpt-5.2`, `gpt-5.2-pro` | $0.02-0.05 / search | | ||
| | OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.025-0.03 / search | |
There was a problem hiding this comment.
Update pricing to match corrected model definitions.
The pricing for OpenAI models should be updated to reflect the correct web search costs. If the pricing fix for gpt-4o-search-preview is applied (changing from $0.03 to $0.01), this table should show $0.01-0.025 / search instead of $0.025-0.03 / search for the search-preview models. Additionally, verify the GPT-5 series pricing range ($0.02-0.05) matches the model definitions.
🔎 Proposed fix (if gpt-4o-search-preview pricing is corrected)
| Provider | Models | Pricing |
| --------- | ------------------------------------------------------------------------ | -------------------- |
| OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.1`, `gpt-5.2`, `gpt-5.2-pro` | $0.02-0.05 / search |
-| OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.025-0.03 / search |
+| OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.01-0.025 / search |
| Anthropic | `claude-sonnet-4-20250514` | $0.01 per search |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.1`, `gpt-5.2`, `gpt-5.2-pro` | $0.02-0.05 / search | | |
| | OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.025-0.03 / search | | |
| | OpenAI | `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.1`, `gpt-5.2`, `gpt-5.2-pro` | $0.02-0.05 / search | | |
| | OpenAI | `gpt-4o-search-preview`, `gpt-4o-mini-search-preview` | $0.01-0.025 / search | |
🤖 Prompt for AI Agents
In apps/docs/content/features/web-search.mdx around lines 29 to 30, the pricing
row for the OpenAI search-preview models is outdated and should reflect the
corrected `gpt-4o-search-preview` cost; change the second table row's price from
`$0.025-0.03 / search` to `$0.01-0.025 / search`. Also verify the GPT-5 series
row (`gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5.1`, `gpt-5.2`, `gpt-5.2-pro`)
matches the authoritative model definitions and update its price range if those
definitions show a different range than `$0.02-0.05 / search`.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/ui/src/components/models/all-models.tsx
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/ui/src/components/models/all-models.tsx
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/ui/src/components/models/all-models.tsx
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/ui/src/components/models/all-models.tsx
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
apps/{ui,playground}/**/*.{ts,tsx}: Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Next.js App Router with React Server Components for frontend developmentUse cookies for user-settings which are not saved in the database to ensure SSR works
Files:
apps/ui/src/components/models/all-models.tsx
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/ui/src/components/models/all-models.tsx
apps/{ui,playground,docs}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
next/linkfor links andnext/navigation's router for programmatic navigation
Files:
apps/ui/src/components/models/all-models.tsx
🔇 Additional comments (6)
apps/ui/src/components/models/all-models.tsx (6)
8-8: LGTM! Globe icon import for web search.The import is correctly added and used consistently throughout the file for web search capability indication.
135-135: LGTM! Consistent filter state initialization.The webSearch filter initialization follows the same pattern as other capability filters and correctly reads from URL search params.
268-273: LGTM! Consistent filtering logic.The webSearch filtering logic correctly follows the established pattern for other capability filters and appropriately checks provider-level capabilities.
597-603: LGTM! Web Search capability icon added consistently.The capability icon definition follows the established pattern and uses an appropriate Globe icon with a distinct color.
618-618: LGTM! Clear filters correctly resets web search.The webSearch filter is properly reset to false in state and removed from URL parameters, consistent with other capability filters.
Also applies to: 638-639
718-723: LGTM! Web Search filter option added correctly.The filter checkbox follows the established UI pattern for capability filters and correctly updates both state and URL parameters.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
apps/ui/src/components/models/all-models.tsx (1)
1203-1218: Verify discount handling for web search pricing.The Web Search pricing display doesn't apply
provider.discountlike other price columns do. This creates an inconsistency:
- Other prices: Use
formatPrice(provider.inputPrice, provider.discount)which applies discounts- Web Search: Displays
provider.webSearchPrice * 1000without discount considerationQuestions:
- Should web search pricing honor provider discounts?
- If discounts don't apply to web search, is this intentional?
💡 Optional: Extract web search price formatting to helper function
For consistency and reusability, consider extracting the web search price formatting logic:
const formatWebSearchPrice = (webSearch: boolean | undefined, price: number | undefined, discount?: number) => { if (!webSearch) return "—"; if (price === undefined) return "Free"; const basePrice = price * 1000; if (discount) { const discountedPrice = (basePrice * (1 - discount)).toFixed(2); return ( <div className="flex flex-col"> <div className="flex items-center gap-1"> <span className="line-through text-muted-foreground text-xs"> ${basePrice.toFixed(2)} </span> <span className="text-green-600 font-semibold"> ${discountedPrice} </span> </div> </div> ); } return `$${basePrice.toFixed(2)}/1K`; };Then use it in the table cell:
<div className="text-sm font-mono"> {formatWebSearchPrice(provider.webSearch, provider.webSearchPrice, provider.discount)} </div>This would:
- Apply discounts consistently if they're applicable
- Make the code more maintainable
- Match the pattern used for other price columns
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/all-models.tsx
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
apps/{ui,playground}/**/*.{ts,tsx}: Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Next.js App Router with React Server Components for frontend developmentUse cookies for user-settings which are not saved in the database to ensure SSR works
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
apps/{ui,playground,docs}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
next/linkfor links andnext/navigation's router for programmatic navigation
Files:
apps/ui/src/components/models/all-models.tsxapps/ui/src/components/dashboard/log-card.tsx
🧬 Code graph analysis (2)
apps/ui/src/components/models/all-models.tsx (2)
packages/db/src/schema.ts (2)
model(658-687)provider(624-656)apps/ui/src/lib/components/table.tsx (2)
TableHead(110-110)TableCell(112-112)
apps/ui/src/components/dashboard/log-card.tsx (1)
packages/db/src/schema.ts (1)
log(381-492)
🔇 Additional comments (8)
apps/ui/src/components/dashboard/log-card.tsx (1)
458-463: LGTM! Web Search Cost display follows existing patterns.The conditional rendering for Web Search Cost is implemented correctly and consistently with the existing
cachedInputCostdisplay pattern. The formatting (toFixed(8)), conditional logic, and placement within the Cost Information section are all appropriate.apps/ui/src/components/models/all-models.tsx (7)
8-8: LGTM: Globe icon import for Web Search.The import is correctly added for the Web Search capability icon.
135-135: LGTM: Web Search filter initialization.Correctly initializes the webSearch filter from URL parameters, consistent with other capability filters.
268-273: LGTM: Web Search filtering logic.The filtering logic correctly checks if any provider supports web search, consistent with other capability filters.
597-603: LGTM: Web Search capability icon.Correctly adds the Globe icon for Web Search capability with appropriate styling and label.
618-618: LGTM: Web Search filter reset.Correctly resets the webSearch filter to false and removes it from URL parameters when clearing filters.
Also applies to: 638-639
718-723: LGTM: Web Search filter UI.The Web Search checkbox is correctly integrated into the capabilities filter panel with consistent styling and behavior.
983-985: LGTM: Web Search table header.The "Web Search" column header is correctly added with appropriate styling and positioning.
![terragon-labs[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
- Introduce WebSearchTool type to unify web search tool config across providers. - Extract and parse web search citations and counts from provider responses (Anthropic, Google, OpenAI, ZAI). - Include web search annotations in transformed OpenAI-compatible responses and streaming deltas. - Add webSearchTool handling in chat core logic for tools and request preparation. - Extend cost calculations and logging to include web search query costs. - Update provider models to mark those supporting web search with pricing info. - Enhance provider headers to support enabling web search beta features. - Support web search in request body preparations for Anthropic, Google, OpenAI, ZAI. This enables native web search capabilities for supported models and tracks usage/costs accordingly. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
- Introduce "web_search" tool type alongside existing function tools in API schema - Support optional user location, search context size, and max uses parameters - Update request validation and model capability checks to allow web search-only requests - Modify request preparation to separate web search tool handling from function tools - Add extensive web search feature documentation in docs/content/features/web-search.mdx - Add type definitions for web search tools across relevant packages - Enhance gateway and model packages to support and process web search tools This enables real-time information retrieval from the internet by models via the new web search tool in the LLM Gateway. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
- Added GPT-5 series models with web search capabilities in openai.ts - Included pricing and support for Responses API - Updated web-search.mdx to reflect GPT-5 model support and pricing - Clarified models that support web search alongside function tools Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
- Include 'webSearch' as a filterable capability for models - Add Globe icon to represent web search capability - Update filters and UI to display web search status - Add 'Web Search' column to models table header Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
Added a web search cost display in the LogCard component to show associated costs when present. Also updated the AllModels component to show web search pricing per provider, supporting free, priced, or unavailable statuses. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
b71b8aa to
71ee641
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/chat/chat.ts (1)
4481-4532: AddwebSearchCostto streaming path logs.The streaming path's
insertLogcall (around line 3818) is missingwebSearchCost, even though it's calculated viacalculateCosts()like the non-streaming path. This creates an inconsistency where web search costs are only tracked for non-streaming requests. AddwebSearchCost: costs.webSearchCostto the streaminginsertLogcall to ensure consistent cost tracking across both request types.
♻️ Duplicate comments (15)
packages/models/src/models/google.ts (1)
40-41: Clarify pricing terminology (duplicate concern).The comment "$35 per 1000 prompts" has been flagged in previous reviews as ambiguous and inconsistent with Google's official terminology. Google documentation refers to these as "grounded prompts" for Gemini 2.5 models, and the billing is per web search query, not per LLM prompt. This concern has already been raised in prior review cycles.
Also applies to: 72-73, 381-382, 400-401
packages/models/src/models/anthropic.ts (3)
177-199: Missing webSearch configuration (duplicate concern).According to previous reviews citing Anthropic's web search API documentation,
claude-sonnet-4-20250514should includewebSearch: trueandwebSearchPrice: 0.01. This critical issue was flagged in earlier review cycles but remains unaddressed in the current code.
361-384: Missing webSearch configuration (duplicate concern).
claude-opus-4-20250514is missingwebSearchandwebSearchPricefields. Previous reviews confirmed this model supports web search per Anthropic's official documentation. This critical issue remains unresolved.
406-428: Missing webSearch configuration (duplicate concern).
claude-opus-4-1-20250805lackswebSearchandwebSearchPriceconfiguration despite supporting web search according to Anthropic's API documentation (as noted in previous reviews). This critical issue needs resolution.packages/models/src/types.ts (1)
391-454: Structural inconsistency in user_location (duplicate concern).Previous reviews identified that
OpenAIWebSearchOptions(lines 444-454) usesuser_location.approximate.city(double-nested), whileWebSearchTool(lines 391-413) usesuser_location.citywithtype: "approximate"at the parent level. This structural inconsistency creates confusion about the actual OpenAI API format. The concern has been raised in prior reviews but remains unaddressed.packages/models/src/models/openai.ts (2)
44-69: Incorrect pricing and release date (duplicate concern).Previous reviews identified that
webSearchPriceshould be0.01(not0.03) and the release date should benew Date("2025-03-11")to match OpenAI's official pricing ($10 per 1,000 web-search calls) and the gpt-4o-search-preview-2025-03-11 snapshot. This critical issue was flagged earlier but remains unresolved.
167-169: Missing webSearchPrice despite webSearch enabled.While Line 167 enables
webSearch: true, there is no correspondingwebSearchPricefield. Although a previous review comment indicated this was "Addressed in commit 7061c92", the field is absent in the current code. Without this field, the cost calculation defaults to zero, making web search free forgpt-4o, which is inconsistent with all other web-search-enabled models.🔎 Add webSearchPrice
webSearch: true, // Supports web_search tool via Responses API +webSearchPrice: 0.01, // $10 per 1000 searches jsonOutputSchema: true,apps/docs/content/features/web-search.mdx (1)
33-33: Z.AI modelzai-searchmay not exist.This was flagged in a previous review. Verify if
zai-searchis a valid model or update to reference an actual supported Z.AI model (e.g.,glm-4.5,glm-4.7).apps/gateway/src/lib/costs.ts (1)
301-307: Remove unnecessaryas anycast forwebSearchPrice.The
ProviderModelMappinginterface should includewebSearchPriceas an optional property (per the PR addingwebSearchPriceto model definitions). This cast bypasses type safety unnecessarily.🔎 Proposed fix
- const webSearchPrice = new Decimal((providerInfo as any).webSearchPrice || 0); + const webSearchPrice = new Decimal(providerInfo.webSearchPrice ?? 0);apps/gateway/src/chat/tools/parse-provider-response.ts (2)
560-583:webSearchCountconflates citations with search calls.This incrementing
webSearchCountfor each annotation (line 562) counts citations rather than actual search API calls. TheMath.maxwithwebSearchCalls.length(line 583) attempts to correct this but creates unreliable logic. For accurate billing, use onlywebSearchCalls.lengthas the authoritative count when available.This issue was flagged in a previous review.
646-661: SamewebSearchCountissue in Chat Completions path.Similar to the Responses API path, incrementing
webSearchCountfor each annotation conflates citations with search calls. Consider tracking these separately or using a more reliable indicator of actual search operations.packages/models/src/prepare-request-body.ts (3)
339-352: Consider defining a typed interface for web search options.Using
anytype for thewebSearchobject bypasses type safety. Per coding guidelines, avoidanyunless absolutely necessary.🔎 Proposed fix
- const webSearch: any = { type: "web_search" }; + const webSearch: { + type: "web_search"; + user_location?: WebSearchTool["user_location"]; + search_context_size?: WebSearchTool["search_context_size"]; + } = { type: "web_search" };Based on learnings, avoid
anyoras anyunless absolutely necessary.
455-466: ZAI web search ignoresuser_locationandsearch_context_size.The ZAI implementation hardcodes
search_engine: "search-prime"and ignores theuser_locationandsearch_context_sizeproperties fromWebSearchTool. If ZAI supports these options, consider passing them through; otherwise, document which properties are supported per provider.This was flagged in a previous review.
943-948: Verify Google web search tool structure.The code pushes
{ google_search: {} }to the tools array. A previous review noted that Google's API may expectgoogleSearchRetrievalinstead ofgoogle_searchfor grounding. Verify this matches Google's current API documentation.Google Gemini API google_search vs googleSearchRetrieval grounding tool formatapps/gateway/src/chat/chat.ts (1)
535-554: Array mutation and unnecessaryas anycasts.Two issues in this segment:
Array mutation (previously flagged):
tools.splice(webSearchToolIndex, 1)mutates the originaltoolsarray from the validated request data. This side effect could cause issues iftoolsis referenced elsewhere.
as anyusage: The coding guidelines state "Never useanyoras anyunless absolutely necessary." A type guard or proper typing would be cleaner.🔎 Suggested refactor to avoid mutation and `as any`
// Extract web_search tool from tools array if present // The web_search tool is a special tool that enables native web search for providers that support it let webSearchTool: WebSearchTool | undefined; + let functionTools = tools; if (tools && Array.isArray(tools)) { - const webSearchToolIndex = tools.findIndex( - (tool: any) => tool.type === "web_search", + const webSearchToolEntry = tools.find( + (tool): tool is { type: "web_search"; user_location?: WebSearchTool["user_location"]; search_context_size?: WebSearchTool["search_context_size"]; max_uses?: number } => + tool.type === "web_search", ); - if (webSearchToolIndex !== -1) { - // Cast to any to access properties since the schema allows both function and web_search tools - const foundTool = tools[webSearchToolIndex] as any; + if (webSearchToolEntry) { webSearchTool = { type: "web_search", - user_location: foundTool.user_location, - search_context_size: foundTool.search_context_size, - max_uses: foundTool.max_uses, + user_location: webSearchToolEntry.user_location, + search_context_size: webSearchToolEntry.search_context_size, + max_uses: webSearchToolEntry.max_uses, }; - // Remove the web_search tool from the tools array so it's not sent as a regular tool - tools.splice(webSearchToolIndex, 1); + // Filter out web_search tool without mutating the original array + functionTools = tools.filter((tool) => tool.type !== "web_search"); } }Then use
functionToolsinstead oftoolsin subsequent code.
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (3)
apps/admin/src/lib/api/v1.d.tsis excluded by!**/v1.d.tsapps/playground/src/lib/api/v1.d.tsis excluded by!**/v1.d.tsapps/ui/src/lib/api/v1.d.tsis excluded by!**/v1.d.ts
📒 Files selected for processing (22)
apps/docs/content/features/web-search.mdxapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/lib/costs.tsapps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/all-models.tsxpackages/db/migrations/1767365168_foamy_the_initiative.sqlpackages/db/migrations/meta/1767365168_snapshot.jsonpackages/db/migrations/meta/_journal.jsonpackages/db/src/schema.tspackages/db/src/types.tspackages/models/src/get-provider-headers.tspackages/models/src/models.tspackages/models/src/models/anthropic.tspackages/models/src/models/google.tspackages/models/src/models/openai.tspackages/models/src/models/zai.tspackages/models/src/prepare-request-body.tspackages/models/src/types.ts
🚧 Files skipped from review as they are similar to previous changes (8)
- apps/ui/src/components/dashboard/log-card.tsx
- packages/db/migrations/1767365168_foamy_the_initiative.sql
- packages/models/src/models.ts
- apps/gateway/src/chat/tools/transform-response-to-openai.ts
- packages/db/src/schema.ts
- packages/db/migrations/meta/_journal.json
- packages/models/src/get-provider-headers.ts
- packages/models/src/models/zai.ts
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/types.tspackages/models/src/models/google.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tspackages/models/src/models/anthropic.tsapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsxpackages/models/src/models/openai.tspackages/db/src/types.tsapps/gateway/src/chat/tools/parse-provider-response.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
packages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/types.tspackages/models/src/models/google.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tspackages/models/src/models/anthropic.tsapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsxpackages/models/src/models/openai.tspackages/db/src/types.tsapps/gateway/src/chat/tools/parse-provider-response.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
packages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/types.tspackages/models/src/models/google.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tspackages/models/src/models/anthropic.tsapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsxpackages/models/src/models/openai.tspackages/db/src/types.tsapps/gateway/src/chat/tools/parse-provider-response.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
packages/models/src/prepare-request-body.tsapps/gateway/src/chat/tools/transform-streaming-to-openai.tspackages/models/src/types.tspackages/models/src/models/google.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tspackages/models/src/models/anthropic.tsapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsxpackages/models/src/models/openai.tspackages/db/src/types.tsapps/gateway/src/chat/tools/parse-provider-response.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/tools/transform-streaming-to-openai.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/parse-provider-response.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/tools/transform-streaming-to-openai.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/chat.tspackages/db/src/types.tsapps/gateway/src/chat/tools/parse-provider-response.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/tools/transform-streaming-to-openai.tsapps/gateway/src/lib/costs.tsapps/gateway/src/chat/tools/types.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/parse-provider-response.ts
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
apps/{ui,playground}/**/*.{ts,tsx}: Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Next.js App Router with React Server Components for frontend developmentUse cookies for user-settings which are not saved in the database to ensure SSR works
Files:
apps/ui/src/components/models/all-models.tsx
apps/{ui,playground,docs}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
next/linkfor links andnext/navigation's router for programmatic navigation
Files:
apps/ui/src/components/models/all-models.tsx
packages/db/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Drizzle ORM with latest object syntax for database operations
Files:
packages/db/src/types.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:14.219Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-03T12:42:14.219Z
Learning: Applies to **/*.{ts,tsx} : Never use `any` or `as any` unless absolutely necessary in TypeScript code
Applied to files:
packages/models/src/prepare-request-body.ts
🧬 Code graph analysis (4)
apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (1)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)
apps/gateway/src/chat/tools/types.ts (1)
packages/models/src/types.ts (1)
ToolCall(54-61)
apps/gateway/src/chat/chat.ts (4)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/types.ts (1)
WebSearchTool(391-413)packages/models/src/models.ts (1)
ProviderModelMapping(50-176)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
apps/gateway/src/chat/tools/parse-provider-response.ts (1)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: lint / run
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: generate / run
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (1)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (4)
- GitHub Check: autofix
🔇 Additional comments (20)
packages/db/src/types.ts (1)
18-37: Clean schema definitions for tool types.The new
functionToolandwebSearchToolschemas are well-structured and follow Zod conventions. The union type correctly enables both tool types while maintaining type safety.apps/gateway/src/chat/tools/types.ts (1)
29-49: LGTM! Well-structured annotation types for web search citations.The
UrlCitationAnnotationinterface correctly models the citation data structure with appropriate optional fields for provider variations. The type aliasAnnotation = UrlCitationAnnotationprovides good extensibility for future annotation types.apps/gateway/src/chat/tools/transform-streaming-to-openai.ts (2)
119-153: LGTM! Anthropic web search citation extraction in streaming.The handler correctly extracts citations from
web_search_tool_resultblocks and transforms them to the unifiedurl_citationannotation format. The conditional spread...(annotations.length > 0 && { annotations })properly avoids adding empty arrays.
456-474: LGTM! Google grounding metadata citation extraction.The handler correctly extracts web search citations from Google's
groundingMetadata.groundingChunksusing the proper field names (chunk.web.uri,chunk.web.title).apps/gateway/src/lib/costs.ts (1)
89-90: LGTM! Web search cost parameter integration.The
webSearchCountparameter is correctly added with a default ofnull, and the cost calculation properly handles the nullable value with appropriate guards.apps/gateway/src/chat/tools/parse-provider-response.ts (2)
103-126: LGTM! Anthropic web search extraction logic.The extraction correctly identifies
web_search_tool_resultblocks and setswebSearchCountto the number of search blocks (not citations), which accurately represents the number of search operations for billing purposes.
697-698: LGTM! Return statement with new fields.The return object correctly exposes
annotations(null when empty) andwebSearchCount(null when zero) for downstream consumption.apps/ui/src/components/models/all-models.tsx (3)
268-273: LGTM! Web search filter integration.The filter correctly checks if any provider in the model's provider details supports web search, consistent with other capability filters.
597-603: LGTM! Web search capability icon.The Globe icon with
text-sky-500color is appropriately added to the capability icons, consistent with the filter panel styling.
1203-1218: LGTM! Web Search column cell implementation.The cell correctly displays:
- Price formatted as
$/1K searcheswhenwebSearchandwebSearchPriceare both present- "Free" when
webSearchis enabled but no price is set (e.g., Google grounding)- "—" when web search is not supported
packages/models/src/prepare-request-body.ts (2)
22-29: LGTM! Type guard for function tools.The
isFunctionTooltype guard correctly narrowsOpenAIToolInputtoOpenAIFunctionToolInput, enabling type-safe filtering of tools. This follows the coding guideline to avoid unnecessaryanyusage.
258-265: LGTM! Filtering tools to function tools only.Correctly separates function tools from web_search tools, allowing web search to be handled via the dedicated
webSearchToolparameter.apps/gateway/src/chat/chat.ts (8)
46-46: LGTM!The
WebSearchTooltype import is correctly placed as a top-level import and aligns with the interface definition inpackages/models/src/types.ts.
848-865: LGTM!The capability checking logic correctly handles the distinction between:
- Models that support general tool calling
- Models that only support web search
The
isWebSearchOnlydetermination and the conditional allow logic are well-structured.
2416-2418: LGTM!The
webSearchEnabledflag is correctly propagated togetProviderHeaders, which will add the appropriate beta headers for providers like Anthropic that require them for web search functionality.
2300-2323: LGTM!The
webSearchToolparameter is correctly passed toprepareRequestBody. Theundefinedvalue when web search is not requested is the expected behavior.
3931-3933: LGTM!Consistent with the streaming path implementation. The
webSearchEnabledflag is correctly propagated for non-streaming requests.
4289-4303: LGTM!The new
annotationsandwebSearchCountfields are correctly destructured fromparseProviderResponse. These enable proper citation propagation and web search billing.
4364-4380: LGTM!The
webSearchCountis correctly passed tocalculateCostsfor per-search billing. When web search is not used, this will beundefinedor0, resulting in no additional cost.
4386-4416: LGTM!The
annotationsparameter is correctly passed totransformResponseToOpenai, enabling web search citations to be included in the OpenAI-format response returned to users.
| "annotations": [ | ||
| { | ||
| "type": "url_citation", | ||
| "url": "https://weather.com/...", | ||
| "title": "San Francisco Weather" | ||
| } | ||
| ] |
There was a problem hiding this comment.
Documentation example shows incorrect annotation structure.
The example response shows a flat annotation format, but the actual UrlCitationAnnotation type (in types.ts) uses a nested structure with url_citation object:
"annotations": [
{
"type": "url_citation",
- "url": "https://weather.com/...",
- "title": "San Francisco Weather"
+ "url_citation": {
+ "url": "https://weather.com/...",
+ "title": "San Francisco Weather"
+ }
}
]The same inconsistency appears in the Citations section (lines 210-221). Please update the examples to match the actual API response structure.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "annotations": [ | |
| { | |
| "type": "url_citation", | |
| "url": "https://weather.com/...", | |
| "title": "San Francisco Weather" | |
| } | |
| ] | |
| "annotations": [ | |
| { | |
| "type": "url_citation", | |
| "url_citation": { | |
| "url": "https://weather.com/...", | |
| "title": "San Francisco Weather" | |
| } | |
| } | |
| ] |
🤖 Prompt for AI Agents
In apps/docs/content/features/web-search.mdx around lines 78-84 and 210-221, the
example JSON shows annotations as a flat object with keys like "type":
"url_citation" and top-level "url"/"title", but the actual UrlCitationAnnotation
type uses a nested structure where the annotation has "type": "url_citation" and
a nested "url_citation" object containing the fields (e.g., url, title). Update
both examples to wrap the url/title inside a "url_citation" object under the
annotation entry to match the API response shape.
| z.union([ | ||
| z.object({ | ||
| type: z.literal("function"), | ||
| function: z.object({ | ||
| name: z.string(), | ||
| description: z.string().optional(), | ||
| parameters: z.record(z.any()).optional(), | ||
| }), | ||
| }), | ||
| }), | ||
| z.object({ | ||
| type: z.literal("web_search"), | ||
| user_location: z | ||
| .object({ | ||
| city: z.string().optional(), | ||
| region: z.string().optional(), | ||
| country: z.string().optional(), | ||
| timezone: z.string().optional(), | ||
| }) | ||
| .optional(), | ||
| search_context_size: z.enum(["low", "medium", "high"]).optional(), | ||
| max_uses: z.number().optional(), | ||
| }), | ||
| ]), |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, find and examine the WebSearchTool interface in packages/models
fd types.ts -g 'packages/models/**' | head -5Repository: theopenco/llmgateway
Length of output: 169
🏁 Script executed:
#!/bin/bash
# Check lines around 545-550 in chat.ts to see how user_location is used
cat -n apps/gateway/src/chat/chat.ts | sed -n '540,555p'Repository: theopenco/llmgateway
Length of output: 736
🏁 Script executed:
#!/bin/bash
# Search for how webSearchTool is constructed and used
rg -n -B 3 -A 8 'webSearchTool' apps/gateway/src/chat/chat.ts | head -40Repository: theopenco/llmgateway
Length of output: 1791
🏁 Script executed:
#!/bin/bash
# Find all WebSearchTool references and definitions
rg -n 'WebSearchTool' --type=ts -g '!**/node_modules/**'Repository: theopenco/llmgateway
Length of output: 546
🏁 Script executed:
#!/bin/bash
# Get the WebSearchTool interface definition
cat -n packages/models/src/types.ts | sed -n '385,410p'Repository: theopenco/llmgateway
Length of output: 880
🏁 Script executed:
#!/bin/bash
# Also check OpenAIWebSearchToolInput to understand the structure better
cat -n packages/models/src/types.ts | sed -n '120,145p'Repository: theopenco/llmgateway
Length of output: 849
🏁 Script executed:
#!/bin/bash
# Check the db types definition as mentioned in original comment
fd types.ts | grep packages/dbRepository: theopenco/llmgateway
Length of output: 88
Fix user_location schema to match WebSearchTool interface.
The Zod validation schema accepts user_location with { city?, region?, country?, timezone? }, but the WebSearchTool interface requires { type: "approximate", city?, region?, country? }.
When constructing webSearchTool at lines 545-550, user_location is passed directly without the required type: "approximate" field, and the timezone field is accepted but not part of the interface.
Update the schema to:
user_location: z
.object({
type: z.literal("approximate"),
city: z.string().optional(),
region: z.string().optional(),
country: z.string().optional(),
})
.optional(),Remove timezone from the schema and add type: "approximate" to match the WebSearchTool interface.
🤖 Prompt for AI Agents
In apps/gateway/src/chat/chat.ts around lines 243 to 265 (and note webSearchTool
construction at ~lines 545-550), the zod schema for user_location currently
accepts { city?, region?, country?, timezone? } which does not match the
WebSearchTool interface; update the schema so user_location is an optional
object with a required type: z.literal("approximate") and optional city, region,
and country fields, and remove timezone from the schema so the validated shape
matches the WebSearchTool interface used when constructing webSearchTool.
…me fallback Update model lookup logic in get-provider-endpoint and prepare-request-body to first try matching by model ID and then fall back to matching by provider modelName. This improves compatibility with models identified by provider-specific names, ensuring correct endpoint selection and request preparation. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
- Added checks to ensure that when the web search tool is requested, only model providers that support web search are considered. - Updated filtering logic for available model providers to respect web search capability. - Prevent usage of providers that do not support web search when the tool is requested. This ensures correct provider selection based on web search tool availability, improving reliability when web search is enabled. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
…arch - Updated documentation to rebrand 'Web Search' as 'Native Web Search' with revised content emphasizing GPT-5.2 as recommended model. - Simplified provider list into a link to models page filtered for native web search. - Updated multiple example usages to use model 'openai/gpt-5.2'. - Adjusted UI labels and tooltips to use 'Native Web Search' instead of 'Web Search' for consistency and clarity. - Added pricing info to tooltip in model provider card. - Clarified usage notes to direct users to native web search enabled models only. This change enhances user understanding and promotes the preferred native web search experience with GPT-5 models. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
…pport-hpf6au' into terragon/add-native-websearch-support-hpf6au
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/ui/src/components/models/model-provider-card.tsx (1)
3-16: Align Native Web Search price units with AllModels tableThe capability chip implementation and tooltip wiring look solid. However, this tooltip shows
($${provider.webSearchPrice.toFixed(3)}/search)whileapps/ui/src/components/models/all-models.tsxrenders the samewebSearchPriceas$${(provider.webSearchPrice * 1000).toFixed(2)}/1K. One of these is likely misrepresenting the unit.Recommend deciding whether
webSearchPriceis “per search” or “per 1K searches” and updating both places to use the same unit and formatting.Also applies to: 517-534
♻️ Duplicate comments (2)
apps/ui/src/components/models/all-models.tsx (1)
920-985: Table: Native Web Search column is now structurally and visually consistent (pricing unit mismatch remains)The table header now includes a “Native Web Search” column and the body renders a corresponding
<TableCell>per model row, so the earlier column-count mismatch is resolved. The per-provider cell logic:provider.webSearch && provider.webSearchPrice ? `$${(provider.webSearchPrice * 1000).toFixed(2)}/1K` : provider.webSearch ? "Free" : "—"is clear and matches other numeric columns.
However, note this pricing unit disagrees with
ModelProviderCard, which renderswebSearchPriceas$price/search. One of these two UIs is necessarily wrong about the units. Please align both to the same interpretation ofwebSearchPrice.Also applies to: 996-1218
apps/gateway/src/chat/chat.ts (1)
46-47: Web search tool extraction works but usesanyand mutates validated dataFunctionally this does the right thing: the Zod schema accepts a
type: "web_search"tool withuser_location,search_context_size, andmax_uses, you extract the first such tool into a typedWebSearchTooland keeptoolsfor function tools only.Two improvement points:
Avoid
anyandas any
You can derive the tool type from the schema instead of dropping toany, e.g.:type IncomingTool = z.infer<typeof completionsRequestSchema>["tools"] extends (infer T)[] ? T : never; type IncomingWebSearchTool = Extract<IncomingTool, { type: "web_search" }>; const webSearchToolIndex = tools.findIndex( (tool): tool is IncomingWebSearchTool => tool.type === "web_search", );Then
foundToolis already strongly typed withoutas anyand matchesWebSearchTool.Avoid mutating
toolsin-place
tools.splice(webSearchToolIndex, 1)mutates the parsed request object, which can be surprising and complicates reuse. A non-mutating approach keeps behavior clearer:if (webSearchToolIndex !== -1) { const [foundTool] = tools.slice(webSearchToolIndex, webSearchToolIndex + 1); webSearchTool = { ... }; tools = tools.filter((_, idx) => idx !== webSearchToolIndex); }This also gracefully handles any future case where multiple
web_searchtools might be present.Neither change is required for correctness, but they tighten type safety and avoid side effects on the validated data.
Check whether TypeScript can narrow the `tools` union from `completionsRequestSchema` sufficiently to avoid `any` by using `z.infer` and `Extract<..., { type: "web_search" }>` patterns.Also applies to: 242-265, 535-554
🧹 Nitpick comments (1)
apps/ui/src/components/dashboard/log-card.tsx (1)
458-465: Native Web Search cost row looks good; consider zero-cost visibilityThe conditional and formatting are consistent with other cost fields. If you ever want to surface free/zero-cost web search usage explicitly (e.g., promotional or bundled search), you might relax the
> 0check to render whenwebSearchCostis non-null.
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
apps/docs/content/features/web-search.mdxapps/gateway/src/chat/chat.tsapps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/all-models.tsxapps/ui/src/components/models/model-provider-card.tsx
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/docs/content/features/web-search.mdx
🧰 Additional context used
📓 Path-based instructions (9)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsx
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsx
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsx
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
apps/{ui,playground}/**/*.{ts,tsx}: Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Next.js App Router with React Server Components for frontend developmentUse cookies for user-settings which are not saved in the database to ensure SSR works
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/ui/src/components/models/all-models.tsx
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/gateway/src/chat/chat.tsapps/ui/src/components/models/all-models.tsx
apps/{ui,playground,docs}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
next/linkfor links andnext/navigation's router for programmatic navigation
Files:
apps/ui/src/components/dashboard/log-card.tsxapps/ui/src/components/models/model-provider-card.tsxapps/ui/src/components/models/all-models.tsx
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/chat.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/chat.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:26.162Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T12:42:26.162Z
Learning: Applies to apps/{gateway,api}/src/**/*.ts : Use Zod schemas for validation in Hono services
Applied to files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (4)
apps/ui/src/components/dashboard/log-card.tsx (1)
packages/db/src/schema.ts (1)
log(381-492)
apps/ui/src/components/models/model-provider-card.tsx (2)
packages/db/src/schema.ts (1)
provider(624-656)packages/shared/src/components/ui/tooltip.tsx (3)
Tooltip(61-61)TooltipTrigger(61-61)TooltipContent(61-61)
apps/gateway/src/chat/chat.ts (2)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
apps/ui/src/components/models/all-models.tsx (2)
apps/ui/src/lib/components/providers-icons.tsx (1)
getProviderIcon(1213-1226)apps/ui/src/components/models/model-card.tsx (1)
ModelCard(44-478)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: generate / run
- GitHub Check: lint / run
- GitHub Check: test / run
- GitHub Check: build / run
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (8)
apps/ui/src/components/models/all-models.tsx (4)
83-90: Web Search filter wiring is consistent across state, URL, and UIAdding
webSearchtoSortField(for cached pricing),filters.capabilities,clearFilters, URL sync, and the capabilities checklist is coherent. The checkbox correctly toggles both local state andwebSearchquery param viaupdateUrlWithFilters, andhasActiveFilterswill count it viaObject.values(filters.capabilities).No issues from a behavior or state-sync perspective.
Also applies to: 126-152, 607-652, 666-771
268-273: Web Search capability filter semantics make senseThe capability gate:
if ( filters.capabilities.webSearch && !model.providerDetails.some((p) => p.provider.webSearch) ) { return false; }is the right behavior: it keeps only models with at least one provider that supports native web search. This aligns with the rest of the capability filters.
546-605: Shared capability icon helper correctly incorporates Native Web SearchExtending
getCapabilityIconsto push a Globe-based “Native Web Search” capability forprovider.webSearchand reusing that in both the table (capabilities column) andModelCardkeeps the UI consistent across views.The ModelCard grid wiring (
getCapabilityIcons,shouldShowStabilityWarning,formatPrice) looks correct.Also applies to: 1303-1318
1330-1331: Header layout and summary cards changes are reasonable
- Wrapping the main content in an inner
container mx-auto py-8under an outercontainer ...keeps things centered and doesn’t introduce obvious layout problems.- The new “Compare” button linking to
/models/comparefollows existing patterns for navigation actions alongside “Table/Grid”.- Card summaries (models, providers, vision, tools, free models) just tighten padding and reuse
modelsWithProviders/filteredProviderCountcorrectly.All good from a behavior/UI-consistency standpoint.
Also applies to: 1373-1378, 1449-1507
apps/gateway/src/chat/chat.ts (4)
1177-1183: Routing and fallback correctly restrict web-search requests to web-search-capable providersThese filters:
In auto-routing’s
suitableProviders:if (webSearchTool && (provider as ProviderModelMapping).webSearch !== true) { return false; }In low-uptime fallback:
if (webSearchTool) { return (provider as ProviderModelMapping).webSearch === true; }In generic “no usedProvider yet” selection:
if (webSearchTool) { return (provider as ProviderModelMapping).webSearch === true; }ensure that when routing is involved, web-search-enabled requests only consider providers whose mappings have
webSearch === true. That’s exactly what you want for auto/credits/hybrid selection and provider failover.Looks correct and consistent with the capability model.
Also applies to: 1334-1349, 1455-1465
4391-4407: OpenAI-compatible response now surfaces annotations and keeps cost breakdowns consistentThe additions:
- Passing
webSearchCountintocalculateCostsand- Passing
annotationsintotransformResponseToOpenaimean OpenAI-style responses can expose both web-search usage metadata and citations, while
shouldIncludeCostsstill gates cost fields based on hosted/pro plan.The way total tokens are computed for the transform (
prompt + completion + reasoning) is unchanged and remains consistent with earlier logic.No issues here.
Also applies to: 4413-4443
2443-2445:webSearchEnabledheader flag is threaded correctly into provider headersPassing
webSearchEnabled: !!webSearchToolintogetProviderHeadersfor both streaming and non-streaming requests is the right hook for providers like Anthropic that toggle web-search betas via headers. The implementation correctly addsweb-search-2025-03-05to the Anthropic beta header when enabled, and unknown providers safely ignore this option, making it a backward-compatible extension.
829-870: Tool support gating: clarify behavior when requests mix web search and function toolsThe updated gating logic correctly allows regular tools and web-search-only requests. However, when a request includes both function tools and a
web_searchtool, the check will pass if the provider supports regular tools (supportsTools=true), even if that provider lacks web search capability (webSearch=false).This can occur with providers like Deepseek, Meta, XAI, and Cerebras, which declare
tools: truebut have nowebSearch: trueflag. While provider selection filtering in auto-routing paths (lines ~1130, ~1300) correctly excludes non-webSearch-capable providers whenwebSearchToolis present, explicit provider requests bypass these filters. TheprepareRequestBodyfunction only handleswebSearchToolfor OpenAI, Anthropic, Google, and ZAI; other tool-capable providers silently ignore it.To prevent confusion and silent failures, consider adding an explicit check:
if (webSearchTool && !supportsWebSearch) { throw new HTTPException(400, { message: `Model ${requestedModel} does not support native web search. Remove the web_search tool or choose a search-capable model.`, }); }This validation should be added before the existing tools capability check to ensure web search requirements are enforced consistently across all code paths.
| annotations, | ||
| webSearchCount, | ||
| } = parseProviderResponse(usedProvider, json, messages); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's find and examine the calculateCosts function in costs.ts
find . -path "*/apps/gateway/src/lib/costs.ts" -type fRepository: theopenco/llmgateway
Length of output: 95
🏁 Script executed:
# Also check the structure of the gateway directory
ls -la apps/gateway/src/lib/ 2>/dev/null || echo "Directory not found"Repository: theopenco/llmgateway
Length of output: 1047
🏁 Script executed:
# Search for calculateCosts function definition
rg "export.*calculateCosts|function calculateCosts" apps/gateway/src/lib/ -A 5Repository: theopenco/llmgateway
Length of output: 410
🏁 Script executed:
# Read the complete calculateCosts function signature and logic
cat -n apps/gateway/src/lib/costs.ts | head -50Repository: theopenco/llmgateway
Length of output: 1650
🏁 Script executed:
# Get the full calculateCosts function
rg "export function calculateCosts" apps/gateway/src/lib/costs.ts -A 30Repository: theopenco/llmgateway
Length of output: 847
🏁 Script executed:
# Get the full calculateCosts function to see how webSearchCount is used
rg "export function calculateCosts" apps/gateway/src/lib/costs.ts -A 100 | head -120Repository: theopenco/llmgateway
Length of output: 2793
🏁 Script executed:
# Search for webSearchCost calculation in costs.ts
rg "webSearchCost" apps/gateway/src/lib/costs.ts -B 2 -A 2Repository: theopenco/llmgateway
Length of output: 978
🏁 Script executed:
# Now let's examine the streaming and non-streaming code paths in chat.ts
# First, let's look at the non-streaming path around line 4328-4330
sed -n '4320,4350p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 856
🏁 Script executed:
# Look at the streaming code around the mentioned lines
sed -n '4391,4450p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 1588
🏁 Script executed:
# Check the final streaming log insert around 4543-4544
sed -n '4535,4555p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 669
🏁 Script executed:
# Let's look at the streaming path code. First, check around line 3766-3781 for streaming chunks
sed -n '3760,3795p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 848
🏁 Script executed:
# Check the other streaming chunk calculation around 3845-3894
sed -n '3840,3900p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 2060
🏁 Script executed:
# Look for calculateCosts calls in streaming context
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts -B 2 -A 8Repository: theopenco/llmgateway
Length of output: 1554
🏁 Script executed:
# Let's check all calculateCosts calls to see which ones include webSearchCount
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts -B 1 -A 15 | grep -E "(calculateCosts|webSearchCount)" | head -40Repository: theopenco/llmgateway
Length of output: 267
🏁 Script executed:
# Check if parseProviderResponse returns webSearchCount
rg "export.*parseProviderResponse|function parseProviderResponse" apps/gateway/src/ -A 5Repository: theopenco/llmgateway
Length of output: 517
🏁 Script executed:
# Look for streaming annotations/deltas to see if webSearchCount is available
rg "webSearchCount" apps/gateway/src/chat/chat.ts -B 3 -A 1Repository: theopenco/llmgateway
Length of output: 283
🏁 Script executed:
# Let's get a comprehensive view of all calculateCosts calls with context
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts -B 1 -A 12 | head -200Repository: theopenco/llmgateway
Length of output: 1960
🏁 Script executed:
# Check the streaming log inserts - look for insertLog calls in streaming context
rg "insertLog\(" apps/gateway/src/chat/chat.ts -B 2 -A 30 | grep -E "(insertLog|webSearchCost|streamed.*true)" | head -30Repository: theopenco/llmgateway
Length of output: 383
🏁 Script executed:
# Let's look at the full streaming insertLog call to confirm webSearchCost is missing
sed -n '3870,3930p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 1635
🏁 Script executed:
# Check line 4543-4544 area for the streaming log insert
sed -n '3885,3945p' apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 1503
🏁 Script executed:
# Search for all insertLog calls and check if they include webSearchCost
rg "insertLog\(" apps/gateway/src/chat/chat.ts -A 40 | grep -E "(insertLog|webSearchCost|streamed)" | head -50Repository: theopenco/llmgateway
Length of output: 522
🏁 Script executed:
# Let's check if parseProviderResponse is called in streaming paths
rg "parseProviderResponse" apps/gateway/src/chat/chat.ts -B 2 -A 2Repository: theopenco/llmgateway
Length of output: 850
🏁 Script executed:
# Look at the non-streaming insertLog to confirm webSearchCost is included
rg "await insertLog\(" apps/gateway/src/chat/chat.ts -A 45 | grep -A 40 "streamed: false" | head -50Repository: theopenco/llmgateway
Length of output: 1623
🏁 Script executed:
# Let's check what the streaming path does - look for how streaming annotations are handled
rg "annotations" apps/gateway/src/chat/chat.ts | grep -i streamRepository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Check if there's any web search handling in streaming chunks
rg "webSearch" apps/gateway/src/chat/chat.tsRepository: theopenco/llmgateway
Length of output: 956
🏁 Script executed:
# Verify the exact line numbers for streaming log insert - check around 3885-3945
sed -n '3880,3920p' apps/gateway/src/chat/chat.ts | cat -nRepository: theopenco/llmgateway
Length of output: 1450
🏁 Script executed:
# Let's trace the full flow - check if streaming paths call parseProviderResponse
rg "parseProviderResponse" apps/gateway/src/chat/chat.ts -C 5Repository: theopenco/llmgateway
Length of output: 1581
🏁 Script executed:
# Double-check all streaming insertLog calls to confirm webSearchCost is missing
rg "streamed: true" apps/gateway/src/chat/chat.ts -B 50 | grep -E "(insertLog|webSearchCost|calculateCosts)" | head -40Repository: theopenco/llmgateway
Length of output: 209
🏁 Script executed:
# Check what happens in streaming chunks - are annotations/deltas parsed?
sed -n '3700,3800p' apps/gateway/src/chat/chat.ts | head -80Repository: theopenco/llmgateway
Length of output: 2212
🏁 Script executed:
# Let's understand the streaming flow better - check if web search is processed in streaming
rg "webSearchTool" apps/gateway/src/chat/chat.ts -B 3 -A 3 | head -50Repository: theopenco/llmgateway
Length of output: 1719
🏁 Script executed:
# Check if there's any web search handling in the streaming SSE handler
rg "writeSSE|stream\(" apps/gateway/src/chat/chat.ts -B 2 -A 10 | grep -E "(webSearch|web search|parseProvider)" | head -20Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Verify - check the line where streaming log is inserted (around line 3910) for webSearchCost
sed -n '3900,3950p' apps/gateway/src/chat/chat.ts | grep -n "webSearch"Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Check the streaming response handling more carefully
sed -n '3600,3700p' apps/gateway/src/chat/chat.ts | cat -nRepository: theopenco/llmgateway
Length of output: 4289
🏁 Script executed:
# Search for annotation handling in streaming
rg "annotation" apps/gateway/src/chat/chat.ts -C 3Repository: theopenco/llmgateway
Length of output: 296
🏁 Script executed:
# Check if streaming responses from providers include web search data
rg "chunk|delta" apps/gateway/src/chat/chat.ts | grep -i "web\|search" | head -10Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Let's check the full streaming handler to see if webSearchCount could be parsed from streaming chunks
sed -n '3500,3600p' apps/gateway/src/chat/chat.ts | cat -n | head -80Repository: theopenco/llmgateway
Length of output: 3384
🏁 Script executed:
# Check if parseProviderResponse is somehow used or if annotations are parsed during streaming
rg "streaming.*annotation|chunk.*annotation" apps/gateway/src/chat/chat.ts -C 5Repository: theopenco/llmgateway
Length of output: 46
🏁 Script executed:
# Let's verify the exact parameters passed to calculateCosts in streaming vs non-streaming
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts | wc -lRepository: theopenco/llmgateway
Length of output: 65
🏁 Script executed:
# Count how many calculateCosts calls don't include webSearchCount parameter
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts -A 12 | grep -c "webSearchCount\|inputImageCount," | head -20Repository: theopenco/llmgateway
Length of output: 65
🏁 Script executed:
# Let's verify by checking all 5 calculateCosts calls and their parameters
rg "calculateCosts\(" apps/gateway/src/chat/chat.ts -A 12 | grep -B 12 "inputImageCount"Repository: theopenco/llmgateway
Length of output: 749
🏁 Script executed:
# Confirm streaming insertLog doesn't have webSearchCost while non-streaming does
rg "await insertLog\(" apps/gateway/src/chat/chat.ts -A 35 | grep -E "(insertLog|webSearchCost|streamed:)" | head -40Repository: theopenco/llmgateway
Length of output: 522
Streaming requests using web search don't include webSearchCost in billing logs
The non-streaming path correctly wires web search billing:
parseProviderResponsereturnswebSearchCountcalculateCostsreceiveswebSearchCountas the final argumentinsertLogincludeswebSearchCost: costs.webSearchCost
However, the streaming path omits web search from cost calculations:
- All streaming
calculateCostscalls lack thewebSearchCountparameter - The streaming
insertLogdoes not setwebSearchCostat all - Since
calculateCostsuseswebSearchCountto derivewebSearchCost, streaming web-search requests will not be billed for or tracked correctly
If streaming with native web search is a supported combination, the streaming pipeline needs to:
- Capture
webSearchCountfrom streaming chunks or responses - Pass it to
calculateCosts - Include
webSearchCostin the streaming log insert
If streaming + web search is intentionally unsupported, this assumption should be asserted in code or documentation.
🤖 Prompt for AI Agents
In apps/gateway/src/chat/chat.ts around lines 4328 to 4330, the streaming branch
omits webSearchCount when calculating costs and does not include webSearchCost
in the streaming insertLog; update the streaming pipeline to capture
webSearchCount from the provider streaming output (accumulate it from streaming
chunks or obtain it from the final streaming response), pass that webSearchCount
into calculateCosts wherever calculateCosts is invoked in the streaming flow,
and set webSearchCost: costs.webSearchCost in the insertLog for streaming
requests; alternatively, if streaming+web search is unsupported, explicitly
assert or guard against that combination and log/throw accordingly.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
packages/db/migrations/1767474314_zippy_norman_osborn.sqlpackages/db/migrations/meta/1767474314_snapshot.jsonpackages/db/migrations/meta/_journal.json
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/db/migrations/meta/_journal.json
| @@ -0,0 +1 @@ | |||
| ALTER TABLE "log" ADD COLUMN "web_search_cost" real; | |||
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find all SQL migration files to understand the schema structure
find packages/db/migrations -name "*.sql" -type f | head -20Repository: theopenco/llmgateway
Length of output: 1183
🏁 Script executed:
#!/bin/bash
# Search for cost-related columns in all SQL files
rg -n 'cost|Cost' packages/db/migrations --type=sql -iRepository: theopenco/llmgateway
Length of output: 884
🏁 Script executed:
#!/bin/bash
# Look for schema definition files
fd 'schema' packages --type f | grep -E '\.(ts|js|sql)$'Repository: theopenco/llmgateway
Length of output: 89
🏁 Script executed:
#!/bin/bash
# Examine the schema.ts file to understand how costs are defined
cat packages/db/src/schema.tsRepository: theopenco/llmgateway
Length of output: 23776
🏁 Script executed:
#!/bin/bash
# Get more context on the data_storage_cost migration that uses numeric
rg -B2 -A2 'data_storage_cost' packages/db/migrations --type=sqlRepository: theopenco/llmgateway
Length of output: 344
Use numeric for web_search_cost to match the pattern established by data_storage_cost.
The real type (4-byte floating-point) can introduce precision errors for financial data. The codebase already addressed this with the data_storage_cost column (migration 1764086480), which uses numeric with fixed precision. This migration should follow that same pattern for consistency and to avoid rounding discrepancies in cost calculations.
Suggested change
ALTER TABLE "log" ADD COLUMN "web_search_cost" numeric DEFAULT '0' NOT NULL;🤖 Prompt for AI Agents
In packages/db/migrations/1767474314_zippy_norman_osborn.sql around line 1, the
migration adds web_search_cost as type real which can cause precision errors and
diverges from the existing data_storage_cost pattern; change the column to use
numeric with a default and non-null constraint (e.g. ALTER TABLE "log" ADD
COLUMN "web_search_cost" numeric DEFAULT '0' NOT NULL) so it matches the
established cost column type and avoids floating-point rounding issues.
- Update billing count logic to count web_search_call items correctly - Change pricing for reasoning models to $10 per 1000 searches - Skip minimal reasoning effort auto-set if web_search tool is present - Add display of web search cost in UI dashboard log card - Clarify distinction between citations (display) and calls (billing) Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (5)
apps/gateway/src/chat/tools/parse-provider-response.ts (1)
309-338: Clarify Google web search count logic.The comment at line 313 states "Google doesn't report individual search counts," yet line 336 overwrites
webSearchCountwithwebSearchQueries.lengthwhen available. This creates ambiguity:
- If Google consistently provides
webSearchQueries, the hardcoded1is always overwritten and the comment is misleading.- If
webSearchQueriesis sometimes absent, the default should be documented as a fallback assumption.Suggest:
- Remove the misleading comment, OR
- Update the comment to clarify: "Default to 1 search; use webSearchQueries.length if available"
🔎 Proposed clarification
- webSearchCount = 1; // Google doesn't report individual search counts + // Default to 1 search; will be updated if webSearchQueries is available + webSearchCount = 1;packages/models/src/models/openai.ts (2)
44-69: Fix incorrect web search pricing for gpt-4o-search-preview.Line 60 sets
webSearchPrice: 0.03($30 per 1,000 searches), but OpenAI's official pricing is $10.00 per 1,000 web-search calls, which equals0.01per search. The current value is 3x higher and will overcharge users.Additionally, the release dates (lines 50-51) may be outdated based on the current model snapshot (gpt-4o-search-preview-2025-03-11 as of March 2025).
🔎 Proposed fix
- webSearchPrice: 0.03, // $30 per 1000 searches + webSearchPrice: 0.01, // $10 per 1000 searches (OpenAI official pricing)Also consider updating the dates if the snapshot has changed:
- releasedAt: new Date("2024-10-01"), - publishedAt: new Date("2024-10-01"), + releasedAt: new Date("2025-03-11"), + publishedAt: new Date("2025-03-11"),
167-167: Add missing webSearchPrice for gpt-4o or document if free.Line 167 enables
webSearch: truefor gpt-4o but lacks awebSearchPricefield. The cost calculation will default to 0, making web search free for this model.If OpenAI charges for web search on gpt-4o via the Responses API, add the appropriate
webSearchPrice. If web search is free for this model, document this explicitly in a comment to clarify the intentional omission.🔎 Proposed fix if priced
tools: true, webSearch: true, // Supports web_search tool via Responses API + webSearchPrice: 0.01, // $10 per 1000 searches (verify official pricing) jsonOutputSchema: true,Or if free:
webSearch: true, // Supports web_search tool via Responses API (no additional charge)apps/gateway/src/chat/chat.ts (2)
243-265: Fix user_location schema to match WebSearchTool interface.The Zod schema (lines 254-260) accepts
{ city?, region?, country?, timezone? }, but theWebSearchToolinterface requires{ type: "approximate", city?, region?, country? }(see packages/models/src/types.ts:395-402).Mismatches:
- Missing required
type: "approximate"field- Extra
timezonefield not in interfaceWhen
webSearchToolis constructed at lines 545-550,user_locationis passed directly without the requiredtypefield.🔎 Proposed fix
z.object({ type: z.literal("web_search"), user_location: z .object({ + type: z.literal("approximate"), city: z.string().optional(), region: z.string().optional(), country: z.string().optional(), - timezone: z.string().optional(), }) .optional(), search_context_size: z.enum(["low", "medium", "high"]).optional(), max_uses: z.number().optional(), }),As per coding guidelines, use Zod schemas for validation in Hono services.
535-554: Avoid mutating the tools array; use filter instead.Line 552 uses
tools.splice(webSearchToolIndex, 1)to remove the web_search tool, mutating the originaltoolsarray from the request. This side effect could cause issues if the array is used elsewhere.🔎 Proposed fix
// Extract web_search tool from tools array if present let webSearchTool: WebSearchTool | undefined; if (tools && Array.isArray(tools)) { const webSearchToolIndex = tools.findIndex( (tool: any) => tool.type === "web_search", ); if (webSearchToolIndex !== -1) { const foundTool = tools[webSearchToolIndex] as any; webSearchTool = { type: "web_search", user_location: foundTool.user_location, search_context_size: foundTool.search_context_size, max_uses: foundTool.max_uses, }; - // Remove the web_search tool from the tools array so it's not sent as a regular tool - tools.splice(webSearchToolIndex, 1); + // Filter out the web_search tool to avoid sending it as a regular tool + tools = tools.filter((_, index) => index !== webSearchToolIndex); } }
🧹 Nitpick comments (1)
packages/models/src/models/openai.ts (1)
685-687: Verify GPT-5 web search pricing when models are released.Multiple GPT-5 variants now include
webSearch: truewithwebSearchPrice: 0.01($10 per 1,000 searches). The comment specifies this pricing is "for reasoning models."Since GPT-5 models are not yet released (as of January 2026), confirm this pricing with OpenAI's official documentation when the models become available. The pricing may differ from the current gpt-4o-search-preview pricing.
Also applies to: 752-754, 818-820, 916-918, 982-984, 1012-1014, 1062-1064, 1110-1112, 1175-1177, 1225-1227
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/parse-provider-response.tsapps/ui/src/components/dashboard/log-card.tsxpackages/models/src/models/openai.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/ui/src/components/dashboard/log-card.tsx
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.tspackages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.tspackages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.tspackages/models/src/models/openai.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.tspackages/models/src/models/openai.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/tools/parse-provider-response.tsapps/gateway/src/chat/chat.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:26.162Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T12:42:26.162Z
Learning: Applies to apps/{gateway,api}/src/**/*.ts : Use Zod schemas for validation in Hono services
Applied to files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (2)
apps/gateway/src/chat/tools/parse-provider-response.ts (1)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)
apps/gateway/src/chat/chat.ts (5)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/types.ts (1)
WebSearchTool(391-413)packages/db/src/schema.ts (1)
provider(624-656)packages/models/src/models.ts (1)
ProviderModelMapping(50-176)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
🔇 Additional comments (10)
apps/gateway/src/chat/tools/parse-provider-response.ts (3)
552-583: LGTM! Clear separation of billing vs display data.The implementation correctly distinguishes between:
- Billing:
webSearchCalls.length(line 557) counts API calls for cost calculation- Display: Annotations (lines 562-582) provide citations for the user
The inline comments clearly document this distinction.
641-666: LGTM! Correct billing for search preview models.The implementation correctly bills search preview models (gpt-4o-search-preview, gpt-4o-mini-search-preview) at 1 search per request when citations are present, matching OpenAI's billing model as documented in the comment at line 643.
103-143: Implementation correctly aligns with Anthropic's billing model.Anthropic bills web search at $10 per 1,000 web searches, meaning billing is per search invocation. The code correctly counts
webSearchBlocks.length(line 109) as the number of billable search calls, where eachweb_search_tool_resultblock represents one search invocation. Inline citations fromtextBlocks(lines 128-143) are content within search results, not separate billable events, so they appropriately don't incrementwebSearchCount.apps/gateway/src/chat/chat.ts (7)
848-870: LGTM! Correct validation for web search tool support.The implementation correctly validates tool support:
- Checks if provider supports
webSearchcapability (lines 849-851)- Distinguishes web-search-only requests from mixed tool requests (lines 853-860)
- Allows requests that either have tool support OR use web search exclusively (lines 862-869)
This properly handles the case where search preview models don't support additional function tools but do support web search.
1177-1183: LGTM! Correct auto-routing filter for web search.The auto-routing logic correctly filters providers to only those with
webSearch === truewhen awebSearchToolis present (lines 1177-1183). This ensures web search requests are routed to capable providers.
1334-1348: LGTM! Fallback routing respects web search requirements.When routing away from a low-uptime provider, the code correctly filters to only providers that support web search if
webSearchToolis present (lines 1344-1346). This maintains web search capability during fallback routing.
1456-1465: LGTM! Provider selection respects web search capability.The provider selection logic correctly filters to web-search-capable providers when
webSearchToolis present (lines 1461-1463), maintaining consistency with other provider selection paths.
2351-2351: LGTM! Web search tool correctly integrated into request flow.The
webSearchToolis properly:
- Passed to
prepareRequestBody(line 2351) for request body construction- Converted to
webSearchEnabledflag for headers (lines 2445-2447, 3960-3962)- Applied to both streaming and non-streaming code paths
The integration is consistent across all execution paths.
Also applies to: 2445-2447, 3960-3962
4330-4332: LGTM! Non-streaming path correctly handles web search billing.The non-streaming path properly:
- Extracts
annotationsandwebSearchCountfromparseProviderResponse(lines 4330-4332)- Passes
webSearchCounttocalculateCosts(line 4408)- Includes
annotationsin the transformed response (line 4444)- Logs
webSearchCostin billing records (line 4545)The web search billing flow is complete for non-streaming requests.
Note: A past review comment flagged that the streaming path lacks
webSearchCostin billing logs (see existing_comments for lines 4330-4332 in past_review_comments). That issue is not addressed in this diff and remains outstanding.Also applies to: 4408-4408, 4444-4444, 4545-4545
1617-1637: Reconsider the reasoning_effort auto-set logic when web search is present.The comment states web search is "incompatible with 'minimal' reasoning effort," but this conflates two separate concerns. The actual API limitation is that parallel function tool calls are restricted with minimal reasoning_effort (noted for Azure), not web search itself. Web search is extracted as a separate capability, not invoked as a traditional function tool.
The code should differentiate between:
- Web search (a native capability) — compatible with minimal reasoning_effort
- Parallel function tool calls — restricted with minimal reasoning_effort
Consider refining the condition to skip reasoning_effort auto-setting only when the request includes function tools that require parallelization, not whenever web search is present. If web search is used alone or alongside non-parallel tools, auto-setting reasoning_effort should be safe.
| // For ZAI, extract web search info if present | ||
| // ZAI includes web_search content in the response | ||
| if (usedProvider === "zai") { | ||
| const webSearchResults = | ||
| json.choices?.[0]?.message?.web_search || null; | ||
| if (webSearchResults && Array.isArray(webSearchResults)) { | ||
| webSearchCount = webSearchResults.length; | ||
| for (const result of webSearchResults) { | ||
| annotations.push({ | ||
| type: "url_citation", | ||
| url_citation: { | ||
| url: result.link || result.url || "", | ||
| title: result.title, | ||
| }, | ||
| }); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
ZAI Z.AI web search API billing model pricing documentation
💡 Result:
Summary — Z.AI (ZAI) Web Search API billing
- Web Search tool: $0.01 per use. [1]
- Z.AI overall API pricing is pay-as-you-go; model calls are billed per-token (prices listed per 1M tokens) and some built-in tools (like Web Search) are billed per call. See the pricing table for model-specific token rates and other tool costs. [1]
- Your account/billing dashboard (Manage → Billing) shows usage, balance, invoices and lets you top up/manage payment methods. [2]
Sources
- Z.AI pricing documentation (Web Search: $0.01/use; model token rates). [1]
- Z.AI billing dashboard / account billing page. [2]
Fix web search usage counting logic.
The code counts webSearchResults.length (line 674) as the billable search count. However, ZAI bills Web Search at $0.01 per use/call, not per result returned. A single search call typically returns multiple results in the array. Counting results instead of search calls inflates the reported usage.
Determine the correct count based on how many actual search queries ZAI executed (not results returned), and adjust the webSearchCount assignment accordingly.
Adjusted webSearchPrice from $0.03 to $0.025 for non-reasoning models in openai.ts to reflect accurate pricing. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
packages/models/src/models/openai.ts (2)
70-95: Verify webSearchPrice and release date.This model has the same
webSearchPriceof0.025and potentially outdatedreleasedAtdate asgpt-4o-search-preview. Please verify against official OpenAI documentation.
167-167: Add missingwebSearchPricefor gpt-4o.The model has
webSearch: truebut lacks awebSearchPricefield. A previous review comment (marked as addressed in commit 7061c92) flagged this issue, but the field is still missing in the current code. WithoutwebSearchPrice, cost calculations will default to 0, making web search free for this model and creating an inconsistency with other web-search-enabled models.🔎 Proposed fix
webSearch: true, // Supports web_search tool via Responses API +webSearchPrice: 0.01, // $10 per 1000 searches jsonOutputSchema: true,
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
packages/models/src/models/openai.ts
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
packages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
packages/models/src/models/openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
packages/models/src/models/openai.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
packages/models/src/models/openai.ts
🔇 Additional comments (1)
packages/models/src/models/openai.ts (1)
685-687: Consistent web search implementation across GPT-5 family.The web search additions across all GPT-5 family models are internally consistent, using
webSearchPrice: 0.01($10 per 1,000 searches for reasoning models) andsupportsResponsesApi: truewhere applicable. The distinction between reasoning model pricing (0.01) and non-reasoning model pricing (0.025) in the search-preview models suggests an intentional pricing tier, but this should be verified against official OpenAI documentation to ensure accuracy.Also applies to: 752-754, 818-820, 916-920, 982-984, 1012-1014, 1062-1064, 1110-1114, 1175-1177, 1225-1227
| { | ||
| id: "gpt-4o-search-preview", | ||
| name: "GPT-4o Search Preview", | ||
| description: | ||
| "GPT-4o with native web search capabilities for up-to-date information.", | ||
| family: "openai", | ||
| releasedAt: new Date("2024-10-01"), | ||
| publishedAt: new Date("2024-10-01"), | ||
| providers: [ | ||
| { | ||
| providerId: "openai", | ||
| modelName: "gpt-4o-search-preview", | ||
| inputPrice: 2.5 / 1e6, | ||
| outputPrice: 10.0 / 1e6, | ||
| requestPrice: 0, | ||
| webSearch: true, | ||
| webSearchPrice: 0.025, // $25 per 1000 searches for non-reasoning models | ||
| contextSize: 128000, | ||
| maxOutput: 16384, | ||
| streaming: true, | ||
| vision: true, | ||
| tools: false, // Search models don't support additional tools | ||
| jsonOutput: false, | ||
| }, | ||
| ], | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenAI gpt-4o-search-preview web search pricing per 1000 searches
💡 Result:
gpt-4o-search-preview web-search tool calls: $10.00 per 1,000 calls, plus search content tokens billed at the model’s input-token rates. [1][2]
Sources:
[1] OpenAI API Pricing — Built-in tools (Web search preview (reasoning models) $10.00 / 1K calls + search content tokens billed at model rates).
[2] GPT-4o Search Preview model docs (notes pricing and search-token billing).
🌐 Web query:
OpenAI gpt-4o-search-preview release date model snapshot 2025
💡 Result:
OpenAI released the GPT-4o Search Preview models in March 2025. The API exposes snapshot aliases including gpt-4o-search-preview-2025-03-11 (and the gpt-4o-mini-search-preview equivalent). Sources: OpenAI model docs and changelog. [1][2]
Sources
- OpenAI Models docs — "GPT-4o Search Preview" (shows snapshots/gpt-4o-search-preview-2025-03-11).
- OpenAI API changelog — March 11, 2025 entry announcing gpt-4o-search-preview and gpt-4o-mini-search-preview.
🏁 Script executed:
sed -n '44,69p' packages/models/src/models/openai.tsRepository: theopenco/llmgateway
Length of output: 770
Correct webSearchPrice and update release date to match official OpenAI specifications.
The webSearchPrice of 0.025 is incorrect per official OpenAI pricing documentation, which specifies $10.00 per 1,000 search calls (0.01). Additionally, releasedAt and publishedAt should be updated to March 11, 2025 (new Date("2025-03-11")) to match the official model snapshot gpt-4o-search-preview-2025-03-11 released in March 2025.
🤖 Prompt for AI Agents
In packages/models/src/models/openai.ts around lines 44 to 69, update the model
metadata to match official OpenAI specs: change webSearchPrice from 0.025 to
0.01 (representing $10.00 per 1,000 searches) and set releasedAt and publishedAt
to new Date("2025-03-11"); keep all other provider fields the same.
Add a check to ensure that if the web_search tool is requested, the selected model supports it. Throws an HTTP 400 error with an explanatory message if the model does not support native web search, directing users to supported models documentation. Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
apps/gateway/src/chat/chat.ts (3)
243-265: [Duplicate] Fix user_location schema to match WebSearchTool interface.This issue was already flagged in a previous review. The Zod schema for
user_locationaccepts{ city?, region?, country?, timezone? }but theWebSearchToolinterface requires{ type: "approximate", city?, region?, country? }. The schema is missing the requiredtypefield and includes an unsupportedtimezonefield.
535-554: [Duplicate] Avoid mutating the original tools array.This issue was already flagged in a previous review. The
tools.splice(webSearchToolIndex, 1)operation mutates the original request's tools array, which can cause issues if the array is used elsewhere. Use a non-mutating operation likefilterinstead.
3775-3790: [Duplicate] Streaming requests missing web search billing.This critical issue was already flagged in a previous review. The streaming path does not include web search billing:
- The
calculateCostscall at line 3775 omits thewebSearchCountparameter (compare to line 4415 in non-streaming)- The
insertLogcall starting at line 3854 does not include thewebSearchCostfield (compare to line 4552 in non-streaming)This means streaming requests using web search will not be tracked or billed correctly. Since
calculateCostsderiveswebSearchCostfromwebSearchCount, and streaming doesn't pass this parameter, web search costs will be zero for streaming requests.Also applies to: 3854-3903
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/gateway/src/chat/chat.ts
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/gateway/src/chat/chat.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/gateway/src/chat/chat.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/chat.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/gateway/src/chat/chat.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/chat.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:26.162Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T12:42:26.162Z
Learning: Applies to apps/{gateway,api}/src/**/*.ts : Use Zod schemas for validation in Hono services
Applied to files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (1)
apps/gateway/src/chat/chat.ts (5)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/types.ts (1)
WebSearchTool(391-413)packages/db/src/schema.ts (1)
provider(624-656)packages/models/src/models.ts (1)
ProviderModelMapping(50-176)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
🔇 Additional comments (4)
apps/gateway/src/chat/chat.ts (4)
848-877: LGTM! Clear validation logic for web search support.The validation logic correctly handles the distinction between function tools and web search tools:
- Determines if the request is web-search-only (has webSearchTool but no function tools)
- Allows the request if the model supports regular tools OR (supports web search AND is web-search-only)
- Provides a clear error message with documentation link when web search is not supported
The error message at lines 873-876 is particularly helpful, directing users to the models page with the webSearch filter.
2358-2358: LGTM! Consistent web search integration in request flow.The web search tool is correctly integrated into both streaming and non-streaming request flows:
webSearchToolis passed toprepareRequestBodyto include in provider-specific payloads (line 2358)webSearchEnabledheader flag is set consistently using!!webSearchTool(lines 2452-2454 for streaming, 3967-3969 for non-streaming)- The boolean coercion pattern matches other similar parameters in the codebase
Also applies to: 2452-2454, 3967-3969
4337-4339: LGTM! Complete web search billing flow in non-streaming path.The non-streaming path correctly implements the full web search billing flow:
- Extracts
annotationsandwebSearchCountfrom provider response (lines 4337-4339)- Passes
webSearchCounttocalculateCostsas the final argument (line 4415)- Includes
annotationsin the transformed OpenAI response (line 4451)- Logs
webSearchCostin the database (line 4552)This ensures web search usage is properly tracked and billed for non-streaming requests.
Also applies to: 4415-4415, 4451-4451, 4552-4552
1624-1644: The comment about web search and reasoning effort incompatibility is misleading. The code sends bothreasoning_effortandwebSearchTooltogether to the OpenAI Responses API without any validation preventing this combination. Model definitions explicitly list reasoning models with web search support (e.g.,webSearchPrice: 0.01, // $10 per 1000 searches for reasoning models). The skip at line 1630 is an auto-routing design choice, not evidence of API incompatibility—explicitreasoning_effortwith web search is not rejected anywhere in the codebase and is actively supported by the provider implementations.Likely an incorrect or invalid review comment.
…ebsearch-support-hpf6au
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/chat/chat.ts (1)
4388-4402: Non‑streaming path correctly wires annotations and web search costs, but surface is incompleteThe non‑streaming flow is mostly well‑integrated:
parseProviderResponsenow returnsannotationsandwebSearchCount.annotationsare passed intotransformResponseToOpenai, which attaches them to the assistant message.webSearchCountis passed as the last argument tocalculateCosts, sowebSearchCostis computed.- The final log insert includes
webSearchCost: costs.webSearchCost, ensuring per‑search billing is persisted.Two gaps remain:
usage.web_search_costnot exposed
Even thoughwebSearchCostis calculated, theCostDatapassed totransformResponseToOpenaionly includes input/output/cached/request/total cost fields. The OpenAPI schema for/v1/chat/completionsusage also lacksweb_search_cost. If the public API is meant to expose per‑search cost (as docs show), you should:
- extend the usage schema to add
web_search_cost: z.number().nullable().optional(), and- extend the
CostData/buildUsageObjectpath (intransform-response-to-openai.ts) to include aweb_search_costfield derived fromcosts.webSearchCost.Response schema missing
annotationsfield
Thecompletionsroute’s response schema definesmessagewithout anannotationsproperty, but the transformed responses now includemessage.annotationswhen available. To keep OpenAPI accurate, add an optionalannotationsfield (even asz.array(z.record(z.unknown())).optional()if you don’t want to fully modelUrlCitationAnnotationin Zod).Also applies to: 4495-4511, 4518-4547, 4652-4652
♻️ Duplicate comments (4)
apps/gateway/src/chat/chat.ts (2)
3818-3833: Streaming + web search still doesn’t bill or logwebSearchCostThe streaming pipeline still omits web search usage from cost calculation and logs:
- All streaming
calculateCostscalls (including cached streaming and non‑cached streaming at lines ~2090 and ~3818) do not passwebSearchCount.- There is no variable tracking web search usage/count derived from streaming chunks.
- The streaming
insertLogcall at ~3902–3951 setsinputCost,outputCost, etc., but nowebSearchCost, so logs don’t record web search billing for streaming requests.As a result, any request that uses native web search in streaming mode will have:
- token and request costs accounted for, but
- web search cost always treated as zero and omitted from logs.
If streaming + web search is a supported combination, the streaming path needs to:
- Capture
webSearchCountfrom provider streaming output (e.g., via annotations or dedicated streaming metadata), and- Pass that
webSearchCountintocalculateCostseverywhere it’s called in the streaming codepath, and- Include
webSearchCost: costs.webSearchCostin the streaminginsertLogcall.If streaming + web search is intentionally unsupported, you should explicitly guard against
stream && webSearchTool(e.g., 400 error) and document that constraint to avoid silent under‑billing.Also applies to: 3902-3951, 2090-2101, 2213-2224
244-266: Fixweb_searchtool schema and extraction to matchWebSearchTooland avoidany/mutationTwo related problems here:
- Schema vs
WebSearchToolmismatch
Zod schema accepts:
{ type: "web_search", user_location?: { city?: string; region?: string; country?: string; timezone?: string; }; search_context_size?: "low" | "medium" | "high"; max_uses?: number; }But
WebSearchTool.user_location(from@llmgateway/models) is:user_location?: { type: "approximate"; city?: string; region?: string; country?: string; };The extraction code then does:
const foundTool = tools[webSearchToolIndex] as any; webSearchTool = { type: "web_search", user_location: foundTool.user_location, ... };This bypasses the required
type: "approximate"and silently forwards a shape that doesn’t satisfy theWebSearchToolcontract (and carriestimezone, which downstream code doesn’t know about).prepareRequestBodyand provider integrations will see an object that doesn’t match their declared type.You should either:
Align the request schema with
WebSearchTool(recommended for a new feature):user_location: z .object({ type: z.literal("approximate"), city: z.string().optional(), region: z.string().optional(), country: z.string().optional(), }) .optional(),and drop
timezone, orKeep accepting the current shape but normalize it when constructing
webSearchTool:webSearchTool = { type: "web_search", user_location: foundTool.user_location ? { type: "approximate", city: foundTool.user_location.city, region: foundTool.user_location.region, country: foundTool.user_location.country, } : undefined, search_context_size: foundTool.search_context_size, max_uses: foundTool.max_uses, };
anyusage and mutatingtools
findIndex((tool: any) => tool.type === "web_search")introducesany, violating the guideline to avoidanyin TS files.tools.splice(webSearchToolIndex, 1)mutates the validatedtoolsarray in place, which is brittle and can surprise future callers that reusevalidationResult.data.A safer, type‑aware pattern is:
Suggested refactor
type ToolsInput = z.infer<typeof completionsRequestSchema>["tools"]; let webSearchTool: WebSearchTool | undefined; let tools = validationResult.data.tools; if (tools && Array.isArray(tools)) { const webSearchTools = tools.filter( (tool): tool is Extract<NonNullable<ToolsInput>[number], { type: "web_search" }> => tool.type === "web_search", ); if (webSearchTools.length > 0) { const foundTool = webSearchTools[0]; webSearchTool = { type: "web_search", user_location: foundTool.user_location ? { type: "approximate", city: foundTool.user_location.city, region: foundTool.user_location.region, country: foundTool.user_location.country, } : undefined, search_context_size: foundTool.search_context_size, max_uses: foundTool.max_uses, }; // Keep `tools` as function tools only, without mutating the original array tools = tools.filter((tool) => tool.type !== "web_search"); } }This removes
any, avoids in‑place mutation, and produces aWebSearchToolthat actually matches the interface used downstream.Also applies to: 553-572
packages/db/migrations/1767485667_faithful_bedlam.sql (1)
1-1: Usenumeric(with default) forweb_search_costinstead ofrealto avoid precision issuesUsing
realfor a cost field will introduce floating‑point rounding and diverges from the existingdata_storage_costpattern, which usesnumericfor precise billing. This column should follow the same approach.Consider:
Suggested migration change
ALTER TABLE "log" ADD COLUMN "web_search_cost" numeric DEFAULT '0' NOT NULL;This keeps all cost columns consistent and avoids subtle billing discrepancies.
apps/docs/content/features/web-search.mdx (1)
53-83: Align annotations and cost-tracking examples with the actual API types and responsesThere are a few mismatches between this doc and the implemented types/behavior:
Annotations shape (lines 53–71 and 197–208)
The examples show flat annotations:{ "type": "url_citation", "url": "https://example.com/article", "title": "Article Title" }but
UrlCitationAnnotationis defined as a nested structure with aurl_citationobject (and includes indices):{ "type": "url_citation", "url_citation": { "url": "https://example.com/article", "title": "Article Title", "start_index": 0, "end_index": 50 } }Both annotation examples here should be updated to match the real
UrlCitationAnnotationshape.
web_search_costinusage(lines 221–231)
The example shows:"usage": { "prompt_tokens": 15, "completion_tokens": 150, "total_tokens": 165, "cost_usd_total": 0.0315, "cost_usd_input": 0.0015, "cost_usd_output": 0.015, "web_search_cost": 0.03 }but the
/v1/chat/completionsOpenAPI schema andbuildUsageObjectcurrently expose onlycost_usd_*fields (noweb_search_cost). Either:
- extend the gateway response to actually include
web_search_costonusage, or- adjust this example to describe web search cost as part of the total/request cost (and make clear that per‑search cost is visible via logs, not
usage).
user_locationshape in tool config examples (lines 93–103, 307–323)
Examples accept:{ "type": "web_search", "user_location": { "city": "San Francisco", "region": "California", "country": "US", "timezone": "America/Los_Angeles" } }while the
WebSearchToolinterface inpackages/models/src/types.tsexpects:user_location?: { type: "approximate"; city?: string; region?: string; country?: string; };Once the server schema is aligned with
WebSearchTool, these examples should be updated to includetype: "approximate"and droptimezone, or the type/interface should be relaxed iftimezoneis intended to be supported.Also applies to: 197-208, 221-231
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
apps/docs/content/features/web-search.mdxapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/ui/src/components/dashboard/log-card.tsxpackages/db/migrations/1767485667_faithful_bedlam.sqlpackages/db/migrations/meta/1767485667_snapshot.jsonpackages/db/migrations/meta/_journal.jsonpackages/db/src/schema.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/db/src/schema.ts
- packages/db/migrations/meta/_journal.json
🧰 Additional context used
📓 Path-based instructions (9)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.tsapps/ui/src/components/dashboard/log-card.tsxapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/gateway/src/chat/chat.tsapps/ui/src/components/dashboard/log-card.tsxapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/gateway/src/chat/chat.tsapps/ui/src/components/dashboard/log-card.tsxapps/gateway/src/chat/tools/transform-response-to-openai.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/gateway/src/chat/chat.tsapps/ui/src/components/dashboard/log-card.tsxapps/gateway/src/chat/tools/transform-response-to-openai.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
apps/{ui,playground}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
apps/{ui,playground}/**/*.{ts,tsx}: Use cookies for user-settings which are not saved in the database to ensure SSR works
Use Next.js App Router with React Server Components for frontend developmentUse cookies for user-settings which are not saved in the database to ensure SSR works
Files:
apps/ui/src/components/dashboard/log-card.tsx
apps/{ui,playground,docs}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
next/linkfor links andnext/navigation's router for programmatic navigation
Files:
apps/ui/src/components/dashboard/log-card.tsx
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:26.162Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T12:42:26.162Z
Learning: Applies to apps/{gateway,api}/src/**/*.ts : Use Zod schemas for validation in Hono services
Applied to files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (3)
apps/gateway/src/chat/chat.ts (4)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/types.ts (1)
WebSearchTool(391-413)packages/db/src/schema.ts (1)
provider(631-663)packages/models/src/models.ts (1)
ProviderModelMapping(50-176)
apps/ui/src/components/dashboard/log-card.tsx (1)
packages/db/src/schema.ts (1)
log(381-499)
apps/gateway/src/chat/tools/transform-response-to-openai.ts (2)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)packages/db/src/schema.ts (1)
message(596-618)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: e2e-shards (1)
- GitHub Check: build / run
- GitHub Check: lint / run
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (3)
- GitHub Check: test / run
- GitHub Check: e2e-shards (5)
- GitHub Check: generate / run
- GitHub Check: autofix
🔇 Additional comments (3)
apps/ui/src/components/dashboard/log-card.tsx (1)
12-12: Native Web Search cost display wiring looks correctThe new
webSearchCosthandling (conditional rendering, Number(...) casting, and separate cost row + Builtin Tools section) is consistent with how other cost fields are treated and correctly avoids rendering when the value is zero/absent. No issues from a typing or UX perspective.Also applies to: 470-477, 831-851
apps/gateway/src/chat/chat.ts (2)
866-895: Web search capability gating and routing logic look soundThe updated tool gating and routing logic integrates web search cleanly:
For non‑auto/custom models, you now:
- Allow requests when either:
- the model supports regular tools, or
- the model supports
webSearchand the request is web‑search‑only (no function tools).- Explicitly reject
web_searchusage whensupportsWebSearchis false with a clear error message and link to supported models.In auto‑routing and fallback paths, you correctly filter provider mappings to those with
webSearch === truewhen awebSearchToolis present, ensuring you don’t silently route a web‑search request to a non‑search‑capable provider.Skipping automatic
reasoning_effortdefaults whenwebSearchToolis set avoids conflicts with providers that may not support minimal reasoning + web search together.Overall, the selection logic matches the intended behavior and maintains backward compatibility for non‑web‑search calls.
Also applies to: 1202-1208, 1358-1373, 1481-1490, 1641-1648
2481-2483: Provider headers and request body correctly receive web search contextThe new wiring:
getProviderHeaders(usedProvider, usedToken, { webSearchEnabled: !!webSearchTool })in both streaming and non‑streaming paths, and- Passing
webSearchToolintoprepareRequestBody,is consistent with the models layer design and ensures providers that support native web search get:
- explicit signaling via headers, and
- the structured
WebSearchToolconfiguration in the body.No issues here.
Also applies to: 4015-4017, 2365-2387
| @@ -1,4 +1,4 @@ | |||
| import type { ImageObject } from "./types.js"; | |||
| import type { Annotation, ImageObject } from "./types.js"; | |||
There was a problem hiding this comment.
Annotations wiring is good, but web_search_cost is never exposed in usage
The new annotations plumbed through transformResponseToOpenai for Google, Anthropic, Bedrock, OpenAI responses, and the default branch looks correct and is safely guarded by a non‑empty check.
Two follow‑ups to consider:
-
Expose web search cost on the usage object
TheCostDatainterface andbuildUsageObjectcurrently only emit:cost_usd_totalcost_usd_inputcost_usd_outputcost_usd_cached_inputcost_usd_request
but docs now show a
web_search_costfield inusage. SincecalculateCostscomputeswebSearchCost, you’ll need to:- extend
CostDatawithwebSearchCost: number | null, - include it in
buildUsageObjectas e.g."web_search_cost": costs.webSearchCost, and - pass
webSearchCost: costs.webSearchCostfrom call sites (e.g.chat.ts).
Without this, clients never see a per‑search cost on the response despite the documentation.
-
Minor style nit
Several blocks use{ annotations: annotations }. These can be simplified to{ annotations }:...(annotations && annotations.length > 0 && { annotations }),
This is purely cosmetic and optional.
Also applies to: 75-101, 160-161, 296-297, 425-426, 463-466, 601-604
🤖 Prompt for AI Agents
apps/gateway/src/chat/tools/transform-response-to-openai.ts lines 1, 75-101,
160-161, 296-297, 425-426, 463-466, 601-604: the response `usage` never exposes
`web_search_cost` even though `calculateCosts` computes `webSearchCost`; extend
the CostData interface to include webSearchCost: number | null, update
buildUsageObject to add "web_search_cost": costs.webSearchCost, and update call
sites (e.g. chat.ts and other places that construct CostData) to pass
webSearchCost: costs.webSearchCost so clients see per-search cost; optionally
simplify object shorthand for annotations by replacing `{ annotations:
annotations }` with `{ annotations }` where used.
- Add /v1/responses endpoint to mock OpenAI server - Include webSearchCost in log entries - Expose cost_usd_web_search in API usage response - Update docs with correct web search cost field name 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/chat/tools/transform-response-to-openai.ts (1)
259-276: Missingcost_usd_web_searchin usage update for existing responses.When updating existing responses (not creating new ones), the code manually spreads cost fields but omits
cost_usd_web_search. This is inconsistent withbuildUsageObjectwhich includes it.🔎 Proposed fix
if (costs !== null) { transformedResponse.usage = { ...transformedResponse.usage, cost_usd_total: costs.totalCost, cost_usd_input: costs.inputCost, cost_usd_output: costs.outputCost, cost_usd_cached_input: costs.cachedInputCost, cost_usd_request: costs.requestCost, + cost_usd_web_search: costs.webSearchCost, }; }Apply this fix to all similar blocks at lines 259-276, 381-398, 479-496, 562-579, and 612-628.
♻️ Duplicate comments (3)
apps/gateway/src/chat/chat.ts (3)
553-572: Avoidanytype casts and mutating the inputtoolsarray.
- Lines 558 and 562 use
as anycasts which violate the coding guideline to avoidanyunless absolutely necessary.- Line 570 uses
tools.splice()which mutates the originaltoolsarray from the request. This could cause issues if the array is used elsewhere.🔎 Proposed refactor using type-safe extraction
let webSearchTool: WebSearchTool | undefined; + let filteredTools = tools; if (tools && Array.isArray(tools)) { - const webSearchToolIndex = tools.findIndex( - (tool: any) => tool.type === "web_search", - ); - if (webSearchToolIndex !== -1) { - // Cast to any to access properties since the schema allows both function and web_search tools - const foundTool = tools[webSearchToolIndex] as any; - webSearchTool = { - type: "web_search", - user_location: foundTool.user_location, - search_context_size: foundTool.search_context_size, - max_uses: foundTool.max_uses, - }; - // Remove the web_search tool from the tools array so it's not sent as a regular tool - tools.splice(webSearchToolIndex, 1); + const webSearchTools = tools.filter( + (tool): tool is { type: "web_search"; user_location?: unknown; search_context_size?: string; max_uses?: number } => + typeof tool === "object" && tool !== null && "type" in tool && tool.type === "web_search" + ); + if (webSearchTools.length > 0) { + const foundTool = webSearchTools[0]; + webSearchTool = { + type: "web_search", + user_location: foundTool.user_location as WebSearchTool["user_location"], + search_context_size: foundTool.search_context_size as WebSearchTool["search_context_size"], + max_uses: foundTool.max_uses, + }; + // Create new array without web_search tools (non-mutating) + filteredTools = tools.filter( + (tool) => !(typeof tool === "object" && tool !== null && "type" in tool && tool.type === "web_search") + ); } }Then use
filteredToolsinstead oftoolsfor subsequent operations.
3823-3838: Streaming path does not capturewebSearchCountfor cost calculation.The
calculateCostscall here doesn't receivewebSearchCountas a parameter (compare with the non-streaming call at line 4519 which passeswebSearchCount). This meanswebSearchCostwill always be 0 for streaming requests, even when web search is actually used.The streaming pipeline needs to:
- Accumulate
webSearchCountfrom streaming chunks/responses- Pass it to
calculateCostsThis was flagged in past reviews but appears unresolved in the streaming path.
253-265: Schema mismatch withWebSearchToolinterface foruser_location.The Zod schema accepts
{ city?, region?, country?, timezone? }but theWebSearchToolinterface requires{ type: "approximate", city?, region?, country? }. Thetimezonefield is accepted but not part of the interface, and the requiredtype: "approximate"field is missing.
🧹 Nitpick comments (2)
apps/gateway/src/test-utils/mock-openai-server.ts (2)
44-44: Add error handling for JSON parsing.Both endpoints parse the request body without error handling. If malformed JSON is sent, it will cause an unhandled promise rejection and potentially flaky tests.
🔎 Proposed fix
For the
/v1/responsesendpoint:mockOpenAIServer.post("/v1/responses", async (c) => { - const body = await c.req.json(); + try { + const body = await c.req.json(); + } catch (error) { + c.status(400); + return c.json({ error: { message: "Invalid JSON in request body", type: "invalid_request_error" } }); + }Similarly for the
/v1/chat/completionsendpoint at line 92.Also applies to: 92-92
47-50: Consider adding proper types instead ofany.The code uses
anytype for message parameters, which violates the coding guideline. While this is test utility code, defining a minimal interface would improve type safety and code clarity.As per coding guidelines, "Never use
anyoras anyunless absolutely necessary in TypeScript code."🔎 Proposed fix
+interface MockMessage { + role: string; + content?: string; +} + +interface MockRequestBody { + input?: MockMessage[]; + model?: string; +} + mockOpenAIServer.post("/v1/responses", async (c) => { - const body = await c.req.json(); + const body = await c.req.json() as MockRequestBody; // Check if this request should trigger an error response const shouldError = body.input?.some?.( - (msg: any) => + (msg: MockMessage) => msg.role === "user" && msg.content?.includes?.("TRIGGER_ERROR"), ); // Get the user's message to include in the response const userMessage = - body.input?.find?.((msg: any) => msg.role === "user")?.content || ""; + body.input?.find?.((msg: MockMessage) => msg.role === "user")?.content || "";Also applies to: 58-59
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
apps/docs/content/features/web-search.mdxapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.tsapps/gateway/src/test-utils/mock-openai-server.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/docs/content/features/web-search.mdx
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Never useanyoras anyunless absolutely necessary in TypeScript code
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{ts,tsx,js,jsx,json,md}
📄 CodeRabbit inference engine (CLAUDE.md)
Always use tabs for indentation
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx,js,jsx}: Always use top-levelimport, never use require or dynamic imports
No unnecessary code comments
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
apps/{gateway,api}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Use Hono framework with Zod validation and OpenAPI documentation for backend APIs
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
**/*.{js,ts,tsx,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Always use top-level
import, never use require or dynamic imports
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
{apps/api,apps/gateway,packages/db}/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
{apps/api,apps/gateway,packages/db}/**/*.ts: Use Drizzle ORM with latest object syntax for database operations
For database reads: Usedb().query.<table>.findMany()ordb().query.<table>.findFirst()
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
apps/{gateway,api}/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
apps/{gateway,api}/src/**/*.ts: Use Hono for backend framework in Gateway and API services
Use Zod schemas for validation in Hono services
Files:
apps/gateway/src/test-utils/mock-openai-server.tsapps/gateway/src/chat/chat.tsapps/gateway/src/chat/tools/transform-response-to-openai.ts
🧠 Learnings (1)
📚 Learning: 2025-12-03T12:42:26.162Z
Learnt from: CR
Repo: theopenco/llmgateway PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-03T12:42:26.162Z
Learning: Applies to apps/{gateway,api}/src/**/*.ts : Use Zod schemas for validation in Hono services
Applied to files:
apps/gateway/src/chat/chat.ts
🧬 Code graph analysis (2)
apps/gateway/src/chat/chat.ts (4)
packages/db/src/types.ts (3)
webSearchTool(23-35)tools(60-60)tool(37-37)packages/models/src/types.ts (1)
WebSearchTool(391-413)packages/models/src/models.ts (1)
ProviderModelMapping(50-176)packages/models/src/get-provider-headers.ts (1)
getProviderHeaders(13-60)
apps/gateway/src/chat/tools/transform-response-to-openai.ts (2)
apps/gateway/src/chat/tools/types.ts (1)
Annotation(40-40)packages/db/src/schema.ts (1)
message(596-618)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: build / run
- GitHub Check: lint / run
- GitHub Check: generate / run
- GitHub Check: test / run
- GitHub Check: e2e-shards (3)
- GitHub Check: e2e-shards (5)
- GitHub Check: e2e-shards (4)
- GitHub Check: e2e-shards (2)
- GitHub Check: e2e-shards (1)
- GitHub Check: autofix
🔇 Additional comments (18)
apps/gateway/src/test-utils/mock-openai-server.ts (1)
62-87: LGTM!The mock Responses API response structure looks correct and appropriately mirrors the expected format. The implementation is clean and well-suited for testing purposes.
apps/gateway/src/chat/chat.ts (12)
46-47: LGTM!The
WebSearchTooltype import is correctly added to support the new web search functionality.
866-894: LGTM!The web search capability validation logic correctly:
- Checks provider support for web search
- Allows web-search-only requests when the provider supports it
- Provides a helpful error message with a link to supported models when web search is unavailable
1202-1208: LGTM!The auto-routing filter correctly excludes providers that don't support web search when a web search tool is requested.
1359-1374: LGTM!The low-uptime fallback logic correctly filters to web-search-capable providers when a web search tool is requested.
1480-1490: LGTM!The provider selection filter correctly applies web search capability requirements.
1642-1648: LGTM!Correctly skips auto-setting minimal reasoning effort when web search is present to avoid incompatible parameter combinations.
2482-2484: LGTM!The
webSearchEnabledoption is correctly passed togetProviderHeadersbased on the presence of the web search tool.
2388-2389: LGTM!The
webSearchToolis correctly passed toprepareRequestBodyto include web search configuration in provider requests.
4408-4411: LGTM!The
annotationsandwebSearchCountare correctly extracted fromparseProviderResponseto support web search citations and billing.
4504-4520: LGTM!The
webSearchCountis correctly passed tocalculateCostsin the non-streaming path, enabling accurate web search billing.
4526-4557: LGTM!The
webSearchCostis included in the costs object andannotationsare passed totransformResponseToOpenai, ensuring web search data is properly included in the response.
4658-4663: LGTM!The
webSearchCostis consistently included in all log entries, with appropriate null values for error cases.apps/gateway/src/chat/tools/transform-response-to-openai.ts (5)
1-11: LGTM!The
Annotationtype import andwebSearchCostfield inCostDatainterface are correctly added to support web search functionality.
41-48: LGTM!The
cost_usd_web_searchfield is correctly added to the usage object when costs are available, allowing clients to see per-search costs.
75-78: LGTM!The
annotationsparameter is correctly added with proper typing and a default value for backward compatibility.
192-278: No action needed - annotations correctly omitted for these providers.The inference.net, together.ai, and groq providers do not have web search capability in their model definitions, so excluding annotations from their response handlers is correct and intentional.
320-401: No action required. Alibaba models do not support web search based on the model configuration, so annotations are not needed for this provider.Likely an incorrect or invalid review comment.
Summary
Adds native web search support with billing across providers. This enables web_search capabilities (via Responses API for non-OpenAI providers and native web search for OpenAI models) and tracks per-search costs, citations, and usage.
Changes
Core flow
Billing & costs
Data model & types
Parsing & transformation
Provider integrations
Model & provider configuration
DB schema & logging
Misc
Documentation
Test plan
Notes for reviewers
Documentation / breaking changes
No breaking changes for existing flows. Documentation should be updated to describe the web_search tool, webSearchEnabled headers, and the new billing behavior for web searches.
Consider updating UI or client SDKs to expose web search capabilities and citations to end users where applicable.
Updated documentation: Added web search feature page at apps/docs/content/features/web-search.mdx with usage guides, model support notes, and billing guidance.
🌿 Generated by Terry
ℹ️ Tag @terragon-labs to ask questions and address PR feedback
Task reference
https://www.terragonlabs.com/task/dafb5a2d-c45b-46cf-aee2-c7165b4556d1
Summary by CodeRabbit
New Features
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.