feat(ai-gateway): add NVIDIA direct BYOK support - #4803
Conversation
Adds nvidia-byok as a direct BYOK provider using NVIDIA's hosted OpenAI-compatible Chat Completions endpoint. Models come only from the live NVIDIA catalog intersected with models.dev capability metadata, so no compiled model list can go stale. Model compatibility was verified with live requests against the hosted API: - Excludes models whose endpoints reject agent requests (Gemma 2/3n reject tools or auto tool choice, Sarvam-M reports tool use as unsupported, Qwen3.5 397B returns a missing-function error). - Sends reasoning_effort only for models with verified support and translates a disabled reasoning request to the documented none effort; gpt-oss and Llama endpoints return 400 for efforts they do not accept. - Overrides context length where NVIDIA serves a smaller window than the catalog advertises (Nemotron Mini 4B, Llama 3.2 90B Vision). - Fails the sync instead of publishing an empty model list. Credential testing follows the existing static test-model convention.
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Executive SummaryThe Overview
Issue Details (click to expand)WARNING
Files Reviewed (4 files)
Notes and assumptions
Fix these issues in Kilo Cloud Previous Review Summaries (12 snapshots, latest commit 6d30087)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 6d30087)Status: No Issues Found | Recommendation: Merge Executive SummaryThe incremental change is a mechanical refactor converting the Files Reviewed (2 files)
Previous review (commit 91a6571)Status: No Issues Found | Recommendation: Merge Executive SummaryThe previously flagged WARNING ( Files Reviewed (3 files)
Previous review (commit 56ae955)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
The three previously reported WARNINGs on this incremental slice have been addressed by this update: request-time Files Reviewed (7 files)
Fix these issues in Kilo Cloud Previous review (commit 29561fc)Status: 4 Issues Found | Recommendation: Address before merge Executive SummaryNVIDIA request-time reasoning validation was decoupled from the model-specific catalog, and NVIDIA's model filter, context-length overrides, and empty-catalog guard remain absent, risking silent 400s and a stale/empty model list for users. Overview
Issue Details (click to expand)WARNING
Files Reviewed (2 files)
Fix these issues in Kilo Cloud Previous review (commit 8b84db1)Status: 3 Issues Found | Recommendation: Address before merge Executive SummaryThe final refactor commit ( Overview
Issue Details (click to expand)WARNING
Files Reviewed (1 file)
Fix these issues in Kilo Cloud Previous review (commit 9464aae)Status: No Issues Found | Recommendation: Merge Executive SummaryCommit Files Reviewed (1 file)
Previous review (commit e4a2cd7)Status: 1 Issue Found | Recommendation: Address before merge Executive SummaryThe latest commit ( Overview
Issue Details (click to expand)WARNING
Files Reviewed (8 files)
Fix these issues in Kilo Cloud Previous review (commit b0efb8b)Status: No Issues Found | Recommendation: Merge Executive SummaryThe latest commits refine the NVIDIA BYOK model metadata (per-model Files Reviewed (9 files)
Previous review (commit 126a7b2)Status: No Issues Found | Recommendation: Merge Executive SummaryThe latest commit correctly extends the NVIDIA BYOK field-stripping list to include Files Reviewed (3 files)
Previous review (commit cbff1c9)Status: 1 Issue Found | Recommendation: Address before merge Executive SummaryThe latest commit reverted a fix that made Overview
Issue Details (click to expand)WARNING
Files Reviewed (9 files)
Fix these issues in Kilo Cloud Previous review (commit 44d5234)Status: No Issues Found | Recommendation: Merge Executive SummaryIncremental review of the two new field deletions ( Files Reviewed (2 files)
Previous review (commit 4ec45d4)Status: No Issues Found | Recommendation: Merge Executive SummaryReviewed the NVIDIA direct BYOK integration (provider definition, reasoning-effort translation, model catalog sync/filtering, gateway-field stripping, and BYOK UI changes) with high confidence and found no correctness, security, or reliability issues in the changed lines. Files Reviewed (13 files)
Reviewed by claude-opus-5 · Input: 54 · Output: 16.2K · Cached: 2M Review guidance: REVIEW.md from base branch |
Local end-to-end testing through the dev server showed NVIDIA rejecting requests with 'Unsupported parameter(s): safety_identifier'. The gateway injects safety_identifier, user, and prompt_cache_key for caller attribution and cache hints, and NVIDIA validates unknown fields. Removes them for nvidia-byok, matching the existing seed provider, and adds regression coverage.
|
Tested end to end against a local dev server with a real client-style request, which caught a bug the unit tests and direct API probes both missed. Bug found and fixed ( Verified locally (fake user, personal NVIDIA key, local Postgres + serverless-redis-http):
Full suite state: 49 targeted tests, typecheck, and lint all pass. |
|
- Remove the NVIDIA terms notice from the BYOK dialog; Kilo's terms already cover agreeing to provider terms. - Drop the per-model supported_parameters override. transformRequest already strips reasoning fields for models without verified efforts, so it changed nothing observable. - Drop the include_reasoning type and delete. Nothing in the client or gateway sets it; it only appears in advertised supported_parameters. - Stop deleting `user`, which NVIDIA accepts, and document why the remaining fields are removed. - Explain why the NVIDIA model list joins models.dev with NVIDIA's own list.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Direct BYOK models advertise include_reasoning in supported_parameters, so a client can send it and NVIDIA rejects it with 'Unsupported parameter(s)'. Removing both the advertisement override and the strip left that mismatch unhandled, so the strip is restored.
Kilobot correctly flagged the getModelVariants nvidia-byok branch as dead: the production call site in convertModel does not pass a provider ID. Rather than changing that shared call site, revert model-settings.ts and openrouter/types.ts entirely. For the 21 models NVIDIA currently hosts, the existing heuristics already return undefined and fall back to the provider's openai-compatible default, so no shared change is needed. include_reasoning is now removed in the NVIDIA transform with a local cast.
Carry optional per-model parameters and OpenCode settings through the direct BYOK model cache. NVIDIA uses this to expose only documented reasoning efforts and to pin every hosted model to OpenAI-compatible Chat Completions, without adding NVIDIA branches to shared model heuristics. Also restore providerOptions stripping, which NVIDIA rejects with a 400.
Keep the direct model extension narrow: supported parameters and variants only. Remove dead NVIDIA helpers, duplicate schemas, broad OpenCode overrides, and the provider-specific empty-catalog policy. Move sync-only compatibility data into the sync module and trim declarative/oversized tests while retaining coverage for filtering, context corrections, reasoning matrices, request normalization, serialization, and final model conversion.
Treat an empty cross-catalog intersection as a sync failure so Redis retains the last known-good NVIDIA model list instead of silently replacing it with an empty catalog.
Use models.dev as the single NVIDIA model catalog, matching existing direct BYOK providers. Keep only non-deprecated tool-calling text models, while accepting that catalog availability can lag NVIDIA. Remove the live catalog fetch, intersection, endpoint denylist, context overrides, and related tests.
Remove the hardcoded NVIDIA model/effort map. Parse models.dev effort options through the shared reasoning schema and expose only those variants. Toggle-only models use their provider default until their model-specific transport is represented. Keep runtime translation generic and reject values outside the shared effort enum.
2 participants
Summary
Adds NVIDIA as a direct BYOK provider (
nvidia-byok). Users save an NVIDIA Developer API key and select namespaced models such asnvidia-byok/nvidia/nemotron-3-super-120b-a12b; Cloud strips the outer prefix and calls NVIDIA's hosted OpenAI-compatible Chat Completions endpoint with bearer authentication.NVIDIA BYOK already exists client-side in
kilocode; this adds the corresponding Cloud routing and catalog support.Why direct instead of Vercel
Vercel serves NVIDIA-authored models through providers such as Baseten, DeepInfra, and Together. It does not expose NVIDIA as an inference-provider slug that can use an NVIDIA API key, so NVIDIA BYOK must route directly to
https://integrate.api.nvidia.com/v1.Model catalog
NVIDIA uses models.dev as its model catalog, consistent with existing direct BYOK providers. Sync keeps models that are:
models.dev is the catalog authority for this integration. Availability may lag NVIDIA, so a stale catalog entry can return an upstream error until models.dev is corrected. No static routable fallback is compiled into Cloud.
Reasoning variants also come from models.dev. Explicit
effortvalues are validated through Cloud's shared reasoning schema and exposed as model variants. Toggle-only metadata does not expose variants because NVIDIA toggle transports differ by model; those models use their provider default.Request routing
NVIDIA supports Chat Completions only. The provider translates Kilo's nested reasoning effort into NVIDIA's top-level
reasoning_effortand removes gateway/OpenRouter fields NVIDIA rejects, includingprovider,providerOptions,transforms,reasoning,include_reasoning,safety_identifier, andprompt_cache_key.Credential testing follows the existing static test-model convention using
nvidia/nemotron-3-nano-30b-a3b. No migration or new Cloud environment variable is required.Verification
direct-byokusage attribution, unsupported protocol rejection, and disabled-key behavior.