feat(scripts): add models.dev export script - #1486
Conversation
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as "export-models-dev.ts"
participant Models as "@llmgateway/models"
participant FS as "File System (exports/)"
participant STD as "Stdout"
CLI->>Models: import providers & models
Models-->>CLI: provider & model data
CLI->>CLI: filter/map models, compute fields (family, cost, status, modalities), escape/format values
CLI->>FS: ensure directories under exports/providers/<provider>/
CLI->>FS: write provider.toml, README.md, logo.svg
CLI->>FS: write nested models/.../*.toml files
CLI->>STD: print progress and summary
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@scripts/export-models-dev.ts`:
- Around line 436-447: The code currently calls sanitizeFilename before
splitting, so slashes are removed and nested directory logic never runs; change
to compute rawName = providerMapping?.modelName || model.id, split rawName on
"/" into parts, then sanitize each part (map sanitizeFilename over parts) and
use the sanitized parts to create nestedDir = join(modelsDir,
...sanitizedParts.slice(0, -1)), mkdirSync that directory, and
writeFileSync(join(nestedDir,
`${sanitizedParts[sanitizedParts.length-1]}.toml`), otherwise write into
modelsDir when parts.length === 1; update references to
modelFilename/parts/modelToml accordingly.
🧹 Nitpick comments (2)
scripts/export-models-dev.ts (2)
184-213: Remove unusedtomlStringifyfunction.This function is defined but never used. The script builds TOML output manually in
generateProviderTomlandgenerateModelTomlinstead.♻️ Proposed fix
-function tomlStringify(obj: Record<string, unknown>, indent = 0): string { - const lines: string[] = []; - const indentStr = " ".repeat(indent); - - for (const [key, value] of Object.entries(obj)) { - if (value === undefined || value === null) continue; - - if (typeof value === "object" && !Array.isArray(value)) { - lines.push(`${indentStr}[${key}]`); - lines.push(tomlStringify(value as Record<string, unknown>, 0)); - } else if (Array.isArray(value)) { - const formattedArray = value.map((v) => (typeof v === "string" ? `"${escapeTomlString(v)}"` : v)).join(", "); - lines.push(`${indentStr}${key} = [${formattedArray}]`); - } else if (typeof value === "string") { - lines.push(`${indentStr}${key} = "${escapeTomlString(value)}"`); - } else if (typeof value === "boolean") { - lines.push(`${indentStr}${key} = ${value}`); - } else if (typeof value === "number") { - // Format numbers with underscore separators for large values - if (Number.isInteger(value) && value >= 1000) { - lines.push(`${indentStr}${key} = ${value.toLocaleString("en-US").replace(/,/g, "_")}`); - } else { - // Format decimal numbers nicely - lines.push(`${indentStr}${key} = ${value}`); - } - } - } - - return lines.join("\n"); -}
315-316: Consider documenting the fallback date.The hardcoded
"2024-01-01"fallback whenreleasedAtis missing could be misleading in the exported data. Consider adding a comment explaining this is a placeholder, or using a more explicit sentinel value if models.dev supports it.
| const providerMapping = model.providers.find((p) => p.providerId === providerId); | ||
| const modelFilename = sanitizeFilename(providerMapping?.modelName || model.id); | ||
|
|
||
| // Handle nested model IDs (e.g., "meta-llama/Meta-Llama-3.1-8B-Instruct") | ||
| const parts = modelFilename.split("/"); | ||
| if (parts.length > 1) { | ||
| const nestedDir = join(modelsDir, ...parts.slice(0, -1)); | ||
| mkdirSync(nestedDir, { recursive: true }); | ||
| writeFileSync(join(nestedDir, `${parts[parts.length - 1]}.toml`), modelToml); | ||
| } else { | ||
| writeFileSync(join(modelsDir, `${modelFilename}.toml`), modelToml); | ||
| } |
There was a problem hiding this comment.
Bug: Nested model ID handling is broken.
The sanitizeFilename function (line 390) replaces / with - before the split on line 440. This means parts.length will always be 1, and the nested directory logic (lines 441-444) will never execute.
For model IDs like "meta-llama/Meta-Llama-3.1-8B-Instruct", the current code produces meta-llama-meta-llama-3.1-8b-instruct.toml instead of creating meta-llama/meta-llama-3.1-8b-instruct.toml.
🐛 Proposed fix: Split before sanitizing
- const modelFilename = sanitizeFilename(providerMapping?.modelName || model.id);
-
- // Handle nested model IDs (e.g., "meta-llama/Meta-Llama-3.1-8B-Instruct")
- const parts = modelFilename.split("/");
- if (parts.length > 1) {
- const nestedDir = join(modelsDir, ...parts.slice(0, -1));
+ const rawModelName = providerMapping?.modelName || model.id;
+
+ // Handle nested model IDs (e.g., "meta-llama/Meta-Llama-3.1-8B-Instruct")
+ const parts = rawModelName.split("/");
+ const sanitizedParts = parts.map(sanitizeFilename);
+
+ if (sanitizedParts.length > 1) {
+ const nestedDir = join(modelsDir, ...sanitizedParts.slice(0, -1));
mkdirSync(nestedDir, { recursive: true });
- writeFileSync(join(nestedDir, `${parts[parts.length - 1]}.toml`), modelToml);
+ writeFileSync(join(nestedDir, `${sanitizedParts.at(-1)}.toml`), modelToml);
} else {
- writeFileSync(join(modelsDir, `${modelFilename}.toml`), modelToml);
+ writeFileSync(join(modelsDir, `${sanitizedParts[0]}.toml`), modelToml);
}🤖 Prompt for AI Agents
In `@scripts/export-models-dev.ts` around lines 436 - 447, The code currently
calls sanitizeFilename before splitting, so slashes are removed and nested
directory logic never runs; change to compute rawName =
providerMapping?.modelName || model.id, split rawName on "/" into parts, then
sanitize each part (map sanitizeFilename over parts) and use the sanitized parts
to create nestedDir = join(modelsDir, ...sanitizedParts.slice(0, -1)), mkdirSync
that directory, and writeFileSync(join(nestedDir,
`${sanitizedParts[sanitizedParts.length-1]}.toml`), otherwise write into
modelsDir when parts.length === 1; update references to
modelFilename/parts/modelToml accordingly.
There was a problem hiding this comment.
Pull request overview
Adds a new TypeScript script to export the repo’s model/provider catalog into models.dev-compatible TOML files, and ignores the generated output directory.
Changes:
- Add
scripts/export-models-dev.tsto generateprovider.tomland per-model TOMLs underexports/providers/<provider>/.... - Add provider metadata mappings (npm package, env vars, docs URL, API base URL) and model metadata mappings (pricing, limits, capabilities).
- Update
.gitignoreto exclude the generatedexports/directory.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| scripts/export-models-dev.ts | Implements the models/providers → models.dev TOML export pipeline and filesystem output layout. |
| .gitignore | Ignores generated exports/ output from the export script. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const inputCost = (providerMapping.inputPrice || 0) * 1e6; | ||
| const outputCost = (providerMapping.outputPrice || 0) * 1e6; | ||
| const cacheReadCost = providerMapping.cachedInputPrice ? providerMapping.cachedInputPrice * 1e6 : undefined; |
There was a problem hiding this comment.
Cost calculation ignores providerMapping.discount (documented as a 0-1 discount multiplier in ProviderModelMapping), so exported prices will be higher than the effective prices used elsewhere in this repo (e.g., cheapest-model selection applies the discount). If the models.dev export is meant to reflect real effective pricing, apply the discount multiplier to input/output/cache prices before converting to per-1M-token values.
| const inputCost = (providerMapping.inputPrice || 0) * 1e6; | |
| const outputCost = (providerMapping.outputPrice || 0) * 1e6; | |
| const cacheReadCost = providerMapping.cachedInputPrice ? providerMapping.cachedInputPrice * 1e6 : undefined; | |
| const discount = providerMapping.discount ?? 1; | |
| const inputCost = (providerMapping.inputPrice || 0) * discount * 1e6; | |
| const outputCost = (providerMapping.outputPrice || 0) * discount * 1e6; | |
| const cacheReadCost = providerMapping.cachedInputPrice ? providerMapping.cachedInputPrice * discount * 1e6 : undefined; |
| function tomlStringify(obj: Record<string, unknown>, indent = 0): string { | ||
| const lines: string[] = []; | ||
| const indentStr = " ".repeat(indent); | ||
|
|
||
| for (const [key, value] of Object.entries(obj)) { | ||
| if (value === undefined || value === null) continue; | ||
|
|
||
| if (typeof value === "object" && !Array.isArray(value)) { | ||
| lines.push(`${indentStr}[${key}]`); | ||
| lines.push(tomlStringify(value as Record<string, unknown>, 0)); | ||
| } else if (Array.isArray(value)) { | ||
| const formattedArray = value.map((v) => (typeof v === "string" ? `"${escapeTomlString(v)}"` : v)).join(", "); | ||
| lines.push(`${indentStr}${key} = [${formattedArray}]`); | ||
| } else if (typeof value === "string") { | ||
| lines.push(`${indentStr}${key} = "${escapeTomlString(value)}"`); | ||
| } else if (typeof value === "boolean") { | ||
| lines.push(`${indentStr}${key} = ${value}`); | ||
| } else if (typeof value === "number") { | ||
| // Format numbers with underscore separators for large values | ||
| if (Number.isInteger(value) && value >= 1000) { | ||
| lines.push(`${indentStr}${key} = ${value.toLocaleString("en-US").replace(/,/g, "_")}`); | ||
| } else { | ||
| // Format decimal numbers nicely | ||
| lines.push(`${indentStr}${key} = ${value}`); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return lines.join("\n"); | ||
| } |
There was a problem hiding this comment.
tomlStringify is defined but never used in this script, which adds dead code and increases maintenance surface. Please remove it or refactor the manual TOML building to use it consistently (ensuring the generated format matches models.dev requirements).
| // Replace invalid filename characters | ||
| return name.replace(/[<>:"/\\|?*]/g, "-").replace(/\s+/g, "-").toLowerCase(); |
There was a problem hiding this comment.
sanitizeFilename replaces "/" with "-", but later code attempts to split on "/" to create nested directories for names like "meta-llama/Meta-Llama-...". As written, parts.length will always be 1, so nested paths are never created and distinct names can collapse into the same filename (risking overwrites). Consider preserving "/" as a path separator (sanitize each segment) or remove the nested-directory logic.
| // Replace invalid filename characters | |
| return name.replace(/[<>:"/\\|?*]/g, "-").replace(/\s+/g, "-").toLowerCase(); | |
| // Replace invalid filename characters in each path segment, preserving "/" as a separator | |
| return name | |
| .split("/") | |
| .map((segment) => segment.replace(/[<>:"\\|?*]/g, "-").replace(/\s+/g, "-").toLowerCase()) | |
| .join("/"); |
| // Cost section | ||
| lines.push("[cost]"); | ||
| lines.push(`input = ${modelData.cost.input.toFixed(2)}`); | ||
| lines.push(`output = ${modelData.cost.output.toFixed(2)}`); | ||
| if (modelData.cost.cache_read !== undefined) { | ||
| lines.push(`cache_read = ${modelData.cost.cache_read.toFixed(2)}`); | ||
| } |
There was a problem hiding this comment.
Cost values are forced to 2 decimal places (toFixed(2)), which loses precision for very low per-1M-token prices (e.g., cached input price of 0.004 becomes 0.00). This will produce inaccurate pricing in the exported TOML. Consider emitting more decimal precision (or a trim-to-significant-digits strategy) so small but non-zero prices round-trip correctly.
e770e04 to
b389711
Compare
Add script to export all models and providers to TOML format compatible with https://github.com/anomalyco/models.dev Usage: npx tsx scripts/export-models-dev.ts Features: - Exports 222 models across 24 providers - Generates provider.toml with npm package, env vars, docs URL - Generates model TOMLs with pricing, limits, capabilities - Auto-detects open weights models (Llama, Qwen, etc.) - Maps provider IDs and family names to models.dev conventions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
b389711 to
d877f9e
Compare
Removes provider subdirectories, exports all models directly to models/ folder for simpler structure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removes empty generate.ts file and scripts/ directory. README now links to llmgateway repo for regeneration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
models.dev schema requires limit.output field. Defaults to 16384 when not specified. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2276d18 to
d48b4ab
Compare
Maps internal family names to valid models.dev families: - moonshot → kimi - bytedance → seed - zai → glm - nvidia → nemotron Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Gemma models now use "gemma" family instead of "gemini" - GPT OSS models now use "gpt-oss" family instead of "gpt" Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
5a50abe to
b419903
Compare
## Summary Add a script to export all models and providers from `@llmgateway/models` to TOML format compatible with [models.dev](https://github.com/anomalyco/models.dev). ## Usage ```bash npx tsx scripts/export-models-dev.ts ``` ## Output ``` exports/providers/ ├── openai/ │ ├── provider.toml │ └── models/ │ ├── gpt-4o.toml │ └── ... ├── anthropic/ │ ├── provider.toml │ └── models/ │ └── ... └── ... (24 providers) ``` ## Features - Exports 222 models across 24 providers - Generates `provider.toml` with: - npm package (`@ai-sdk/openai`, etc.) - Environment variables - Documentation URL - API endpoint (for OpenAI-compatible providers) - Generates model TOMLs with: - Pricing (input/output/cache per million tokens) - Context and output limits - Capabilities (vision, tools, reasoning, structured output) - Modalities (text, image) - Status (alpha/beta/deprecated) - Auto-detects open weights models (Llama, Qwen, Gemma, etc.) - Maps provider IDs and family names to models.dev conventions ## Test plan - [x] Script runs successfully - [x] Output matches models.dev schema - [x] PR submitted to models.dev: anomalyco/models.dev#698 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a capability to export provider and model metadata into a standardized TOML format with per-model files and a summary of outputs. * **Chores** * Updated version control ignore rules to exclude generated export outputs. * **Refactor** * Cleaned up component imports to remove a duplicate import and streamline module organization. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Summary
Add a script to export all models and providers from
@llmgateway/modelsto TOML format compatible with models.dev.Usage
Output
Features
provider.tomlwith:@ai-sdk/openai, etc.)Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Chores
Refactor
✏️ Tip: You can customize this high-level summary in your review settings.