Skip to content

feat(ai-gateway): expose Terminal Bench model stats - #3703

Merged
lambertjosh merged 4 commits into
mainfrom
feat/terminal-bench-model-details
Jun 8, 2026
Merged

feat(ai-gateway): expose Terminal Bench model stats#3703
lambertjosh merged 4 commits into
mainfrom
feat/terminal-bench-model-details

Conversation

@lambertjosh

@lambertjosh lambertjosh commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Enrich the existing OpenRouter model catalog with optional Terminal Bench 2.0 completion and per-attempt cost metadata.
  • Publish active non-internal results with at least five attempts and a complete-attempt cost; stealth models remain eligible.
  • Cache the compact benchmark map for five minutes and fail open to stale or empty data so stats lookup failures never block model selection.

Verification

Visual Changes

N/A

Reviewer Notes

  • Prefer deploying this catalog enrichment before the client PR, although either order is safe because the new field is optional.
  • Metrics attach only to canonical OpenRouter catalog entries, with the existing safe outer kilo/ prefix fallback.

@lambertjosh
lambertjosh marked this pull request as ready for review June 5, 2026 20:49
@kilo-code-bot

kilo-code-bot Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The new fix commit correctly resolves the previously flagged concern — internal (kilo-internal/*) benchmark summaries are now excluded at both the DB query layer (notLike) and the in-memory filter (startsWith(CUSTOM_LLM_PREFIX)), while stealth models are correctly made eligible as intended.

Files Reviewed (5 files)
  • apps/web/src/lib/model-stats/terminal-bench.ts
  • apps/web/src/lib/model-stats/terminal-bench.test.ts
  • apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
  • apps/web/src/lib/organizations/organization-types.ts
  • apps/web/src/tests/openrouter-models.test.ts
Changes since last review (commit 3a2bd50)

fix(ai-gateway): exclude internal benchmark summaries

  • Removes isStealth from the DB query and eligibility filter; replaces it with a CUSTOM_LLM_PREFIX (kilo-internal/) prefix check at both the SQL level (notLike) and in-memory (startsWith).
  • Stealth models are now correctly eligible for Terminal Bench stats.
  • Tests updated to verify kilo-internal/* models are excluded and stealth models are included.
  • All previous review notes remain non-blocking; no new issues introduced.

Reviewed by claude-4.6-sonnet-20260217 · 597,038 tokens

Review guidance: REVIEW.md from base branch main

…h-cloud

# Conflicts:
#	apps/web/src/lib/ai-gateway/providers/openrouter/index.ts
Comment thread apps/web/src/lib/model-stats/terminal-bench.ts Outdated
@lambertjosh
lambertjosh merged commit e68741e into main Jun 8, 2026
16 checks passed
@lambertjosh
lambertjosh deleted the feat/terminal-bench-model-details branch June 8, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants