Skip to content

feat(dashboard): expose full auto-routing catalog with Control Center… - #15397

Merged
diegosouzapw merged 12 commits into
cheaper-inference:release/v3.8.52from
afonsoft:feat/combos-auto-catalog-actions
Oct 8, 2026
Merged

diegosouzapw merged 12 commits into
cheaper-inference:release/v3.8.52from
afonsoft:feat/combos-auto-catalog-actions

Conversation

@afonsoft

@afonsoft afonsoft commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The /dashboard/combos Auto-routing catalog previously rendered only a static list of ~17 template combos, hiding the ~50 virtual combos the runtime actually resolves (auto, variants, category:tier entries, and 11 model families like auto/gemini, auto/glm, auto/kimi, auto/qwen, auto/deepseek, auto/gpt). This PR lazy-loads the full live catalog and wires the two missing per-card actions — Control Center and Test — so every runtime combo is inspectable and testable from the UI.

What changed

Catalog (UI) — AutoComboCatalog.tsx

  • Expands on first open via GET /api/combos/auto and renders all runtime combos with candidates (47 in docker smoke) grouped with kind badges (variant / template / category / family) and live candidate counts.
  • Keeps the static 17-template list as an offline/failure fallback only — an empty live list is now authoritative (all combos genuinely have no candidates) and shows an empty-state message instead of falling back to stale templates.

Zero-candidate filter — route.ts, catalog.ts, AutoComboCatalog.tsx

  • Combos whose materialized candidate pool is empty (no connected provider + model can ever dispatch) are skipped in both listings: GET /api/combos/auto (all 4 enumeration phases) and GET /v1/models (auto/* synthesis). E.g. auto/minimax/auto/zai disappear from every listing while auto/gemini stays once a gemini provider is connected.
  • Explicit ?id= materialization is unchanged — an already-known id still resolves for the Control Center.

Per-card actions — AutoComboCatalog.tsx, page.tsx

  • monitoring → Control Center at /dashboard/combos/<encodeURIComponent(auto/…)>
  • play_arrow → existing test modal via handleTestCombo, with per-item spinner
  • content_copy → /api/combos/duplicate (strategy auto for non-templates); hidden on bare auto

Materialization — new src/lib/combos/autoVirtual.ts

  • isAutoComboId (auto | auto/*) + materializeAutoCombo, reusing the runtime createVirtualAutoCombo / createBuiltinAutoCombo so UI/API see the same shape the router resolves.

APIs

  • GET /api/combos/auto?id=auto/… materializes one virtual combo in Control Center shape; ComboControlCenterClient routes auto* ids there and skips the UUID-only /api/usage/combo-health call.
  • POST /api/combos/test falls back to virtual materialization for auto/*, probes only the 8 highest-weighted candidates (AUTO_COMBO_TEST_MAX_PROBES), and returns comboType: "auto" + totalCandidates; the modal shows "Tested N of M live candidates".
  • route.ts materializes through { ...virtual } so the result satisfies ComboRecord (Record<string, unknown>) — interfaces don't get implicit index signatures (api-typecheck gate).

i18n — 9 new combos.* keys in en.json, pt-BR.json, and vi.json (the vi key-parity unit test requires strict en↔vi parity; caught by npm run check and backfilled).

Cleanup — fixed 6 suppressed @typescript-eslint/no-unused-vars violations in page.tsx and pruned config/quality/eslint-suppressions.json.

Related Issues

Screenshots (docker smoke, omniroute:base)

Catalog top — 47 runtime combos with the new actions on every card:

auto-catalog-top

FAMILY + CATEGORY tiers — auto/gemini, auto/glm, auto/kimi, auto/qwen, auto/deepseek, auto/gpt, auto/coding:pro, auto/reasoning, …, previously invisible:

auto-catalog-families

Validation

  • npm run typecheck:core — clean
  • tests/unit/combo-test-auto-route.test.ts — 6/6
  • npx vitest run tests/unit/AutoComboCatalog.test.tsx — 12/12
  • tests/unit/combos-auto-hide-empty-candidates.test.ts — 4/4 (new)
  • Regressions: combo-test-route 26/26, 14889-combos-auto-*, combos-duplicate-route, models-catalog-auto-combos-4164 4/4, models-catalog-block-auto-5192 3/3, models-catalog-route 39/39, auto-combos-free-models-routes 8/8, catalog-*/api-key-allow-auto-combos/11947/4235/13168 — all pass
  • i18n:check-ui-coverage, lint-staged / docs-sync / any-budget hooks — pass
  • api-typecheck gate — within frozen baseline (fixed TS2322 via fresh-object spread)
  • npm run check (lint + full unit suite) — lint clean; 45,063 tests, 45,029 pass. The only PR-caused failures were the 3 vi.json parity tests (fixed in c0f62b4). Remaining 3 failures reproduce identically on release/v3.8.52 and touch code this PR doesn't modify: memory-tools add/clear (2), videoBridgeFu07 real-FFmpeg evidence (1)
  • npm run test:coverage — gate 60/60/60/60 cleared with margin: statements 81.46%, branches 78.90%, functions 83.04%, lines 81.46%
  • npm run test:e2e (Playwright, against the running Docker container on :20128) — 110 passed, 36 conditional skips (opt-in suites: RUN_CROSS_E2E, missing @axe-core/playwright), 0 failed
  • check:cycles — advisory run, no new cycles from this diff
  • Docker smoke: docker compose --profile base up — build passes (OMNIROUTE_BUILD_WORKERS=1 OMNIROUTE_BUILD_MEMORY_MB=8192 on a 7.8 GB box), dashboard served, catalog verified live (screenshots)

Tests Added/Updated

  • tests/unit/AutoComboCatalog.test.tsx — expanded for lazy fetch, kind badges, per-card actions, fallback
  • tests/unit/combo-test-auto-route.test.ts — materialization + probe cap + comboType:"auto" response
  • tests/unit/combos-auto-hide-empty-candidates.test.ts — empty pool: every listed combo has candidateCount>0 + zero-candidate ids absent from /api/combos/auto and /v1/models, before and after seeding providers
  • tests/unit/models-catalog-auto-combos-4164.test.ts, models-catalog-block-auto-5192.test.ts — seed providers so baselines reflect "combos with candidates"

Coverage Notes

npm run test:coverage clears the 60/60/60/60 gate with margin (see Validation). The run's exit code was non-zero only because two tail test files (radar-*, rate-limit-*) hit a resource-exhaustion "Promise resolution is still pending but the event loop has already resolved" teardown issue — both pass 90/90 when run in isolation and touch code unrelated to this PR.

Reviewer Notes

  • Bare auto intentionally hides content_copy (nothing to duplicate — it IS the virtual root).
  • AUTO_COMBO_TEST_MAX_PROBES=8 bounds the probe loop; candidates are pre-sorted by weight so the probe covers the likely winners.
  • vi.json carries translated strings rather than __MISSING__ markers because i18n-vi-completeness.test.ts asserts strict key parity with en.json.

⚠️ base-red inherited: release/v3.8.52 tip 23a11484 currently fails Lint (npm audit critical advisory on next — deps untouched by this PR), Docs Sync (Strict) (QUALITY_GATES.md translation drift — file untouched by this PR), check:cycles (14 pre-existing SCCs, none containing PR-touched files), Gate / CI, Gate / Quality and Vitest (MCP) — all reproduce on the base branch's own check runs. On the latest push (f14f3304), Fast Quality Gates fails only on cycles; Build was killed by a runner shutdown mid-build (infra cancellation), and Unit Tests (2/8) flaked on usage-pending-sweep (timing-sensitive eviction assert, passes 16/16 locally, untouched by this diff). This PR introduces no new failures; api-typecheck, Merge integrity (generated skills), file-size and complexity/cognitive ratchets are green after c0f62b4 + 72e8a33 + f14f330.

… and Test actions

The /dashboard/combos auto-routing catalog rendered only the 17 static
AUTO_COMBO_TEMPLATES, hiding runtime-supported combos such as auto/gemini
and the other model families, variants, and category:tier entries. The
catalog now lazy-loads GET /api/combos/auto so it lists every supported
virtual combo with its kind and live candidate count, falling back to the
static templates when the endpoint is unavailable.

Each entry gains monitoring (Control Center) and play_arrow (Test) actions
next to content_copy. Since virtual combos have no persisted UUID, a shared
materializeAutoCombo helper backs a new GET /api/combos/auto?id= lookup for
the Control Center (skipping the UUID-only combo-health call) and lets
POST /api/combos/test probe the eight highest-weighted candidates instead
of 404ing.
@afonsoft
afonsoft requested a review from diegosouzapw as a code owner October 2, 2026 22:37
VirtualAutoCombo is an interface without an implicit index signature, so it
is not assignable to ComboRecord (Record<string, unknown>). Spreading the
materialized combo yields a fresh object type that satisfies the index
signature, clearing the api-typecheck regression (TS2322) flagged by CI.

Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
@afonsoft

afonsoft commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Screenshots — docker smoke (omniroute:base, docker compose --profile base up)

Auto-routing catalog — top (47 runtime combos lazy-loaded from GET /api/combos/auto; monitoring → Control Center, play_arrow → Test, content_copy → duplicate; copy hidden on bare auto):

auto-catalog-top

Auto-routing catalog — FAMILY + CATEGORY tiers (auto/glm, auto/gemini, auto/kimi, auto/qwen, auto/deepseek, auto/gpt, auto/minimax, auto/mimo, auto/zai, auto/gemma, auto/llama, auto/coding:pro, auto/reasoning, auto/vision, auto/multimodal — previously invisible in the UI):

auto-catalog-families

devin-ai-integration Bot and others added 2 commits October 3, 2026 00:43
Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
@afonsoft

afonsoft commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Hi @diegosouzapw — this PR is ready for your review/validation when you have a moment.

TL;DR: the Auto-routing catalog on /dashboard/combos now lazy-loads GET /api/combos/auto and lists all ~47 runtime virtual combos (families like auto/gemini, auto/glm, auto/kimi, auto/qwen, auto/deepseek, auto/gpt, plus variants and category:tier entries) instead of only the 17 static templates — and every card gains monitoring (Control Center) and play_arrow (Test) actions. Screenshots in the description.

  • 5 commits: the feature (a3e9c18), an api-typecheck fix (c09e631), a vi.json parity backfill (c0f62b4), and a generated-skills regen (72e8a33) needed by the merge-integrity gate.
  • The remaining red checks (Lint audit, Docs Sync, Gate, Vitest MCP) fail identically on the release/v3.8.52 base — flagged as ⚠️ base-red inherited in the description.

Happy to address any feedback. Thanks!

/auto/combos/auto and /v1/models no longer advertise auto/* ids whose
materialized candidate pool is empty (no provider + model can ever
dispatch). The dashboard catalog treats an empty live list as
authoritative and shows an empty-state message instead of static
fallback templates.

Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
@afonsoft

afonsoft commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Atualização: adicionei um filtro de candidatos em todo o pipeline dos combos automáticos (commit 66be061).

Antes: o GET /api/combos/auto e o GET /v1/models listavam todos os ~50 ids auto/* incondicionalmente — incluindo famílias/categorias sem nenhum provider conectado. Os cards apareciam com "No matching providers" e clientes recebiam ids que nunca conseguem despachar.

Agora:

  • /api/combos/auto pula combos cujo pool de candidatos materializado está vazio (sem provider + modelo elegível), nas 4 fases de enumeração (variants, templates, category:tier, famílias).
  • /v1/models aplica o mesmo filtro ao sintetizar os ids auto/* — modelo não é mais anunciado para clientes quando não há candidato.
  • A tela /dashboard/combos deixa de exibir cards vazios; se a lista viva vier vazia, mostra uma mensagem de estado vazio em vez de cair nos templates estáticos.
  • GET /api/combos/auto?id=auto/… continua materializando um id já conhecido (necessário pro Control Center).

Ex.: com providers openai/anthropic/gemini/… conectados, auto/gemini continua listado, mas auto/minimax e auto/zai somem de todas as listagens até existir um modelo da família conectado.

Cobertura: novo tests/unit/combos-auto-hide-empty-candidates.test.ts (4 testes — pool vazio e pool semeado, ambas as APIs), e baselines #4164/#5192 atualizados para refletir o novo contrato. Typecheck clean, lint 0 erros, 100+ testes afetados verificados.

devin-ai-integration Bot and others added 3 commits October 3, 2026 01:47
Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
…ve candidates

The zero-candidate filter hides auto ids whose materialized pool is empty.
Update the suffix-variants baseline to the new contract: seed providers and
assert each AUTO_SUFFIX_VARIANTS id is present iff its pool is non-empty.

Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
…lit POST/card)

Co-Authored-By: Afonso Dutra Nogueira Filho <afonsoft@gmail.com>
@afonsoft

afonsoft commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Análise das falhas de CI

Não há erro de compilação. O job Build foi cancelado ~5 min após iniciar o next build (Turbopack) — runner has received a shutdown signal / operation was canceled. Não é erro de código; precisa de re-run.

As 6 falhas, separadas:

Check Causa Causado pelo PR?
Build Cancelado no meio do build Não — infra/re-run
Lint npm audit --audit-level=critical → next@16.3.5 com RCE crítico (GHSA-vcvr-r3jv-pc5j, fix em 16.3.8) + 41 high Não — o PR não toca package.json/lock
Fast Quality Gates check:cycles → 14 SCCs (maior: 43 arquivos em src/lib/db/*) Não — PRs mergeados na base (#15253, #15254, #15140) falham no mesmo gate
Docs Sync (Strict) i18n drift: docs/architecture/QUALITY_GATES.md mudou e a tradução ficou stale (npm run i18n:run) Não — o PR não toca esse arquivo
Gate/CI, Gate/Quality Agregadores refletindo as falhas acima downstream

Detalhe: o check:cycles roda sem --ratchet, então falha com qualquer ciclo — e os ciclos estão em open-sse/ e src/lib/db/, arquivos que o PR nem toca. O gate já está vermelho na release/v3.8.52 (o Release-Green (continuous) também falha na base).

Conclusão: nenhuma falha vem do diff do PR. Para destravar: re-rodar o Build cancelado; o resto é dívida da base (bump next ≥16.3.8 para o audit, re-baseline ou fix dos ciclos, npm run i18n:run para o drift de docs).

Resolve src/app/api/combos/auto/route.ts: keep the pushVariant(id, build)
helper structure from release (skipped-variant logging) and translate the
branch's empty-pool `continue` skips into `return null` inside the build
callbacks — `continue` is illegal inside a non-loop function. Empty pools
stay unlisted and unmarked in seenIds/skipped, matching the branch's
hide-empty semantics (combos-auto-hide-empty-candidates) and release's
skipped-variant logging (combos-auto-skipped-variants).
…acks (cheaper-inference#15675)

auto/claude-opus|sonnet|haiku lived in AUTO_TEMPLATE_VARIANTS mapped to the
smart/coding/fast weight packs, so the whole connected provider pool was
scored and auto/claude-opus could route to an unrelated model (observed:
cheaperinference/aion-3.0). The ids now live on the auto/<family> axis:
claude-opus/claude-sonnet/claude-haiku are registered ModelFamily values
detected via word-bounded tier tokens (\bopus/\bsonnet/\bhaiku), so the
candidate pool is restricted to matching Claude tier models before scoring.
@afonsoft

afonsoft commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Fix — auto/claude-* aliases now resolve as model families (#15675)

Commit f1f0fce fixes #15675 — a request to auto/claude-opus could route to an unrelated model (observed: cheaperinference/aion-3.0) instead of a Claude Opus.

Root cause: auto/claude-opus, auto/claude-sonnet, and auto/claude-haiku lived in AUTO_TEMPLATE_VARIANTS mapped to the smart/coding/fast weight packs. The tier names were discarded and the entire connected provider pool was scored, so any model could win — the aliases implied a Claude guarantee that never existed.

Change (reuses the existing auto/<family> axis — no new routing mechanism):

  • open-sse/services/autoCombo/modelFamily.ts — claude-opus, claude-sonnet, claude-haiku added to ModelFamily/MODEL_FAMILIES, detected via word-bounded tier tokens (/\bopus/i, /\bsonnet/i, /\bhaiku/i). Covers real provider id shapes (claude-opus-4.8, anthropic/claude-3-opus-20240229, us.anthropic.claude-opus-4-1, claude-3-5-sonnet-*) while \b keeps lookalikes like octopus-v2 out.
  • open-sse/services/autoCombo/builtinCatalog.ts — the three ids removed from AUTO_TEMPLATE_VARIANTS; AUTO_FAMILY_IDS (derived from MODEL_FAMILIES) advertises them automatically, so they resolve through the family path and the candidate pool is filtered before scoring.

Net effect: auto/claude-opus can only dispatch to *opus* models — aion-3.0 can no longer enter its pool regardless of scoring weights. Same for *sonnet*/*haiku*.

Tests (tests/unit/autoCombo/provider-family-combos.test.ts):

  • Repro of the issue: connected cheaperinference serving aion-3.0 + a Claude provider → materialized auto/claude-opus pool contains only *opus* candidates, aion excluded, Claude provider contributes candidates.
  • Family detection across provider id shapes; claude-fable-5, aion-3.0, octopus-v2 correctly match nothing.
  • The three ids asserted absent from AUTO_TEMPLATE_VARIANTS, present in AUTO_FAMILY_IDS, and resolving to { family } specs (no weight variant).

Validation: 18/18 in the updated file, 263/263 across the autoCombo vitest scope, 23/23 in adjacent catalog/route suites; typecheck:core and ESLint clean.

diegosouzapw and others added 2 commits October 8, 2026 07:08
# Conflicts:
#	open-sse/services/autoCombo/builtinCatalog.ts
#	open-sse/services/autoCombo/modelFamily.ts
#	tests/unit/autoCombo/provider-family-combos.test.ts
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@diegosouzapw
diegosouzapw merged commit 734090c into cheaper-inference:release/v3.8.52 Oct 8, 2026
8 of 9 checks passed
@afonsoft
afonsoft deleted the feat/combos-auto-catalog-actions branch October 20, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants