Skip to content

fix(autoCombo): inherit task fitness from base model for effort/alias variants (#11489) - #11492

Merged
diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
MumuTW:fix/taskfitness-variant-fitness-base
Aug 25, 2026
Merged

diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
MumuTW:fix/taskfitness-variant-fitness-base

Conversation

@MumuTW

@MumuTW MumuTW commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #11489

⚠️ base-red inherited: #11449 — every red check on this PR reproduces on the pristine base tip 6435f618f4 with this branch's changes absent (each verified in a detached worktree or by git show base:):

  • Docs Gates: README.md / AGENTS.md / llm.txt say "159 migrations", code has 160
  • No new ESLint warnings: run-eslint-json.mjs exits 2 on stale entries in config/quality/eslint-suppressions.json (none in files this PR touches)
  • Vitest (fast-path): glmCodingProviderConfig.test.ts — 2 failures (glm-5.3-max tiers, 15-vs-14 inventory)
  • Unit Tests fast-path 1–4: failures across A2A agent card, provider-nodes, APIKEY_PROVIDERS count, openapi coverage floor, DB handles — none in taskFitness / scoresAs; the only unit test this PR adds passes
  • Fast Quality Gates → step "Quality gates (all)": check:lockfile (libxmljs2/…/brace-expansion resolved from registry.npmmirror.com in the base lockfile), check:mutation-test-coverage (2 base test files missing from stryker.conf.json), check:open-sse-typecheck (src/app/api/v1/models/catalog.ts TS2367, baseline 0 → live 2). All other 20 gates pass locally; file-size is base-relative and every touched file is under the 1000-line cap
  • Build (advisory): resolveLiveWsUrl / sanitizeLiveWsPort missing exports — untouched by this PR

Problem

Auto-combo task fitness scores every catalog id by exact string match, but the gateway already resolves <model>-<effort> ids to a base model at dispatch. The scoring layer never sees that resolution, so a variant like gpt-5.6-sol-xhigh misses every DB layer and lands on the wildcard 0.5 — while its own base model is scored properly.

Mechanism

getTaskFitnessWithSource() layers 1–2 call getModelIntelligenceBySource(), which is WHERE model = ?. Combo candidate scoring passes the raw registry id (buildAutoCandidates() → parseModel(target.modelStr).model); no effort splitter is imported anywhere under open-sse/services/combo*. Every DB-backed source (arena, models.dev, user overrides) publishes scores for base models only, so no data source can close this gap.

Fix

New open-sse/services/autoCombo/scoresAs.ts exports one resolver:

resolveScoresAs(modelId): { base: string; via: "explicit" | "effort-suffix" | "free-suffix" | null }

Three tiers, in order, and it never guesses:

  1. explicit — the registry entry declares the new RegistryModel.scoresAs. One hop only; the target must itself be a catalog id.
  2. effort-suffix — a trailing reasoning-effort token stripped by an existing dispatch splitter (splitClaudeEffortSuffix, splitCodexReasoningSuffix). No new regex is introduced.
  3. free-suffix — a trailing -free tier marker.

Tiers 2–3 are catalog-anchored: a stripped base is accepted only when findRegistryModelById(base) (newly exported from providerModels.ts — getGlobalModel's exact-id steps 1–2, deliberately not its startsWith step 3) finds it. On today's catalog, 57 of the 201 strippable effort ids strip to a base that does not exist — qwen3.7-max → qwen3.7 (-max is the model name here), grok-4.6-fast-high → grok-4.6-fast, extra-high → extra. The anchor rejects all of them.

getTaskFitnessWithSource() retries user_override then arena_elo against the resolved base on a literal miss and reports <source>:inherited. The score is inherited verbatim — the 12-factor scoring already prices cost and latency per variant, so there is no basis for inventing an effort delta.

The former -free arena-elo special case (lookupFreeAliasArenaElo, source arena_elo_free_alias, #4517) is deleted and folded into tier 3, which also extends it to user_override (it was arena-elo-only before).

Cache keys are the queried model id, so an inherited hit for a variant cannot poison the base's entry or vice versa; invalidateFitnessCache() still clears everything.

FITNESS_TABLE contents are untouched (that is #8777).

Measured effect

Resolving every catalog id + alias (1327 ids) through getTaskFitnessWithSource(id, "coding"), DATA_DIR isolated:

Structural tier breakdown of resolveScoresAs:

tier ids
explicit 16
effort-suffix 144
free-suffix 7
unresolved 1160

Coverage, with arena_elo seeded on each of the 47 distinct base ids the variants point at (the mechanism's ceiling given full base coverage):

source before after
arena_elo 47 47
arena_elo_free_alias 7 — (folded in)
arena_elo:inherited — 167
fitness_table 511 456
wildcard_boost 762 657

105 ids leave the wildcard entirely; a further 55 move off the static fitness_table onto the real benchmarked signal of the model they actually are. This number rises with every base row any data source adds, which is the point.

scoresAs entries populated

Only relations tiers 2–3 cannot express — 15 entries:

id scoresAs justification
gpt-5.6 (openai) gpt-5.6-sol Per OpenAI's model reference gpt-5.6 is an alias of gpt-5.6-sol, not a model. Points forward; no suffix stripper can produce this. Siblings -terra/-luna deliberately left alone.
claude-4.6-opus-high (cursor/agy) claude-opus-4-6 cursor/agy spell Claude ids <version>-<family>; strips to the ghost claude-4.6-opus.
claude-4.6-opus-max claude-opus-4-6 same
claude-4.6-opus-high-thinking claude-opus-4-6 same
claude-4.6-opus-max-thinking claude-opus-4-6 same
claude-4.6-opus-high-thinking-fast claude-opus-4-6 same
claude-4.6-opus-max-thinking-fast claude-opus-4-6 same
claude-4.5-opus-high claude-opus-4-5 same
claude-4.5-opus-high-thinking claude-opus-4-5 same
claude-4.6-sonnet-medium claude-sonnet-4-6 same
claude-4.6-sonnet-medium-thinking claude-sonnet-4-6 same
claude-4.5-sonnet claude-sonnet-4-5 same (no effort suffix to strip at all)
claude-4.5-sonnet-thinking claude-sonnet-4-5 same
claude-4-sonnet claude-sonnet-4 same
claude-4-sonnet-thinking claude-sonnet-4 same

Everything else stays unresolved by design.

Tests

TDD per Hard Rule #18 — both files were written first and observed failing (Cannot find module '../scoresAs' / 2 of 4 subtests failing) before the implementation landed.

  • open-sse/services/autoCombo/__tests__/scoresAs-11489.test.ts (vitest, 10 tests, no DB) — pure resolver. Covers gpt-5.6-sol-xhigh → effort-suffix, gpt-5.6 → explicit, gpt-5.6-luna unresolved (and specifically not gpt-5.6/gpt-5.6-sol), qwen3.7-max unresolved, claude-sonnet-5 never collapsing to its family, the cursor spellings, mimo-v2.5-free → free-suffix, ox-alpha-free unresolved. grok-4.6-fast-high is asserted against findRegistryModelById rather than a hardcoded expectation, so the test stays true as the catalog moves.
  • tests/unit/task-fitness-scores-as-11489.test.ts (node:test, 4 tests, DB-backed) — seeds a user_override on gpt-5.6-sol, asserts gpt-5.6-sol-xhigh and gpt-5.6 return it as user_override:inherited and that gpt-5.6-luna does not. Cleans up and calls resetDbInstance() in after().
  • open-sse/services/autoCombo/__tests__/autoCombo.test.ts — the feat: qualify ambiguous model names with provider prefix in /v1/models #4517 -free case updated for the new source string (arena_elo:inherited) and moved off synthetic ids (alias-base-test-4517) onto a real catalog pair (mimo-v2.5 / mimo-v2.5-free), since the base lookup is now catalog-anchored.

Verification

$ npx vitest run --config vitest.mcp.config.ts open-sse/services/autoCombo/__tests__/
 Test Files  7 passed (7)
      Tests  112 passed (112)

$ node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts \
    --import ./tests/_setup/isolateDataDir.ts --test \
    tests/unit/task-fitness-scores-as-11489.test.ts tests/unit/task-fitness-missing-table-8603.test.ts
# tests 5
# pass 5
# fail 0

$ npx eslint <9 touched files>
✖ 3 problems (3 errors, 0 warnings)
  open-sse/config/providers/shared.ts  784:51  'model' is defined but never used
  open-sse/services/autoCombo/__tests__/autoCombo.test.ts  5:44 'vi' / 7:34 'ScoringWeights' unused

All 3 are pre-existing on the base branch (#11449) and are already frozen in config/quality/eslint-suppressions.json with exactly these counts (shared.ts: 1, autoCombo.test.ts: 2), which is what npm run lint (and CI) applies. 0 new errors.

$ npm run typecheck:core
open-sse/services/compression/engines/omniglyphAdapter.ts  (6 errors)
open-sse/services/compression/omniglyphTelemetry.ts        (3 errors)

Both are pre-existing omniglyph package-export mismatches in files this PR does not touch; no touched file reports an error.

… variants (cheaper-inference#11489)

Auto-combo task fitness scored every catalog id by exact string match, but
dispatch already resolves <model>-<effort> ids to a base model. A variant like
gpt-5.6-sol-xhigh missed both DB layers and fell to the wildcard 0.5 while its
base model was scored properly.

Adds resolveScoresAs(), a catalog-anchored resolver with three tiers: an
explicit scoresAs declared on the registry entry, a reasoning-effort suffix
stripped by one of the EXISTING dispatch splitters, and a -free tier marker.
Tiers 2-3 accept a stripped base only when it is itself a routable catalog id,
which keeps qwen3.7-max (where -max is the model) and grok-4.6-fast-high (whose
base does not exist) unresolved. No new suffix regex is introduced.

getTaskFitnessWithSource retries user_override and arena_elo against the
resolved base on a literal miss and reports <source>:inherited. The former
-free arena_elo special case (cheaper-inference#4517) is folded into the same path and extended
to user_override.

Populates scoresAs for the relations stripping cannot express: the forward
vendor alias gpt-5.6 -> gpt-5.6-sol, and cursor/agy's <version>-<family>
spelling of the Claude ids.

Fixes cheaper-inference#11489
@MumuTW
MumuTW requested a review from diegosouzapw as a code owner August 25, 2026 10:00
@diegosouzapw
diegosouzapw merged commit 78d10da into cheaper-inference:release/v3.8.51 Aug 25, 2026
7 of 16 checks passed
diegosouzapw pushed a commit that referenced this pull request Aug 25, 2026
…ss variants; drop MODEL_ALIAS_MAP (#11504) (#11506)

Validated in a combined 4-PR batch worktree off release/v3.8.51 tip (stacked on #11492, merged first).
- TDD-first: the new base-model-synthesis describe block failed 5/5 pre-fix, passes now
- Focused test: tests/unit/arena-elo-sync.test.ts — 56/56 pass, part of batch's 246/246 node:test + 126/126 vitest runs
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK

Thanks for closing the arena-lookup gap for harness/effort-annotated leaderboard rows and retiring MODEL_ALIAS_MAP's cross-generation score copying in favor of scoresAs + registry-owned aliases.
diegosouzapw pushed a commit that referenced this pull request Aug 25, 2026
… fix 7 BUILT_IN_ALIASES targets, add model-lifecycle gate (#11503) (#11507)

Validated in a combined 4-PR batch worktree off release/v3.8.51 tip. This PR's diff overlapped taskFitness.ts and autoCombo.test.ts with the already-merged #11492/#11506 — git's merge auto-resolved both hunks cleanly (non-overlapping layers: #11492/#11506 touch layer 2 arena lookup, this PR touches layer 4 static-table hygiene); verified no conflict markers remained and re-ran the full suite after boarding.
- npm run check:model-lifecycle — PASS, 68 retired ids, 1327 catalog ids, 0 violations (re-ran with the correct `node --import tsx/esm` loader after an initial bare-node invocation mistakenly failed on path-alias resolution — that was my invocation error, not the gate)
- Focused tests: fitness-table-hygiene-11503.test.ts, taskFitness-pattern-order-8603.test.ts, model-deprecation-aliases-11503.test.ts, check-model-lifecycle-gate.test.ts, model-deprecation.test.ts, autoCombo.test.ts — part of batch's 126/126 vitest + 246/246 node:test runs
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK
- Full-repo lint: 228 problems remaining, all pre-existing dashboard react-hooks/* findings unrelated to this diff (zero errors in any file this PR touches)

Thanks for this — genuinely thorough methodology (segment-boundary matching, provider-scoped alias guard, offline lifecycle gate with a documented burn-down list for the 6 remaining catalog offenders).
@MumuTW
MumuTW deleted the fix/taskfitness-variant-fitness-base branch August 27, 2026 01:47
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
… variants (cheaper-inference#11489) (cheaper-inference#11492)

Validated in a combined 4-PR batch worktree off release/v3.8.51 tip.
- Focused tests: scoresAs-11489.test.ts + task-fitness-scores-as-11489.test.ts + autoCombo.test.ts — pass as part of batch's 126/126 vitest + 246/246 node:test runs
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK

Thanks for closing the gap where effort/alias variants fell back to the wildcard score instead of inheriting their base model's task fitness.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ss variants; drop MODEL_ALIAS_MAP (cheaper-inference#11504) (cheaper-inference#11506)

Validated in a combined 4-PR batch worktree off release/v3.8.51 tip (stacked on cheaper-inference#11492, merged first).
- TDD-first: the new base-model-synthesis describe block failed 5/5 pre-fix, passes now
- Focused test: tests/unit/arena-elo-sync.test.ts — 56/56 pass, part of batch's 246/246 node:test + 126/126 vitest runs
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK

Thanks for closing the arena-lookup gap for harness/effort-annotated leaderboard rows and retiring MODEL_ALIAS_MAP's cross-generation score copying in favor of scoresAs + registry-owned aliases.
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
… fix 7 BUILT_IN_ALIASES targets, add model-lifecycle gate (cheaper-inference#11503) (cheaper-inference#11507)

Validated in a combined 4-PR batch worktree off release/v3.8.51 tip. This PR's diff overlapped taskFitness.ts and autoCombo.test.ts with the already-merged cheaper-inference#11492/cheaper-inference#11506 — git's merge auto-resolved both hunks cleanly (non-overlapping layers: cheaper-inference#11492/cheaper-inference#11506 touch layer 2 arena lookup, this PR touches layer 4 static-table hygiene); verified no conflict markers remained and re-ran the full suite after boarding.
- npm run check:model-lifecycle — PASS, 68 retired ids, 1327 catalog ids, 0 violations (re-ran with the correct `node --import tsx/esm` loader after an initial bare-node invocation mistakenly failed on path-alias resolution — that was my invocation error, not the gate)
- Focused tests: fitness-table-hygiene-11503.test.ts, taskFitness-pattern-order-8603.test.ts, model-deprecation-aliases-11503.test.ts, check-model-lifecycle-gate.test.ts, model-deprecation.test.ts, autoCombo.test.ts — part of batch's 126/126 vitest + 246/246 node:test runs
- typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK
- Full-repo lint: 228 problems remaining, all pre-existing dashboard react-hooks/* findings unrelated to this diff (zero errors in any file this PR touches)

Thanks for this — genuinely thorough methodology (segment-boundary matching, provider-scoped alias guard, offline lifecycle gate with a documented burn-down list for the 6 remaining catalog offenders).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(autoCombo): task fitness scores effort/mode variant ids raw; 219 static variants fall to wildcard 0.5 while their base model is scored

2 participants