Repository navigation
fix(autoCombo): inherit task fitness from base model for effort/alias variants (#11489) - #11492
Merged
diegosouzapw merged 2 commits intoAug 25, 2026
Conversation
… variants (cheaper-inference#11489) Auto-combo task fitness scored every catalog id by exact string match, but dispatch already resolves <model>-<effort> ids to a base model. A variant like gpt-5.6-sol-xhigh missed both DB layers and fell to the wildcard 0.5 while its base model was scored properly. Adds resolveScoresAs(), a catalog-anchored resolver with three tiers: an explicit scoresAs declared on the registry entry, a reasoning-effort suffix stripped by one of the EXISTING dispatch splitters, and a -free tier marker. Tiers 2-3 accept a stripped base only when it is itself a routable catalog id, which keeps qwen3.7-max (where -max is the model) and grok-4.6-fast-high (whose base does not exist) unresolved. No new suffix regex is introduced. getTaskFitnessWithSource retries user_override and arena_elo against the resolved base on a literal miss and reports <source>:inherited. The former -free arena_elo special case (cheaper-inference#4517) is folded into the same path and extended to user_override. Populates scoresAs for the relations stripping cannot express: the forward vendor alias gpt-5.6 -> gpt-5.6-sol, and cursor/agy's <version>-<family> spelling of the Claude ids. Fixes cheaper-inference#11489
This was referenced Aug 25, 2026
diegosouzapw
merged commit Aug 25, 2026
78d10da
into
cheaper-inference:release/v3.8.51
7 of 16 checks passed
diegosouzapw
pushed a commit
that referenced
this pull request
Aug 25, 2026
…ss variants; drop MODEL_ALIAS_MAP (#11504) (#11506) Validated in a combined 4-PR batch worktree off release/v3.8.51 tip (stacked on #11492, merged first). - TDD-first: the new base-model-synthesis describe block failed 5/5 pre-fix, passes now - Focused test: tests/unit/arena-elo-sync.test.ts — 56/56 pass, part of batch's 246/246 node:test + 126/126 vitest runs - typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK Thanks for closing the arena-lookup gap for harness/effort-annotated leaderboard rows and retiring MODEL_ALIAS_MAP's cross-generation score copying in favor of scoresAs + registry-owned aliases.
diegosouzapw
pushed a commit
that referenced
this pull request
Aug 25, 2026
… fix 7 BUILT_IN_ALIASES targets, add model-lifecycle gate (#11503) (#11507) Validated in a combined 4-PR batch worktree off release/v3.8.51 tip. This PR's diff overlapped taskFitness.ts and autoCombo.test.ts with the already-merged #11492/#11506 — git's merge auto-resolved both hunks cleanly (non-overlapping layers: #11492/#11506 touch layer 2 arena lookup, this PR touches layer 4 static-table hygiene); verified no conflict markers remained and re-ran the full suite after boarding. - npm run check:model-lifecycle — PASS, 68 retired ids, 1327 catalog ids, 0 violations (re-ran with the correct `node --import tsx/esm` loader after an initial bare-node invocation mistakenly failed on path-alias resolution — that was my invocation error, not the gate) - Focused tests: fitness-table-hygiene-11503.test.ts, taskFitness-pattern-order-8603.test.ts, model-deprecation-aliases-11503.test.ts, check-model-lifecycle-gate.test.ts, model-deprecation.test.ts, autoCombo.test.ts — part of batch's 126/126 vitest + 246/246 node:test runs - typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK - Full-repo lint: 228 problems remaining, all pre-existing dashboard react-hooks/* findings unrelated to this diff (zero errors in any file this PR touches) Thanks for this — genuinely thorough methodology (segment-boundary matching, provider-scoped alias guard, offline lifecycle gate with a documented burn-down list for the 6 remaining catalog offenders).
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
… variants (cheaper-inference#11489) (cheaper-inference#11492) Validated in a combined 4-PR batch worktree off release/v3.8.51 tip. - Focused tests: scoresAs-11489.test.ts + task-fitness-scores-as-11489.test.ts + autoCombo.test.ts — pass as part of batch's 126/126 vitest + 246/246 node:test runs - typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK Thanks for closing the gap where effort/alias variants fell back to the wildcard score instead of inheriting their base model's task fitness.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ss variants; drop MODEL_ALIAS_MAP (cheaper-inference#11504) (cheaper-inference#11506) Validated in a combined 4-PR batch worktree off release/v3.8.51 tip (stacked on cheaper-inference#11492, merged first). - TDD-first: the new base-model-synthesis describe block failed 5/5 pre-fix, passes now - Focused test: tests/unit/arena-elo-sync.test.ts — 56/56 pass, part of batch's 246/246 node:test + 126/126 vitest runs - typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK Thanks for closing the arena-lookup gap for harness/effort-annotated leaderboard rows and retiring MODEL_ALIAS_MAP's cross-generation score copying in favor of scoresAs + registry-owned aliases.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
… fix 7 BUILT_IN_ALIASES targets, add model-lifecycle gate (cheaper-inference#11503) (cheaper-inference#11507) Validated in a combined 4-PR batch worktree off release/v3.8.51 tip. This PR's diff overlapped taskFitness.ts and autoCombo.test.ts with the already-merged cheaper-inference#11492/cheaper-inference#11506 — git's merge auto-resolved both hunks cleanly (non-overlapping layers: cheaper-inference#11492/cheaper-inference#11506 touch layer 2 arena lookup, this PR touches layer 4 static-table hygiene); verified no conflict markers remained and re-ran the full suite after boarding. - npm run check:model-lifecycle — PASS, 68 retired ids, 1327 catalog ids, 0 violations (re-ran with the correct `node --import tsx/esm` loader after an initial bare-node invocation mistakenly failed on path-alias resolution — that was my invocation error, not the gate) - Focused tests: fitness-table-hygiene-11503.test.ts, taskFitness-pattern-order-8603.test.ts, model-deprecation-aliases-11503.test.ts, check-model-lifecycle-gate.test.ts, model-deprecation.test.ts, autoCombo.test.ts — part of batch's 126/126 vitest + 246/246 node:test runs - typecheck:core, file-size, changelog-integrity, complexity, cognitive-complexity, check:cycles — all OK - Full-repo lint: 228 problems remaining, all pre-existing dashboard react-hooks/* findings unrelated to this diff (zero errors in any file this PR touches) Thanks for this — genuinely thorough methodology (segment-boundary matching, provider-scoped alias guard, offline lifecycle gate with a documented burn-down list for the 6 remaining catalog offenders).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11489
6435f618f4with this branch's changes absent (each verified in a detached worktree or bygit show base:):run-eslint-json.mjsexits 2 on stale entries inconfig/quality/eslint-suppressions.json(none in files this PR touches)glmCodingProviderConfig.test.ts— 2 failures (glm-5.3-maxtiers, 15-vs-14 inventory)APIKEY_PROVIDERScount, openapi coverage floor, DB handles — none intaskFitness/scoresAs; the only unit test this PR adds passescheck:lockfile(libxmljs2/…/brace-expansionresolved fromregistry.npmmirror.comin the base lockfile),check:mutation-test-coverage(2 base test files missing fromstryker.conf.json),check:open-sse-typecheck(src/app/api/v1/models/catalog.tsTS2367, baseline 0 → live 2). All other 20 gates pass locally;file-sizeis base-relative and every touched file is under the 1000-line capresolveLiveWsUrl/sanitizeLiveWsPortmissing exports — untouched by this PRProblem
Auto-combo task fitness scores every catalog id by exact string match, but the gateway already resolves
<model>-<effort>ids to a base model at dispatch. The scoring layer never sees that resolution, so a variant likegpt-5.6-sol-xhighmisses every DB layer and lands on the wildcard0.5— while its own base model is scored properly.Mechanism
getTaskFitnessWithSource()layers 1–2 callgetModelIntelligenceBySource(), which isWHERE model = ?. Combo candidate scoring passes the raw registry id (buildAutoCandidates()→parseModel(target.modelStr).model); no effort splitter is imported anywhere underopen-sse/services/combo*. Every DB-backed source (arena, models.dev, user overrides) publishes scores for base models only, so no data source can close this gap.Fix
New
open-sse/services/autoCombo/scoresAs.tsexports one resolver:Three tiers, in order, and it never guesses:
explicit— the registry entry declares the newRegistryModel.scoresAs. One hop only; the target must itself be a catalog id.effort-suffix— a trailing reasoning-effort token stripped by an existing dispatch splitter (splitClaudeEffortSuffix,splitCodexReasoningSuffix). No new regex is introduced.free-suffix— a trailing-freetier marker.Tiers 2–3 are catalog-anchored: a stripped base is accepted only when
findRegistryModelById(base)(newly exported fromproviderModels.ts—getGlobalModel's exact-id steps 1–2, deliberately not itsstartsWithstep 3) finds it. On today's catalog, 57 of the 201 strippable effort ids strip to a base that does not exist —qwen3.7-max → qwen3.7(-maxis the model name here),grok-4.6-fast-high → grok-4.6-fast,extra-high → extra. The anchor rejects all of them.getTaskFitnessWithSource()retriesuser_overridethenarena_eloagainst the resolved base on a literal miss and reports<source>:inherited. The score is inherited verbatim — the 12-factor scoring already prices cost and latency per variant, so there is no basis for inventing an effort delta.The former
-freearena-elo special case (lookupFreeAliasArenaElo, sourcearena_elo_free_alias, #4517) is deleted and folded into tier 3, which also extends it touser_override(it was arena-elo-only before).Cache keys are the queried model id, so an inherited hit for a variant cannot poison the base's entry or vice versa;
invalidateFitnessCache()still clears everything.FITNESS_TABLEcontents are untouched (that is #8777).Measured effect
Resolving every catalog id + alias (1327 ids) through
getTaskFitnessWithSource(id, "coding"), DATA_DIR isolated:Structural tier breakdown of
resolveScoresAs:expliciteffort-suffixfree-suffixCoverage, with
arena_eloseeded on each of the 47 distinct base ids the variants point at (the mechanism's ceiling given full base coverage):arena_eloarena_elo_free_aliasarena_elo:inheritedfitness_tablewildcard_boost105 ids leave the wildcard entirely; a further 55 move off the static
fitness_tableonto the real benchmarked signal of the model they actually are. This number rises with every base row any data source adds, which is the point.scoresAsentries populatedOnly relations tiers 2–3 cannot express — 15 entries:
gpt-5.6(openai)gpt-5.6-solgpt-5.6is an alias ofgpt-5.6-sol, not a model. Points forward; no suffix stripper can produce this. Siblings-terra/-lunadeliberately left alone.claude-4.6-opus-high(cursor/agy)claude-opus-4-6<version>-<family>; strips to the ghostclaude-4.6-opus.claude-4.6-opus-maxclaude-opus-4-6claude-4.6-opus-high-thinkingclaude-opus-4-6claude-4.6-opus-max-thinkingclaude-opus-4-6claude-4.6-opus-high-thinking-fastclaude-opus-4-6claude-4.6-opus-max-thinking-fastclaude-opus-4-6claude-4.5-opus-highclaude-opus-4-5claude-4.5-opus-high-thinkingclaude-opus-4-5claude-4.6-sonnet-mediumclaude-sonnet-4-6claude-4.6-sonnet-medium-thinkingclaude-sonnet-4-6claude-4.5-sonnetclaude-sonnet-4-5claude-4.5-sonnet-thinkingclaude-sonnet-4-5claude-4-sonnetclaude-sonnet-4claude-4-sonnet-thinkingclaude-sonnet-4Everything else stays unresolved by design.
Tests
TDD per Hard Rule #18 — both files were written first and observed failing (
Cannot find module '../scoresAs'/ 2 of 4 subtests failing) before the implementation landed.open-sse/services/autoCombo/__tests__/scoresAs-11489.test.ts(vitest, 10 tests, no DB) — pure resolver. Coversgpt-5.6-sol-xhigh→ effort-suffix,gpt-5.6→ explicit,gpt-5.6-lunaunresolved (and specifically notgpt-5.6/gpt-5.6-sol),qwen3.7-maxunresolved,claude-sonnet-5never collapsing to its family, the cursor spellings,mimo-v2.5-free→ free-suffix,ox-alpha-freeunresolved.grok-4.6-fast-highis asserted againstfindRegistryModelByIdrather than a hardcoded expectation, so the test stays true as the catalog moves.tests/unit/task-fitness-scores-as-11489.test.ts(node:test, 4 tests, DB-backed) — seeds auser_overrideongpt-5.6-sol, assertsgpt-5.6-sol-xhighandgpt-5.6return it asuser_override:inheritedand thatgpt-5.6-lunadoes not. Cleans up and callsresetDbInstance()inafter().open-sse/services/autoCombo/__tests__/autoCombo.test.ts— the feat: qualify ambiguous model names with provider prefix in /v1/models #4517-freecase updated for the new source string (arena_elo:inherited) and moved off synthetic ids (alias-base-test-4517) onto a real catalog pair (mimo-v2.5/mimo-v2.5-free), since the base lookup is now catalog-anchored.Verification
All 3 are pre-existing on the base branch (#11449) and are already frozen in
config/quality/eslint-suppressions.jsonwith exactly these counts (shared.ts: 1,autoCombo.test.ts: 2), which is whatnpm run lint(and CI) applies. 0 new errors.Both are pre-existing
omniglyphpackage-export mismatches in files this PR does not touch; no touched file reports an error.