Skip to content

fix(autoCombo,sse): drop vendor-retired ids from the auto-combo pool and skip leftover arena_elo (#11625) - #11626

Merged
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.51from
MumuTW:fix/11625-retired-pool-fitness
Aug 26, 2026
Merged

diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.51from
MumuTW:fix/11625-retired-pool-fitness

Conversation

@MumuTW

@MumuTW MumuTW commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Fixes #11625

Not a replacement for #11598. That PR's layer-3 veto (prefix / inherited-base) stays on #11508. This is the candidate pool + layers 1–2, which still select a retired model even after a layer-3 veto.

⚠️ base-red inherited: #11449 — remaining red checks on this PR reproduce on the pristine base tip.

Problem

getTaskFitnessWithSource() is user_override → arena_elo → inherited 1–2 → models_dev_tier → FITNESS_TABLE → wildcard.

#11507 stopped layer 4 from ranking retired ids at 0.98. #11508 / #11598 veto layer 3. Neither stops auto-combo from picking a retired model:

  1. buildAutoCandidates() only drops dashboard-hidden models. filterSelectableModels() exists but reads MODEL_LIFECYCLE_RECORDS (OpenAI-only hardcoded). An Anthropic/Google retired id is untracked → allow.
  2. A leftover arena_elo / user_override row returns at layer 1–2, so the layer-3 veto never runs. lookupInheritedFitness also copies a dead base's arena score onto an effort variant.
  3. Aggregator traffic (openrouter + gpt-5.2-codex) is untracked because the dated table is provider-scoped.

allowedRetiredInCatalog (6 ids) stays a catalog/CI burn-down. This PR does not delete catalog rows; it stops auto-combo from selecting them and stops leftover intelligence rows from scoring above the 0.5 neutral.

Fix

One prefix-stripped predicate, isVendorRetiredId(), reads config/quality/model-lifecycle.json (status === "retired" only — retiring with retiredOn: null is left alone).

  • rejectRetiredAutoComboCandidates() runs after the hidden-model filter in buildAutoCandidates.
  • getTaskFitnessWithSource skips layers 1–3 when the request id or its resolveScoresAs() base is retired, and returns wildcard 0.5 with no *codex / code boost.
  • getModelLifecycleDecision consults the same snapshot after the dated OpenAI table misses, so aggregator traffic is not untracked for ids the JSON already knows. Dated OpenAI warn-before-shutdown rows stay provider-scoped.

Tests

TDD (Hard Rule #18): tests/unit/retired-pool-11625.test.ts was written first and failed 9/11 on the pre-fix chain.

  • isVendorRetiredId("openai/gpt-5.2-codex") and the bare form
  • rejectRetiredAutoComboCandidates keeps gpt-5.6-sol, drops snapshot-retired ids
  • isModelSelectable("anthropic", "claude-3-7-sonnet-20250219") === false
  • seeded arena_elo on gpt-5.2-codex / openai/gpt-5.2-codex is not returned
  • claude-3-7-sonnet-20250219-high does not inherit the dead base's arena row
  • live gpt-5.6-sol arena row still wins

Existing model-lifecycle.test.ts updated: dated OpenAI deprecations stay provider-scoped; snapshot-retired ids are rejected on every provider. #11489 inherit tests still pass.

Verification

DISABLE_SQLITE_AUTO_BACKUP=true node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test \
  tests/unit/retired-pool-11625.test.ts tests/unit/model-lifecycle.test.ts tests/unit/task-fitness-scores-as-11489.test.ts
# tests 22 | pass 22 | fail 0

DISABLE_SQLITE_AUTO_BACKUP=true node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test \
  tests/unit/model-lifecycle-integration.test.ts tests/unit/auto-combo-hidden-models-4558.test.ts tests/unit/model-deprecation-aliases-11503.test.ts
# tests 73 | pass 73 | fail 0

npm run check:model-lifecycle
# PASS — snapshot 2026-08-25, 68 retired id(s), 1327 catalog id(s)

npm run check:cycles
# OK - no cycles detected across 429 files

npm run typecheck:core reports the pre-existing omniglyph package-export mismatches on files this PR does not touch (same baseline as #11492 / #11507). Touched files are under the 1000-line cap.

…and skip leftover arena_elo (diegosouzapw#11625)

Auto-combo still selected vendor-retired catalog ids after diegosouzapw#11507/diegosouzapw#11508
because buildAutoCandidates only filtered dashboard-hidden models, and
getTaskFitnessWithSource returned leftover arena_elo / user_override rows
before the layer-3 veto could run. A variant could also inherit a dead
base's arena score via resolveScoresAs.

One prefix-stripped predicate reads model-lifecycle.json (status=retired
only; retiring-without-date is left alone). The candidate pool drops
those ids, layers 1–3 miss so a leftover row cannot short-circuit, and
getModelLifecycleDecision consults the same snapshot so aggregator
traffic is not untracked for ids the JSON already knows. Dated OpenAI
rows stay provider-scoped.

Fixes diegosouzapw#11625
# Conflicts:
#	open-sse/services/autoCombo/taskFitness.ts
@diegosouzapw
diegosouzapw merged commit 00d385b into diegosouzapw:release/v3.8.51 Aug 26, 2026
4 of 7 checks passed
@MumuTW
MumuTW deleted the fix/11625-retired-pool-fitness branch September 5, 2026 10:19
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…and skip leftover arena_elo (diegosouzapw#11625) (diegosouzapw#11626)

Merged via /merge-batch (lote 2026-08-26 batch 2, v3.8.51). Conflito real em taskFitness.ts: seu código pula corretamente as camadas 1-3 para ids vendor-retired (a versão anterior fazia lookup redundante de arena_elo/inherited/tier mesmo quando já sabia que o id era retired). Resolvido mantendo o skip antecipado. Validado: 23/23 testes passando. Obrigado pela contribuição.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(autoCombo): vendor-retired catalog ids stay in the candidate pool and still win on arena_elo after the layer-3 veto

3 participants