fix(auto-combo): give every mode pack quality and reliability weights - #12731
Merged
diegosouzapw merged 1 commit intoSep 10, 2026
Conversation
maxmad64bis
force-pushed
the
fix/batch2-scoring-packs-quality-reliability
branch
2 times, most recently
from
September 4, 2026 14:07
9c94ffd to
1aa284b
Compare
maxmad64bis
changed the title
5 tasks done
maxmad64bis
force-pushed
the
fix/batch2-scoring-packs-quality-reliability
branch
2 times, most recently
from
September 6, 2026 01:32
3b0b960 to
d08fe12
Compare
maxmad64bis
force-pushed
the
fix/batch2-scoring-packs-quality-reliability
branch
from
September 10, 2026 11:51
d08fe12 to
b4e8b65
Compare
diegosouzapw
pushed a commit
that referenced
this pull request
Sep 10, 2026
…ty/breaker (#12794) Validado numa worktree combinada com a onda de roteamento/free-tier desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-api-typecheck OK (288), check-file-size OK após rebaseline, 84/85 nos testes focados. Tratar breaker aberto como equivalente a fechado no scoring de snapshot é pior que não pontuar: afirma saúde onde há falha conhecida. Trocar as três constantes neutras por valor observado é a correção, e manter preço e orçamento fora do escopo mantém a PR revisável. **Integração:** o `computeSnapshotWeights` conflitou com o #12731, que adiciona peso de `reliability` a partir de `failureRate`/`errorRate`. Os dois cobrem chaves diferentes e compõem — ficaram ambos: reliability do #12731, health via breaker e quality desta PR, quota neutro nos dois. Nenhum dos dois lados foi descartado.
diegosouzapw
merged commit Sep 10, 2026
9795372
into
diegosouzapw:release/v3.8.51
9 of 16 checks passed
diegosouzapw
pushed a commit
that referenced
this pull request
Sep 10, 2026
…eights, quality gate and scoring diagram now covered (#12507) Estender o gate de contagens para headings, rankings, catálogo, pesos, quality gate e o diagrama de scoring é exatamente o tipo de trabalho que evita a classe inteira em vez de um caso. Falo por experiência desta campanha: o `check:docs-counts` caiu **duas vezes** hoje pela mesma causa — contagem de migration escrita à mão em três arquivos mais 41 mirrors, desatualizando a cada migration nova (#12970 e #13209). Cada superfície que este PR passa a cobrir é uma que deixa de virar base-red na mão de quem vier depois. Revalidei sobre o tip: **19/19**, `check:docs-counts-sync` com 0 drifts, `check:docs-all` PASS, `check:doc-links` PASS. **Integração:** dois conflitos. 1. `scripts/check/check-docs-counts-sync.mjs` — o bloco de leitura de fatos conflitou com os imports de free-tier que entraram pelo #12786/#12744 nesta campanha. Aditivo, os dois conjuntos ficaram. 2. `docs/diagrams/auto-combo-scoring.mmd` — o seu rótulo dizia `reliability (0.0000)`, mas o #12731 mergeou horas antes e passou a dar peso de reliability a todo mode pack. Ficou o rótulo do tip, `reliability (0.0000 DEFAULT, 0.03 packs, 0.04 reliable)`, que é o número real agora.
diegosouzapw
added a commit
that referenced
this pull request
Sep 10, 2026
#12731 added the "Mode pack ranking gates (cold/warm/health)" block to open-sse/services/autoCombo/__tests__/autoCombo.test.ts with seven ProviderCandidate literals that omit three required fields (provider, model, errorRate). check:open-sse-typecheck reports them as 'TS2739 (baseline 0, live 7)', which fails Fast Quality Gates on the tip and therefore on every open PR. The fields are filled with neutral values that preserve each case's intent: provider/model are identical across every candidate (so classifyTier cannot skew a comparison), and errorRate is 0 — which is only read when failureRate is absent, i.e. the two 'cold pool' candidates the test itself describes as 'reliability 1'. Validation: npm run check:open-sse-typecheck goes from 'FAIL - 1 new/regressed error' to 'OK - 0 pre-existing error(s)'. The suite itself does not execute in CI: vitest.config.ts excludes this file (#8618), so typecheck is its only gate. Refs #12732
diegosouzapw
added a commit
that referenced
this pull request
Sep 11, 2026
#12731 added the "Mode pack ranking gates (cold/warm/health)" block to open-sse/services/autoCombo/__tests__/autoCombo.test.ts with seven ProviderCandidate literals that omit three required fields (provider, model, errorRate). check:open-sse-typecheck reports them as 'TS2739 (baseline 0, live 7)', which fails Fast Quality Gates on the tip and therefore on every open PR. The fields are filled with neutral values that preserve each case's intent: provider/model are identical across every candidate (so classifyTier cannot skew a comparison), and errorRate is 0 — which is only read when failureRate is absent, i.e. the two 'cold pool' candidates the test itself describes as 'reliability 1'. Validation: npm run check:open-sse-typecheck goes from 'FAIL - 1 new/regressed error' to 'OK - 0 pre-existing error(s)'. The suite itself does not execute in CI: vitest.config.ts excludes this file (#8618), so typecheck is its only gate. Refs #12732
7 tasks done
maxmad64bis
deleted the
fix/batch2-scoring-packs-quality-reliability
branch
September 24, 2026 21:12
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ty/breaker (diegosouzapw#12794) Validado numa worktree combinada com a onda de roteamento/free-tier desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-api-typecheck OK (288), check-file-size OK após rebaseline, 84/85 nos testes focados. Tratar breaker aberto como equivalente a fechado no scoring de snapshot é pior que não pontuar: afirma saúde onde há falha conhecida. Trocar as três constantes neutras por valor observado é a correção, e manter preço e orçamento fora do escopo mantém a PR revisável. **Integração:** o `computeSnapshotWeights` conflitou com o diegosouzapw#12731, que adiciona peso de `reliability` a partir de `failureRate`/`errorRate`. Os dois cobrem chaves diferentes e compõem — ficaram ambos: reliability do diegosouzapw#12731, health via breaker e quality desta PR, quota neutro nos dois. Nenhum dos dois lados foi descartado.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…diegosouzapw#12731) Validado numa worktree combinada com a onda de roteamento/free-tier desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-api-typecheck OK (288), check-file-size OK após rebaseline, 84/85 nos testes focados. `reliability-first` que não pesava reliability é o defeito mais constrangedor possível num mode pack, e a causa é clara: `modePacks.ts:13` substituía os defaults por inteiro. Financiar os novos pesos com `quota`/`costInv`/`tierPriority` mantendo cada pack somando 1.0 é a parte que exige cuidado e você fez. Manter `quality-first` em 0.03, igual ao default, para que ele não fique mais fraco que `balanced`, é o tipo de detalhe que só aparece quando se checa a tabela inteira. **Integração:** conflitou com o diegosouzapw#12794 no `computeSnapshotWeights`; os dois compõem e ambos ficaram.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…eights, quality gate and scoring diagram now covered (diegosouzapw#12507) Estender o gate de contagens para headings, rankings, catálogo, pesos, quality gate e o diagrama de scoring é exatamente o tipo de trabalho que evita a classe inteira em vez de um caso. Falo por experiência desta campanha: o `check:docs-counts` caiu **duas vezes** hoje pela mesma causa — contagem de migration escrita à mão em três arquivos mais 41 mirrors, desatualizando a cada migration nova (diegosouzapw#12970 e diegosouzapw#13209). Cada superfície que este PR passa a cobrir é uma que deixa de virar base-red na mão de quem vier depois. Revalidei sobre o tip: **19/19**, `check:docs-counts-sync` com 0 drifts, `check:docs-all` PASS, `check:doc-links` PASS. **Integração:** dois conflitos. 1. `scripts/check/check-docs-counts-sync.mjs` — o bloco de leitura de fatos conflitou com os imports de free-tier que entraram pelo diegosouzapw#12786/diegosouzapw#12744 nesta campanha. Aditivo, os dois conjuntos ficaram. 2. `docs/diagrams/auto-combo-scoring.mmd` — o seu rótulo dizia `reliability (0.0000)`, mas o diegosouzapw#12731 mergeou horas antes e passou a dar peso de reliability a todo mode pack. Ficou o rótulo do tip, `reliability (0.0000 DEFAULT, 0.03 packs, 0.04 reliable)`, que é o número real agora.
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Picking a mode like
reliableorqualityused to mute two signals the scorer already knows: how good the output was (quality) and how often the provider failed in the last 24 h (reliability). Soreliability-firstdidn't actually weigh reliability at all (open-sse/services/autoCombo/modePacks.ts:13replaces the defaults wholesale). Now every pack carries both —quality 0.02exceptquality-first 0.03(same as the default, soquality-firstisn't weaker thanbalanced),reliability 0.03exceptreliability-first 0.04. Funding comes fromquota/costInv/tierPriority; health (the circuit-breaker — open/half-open/closed) and stability (how steady the latency is) stay untouched, and each pack still sums to1.0(0.9999as printed at four decimals).Related Issues
AUTO-COMBO.mdarea, no overlap.Validation
npm run lint(pre-existing suppressions only; touched files clean when grepped by relative path)Tests Added Or Updated
open-sse/services/autoCombo/__tests__/autoCombo.test.ts— gates that each pack hasqualityandreliability, a cold pool (no observations) doesn't change ranking, a warm pool does,NaNmaps to neutral, and a healthy provider still beats a flaky one. (63/63 green).tests/unit/auto-combos-suffixes-4235.test.ts— same per-pack gate (9/9).tests/unit/check-docs-counts-quality-pinning.test.ts(new) — locks that every pack pinsqualityexplicitly and the scoring fallback stays neutral, plus a strict gate (6/6).docs/routing/AUTO-COMBO.md— emphasis prose rebased on the rebalanced packs.Coverage Notes
Touches
open-sse/services/autoCombo/modePacks.tsandscoring.ts; covered by the two files above. No change to the default path (reliabilitystays0there).Reviewer Notes
Docs (
AUTO-COMBO.md:187/210/218/236/243andauto-combo-scoring.mmdf16) updated alongside the code. When a pool has no observations, packs add a flat baseline — ranking only shifts once real failure-rate data fills up, and a closed breaker still outweighs a bad failure rate.