Repository navigation
feat(rankings): order free providers by measured reliability - #12218
Merged
diegosouzapw merged 2 commits intoSep 1, 2026
Merged
diegosouzapw merged 2 commits into
diegosouzapw merged 2 commits into
Conversation
maxmad64bis
force-pushed
the
feat/rankings-reliability-sort
branch
2 times, most recently
from
August 31, 2026 18:55
6855361 to
5c67330
Compare
maxmad64bis
force-pushed
the
feat/rankings-reliability-sort
branch
from
August 31, 2026 19:21
5c67330 to
a566cf6
Compare
…new sort keys - freeProviderRankings.ts: extract needsUsageAggregate() and applyRankingOrder() so computeFreeProviderRankings stays within the cyclomatic ratchet (file back to the base's single pre-existing cognitive violation) - vi.json / pt-BR.json: real translations for the two new keys (sortByReliability, sortByReliabilityHelp) — the strict Vietnamese completeness gate rejects __MISSING__ markers
diegosouzapw
merged commit Sep 1, 2026
5253b93
into
cheaper-inference:release/v3.8.51
15 of 16 checks passed
7 tasks done
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…-inference#12218) Free Provider Rankings sorted by the top model's Arena score, so a provider stayed first even if it failed every call — the reliability column from cheaper-inference#11546 already showed what each one actually served, but ordering ignored it. Adds an opt-in ?sortBy=reliability (API) and a "Most reliable first" toggle (page) sharing one comparator in freeProviderRankingsUsage.ts: measured providers first by successRate desc with ELO on ties, then unmeasured in their incoming order. The default is unchanged and locked by tests. successRate is null below MIN_USAGE_REQUESTS = 5 (existing, never zero), and ordering runs before slice(0, limit) so limit counts in the requested order. The toggle composes with the existing sortTypeFirst/groupByType grouping — a stable sort keeps reliability order within each group. Opt-in is the right default here: the page is for discovery, including providers never called. 13 tests across three files (6 new for the comparator in isolation, plus filter and route coverage including unknown sortBy → 400 and the default path staying off call_logs). Verified in a combined batch worktree with all 11 PRs of this batch: 174/174 focused tests, typecheck:core clean, complexity 2706/3218, cognitive-complexity 1221/1437, check:cycles and check:docs-counts green. The 42-locale i18n pass was checked with the CI gates: check-ui-keys-coverage PASS (all 42 locales at or above 65%) and check-ui-value-drift PASS against the release tip. Thanks @maxmad64bis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Free Provider Rankings sorts by the top model's Arena score — a provider stays first even if it fails every call. The reliability column (#11546) already shows what each provider actually served, but ordering ignores it.
This PR adds an opt-in
?sortBy=reliability(API) and a "Most reliable first" toggle (page), sharing one comparator infreeProviderRankingsUsage.ts— measured providers first (bysuccessRatedesc, ELO on ties), then unmeasured in their incoming order. Default unchanged (locked by tests).successRateisnullbelowMIN_USAGE_REQUESTS = 5(existing, never zero), and ordering runs beforeslice(0, limit)solimitcounts in the requested order.The toggle is deliberately opt-in — the page is for discovery, including providers never called — and composes with the existing
sortTypeFirst/groupByTypetype-grouping (stable sort keeps reliability order within each group).Related Issues
Validation
npm run lintupstream/release/v3.8.51)Tests Added Or Updated
tests/unit/freeProviderRankings-reliability-sort.test.ts— 6 tests forsortRankingsByReliabilityin isolation (measured first, rate desc, ELO fallback on ties,nullcounts as unmeasured, stable unmeasured, no mutation).tests/unit/freeProviderRankings-filters.test.ts— 3 tests: nosortBypreserves ELO order,sortBy=reliabilitypulls usage withoutwithUsage, and limit + combinedsortRankingsAuthTypeFirststability.tests/unit/free-provider-rankings-usage-route.test.ts— 4 tests: unknownsortBy→ 400,sortBy=elo/reliabilityaccepted,sortBy=reliabilityhitscall_logswithoutwithUsage, default path stays offcall_logs.Hard Rule #18 red/green: stubbing the comparator to
return [...rankings]→freeProviderRankings-reliability-sort2/6, restore → all green (6/6, 29/29, 8/8).Coverage Notes
Touches
src/lib/,src/app/,src/i18n/. Covered by suites above plus existing free-provider integration tests. i18n gates:i18n:check-ui-coveragePASS,i18n:check-value-drift(BASE_REF=upstream/release/v3.8.51) PASS,i18n:check-glossaryPASS,check:dashboard-typecheckwithin baseline.Reviewer Notes
Diff is two halves — read behavior first, then catalogs:
src/lib/freeProviderRankingsUsage.ts(comparator),src/lib/freeProviderRankings.ts(sortBy,needsConnectionSnapshot, ordering before slice, usage aggregate gated onsortBy),src/app/api/free-provider-rankings/route.ts(sortByenum, 400 on typo),src/app/(dashboard)/dashboard/free-provider-rankings/page.tsx(toggle,useMemocomposition).en.jsongainssortByReliability/sortByReliabilityHelp(2 keys). 42 locales viascripts/i18n/sync-ui-keys.mjsgain those 2 plus 23 prioren_onlykeys already inen.jsonat base but missing in those locales (40 × +25,vi/pt-BR+2) — all__MISSING__sentinels, no hand editing. All i18n gates PASS.No new threshold, no silent coercion, no extra network call (
withUsagealready fetched).