Skip to content

fix(auth): prefer accounts without backoff in least-used rotation - #12375

Merged
diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
pacocartones:fix/least-used-backoff-tiebreak
Sep 2, 2026
Merged

diegosouzapw merged 2 commits into
cheaper-inference:release/v3.8.51from
pacocartones:fix/least-used-backoff-tiebreak

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

Summary

  • The least-used account strategy ranked candidates by lastUsedAt only. After a 429 excluded the active account, the replacement could be an account that was merely the oldest by lastUsedAt while still carrying a backoff from its own recent rate limit; it served one request and the next request settled on a healthy account (the one-request detour with two cache misses reported on Codex). least-used now applies the same backoffLevel tie-break the round-robin fallback branch already has: lower backoff first, then the existing never-used / oldest / priority order.

Related Issues

Validation

  • Change type: routing
  • Focused tests and category gates from the golden path: tests/unit/sse-auth.test.ts (67/67), npm run typecheck:core 0, npm run check:changelog-integrity OK, git diff --check clean
  • npm run lint — repository-wide eslint exit 0 (run with --pass-on-unpruned-suppressions; the literal command reports only pre-existing unused global suppressions on this base)
  • Reconciled with the current active release base release/v3.8.51@158647618; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR
  • SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Tests Added Or Updated

  • tests/unit/sse-auth.test.ts: new case getProviderCredentials least-used prefers an account without backoff over the least recently used one (#12279) (RED on the base: backed-off account selected; GREEN with the fix).

Coverage Notes

  • src/sse/services/auth.ts: the added comparator lines are covered by the new test; the two existing least-used tests (never-used preference, oldest-timestamp preference) still pass and cover the equal-backoff path.

Reviewer Notes

  • This changes selection for every provider configured with least-used, not only Codex. With all candidates at backoffLevel 0 (the common case) the order is identical to before.
  • backoffLevel is auto-decayed to 0 in getProviderCredentials once rateLimitedUntil has passed, and rate-limited accounts are pre-filtered unless the caller sets allowRateLimitedConnections (combo targets with allowRateLimitedConnection). The tie-break therefore takes effect for accounts still inside their rate-limit window that a caller deliberately keeps in the pool; the test seeds exactly that shape. createProviderConnection does not persist backoffLevel, so the test writes it via updateProviderConnection.
  • The Codex mid-request failover path (chatCore.ts calling the plain selector instead of the quota-preflight one) is the other half of the triage and is not changed here.

The least-used strategy in getProviderCredentials() sorted candidates by
lastUsedAt only. round-robin's fallback branch in the same function
already ranks backoffLevel first (cheaper-inference#340), but least-used had no such
tie-break, so after a 429 excluded the active account the replacement
could be an account that was simply the oldest by lastUsedAt while
still carrying a backoff from its own recent rate limit. That account
served one request, its lastUsedAt moved to now, and the next request
settled on the healthy account, which is the one-request detour with
two cache misses reported in cheaper-inference#12279 on Codex.

Prepend the same backoffLevel comparison to the least-used comparator:
lower backoff first, then the existing never-used / oldest-lastUsedAt /
priority order. Accounts with equal backoff (the common case, all zero)
sort exactly as before.

Test: tests/unit/sse-auth.test.ts seeds two accounts where the least
recently used one carries backoffLevel 2 with an active rateLimitedUntil
(so the auto-decay does not clear it) and selects with
allowRateLimitedConnections so it stays in the pool; the healthy account
must win. Fails on the base (backed-off account selected), passes with
this change.

Closes cheaper-inference#12279

Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
@diegosouzapw
diegosouzapw merged commit 0389b07 into cheaper-inference:release/v3.8.51 Sep 2, 2026
16 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…eaper-inference#12375)

The least-used strategy ranked candidates by lastUsedAt alone, so after a 429 excluded the active account the replacement could be one that was merely oldest while still carrying its own backoff — it served a single request before the next one settled on a healthy account, the one-request detour with two cache misses reported on Codex. least-used now applies the backoffLevel tie-break the round-robin fallback branch already had, ahead of the existing never-used / oldest / priority order.

Validated in a combined worktree with all 25 PRs of this batch boarded together: typecheck:core clean, 443/443 node-runner tests plus 14/14 vitest across every test file the batch touches, and check-changelog-integrity, check:cycles (418 files), check:provider-consistency (272 REGISTRY entries, 355 canonical providers), check:docs-counts, check:docs-sync (42 locales) and check-file-size all green.

Thanks @pacocartones.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(resilience): Codex accounts rotate for a single query before settling on the next account

2 participants