fix(auth): do not park healthy quota accounts as expired by RaviTharuma · Pull Request #12452 · diegosouzapw/OmniRoute

RaviTharuma · 2026-09-02T09:17:58Z

Summary

Healthy accounts with remaining quota were persisted as expired / credits_exhausted and then never retried.

HTTP 401 with a credits/quota body is quota, not UNAUTHORIZED.
A 401 while tokenExpiresAt is still in the future is a cooldown, not expired.
Connection recovery re-probes stale expired rows after 30m (unless invalid_grant / deactivated).
Token health sweep no longer skips credits_exhausted, so OAuth refresh can clear a false no-quota mark.

Real invalid keys (unauthorized on API-key providers) still terminal-expire (#8200 contract).

Related Issues

Validation

Change type: routing
Focused tests
npm run lint (CI)
Production-code changes include automated tests

node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test \
  tests/unit/false-terminal-401-quota.test.ts \
  tests/unit/quota-connection-recovery.test.ts \
  tests/unit/8200-perplexity-web-401-cooldown.test.ts

22 pass, 0 fail.

Tests Added Or Updated

tests/unit/false-terminal-401-quota.test.ts
tests/unit/quota-connection-recovery.test.ts

Coverage Notes

Covers 401 credits body, 401 with a still-valid access token, and expired reprobe selection.

Reviewer Notes

Does not re-enable the credential-health scheduler. Operators who set OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK=true still get recovery via the connection-recovery tick.

Copilot

🟡 Changes recommended

The required npm run test:coverage gate and final coverage result (60/60/60/60 minimum) are not demonstrated in the PR validation notes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes a routing/auth resilience regression where healthy provider connections could be persisted into terminal states (expired / credits_exhausted) after certain 401 responses, causing combo routing to permanently pre-skip them without reattempting.

Changes:

Classify 401 responses with credits/quota bodies as QUOTA_EXHAUSTED (instead of generic UNAUTHORIZED) and persist them as credits_exhausted.
Avoid terminal-parking OAuth connections as expired when the stored token expiry is still in the future (cooldown instead of terminal).
Extend connection-recovery to re-probe stale expired rows after 30 minutes (with a blocklist for true deactivation/invalid-grant cases) and adjust token health sweep behavior to include credits_exhausted.

Commands run (review): none
Tests changed: tests/unit/quota-connection-recovery.test.ts, tests/unit/false-terminal-401-quota.test.ts
Coverage result (custom): not provided / not verified in this review (npm run test:coverage not run)

File summaries

File	Description
`tests/unit/quota-connection-recovery.test.ts`	Adds assertions for the new “expired reprobe” selection behavior and inclusion in recoverable selection.
`tests/unit/false-terminal-401-quota.test.ts`	New regression tests ensuring 401 quota bodies don’t park connections as `expired`, and valid-token 401s don’t terminal-expire OAuth connections.
`src/sse/services/auth.ts`	Updates terminal-status resolution to detect quota signals (including 401 quota cases) and prevents “still-valid token” OAuth 401s from being persisted as `expired`.
`src/lib/tokenHealthCheck.ts`	Stops skipping `credits_exhausted` so OAuth refresh can clear false quota marks; keeps banned/expired as terminal (with existing exceptions).
`src/lib/quota/connectionRecovery.ts`	Adds `expired` reprobe candidate logic (with lastErrorType blocklist) and threads `lastErrorType` through the tick wiring.
`open-sse/services/errorClassifier.ts`	Expands quota classification to include HTTP 401 when the body indicates credits/quota exhaustion.

Review details

Files reviewed: 6/6 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

RaviTharuma · 2026-09-02T10:02:25Z

Conflicts resolved — rebased onto current release/v3.8.51.

Kept the per-model 402 connection-wide guard from #12242 and still classify 401 quota/credits bodies as credits_exhausted (not expired) for non-per-model providers.

RaviTharuma · 2026-09-02T10:22:51Z

Follow-up pushed: src/sse/services/authTerminalStatus.ts extracts the terminal-status helpers so auth.ts stays under the frozen file-size cap (3445 -> 3374). Cursor health-check now expects credits_exhausted rows to be swept, matching the #12452 change.

If No new ESLint warnings still fails on CodeQL ratchet (12 open > baseline 11), that count is repo-wide Security alerts, not this diff.

diegosouzapw · 2026-09-03T11:08:11Z

sweep-reds / babysit: reds on this PR are inherited from base-red #12581, not this 401-quota classification diff. Skipping a code change; did not merge origin/release/v3.8.51.

ESLint: CodeQL ratchet 12 open > baseline 11 (repo-wide Security alerts). No unused/@eloqnt in this diff.
FQG mutation-test-coverage: missing tests/unit/video-bridge-memory-suppression.test.ts on open-sse/handlers/chatCore/memoryExtraction.ts — already listed on the current origin/release/v3.8.51 tip; not a file this PR touches.
Unit Tests 1/4: tests/unit/build/npm-ci-retry-composite.test.ts (fixTlsClientNodeBinary.mjs vanished / wreqJsNative.mjs omitted). Not in this PR's files; already corrected on the release tip.

diegosouzapw · 2026-09-03T12:37:08Z

sweep-reds round 6 / babysit: re-verified — every remaining red is inherited from base-red #12581, not this 401-quota classification diff. Skipping a code change; did not merge origin/release/v3.8.51.

ESLint: eslintWarnings=0; job fails CodeQL ratchet 12 open > baseline 11.
FQG: mutation-test-coverage missing tests/unit/video-bridge-memory-suppression.test.ts covering memoryExtraction.ts — not this PR. Own false-terminal-401-quota.test.ts is already in stryker.conf.json. Complexity new-code was OK (5 files).
Units 1/4: fixTlsClientNodeBinary.mjs vanished / wreqJsNative.mjs cache-key — postinstall helpers this PR does not touch.

STOP. Resume after #12581 drains.

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

…size cap

Shrink tokenHealthCheck comments under the frozen LOC cap, split authTerminalStatus helpers so cyclomatic stays at the base count, and register the new 401-quota unit file in tap.testFiles. Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

The helper moved to authTerminalStatus.ts; the leftover import trips @typescript-eslint/no-unused-vars on parent Quality Gates. Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

@RaviTharuma

…w#12452) Validado em lote numa worktree combinada com os 10 PRs desta leva sobre o tip de `release/v3.8.51`: `typecheck:core` limpo, `check-file-size` OK e **241/242** nos 29 arquivos de teste que os PRs tocam. A única "falha" não é falha: `tests/unit/autoCombo/strict-zero-cost-filter.test.ts` é um teste em estilo Vitest que eu incluí por engano na invocação do runner nativo do Node — ele quebra no import (`@vitest/runner`), não numa asserção. Ao investigar, descobri que esse arquivo não roda em nenhum dos dois runners hoje (o glob do `test:unit` não lista `autoCombo` e o `include` do Vitest só pega `.tsx` nessa pasta); é um problema pré-existente do repositório, sem relação com esta leva, e vou registrá-lo separadamente. O diegosouzapw#12636 conflitava apenas na lista de testes do `@omniroute/opencode-plugin/package.json`, de forma aditiva: o tip já tinha `models-fetcher.test.ts` (do diegosouzapw#12607, irmão desta mesma leva) e o diegosouzapw#12636 acrescenta `telemetry.test.ts`. Fiz a união dos dois lados (25 arquivos contra 24 de cada) em vez de escolher um, o que teria removido um arquivo da suíte do plugin em silêncio. Obrigado, @RaviTharuma.

A non-terminal 401 on an OAuth connection (a still-valid token after a refresh, which diegosouzapw#12452 rightly keeps out of `expired`, or an invalid-token class like diegosouzapw#12594) got cooldown 0: the status_401 rule declares no cooldown, although the diegosouzapw#12452 comment says such a 401 "must cooldown". Every request then re-selected the account, refreshed the token and failed again. Seen in production during a 21-minute upstream 401 burst on two Codex accounts: 273 token refreshes and 216 requests losing ~3 s each before falling back. - Cool such a connection down with exponential backoff: the runtime OAuth base cooldown (5 s by default), doubling per consecutive 401, capped at the existing but unused COOLDOWN_MS.unauthorized (2 min). The streak is kept in memory because the 401 rule does not raise backoffLevel and selection resets it once a cooldown passes; a success (clearAccountError) or a quiet window ends it. testStatus is left untouched so the connection stays refreshable (diegosouzapw#12594), and a 401 that only names an unsupported model is excluded (diegosouzapw#7268). - Do not wait out an auth cooldown in the single-model cooldown-aware retry: it has no known end, so each retry would hit the same 401 and refresh again (a single-account setup would wait ~155 s and refresh 6 times, then fail).

RaviTharuma requested a review from diegosouzapw as a code owner September 2, 2026 09:17

Copilot AI lite review requested due to automatic review settings September 2, 2026 09:17

RaviTharuma mentioned this pull request Sep 2, 2026

[BUG] Healthy accounts with remaining quota are persisted as expired/credits_exhausted and never retried #12451

Closed

Copilot started reviewing on behalf of RaviTharuma September 2, 2026 09:18 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

RaviTharuma force-pushed the fix/false-terminal-quota-from-401 branch from 2f266b4 to 6eb763b Compare September 2, 2026 10:03

cursor Bot force-pushed the fix/false-terminal-quota-from-401 branch from 6eb763b to 1bb648e Compare September 2, 2026 10:20

RaviTharuma mentioned this pull request Sep 2, 2026

[BUG] Successful OAuth refresh and non-auth failures (quota, 403, combo timeout) are classified as authentication expired and skip remaining combo targets #12441

Closed

RaviTharuma and others added 4 commits September 3, 2026 23:25

fix(auth): do not park healthy quota accounts as expired

f6fd5e7

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

fix: extract terminal status helpers so auth.ts stays under the file-…

3dd8524

…size cap

fix(auth): drop unused isCreditsExhausted import from auth.ts

ccf25b9

The helper moved to authTerminalStatus.ts; the leftover import trips @typescript-eslint/no-unused-vars on parent Quality Gates. Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>

RaviTharuma force-pushed the fix/false-terminal-quota-from-401 branch from 04e0625 to ccf25b9 Compare September 3, 2026 23:26

diegosouzapw merged commit 85b8d12 into diegosouzapw:release/v3.8.51 Sep 4, 2026
15 of 16 checks passed

This was referenced Sep 21, 2026

fix(domain): qwen-cloud-token-plan parked as "all accounts exhausted" after a credit-pack top-up #14359

Closed

fix(domain): bound quota park windows and add healthy override (#14359) #14555

Merged

RaviTharuma deleted the fix/false-terminal-quota-from-401 branch September 23, 2026 19:36

QuangBlue mentioned this pull request Sep 27, 2026

fix(auth): back off OAuth connections that keep failing with 401 #14917

Merged

diegosouzapw mentioned this pull request Sep 29, 2026

fix(release): drain the v3.8.51 base-reds after the 2026-09-28/29 merge wave #15052

Merged

Navigation Menu

Uh oh!

fix(auth): do not park healthy quota accounts as expired - #12452

fix(auth): do not park healthy quota accounts as expired#12452
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.51from
RaviTharuma:fix/false-terminal-quota-from-401

RaviTharuma commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

RaviTharuma commented Sep 2, 2026

Uh oh!

RaviTharuma commented Sep 2, 2026

Uh oh!

diegosouzapw commented Sep 3, 2026

Uh oh!

diegosouzapw commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

RaviTharuma commented Sep 2, 2026

Summary

Related Issues

Validation

Tests Added Or Updated

Coverage Notes

Reviewer Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

RaviTharuma commented Sep 2, 2026

Uh oh!

RaviTharuma commented Sep 2, 2026

Uh oh!

diegosouzapw commented Sep 3, 2026

Uh oh!

diegosouzapw commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants