Skip to content

fix(provider-limits): clear transient rate-limit state when quota recovers - #6128

Merged
diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.44from
janeza2:fix/quota-cooldown-recovery
Jul 3, 2026
Merged

diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.44from
janeza2:fix/quota-cooldown-recovery

Conversation

@janeza2

@janeza2 janeza2 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Clear stale transient rate-limit state on a provider connection after a successful quota refresh returns usable quota, so priority combos pick the primary connection again without manual Retest.

Problem

When a provider account hits a transient rate limit and recovers later, the provider-limits path fetches live quota and caches it, but does not clear the connection's transient cooldown/error DB fields (testStatus: "unavailable", rateLimitedUntil, lastError*, errorCode, backoffLevel). Priority combos still skip the primary because credential selection (getProviderCredentials) still sees the stale cooldown state, forcing the operator to hit Retest in the dashboard.

This affects every provider that goes through the provider-limits quota refresh path, not just GLM.

Fix

Add a generic recovery step in src/lib/usage/providerLimits.ts after a successful quota fetch:

  • hasUsableQuota(usage) returns true when any quota window has remaining > 0, remainingPercentage > 0, or unlimited: true.
  • maybeClearRecoveredQuotaState(connection, usage) calls clearRecoveredProviderState from src/sse/services/auth.ts only when:
    • quota is usable,
    • the connection is NOT in a terminal status (credits_exhausted, banned, expired, deactivated),
    • the connection currently has transient state worth clearing.
  • Wired into both API-key and OAuth branches of fetchLiveProviderLimitsWithOptions.

Side fix: syncExpiredStatusIfNeeded now returns the updated connection so a recovered-quota path cannot overwrite a freshly written expired terminal status (race between auth-error message and usable quota).

Test plan

  • New tests/unit/provider-limits-recovery.test.ts covers:
    • GLM transient cooldown state cleared on usable quota
    • terminal credits_exhausted status NOT cleared
    • terminal banned status NOT cleared
    • terminal expired status NOT cleared
    • error-only quota response (HTTP 429) does NOT clear transient state
  • ReadLints clean on changed files
  • git diff --check clean
  • pre-push hooks passed (check:any-budget:t11, check:tracked-artifacts)
  • CI test-unit (local env cannot resolve tsx package)

diegosouzapw and others added 2 commits July 3, 2026 02:02
…substring sanitization (cheaper-inference#6048)

Port the release/v3.8.44 fix to main so the code-scanning alert closes on
the default branch. Parse the URL and assert on the exact hostname instead
of a substring match — `includes("www.kimi.com")` would also accept a
hostile host like `www.kimi.com.evil.net` or `evil.net/?x=www.kimi.com`
(js/incomplete-url-substring-sanitization).
…overs

After a successful provider-limits quota refresh returns usable quota
(remaining > 0, remainingPercentage > 0, or unlimited: true), clear stale
transient cooldown/error state on the connection row so priority combos
pick the primary connection again without manual Retest.

The recovery is generic across providers and only fires when:
- usage.quotas has at least one window with usable quota, and
- the connection is NOT in a terminal status (credits_exhausted,
  banned, expired, deactivated), and
- the connection currently has transient state (testStatus unavailable,
  rateLimitedUntil, lastError*, errorCode, or backoffLevel > 0).

Reuses clearRecoveredProviderState from src/sse/services/auth.ts.

syncExpiredStatusIfNeeded now returns the updated connection so a
recovered-quota path cannot overwrite a freshly written "expired"
terminal status.

Tests cover: GLM transient cooldown recovery, terminal-status guard
(credits_exhausted/banned/expired), and error-only quota response
non-recovery.
@janeza2
janeza2 requested a review from diegosouzapw as a code owner July 3, 2026 19:11
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

diegosouzapw and others added 2 commits July 3, 2026 20:26
The live GLM usage path throws on a 429 rather than returning an error
envelope, so fetchAndPersistProviderLimits rejects. Assert the rejection and
keep the transient-state guard — the throw precedes maybeClearRecoveredQuotaState,
so an errored refresh still never clears the connection's cooldown.

Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.44 July 3, 2026 23:28
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Merged — thank you, @janeza2! 🙏 Solid resilience fix: after a successful quota refresh returns usable quota, maybeClearRecoveredQuotaState clears the stale transient rate-limit state (testStatus/rateLimitedUntil/error fields/backoffLevel) so priority combos re-pick the recovered primary connection without a manual Retest — while correctly leaving terminal statuses (credits_exhausted/banned/expired) untouched. I made one small test adjustment: the live GLM usage path throws on a 429 (it doesn't return an error envelope), so the 'error-only' case now asserts the rejection with assert.rejects — the transient-state guard still confirms an errored refresh never clears the cooldown (the throw precedes the clear). 5/5 tests green, lint + typecheck clean. Base redirected main→release/v3.8.44. Ships in the next release.

@diegosouzapw
diegosouzapw merged commit 7160412 into cheaper-inference:release/v3.8.44 Jul 3, 2026
1 of 3 checks passed
diegosouzapw added a commit that referenced this pull request Jul 4, 2026
Residual drift on release tip 7160412 (moving target): eslintWarnings 4270->4279
(+9 as the branch advanced past the prior rebaseline) and testFrozen/frozen file-size
caps for providerLimits.ts (955->982), accountFallback.ts (1790->1864) and
sse-auth.test.ts (1553->1600). All inherited from parallel-session merges (e.g. #6128);
the two production god-files ideally warrant decomposition rather than a bump (tracked
as debt). No production code touched.
diegosouzapw pushed a commit that referenced this pull request Jul 4, 2026
…6139)

Close TOCTOU race in quota recovery clear via CAS primitive (I2 from #6128). Integrated into release/v3.8.44.
@diegosouzapw diegosouzapw mentioned this pull request Jul 4, 2026
tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
…overs (cheaper-inference#6128)

Clear transient rate-limit state when quota recovers. Integrated into release/v3.8.44.
tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
Residual drift on release tip b8a928e (moving target): eslintWarnings 4270->4279
(+9 as the branch advanced past the prior rebaseline) and testFrozen/frozen file-size
caps for providerLimits.ts (955->982), accountFallback.ts (1790->1864) and
sse-auth.test.ts (1553->1600). All inherited from parallel-session merges (e.g. cheaper-inference#6128);
the two production god-files ideally warrant decomposition rather than a bump (tracked
as debt). No production code touched.
tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
…heaper-inference#6139)

Close TOCTOU race in quota recovery clear via CAS primitive (I2 from cheaper-inference#6128). Integrated into release/v3.8.44.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants