Repository navigation
fix(provider-limits): clear transient rate-limit state when quota recovers - #6128
Conversation
…substring sanitization (cheaper-inference#6048) Port the release/v3.8.44 fix to main so the code-scanning alert closes on the default branch. Parse the URL and assert on the exact hostname instead of a substring match — `includes("www.kimi.com")` would also accept a hostile host like `www.kimi.com.evil.net` or `evil.net/?x=www.kimi.com` (js/incomplete-url-substring-sanitization).
…overs After a successful provider-limits quota refresh returns usable quota (remaining > 0, remainingPercentage > 0, or unlimited: true), clear stale transient cooldown/error state on the connection row so priority combos pick the primary connection again without manual Retest. The recovery is generic across providers and only fires when: - usage.quotas has at least one window with usable quota, and - the connection is NOT in a terminal status (credits_exhausted, banned, expired, deactivated), and - the connection currently has transient state (testStatus unavailable, rateLimitedUntil, lastError*, errorCode, or backoffLevel > 0). Reuses clearRecoveredProviderState from src/sse/services/auth.ts. syncExpiredStatusIfNeeded now returns the updated connection so a recovered-quota path cannot overwrite a freshly written "expired" terminal status. Tests cover: GLM transient cooldown recovery, terminal-status guard (credits_exhausted/banned/expired), and error-only quota response non-recovery.
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
…cooldown-recovery
The live GLM usage path throws on a 429 rather than returning an error envelope, so fetchAndPersistProviderLimits rejects. Assert the rejection and keep the transient-state guard — the throw precedes maybeClearRecoveredQuotaState, so an errored refresh still never clears the connection's cooldown. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
|
Merged — thank you, @janeza2! 🙏 Solid resilience fix: after a successful quota refresh returns usable quota, |
7160412
into
cheaper-inference:release/v3.8.44
Residual drift on release tip 7160412 (moving target): eslintWarnings 4270->4279 (+9 as the branch advanced past the prior rebaseline) and testFrozen/frozen file-size caps for providerLimits.ts (955->982), accountFallback.ts (1790->1864) and sse-auth.test.ts (1553->1600). All inherited from parallel-session merges (e.g. #6128); the two production god-files ideally warrant decomposition rather than a bump (tracked as debt). No production code touched.
…overs (cheaper-inference#6128) Clear transient rate-limit state when quota recovers. Integrated into release/v3.8.44.
Residual drift on release tip b8a928e (moving target): eslintWarnings 4270->4279 (+9 as the branch advanced past the prior rebaseline) and testFrozen/frozen file-size caps for providerLimits.ts (955->982), accountFallback.ts (1790->1864) and sse-auth.test.ts (1553->1600). All inherited from parallel-session merges (e.g. cheaper-inference#6128); the two production god-files ideally warrant decomposition rather than a bump (tracked as debt). No production code touched.
…heaper-inference#6139) Close TOCTOU race in quota recovery clear via CAS primitive (I2 from cheaper-inference#6128). Integrated into release/v3.8.44.
Summary
Clear stale transient rate-limit state on a provider connection after a successful quota refresh returns usable quota, so priority combos pick the primary connection again without manual Retest.
Problem
When a provider account hits a transient rate limit and recovers later, the provider-limits path fetches live quota and caches it, but does not clear the connection's transient cooldown/error DB fields (
testStatus: "unavailable",rateLimitedUntil,lastError*,errorCode,backoffLevel). Priority combos still skip the primary because credential selection (getProviderCredentials) still sees the stale cooldown state, forcing the operator to hit Retest in the dashboard.This affects every provider that goes through the provider-limits quota refresh path, not just GLM.
Fix
Add a generic recovery step in
src/lib/usage/providerLimits.tsafter a successful quota fetch:hasUsableQuota(usage)returns true when any quota window hasremaining > 0,remainingPercentage > 0, orunlimited: true.maybeClearRecoveredQuotaState(connection, usage)callsclearRecoveredProviderStatefromsrc/sse/services/auth.tsonly when:credits_exhausted,banned,expired,deactivated),fetchLiveProviderLimitsWithOptions.Side fix:
syncExpiredStatusIfNeedednow returns the updated connection so a recovered-quota path cannot overwrite a freshly writtenexpiredterminal status (race between auth-error message and usable quota).Test plan
tests/unit/provider-limits-recovery.test.tscovers:credits_exhaustedstatus NOT clearedbannedstatus NOT clearedexpiredstatus NOT clearedReadLintsclean on changed filesgit diff --checkcleancheck:any-budget:t11,check:tracked-artifacts)test-unit(local env cannot resolvetsxpackage)