[brendandebeasi]

Summary

• Accounts stuck at high backoffLevel (up to 15) after a burst of 429s were permanently deprioritized by the account selector, even after the rate limit window expired
• Add auto-decay: during account selection, reset backoffLevel to 0 for any non-terminal connection whose rateLimitedUntil has passed

Problem

When a provider returns repeated 429s, backoffLevel increments up to maxLevel: 15 and testStatus is set to unavailable. Once rateLimitedUntil expires (max 2 min), the account becomes eligible for selection again — but backoffLevel: 15 persists permanently in the DB.

The account health score is calculated as 100 - (backoffLevel * 10), so a level-15 account scores -50. The account selector always picks healthier accounts first, meaning this account is never selected. The only way to reset backoffLevel is via clearAccountError, which requires a successful request through that account — creating a deadlock:

429 burst → backoffLevel=15 → health score=-50 → never selected → never succeeds → never resets

This is what causes the "Quota exceeded" error in clients even when the upstream provider has plenty of quota available.

Solution

During account selection in getProviderCredentials, before filtering available connections, check each connection:

• If backoffLevel > 0 AND not a terminal status AND rateLimitedUntil has passed (or is null)
• → Reset backoffLevel: 0, testStatus: "active", clear error fields

The DB update is fire-and-forget (.catch(() => {})) to avoid blocking the hot path. The in-memory backoffLevel is also set to 0 immediately so the current request benefits.

Verification

• TypeScript diagnostics: ✅ no errors
• 938 unit tests: ✅ all pass, 0 failures
• Pre-commit hooks (prettier, eslint, docs-sync): ✅ all pass

[gemini-code-assist]

Code Review

This pull request introduces an auto-decay mechanism for account backoff levels in src/sse/services/auth.ts. When a rate limit window expires, the code now resets the backoff level and clears error-related fields to ensure accounts can be re-selected for requests. Feedback suggests refactoring the update logic to avoid duplication with existing functions, explicitly clearing the rateLimitedUntil field, and implementing error logging for the background update operation instead of swallowing exceptions.

[gemini-code-assist]

This update object is very similar to the one used in clearAccountError (lines 859-868), creating code duplication. This could become a maintenance concern if new error-related fields are added in the future. It would be beneficial to refactor this to a shared helper or constant.

I have a couple of suggestions for this block:

1. For consistency with clearAccountError, consider adding rateLimitedUntil: null to the update payload, since the rate limit has passed.
2. Swallowing errors with an empty .catch(() => {}) can hide underlying problems. While this is a fire-and-forget operation, logging the error would be valuable for debugging without blocking the request.

        updateProviderConnection(c.id, {
          backoffLevel: 0,
          testStatus: "active",
          lastError: null,
          lastErrorAt: null,
          lastErrorType: null,
          lastErrorSource: null,
          errorCode: null,
          rateLimitedUntil: null,
        }).catch((err) =>
          log.warn("AUTH", `Failed to auto-decay backoffLevel for ${c.id.slice(0, 8)}`, err)
        );

[kilo-code-bot]

Code Review Summary

Status: Issues Addressed | Recommendation: Address inline feedback before merge

Overview

An existing review from gemini-code-assist[bot] has already identified the key issues with this PR. The inline comment on line 390 is comprehensive and covers:

• Code duplication with clearAccountError function (lines 859-868)
• Missing rateLimitedUntil: null in the update payload
• Silent error swallowing with .catch(() => {}) instead of proper error logging

Recommendation

Address the suggestions in the existing inline comment before merging:

1. Refactor to use a shared helper or constant (to avoid code duplication with clearAccountError)
2. Add rateLimitedUntil: null to the update payload for consistency
3. Replace the empty catch with proper error logging

Files Reviewed (1 file)

• src/sse/services/auth.ts - Logic is correct, addresses the deadlock issue as described in comments

Additional Notes

• The auto-decay logic correctly identifies accounts eligible for backoff reset (non-terminal status, rate limit has passed)
• The fire-and-forget approach (not awaiting DB update) is acceptable for this use case but requires the error logging improvement

[diegosouzapw]

Thanks @brendandebeasi for this excellent bug fix! 🎉 The backoff deadlock was a real issue — accounts would get permanently deprioritized after a rate limit even when the window expired. Your auto-decay logic is clean and well-placed. This will be part of v3.1.5. We really appreciate this contribution! 🚀