Skip to content

fix(sse): bound Antigravity 429 retry loop and lock quota-exhausted accounts for full reset window - #3122

Merged
diegosouzapw merged 5 commits into
diegosouzapw:release/v3.8.9from
ahmet-cetinkaya:fix/antigravity-429-retry-loop
Jun 3, 2026
Merged

diegosouzapw merged 5 commits into
diegosouzapw:release/v3.8.9from
ahmet-cetinkaya:fix/antigravity-429-retry-loop

Conversation

@ahmet-cetinkaya

@ahmet-cetinkaya ahmet-cetinkaya commented Jun 3, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Motivation and Context

Fixes two critical gaps in Antigravity 429 handling:

  1. Executor retry loop — A persistent 429 on the short-retry branch (retryAfterMs ≤ 60s) looped forever on the same endpoint/account because the branch did urlIndex--; continue without checking the shared retry counter. Production log showed 77 consecutive 429s on one daily endpoint with zero fallback.

  2. Quota-exhausted lockout — After the retry-loop bound, OmniRoute fell over to the next account but re-selected the exhausted one first on every subsequent request (~60s wasted per request). The 429 body "Individual quota reached. Contact your administrator to enable overages. Resets in 164h27m24s." was not recognized as quota exhaustion, so the model was locked for only ~5s instead of the real 6.8-day reset window.

⚙️ Implementation Details

Commit 1 — fix(sse): bound Antigravity short-retry 429 loop per endpoint

  • Gate the short-retry branch on retryAttemptsByUrl[urlIndex] < MAX_AUTO_RETRIES (mirroring the already-bounded sibling)
  • A persistent 429 now retries at most 3× per endpoint across all 3 base URLs, then returns the 429 to account-fallback
  • Regression test: tests/unit/executor-antigravity.test.ts — asserts 12 total attempts (3 × 4) and a returned 429

Commit 2 — fix(sse): lock Antigravity quota-exhausted account for full reset window

  • classify429.ts — add QUOTA_PATTERNS: /individual quota reached/i, /enable overages/i
  • accountFallback.ts — delegate to looksLikeQuotaExhausted() for quota patterns; extend parseRetryFromErrorText() to parse "Resets? in XhYmZs"
  • The exact reset duration reaches recordModelLockoutFailure as exactCooldownMs (uncapped, per user choice)
  • Verified: classify429 → quota_exhausted; parseRetryFromErrorText → 592044000 ms (164h27m24s exactly); checkFallbackError → usedUpstreamRetryHint: true

Mirrors Antigravity-Manager src-tauri/src/proxy/rate_limit.rs set_lockout_until behavior. Patterns are specific — plain rate-limit messages still classify as rate_limit.

Commit 3 — refactor(sse): address review findings from PR #3122

  • Removed overly broad /quota reached/i pattern (retained only /individual quota reached/i)
  • Delegated classifyErrorText() to looksLikeQuotaExhausted() for quota patterns (eliminates duplication)
  • Added 30-day cap to computeDurationMs (prevents adversarial/buggy upstream from locking indefinitely)
  • Flattened parseRetryFromErrorText nested structure for readability
  • Added test for CREDITS_EXHAUSTED_SIGNALS "free tier of the model has been exhausted" entry
  • Added test for 30-day cooldown cap

📋 Checklist for Reviewer

  • Tests passed locally (10768 pass / 0 fail; coverage 79.88/75.18/82.4/79.88)
  • Commit history is clean and descriptive (3 commits, all Conventional Commits)
  • Code quality standards met (typecheck clean, lint 0 errors)
  • Code review findings addressed (5 MAJOR, 3 MINOR fixed in commit 3)

🔗 Related

A persistent 429 on the short-retry branch (retryAfterMs ≤ 60s) looped
forever on the same endpoint because the branch did `urlIndex--; continue`
without checking the shared retry counter. Production log showed 77
consecutive 429s on one daily endpoint/account with zero fallback.

Gate the short-retry branch on `retryAttemptsByUrl[urlIndex] < MAX_AUTO_RETRIES`
(mirroring the already-bounded sibling), so a persistent 429 retries at most
3× per endpoint across all 3 base URLs then returns the 429 to the account-
fallback layer.

Regression test: 'bounds a persistent short-retry 429' in
tests/unit/executor-antigravity.test.ts — asserts 12 total attempts
(3 endpoints × 4) and a returned 429 with zero hang.
After the retry-loop bound, OmniRoute fell over to the next account but
re-selected the exhausted one first on every subsequent request (~60s wasted
per request). Root cause: the 429 body 'Individual quota reached. Contact
your administrator to enable overages. Resets in 164h27m24s.' was not
recognized as quota exhaustion, so the model was locked for only ~5s instead
of the real 6.8-day reset window.

Two detector fixes (mirrors Antigravity-Manager rate_limit.rs set_lockout_until):

1. classify429.ts — add QUOTA_PATTERNS: /individual quota reached/i,
   /quota reached/i, /enable overages/i so looksLikeQuotaExhausted() fires.
2. accountFallback.ts — same patterns in classifyErrorText(); extend
   parseRetryFromErrorText() to parse 'Resets? in XhYmZs' (reusing the
   existing computeDurationMs helper) so the exact reset duration reaches
   recordModelLockoutFailure as exactCooldownMs (uncapped, per user choice).

The lockout machinery already stores until = now + cooldownMs with no clamp,
bypasses getScaledCooldown when exactCooldownMs > 0, and keeps the longer of
existing/new, so the full 164h window flows through intact.

New patterns stay specific — plain 'too many requests'/'rate limit exceeded'
messages still classify as rate_limit.

Verified end-to-end against the real message:
- classify429 → quota_exhausted
- parseRetryFromErrorText → 592044000 ms (164h27m24s exactly)
- checkFallbackError → usedUpstreamRetryHint: true, cooldownMs: 592044000

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces changes to handle Antigravity / Cloud Code quota exhaustion and prevent infinite retry loops. Specifically, it bounds short-retries in AntigravityExecutor using a per-URL attempt counter to avoid infinite loops on persistent 429 errors. It also updates error parsing and classification to recognize Antigravity's 'Resets in XhYmZs' phrasing and quota-related error messages, ensuring they are correctly classified as quota exhaustion with the appropriate long cooldown rather than transient rate limits. Comprehensive unit tests have been added to verify these behaviors. There are no review comments, so I have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

ahmet-cetinkaya added a commit to ahmet-cetinkaya/OmniRoute that referenced this pull request Jun 3, 2026
- Remove /quota reached/i pattern (subsumed by /individual quota reached/i)
- Delegate classifyErrorText to looksLikeQuotaExhausted for quota patterns
- Add 30-day cap to computeDurationMs (prevent adversarial lockouts)
- Flatten parseRetryFromErrorText nested structure
- Add test for CREDITS_EXHAUSTED_SIGNALS 'free tier exhausted' entry
- Add test for 30-day cooldown cap

Addressed MAJOR findings from code review:
- Pattern duplication across classify429.ts and accountFallback.ts
- Missing test coverage for CREDITS_EXHAUSTED_SIGNALS entry
- Overly broad /quota reached/i pattern (false-positive risk)
- Unbounded cooldown injection (30-day cap added)
- Nested if chain flattened for readability
@ahmet-cetinkaya
ahmet-cetinkaya force-pushed the fix/antigravity-429-retry-loop branch from 6c63884 to 2d95845 Compare June 3, 2026 19:47
…afety

- Implement a 30-day cap on parsed retry durations to prevent indefinite account lockouts.
- Replace manual string matching with `looksLikeQuotaExhausted` for more robust quota detection.
- Streamline regex logic in `parseRetryFromErrorText` for better readability.
- Add unit tests for extreme cooldown values and free-tier exhaustion scenarios.
@ahmet-cetinkaya
ahmet-cetinkaya marked this pull request as ready for review June 3, 2026 19:55
@diegosouzapw
diegosouzapw changed the base branch from main to release/v3.8.9 June 3, 2026 21:01
The bare /quota reached/ would also flag transient per-minute limits like
'request quota reached, retry in 60s' as quota_exhausted (multi-hour lock).
The Antigravity message is still caught by /individual quota reached/. Added a
regression assertion proving the transient case stays a rate_limit.
@diegosouzapw

Copy link
Copy Markdown
Owner

Thanks, @ahmet-cetinkaya! 🙏 Two solid fixes: bounding the Antigravity 429 short-retry branch with the per-URL MAX_AUTO_RETRIES guard (no more infinite loop on a persistent 429), and locking quota-exhausted accounts for the full "Resets in XhYmZs" window via model lockout (right resilience layer — never trips the provider breaker). I pushed a small follow-up dropping the broad /quota reached/i pattern (it would also flag transient per-minute limits) — the Antigravity message is still caught by /individual quota reached/i, with a regression assertion added. Merged into release/v3.8.9.

@diegosouzapw
diegosouzapw merged commit 84b5cae into diegosouzapw:release/v3.8.9 Jun 3, 2026
2 checks passed
@kilo-code-bot

kilo-code-bot Bot commented Jun 3, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (4 files)
  • open-sse/executors/antigravity.ts - bounds retry loop, test added
  • open-sse/services/accountFallback.ts - parsing/classification fixes
  • src/shared/utils/classify429.ts - new Antigravity patterns, capped cooldown
  • tests/unit/account-fallback-service.test.ts - regression tests
  • tests/unit/classify429.test.ts - pattern coverage
  • tests/unit/executor-antigravity.test.ts - bounded retry test

Reviewed by step-3.7-flash-20260528 · 708,913 tokens

@diegosouzapw diegosouzapw mentioned this pull request Jun 3, 2026
diegosouzapw added a commit that referenced this pull request Jun 3, 2026
…ors hall

Adds entries for #3097, #3101 (deepseek-web #2942/#2820), #3104, #3105, #3107,
#3109, #3111, #3113, #3115, #3122, #3125, #3127, #3129, plus a Contributors
section crediting all v3.8.9 contributors. Stamps the 3.8.9 release date.
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…ccounts for full reset window (diegosouzapw#3122)

* fix(sse): bound Antigravity short-retry 429 loop per endpoint

A persistent 429 on the short-retry branch (retryAfterMs ≤ 60s) looped
forever on the same endpoint because the branch did `urlIndex--; continue`
without checking the shared retry counter. Production log showed 77
consecutive 429s on one daily endpoint/account with zero fallback.

Gate the short-retry branch on `retryAttemptsByUrl[urlIndex] < MAX_AUTO_RETRIES`
(mirroring the already-bounded sibling), so a persistent 429 retries at most
3× per endpoint across all 3 base URLs then returns the 429 to the account-
fallback layer.

Regression test: 'bounds a persistent short-retry 429' in
tests/unit/executor-antigravity.test.ts — asserts 12 total attempts
(3 endpoints × 4) and a returned 429 with zero hang.

* fix(sse): lock Antigravity quota-exhausted account for full reset window

After the retry-loop bound, OmniRoute fell over to the next account but
re-selected the exhausted one first on every subsequent request (~60s wasted
per request). Root cause: the 429 body 'Individual quota reached. Contact
your administrator to enable overages. Resets in 164h27m24s.' was not
recognized as quota exhaustion, so the model was locked for only ~5s instead
of the real 6.8-day reset window.

Two detector fixes (mirrors Antigravity-Manager rate_limit.rs set_lockout_until):

1. classify429.ts — add QUOTA_PATTERNS: /individual quota reached/i,
   /quota reached/i, /enable overages/i so looksLikeQuotaExhausted() fires.
2. accountFallback.ts — same patterns in classifyErrorText(); extend
   parseRetryFromErrorText() to parse 'Resets? in XhYmZs' (reusing the
   existing computeDurationMs helper) so the exact reset duration reaches
   recordModelLockoutFailure as exactCooldownMs (uncapped, per user choice).

The lockout machinery already stores until = now + cooldownMs with no clamp,
bypasses getScaledCooldown when exactCooldownMs > 0, and keeps the longer of
existing/new, so the full 164h window flows through intact.

New patterns stay specific — plain 'too many requests'/'rate limit exceeded'
messages still classify as rate_limit.

Verified end-to-end against the real message:
- classify429 → quota_exhausted
- parseRetryFromErrorText → 592044000 ms (164h27m24s exactly)
- checkFallbackError → usedUpstreamRetryHint: true, cooldownMs: 592044000

* refactor(account-fallback): simplify error parsing and add cooldown safety

- Implement a 30-day cap on parsed retry durations to prevent indefinite account lockouts.
- Replace manual string matching with `looksLikeQuotaExhausted` for more robust quota detection.
- Streamline regex logic in `parseRetryFromErrorText` for better readability.
- Add unit tests for extreme cooldown values and free-tier exhaustion scenarios.

* fix(429): drop over-broad /quota reached/ pattern, keep specific matches

The bare /quota reached/ would also flag transient per-minute limits like
'request quota reached, retry in 60s' as quota_exhausted (multi-hour lock).
The Antigravity message is still caught by /individual quota reached/. Added a
regression assertion proving the transient case stays a rate_limit.

---------

Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
…ccounts for full reset window (diegosouzapw#3122)

* fix(sse): bound Antigravity short-retry 429 loop per endpoint

A persistent 429 on the short-retry branch (retryAfterMs ≤ 60s) looped
forever on the same endpoint because the branch did `urlIndex--; continue`
without checking the shared retry counter. Production log showed 77
consecutive 429s on one daily endpoint/account with zero fallback.

Gate the short-retry branch on `retryAttemptsByUrl[urlIndex] < MAX_AUTO_RETRIES`
(mirroring the already-bounded sibling), so a persistent 429 retries at most
3× per endpoint across all 3 base URLs then returns the 429 to the account-
fallback layer.

Regression test: 'bounds a persistent short-retry 429' in
tests/unit/executor-antigravity.test.ts — asserts 12 total attempts
(3 endpoints × 4) and a returned 429 with zero hang.

* fix(sse): lock Antigravity quota-exhausted account for full reset window

After the retry-loop bound, OmniRoute fell over to the next account but
re-selected the exhausted one first on every subsequent request (~60s wasted
per request). Root cause: the 429 body 'Individual quota reached. Contact
your administrator to enable overages. Resets in 164h27m24s.' was not
recognized as quota exhaustion, so the model was locked for only ~5s instead
of the real 6.8-day reset window.

Two detector fixes (mirrors Antigravity-Manager rate_limit.rs set_lockout_until):

1. classify429.ts — add QUOTA_PATTERNS: /individual quota reached/i,
   /quota reached/i, /enable overages/i so looksLikeQuotaExhausted() fires.
2. accountFallback.ts — same patterns in classifyErrorText(); extend
   parseRetryFromErrorText() to parse 'Resets? in XhYmZs' (reusing the
   existing computeDurationMs helper) so the exact reset duration reaches
   recordModelLockoutFailure as exactCooldownMs (uncapped, per user choice).

The lockout machinery already stores until = now + cooldownMs with no clamp,
bypasses getScaledCooldown when exactCooldownMs > 0, and keeps the longer of
existing/new, so the full 164h window flows through intact.

New patterns stay specific — plain 'too many requests'/'rate limit exceeded'
messages still classify as rate_limit.

Verified end-to-end against the real message:
- classify429 → quota_exhausted
- parseRetryFromErrorText → 592044000 ms (164h27m24s exactly)
- checkFallbackError → usedUpstreamRetryHint: true, cooldownMs: 592044000

* refactor(account-fallback): simplify error parsing and add cooldown safety

- Implement a 30-day cap on parsed retry durations to prevent indefinite account lockouts.
- Replace manual string matching with `looksLikeQuotaExhausted` for more robust quota detection.
- Streamline regex logic in `parseRetryFromErrorText` for better readability.
- Add unit tests for extreme cooldown values and free-tier exhaustion scenarios.

* fix(429): drop over-broad /quota reached/ pattern, keep specific matches

The bare /quota reached/ would also flag transient per-minute limits like
'request quota reached, retry in 60s' as quota_exhausted (multi-hour lock).
The Antigravity message is still caught by /individual quota reached/. Added a
regression assertion proving the transient case stays a rate_limit.

---------

Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ccounts for full reset window (diegosouzapw#3122)

* fix(sse): bound Antigravity short-retry 429 loop per endpoint

A persistent 429 on the short-retry branch (retryAfterMs ≤ 60s) looped
forever on the same endpoint because the branch did `urlIndex--; continue`
without checking the shared retry counter. Production log showed 77
consecutive 429s on one daily endpoint/account with zero fallback.

Gate the short-retry branch on `retryAttemptsByUrl[urlIndex] < MAX_AUTO_RETRIES`
(mirroring the already-bounded sibling), so a persistent 429 retries at most
3× per endpoint across all 3 base URLs then returns the 429 to the account-
fallback layer.

Regression test: 'bounds a persistent short-retry 429' in
tests/unit/executor-antigravity.test.ts — asserts 12 total attempts
(3 endpoints × 4) and a returned 429 with zero hang.

* fix(sse): lock Antigravity quota-exhausted account for full reset window

After the retry-loop bound, OmniRoute fell over to the next account but
re-selected the exhausted one first on every subsequent request (~60s wasted
per request). Root cause: the 429 body 'Individual quota reached. Contact
your administrator to enable overages. Resets in 164h27m24s.' was not
recognized as quota exhaustion, so the model was locked for only ~5s instead
of the real 6.8-day reset window.

Two detector fixes (mirrors Antigravity-Manager rate_limit.rs set_lockout_until):

1. classify429.ts — add QUOTA_PATTERNS: /individual quota reached/i,
   /quota reached/i, /enable overages/i so looksLikeQuotaExhausted() fires.
2. accountFallback.ts — same patterns in classifyErrorText(); extend
   parseRetryFromErrorText() to parse 'Resets? in XhYmZs' (reusing the
   existing computeDurationMs helper) so the exact reset duration reaches
   recordModelLockoutFailure as exactCooldownMs (uncapped, per user choice).

The lockout machinery already stores until = now + cooldownMs with no clamp,
bypasses getScaledCooldown when exactCooldownMs > 0, and keeps the longer of
existing/new, so the full 164h window flows through intact.

New patterns stay specific — plain 'too many requests'/'rate limit exceeded'
messages still classify as rate_limit.

Verified end-to-end against the real message:
- classify429 → quota_exhausted
- parseRetryFromErrorText → 592044000 ms (164h27m24s exactly)
- checkFallbackError → usedUpstreamRetryHint: true, cooldownMs: 592044000

* refactor(account-fallback): simplify error parsing and add cooldown safety

- Implement a 30-day cap on parsed retry durations to prevent indefinite account lockouts.
- Replace manual string matching with `looksLikeQuotaExhausted` for more robust quota detection.
- Streamline regex logic in `parseRetryFromErrorText` for better readability.
- Add unit tests for extreme cooldown values and free-tier exhaustion scenarios.

* fix(429): drop over-broad /quota reached/ pattern, keep specific matches

The bare /quota reached/ would also flag transient per-minute limits like
'request quota reached, retry in 60s' as quota_exhausted (multi-hour lock).
The Antigravity message is still caught by /individual quota reached/. Added a
regression assertion proving the transient case stays a rate_limit.

---------

Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants