Skip to content

fix(antigravity): skip credits retry on full_quota_exhausted, persist 24h cooldown to DB (#3707) - #3754

Merged
diegosouzapw merged 1 commit into
release/v3.8.24from
fix/3707-quota-skip
Jun 13, 2026
Merged

diegosouzapw merged 1 commit into
release/v3.8.24from
fix/3707-quota-skip

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Closes #3707

Root Causes

Three interconnected bugs, all in the antigravity 429 path:

Bug 1 — decide429() called too late (~41s hang)
The Google One AI credits retry HTTP call was attempted before decide429() was ever consulted. For a quota-exhausted account, this meant a live upstream request was made even though the final verdict was always going to be full_quota_exhausted. On slow/loaded upstream, this caused ~41s request hangs before the account was skipped.

Bug 2 — learned cooldown lost on restart
setConnectionRateLimitUntil() was exported but never called from the antigravity executor. The in-memory providerCooldownTracker cleared on restart, forcing every post-restart request to re-discover exhaustion via a live 429.

Bug 3 — real AG message never classified as quota_exhausted
classify429() in antigravity429Engine only checked for "quota_exhausted" (literal) and "quota exhausted". The actual Antigravity error message — "Individual quota reached. Contact your administrator to enable overages. Resets in Xh." — fell through to unknown, meaning the full_quota_exhausted path never fired for real AG quota hits.

Fix

  • Move decide429() call to before the credits retry check
  • If decision.kind === "full_quota_exhausted" → skip credits retry entirely + call markConnectionQuotaExhausted(accountId, retryAfterMs) to persist the cooldown
  • Add "quota reached", "enable overages", "individual quota" keywords to QUOTA_EXHAUSTED_KEYWORDS

Validation

5 new TDD tests in tests/unit/antigravity-429-quota-cooldown.test.ts, failing before fix → passing after:

  • decide429("quota_exhausted") → full_quota_exhausted (engine contract)
  • AG "Individual quota reached" message → quota_exhausted (new keyword coverage)
  • markConnectionQuotaExhausted persists 24h cooldown; isConnectionRateLimited returns true
  • Expired cooldown does not block the connection

typecheck:core clean (0 errors). 44/44 tests green on affected files.

… 24h cooldown to DB (#3707)

Three bugs fixed together:
1. decide429() was called AFTER the Google One AI credits retry — a quota-exhausted
   account could hang ~41s on the credits HTTP call before the full_quota_exhausted
   verdict was ever computed. Now decide429() runs first; credits retry is skipped
   when kind === full_quota_exhausted.
2. setConnectionRateLimitUntil() was never called from the antigravity executor —
   the 24h cooldown state lived only in memory and was lost on restart, causing
   post-restart requests to re-learn exhaustion the hard way (7,412 upstream 429s
   documented in #3707). markConnectionQuotaExhausted() now persists it to the DB.
3. antigravity429Engine classify429() did not recognise the real Antigravity quota
   message ("Individual quota reached. Contact your administrator to enable overages.")
   — it fell through to the unknown category and never triggered the quota_exhausted
   path. Added "quota reached", "enable overages", "individual quota" to the keywords.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the Antigravity executor by skipping Google One AI credits retry on a full quota exhaustion verdict and persisting the 24-hour cooldown in the database. It also updates the 429 engine to recognize native Antigravity quota-exhausted error messages and adds unit tests. However, the reviewer identified a critical logic bug where calling decide429 before the credits retry check completely disables the credits retry mechanism, making it dead code. The reviewer provided actionable suggestions to refactor the logic and pre-compute whether a credits retry is possible before marking the connection as exhausted.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +1200 to 1216
// 3. Decide final retry time BEFORE the credits retry so that
// full_quota_exhausted can skip the credits attempt entirely
// (avoids ~41s hold on an already-exhausted account) and
// persist the cooldown to DB for post-restart routing.
const decision: Decision = decide429(category, parsedRetryMs);
retryMs = decision.retryAfterMs;
log?.debug?.(
"AG_429",
`Category: ${category}, Decision: ${decision.kind} — ${decision.reason}`
);

if (decision.kind === "full_quota_exhausted" && retryMs) {
markConnectionQuotaExhausted(accountId, retryMs);
}

const creditsAlreadyInjected =
(transformedBody as { enabledCreditTypes?: unknown }).enabledCreditTypes != null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

⚠️ Critical Logic Bug: Google One AI Credits Retry is completely disabled

In this PR, decide429() is called before the credits retry check. However, decide429(category, ...) always returns decision.kind === "full_quota_exhausted" when category is "quota_exhausted" (as verified by the unit test decide429: quota_exhausted category → full_quota_exhausted kind).

Because of this:

  1. decision.kind !== "full_quota_exhausted" will always evaluate to false when category === "quota_exhausted".
  2. As a result, the if block for retrying with credits (lines 1224-1229) is now dead code and will never be executed.
  3. On the very first quota_exhausted error, the connection is immediately marked as fully exhausted in the DB for 24 hours, completely bypassing the Google One AI credits retry mechanism.

Solution

We should only mark the connection as full_quota_exhausted and skip the credits retry if we cannot retry with credits (i.e., credits are already injected, or credits mode is off/not supported). Let's define canRetryWithCredits first, and then use it to determine whether we should mark the connection as exhausted.

              const creditsAlreadyInjected =
                (transformedBody as { enabledCreditTypes?: unknown }).enabledCreditTypes != null;

              const canRetryWithCredits =
                category === "quota_exhausted" &&
                !creditsAlreadyInjected &&
                shouldRetryWithCredits(credentials?.accessToken || "", creditsMode !== "off");

              // 3. Decide final retry time BEFORE the credits retry so that
              //    full_quota_exhausted can skip the credits attempt entirely
              //    (avoids ~41s hold on an already-exhausted account) and
              //    persist the cooldown to DB for post-restart routing.
              const decision: Decision = decide429(category, parsedRetryMs);
              retryMs = decision.retryAfterMs;
              log?.debug?.(
                "AG_429",
                `Category: ${category}, Decision: ${decision.kind} — ${decision.reason}`
              );

              const isFullQuotaExhausted =
                decision.kind === "full_quota_exhausted" && !canRetryWithCredits;

              if (isFullQuotaExhausted && retryMs) {
                markConnectionQuotaExhausted(accountId, retryMs);
              }

Comment on lines 1224 to 1229
if (
category === "quota_exhausted" &&
decision.kind !== "full_quota_exhausted" &&
!creditsAlreadyInjected &&
shouldRetryWithCredits(credentials?.accessToken || "", creditsMode !== "off")
) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Simplify the condition to use the pre-computed canRetryWithCredits flag. This avoids the logical contradiction where decision.kind !== "full_quota_exhausted" was always evaluating to false for quota-exhausted errors.

              if (canRetryWithCredits) {

tkgo11 pushed a commit to tkgo11/OmniRoute that referenced this pull request Sep 23, 2026
… 24h cooldown to DB (cheaper-inference#3707) (cheaper-inference#3754)

Three bugs fixed together:
1. decide429() was called AFTER the Google One AI credits retry — a quota-exhausted
   account could hang ~41s on the credits HTTP call before the full_quota_exhausted
   verdict was ever computed. Now decide429() runs first; credits retry is skipped
   when kind === full_quota_exhausted.
2. setConnectionRateLimitUntil() was never called from the antigravity executor —
   the 24h cooldown state lived only in memory and was lost on restart, causing
   post-restart requests to re-learn exhaustion the hard way (7,412 upstream 429s
   documented in cheaper-inference#3707). markConnectionQuotaExhausted() now persists it to the DB.
3. antigravity429Engine classify429() did not recognise the real Antigravity quota
   message ("Individual quota reached. Contact your administrator to enable overages.")
   — it fell through to the unknown category and never triggered the quota_exhausted
   path. Added "quota reached", "enable overages", "individual quota" to the keywords.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant