[diegosouzapw]

Summary

• A Gemini 429 whose google.rpc.QuotaFailure names a per-day quotaId (e.g. GenerateRequestsPerDayPerProjectPerModel-FreeTier) now locks that model until the daily reset. Before, the ~20s RetryInfo attached to the same body was the only signal used, so an exhausted free-tier key went back into rotation every ~20s until midnight Pacific.
• Per-minute violations keep the short RetryInfo cooldown (fix(resilience): Model Lockouts ignore configured cap and upstream retry hint (freeze until midnight) #7940 behavior unchanged).
• Only the model is locked (Gemini quotas are per project and model). Other models on the same key keep serving. The lock has a finite expiry and is never a terminal state.

Attribution

Thanks to @scottchristian for the original implementation.

Changes

• open-sse/services/retryAfterJson.ts: googleDailyQuotaIds() / hasGoogleDailyQuotaViolation() read error.details[] with a strict google.rpc.QuotaFailure @type and match /perday/i on quotaId only, never on free text. A bounded raw-text fallback (ReDoS-safe) handles a prose message that has the detail appended.
• open-sse/services/googleDailyQuota.ts (new):
  • googleDailyQuotaCooldownMs() uses the operator's node daily-reset clock when one is configured. Otherwise it waits until the next midnight America/Los_Angeles, Google's documented reset for Gemini's requests-per-day quota. The result is capped at 25h.
  • appendGoogleDailyQuotaViolation() adds a compact QuotaFailure line, holding the PerDay quota ids only, to the classification text.
• open-sse/services/accountFallback.ts:
  • checkFallbackError returns quota_exhausted + dailyQuotaExhausted + quotaResetHintMs for these 429s. The check runs before the Gemini metric and credits checks and is keyed on the body shape, so it covers gemini, vertex and AI-Studio-style connections. For per-model providers, markAccountUnavailable turns quotaResetHintMs into a model lockout.
  • parseRetryFromErrorText returns the same daily window instead of 26s.
• open-sse/handlers/chatCore/providerExecutionPipeline.ts (non-streaming) and open-sse/handlers/chatCore/streamingResponse.ts (streaming): classification only ever saw error.message. For Gemini that message reads the same for per-minute and per-day exhaustion. Both paths now carry the compact QuotaFailure line in the classification text. Client-facing error bodies keep only the first line, so their content does not change. I checked this with createErrorResult.
• Design choices:
  • The model lockout goes through the existing markAccountUnavailable path, so the operator's model-lockout maxCooldownMs (30 min by default) still bounds it. That is the same policy the existing daily-quota (isDailyQuotaExhausted) lockouts follow. The connection-level cooldownMs / quotaResetHintMs carry the full daily window.
  • No random multi-hour park for 429s that carry no hint: exponential backoff stays the house policy.
  • rateLimitManager is left as is, because a multi-hour reservoir would queue requests instead of failing over.

Test plan

• node --import tsx/esm --test tests/unit/gemini-daily-quota-violation.test.ts (new, 11 tests): before the wiring, 4 behavior tests failed (parseRetryFromErrorText daily window, checkFallbackError daily park, prose+append path, model-only lockout via markAccountUnavailable). The streaming source-wiring assertion and the new pipeline test also failed with the call sites reverted. After the change: ℹ tests 11 ℹ pass 11 ℹ fail 0.
• Focused suites: gemini-daily-quota-violation, provider-execution-pipeline, bug-7940-gemini-retrydelay, gemini-429-retry-hint-skips-intra-retry, account-fallback-service, account-fallback-retry-after-json → ℹ tests 153 ℹ pass 153 ℹ fail 0.
• All 187 test files that import the touched modules (accountFallback|retryAfterJson|providerExecutionPipeline|streamingResponse|parseRetryFromErrorText|checkFallbackError|dailyQuotaReset) → ℹ tests 1865 ℹ pass 1861 ℹ fail 4. Two of the 4 pass in isolation (load-sensitive). The other two (copilot-learned-catalog-15634 "a listed, supported Copilot model still reaches the HTTP upstream" and provider-request-failure-pipeline "network failure persisted call log…") fail identically with this PR's source changes reverted, so they come from the base.
• npm run typecheck:core → exit 0.
• npx eslint --suppressions-location config/quality/eslint-suppressions.json <changed files> → clean; npx prettier --check <changed files> → clean.
• node scripts/check/check-pr-test-policy.mjs (PR mode) → PASS; GITHUB_BASE_REF=release/v3.8.52 npm run check:test-masking → OK (no weakening).
• npm run check:file-size → no violation in touched files. accountFallback.ts is at 2487 against its 2504 ceiling and streamingResponse.ts at 1308 against 1308. The only reported violations are two untouched test files (base drift).

⚠️ base-red inherited: #15306 — unit/vitest/integration/package-artifact gates on release/v3.8.52

[github-actions]

CI Coverage Report

• Coverage job: skipped
• PR test policy: success

Coverage artifact was not available for this run.