fix(resilience): lock permanently retired models instead of short backoff (Gemini ban prevention)

[turbolego]

Why this happened

We had a Gemini API key get banned after a burst of "spam-looking" traffic. Filtering our 24h request logs for provider: "gemini" showed three repeating patterns:

1. Genuine free-tier 429 quota errors (expected/handled correctly, not the bug):

[429]: You exceeded your current quota, please check your plan and billing details. ...
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests,
  limit: 20, model: gemini-3-flash
Please retry in 20.551466945s.

2. The actual bug — deprecated-model 404s, recurring every ~15–45 minutes, all day:

"model": "gemini-2.5-flash",
"status": 404,
"error": "[404]: This model models/gemini-2.5-flash is no longer available to new users.
          Please update your code to use models/gemini-3.6-flash for the latest features
          and improvements."

"model": "gemini-2.5-flash-lite",
"status": 404,
"error": "[404]: This model models/gemini-2.5-flash-lite is no longer available to new users.
          Please update your code to use models/gemini-3.5-flash-lite for the latest features
          and improvements."

3. Symptom of the above — the local rate-limit queue repeatedly getting congested:

"status": 502,
"error": "[502]: Request dropped: the local rate-limit queue for gemini/gemini-3.1-flash-lite
          was detected as wedged (stalled with nothing executing) and force-reset. This is
          OmniRoute's own queue recovering, not an upstream error."

Root cause

I traced checkFallbackError() in open-sse/services/accountFallback.ts end-to-end. Contrary to my first hypothesis, Gemini's "Please retry in Ns" 429 hint is already parsed and honored correctly (parseRetryFromErrorText + useUpstreamRetryHints, which defaults to true for apikey-category providers) — that part was not the bug.

The real gap: a 404 for a permanently retired model (Google explicitly says "no longer available to new users") had no dedicated classification. It fell through to the generic "all other errors" branch at the bottom of checkFallbackError, which only applies a short transient cooldown (a few seconds, escalating to a ~20 minute cap via BACKOFF_STEPS_MS). Because Gemini is a per-model-quota provider (hasPerModelQuota("gemini") === true), combo/auto-routing kept re-selecting the same dead model roughly every cooldown window — forever, since the model will never succeed. That produced a steady drumbeat of guaranteed-to-fail requests against the account, on top of the genuine 429 bursts on other Gemini model aliases. At volume, this combination looks like abusive/spam traffic to Google and is what got the account flagged and banned.

The fix

In open-sse/services/accountFallback.ts:

• Added MODEL_PERMANENTLY_UNAVAILABLE_PATTERNS — matches phrasing like "no longer available", "no longer supported", "has reached its end of life", "model ... deprecated/retired/discontinued" (also fixes the same class of bug for Fireworks/etc. end-of-life 410s, e.g. minimaxai/minimax-m2.7 seen in the same logs).
• Added isModelPermanentlyUnavailable() to test error text against those patterns.
• checkFallbackError() now classifies a 404/410 matching those patterns with a 24-hour lockout (reason: "not_found"), instead of the short generic transient cooldown. The lockout is surfaced via quotaResetHintMs, which flows into combo's per-request model-lockout (open-sse/services/combo.ts) as an upstream-verified reset — so it is honored in full and is not clamped down to the normal ~20 minute model-lockout ceiling.
• A generic/unrelated 404 (e.g. "Model not found, inaccessible, and/or not deployed") is untouched and still falls through to the existing short-backoff path — this change is narrowly scoped to the specific "permanently retired" phrasing.

How this prevents future bans

• Once Google tells us a model is retired, OmniRoute now stops hammering it for 24h instead of retrying every few minutes forever — cutting the volume of guaranteed-to-fail requests against a Gemini account from "dozens per day, indefinitely" down to effectively zero after the first failure.
• Combined with the free-tier 429 handling (already correct), this removes the main source of sustained abusive-looking traffic against a single Gemini key, which is the behavior Google's anti-abuse systems flag for bans.

Testing

Added tests/unit/gemini-deprecated-model-lockout.test.ts (6 tests, all passing):

• Matches Gemini's deprecated-model phrasing and generic end-of-life 410 phrasing.
• Does not match an unrelated/generic 404.
• checkFallbackError returns a 24h lockout (reason: "not_found", quotaResetHintMs) for both the Gemini 404 and a Fireworks-style end-of-life 410.
• A generic 404 still gets the short transient cooldown (no regression).

Ran locally:

node --import tsx/esm --test tests/unit/gemini-deprecated-model-lockout.test.ts
# 6/6 pass

node --import tsx/esm --test tests/integration/gemini-rate-limit-classification.test.ts \
  tests/unit/account-fallback-service.test.ts tests/unit/error-classification.test.ts
# 133/133 pass, no regressions

Files changed

• open-sse/services/accountFallback.ts — new pattern set + classification branch.
• tests/unit/gemini-deprecated-model-lockout.test.ts — new regression test.