[ggdayup]

What

Adds a payload-aware anti-avalanche guard so a single oversized request cannot avalanche the whole provider pool. On an identified avalanche risk (>=50k prompt tokens or a TPM-class 429) the guard suppresses cross-pool fallback (shouldFallback: false), bounds the per-key transient cooldown via avalancheCooldownMs, and deliberately skips the 1800s model-lockout path so a heavy prompt on one key does not blackhole the model for 30 minutes.

Why

The existing resilience layers (connection cooldown, provider circuit-breaker, model lockout) do not recognise that an oversized payload failing repeatedly across pools is an avalanche rather than a normal 429. Without this, one heavy client can exhaust every pool's headroom at once.

Scope

• new module open-sse/services/accountFallback/antiAvalanche.ts (risk classification + bounded cooldown)
• src/sse/services/auth.ts: options surface (isAvalancheRisk, avalancheCooldownMs, suppressFallback), exactCooldownMs avalanche-first precedence, early return when suppressFallback
• src/sse/handlers/chat.ts: wire the classifier into the request pipeline
• tests/unit/anti-avalanche-failover-protection.test.ts — 12 assertions

Test evidence

node --import tsx/esm --test tests/unit/anti-avalanche-failover-protection.test.ts -> 12/12 pass on this branch.

⚠ base-red inherited: #15306

[diegosouzapw]

Hi @ggdayup — closing this pull request to clear the open queue. The branch fix/payload-aware-anti-avalanche-guard is kept, so the commits stay. Comment here if it should be reopened.

[diegosouzapw]

Hi @ggdayup — reopening this pull request. The close was a mistake, and it is back in the merge queue. The branch fix/payload-aware-anti-avalanche-guard was kept.

[shipsfromrio]

Thanks for this, @ggdayup. Short-circuiting 413 and context-overflow 400s makes sense to me. One concern about the size-only 429 branches, from a gateway I run: our agent traffic sends prompts that routinely sit above the 50k threshold through a pool of Gemini keys, and most of the 429s on those are the daily quota, not TPM. Of the 646 retry hints Gemini put in 429 bodies between Oct 9 and 12, 488 were longer than 1 h, up to 86,399 s (the numbers in #16525).

On the PR head (297d4d5), branch 3c of evaluateAvalancheRisk (antiAvalanche.ts L212-225) returns isAvalancheRisk: true on the first 429 of any prompt of 50k tokens or more, even while it still allows fallback, and 3b (L200-210) does the same at 100k. markAccountUnavailable (auth.ts L3142-3153) then replaces both exactCooldownMs and maxCooldownMs with the 60 s avalancheCooldownMs, ahead of the usedUpstreamRetryHint / quotaResetHintMs values the failure path parsed, and because exactCooldownMs is now a number, recordModelLockoutFailure also skips its until-midnight default for quota_exhausted (accountFallback.ts L857-861 on release/v3.8.52, 9ecddd1). On the release branch today the same 429 keeps what the failure path computed: a daily-quota body classified quota_exhausted (by the Gemini block in checkFallbackError) locks the model for maxCooldownMs (1,800 s by default), and a reset that came from a Retry-After header or a typed RetryInfo locks it for the full stated wait (exactCooldownIsUpstreamReset, auth.ts L3036-3047). With this PR both become 60 s, so a key whose quota resets in 24 h is dispatched again every minute.

The 115,618-token "incident repro" test uses the generic "Resource has been exhausted (e.g. check quota)." text with no retry hint, which classify429 reads as rate_limit. The daily-quota body is the one that names its metric and its retry delay in the message text, and the size of the prompt cannot tell a one-day window from a one-minute one.

Would two small changes work for you when you rebase for 3.8.53?

• pass the failureKind that handleSingleModelChat already computes (L2429 on the PR head, L2525 on the release branch) into evaluateAvalancheRisk, and skip 3b/3c when it is "quota_exhausted". classify429 keeps a 429 whose declared retry window is under 1 h as "rate_limit" (classify429.ts L353-356), so TPM/RPM 429s with short hints, and the generic text of the repro, still get the guard;
• in markAccountUnavailable, apply avalancheCooldownMs only when fallbackResult.usedUpstreamRetryHint !== true and the lockout reason is not quota_exhausted, so header, RetryInfo and prose resets, and the quota default, keep precedence.

A test with a 60k-token prompt and a Gemini body whose message says Please retry in 86399s (or a typed RetryInfo of 86399s), expecting a lockout longer than 60 s, would pin this. One question on 3b: a daily-quota 429 is per key, so a sibling key may still have headroom. Should prompt size alone suppress rotation in that case? And for the generic text, where only the typed details[].retryDelay tells the two apart, would you rather have the chat path read that detail than decide from size? If you would rather keep the guard as it is and handle the daily quota elsewhere, that works for me too.

Two related changes, for context. #16360, merged today, waits out a short transient model lock that would otherwise skip every target before dispatch (maxWaitMs defaults to 90 s), so when the generic text leaves the lock reason at rate_limited, a 60 s lock from this guard is now one the combo waits for and then dispatches into again, instead of moving on. And #16525, opened today, locks the model from the queue side when a typed RetryInfo is longer than the queue budget; since lockModel keeps the longer of two locks, that lock would survive the 60 s relock here. It only covers typed RetryInfo on a model-scoped limiter, though, so Retry-After headers and connection-wide limiters still go through the override above.

One more thing for the rebase: on today's release/v3.8.52 (9ecddd1), src/sse/handlers/chat.ts is at 2,696 lines against a check:file-size ceiling of 2,697 and src/sse/services/auth.ts at 3,587 against 3,587, so the +32 and +18 lines here will trip that gate unless the wiring moves into a module.

Happy to run the rebased branch against the same traffic and report the lockout durations back here.

[diegosouzapw]

Re-homed to release/v3.8.53: v3.8.52 entered its release freeze, so the branch now belongs to the release captain and development continues on the next cycle. Nothing is wrong with this PR — it just needed a live base. No action needed from you; CI will re-run against the new base.