feat(resilience): payload-aware anti-avalanche failover guard

.env.example

@@ -2143,6 +2143,16 @@ CURSOR_USER_AGENT="Cursor/3.4"
 # Retry-After. Used by: open-sse/services/apiKeyRotator.ts (#14573). Default 60000.
 # OMNIROUTE_API_KEY_COOLDOWN_MS=60000
 
+# ── Anti-avalanche failover guard (opt-in, off by default) ──
+# Used by: open-sse/services/accountFallback/antiAvalanche.ts → src/sse/handlers/chat.ts.
+# Only an explicit "true" enables it. When on, a 429 that names a tokens-per-minute limit
+# (or a context-overflow 400) stops the cascading failover across the remaining keys of the
+# provider. A quota-exhausted 429 keeps failing over however large the prompt is, 413 keeps
+# the #13350 retry-on-another-account behaviour, and the burst cooldown never shortens a
+# longer upstream reset.
+# OMNIROUTE_AVALANCHE_PROTECTION_ENABLED=false
+# OMNIROUTE_AVALANCHE_BURST_COOLDOWN_MS=60000
+
 # ── Provider-level circuit breaker thresholds and cooldowns ──
 # Used by: open-sse/config/constants.ts (PROVIDER_PROFILES → accountFallback).
 # These control the provider-level fuse (entire provider cooldown after repeated

changelog.d/features/15775-payload-aware-anti-avalanche-guard.md

@@ -0,0 +1 @@
+- **feat(resilience):** opt-in anti-avalanche failover guard (`OMNIROUTE_AVALANCHE_PROTECTION_ENABLED=true`, off by default) — when enabled, a 429 that names a tokens-per-minute limit no longer cascades over the provider's remaining keys and the model lockout uses a short cooldown instead of escalating, while a longer upstream reset always wins; quota-exhausted 429s keep failing over however large the prompt is and HTTP 413 keeps its retry-on-another-account behaviour ([#15775](https://github.com/cheaper-inference/OmniRoute/pull/15775) — thanks @ggdayup)

docs/reference/ENVIRONMENT.md

@@ -986,6 +986,8 @@ Provider-level circuit breaker tuning. Defaults reflect the scaled values used s
 | `OMNIROUTE_PROVIDER_BREAKER_LOCAL_FAILURE_WINDOW_MS`          | `300000`  | `open-sse/config/constants.ts`       | Provider-level breaker: rolling failure-count window (ms) for local providers.                                                                                                   |
 | `OMNIROUTE_PROVIDER_BREAKER_LOCAL_COOLDOWN_MS`                | `60000`   | `open-sse/config/constants.ts`       | Provider-level breaker: cooldown (ms) once the local provider threshold is reached.                                                                                              |
 | `OMNIROUTE_API_KEY_COOLDOWN_MS`                               | `60000`   | `open-sse/services/apiKeyRotator.ts` | Per-key cooldown (ms) applied by multi-key API-key rotation when an upstream 429 carries no `Retry-After` (#14573). Non-positive or non-numeric values fall back to the default. |
+| `OMNIROUTE_AVALANCHE_PROTECTION_ENABLED` | `false` | `open-sse/services/accountFallback/antiAvalanche.ts` | Opt-in anti-avalanche guard: only an explicit `true` enables it. When on, a 429 that names a tokens-per-minute limit (and a context-overflow 400) stops the cascading failover across the provider's remaining keys. A quota-exhausted 429 keeps failing over however large the prompt is, and HTTP 413 is not affected (it keeps the #13350 retry-on-another-account behaviour). |
+| `OMNIROUTE_AVALANCHE_BURST_COOLDOWN_MS` | `60000` | `open-sse/services/accountFallback/antiAvalanche.ts` | Model-lockout cooldown (ms) applied when the guard fires, instead of the escalating lockout. A longer upstream reset always wins. Non-positive or non-numeric values fall back to the default. |
 | `PIN_DROP_BACKOFF_LEVEL`                                      | `2`       | `open-sse/services/combo.ts`         | Backoff depth at which a context-cache pin's provider is deemed durably unhealthy and the pin is dropped for failover.                                                           |
 | `PIN_DROP_GRACE_MS`                                           | `20000`   | `open-sse/services/combo.ts`         | Anti-flap window (ms) tolerating brief transient cooldowns before dropping a context-cache pin.                                                                                  |
 

open-sse/services/accountFallback/antiAvalanche.ts

@@ -0,0 +1,205 @@
+/**
+ * open-sse/services/accountFallback/antiAvalanche.ts
+ *
+ * Anti-Avalanche Guard: opt-in cascading-failover protection
+ * (OMNIROUTE_AVALANCHE_PROTECTION_ENABLED=true, off by default).
+ *
+ * Prevents domino exhaustion across multiple credentials when an upstream
+ * rejection comes from a limit every sibling key would hit too:
+ *  - an HTTP 429 that names a tokens-per-minute limit (e.g. Gemini
+ *    generate_content_free_tier_input_token_count, or an OpenAI/Anthropic TPM
+ *    limit). This stops 1 oversized request from knocking down all N keys and
+ *    triggering escalated 1800s model lockouts on healthy accounts;
+ *  - an HTTP 400 context-overflow rejection.
+ *
+ * Prompt size alone never suppresses failover: a quota-exhausted 429 with a large
+ * prompt says nothing about the sibling accounts, so it fails over as usual.
+ *
+ * HTTP 413 is deliberately NOT handled here: accountFallback.ts treats
+ * PAYLOAD_TOO_LARGE as a retryable fallback (#13350), so another account or
+ * provider may still accept the payload.
+ *
+ * Tuning is env-only (OMNIROUTE_AVALANCHE_*, see docs/reference/ENVIRONMENT.md).
+ */
+
+import { classifyGeminiQuotaMetricFromText } from "../geminiRateLimitTracker.ts";
+import { isContextOverflow400 } from "../combo/comboPredicates.ts";
+import { isSubscriptionQuotaText } from "../quotaTextCooldowns.ts";
+
+export const DEFAULT_AVALANCHE_BURST_COOLDOWN_MS = 60_000;
+
+const EXPLICIT_TPM_PATTERNS = [
+  /free_tier_input_token_count/i,
+  /\bgenerate_content_.*_token_count\b/i,
+  /\b(?:tpm|tokens?\s*per\s*minute)\b/i,
+  /\btoken\s*rate\s*limit\b/i,
+  /\brate_limit_exceeded.*tokens?\b/i,
+];
+
+// A window longer than a minute is a quota: sibling accounts keep their own, so it must fail over
+// even when the body also carries a token wording (OpenAI "tokens per day", Gemini "...PerDay").
+const LONG_WINDOW_QUOTA_PATTERNS = [
+  /\b(?:tpd|rpd)\b/i,
+  /per[\s_-]*(?:hour|day|week|month)/i,
+  /\b(?:hourly|daily|weekly|monthly)\b/i,
+  /\binsufficient_quota\b/i,
+];
+
+export type AvalancheFailureReason = "explicit_tpm_limit" | "context_length_exceeded";
+
+export interface AvalancheEvaluationContext {
+  status: number;
+  errorText: string | null | undefined;
+  provider?: string | null;
+  model?: string | null;
+  promptTokens?: number | null;
+}
+
+export interface AvalancheRiskResult {
+  isAvalancheRisk: boolean;
+  reason?: AvalancheFailureReason;
+  shouldSuppressFallback: boolean;
+  suggestedCooldownMs: number;
+  promptTokens: number;
+  explanation?: string;
+}
+
+export interface AvalancheSettings {
+  enabled: boolean;
+  burstCooldownMs: number;
+}
+
+function readPositiveNumber(raw: string | undefined, fallback: number): number {
+  if (raw === undefined || raw.trim() === "") return fallback;
+  const parsed = Number(raw);
+  return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+export function resolveAvalancheSettings(): AvalancheSettings {
+  return {
+    // Opt-in: it changes failover semantics, so only an explicit "true" enables it.
+    enabled: process.env.OMNIROUTE_AVALANCHE_PROTECTION_ENABLED === "true",
+    burstCooldownMs: readPositiveNumber(
+      process.env.OMNIROUTE_AVALANCHE_BURST_COOLDOWN_MS,
+      DEFAULT_AVALANCHE_BURST_COOLDOWN_MS
+    ),
+  };
+}
+
+/**
+ * True only when the 429 itself names a tokens-per-minute limit. Local usage counters do not
+ * count: they are per model across every key, so they cannot describe this key's rejection.
+ */
+export function isExplicitTokenRateLimitError(
+  status: number,
+  errorText: string | null | undefined,
+  provider?: string | null
+): boolean {
+  if (status !== 429) return false;
+  const text = String(errorText || "");
+
+  if (
+    isSubscriptionQuotaText(text.toLowerCase(), provider) ||
+    LONG_WINDOW_QUOTA_PATTERNS.some((pattern) => pattern.test(text))
+  ) {
+    return false;
+  }
+
+  if (provider === "gemini") {
+    const metric = classifyGeminiQuotaMetricFromText(text);
+    if (metric) return metric === "tpm";
+  }
+
+  return EXPLICIT_TPM_PATTERNS.some((pattern) => pattern.test(text));
+}
+
+function nonNegativeOrZero(value: number | null | undefined): number {
+  return typeof value === "number" && value >= 0 ? value : 0;
+}
+
+function noAvalancheRisk(promptTokens: number): AvalancheRiskResult {
+  return {
+    isAvalancheRisk: false,
+    shouldSuppressFallback: false,
+    suggestedCooldownMs: 0,
+    promptTokens,
+  };
+}
+
+export function evaluateAvalancheRisk(ctx: AvalancheEvaluationContext): AvalancheRiskResult {
+  const settings = resolveAvalancheSettings();
+  const promptTokens = nonNegativeOrZero(ctx.promptTokens);
+  if (!settings.enabled) return noAvalancheRisk(promptTokens);
+
+  const label = `${ctx.provider || "unknown"}/${ctx.model || "unknown"}`;
+
+  // 1. HTTP 400: Context length / overflow
+  if (ctx.status === 400 && isContextOverflow400(ctx.errorText)) {
+    return {
+      isAvalancheRisk: true,
+      reason: "context_length_exceeded",
+      shouldSuppressFallback: true,
+      suggestedCooldownMs: 0,
+      promptTokens,
+      explanation: `[OmniRoute Anti-Avalanche] Context length exceeded for ${label}. Cascading fallback across same-model accounts suppressed.`,
+    };
+  }
+
+  // 2. HTTP 429 naming a tokens-per-minute limit
+  if (isExplicitTokenRateLimitError(ctx.status, ctx.errorText, ctx.provider)) {
+    return {
+      isAvalancheRisk: true,
+      reason: "explicit_tpm_limit",
+      shouldSuppressFallback: true,
+      suggestedCooldownMs: settings.burstCooldownMs,
+      promptTokens,
+      explanation: `[OmniRoute Anti-Avalanche] Token rate limit (TPM/token count) hit for ${label}. Cascading fallback suppressed across remaining keys to protect pool.`,
+    };
+  }
+
+  return noAvalancheRisk(promptTokens);
+}
+
+/** The guard's burst cooldown is a floor: it never shortens an upstream reset. */
+export function resolveAvalancheCooldownMs(
+  avalancheCooldownMs: number | null | undefined,
+  upstreamResetMs: number | null | undefined
+): number | null {
+  if (typeof avalancheCooldownMs !== "number") return upstreamResetMs ?? null;
+  return Math.max(avalancheCooldownMs, upstreamResetMs ?? 0);
+}
+
+export function formatAntiAvalancheMessage(
+  provider: string,
+  model: string,
+  promptTokens: number,
+  reason: string,
+  originalError?: string
+): string {
+  const base = `[OmniRoute Anti-Avalanche] Request to ${provider}/${model} (~${promptTokens.toLocaleString()} tokens) triggered a payload-induced rate limit (${reason}). Cascading fallback across remaining credentials was suppressed to preserve pool availability.`;
+  const suggestion =
+    provider === "gemini"
+      ? ` Tip: Large context requests (>50k tokens) with deep agent history should be routed to 'agy/*' (Antigravity OAuth) which has higher burst token limits, or wait 60s for the TPM sliding window to reset.`
+      : ` Tip: Consider reducing context size or waiting for rate limit window to clear.`;
+  const upstream = originalError ? ` Upstream error: ${originalError}` : "";
+  return `${base}${suggestion}${upstream}`;
+}
+
+/**
+ * Tags a suppressed failover on the client-facing response. Best-effort: the headers of
+ * some Response objects (e.g. Response.error() or a response built by a fetch wrapper) are
+ * immutable and .set() throws, which must never break the failover/error path.
+ */
+export function applyAvalancheSuppressionHeaders(
+  response: Response | null | undefined,
+  reason?: AvalancheFailureReason
+): void {
+  const headers = response?.headers;
+  if (!headers) return;
+  try {
+    headers.set("x-omniroute-anti-avalanche", "suppressed");
+    if (reason) headers.set("x-omniroute-avalanche-reason", reason);
+  } catch {
+    // immutable headers: the suppression is still enforced, only the debug tag is dropped.
+  }
+}

src/sse/handlers/chat.ts

@@ -28,6 +28,12 @@ import {
   recordModelLockoutFailure,
   isDailyQuotaExhausted,
 } from "@omniroute/open-sse/services/accountFallback.ts";
+import {
+  applyAvalancheSuppressionHeaders,
+  evaluateAvalancheRisk,
+  resolveAvalancheSettings,
+} from "@omniroute/open-sse/services/accountFallback/antiAvalanche.ts";
+import { estimateTokens } from "@omniroute/open-sse/services/contextManager.ts";
 import { getCombo, getComboForModel, getModelInfo } from "../services/model";
 import { stripContextWindowSuffix } from "@omniroute/open-sse/services/model.ts";
 import { resolveBareModelToConnectionDefault } from "@omniroute/open-sse/services/model.ts";
@@ -2727,6 +2733,18 @@ async function handleSingleModelChat(
       const skipConnectionDisable =
         isRefusalReplay || shouldSkipConnDisable(result, is401, hasExtraKeys, provider);
 
+      // Anti-Avalanche Guard (opt-in): detect TPM / context-overflow rejections to suppress domino failover
+      const avalancheRisk =
+        (result.status === 429 || result.status === 400) && resolveAvalancheSettings().enabled
+          ? evaluateAvalancheRisk({
+              status: result.status,
+              errorText: errorStr,
+              provider,
+              model: effectiveModel,
+              promptTokens: estimateTokens(requestBody?.messages ?? requestBody),
+            })
+          : null;
+
       const { shouldFallback, cooldownMs } = skipConnectionDisable
         ? { shouldFallback: false, cooldownMs: 0 }
         : await markAccountUnavailable(
@@ -2744,10 +2762,21 @@ async function handleSingleModelChat(
               ),
               isCombo,
               headers: result.response.headers,
+              isAvalancheRisk: avalancheRisk?.isAvalancheRisk ?? false,
+              avalancheCooldownMs: avalancheRisk?.suggestedCooldownMs,
+              suppressFallback: avalancheRisk?.shouldSuppressFallback ?? false,
               structuredError: { code: result.errorCode, type: result.errorType },
             })
           );
 
+      if (avalancheRisk?.shouldSuppressFallback) {
+        log.warn(
+          "ANTI_AVALANCHE",
+          `${provider}/${model} cascading failover suppressed (${avalancheRisk.reason}, ~${avalancheRisk.promptTokens} tokens, attempt ${excludedConnectionIds.size + 1}) to prevent key exhaustion`
+        );
+        applyAvalancheSuppressionHeaders(result.response, avalancheRisk.reason);
+      }
+
       // An explicit pin (combo step `connectionId` / `x-omniroute-connection`) is an
       // operator instruction, not a suggestion: the account cooldown above is still
       // recorded, but selection must NOT silently rotate to a sibling account of the

src/sse/services/auth.ts

@@ -106,6 +106,7 @@ import {
 } from "@omniroute/open-sse/services/accountFallback.ts";
 import { copilotRejectionFlag, lockCopilotModelNotSupported } from "./copilotModelNotSupportedLock";
 import { isSharedWalletCredits402 } from "@omniroute/open-sse/services/accountFallback/sharedWalletCredits.ts";
+import { resolveAvalancheCooldownMs } from "@omniroute/open-sse/services/accountFallback/antiAvalanche.ts";
 import { isModelScopedAccessDenial } from "@omniroute/open-sse/services/accountFallback/modelScopedAccessDenial.ts";
 import { isModelUnsupported401 } from "@omniroute/open-sse/services/accountFallback/perModelFailureScope.ts";
 import { postOutputFailureReachesLockout } from "@omniroute/open-sse/services/accountFallback/postOutputFailureStreak.ts";
@@ -2332,6 +2333,16 @@ function isStreamContentStall(errorText: string): boolean {
   return /stream content stall/i.test(String(errorText || ""));
 }
 
+/** The reset an upstream announced for this failure, as opposed to a locally computed backoff. */
+function resolveUpstreamResetMs(fallback: {
+  usedUpstreamRetryHint?: boolean;
+  cooldownMs?: number;
+  quotaResetHintMs?: number | null;
+}): number | null {
+  if (fallback.usedUpstreamRetryHint === true) return fallback.cooldownMs ?? null;
+  return fallback.quotaResetHintMs ?? null;
+}
+
 export async function markAccountUnavailable(
   connectionId: string,
   status: number,
@@ -2346,6 +2357,9 @@ export async function markAccountUnavailable(
     headers?: Headers | Record<string, string> | null;
     correlationId?: string | null;
     streamOutputEmitted?: boolean;
+    isAvalancheRisk?: boolean;
+    avalancheCooldownMs?: number;
+    suppressFallback?: boolean;
     structuredError?: { code?: string | null; type?: string | null } | null;
   } = {}
 ) {
@@ -2801,6 +2815,12 @@ export async function markAccountUnavailable(
         status === 429
           ? ANTIGRAVITY_FAMILY_INFERRED_BASE_COOLDOWN_MS
           : null;
+      const upstreamCooldownMs = isModelScopedClaudeQuota
+        ? claudeQuotaScope.cooldownMs
+        : resolveUpstreamResetMs(fallbackResult);
+      const exactCooldownMs = options.isAvalancheRisk
+        ? resolveAvalancheCooldownMs(options.avalancheCooldownMs, upstreamCooldownMs)
+        : upstreamCooldownMs;
       const lockout = recordModelLockoutFailure(
         provider,
         connectionId,
@@ -2817,18 +2837,16 @@ export async function markAccountUnavailable(
         effectiveProviderProfile,
         {
           ...modelLockoutOptions,
-          exactCooldownMs: isModelScopedClaudeQuota
-            ? claudeQuotaScope.cooldownMs
-            : fallbackResult.usedUpstreamRetryHint === true
-              ? fallbackResult.cooldownMs
-              : (fallbackResult.quotaResetHintMs ?? null),
+          exactCooldownMs,
           maxCooldownMs: mlSettings.maxCooldownMs,
           scope: usesExactAntigravityLock ? "exact" : undefined,
           // Only a transport header, google.rpc.RetryInfo, or cached Claude reset can bypass
-          // maxCooldownMs. Prose and generic JSON hints remain exact but operator-capped.
+          // maxCooldownMs. Prose and generic JSON hints remain exact but operator-capped, and
+          // so does the avalanche burst cooldown when it outlasts the upstream reset.
           exactCooldownIsUpstreamReset:
-            retryHintBypassesMaxCooldownMs(fallbackResult.retryHintSource) ||
-            isModelScopedClaudeQuota,
+            exactCooldownMs === upstreamCooldownMs &&
+            (retryHintBypassesMaxCooldownMs(fallbackResult.retryHintSource) ||
+              isModelScopedClaudeQuota),
         }
       );
       // Update last error for observability (without changing terminal status)
@@ -2849,7 +2867,10 @@ export async function markAccountUnavailable(
         cooldownMs: lockout.cooldownMs,
         reason,
       });
-      return { shouldFallback: true, cooldownMs: lockout.cooldownMs };
+      return {
+        shouldFallback: options.suppressFallback ? false : true,
+        cooldownMs: lockout.cooldownMs,
+      };
     }
     const result = fallbackResult;
     if (isSharedWalletCredits402(provider, status, errorText)) {
@@ -2859,6 +2880,16 @@ export async function markAccountUnavailable(
     }
     const { shouldFallback, cooldownMs: rawCooldownMs, newBackoffLevel, reason } = result;
     if (!shouldFallback) return { shouldFallback: false, cooldownMs: 0 };
+    if (options.suppressFallback) {
+      const suppressedCooldownMs =
+        resolveAvalancheCooldownMs(options.avalancheCooldownMs, resolveUpstreamResetMs(result)) ??
+        rawCooldownMs;
+      // This path skips the connection cooldown below, so the lock is what keeps the key out.
+      if (provider && suppressedCooldownMs > 0 && !disableCooling) {
+        lockModel(provider, connectionId, model, reason || "unknown", suppressedCooldownMs);
+      }
+      return { shouldFallback: false, cooldownMs: suppressedCooldownMs };
+    }
     const providerErrorType = classifyProviderError(status, errorText, provider);
 
     if (

stryker.conf.json

@@ -148,6 +148,8 @@
       "tests/unit/anthropic-thinking-signature-recovery.test.ts",
       "tests/unit/anthropic-request-not-allowed-cooldown-escalation.test.ts",
       "tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts",
+      "tests/unit/anti-avalanche-chat-wiring.test.ts",
+      "tests/unit/anti-avalanche-failover-protection.test.ts",
       "tests/unit/grok-request-refusal-not-a-ban.test.ts",
       "tests/unit/agy-family-not-connection-cooldown.test.ts",
       "tests/unit/antigravity-family-cooldown-reader.test.ts",