fix(resilience): recover idle-capacity limiter wedges early

changelog.d/fixes/9041-rate-limit-idle-capacity-wedge.md

@@ -0,0 +1 @@
+- **fix(resilience):** Detect and reset idle-capacity rate-limit queue wedges on an eligible watchdog scan so routing can fall back promptly ([#9041](https://github.com/diegosouzapw/OmniRoute/pull/9041))

config/quality/file-size-baseline.json

@@ -577,4 +577,4 @@
   "_rebaseline_2026_08_07_9653_disconnect_grace_period": "Extracted fix(sse): grace period before finalizing a client disconnect as 499 (#9653) \u2014 a client that closes its connection right after reading a fully-completed SSE stream can race OmniRoute's own completion bookkeeping, getting persisted as a false 499/0-tokens even though it delivered the full response (live-confirmed: a real disconnect at 18236ms was corrected to 200/82814+1292 tokens). Own growth: open-sse/handlers/chatCore.ts 5030->5039 (+9, wiring createClientDisconnectGraceHandler at the existing onClientDisconnectFinalize call site) \u2014 irreducible call-site wiring, the actual grace-period logic lives in the new leaf createClientDisconnectGraceHandler (open-sse/utils/streamFailureFinalization.ts, not frozen). Re-measured to 5042 after rebasing onto a newer release/v3.8.50 tip: the file carries an unrelated +3 base drift from already-merged upstream commits between this PR's original branch point and the rebase target, not covered by this entry. Covered by tests/unit/stream-disconnect-grace-period-9653.test.ts (4/4, fake-timer driven). Other file-size gate violations present on this base tip are pre-existing/unrelated to this change (base-red #9679, re-verify current issue number at merge time).",
   "_rebaseline_2026_08_04_9268_gemini_schema_empty_choices": "Feature #9268 own growth: open-sse/utils/stream.ts 2889->2915 (+26 = irreducible call-site wiring for the empty-choices interceptor). The translate-mode flush now rejects a stream that completed without forwarding any valuable chunk (all-empty `choices: []`, no content/tool_calls/finish_reason) as a retryable 502 \"empty content\" instead of a clean empty 200 \u2014 the missing streaming counterpart of chatCore.ts's non-streaming isEmptyContentResponse. All rejection logic lives in the NEW leaf module open-sse/utils/streamEmptyChoices.ts (<cap, not frozen, unit-tested via tests/unit/stream-empty-choices-interceptor.test.ts); stream.ts only carries the `forwardedValuableChunk` boolean (declared at createSSEStream scope, set in emitTranslatedClientItem where the sole hasValuableContent check passes) plus the one flush-time rejectEmptyChoicesStream() call \u2014 the wait/orchestration at the chokepoint, not a movable block (mirrors the comboCooldownRetry.ts precedent). Schema-side twin fix: recursive type:\"object\" injection in open-sse/translator/helpers/geminiHelper.ts (not frozen, +33) for nested schemas with properties but no type (Gemini 400).",
   "_rebaseline_2026_08_09_5696_capability_filter": "PR #9424 own growth: open-sse/handlers/chatCore.ts 5050->5061 (+11). The Layer A capability gate is irreducible wiring at the existing pre-dispatch chokepoint: feature-flag check, capability derivation, compatibility decision, sanitized 400 response, pending-request cleanup, and warning telemetry. All matching and message logic lives outside the god-file in src/shared/constants/capabilities/capabilityFilter.ts; only orchestration remains here. Covered by tests/unit/capability-filter.test.ts (20 cases, including flag-off and sanitized error behavior). Structural shrink remains tracked separately."
-}
+}

docs/architecture/RESILIENCE_GUIDE.md

@@ -224,12 +224,16 @@ rate limit. Bounded by `comboCooldownWait` (`enabled`, `maxWaitMs`, `maxAttempts
 **Scope**: the local per-provider+connection rate-limit queue (`open-sse/services/rateLimitManager.ts`,
 backed by Bottleneck), one layer below the three mechanisms above.
 
-**`maxWaitMs` default lowered 120s → 15s.** `resilienceSettings.requestQueue.maxWaitMs`
-bounds how long a request may wait in the local queue before it is dropped
-(`code: "RATE_LIMIT_QUEUE_TIMEOUT"`, #4165). The factory default fell from 120000ms to
-15000ms so a saturated queue fails fast instead of holding a caller for two
-minutes; override via `RATE_LIMIT_MAX_WAIT_MS` (env) or the dashboard
-(**Settings → Resilience**, 1–30000ms UI ceiling).
+**`maxWaitMs` is a legacy persisted name for execution expiration.**
+`resilienceSettings.requestQueue.maxWaitMs` is passed to Bottleneck as a job
+`expiration`, whose timer starts only after dispatch. It therefore bounds
+limiter-managed execution, not time spent in the local queue. Expiration is
+surfaced as trusted local `code: "RATE_LIMIT_EXECUTION_TIMEOUT"` (HTTP 504);
+the former queue-timeout code name is accepted only for trusted internal
+backward compatibility. The default is 15000ms; override via
+`RATE_LIMIT_MAX_WAIT_MS` (env) or the dashboard (**Settings → Resilience**,
+1–30000ms UI ceiling). Queue residence has no time deadline; use
+`maxQueueDepth` below to bound queued callers.
 
 **`maxQueueDepth` — opt-in admission cap (new).** `resilienceSettings.requestQueue.maxQueueDepth`
 bounds how many requests may sit queued (not yet dispatched) for one
@@ -252,7 +256,7 @@ it is unit-testable without a real Bottleneck limiter.
 > around the `resolveCompressionSettings`/`selectCompressionStrategy` block),
 > not HTTP response compression on synthesized 429 bodies — there is no
 > matching code path for a literal bypass flag. That prompt-compression step
-> also currently runs *before* `withRateLimit()` in the request pipeline, so
+> also currently runs _before_ `withRateLimit()` in the request pipeline, so
 > reordering to skip it on a queue-full rejection is a separate, larger
 > change than this issue's scope; it was intentionally **not** implemented
 > here and is left as a follow-up if the CPU-saving win is worth the

docs/security/AGENTROUTER_WAF.md

@@ -1,5 +1,5 @@
 ---
-title: "AgentRouter WAF"
+title: "agentrouter.org WAF (Web Application Firewall)"
 version: 3.8.50
 lastUpdated: 2026-08-03
 ---
@@ -94,4 +94,4 @@ The current filter is overly aggressive — it blocks "Lorem ipsum" in
 `tool_result` blocks even though the operator clearly did not intend to
 inject a prompt. Operators who want this fixed at the source should
 contact `agentrouter.org` to report the false positives. The blocklist
-above is the empirical result of probing the upstream as of 2026-08-03.
+above is the empirical result of probing the upstream as of 2026-08-03.

open-sse/handlers/chatCore.ts

@@ -302,6 +302,7 @@ import {
   updateFromResponseBody,
   initializeRateLimits,
 } from "../services/rateLimitManager.ts";
+import * as localLimiterErrors from "../services/rateLimitManager/errors.ts";
 import {
   acquire as acquireAccountSemaphore,
   markBlocked as markAccountSemaphoreBlocked,
@@ -3363,36 +3364,32 @@ export async function handleChatCore({
         errorCode: error.code,
       };
     }
-    // abort(reason) can reject the upstream fetch with a raw string reason
-    // (e.g. "request_signal_aborted") that has no `name`/`status`; classify
-    // via isLocalStreamLifecycleError so those map to 499 instead of falling
-    // through to the 502 provider-failure default.
+    // abort(reason) can reject with a raw string lacking `name`/`status`; classify
+    // it through isLocalStreamLifecycleError so it maps to 499 rather than the
+    // 502 provider-failure default.
     const isRequestAborted = isLocalStreamLifecycleError(error);
-    // #8376: an unreachable upstream proxy (ECONNREFUSED/ECONNRESET/...) is tagged by
-    // proxyFetch.ts (tagProxyUnreachable) with `.errorCode = "proxy_unreachable"` before
-    // it reaches this catch. Classify it explicitly to 502 instead of falling through
-    // the generic `error.status` branch (a raw connect-refused error has no `.status` at
-    // all, so it used to collapse into an ordinary 502/504 the provider-breaker predicate
-    // can't tell apart from a per-model 5xx).
+    // #8376: proxyFetch tags unreachable transport failures so they remain
+    // distinguishable from ordinary provider 5xx responses.
     const isProxyUnreachableFailure =
       !isRequestAborted && (error as { errorCode?: unknown })?.errorCode === "proxy_unreachable";
     const errorCode = getUpstreamErrorIdentifier(error);
-    const isLocalQueueTimeout = errorCode === "RATE_LIMIT_QUEUE_TIMEOUT";
+    const localRateLimitFailure = localLimiterErrors.getClientSafeLocalRateLimitError(error);
     const failureStatus = isRequestAborted
       ? 499
       : isProxyUnreachableFailure
         ? HTTP_STATUS.BAD_GATEWAY
-        : isLocalQueueTimeout
-          ? HTTP_STATUS.SERVICE_UNAVAILABLE
+        : localRateLimitFailure
+          ? localRateLimitFailure.status
           : error.name === "TimeoutError" || error.name === "BodyTimeoutError"
             ? HTTP_STATUS.GATEWAY_TIMEOUT
             : error.status && typeof error.status === "number"
               ? error.status
               : HTTP_STATUS.BAD_GATEWAY;
     const failureMessage = isRequestAborted
       ? "Request aborted"
-      : formatProviderError(error, provider, model, failureStatus);
-    const upstreamErrorCode = isProxyUnreachableFailure ? "proxy_unreachable" : errorCode;
+      : formatProviderError(localRateLimitFailure ?? error, provider, model, failureStatus);
+    const upstreamErrorCode =
+      localRateLimitFailure?.code ?? (isProxyUnreachableFailure ? "proxy_unreachable" : errorCode);
     // Tag our own deadline timeouts (fetch-start TimeoutError / body BodyTimeoutError,
     // both surfaced as a 504) as "upstream_timeout" so the cooldown layer can tell a
     // slow-but-not-failed request apart from a real provider 5xx. (Antigravity already
@@ -3442,19 +3439,22 @@ export async function handleChatCore({
         upstreamErrorCode,
         upstreamErrorType
       );
+      localLimiterErrors.markTrustedLocalRateLimitResponse(result.response, error);
       return {
         ...result,
         errorType: upstreamErrorType,
         errorCode: upstreamErrorCode,
       };
     }
-    return createErrorResult(
+    const result = createErrorResult(
       failureStatus,
       failureMessage,
       null,
       upstreamErrorCode,
       upstreamErrorType
     );
+    localLimiterErrors.markTrustedLocalRateLimitResponse(result.response, error);
+    return result;
   }
   let upstreamErrorParsed = false;
   let parsedStatusCode = providerResponse.status;

open-sse/services/combo.ts

@@ -1591,7 +1591,7 @@ export async function handleComboChat({
                       : undefined,
                 }
               : undefined;
-          const scopedFailure = isScopedFailure(result.status, errorText, structuredError);
+          const scopedFailure = isScopedFailure(result, errorText, structuredError);
 
           // #8375: input-bound request-scoped failures (context_length_exceeded) are
           // deterministic for the same input — retrying on other accounts of the same
@@ -1672,6 +1672,7 @@ export async function handleComboChat({
             rawModel,
             isTokenLimitBreach,
             allAccountsRateLimited: false,
+            requestScopedFailure: scopedFailure,
             sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
             log,
             tag: "COMBO",
@@ -1764,6 +1765,7 @@ export async function handleComboChat({
           const isTransient =
             !isStreamReadinessFailure &&
             !isTokenLimitBreach &&
+            !scopedFailure &&
             [408, 429, 500, 502, 503, 504].includes(result.status);
           if (retry < maxRetries && isTransient && !providerExhausted) {
             if (
@@ -2834,7 +2836,7 @@ async function handleRoundRobinCombo({
                     : undefined,
               }
             : undefined;
-        const scopedFailure = isScopedFailure(result.status, errorText, structuredError);
+        const scopedFailure = isScopedFailure(result, errorText, structuredError);
         const fallbackResult = checkFallbackError(
           result.status,
           errorText,
@@ -2873,6 +2875,7 @@ async function handleRoundRobinCombo({
           rawModel: parseModel(modelStr).model || modelStr,
           isTokenLimitBreach,
           allAccountsRateLimited: isAllAccountsRateLimited,
+          requestScopedFailure: scopedFailure,
           sets: { exhaustedProviders, exhaustedConnections, transientRateLimitedProviders },
           log,
           tag: "COMBO-RR",
@@ -2906,6 +2909,7 @@ async function handleRoundRobinCombo({
         const isTransient =
           !isStreamReadinessFailure &&
           !isTokenLimitBreach &&
+          !scopedFailure &&
           [408, 429, 500, 502, 503, 504].includes(result.status);
         if (retry < maxRetries && isTransient && !providerExhausted) {
           continue;

open-sse/services/combo/comboPredicates.ts

@@ -12,6 +12,7 @@ import { isSelfInflictedUpstreamTimeout } from "../../handlers/chatCore/cooldown
 import { isLocalStreamLifecycleError } from "@/shared/utils/circuitBreaker";
 import { CONTEXT_OVERFLOW_PATTERNS, MODEL_ACCESS_DENIED_PATTERNS } from "../accountFallback.ts";
 import { isResourceNotFoundResponse } from "../errorClassifier.ts";
+import { getTrustedLocalRateLimitResponse } from "../rateLimitManager/errors.ts";
 import type { ResolvedComboTarget } from "./types.ts";
 
 // Status codes that should mark round-robin target semaphores as cooling down.
@@ -190,13 +191,13 @@ export function shouldRecordProviderBreakerFailure(args: {
   );
 }
 
-const REQUEST_SCOPED_UPSTREAM_ERROR_CODES = new Set([
-  "context_length_exceeded",
-  "upstream_empty_response",
-  "upstream_response_failed",
+const REQUEST_SCOPED_UPSTREAM_ERROR_CODES: Record<string, true> = {
+  context_length_exceeded: true,
+  upstream_empty_response: true,
+  upstream_response_failed: true,
   // Local combo per-target timer (targetTimeoutRunner) — not a connection health signal.
-  "combo_target_timeout",
-]);
+  combo_target_timeout: true,
+};
 
 /** Request/model-specific failures must not poison provider-wide resilience state. */
 export function isRequestScopedUpstreamFailure(error?: {
@@ -205,18 +206,19 @@ export function isRequestScopedUpstreamFailure(error?: {
 }): boolean {
   const code = typeof error?.code === "string" ? error.code.toLowerCase() : "";
   const type = typeof error?.type === "string" ? error.type.toLowerCase() : "";
-  return REQUEST_SCOPED_UPSTREAM_ERROR_CODES.has(code) || type === "context_length_exceeded";
+  return REQUEST_SCOPED_UPSTREAM_ERROR_CODES[code] === true || type === "context_length_exceeded";
 }
 
 /** Request-scoped classification that also has access to the HTTP body. */
 export function isComboRequestScopedFailure(
-  status: number,
+  response: Response,
   errorText: string,
   error?: { code?: string | null; type?: string | null }
 ): boolean {
   return (
+    getTrustedLocalRateLimitResponse(response) !== null ||
     isRequestScopedUpstreamFailure(error) ||
-    (status === 404 && isResourceNotFoundResponse(errorText))
+    (response.status === 404 && isResourceNotFoundResponse(errorText))
   );
 }
 
@@ -255,6 +257,7 @@ export function isInputBoundRequestFailure(error?: {
 export function shouldSkipConnDisable(
   result: {
     status: number;
+    response?: Response;
     errorCode?: string | null;
     errorType?: string | null;
     error?: unknown;
@@ -270,6 +273,7 @@ export function shouldSkipConnDisable(
     // Client abort surfaced as a bare error (no statusCode → defaults to 502):
     // a local lifecycle event, not a provider failure (#4602 policy).
     isLocalStreamLifecycleError(result.error) ||
+    (result.response ? getTrustedLocalRateLimitResponse(result.response) !== null : false) ||
     result.errorCode === "plugin_block" ||
     result.errorType === "plugin_block" ||
     (is401 && hasExtraKeys) ||

open-sse/services/combo/targetExhaustion.ts

@@ -61,6 +61,7 @@ export type ApplyComboTargetExhaustionOptions = {
   rawModel: string;
   isTokenLimitBreach: boolean;
   allAccountsRateLimited: boolean;
+  requestScopedFailure: boolean;
   sets: ComboExhaustionSets;
   log: ComboLogger;
   tag: string;
@@ -142,12 +143,25 @@ function isProviderQuotaExhausted(
   provider: string | null | undefined,
   opts: Pick<
     ApplyComboTargetExhaustionOptions,
-    "rawModel" | "fallbackResult" | "structuredError" | "errorText" | "allAccountsRateLimited"
+    | "rawModel"
+    | "fallbackResult"
+    | "structuredError"
+    | "errorText"
+    | "allAccountsRateLimited"
+    | "requestScopedFailure"
   >
 ): boolean {
-  const { rawModel, fallbackResult, structuredError, errorText, allAccountsRateLimited } = opts;
+  const {
+    rawModel,
+    fallbackResult,
+    structuredError,
+    errorText,
+    allAccountsRateLimited,
+    requestScopedFailure,
+  } = opts;
   return (
     Boolean(provider && provider !== "unknown") &&
+    !(requestScopedFailure || isRequestScopedUpstreamFailure(structuredError)) &&
     !hasPerModelQuota(provider as string, rawModel) &&
     (isProviderExhaustedReason(fallbackResult) ||
       classifyErrorText(structuredError?.code || errorText) === RateLimitReason.QUOTA_EXHAUSTED ||
@@ -177,7 +191,17 @@ function markTransientOrConnectionLevel(
   target: ResolvedComboTarget,
   opts: ApplyComboTargetExhaustionOptions
 ): void {
-  const { result, errorText, rawModel, isTokenLimitBreach, sets, log, tag, structuredError } = opts;
+  const {
+    result,
+    errorText,
+    rawModel,
+    isTokenLimitBreach,
+    requestScopedFailure,
+    sets,
+    log,
+    tag,
+    structuredError,
+  } = opts;
   const provider = target.provider;
   if (result.status === 429 && !isTokenLimitBreach && provider && provider !== "unknown") {
     sets.transientRateLimitedProviders.add(provider);
@@ -189,6 +213,7 @@ function markTransientOrConnectionLevel(
     log,
     tag,
     rawModel,
+    requestScopedFailure,
     structuredError,
   });
 }
@@ -232,16 +257,25 @@ function markConnectionLevelExhaustion(
   target: ResolvedComboTarget,
   opts: Pick<
     ApplyComboTargetExhaustionOptions,
-    "result" | "errorText" | "sets" | "log" | "tag" | "rawModel" | "structuredError"
+    | "result"
+    | "errorText"
+    | "sets"
+    | "log"
+    | "tag"
+    | "rawModel"
+    | "requestScopedFailure"
+    | "structuredError"
   >
 ): void {
-  const { result, errorText, sets, log, tag, rawModel, structuredError } = opts;
+  const { result, errorText, sets, log, tag, rawModel, requestScopedFailure, structuredError } =
+    opts;
   const provider = target.provider;
   if (
     !provider ||
     provider === "unknown" ||
     !CONNECTION_LEVEL_ERROR_STATUSES.includes(result.status) ||
     isProviderCircuitOpenResult(result, errorText) ||
+    requestScopedFailure ||
     isRequestScopedUpstreamFailure(structuredError) ||
     // #5085: empty-content 502 is a healthy connection returning no body — model-level, not
     // connection-level. Don't exhaust the provider; let the remaining legs (incl. same-provider)