fix(proxies): shorten refused-egress set-aside and rest slow egress

.env.example

@@ -826,15 +826,15 @@ NEXT_PUBLIC_ENABLE_SOCKS5_PROXY=true
 
 # Curve of the 429 set-aside (read once at startup; restart to apply).
 # Base period for the first refusal through a proxy, doubled on each repeat.
-# Bounded to 1000..3600000; out-of-range or unreadable falls back to 120000 with a warning.
+# Bounded to 1000..3600000; out-of-range or unreadable falls back to 300000 with a warning.
 # Used by: open-sse/utils/proxyRefusalMemory.ts.
-# PROXY_QUOTA_429_BASE_MS=120000
+# PROXY_QUOTA_429_BASE_MS=300000
 
 # Cap of the 429 set-aside (read once at startup; restart to apply).
-# Bounded to 1000..3600000; out-of-range or unreadable falls back to 3600000 with a warning.
-# When the cap is below the base, it falls back to the default 3600000.
+# Bounded to 1000..3600000; out-of-range or unreadable falls back to 900000 with a warning.
+# When the cap is below the base, it falls back to the default 900000.
 # Lowering the cap also shortens streak forgetting: entries purge 2 x max after expiry.
-# PROXY_QUOTA_429_MAX_MS=3600000
+# PROXY_QUOTA_429_MAX_MS=900000
 
 # Opt-in feature flag (default off; a dashboard DB override wins over this value): for
 # providers whose quota is bucketed by egress address, rank a pool member sharing a recently

changelog.d/fixes/14891-refused-egress-setaside-tuning.md

@@ -0,0 +1 @@
+- **fix(proxies):** refused-egress set-aside defaults to a 300s base with a 900s cap, and an egress that repeatedly waits out the response-headers window is set aside under a short 60s/600s curve after three settled overruns in five minutes ([#14891](https://github.com/diegosouzapw/OmniRoute/pull/14891)) — thanks @maxmad64bis

docs/reference/ENVIRONMENT.md

@@ -379,8 +379,8 @@ Route upstream LLM provider calls through an HTTP or SOCKS5 proxy for egress con
 | `NEXT_PUBLIC_ENABLE_SOCKS5_PROXY`        | `true`    | Client-side                                  | Client-side awareness of SOCKS5 availability.                                                                                                                                                                                                                                                                                                       |
 | `PROXY_SKIP_RECENTLY_FAILED`             | `true`    | `src/shared/utils/featureFlags.ts`           | On-by-default feature flag (see [FEATURE_FLAGS.md](./FEATURE_FLAGS.md); a dashboard DB override wins). Proxy pools and per-account rotation stop re-serving a member that just failed (refused TCP probe, or a 429 through it) for a period that doubles on each repeat, up to a cap. `false` restores plain selection.                                          |
 | `SELECTOR_CONTROL_ALLOWLIST`             | _(unset)_ | `src/lib/proxySubscription/selectorGuard.ts` | Allow-list for local-core selector-control targets beyond loopback (comma/space separated hosts or IPs, `*` opens LAN/private but never metadata/link-local). Loopback over http/https is allowed by default. A dashboard DB value wins over this variable when set and non-empty.                                                                                                                                                                          |
-| `PROXY_QUOTA_429_BASE_MS`                | `120000`  | `open-sse/utils/proxyRefusalMemory.ts`       | Base period (ms) of the 429 set-aside, doubled on each repeat. Read once at startup. Bounded to `1000..3600000`; out-of-range or unreadable falls back to the default with a warning. |
-| `PROXY_QUOTA_429_MAX_MS`                 | `3600000` | `open-sse/utils/proxyRefusalMemory.ts`       | Cap (ms) of the 429 set-aside. Read once at startup. Bounded to `1000..3600000`; out-of-range or unreadable falls back to the default with a warning. When the cap is below the base, it falls back to the default. Lowering the cap also shortens streak forgetting: entries purge `2 x max` after expiry.                                                                                                                                                                          |
+| `PROXY_QUOTA_429_BASE_MS`                | `300000`  | `open-sse/utils/proxyRefusalMemory.ts`       | Base period (ms) of the 429 set-aside, doubled on each repeat. Read once at startup. Bounded to `1000..3600000`; out-of-range or unreadable falls back to the default with a warning. |
+| `PROXY_QUOTA_429_MAX_MS`                 | `900000` | `open-sse/utils/proxyRefusalMemory.ts`       | Cap (ms) of the 429 set-aside. Read once at startup. Bounded to `1000..3600000`; out-of-range or unreadable falls back to the default with a warning. When the cap is below the base, it falls back to the default. Lowering the cap also shortens streak forgetting: entries purge `2 x max` after expiry.                                                                                                                                                                          |
 | `PROXY_POOL_SHARED_EGRESS_ORDER`         | `false`   | `src/shared/utils/featureFlags.ts`           | Opt-in feature flag (see [FEATURE_FLAGS.md](./FEATURE_FLAGS.md); a dashboard DB override wins). For providers whose quota is bucketed by egress address, rank a pool member sharing a recently refused member's observed egress address just below healthy members. Order only, never excluded. Needs `PROXY_SKIP_RECENTLY_FAILED`, which produces the refusal signal it reads. |
 | `PROXY_WEBHOOK_REBOUND_MS`               | `300000`  | `src/lib/proxyEvents/proxyTransitionBridge.ts` | Rebound window (ms) capping repeat `proxy.set_aside` webhook alerts per pool member so a burst of refusals sends one event. Read on each emit. Bounded to `1000..3600000`; out-of-range or unreadable falls back to the default silently. |
 | `HTTP_PROXY`                             | _(unset)_ | Node.js standard                             | HTTP proxy for upstream calls.                                                                                                                                                                                                                                                                                                                      |

open-sse/executors/opencode.ts

@@ -446,8 +446,7 @@ export class OpencodeExecutor extends BaseExecutor {
               if (isResponsesTerminalLine(line)) {
                 // OpenCode Zen sends a ping after response.completed and may keep
                 // the HTTP connection alive. The Responses terminal event is
-                // authoritative; do not let those post-completion pings hold Chat
-                // Completions open.
+                // authoritative; do not let those post-completion pings hold Chat Completions open.
                 closed = true;
                 void reader.cancel().catch(() => undefined);
                 controller.close();
@@ -865,6 +864,7 @@ export class OpencodeExecutor extends BaseExecutor {
               stalled: headersWait.spent,
               cooldown: markCooldown,
               markDirect: () => (directTried = true),
+              slow: { account, enabled: skipRecentlyFailed, read: readAppliedKey },
             }); // same settle as the stall arm
             log?.warn?.(
               "OPENCODE",

open-sse/executors/opencodeEgressThrottle.ts

@@ -19,7 +19,12 @@ import { sleepAbortable } from "./opencodeTransientFailure.ts";
 import { classifyUpstream429, type RateLimit429Verdict } from "./opencodeRateLimited.ts";
 import { proxyKeyOf } from "./opencodeGeoBlock.ts";
 import { pickAccount, type RotatableAccount } from "./accountRotation.ts";
-import { noteProxyRefusal, proxyEgressKey } from "../utils/proxyRefusalMemory.ts";
+import {
+  hasSlowOverrunEvidence,
+  noteProxyRefusal,
+  proxyEgressKey,
+  recordSlowOverrun,
+} from "../utils/proxyRefusalMemory.ts";
 
 export const DIRECT_EGRESS_SENTINEL = "direct";
 
@@ -500,6 +505,37 @@ export function noteRefusedMember(
   return noteProxyRefusal(key, "ip_quota_429");
 }
 
+/**
+ * Set-aside note for one settled headers-wait overrun: records the overrun,
+ * then hands the refusal to the proxy memory only on repetition (three settled
+ * overruns through the same egress key inside five minutes) and only when the
+ * opt-in is on. A lone slow wait is the upstream queue, not the member.
+ * The key resolves through the same seam as the 429 note: the effective egress
+ * really applied to the attempt (dedicated proxy fast path, ambient reader for
+ * proxyless accounts, direct sentinel as no-op). Returns the set-aside
+ * duration for the log, or null when nothing was set aside.
+ */
+export function noteSlowOverrun(
+  account: AppliedEgressAccount,
+  skipRecentlyFailed: boolean,
+  readApplied?: AppliedEgressReader | null,
+  nowMs: number = Date.now()
+): number | null {
+  if (!skipRecentlyFailed) return null;
+  if (account.proxy !== null) {
+    const key = proxyEgressKey(account.proxy);
+    if (key === null) return null;
+    recordSlowOverrun(key, nowMs);
+    if (!hasSlowOverrunEvidence(key, nowMs)) return null;
+    return noteProxyRefusal(key, "slow", nowMs);
+  }
+  const key = resolveAppliedEgressKey(account, readApplied);
+  if (key === DIRECT_EGRESS_SENTINEL) return null;
+  recordSlowOverrun(key, nowMs);
+  if (!hasSlowOverrunEvidence(key, nowMs)) return null;
+  return noteProxyRefusal(key, "slow", nowMs);
+}
+
 /**
  * Resolve the 429 verdict for one refused dispatch. When the rate-limited
  * early-stop is on, the classifier decides; a rate-limit verdict stays on the
@@ -600,13 +636,15 @@ export function throwPacedError(release: (() => void) | null, err: unknown): nev
 /**
  * Rotation-loop wiring for the stall arm: the tried-set plus a mutable stall
  * counter, bundled so the arm holds one call. The loop owns `stalled` and the
- * helper reads-then-bumps it.
+ * helper reads-then-bumps it. The optional slow note records one settled
+ * headers-wait overrun per call; the stall arm never passes it.
  */
 export interface StallLoopWiring {
   tried: Set<string>;
   stalled: { attempts: number };
   cooldown: (account: { proxy: { host: string; port: number } | null }) => void;
   markDirect: () => void;
+  slow?: { account: AppliedEgressAccount; enabled: boolean; read: AppliedEgressReader | null };
 }
 
 /**
@@ -627,6 +665,8 @@ export function settleStalledDispatch(
   else loop.markDirect();
   const first = loop.stalled.attempts === 0;
   loop.stalled.attempts++;
+  const slow = loop.slow;
+  if (slow) noteSlowOverrun(slow.account, slow.enabled, slow.read ?? null);
   return first;
 }
 

open-sse/utils/proxyRefusalMemory.ts

@@ -10,12 +10,14 @@
  * (callers gate writes and decisions on it) and it stays free of the proxy dispatcher, so
  * the DB layer can consult it without loading undici or the SOCKS connector.
  */
-import { COOLDOWN_MS } from "../config/errorConfig.ts";
 import { notifyProxyTransition } from "./proxyTransitionListeners.ts";
 import { stripIpv6Brackets } from "./proxyFamily.ts";
 
-const DEFAULT_QUOTA_429_BASE_MS = COOLDOWN_MS.rateLimit;
-const DEFAULT_QUOTA_429_MAX_MS = 3_600_000;
+// Field trends show a refused egress rarely recovers within minutes, so the
+// first set-aside lasts five minutes and doubles from there; the cap lets a
+// recovered member return within the quarter-hour.
+const DEFAULT_QUOTA_429_BASE_MS = 300_000;
+const DEFAULT_QUOTA_429_MAX_MS = 900_000;
 const MIN_QUOTA_429_MS = 1_000;
 const MAX_QUOTA_429_MS = 3_600_000;
 
@@ -57,6 +59,7 @@ export const REFUSAL_POLICIES: {
   proxy_unreachable: { baseMs: 60_000; maxMs: 600_000 };
   ip_quota_429: RefusalPolicy;
   transport: { baseMs: 60_000; maxMs: 600_000 };
+  slow: { baseMs: 60_000; maxMs: 600_000 };
 } = {
   /** The TCP probe could not open a connection to the proxy. */
   proxy_unreachable: { baseMs: 60_000, maxMs: 600_000 },
@@ -68,6 +71,12 @@ export const REFUSAL_POLICIES: {
    * not the destination — is at fault. Same curve as a refused probe.
    */
   transport: { baseMs: 60_000, maxMs: 600_000 },
+  /**
+   * Repeated settled waits for upstream response headers through this egress:
+   * one slow wait is the upstream queue, not the member. Same short curve as
+   * a refused probe, kept apart from the quota curve.
+   */
+  slow: { baseMs: 60_000, maxMs: 600_000 },
 };
 
 export type ProxyRefusalKind = keyof typeof REFUSAL_POLICIES;
@@ -390,6 +399,65 @@ export function __transportEvidenceSizeForTesting(): { failures: number; success
   return { failures: transportFailures.length, successes: transportSuccesses.length };
 }
 
+/**
+ * Settled waits for upstream response headers, per egress key. One slow wait
+ * is the upstream queue, not the member — only repetition condemns it, so the
+ * store keeps bare timestamps and the gated helper decides at k=3 / 5 min.
+ */
+export const SLOW_OVERRUN_WINDOW_MS = 300_000;
+export const SLOW_OVERRUN_THRESHOLD = 3;
+const MAX_SLOW_OVERRUNS = 1000;
+
+type SlowOverrun = { key: string; at: number };
+
+const slowOverruns: SlowOverrun[] = [];
+
+// Lazy purge mirrors readState: entries older than the evidence window plus
+// twice the slow cap can no longer contribute, so drop them on record.
+function purgeSlowOverruns(nowMs: number): void {
+  const cutoff = nowMs - SLOW_OVERRUN_WINDOW_MS - 2 * REFUSAL_POLICIES.slow.maxMs;
+  while (slowOverruns.length > 0 && slowOverruns[0].at < cutoff) {
+    slowOverruns.shift();
+  }
+}
+
+/** Record one settled headers-wait overrun. Never throws, never writes refusal memory. */
+export function recordSlowOverrun(key: string | null, nowMs: number = Date.now()): void {
+  if (key === null || key === "") return;
+  purgeSlowOverruns(nowMs);
+  slowOverruns.push({ key, at: nowMs });
+  while (slowOverruns.length > MAX_SLOW_OVERRUNS) slowOverruns.shift();
+}
+
+/**
+ * True when this egress waited out the headers window at least
+ * SLOW_OVERRUN_THRESHOLD times inside SLOW_OVERRUN_WINDOW_MS. Overruns for
+ * other keys never count.
+ */
+export function hasSlowOverrunEvidence(key: string | null, nowMs: number = Date.now()): boolean {
+  if (key === null || key === "") return false;
+  purgeSlowOverruns(nowMs);
+  const from = nowMs - SLOW_OVERRUN_WINDOW_MS;
+  let overruns = 0;
+  for (const o of slowOverruns) {
+    if (o.key === key && o.at >= from) {
+      overruns++;
+      if (overruns >= SLOW_OVERRUN_THRESHOLD) return true;
+    }
+  }
+  return false;
+}
+
+/** Test-only: forget slow overruns (refusal memory is separate). */
+export function __resetSlowOverrunsForTesting(): void {
+  slowOverruns.length = 0;
+}
+
+/** Test-only: current slow-overrun store size. */
+export function __slowOverrunSizeForTesting(): number {
+  return slowOverruns.length;
+}
+
 /** Test-only: number of (key, kind) entries held. */
 export function __proxyRefusalMemorySizeForTesting(): number {
   return memory.size;

open-sse/utils/proxyTransitionListeners.ts

@@ -7,7 +7,7 @@
  * in this file so the store stays pure and cycle-free.
  */
 
-export type ProxyTransitionKind = "ip_quota_429" | "proxy_unreachable" | "transport";
+export type ProxyTransitionKind = "ip_quota_429" | "proxy_unreachable" | "transport" | "slow";
 
 export interface ProxyTransition {
   key: string;

src/app/(dashboard)/dashboard/settings/components/PoolMemberEgressLines.tsx

@@ -104,6 +104,8 @@ const SET_ASIDE_KIND_KEYS: Record<string, string> = {
   // Repeated transport failures with cross-egress evidence (#14802): the proxy
   // path is what failed, so it reads as unreachable to the operator.
   transport: "poolSetAsideKindProxyUnreachable",
+  // Repeated waits for response headers through this egress: slow, not refused.
+  slow: "poolSetAsideKindSlow",
 };
 
 function setAsideKindLabel(

src/i18n/messages/am.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "ወደ ጎን ተቀምጧል: {kind}",
     "poolSetAsideKindProxyUnreachable": "ፕሮክሲው ሊደረስበት አልቻለም",
     "poolSetAsideKindIpQuota429": "የIP ፍጥነት ገደብ (429)",
+    "poolSetAsideKindSlow": "ዝግታ ምላሾች",
     "poolSetAsideUntil": "ወደ ሽክርክር ተመልሷል {endsAt}",
     "poolSetAsideRepeat": "ወደ ጎን ተቀምጧል {count, plural, one {# ጊዜ} other {# ጊዜ}}",
     "poolUnknownAddress": "ያልታወቀ አድራሻ",

src/i18n/messages/ar.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "مستبعد: {kind}",
     "poolSetAsideKindProxyUnreachable": "تعذّر الوصول إلى الوكيل",
     "poolSetAsideKindIpQuota429": "تقييد معدل عنوان IP ‏(429)",
+    "poolSetAsideKindSlow": "استجابات بطيئة",
     "poolSetAsideUntil": "يعود إلى التناوب {endsAt}",
     "poolSetAsideRepeat": "مستبعد {count, plural, one {# مرة} other {# مرات}}",
     "poolUnknownAddress": "عنوان غير معروف",

src/i18n/messages/az.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Kənara qoyuldu: {kind}",
     "poolSetAsideKindProxyUnreachable": "proksiyə çatmaq mümkün deyil",
     "poolSetAsideKindIpQuota429": "IP sürət limiti (429)",
+    "poolSetAsideKindSlow": "yavaş cavablar",
     "poolSetAsideUntil": "Rotasiyaya qayıtdı {endsAt}",
     "poolSetAsideRepeat": "Kənara qoyuldu {count, plural, one {# dəfə} other {# dəfə}}",
     "poolUnknownAddress": "Naməlum ünvan",

src/i18n/messages/bg.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Оставено настрана: {kind}",
     "poolSetAsideKindProxyUnreachable": "проксито е недостъпно",
     "poolSetAsideKindIpQuota429": "ограничение на скоростта по IP (429)",
+    "poolSetAsideKindSlow": "бавни отговори",
     "poolSetAsideUntil": "Обратно в ротация {endsAt}",
     "poolSetAsideRepeat": "Оставено настрана {count, plural, one {# път} other {# пъти}}",
     "poolUnknownAddress": "Неизвестен адрес",

src/i18n/messages/bn.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "সরিয়ে রাখা হয়েছে: {kind}",
     "poolSetAsideKindProxyUnreachable": "প্রক্সিতে পৌঁছানো যাচ্ছে না",
     "poolSetAsideKindIpQuota429": "IP রেট সীমা (429)",
+    "poolSetAsideKindSlow": "ধীর প্রতিক্রিয়া",
     "poolSetAsideUntil": "রোটেশনে ফিরেছে {endsAt}",
     "poolSetAsideRepeat": "সরিয়ে রাখা হয়েছে {count, plural, one {# বার} other {# বার}}",
     "poolUnknownAddress": "অজানা ঠিকানা",

src/i18n/messages/bs.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Sklonjeno: {kind}",
     "poolSetAsideKindProxyUnreachable": "proxy nedostupan",
     "poolSetAsideKindIpQuota429": "ograničenje brzine po IP-u (429)",
+    "poolSetAsideKindSlow": "spori odgovori",
     "poolSetAsideUntil": "Nazad u rotaciji {endsAt}",
     "poolSetAsideRepeat": "Sklonjeno {count, plural, one {# put} other {# puta}}",
     "poolUnknownAddress": "Nepoznata adresa",

src/i18n/messages/cs.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Odloženo: {kind}",
     "poolSetAsideKindProxyUnreachable": "proxy je nedostupná",
     "poolSetAsideKindIpQuota429": "omezení rychlosti podle IP (429)",
+    "poolSetAsideKindSlow": "pomalé odpovědi",
     "poolSetAsideUntil": "Zpět v rotaci {endsAt}",
     "poolSetAsideRepeat": "Odloženo {count, plural, one {#krát} other {#krát}}",
     "poolUnknownAddress": "Neznámá adresa",

src/i18n/messages/da.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Sat til side: {kind}",
     "poolSetAsideKindProxyUnreachable": "proxy kan ikke nås",
     "poolSetAsideKindIpQuota429": "IP-hastighedsbegrænsning (429)",
+    "poolSetAsideKindSlow": "langsomme svar",
     "poolSetAsideUntil": "Tilbage i rotation {endsAt}",
     "poolSetAsideRepeat": "Sat til side {count, plural, one {# gang} other {# gange}}",
     "poolUnknownAddress": "Ukendt adresse",

src/i18n/messages/de.json

@@ -11082,6 +11082,7 @@
     "poolSetAsideReason": "Zurückgestellt: {kind}",
     "poolSetAsideKindProxyUnreachable": "Proxy nicht erreichbar",
     "poolSetAsideKindIpQuota429": "IP-Ratenlimit (429)",
+    "poolSetAsideKindSlow": "langsame Antworten",
     "poolSetAsideUntil": "Zurück in Rotation {endsAt}",
     "poolSetAsideRepeat": "Zurückgestellt {count, plural, one {# Mal} other {# Mal}}",
     "poolUnknownAddress": "Unbekannte Adresse",