fix(proxy-health): skip sweep probes already proven by recent production successes

.env.example

@@ -2488,6 +2488,13 @@ APP_LOG_TO_FILE=true
 # never used for routing). "true" (or 1, yes) enables it.
 # PROXY_POOL_EGRESS_OBSERVATION=false
 
+# Opt-in: skip the scheduler probe for proxies already proven by recent
+# production traffic (single provider, no recent proxy failure). Default: off.
+# PROXY_HEALTH_PASSIVE_SKIP=false
+# Window in ms of production traffic the passive skip looks at (60000..599999).
+# Default: 300000 (5min).
+# PROXY_PASSIVE_WINDOW_MS=300000
+
 # Allow OAuth and provider validation flows to bypass a pinned proxy and connect
 # directly when proxy reachability pre-checks fail. Default: false.
 # Also configurable from Dashboard > Settings > Feature Flags.

changelog.d/fixes/14805-passive-proxy-health.md

@@ -0,0 +1 @@
+- **fix(proxy-health):** skip sweep probes already proven by recent single-provider production successes (a recent failure still gets its live probe), opt-in via `PROXY_HEALTH_PASSIVE_SKIP` (default off: sweep probes every proxy as before) ([#14805](https://github.com/diegosouzapw/OmniRoute/pull/14805)) — thanks @maxmad64bis

docs/reference/ENVIRONMENT.md

@@ -1228,6 +1228,8 @@ Anthropic-compatible provider instead.
 | `PROXY_AUTO_REMOVE`                             | `false`                 | `src/lib/proxyHealth/scheduler.ts`                                                      | Set `true` to let the scheduler auto-remove proxies after repeated consecutive failures.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | `PROXY_AUTO_REMOVE_AFTER`                       | `3`                     | `src/lib/proxyHealth/scheduler.ts`                                                      | Consecutive failures before the scheduler auto-removes a proxy (when `PROXY_AUTO_REMOVE=true`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
 | `PROXY_AUTO_DISABLE`                            | `false`                 | `src/lib/proxyHealth/scheduler.ts`                                                      | Set `true` to let the scheduler soft-disable (status `dead`, never deleted) a proxy after repeated consecutive failures, instead of removing it. Non-destructive alternative to `PROXY_AUTO_REMOVE`: the proxy drops out of pool/rotation resolution immediately (the alive-status filter used by scope-pool resolution already excludes it) and is automatically re-activated once it starts passing probes again. Shares the `PROXY_AUTO_REMOVE_AFTER` threshold. If both flags are `true`, `PROXY_AUTO_REMOVE` wins.                                                  |
+| `PROXY_HEALTH_PASSIVE_SKIP`                     | `false`                 | `src/lib/proxyHealth/scheduler.ts`                                                      | Opt-in passive skip: when `true`, the scheduler skips the live probe for an endpoint whose recent production traffic already proves it (at least one success and no attributed proxy failure in the window, exactly one provider observed). Off by default: every proxy is probed. |
+| `PROXY_PASSIVE_WINDOW_MS`                       | `300000`                | `src/lib/proxyHealth/passiveVerdict.ts`                                                 | Window in ms of production traffic the passive skip looks at (bounded `60000..599999`). Shorter than the 10-minute sweep interval so a recent failure dominates instead of being diluted. |
 | `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | `false`                 | `src/shared/constants/featureFlagDefinitions.ts`                                        | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Effective precedence is Feature Flags DB override > env var > default.                                                                                                                                                                                                                                                                                                                                                                   |
 | `RATE_LIMIT_MAX_WAIT_MS`                        | `30000` (30s)           | `src/lib/resilience/settings.ts`                                                 | Default **queue-wait** budget: how long a request may wait for a provider slot and sit queued before it is rejected, never reaching the upstream. The timer is cleared once the job starts executing — execution is bounded separately by `RATE_LIMIT_EXECUTION_MAX_WAIT_MS`. Env sets the default only: a persisted `resilienceSettings.requestQueue.maxWaitMs`, and then a per-connection `rateLimitOverrides.maxWaitMs`, each win over it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | `RATE_LIMIT_EXECUTION_MAX_WAIT_MS`              | `600000` (10 min)       | `open-sse/services/rateLimitManager.ts`                                                 | Ceiling for how long an admitted request may stay in execution before its rate-limit reservation expires — decoupled from the queue-wait budget so slow fetch-start on non-incremental gateways does not time out (#12027). |

src/lib/proxyHealth/passiveVerdict.ts

@@ -0,0 +1,137 @@
+/**
+ * Passive verdict from production traffic.
+ *
+ * Reads health from real request history instead of probing what production
+ * already exercises: an endpoint with a recent attributed proxy failure keeps
+ * its live probe (only the probe can confirm it), while an endpoint whose
+ * window shows at least one success and no attributed failure is already
+ * proven and skips the redundant probe.
+ *
+ * Pure aggregation over already-read rows (no SQL here — the sweep reads via
+ * the existing `proxyLogs.ts` readers and this module only folds rows into a
+ * boolean). Never mutates a status, never removes: the sweep only uses the
+ * fold to skip or keep a probe (plus a single-provider guard decided in the
+ * scheduler).
+ */
+
+/** Minimal row shape consumed from `proxy_logs` readers. */
+export interface PassiveLogRow {
+  proxy_host?: string | null;
+  proxy_port?: number | null;
+  provider?: string | null;
+  status?: string | null;
+  error?: string | null;
+  upstream_status?: number | null;
+  attempt_issue?: string | null;
+  timestamp?: string | null;
+}
+
+export interface AttributedFailureSignals {
+  error?: string | null;
+  upstream_status?: number | null;
+  status?: string | null;
+}
+
+// Connection-level failures that prove the proxy is at fault: the request
+// never reached a provider (DNS never resolved, connection refused, or the
+// socket died on the proxy leg). Only bounded syscall codes blame the proxy,
+// matched at word granularity so provider text never matches. Matched against
+// the `error` column of `proxy_logs`.
+const ATTRIBUTED_ERROR_PATTERNS =
+  /\b(?:ENOTFOUND|EAI_AGAIN|ECONNREFUSED|EHOSTUNREACH|ENETUNREACH|ENOTCONN|EHOSTDOWN|ENETDOWN|ERR_SOCKET_CLOSED|ERR_SOCKET_TIMEOUT)\b|socket hang up/i;
+
+// Explicitly neutral (never attributed — each pinned by a test): mid-flight
+// drops (ECONNRESET, EPIPE, ETIMEDOUT, ECONNABORTED, ERR_NETWORK, bare
+// ERR_SOCKET), bare provider-side wording (TLS, DNS, handshake, fetch
+// failed), and coded upstream TLS faults (CERT_*, ERR_TLS_*, TLSV1_ALERT_*).
+// The matcher above fires only on bounded proxy codes, so anything listed
+// here falls through to `false` — except a message that ALSO carries a
+// bounded proxy code (`fetch failed: connect ECONNREFUSED ...`), where the
+// code decides and the row degrades.
+
+// Relayed provider refusals (the TARGET refused this egress IP) prove the
+// proxy relayed fine — neutral, exactly like the sweep's `blocked` outcome.
+const TARGET_REFUSAL_STATUSES: ReadonlySet<number> = new Set([401, 403, 429]);
+
+/**
+ * PURE: does this logged row prove the PROXY failed (not the provider, not
+ * the network)? Bounded syscall codes / stalled proxy-leg handshake → true
+ * (degrade). Mid-flight drops, bare provider-side tokens, coded upstream TLS
+ * faults, relayed 5xx, target refusals → false (neutral).
+ * A row with `upstream_status` NULL carries no proof either way → false.
+ */
+export function isProxyAttributedFailure(signals: AttributedFailureSignals): boolean {
+  const { error = null, upstream_status = null, status = null } = signals;
+  // A provider answer (any upstream status) means the proxy relayed: a 5xx
+  // blames the provider, a 401/403/429 refusal blames the target — never the
+  // proxy. Only the absence of an upstream answer can implicate the proxy.
+  if (typeof upstream_status === "number") return false;
+  if (status === "blocked" || (typeof status === "string" && status.startsWith("refused"))) {
+    return false;
+  }
+  // Relayed target refusals (401/403/429 carried on the row) prove the proxy
+  // relayed — neutral, like the sweep's `blocked` outcome.
+  if (typeof status === "string" && TARGET_REFUSAL_STATUSES.has(Number(status))) {
+    return false;
+  }
+  if (typeof error !== "string" || error.length === 0) return false;
+  // A bounded proxy code anywhere in the message decides, even behind a
+  // generic prefix (`fetch failed: connect ECONNREFUSED ...` still degrades).
+  // Bare provider-side tokens (TLS/DNS/handshake/fetch wording, coded
+  // upstream TLS faults) match nothing here and stay neutral below.
+  return ATTRIBUTED_ERROR_PATTERNS.test(error);
+}
+
+/**
+ * PURE: fold windowed rows into a skip proof. `abandoned` sends are excluded
+ * (an abandoned send proves nothing about the proxy). True only when the
+ * window holds at least one `success` and no attributed proxy failure —
+ * order-independent, so the callers' most-recent-first ordering has no
+ * effect. A row with NULL `upstream_status` and no attributed error is
+ * simply not a success and carries no veto either.
+ */
+export function hasProvenSuccessWithoutAttributedFailure(rows: PassiveLogRow[]): boolean {
+  let hasSuccess = false;
+  for (const row of rows) {
+    if (row.attempt_issue === "abandoned") continue;
+    if (isProxyAttributedFailure(row)) return false;
+    if (row.status === "success") hasSuccess = true;
+  }
+  return hasSuccess;
+}
+
+/** Cache key: one proxy endpoint (skips additionally gate on the providers seen). */
+export function passiveVerdictKey(host: string, port: number): string {
+  return `${host}:${port}`;
+}
+
+type PassiveEnv = Record<string, string | undefined>;
+
+function resolveBoundedMs(
+  raw: string | undefined,
+  fallback: number,
+  min: number,
+  max: number
+): number {
+  const parsed = parseInt(raw ?? "", 10);
+  if (!Number.isFinite(parsed)) return fallback;
+  return Math.min(Math.max(parsed, min), max);
+}
+
+/**
+ * Sliding window for passive rows, in ms. Shorter than the 10-minute sweep
+ * interval so a recent failure dominates instead of being diluted.
+ */
+export function resolvePassiveWindowMs(env: PassiveEnv = process.env): number {
+  return resolveBoundedMs(env.PROXY_PASSIVE_WINDOW_MS, 5 * 60 * 1000, 60_000, 10 * 60_000 - 1);
+}
+
+/**
+ * Opt-in gate for the passive sweep skip. Off by default: with the flag
+ * unset the sweep probes every proxy exactly as before (no passive skip).
+ * Local env read (same convention as the recovery pass gate) so no registry
+ * file is touched.
+ */
+export function isPassiveSweepSkipEnabled(env: PassiveEnv = process.env): boolean {
+  return env.PROXY_HEALTH_PASSIVE_SKIP === "true";
+}

src/lib/proxyHealth/scheduler.ts

@@ -23,6 +23,7 @@
  */
 
 import { deleteProxyById, listProxies, updateProxy } from "@/lib/db/proxies";
+import { exportProxyLogsSince } from "@/lib/db/proxyLogs";
 import { isProxyLogIncludeIps } from "@/lib/proxyLogger";
 import {
   getRecentEgressSharingSummary,
@@ -67,6 +68,13 @@ import {
   isProxyHealthBlockedResetsStreakEnabled,
   isProxySkipRecentlyFailedEnabled,
 } from "@/shared/utils/featureFlags";
+import {
+  hasProvenSuccessWithoutAttributedFailure,
+  isPassiveSweepSkipEnabled,
+  passiveVerdictKey,
+  resolvePassiveWindowMs,
+  type PassiveLogRow,
+} from "./passiveVerdict.ts";
 
 // #6246: a HEAD to the public probe target through a legit (often loaded) proxy
 // can exceed a few seconds; the old 5s ceiling produced false negatives that
@@ -538,6 +546,89 @@ async function testOneProxy(proxy: {
   }
 }
 
+/**
+ * Passive skip proof from production traffic, grouped by proxy endpoint.
+ * Filled from the persisted request log once per sweep: an endpoint whose
+ * window shows recent single-provider success and no attributed failure
+ * skips the redundant live probe. A recent failure never skips: it keeps
+ * the probe that alone can confirm it.
+ */
+interface EndpointVerdict {
+  skip: boolean;
+  /** Distinct providers observed for the endpoint; skips need exactly one. */
+  providers: string[];
+}
+
+function readPassiveVerdicts(
+  proxies: Array<{ host: string; port: number }>
+): Map<string, EndpointVerdict> {
+  const verdicts = new Map<string, EndpointVerdict>();
+  const windowMs = resolvePassiveWindowMs();
+  const now = Date.now();
+  const since = new Date(now - windowMs).toISOString();
+  let rows: Record<string, unknown>[] = [];
+  try {
+    // Bounded by the short window via idx_pl_timestamp; never let a DB
+    // hiccup fail the sweep — fall back to live probes (no skip anywhere).
+    rows = exportProxyLogsSince(since);
+  } catch (error) {
+    console.error(`${LOG_PREFIX} Passive verdicts skipped:`, error);
+    return verdicts;
+  }
+  const endpoints = new Set(proxies.map((p) => passiveVerdictKey(p.host, p.port)));
+  const byEndpoint = new Map<string, PassiveLogRow[]>();
+  for (const row of rows) {
+    const host = typeof row.proxy_host === "string" ? row.proxy_host : null;
+    const port = typeof row.proxy_port === "number" ? row.proxy_port : null;
+    if (!host || !port || !endpoints.has(passiveVerdictKey(host, port))) continue;
+    const key = passiveVerdictKey(host, port);
+    const list = byEndpoint.get(key) ?? [];
+    list.push(row as PassiveLogRow);
+    byEndpoint.set(key, list);
+  }
+  for (const [key, keyedRows] of byEndpoint) {
+    const providers = [...new Set(keyedRows.map((r) => r.provider).filter((p) => p))];
+    const skip = hasProvenSuccessWithoutAttributedFailure(keyedRows) && providers.length === 1;
+    verdicts.set(key, { skip, providers: providers as string[] });
+  }
+  return verdicts;
+}
+
+/** Test-only: skip lookup seam for sweep-level tests. */
+export type PassiveVerdictReader = (proxyId: string) => { skip: boolean; providers: string[] };
+
+let passiveReader: PassiveVerdictReader | null = null;
+
+/** Test-only: override the skip lookup. */
+export function __setPassiveVerdictReaderForTesting(fn: PassiveVerdictReader | null): void {
+  passiveReader = fn;
+}
+
+/**
+ * Passive skip check for one proxy: recent production traffic already proves
+ * this endpoint, so the live probe is redundant — but only when the window
+ * shows a single provider (a success for one provider never skips the probe
+ * another provider would need). A recent failure never skips: the live probe
+ * stays the one that can confirm it and feed auto-disable/auto-remove.
+ */
+function resolvePassiveSkip(
+  proxy: { id: string; host: string; port: number },
+  passiveVerdicts: Map<string, EndpointVerdict>
+): boolean {
+  const endpointVerdict = passiveReader
+    ? passiveReader(proxy.id)
+    : (passiveVerdicts.get(`${proxy.host}:${proxy.port}`) ?? {
+        skip: false,
+        providers: [] as string[],
+      });
+  return endpointVerdict.skip && isSingleProviderSkip(endpointVerdict);
+}
+
+/** A proven endpoint skips the probe only with exactly one observed provider. */
+function isSingleProviderSkip(endpointVerdict: EndpointVerdict): boolean {
+  return endpointVerdict.providers.length === 1;
+}
+
 async function sweep(): Promise<void> {
   // #10677: anonymous egress-sharing signal from persisted proxy_logs (no live
   // probes). Logged only when sharing exists — the sweep line is a warning
@@ -559,6 +650,13 @@ async function sweep(): Promise<void> {
   pruneBlockedHistory(proxies.map((proxy) => proxy.id));
   if (proxies.length === 0) return;
 
+  // Passive verdicts from production traffic (opt-in): an endpoint already
+  // judged by recent traffic skips the live probe. Scheduling only — skips
+  // are partitioned before the probe phase, so they never touch the failure
+  // streak, the cross-proxy evidence, the promotion tally, or the sweep
+  // verdict display.
+  const passiveEnabled = isPassiveSweepSkipEnabled();
+  const passiveVerdicts = passiveEnabled ? readPassiveVerdicts(proxies) : new Map();
   const failureMap = getFailureMap();
   const removeAfter = getRemoveAfter();
   const autoRemove = isAutoRemoveEnabled();
@@ -567,7 +665,18 @@ async function sweep(): Promise<void> {
 
   // Phase 1 — collect raw probe results across all batches WITHOUT deciding
   // (cross-proxy evidence requires every response of the target first).
-  const collected = await collectProbeResults(proxies, async (proxy) => {
+  // Passive skips are partitioned before collecting: they never enter the
+  // batching, the failure streak, the cross-proxy evidence, the promotion
+  // tally, or the sweep verdict display — they are only counted apart below.
+  const skippedIds = new Set<string>();
+  if (passiveEnabled) {
+    for (const proxy of proxies) {
+      if (resolvePassiveSkip(proxy, passiveVerdicts)) skippedIds.add(proxy.id);
+    }
+  }
+  const toProbe = proxies.filter((proxy) => !skippedIds.has(proxy.id));
+  const passiveSkipped = skippedIds.size;
+  const collected = await collectProbeResults(toProbe, async (proxy) => {
     const { outcome, status, target } = await testOneProxy(proxy);
     // Ledger: only a sweep-observed 429 with a usable key is recorded.
     // No memory write happens on fail/hang here, except a promoted
@@ -578,11 +687,10 @@ async function sweep(): Promise<void> {
     return { id: proxy.id, proxy, outcome, status, target };
   });
 
-  // Previous-generation evidence is replaced wholesale (never merged): the
-  // current sweep's answered targets become generation N-1 for the next sweep.
-  // Lift the abstention only where proof exists (same sweep or the
-  // immediately previous generation). Received HTTP statuses are never
-  // reclassified — only status-less `inconclusive` probes can become `fail`.
+  // Only live probes feed the decision phase and the evidence: skipped
+  // endpoints contribute nothing. Previous-generation evidence is replaced
+  // wholesale (never merged): the current sweep's answered targets become
+  // generation N-1 for the next sweep.
   const { tested, alive, inconclusive, blocked, hangs, removed, disabled, promoted } =
     await decideCollectedResults(collected, {
       failureMap,
@@ -596,7 +704,8 @@ async function sweep(): Promise<void> {
   console.log(
     `${LOG_PREFIX} Sweep complete: ${tested} tested, ${alive} alive, ` +
       `${blocked} refused by target, ${inconclusive} inconclusive, ${promoted} promoted, ` +
-      `${removed} auto-removed, ${disabled} auto-disabled`
+      `${removed} auto-removed, ${disabled} auto-disabled` +
+      `${passiveSkipped > 0 ? `, ${passiveSkipped} passive-skipped (recent success)` : ""}`
   );
   if (hangs > 0) {
     console.debug(`${LOG_PREFIX} stalled handshakes observed: ${hangs}`);