Skip to content

fix(proxies): set aside egress after repeated transport failures with cross-egress success - #14802

Merged
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/transport-setaside-cross-evidence
Sep 25, 2026
Merged

diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/transport-setaside-cross-evidence

Conversation

@maxmad64bis

@maxmad64bis maxmad64bis commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #14547

Summary

A pool member that is dead behind a healthy local probe was never set aside: tagged transport failures carry no upstream status, so the outcome path ignores them and the refusal memory never learns. Final tagged failures are now recorded per egress and destination host at the single end-of-request site, with successes through a different egress as counter-proof, and a member is set aside under a new transport refusal kind only when at least 3 tagged failures coincide with a completed response through a different egress inside a 5-minute window. Opt-in via the existing flag; retried-then-recovered attempts, isolated failures, globally unreachable destinations, direct egress and single-member pools never write.

Related Issues

  • No linked issue — no existing report covers transport-tagged failures condemned only with cross-egress evidence (upstream searches transport set-aside / tagProxyUnreachable returned 0 covering issues).
  • Related to fix(proxyHealth): promote connection failures with cross-proxy evidence #14776 (open, disjoint files): same cross-proof idea on sweep verdicts rather than the transport path; no collision, no action required.

Validation

  • Change type: routing
  • Focused tests and category gates from the golden path
  • npm run lint (0 errors on touched files)
  • Reconciled with the current active release base (release/v3.8.51); focused checks rerun afterward
  • Production-code changes include a new automated test in this PR

Tests Added Or Updated

  • New tests/unit/proxy-transport-setaside-cross-evidence.test.ts (10 cases): condemnation on 3 failures plus cross-egress success; no set-aside without cross-evidence; isolated failures (1-2) ignored; success through the same egress is not evidence; expired window ignored; flag-off writes nothing; null key ignored; single-member pool ignored with documented fallback when size is unknown; cooldown curve 60s then 120s through the real writer; store bounds. Failing before the change (helpers absent), 10/10 green after.
  • Traffic simulation through the real fetch path (4/4).

Coverage Notes

  • The change (open-sse/utils/proxyRefusalMemory.ts, open-sse/utils/proxyFetch.ts, src/sse/handlers/proxyOutcomeMemory.ts) is covered by the 2 new files above; neighbors kept green (single-file runs): refusal-memory, quota-tuning, outcome-memory, health-refusal-memory, and health-recovery suites 48/48; outcome-memory pair 18/18; undici-retry 10/10; pool-skips-refused 12/12; opencode refusal-memory 7/7; health-refusal 4/4. Lint clean on all touched files; core typecheck clean; cycle check OK.

Reviewer Notes

  • Pointers: failure decision at open-sse/utils/proxyFetch.ts:1239 (tag read off the sanitized error) calling the helper at proxyRefusalMemory.ts:357, success hook at :1203 calling the helper at proxyRefusalMemory.ts:374; evidence stores and threshold predicate in open-sse/utils/proxyRefusalMemory.ts (:267-269 constants, :290/:302/:319 helpers, policy entry at :69, null-key cases at proxyEgressKey :133-153); single gated writer in src/sse/handlers/proxyOutcomeMemory.ts:56-66. Counter-proof is any completed dispatcher response, not strictly 2xx — deliberately broader than the 5-minute/2xx sketch in the design; template-style destinations remain the residual (conservative under-count, see Coverage Notes).
  • The poolSize guard is advisory in production (size not yet threaded); condemning still requires another key's success.
  • Docs Gates red is inherited (🔴 Release branch not green: release/v3.8.51 #14547); file-size rebaselined 1334→1341 (own key).

@maxmad64bis
maxmad64bis force-pushed the fix/transport-setaside-cross-evidence branch from 1d2a273 to 29afea5 Compare September 24, 2026 20:42
@maxmad64bis
maxmad64bis marked this pull request as draft September 24, 2026 20:43
@maxmad64bis
maxmad64bis force-pushed the fix/transport-setaside-cross-evidence branch from 29afea5 to 4ca04a6 Compare September 24, 2026 21:59
@maxmad64bis
maxmad64bis force-pushed the fix/transport-setaside-cross-evidence branch from 4ca04a6 to 44a3f23 Compare September 24, 2026 22:11
@maxmad64bis
maxmad64bis marked this pull request as ready for review September 25, 2026 08:08
@diegosouzapw

Copy link
Copy Markdown
Owner

Thanks — the cross-egress evidence rule is careful and the traffic-simulation test is a great addition. Three things: (1) open-sse now lazily imports @/sse/handlers/proxyOutcomeMemory, which inverts the layering — could the callback be injected instead? (2) success evidence is recorded on every proxied request even with the flag off; gating it would avoid the hot-path cost. (3) poolSize is never passed, so the single-member guard described in the PR body isn't actually active. Please also confirm the new tests fail on the base version.

…ort-setaside-cross-evidence

Moves the transport outcome decision into open-sse/utils/proxyTransportOutcome.ts
so open-sse no longer imports @/sse, and gates evidence recording on
PROXY_SKIP_RECENTLY_FAILED.
… kind

The tip's transition listeners (diegosouzapw#14755) typed the kind as ip_quota_429 |
proxy_unreachable only, so the new transport refusal kind no longer compiled.
Widen ProxyTransitionKind and the proxy.set_aside payload reason, and cover
the listener announcement with a test.
# Conflicts:
#	config/quality/file-size-baseline.json
@diegosouzapw
diegosouzapw merged commit d6293f4 into diegosouzapw:release/v3.8.51 Sep 25, 2026
6 of 7 checks passed
diegosouzapw added a commit to maxmad64bis/OmniRoute that referenced this pull request Sep 25, 2026
Pin the two diegosouzapw#14802 flag-off cases in proxy-transport-setaside-cross-evidence
to PROXY_SKIP_RECENTLY_FAILED=false: with this PR the flag defaults to on, so
deleting the variable no longer means off.
diegosouzapw pushed a commit that referenced this pull request Sep 25, 2026
Flips PROXY_SKIP_RECENTLY_FAILED to default on (=false restores plain rotation). Maintainer rework: reconciled with release/v3.8.51 after #14809 (kept its lastResort429 spare alongside the default-on comment) and #14802/#14755 (their flag-off test cases re-pinned from 'delete env' to an explicit =false, since unset now means on). Red->green: proxy-refusal-memory-default 2/3 red on base, 3/3 green. Focused proxy/opencode/refusal-memory suites 258/261 then 36/36 on the 3 reruns, plus 127/127 after the #14802 merge; typecheck:core and open-sse typecheck clean; file-size only the inherited opencode.ts tip ceiling (1354>1338, red on the pure tip). Thank you @maxmad64bis!
diegosouzapw added a commit to maxmad64bis/OmniRoute that referenced this pull request Sep 25, 2026
…gress-visibility

Locales rebuilt as the release tip plus only this PR's keys. NoAuthAccountCard
keeps the tip's effective-egress shield (diegosouzapw#14796) and appends the set-aside note
to its title. The transport set-aside kind from diegosouzapw#14802 now maps to the
proxy-unreachable label instead of printing the raw kind.
diegosouzapw pushed a commit that referenced this pull request Sep 25, 2026
…nd preference order (#14754)

Maintainer rework: reconciled with release/v3.8.51 — locales rebuilt as the tip plus only this PR's keys (proxyRegistry.pool*, noAuthProvider.proxySetAside; bs also gets the 5 requestTimeline translations the tip was missing), 0 __MISSING__; NoAuthAccountCard keeps the tip's effective-egress shield (#14796) and appends the set-aside note to its title; the new 'transport' set-aside kind from #14802 maps to the proxy-unreachable label instead of the raw kind (new vitest case, red without the mapping). Tests: pool-visibility + refusal/transport suites 79/79, UI vitest 16/16; typecheck:core clean, open-sse typecheck OK, i18n ratio/keys/new-keys PASS, file-size OK. Thank you @maxmad64bis!
@maxmad64bis
maxmad64bis deleted the fix/transport-setaside-cross-evidence branch September 30, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants