fix(sse): repair orphaned PROVIDER_BREAKER_FAILURE_STATUSES reference on the all-rate-limited path - #8390
Merged
diegosouzapw merged 1 commit intoJul 24, 2026
Conversation
…d all-rate-limited breaker path Root cause: #8013 extracted shouldTripProviderBreakerForResult() from src/sse/handlers/chat.ts into the new src/sse/handlers/chatPredicates.ts, taking the (non-exported) const PROVIDER_BREAKER_FAILURE_STATUSES with it. A second, independent use of that const survived in chat.ts's handleSingleModelChat(), in the "all credentials rate-limited" block (~line 1340) — that reference was left orphaned by the extraction. Production impact: any request where every credential for a provider+model is simultaneously rate-limited throws `ReferenceError: PROVIDER_BREAKER_FAILURE_STATUSES is not defined` at runtime in that code path. Concretely this meant: - breaker._onFailure() was unreachable on the all-rate-limited path, so the provider circuit breaker could not trip from it - the ReferenceError propagated up and got mapped to a generic 502, masking the real 503 upstream-unavailable status in combo responses - the issue-agent route surfaced a generic 400 instead of the actual 429 provider-rate-limited response Fix: export PROVIDER_BREAKER_FAILURE_STATUSES from chatPredicates.ts and add it to chat.ts's existing import block from that module. No behavior change — the classification set ([408, 500, 502, 503, 504]) is unchanged, this only repairs the broken reference. Also re-points tests/unit/nvidia-quota-phase1.test.ts's regex-based declaration check at chatPredicates.ts, where the const now actually lives (it previously read chat.ts via fs+regex and silently failed to find the declaration). The regex and the classification assertions themselves are unchanged — this test still proves 429 is excluded from the whole-provider breaker. Refs #8013
This was referenced Jul 24, 2026
MumuTW
added a commit
to MumuTW/OmniRoute
that referenced
this pull request
Jul 25, 2026
… import Rebased onto release/v3.8.49 and reduced to the test file only — the production fix landed in diegosouzapw#8390. Asserts chatPredicates exports the Set with the documented statuses and that chat.ts imports the symbol cleanly.
MumuTW
added a commit
to MumuTW/OmniRoute
that referenced
this pull request
Jul 26, 2026
… import Rebased onto release/v3.8.49 and reduced to the test file only — the production fix landed in diegosouzapw#8390. Asserts chatPredicates exports the Set with the documented statuses and that chat.ts imports the symbol cleanly.
diegosouzapw
pushed a commit
that referenced
this pull request
Jul 27, 2026
…8611) The `compat-build-26` job in nightly-compat.yml is the only place in the CI matrix that runs `npm run build` on Node 26 (ci.yml pins CI_NODE_VERSION=24). It failed every nightly with the runner-reclaimed signature ("The runner has received a shutdown signal" / "The operation was canceled", no exit code), always at the same Turbopack compile phase — the classic OOM-kill pattern on the memory-constrained ubuntu-latest runner. Root cause: Turbopack's native (Rust, off-V8-heap) allocation is not bounded by --max-old-space-size and peaks far higher than webpack on OmniRoute's large module graph (#6409), heavier still under Node 26. Raising the heap does not help — the codebase's own documented escape hatch for RAM-constrained environments is the webpack fallback (OMNIROUTE_USE_TURBOPACK=0; see docs/reference/ENVIRONMENT.md and scripts/build/build-next-isolated.mjs). Wire that fallback into the Node 26 compat build: it still validates the app builds on Node 26 (the point of the job) at a much lower memory peak. Turbopack-on-Node-24 stays covered by ci.yml's build job. Adds a regression guard (tests/unit/nightly-compat-node26-webpack-8090.test.ts) asserting the job keeps the webpack fallback so it cannot silently regress. Class 1 of the triage (shard test failures) was already resolved by #8390, #8386, #8381, #8383. Closes #8090 Refs #6949 #6409
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…d all-rate-limited breaker path (diegosouzapw#8390) Root cause: diegosouzapw#8013 extracted shouldTripProviderBreakerForResult() from src/sse/handlers/chat.ts into the new src/sse/handlers/chatPredicates.ts, taking the (non-exported) const PROVIDER_BREAKER_FAILURE_STATUSES with it. A second, independent use of that const survived in chat.ts's handleSingleModelChat(), in the "all credentials rate-limited" block (~line 1340) — that reference was left orphaned by the extraction. Production impact: any request where every credential for a provider+model is simultaneously rate-limited throws `ReferenceError: PROVIDER_BREAKER_FAILURE_STATUSES is not defined` at runtime in that code path. Concretely this meant: - breaker._onFailure() was unreachable on the all-rate-limited path, so the provider circuit breaker could not trip from it - the ReferenceError propagated up and got mapped to a generic 502, masking the real 503 upstream-unavailable status in combo responses - the issue-agent route surfaced a generic 400 instead of the actual 429 provider-rate-limited response Fix: export PROVIDER_BREAKER_FAILURE_STATUSES from chatPredicates.ts and add it to chat.ts's existing import block from that module. No behavior change — the classification set ([408, 500, 502, 503, 504]) is unchanged, this only repairs the broken reference. Also re-points tests/unit/nvidia-quota-phase1.test.ts's regex-based declaration check at chatPredicates.ts, where the const now actually lives (it previously read chat.ts via fs+regex and silently failed to find the declaration). The regex and the classification assertions themselves are unchanged — this test still proves 429 is excluded from the whole-provider breaker. Refs diegosouzapw#8013
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…iegosouzapw#8611) The `compat-build-26` job in nightly-compat.yml is the only place in the CI matrix that runs `npm run build` on Node 26 (ci.yml pins CI_NODE_VERSION=24). It failed every nightly with the runner-reclaimed signature ("The runner has received a shutdown signal" / "The operation was canceled", no exit code), always at the same Turbopack compile phase — the classic OOM-kill pattern on the memory-constrained ubuntu-latest runner. Root cause: Turbopack's native (Rust, off-V8-heap) allocation is not bounded by --max-old-space-size and peaks far higher than webpack on OmniRoute's large module graph (diegosouzapw#6409), heavier still under Node 26. Raising the heap does not help — the codebase's own documented escape hatch for RAM-constrained environments is the webpack fallback (OMNIROUTE_USE_TURBOPACK=0; see docs/reference/ENVIRONMENT.md and scripts/build/build-next-isolated.mjs). Wire that fallback into the Node 26 compat build: it still validates the app builds on Node 26 (the point of the job) at a much lower memory peak. Turbopack-on-Node-24 stays covered by ci.yml's build job. Adds a regression guard (tests/unit/nightly-compat-node26-webpack-8090.test.ts) asserting the job keeps the webpack fallback so it cannot silently regress. Class 1 of the triage (shard test failures) was already resolved by diegosouzapw#8390, diegosouzapw#8386, diegosouzapw#8381, diegosouzapw#8383. Closes diegosouzapw#8090 Refs diegosouzapw#6949 diegosouzapw#6409
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…d all-rate-limited breaker path (diegosouzapw#8390) Root cause: diegosouzapw#8013 extracted shouldTripProviderBreakerForResult() from src/sse/handlers/chat.ts into the new src/sse/handlers/chatPredicates.ts, taking the (non-exported) const PROVIDER_BREAKER_FAILURE_STATUSES with it. A second, independent use of that const survived in chat.ts's handleSingleModelChat(), in the "all credentials rate-limited" block (~line 1340) — that reference was left orphaned by the extraction. Production impact: any request where every credential for a provider+model is simultaneously rate-limited throws `ReferenceError: PROVIDER_BREAKER_FAILURE_STATUSES is not defined` at runtime in that code path. Concretely this meant: - breaker._onFailure() was unreachable on the all-rate-limited path, so the provider circuit breaker could not trip from it - the ReferenceError propagated up and got mapped to a generic 502, masking the real 503 upstream-unavailable status in combo responses - the issue-agent route surfaced a generic 400 instead of the actual 429 provider-rate-limited response Fix: export PROVIDER_BREAKER_FAILURE_STATUSES from chatPredicates.ts and add it to chat.ts's existing import block from that module. No behavior change — the classification set ([408, 500, 502, 503, 504]) is unchanged, this only repairs the broken reference. Also re-points tests/unit/nvidia-quota-phase1.test.ts's regex-based declaration check at chatPredicates.ts, where the const now actually lives (it previously read chat.ts via fs+regex and silently failed to find the declaration). The regex and the classification assertions themselves are unchanged — this test still proves 429 is excluded from the whole-provider breaker. Refs diegosouzapw#8013
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…iegosouzapw#8611) The `compat-build-26` job in nightly-compat.yml is the only place in the CI matrix that runs `npm run build` on Node 26 (ci.yml pins CI_NODE_VERSION=24). It failed every nightly with the runner-reclaimed signature ("The runner has received a shutdown signal" / "The operation was canceled", no exit code), always at the same Turbopack compile phase — the classic OOM-kill pattern on the memory-constrained ubuntu-latest runner. Root cause: Turbopack's native (Rust, off-V8-heap) allocation is not bounded by --max-old-space-size and peaks far higher than webpack on OmniRoute's large module graph (diegosouzapw#6409), heavier still under Node 26. Raising the heap does not help — the codebase's own documented escape hatch for RAM-constrained environments is the webpack fallback (OMNIROUTE_USE_TURBOPACK=0; see docs/reference/ENVIRONMENT.md and scripts/build/build-next-isolated.mjs). Wire that fallback into the Node 26 compat build: it still validates the app builds on Node 26 (the point of the job) at a much lower memory peak. Turbopack-on-Node-24 stays covered by ci.yml's build job. Adds a regression guard (tests/unit/nightly-compat-node26-webpack-8090.test.ts) asserting the job keeps the webpack fallback so it cannot silently regress. Class 1 of the triage (shard test failures) was already resolved by diegosouzapw#8390, diegosouzapw#8386, diegosouzapw#8381, diegosouzapw#8383. Closes diegosouzapw#8090 Refs diegosouzapw#6949 diegosouzapw#6409
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
#8013 extracted
shouldTripProviderBreakerForResult()fromsrc/sse/handlers/chat.tsinto the newsrc/sse/handlers/chatPredicates.ts, taking the (non-exported)const PROVIDER_BREAKER_FAILURE_STATUSES = new Set([408, 500, 502, 503, 504])with it.A second, independent use of that const survived in
chat.ts'shandleSingleModelChat(), in the "all credentials rate-limited" block (~line 1340):The extraction removed the local declaration but left this second reference behind, orphaned.
Production impact
Any request where every credential for a provider+model is simultaneously rate-limited hits
ReferenceError: PROVIDER_BREAKER_FAILURE_STATUSES is not definedat runtime in this code path. Concretely:breaker._onFailure()was unreachable on the all-rate-limited path — the provider circuit breaker could never trip from it.ReferenceErrorpropagated up and got mapped to a generic502, masking the real503upstream-unavailable status in combo responses.400instead of the actual429provider-rate-limited response.This sits in the central chat resilience path (
handleSingleModelChat), so it fires for any provider whenever all of its credentials are simultaneously cooling down — not an edge case.Fix
Two-line production fix, no behavior change:
src/sse/handlers/chatPredicates.ts: exportPROVIDER_BREAKER_FAILURE_STATUSES.src/sse/handlers/chat.ts: add it to the existing import block from./chatPredicates.The classification set (
[408, 500, 502, 503, 504]) is unchanged — this only repairs the broken reference.Also re-points
tests/unit/nvidia-quota-phase1.test.ts's regex-based declaration check atchatPredicates.ts, where the const now actually lives (it previously readchat.tsvia fs+regex and silently failed to find the declaration — a stale test left behind by the same #8013 extraction). The regex and the classification assertions are unchanged; the test still proves429is excluded from the whole-provider breaker.Validation (TDD — red before fix, green after)
All 6 affected files run sequentially with
env -u OMNIROUTE_API_KEY:tests/unit/chat-rate-limit-body-lock.test.tsReferenceError)tests/unit/chat-cooldown-aware-retry.test.tsReferenceError)tests/unit/chat-combo-live-test.test.tsReferenceError)tests/unit/chat-route-coverage.test.tsReferenceError)tests/unit/issue-agent-route-execution.test.tstests/unit/nvidia-quota-phase1.test.tsAlso verified:
npm run typecheck:core— clean.npx eslinton the 3 changed files — clean.tests/unit/combo-breaker-429.test.ts(uses a deliberately independent, locally-declared const of the same name inopen-sse/services/combo/comboPredicates.ts, to avoid anopen-sse→src/sseimport cycle) — unaffected, still 9/9 green. Confirms this fix is contained to the orphaned single-model path and does not touch the separate combo-layer classification.Refs #8013