fix(resilience): per-model 402 on a passthrough gateway no longer terminalizes the whole connection - #12266
Merged
diegosouzapw merged 2 commits intoSep 1, 2026
Conversation
…minalizes the whole connection 402 variant of diegosouzapw#3027. Passthrough/gateway providers that multiplex many models behind one credential (kilo-gateway, ollama-cloud, etc.) can 402 on a single PAID model while free models on the same key remain perfectly usable. Previously any 402 unconditionally set the connection to a terminal `credits_exhausted` status, which is never auto-recovered without an operator reset — taking out every remaining model on that provider, amplified further inside combo routing (measured: one 402 removed 9 of 14 fallback targets in a real combo, dropping success rate from 98.3% to 74.2% on a fixed load test per the issue report). Root cause (matches the issue's own analysis): 1. resolveTerminalConnectionStatus() returned "credits_exhausted" for ANY status === 402, with no per-model/passthrough check. 2. The generic per-model lockout gate (404/429/>=500) excluded 402. 3. The diegosouzapw#3027 403-branch is gated on `!terminalStatus` — since (1) already resolves a terminal status for any 402 before that branch runs, simply adding 402 to its condition alone would not have fired. Fix: - resolveTerminalConnectionStatus() now takes isPerModelQuotaProvider and skips the connection-wide terminal path for a bare `status === 402` when true, letting it fall through to the per-model lockout branch instead. An explicit result.creditsExhausted (a provider's own classification, independent of HTTP status) is untouched and remains unconditionally terminal. - Extended the existing diegosouzapw#3027 per-model lockout branch to also handle 402 (reason "credits" vs "forbidden" for 403), reusing the same cooldown/lockout machinery and log format. - Single-credential (non-passthrough) providers are unaffected: isPerModelQuotaProvider is false there, so a 402 still terminalizes the connection as before — that behavior is deliberate for prepaid API keys (diegosouzapw#5239 / diegosouzapw#10616). Also checked the issue's 4th root cause (terminal statuses never auto-recovering) against the current codebase: connectionRecovery.ts already has a 30-minute credits_exhausted reprobe (isCreditsExhaustedReprobeCandidate) that the issue's report — filed against v3.8.49 — didn't account for. The other two files it names (rateLimit.ts's clearStaleCrashCooldowns, tokenHealthCheck.ts's OAuth-refresh skip) legitimately exclude credits_exhausted for unrelated reasons and are not bugs. Moot regardless: this fix prevents credits_exhausted from being set at all for the passthrough case, so no recovery wait is needed in the first place. Tests: tests/unit/auth-passthrough-per-model-402-12242.test.ts, modeled on the existing diegosouzapw#3027 precedent test (real DB-backed integration test via auth.markAccountUnavailable). Covers: paid-model-only lockout with free model unaffected, a subsequent free-model request succeeding after a sibling paid-model 402, single-credential 402 still fully terminal, and no connection-wide backoff escalation on repeated 402s. Verified: - node --import tsx/esm --test tests/unit/auth-passthrough-per-model-402-12242.test.ts: 4/4 pass - All related pre-existing tests (auth-ollama-cloud-per-model-403-3027, auth-terminal-status, openrouter-free-model-credits-exhausted, vertex-passthrough-model-lockout, 10347-embed-402-cooldown): 27/27 pass, no regressions - npm run typecheck:core: 0 errors - npm run check:cycles: no cycles - eslint (auth.ts + new test file, with project suppressions): 0 errors Fixes diegosouzapw#12242
…chet auth.ts - stryker.conf.json: add tests/unit/auth-passthrough-per-model-402-12242.test.ts to tap.testFiles in its alphabetical slot - auth.ts: extract the diegosouzapw#12242 connection-wide 402 decision into the pure helper isConnectionWideCreditsExhausted() so resolveTerminalConnectionStatus stays within the cyclomatic ratchet (file back to the base's 11 violations)
jacobsparts
pushed a commit
to jacobsparts/OmniRoute
that referenced
this pull request
Sep 9, 2026
…minalizes the whole connection (diegosouzapw#12266) * fix(resilience): per-model 402 on a passthrough gateway no longer terminalizes the whole connection 402 variant of diegosouzapw#3027. Passthrough/gateway providers that multiplex many models behind one credential (kilo-gateway, ollama-cloud, etc.) can 402 on a single PAID model while free models on the same key remain perfectly usable. Previously any 402 unconditionally set the connection to a terminal `credits_exhausted` status, which is never auto-recovered without an operator reset — taking out every remaining model on that provider, amplified further inside combo routing (measured: one 402 removed 9 of 14 fallback targets in a real combo, dropping success rate from 98.3% to 74.2% on a fixed load test per the issue report). Root cause (matches the issue's own analysis): 1. resolveTerminalConnectionStatus() returned "credits_exhausted" for ANY status === 402, with no per-model/passthrough check. 2. The generic per-model lockout gate (404/429/>=500) excluded 402. 3. The diegosouzapw#3027 403-branch is gated on `!terminalStatus` — since (1) already resolves a terminal status for any 402 before that branch runs, simply adding 402 to its condition alone would not have fired. Fix: - resolveTerminalConnectionStatus() now takes isPerModelQuotaProvider and skips the connection-wide terminal path for a bare `status === 402` when true, letting it fall through to the per-model lockout branch instead. An explicit result.creditsExhausted (a provider's own classification, independent of HTTP status) is untouched and remains unconditionally terminal. - Extended the existing diegosouzapw#3027 per-model lockout branch to also handle 402 (reason "credits" vs "forbidden" for 403), reusing the same cooldown/lockout machinery and log format. - Single-credential (non-passthrough) providers are unaffected: isPerModelQuotaProvider is false there, so a 402 still terminalizes the connection as before — that behavior is deliberate for prepaid API keys (diegosouzapw#5239 / diegosouzapw#10616). Also checked the issue's 4th root cause (terminal statuses never auto-recovering) against the current codebase: connectionRecovery.ts already has a 30-minute credits_exhausted reprobe (isCreditsExhaustedReprobeCandidate) that the issue's report — filed against v3.8.49 — didn't account for. The other two files it names (rateLimit.ts's clearStaleCrashCooldowns, tokenHealthCheck.ts's OAuth-refresh skip) legitimately exclude credits_exhausted for unrelated reasons and are not bugs. Moot regardless: this fix prevents credits_exhausted from being set at all for the passthrough case, so no recovery wait is needed in the first place. Tests: tests/unit/auth-passthrough-per-model-402-12242.test.ts, modeled on the existing diegosouzapw#3027 precedent test (real DB-backed integration test via auth.markAccountUnavailable). Covers: paid-model-only lockout with free model unaffected, a subsequent free-model request succeeding after a sibling paid-model 402, single-credential 402 still fully terminal, and no connection-wide backoff escalation on repeated 402s. Verified: - node --import tsx/esm --test tests/unit/auth-passthrough-per-model-402-12242.test.ts: 4/4 pass - All related pre-existing tests (auth-ollama-cloud-per-model-403-3027, auth-terminal-status, openrouter-free-model-credits-exhausted, vertex-passthrough-model-lockout, 10347-embed-402-cooldown): 27/27 pass, no regressions - npm run typecheck:core: 0 errors - npm run check:cycles: no cycles - eslint (auth.ts + new test file, with project suppressions): 0 errors Fixes diegosouzapw#12242 * chore(quality): register 402 per-model test in stryker tap and de-ratchet auth.ts - stryker.conf.json: add tests/unit/auth-passthrough-per-model-402-12242.test.ts to tap.testFiles in its alphabetical slot - auth.ts: extract the diegosouzapw#12242 connection-wide 402 decision into the pure helper isConnectionWideCreditsExhausted() so resolveTerminalConnectionStatus stays within the cyclomatic ratchet (file back to the base's 11 violations) --------- Co-authored-by: OmniRoute Dev <dev@local> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…minalizes the whole connection (diegosouzapw#12266) * fix(resilience): per-model 402 on a passthrough gateway no longer terminalizes the whole connection 402 variant of diegosouzapw#3027. Passthrough/gateway providers that multiplex many models behind one credential (kilo-gateway, ollama-cloud, etc.) can 402 on a single PAID model while free models on the same key remain perfectly usable. Previously any 402 unconditionally set the connection to a terminal `credits_exhausted` status, which is never auto-recovered without an operator reset — taking out every remaining model on that provider, amplified further inside combo routing (measured: one 402 removed 9 of 14 fallback targets in a real combo, dropping success rate from 98.3% to 74.2% on a fixed load test per the issue report). Root cause (matches the issue's own analysis): 1. resolveTerminalConnectionStatus() returned "credits_exhausted" for ANY status === 402, with no per-model/passthrough check. 2. The generic per-model lockout gate (404/429/>=500) excluded 402. 3. The diegosouzapw#3027 403-branch is gated on `!terminalStatus` — since (1) already resolves a terminal status for any 402 before that branch runs, simply adding 402 to its condition alone would not have fired. Fix: - resolveTerminalConnectionStatus() now takes isPerModelQuotaProvider and skips the connection-wide terminal path for a bare `status === 402` when true, letting it fall through to the per-model lockout branch instead. An explicit result.creditsExhausted (a provider's own classification, independent of HTTP status) is untouched and remains unconditionally terminal. - Extended the existing diegosouzapw#3027 per-model lockout branch to also handle 402 (reason "credits" vs "forbidden" for 403), reusing the same cooldown/lockout machinery and log format. - Single-credential (non-passthrough) providers are unaffected: isPerModelQuotaProvider is false there, so a 402 still terminalizes the connection as before — that behavior is deliberate for prepaid API keys (diegosouzapw#5239 / diegosouzapw#10616). Also checked the issue's 4th root cause (terminal statuses never auto-recovering) against the current codebase: connectionRecovery.ts already has a 30-minute credits_exhausted reprobe (isCreditsExhaustedReprobeCandidate) that the issue's report — filed against v3.8.49 — didn't account for. The other two files it names (rateLimit.ts's clearStaleCrashCooldowns, tokenHealthCheck.ts's OAuth-refresh skip) legitimately exclude credits_exhausted for unrelated reasons and are not bugs. Moot regardless: this fix prevents credits_exhausted from being set at all for the passthrough case, so no recovery wait is needed in the first place. Tests: tests/unit/auth-passthrough-per-model-402-12242.test.ts, modeled on the existing diegosouzapw#3027 precedent test (real DB-backed integration test via auth.markAccountUnavailable). Covers: paid-model-only lockout with free model unaffected, a subsequent free-model request succeeding after a sibling paid-model 402, single-credential 402 still fully terminal, and no connection-wide backoff escalation on repeated 402s. Verified: - node --import tsx/esm --test tests/unit/auth-passthrough-per-model-402-12242.test.ts: 4/4 pass - All related pre-existing tests (auth-ollama-cloud-per-model-403-3027, auth-terminal-status, openrouter-free-model-credits-exhausted, vertex-passthrough-model-lockout, 10347-embed-402-cooldown): 27/27 pass, no regressions - npm run typecheck:core: 0 errors - npm run check:cycles: no cycles - eslint (auth.ts + new test file, with project suppressions): 0 errors Fixes diegosouzapw#12242 * chore(quality): register 402 per-model test in stryker tap and de-ratchet auth.ts - stryker.conf.json: add tests/unit/auth-passthrough-per-model-402-12242.test.ts to tap.testFiles in its alphabetical slot - auth.ts: extract the diegosouzapw#12242 connection-wide 402 decision into the pure helper isConnectionWideCreditsExhausted() so resolveTerminalConnectionStatus stays within the cyclomatic ratchet (file back to the base's 11 violations) --------- Co-authored-by: OmniRoute Dev <dev@local> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12242
402 variant of #3027. Passthrough/gateway providers that multiplex many models behind one credential (kilo-gateway, ollama-cloud, etc.) can 402 on a single paid model while free models on the same key remain perfectly usable. Previously any 402 unconditionally set the connection to a terminal
credits_exhaustedstatus, taking out every remaining model on that provider.Root cause (matches the issue's own analysis)
resolveTerminalConnectionStatus()returned"credits_exhausted"for ANYstatus === 402, with no per-model/passthrough check.!terminalStatus— since (1) already resolves a terminal status for any 402, adding 402 to that branch's condition alone would not have fired.Fix
resolveTerminalConnectionStatus()now takesisPerModelQuotaProviderand skips the connection-wide terminal path for a barestatus === 402when true, falling through to the per-model lockout branch instead. Explicitresult.creditsExhausted(a provider's own classification) is untouched and remains unconditionally terminal.reason: "credits"vs"forbidden"for 403), reusing the same cooldown/lockout machinery.One correction to the original report
The issue's 4th cited root cause (terminal statuses never auto-recovering) doesn't hold against the current
release/v3.8.51—connectionRecovery.tsalready has a 30-minutecredits_exhaustedreprobe (isCreditsExhaustedReprobeCandidate) that the original report (filed against v3.8.49) didn't account for. The other two files it names legitimately excludecredits_exhaustedfor unrelated reasons (crash-cooldown reset, OAuth token refresh) and aren't bugs. Moot either way — this fix preventscredits_exhaustedfrom being set at all for the passthrough case.Testing
tests/unit/auth-passthrough-per-model-402-12242.test.ts— new, modeled on the existing [BUG] ollama-cloud per-model subscription 403 cools down the whole connection (free models knocked out) #3027 precedent (real DB-backed integration test viaauth.markAccountUnavailable). 4/4 pass: paid-model-only lockout, free model on same connection unaffected, single-credential 402 still fully terminal, no connection-wide backoff escalation on repeats.auth-ollama-cloud-per-model-403-3027,auth-terminal-status,openrouter-free-model-credits-exhausted,vertex-passthrough-model-lockout,10347-embed-402-cooldown): 27/27 pass, no regressions.npm run lint: cleannpm run typecheck:core: 0 errorsnpm run check:cycles: no cyclesChecklist
changelog.d/fixes/12242-passthrough-402-per-model-lockout.md