Repository navigation
fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal — cooldown with backoff instead of an instant ban (#12859) - #12864
Merged
diegosouzapw merged 10 commits intoSep 17, 2026
Conversation
added 2 commits
September 6, 2026 02:12
…refusal, not a ban
A single upstream 403 on the `claude` OAuth connection was classified
FORBIDDEN and written as the terminal `banned` connection state
(chatCore -> writeTerminalStatus). From then on every request to that
provider was short-circuited with "All 1 connection(s) banned by
upstream - please reconnect in the dashboard" without touching Anthropic,
until an operator reconnected.
Anthropic's OAuth surface answers a small fraction of otherwise-valid
requests with 403 {"type":"permission_error","message":"Request not
allowed"}. On the reporting install the same token returned 200 forty
seconds before the 403 and again right after the connection was
re-enabled; a revoked or expired token is a 401 authentication_error, not
this. It is a refusal of one request, not of the credential.
Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED
(scoped to provider `claude` and the "Request not allowed" body) and list
that type in authTerminalStatus.isNonTerminalProviderError, mirroring the
Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls
through to the next target for the failing request; the connection stays
active for the next one. Any other claude 403 keeps its previous
classification.
Tests: error-classifier.test.ts covers the Anthropic body, the
gateway-flattened "[403]: Request not allowed" message, the same body from
a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged),
and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins
resolveTerminalConnectionStatus() -> null for the new type even with a
`permanent` fallback verdict, and `banned` for a generic claude 403.
…ECTED (cheaper-inference#12859) Not "ignore the 403" either: if Anthropic ever made "Request not allowed" systematic, re-sending every request into it would be the wrong thing to do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly: - exclude the connection via setConnectionRateLimitUntil for a growing cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a reconnect, and a systematic one cannot become a stream of 403s; - escalate to the terminal `banned` state only for 3 refusals within a 60-minute window (services/requestRejectedStreak.ts, in-memory per connection; a restart forgets the streak, erring towards more cooldowns rather than an operator-undone ban), with a last_error that says so; - probe-origin failures record but never cool down or ban (cheaper-inference#9817). The existing "request not allowed" text rule (5 s) is unaffected: markAccountUnavailable skips a connection that already has a future rateLimitedUntil, so the minute-scale cooldown written here wins. Tests: request-rejected-streak.test.ts pins the window/threshold/backoff arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives the real chat route against a mocked 403 upstream on a `claude` OAuth connection: 300 s cooldown, then 900 s, then banned on the third refusal; a different claude 403 body still bans on the first response.
added 6 commits
September 6, 2026 17:22
… register its tests for mutation coverage chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts (chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new DB-backed tests so their mutant kills count (check:mutation-test-coverage).
…d honest (cheaper-inference#12859 review) Review findings on the first cut of the REQUEST_REJECTED handling: - A burst of in-flight requests that all got the 403 within seconds produced streak 1, 2, 3 and a ban from one upstream event. The streak now counts cooldown *episodes*: a refusal that lands while the connection is already excluded is the same event and is not counted. - Nothing reset the streak on a healthy response, so sporadic refusals on a busy install could still accumulate to a ban. chatHelpers' onRequestSuccess now clears it (only a real success does - the recovery tick's clearAccountError is an elapsed cooldown, not a success). Clearing the cooldown by hand in the dashboard clears it too. - The third rung of the ladder was unreachable (the third refusal escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next to the existing 5 s "request not allowed" rule, with a note on why that rule is superseded for claude. The 60-min window becomes a 24 h staleness bound - "consecutive" is defined by successes, not by time. - Probe-origin refusals no longer touch the streak (cheaper-inference#9817). - The cooldown is written like every other connection-level cooldown: ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the dashboard shows the countdown and the recovery tick restores "active". - One refusal is re-seeded from the persisted row after a restart so a crash loop cannot reset the count on every boot. Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience row mention the streak module. Tests cover the burst, the success reset, the seed, and the ISO/unavailable shape end-to-end through the chat route.
…ween GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch The release tip moved chatCore.ts to its frozen 5984 lines, so the REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and GCP_PROJECT_REQUIRED branches were the same eight statements with different constants and log wording; both now call open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown (behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends 9 lines below the base it branched from.
This was referenced Sep 10, 2026
Collaborator
|
This is a strong fix for a real production incident — the "one 403 among One blocker before merge: GitHub currently reports this PR as |
Resolve base-drift merge conflict in open-sse/handlers/chatCore.ts and open-sse/services/errorClassifier.ts against the current tip: the GEO_BLOCKED/GCP_PROJECT_REQUIRED/REQUEST_REJECTED branch now uses the shared excludeConnectionForCooldown()/handleRequestRejectedFailure() leaves, and PROVIDER_ERROR_TYPES keeps the `as const` + ERROR_TYPE_CONTRACT vocabulary landed on the tip since this branch was cut. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
… its size ceiling After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a frozen ceiling of 6146. Condense the explanatory comments this PR added to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
diegosouzapw
merged commit Sep 17, 2026
dd70dbd
into
cheaper-inference:release/v3.8.51
11 of 16 checks passed
diegosouzapw
added a commit
that referenced
this pull request
Sep 18, 2026
…the write, drain the 09-18 base-reds (#14101) * fix(quality): drain the 09-18 base-reds, part 1 — thinking gate parity, inventory, webpack externals Reproduced on the clean tip 7cc454d before touching anything. Five of the failures trace to one commit, #12905 (b7192b7): it gated thinking-block emission on `requestedThinking === true` in the streaming translator, while its own non-streaming path documents `undefined` as the legacy caller shape that keeps "always a thinking block". The two paths disagreed on the same input, and the streaming side also synthesized the reasoning into a TEXT block for that legacy shape. chatCore always resolves a boolean, so production never sends `undefined` — but every direct caller and the older #5786 suites do. Aligned the streaming gate to the documented tri-state: `false` suppresses, `true` and `undefined` relay, and the fix-B text synthesis fires only on an explicit opt-out. The #12905 test that asserted suppression used a bare createState() (`undefined`) to mean "client did not request thinking"; it now passes `requestedThinking: false`, which is what that sentence resolves to in production. The whole thinking family — dsml, adapter, translator, non-stream parity, #13620, #5786, markdown boundary — is 77/77. #12864 added requestRejectedFailure.ts with a getProviderConnectionById read that seeds the refusal streak across restarts; inventoried as a connection state read next to the family-cooldown site it resembles. #13909 made machineToken.ts import ./dataPaths; the isolated webpack compile has no repo tree, so it joins the sibling externals. The free-tier budget card SVG was one wave behind again (482 -> 491 models). Refs #13866 * fix(sse): restore maxQueueDepth=0 as unbounded, sanitize refusals at the write, drain the rest Part 2 of the 09-18 base-red drain. Two of the remaining failures were not stale tests but production defects the tests had caught. #12911 taught accountSemaphore to read `maxQueueSize: 0` as "reject when the slot is busy", which is what its Codex WS lease wants. But chatCore forwards `resilienceSettings.requestQueue.maxQueueDepth` into that option, and that setting's documented default since #6593 is `0 = disabled`. Under default settings every request that found its account slot occupied was answered 429 "Semaphore queue full (0)" instead of waiting — the managed-lease routing test saw exactly that. `0` (and any non-positive value) is unbounded again; the lease gets an explicit `failFast` option and its four tests stay green, so the #12911 behaviour is preserved where it was meant to apply. A contract test pins the #6593 semantics on the semaphore itself. #12864 moved two providerFailure persistence branches out of chatCore into requestRejectedFailure.ts and the sanitization did not travel with them: three `lastError` writes stored the message as received. The only caller already hands in the projected persistentMessage, so nothing leaks today, but a persistence branch must be safe at its own write (docs/security/ ERROR_SANITIZATION.md) rather than trust whoever calls it. The module now sanitizes on entry, and the public-boundary guard — which caught this by counting sanitized writes in chatCore and coming up two short — covers the extracted module too, verified by mutating one write back to raw. The rest are tests that had fallen behind legitimate changes: - #12905 inserted `requestedThinking` as the 14th positional argument of createSSETransformStreamWithLogger; two tests passed customToolNames or the buffer budget at their old positions. Both production callers were already correct. - #12754 added a per-connection reset-card fetch after the quota fetch; the spacing test now marks a chunk at the quota request only. - #13910 renamed `error` to `errorMetadata` in the timeout classification; the probe matches the identifier with a backreference and still fails when BodyTimeoutError is removed from both sites. Refs #13866 * fix(test): pin the opt-out thinking cases to requestedThinking=false; keep acquireMany under the complexity ceiling The #12905 gate-restore suite encoded 'requestedThinking absent' as opt-out, the same undefined-means-false shape its non-streaming twin documents the other way and that the two-month-old #5786 suites contradict. The three opt-out cases now set the flag explicitly, which is what chatCore resolves for an opted-out client; the two opt-in cases already did. Both suites pass together (27/27). The failFast branch pushed acquireMany over the complexity ceiling it already sat on; the admission policy (fail-fast / bounded / unbounded queue) moves to findQueueRejection() and the new-code ratchet is back at its base. Refs #13866 * fix(test): suspend the #14110 redaction assertion inline; refresh the budget card The 57 commits merged since the previous validation moved two things. #13295 changed how an unknown-root path with an ambiguous tail is answered: where `Provider failed at /custom/internal secret directory` used to become `Provider failed at <path>` it now ships verbatim. The #12506 boundary guard caught it. Two candidate fixes were tried and each breaks one of the two live contracts — #12506's fail-closed swallow, or #13144's rule that a route in prose must survive — so the choice is the owner's (#14110). The one contested assertion is suspended inline with the exact line and the issue; the other nine stay active. The isolated-child harness requires tests == pass, which is why it is a comment and not a todo. The free-tier budget card was one wave behind again (491 -> 489 models). Refs #13866, #14110 * fix(providers): type the TinyCMS DOM stub global as a loose record #13957 typed the mock global as `typeof globalThis & Record<string, unknown>`. The api-route typecheck loads lib.dom, so that intersection carries the real Window / HTMLCanvasElement / document signatures — every stub assignment fails against a DOM constructor, and `delete g.window` narrows the object to `never` (13 diagnostics, the API Route Typecheck base-red on the tip). The function exists to overwrite those globals with stubs; it is now typed as the plain record it manipulates. 29/29 tinycms tests unchanged. Refs #13866 * fix(test): pin the last opt-out thinking sibling to requestedThinking=false translator-reasoning-gate-502-repro is the third #12905 test that encoded a bare state as opt-out; the previous sweep matched files by glob and missed it. The family is now enumerated by grep on requestedThinking (7 files) plus the two pre-#12905 suites: 83/83 together. Refs #13866
sprintberlin
added a commit
to sprintberlin/OmniRoute
that referenced
this pull request
Sep 22, 2026
…t rejection, not a ban xAI's Grok Build upstream declines safety-sensitive prompts with 403 "I can't help with that request." — a refusal of THAT request, not of the credential. Because grok-cli is an OAuth provider, the 403 fell through to the generic FORBIDDEN classification and chatCore wrote the terminal banned state, disabling the whole connection permanently. Measured in production (2026-09-19): the same token answered 200 16 s before the refusal, a token refresh succeeded 3 s before it, and an in-flight call completed 200 2 s after it; the account was never banned by xAI. The single refusal took the provider out of the tier-1 combo for 16.75 h until a manual dashboard reconnect. Same defect shape and same resolution as cheaper-inference#12859/cheaper-inference#12864 (Anthropic OAuth "Request not allowed"): classify the verbatim refusal as REQUEST_REJECTED before the terminal 403 fall-through. The existing requestRejectedFailure leaf then applies the bounded cooldown ladder (5 min, 15 min, terminal only on 3 consecutive refusals) and the failing request falls through to the next combo target. Closes cheaper-inference#14258
sprintberlin
added a commit
to sprintberlin/OmniRoute
that referenced
this pull request
Sep 22, 2026
…tap files Keeps tap.testFiles aligned with the new refusal regression suite matching the pattern established in cheaper-inference#12864.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…refusal — cooldown with backoff instead of an instant ban (cheaper-inference#12859) (cheaper-inference#12864) * fix(sse): Anthropic OAuth 403 "Request not allowed" is a per-request refusal, not a ban A single upstream 403 on the `claude` OAuth connection was classified FORBIDDEN and written as the terminal `banned` connection state (chatCore -> writeTerminalStatus). From then on every request to that provider was short-circuited with "All 1 connection(s) banned by upstream - please reconnect in the dashboard" without touching Anthropic, until an operator reconnected. Anthropic's OAuth surface answers a small fraction of otherwise-valid requests with 403 {"type":"permission_error","message":"Request not allowed"}. On the reporting install the same token returned 200 forty seconds before the 403 and again right after the connection was re-enabled; a revoked or expired token is a 401 authentication_error, not this. It is a refusal of one request, not of the credential. Classify it as the new non-terminal PROVIDER_ERROR_TYPES.REQUEST_REJECTED (scoped to provider `claude` and the "Request not allowed" body) and list that type in authTerminalStatus.isNonTerminalProviderError, mirroring the Cloudflare FINGERPRINT_REJECTION precedent. The combo layer still falls through to the next target for the failing request; the connection stays active for the next one. Any other claude 403 keeps its previous classification. Tests: error-classifier.test.ts covers the Anthropic body, the gateway-flattened "[403]: Request not allowed" message, the same body from a non-Anthropic provider (still FORBIDDEN), other claude 403s (unchanged), and the helpers; anthropic-request-not-allowed-not-a-ban.test.ts pins resolveTerminalConnectionStatus() -> null for the new type even with a `permanent` fallback verdict, and `banned` for a generic claude 403. * fix(sse): cooldown with backoff and streak escalation for REQUEST_REJECTED (cheaper-inference#12859) Not "ignore the 403" either: if Anthropic ever made "Request not allowed" systematic, re-sending every request into it would be the wrong thing to do to an OAuth account. chatCore now handles REQUEST_REJECTED explicitly: - exclude the connection via setConnectionRateLimitUntil for a growing cooldown (5 -> 15 -> 45 min) so a sporadic refusal costs minutes, not a reconnect, and a systematic one cannot become a stream of 403s; - escalate to the terminal `banned` state only for 3 refusals within a 60-minute window (services/requestRejectedStreak.ts, in-memory per connection; a restart forgets the streak, erring towards more cooldowns rather than an operator-undone ban), with a last_error that says so; - probe-origin failures record but never cool down or ban (cheaper-inference#9817). The existing "request not allowed" text rule (5 s) is unaffected: markAccountUnavailable skips a connection that already has a future rateLimitedUntil, so the minute-scale cooldown written here wins. Tests: request-rejected-streak.test.ts pins the window/threshold/backoff arithmetic; anthropic-request-not-allowed-cooldown-escalation.test.ts drives the real chat route against a mocked 403 upstream on a `claude` OAuth connection: 300 s cooldown, then 900 s, then banned on the third refusal; a different claude 403 body still bans on the first response. * chore(changelog): name the cheaper-inference#12859 fragment after its PR (cheaper-inference#12864) * refactor(sse): move the REQUEST_REJECTED branch into a chatCore leaf; register its tests for mutation coverage chatCore.ts is frozen at 5984 lines by the file-size ratchet; the branch body now lives in open-sse/handlers/chatCore/requestRejectedFailure.ts (chatCore: 5974 -> 5983). stryker.conf.json tap.testFiles gains the two new DB-backed tests so their mutant kills count (check:mutation-test-coverage). * fix(sse): count refusal episodes, reset on success, keep the dashboard honest (cheaper-inference#12859 review) Review findings on the first cut of the REQUEST_REJECTED handling: - A burst of in-flight requests that all got the 403 within seconds produced streak 1, 2, 3 and a ban from one upstream event. The streak now counts cooldown *episodes*: a refusal that lands while the connection is already excluded is the same event and is not counted. - Nothing reset the streak on a healthy response, so sporadic refusals on a busy install could still accumulate to a ban. chatHelpers' onRequestSuccess now clears it (only a real success does - the recovery tick's clearAccountError is an elapsed cooldown, not a success). Clearing the cooldown by hand in the dashboard clears it too. - The third rung of the ladder was unreachable (the third refusal escalates): the ladder is now 5 -> 15 min, sourced from COOLDOWN_MS next to the existing 5 s "request not allowed" rule, with a note on why that rule is superseded for claude. The 60-min window becomes a 24 h staleness bound - "consecutive" is defined by successes, not by time. - Probe-origin refusals no longer touch the streak (cheaper-inference#9817). - The cooldown is written like every other connection-level cooldown: ISO rateLimitedUntil + testStatus "unavailable" (+ lastErrorAt), so the dashboard shows the countdown and the recovery tick restores "active". - One refusal is re-seeded from the persisted row after a restart so a crash loop cannot reset the count on every boot. Docs: RESILIENCE_GUIDE terminal states + CODEBASE_DOCUMENTATION resilience row mention the streak module. Tests cover the burst, the success reset, the seed, and the ISO/unavailable shape end-to-end through the chat route. * chore(sse): drop unrelated Prettier churn in auth.ts / providers route * style(api): keep providers route Prettier-clean * refactor(sse): share the "exclude connection for a cooldown" leaf between GEO_BLOCKED, GCP_PROJECT_REQUIRED and the new branch The release tip moved chatCore.ts to its frozen 5984 lines, so the REQUEST_REJECTED branch cannot add a single net line. The GEO_BLOCKED and GCP_PROJECT_REQUIRED branches were the same eight statements with different constants and log wording; both now call open-sse/handlers/chatCore/connectionCooldown.ts::excludeConnectionForCooldown (behaviour, probe guard and log lines preserved verbatim). chatCore.ts ends 9 lines below the base it branched from. * chore(chatCore): tighten the cooldown comments to keep the file under its size ceiling After merging release/v3.8.51, chatCore.ts sat at 6150 lines against a frozen ceiling of 6146. Condense the explanatory comments this PR added to the GEO_BLOCKED and GCP_PROJECT_REQUIRED branches; no code change. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: insoln <is@careerum.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…the write, drain the 09-18 base-reds (cheaper-inference#14101) * fix(quality): drain the 09-18 base-reds, part 1 — thinking gate parity, inventory, webpack externals Reproduced on the clean tip 7784784 before touching anything. Five of the failures trace to one commit, cheaper-inference#12905 (8b5f4dd): it gated thinking-block emission on `requestedThinking === true` in the streaming translator, while its own non-streaming path documents `undefined` as the legacy caller shape that keeps "always a thinking block". The two paths disagreed on the same input, and the streaming side also synthesized the reasoning into a TEXT block for that legacy shape. chatCore always resolves a boolean, so production never sends `undefined` — but every direct caller and the older cheaper-inference#5786 suites do. Aligned the streaming gate to the documented tri-state: `false` suppresses, `true` and `undefined` relay, and the fix-B text synthesis fires only on an explicit opt-out. The cheaper-inference#12905 test that asserted suppression used a bare createState() (`undefined`) to mean "client did not request thinking"; it now passes `requestedThinking: false`, which is what that sentence resolves to in production. The whole thinking family — dsml, adapter, translator, non-stream parity, cheaper-inference#13620, cheaper-inference#5786, markdown boundary — is 77/77. cheaper-inference#12864 added requestRejectedFailure.ts with a getProviderConnectionById read that seeds the refusal streak across restarts; inventoried as a connection state read next to the family-cooldown site it resembles. cheaper-inference#13909 made machineToken.ts import ./dataPaths; the isolated webpack compile has no repo tree, so it joins the sibling externals. The free-tier budget card SVG was one wave behind again (482 -> 491 models). Refs cheaper-inference#13866 * fix(sse): restore maxQueueDepth=0 as unbounded, sanitize refusals at the write, drain the rest Part 2 of the 09-18 base-red drain. Two of the remaining failures were not stale tests but production defects the tests had caught. cheaper-inference#12911 taught accountSemaphore to read `maxQueueSize: 0` as "reject when the slot is busy", which is what its Codex WS lease wants. But chatCore forwards `resilienceSettings.requestQueue.maxQueueDepth` into that option, and that setting's documented default since cheaper-inference#6593 is `0 = disabled`. Under default settings every request that found its account slot occupied was answered 429 "Semaphore queue full (0)" instead of waiting — the managed-lease routing test saw exactly that. `0` (and any non-positive value) is unbounded again; the lease gets an explicit `failFast` option and its four tests stay green, so the cheaper-inference#12911 behaviour is preserved where it was meant to apply. A contract test pins the cheaper-inference#6593 semantics on the semaphore itself. cheaper-inference#12864 moved two providerFailure persistence branches out of chatCore into requestRejectedFailure.ts and the sanitization did not travel with them: three `lastError` writes stored the message as received. The only caller already hands in the projected persistentMessage, so nothing leaks today, but a persistence branch must be safe at its own write (docs/security/ ERROR_SANITIZATION.md) rather than trust whoever calls it. The module now sanitizes on entry, and the public-boundary guard — which caught this by counting sanitized writes in chatCore and coming up two short — covers the extracted module too, verified by mutating one write back to raw. The rest are tests that had fallen behind legitimate changes: - cheaper-inference#12905 inserted `requestedThinking` as the 14th positional argument of createSSETransformStreamWithLogger; two tests passed customToolNames or the buffer budget at their old positions. Both production callers were already correct. - cheaper-inference#12754 added a per-connection reset-card fetch after the quota fetch; the spacing test now marks a chunk at the quota request only. - cheaper-inference#13910 renamed `error` to `errorMetadata` in the timeout classification; the probe matches the identifier with a backreference and still fails when BodyTimeoutError is removed from both sites. Refs cheaper-inference#13866 * fix(test): pin the opt-out thinking cases to requestedThinking=false; keep acquireMany under the complexity ceiling The cheaper-inference#12905 gate-restore suite encoded 'requestedThinking absent' as opt-out, the same undefined-means-false shape its non-streaming twin documents the other way and that the two-month-old cheaper-inference#5786 suites contradict. The three opt-out cases now set the flag explicitly, which is what chatCore resolves for an opted-out client; the two opt-in cases already did. Both suites pass together (27/27). The failFast branch pushed acquireMany over the complexity ceiling it already sat on; the admission policy (fail-fast / bounded / unbounded queue) moves to findQueueRejection() and the new-code ratchet is back at its base. Refs cheaper-inference#13866 * fix(test): suspend the cheaper-inference#14110 redaction assertion inline; refresh the budget card The 57 commits merged since the previous validation moved two things. cheaper-inference#13295 changed how an unknown-root path with an ambiguous tail is answered: where `Provider failed at /custom/internal secret directory` used to become `Provider failed at <path>` it now ships verbatim. The cheaper-inference#12506 boundary guard caught it. Two candidate fixes were tried and each breaks one of the two live contracts — cheaper-inference#12506's fail-closed swallow, or cheaper-inference#13144's rule that a route in prose must survive — so the choice is the owner's (cheaper-inference#14110). The one contested assertion is suspended inline with the exact line and the issue; the other nine stay active. The isolated-child harness requires tests == pass, which is why it is a comment and not a todo. The free-tier budget card was one wave behind again (491 -> 489 models). Refs cheaper-inference#13866, cheaper-inference#14110 * fix(providers): type the TinyCMS DOM stub global as a loose record cheaper-inference#13957 typed the mock global as `typeof globalThis & Record<string, unknown>`. The api-route typecheck loads lib.dom, so that intersection carries the real Window / HTMLCanvasElement / document signatures — every stub assignment fails against a DOM constructor, and `delete g.window` narrows the object to `never` (13 diagnostics, the API Route Typecheck base-red on the tip). The function exists to overwrite those globals with stubs; it is now typed as the plain record it manipulates. 29/29 tinycms tests unchanged. Refs cheaper-inference#13866 * fix(test): pin the last opt-out thinking sibling to requestedThinking=false translator-reasoning-gate-502-repro is the third cheaper-inference#12905 test that encoded a bare state as opt-out; the previous sweep matched files by glob and missed it. The family is now enumerated by grep on requestedThinking (7 files) plus the two pre-cheaper-inference#12905 suites: 83/83 together. Refs cheaper-inference#13866
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12859.
Problem
One upstream
403 {"type":"permission_error","message":"Request not allowed"}on aclaudeOAuth connection is classifiedFORBIDDENinclassifyProviderError()and written bychatCoreas the terminalbannedconnection state. Every later request is then answered locally with[claude] All 1 connection(s) banned by upstream — please reconnect in the dashboard(401) until an operator reconnects. The response is a per-request refusal on a healthy token (200 before and after it on the same connection; a dead token is a401 authentication_error), and it is rare — order of one per several thousand requests.Fix
Neither "ban on first sight" nor "ignore it": a run of these refusals would be enforcement, and re-sending every request into it is the wrong thing to do to an OAuth account.
open-sse/services/errorClassifier.ts: new non-terminalPROVIDER_ERROR_TYPES.REQUEST_REJECTED("the upstream refused this request, not the credential"). In the 403 branch, a body matchingRequest not allowedfrom providerclaudereturns it instead of falling through toFORBIDDEN(helpersisAnthropicOAuthProvider()/isAnthropicRequestNotAllowed()keep the scope explicit).src/sse/services/authTerminalStatus.ts:REQUEST_REJECTEDjoinsisNonTerminalProviderError(), soresolveTerminalConnectionStatus()yieldsnullfor it even when the fallback analysis sayspermanent— theFINGERPRINT_REJECTIONprecedent.open-sse/handlers/chatCore/requestRejectedFailure.ts(leaf;chatCore.tsis at its frozen size): onREQUEST_REJECTEDCOOLDOWN_MS.requestRejected/requestRejectedRepeat, next to the existing 5 srequest_not_allowedtext rule, which this supersedes forclaude) — written like every other connection-level cooldown: ISOrateLimitedUntil+testStatus: "unavailable"+lastErrorAt, so the dashboard shows the countdown and the recovery tick restoresactive;bannedonly on the third consecutive refusal, with alast_errorthat says so.open-sse/services/requestRejectedStreak.ts: the per-connection streak. It counts cooldown episodes: a refusal that lands while the connection is already excluded (parallel sessions / fan-out that were in flight when the first 403 came back) is the same event and is not counted. A successful response on the connection clears the streak (chatHelpersonRequestSuccess), as does clearing the cooldown by hand in the dashboard; the recovery tick does not (an elapsed cooldown is not a success). A 24 h staleness bound keeps an ancient refusal from starting a streak. State is in-memory; after a restart one refusal is re-seeded from the persisted row (last_error_type/last_error_at) so a crash loop cannot reset the count on every boot.open-sse/handlers/chatCore/connectionCooldown.ts: theGEO_BLOCKEDandGCP_PROJECT_REQUIREDbranches were the same eight statements with different constants and wording; both now go throughexcludeConnectionForCooldown()(behaviour, probe guard and log lines preserved verbatim).chatCore.tsis at the file-size ratchet's frozen 5984 lines, so the new branch could not add a net line — after this extraction it ends 9 lines below the base.The failing request itself still falls through to the next combo target, as before. Every other
claude403 body keeps its previous classification, and the same body from any other provider still classifies as before.Bound on upstream hits per refusing connection: 1 per cooldown episode, 3 episodes then
banned— versus 1 then a manual reconnect before this PR.Tests
tests/unit/anthropic-request-not-allowed-cooldown-escalation.test.ts(new, end-to-end through the real chat route with a mocked upstream on aclaudeOAuth connection): 5 min cooldown after the 1st refusal, 15 min after the 2nd,banned+is_active = 0on the 3rd; a burst of refusals inside the first cooldown is one episode (no re-persist, no ban) and the next episode is the 2nd; a 200 clears the streak so the next refusal starts at 5 min again; a different claude 403 body still bans on the first response.tests/unit/request-rejected-streak.test.ts(new): ladder sourced fromCOOLDOWN_MS, episode de-duplication, escalation on the 3rd consecutive refusal, staleness window, success reset, per-connection isolation, restart seed.tests/unit/anthropic-request-not-allowed-not-a-ban.test.ts(new):classifyProviderError→resolveTerminalConnectionStatusgivesnullfor the new type withpermanent: falseandpermanent: true; a genericclaude403 still resolves tobanned.tests/unit/error-classifier.test.ts: Anthropic body →REQUEST_REJECTED, neverFORBIDDEN; the gateway-flattened message; the same body from a non-Anthropic provider staysFORBIDDEN; otherclaude403s unchanged; helper edge cases.All fail against the previous code. Also green locally:
check:file-size(PR mode),check:mutation-test-coverage --strict(new tests registered instryker.conf.json),check:cycles,typecheck:core,eslinton the changed files,check:any-budget:t11,check:doc-links,check:fabricated-docs, and the neighbouring suitesprobe-gate-autodisable,8200-perplexity-web-401-cooldown,error-config,call-log-error-type.Docs:
docs/architecture/RESILIENCE_GUIDE.md(terminal states) anddocs/architecture/CODEBASE_DOCUMENTATION.md(resilience row) mention the new path.glm.tsTS2554, stale "169 migrations" doc counts,reset-aware-model-family.mdfragment,reset-aware-request-scope-12600missing fromstryker.conf.json, and the unrelated unit-test failures listed on #12830).