Repository navigation
fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban (#13157) - #13161
Conversation
…t rejection, not a ban
A Cloudflare managed/JS challenge served in front of an upstream provider is the
same class of block as a Cloudflare 1010 — the edge refused the CLIENT's
signature — but it is a different product surface and carries none of the 1010
markers isCloudflareFingerprintRejection() looks for.
It therefore fell through the entire 403 ladder in classifyProviderError() to the
terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as
testStatus=banned / isActive=false. That state never auto-recovers, so a single
challenge takes the whole provider offline until an operator reconnects in the
dashboard.
Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a
healthy Codex OAuth account: the response carried cf-mitigated: challenge,
server: cloudflare and a ~12KB text/html interstitial with
window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same
connection refreshed its OAuth token successfully in the same second and served
normal /responses traffic seconds before and after, so the account was never
banned upstream.
Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing
non-terminal precedent from cheaper-inference#9929: authTerminalStatus already treats that type as
non-terminal, so the request falls through to the next combo target and the
account state stays untouched.
The markers are matched as full, distinctive Cloudflare-internal strings
(_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id
including its escaped-quote nested form) and never as the loose word
"challenge", so provider bodies discussing a challenge in prose are unaffected.
Tests cover the full interstitial, the gateway-nested error.message form, each
marker individually, prose false-positive guards, and regression guards proving
a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (cheaper-inference#8813) both
still classify as FORBIDDEN.
|
Thanks for tracking down the exact interstitial markers — the distinction between the |
…#14090) * fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban A Cloudflare managed/JS challenge served in front of an upstream provider is the same class of block as a Cloudflare 1010 — the edge refused the CLIENT's signature — but it is a different product surface and carries none of the 1010 markers isCloudflareFingerprintRejection() looks for. It therefore fell through the entire 403 ladder in classifyProviderError() to the terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as testStatus=banned / isActive=false. That state never auto-recovers, so a single challenge takes the whole provider offline until an operator reconnects in the dashboard. Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a healthy Codex OAuth account: the response carried cf-mitigated: challenge, server: cloudflare and a ~12KB text/html interstitial with window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same connection refreshed its OAuth token successfully in the same second and served normal /responses traffic seconds before and after, so the account was never banned upstream. Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing non-terminal precedent from #9929: authTerminalStatus already treats that type as non-terminal, so the request falls through to the next combo target and the account state stays untouched. The markers are matched as full, distinctive Cloudflare-internal strings (_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id including its escaped-quote nested form) and never as the loose word "challenge", so provider bodies discussing a challenge in prose are unaffected. Tests cover the full interstitial, the gateway-nested error.message form, each marker individually, prose false-positive guards, and regression guards proving a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (#8813) both still classify as FORBIDDEN. (cherry picked from commit 8204da6) * fix(translator): skip replayed web_search_call metadata in Responses-to-Chat OmniRoute's web-search fallback emits a native web_search_call output item alongside function_call/function_call_output. Responses clients keep that item in conversation history and replay it in the next request's input. When the follow-up turn routes to a Chat Completions target (Claude), the translator hit its default unsupported-feature branch and returned a deterministic HTTP 400: Unsupported Responses API feature: input item type 'web_search_call' cannot be represented in Chat Completions Skip the replayed metadata next to tool_search_call/tool_search_result. The paired function_call_output still carries the search results, so no context is lost and the sources are not duplicated into assistant history. (cherry picked from commit 2b3e63a) * chore(changelog): credit the locked-fork landings of #13161 and #13304 Both PRs come from an organization fork (azox-ai) that refuses maintainer pushes even with maintainerCanModify=true, so they cannot be re-synced in place and are landed here by cherry-pick with the contributor's authorship preserved (merge-gates §6). GitHub may not mark the PRs Merged, hence the explicit credit in the fragments. --------- Co-authored-by: anhth2 <anhth2@vng.com.vn>
|
Thank you @anhtran-ai — your fix has landed on |
|
Thank you @anhtran-ai! Your fork belongs to an organization that doesn't allow maintainer pushes, so we couldn't merge this PR directly. Your change landed in Closing this as landed via #14090. Thanks again for the fix. |
…ry-pick (locked organization fork) (cheaper-inference#14090) * fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban A Cloudflare managed/JS challenge served in front of an upstream provider is the same class of block as a Cloudflare 1010 — the edge refused the CLIENT's signature — but it is a different product surface and carries none of the 1010 markers isCloudflareFingerprintRejection() looks for. It therefore fell through the entire 403 ladder in classifyProviderError() to the terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as testStatus=banned / isActive=false. That state never auto-recovers, so a single challenge takes the whole provider offline until an operator reconnects in the dashboard. Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a healthy Codex OAuth account: the response carried cf-mitigated: challenge, server: cloudflare and a ~12KB text/html interstitial with window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same connection refreshed its OAuth token successfully in the same second and served normal /responses traffic seconds before and after, so the account was never banned upstream. Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing non-terminal precedent from cheaper-inference#9929: authTerminalStatus already treats that type as non-terminal, so the request falls through to the next combo target and the account state stays untouched. The markers are matched as full, distinctive Cloudflare-internal strings (_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id including its escaped-quote nested form) and never as the loose word "challenge", so provider bodies discussing a challenge in prose are unaffected. Tests cover the full interstitial, the gateway-nested error.message form, each marker individually, prose false-positive guards, and regression guards proving a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (cheaper-inference#8813) both still classify as FORBIDDEN. (cherry picked from commit 8204da6) * fix(translator): skip replayed web_search_call metadata in Responses-to-Chat OmniRoute's web-search fallback emits a native web_search_call output item alongside function_call/function_call_output. Responses clients keep that item in conversation history and replay it in the next request's input. When the follow-up turn routes to a Chat Completions target (Claude), the translator hit its default unsupported-feature branch and returned a deterministic HTTP 400: Unsupported Responses API feature: input item type 'web_search_call' cannot be represented in Chat Completions Skip the replayed metadata next to tool_search_call/tool_search_result. The paired function_call_output still carries the search results, so no context is lost and the sources are not duplicated into assistant history. (cherry picked from commit 2b3e63a) * chore(changelog): credit the locked-fork landings of cheaper-inference#13161 and cheaper-inference#13304 Both PRs come from an organization fork (azox-ai) that refuses maintainer pushes even with maintainerCanModify=true, so they cannot be re-synced in place and are landed here by cherry-pick with the contributor's authorship preserved (merge-gates §6). GitHub may not mark the PRs Merged, hence the explicit credit in the fragments. --------- Co-authored-by: anhth2 <anhth2@vng.com.vn>
Closes #13157
Problem
A Cloudflare managed challenge served in front of an upstream provider is classified
FORBIDDENand persisted as the terminalbannedconnection state, taking the whole provider offline until an operator reconnects in the dashboard.Observed on
POST chatgpt.com/backend-api/codex/responses/input_tokensfor a healthy Codex OAuth account:The OAuth token refreshing successfully in the same second, and normal
/responsestraffic succeeding seconds before and after on the same connection, show the account was never banned upstream.Root cause
isCloudflareFingerprintRejection()only recognises the Cloudflare 1010 family (error_code:1010,browser_signature_banned,fingerprint_rejection). A managed challenge is the same class of block — the edge refused the client's signature, not the credential — but it is a different product surface and carries none of those markers, so the guard at the top of the403branch misses it:Execution then falls through the rest of the ladder — not geo-blocked, not
accountDeactivated, not a Cloud Code project error, notSENTINEL_BLOCKED/Turnstile required,codexis neither anapikey-category provider norauthType: "none"— and lands onreturn PROVIDER_ERROR_TYPES.FORBIDDEN, whichchatCorepersists throughwriteTerminalStatus().Measured response, captured against the live account:
The ~12KB HTML body is also why the operator-visible error degrades to
[403]: <html>.Fix
Recognise the challenge interstitial and classify it as
FINGERPRINT_REJECTION, reusing the existing non-terminal precedent from #9929 —authTerminalStatus.isNonTerminalProviderError()already treats that type as non-terminal, so the failing request falls through to the next combo target and the account state stays untouched.Markers are matched as full, distinctive Cloudflare-internal strings —
_cf_chl_opt,cdn-cgi/challenge-platform, and thechallenge-error-textspan id including its escaped-quote nested form — never as the loose word "challenge", which a provider body may legitimately use in prose.This is an additive branch on an existing carve-out; the genuine-
403path is untouched.Note for reviewers
ENABLE_TLS_FINGERPRINT=trueis not a workaround for this. On the affected deployment all2516rows inproxy_logs— bothcodexandclaude, successes included — carrytls_fingerprint = 0even though the flag is set andwreq-jsis installed, so the fingerprint transport is never actually taken there. That looks like a separate defect (possibly related to #12491) and is out of scope for this PR.Tests
tests/unit/error-classifier.test.ts— 39/39 pass (6 new):FINGERPRINT_REJECTION, notFORBIDDEN;error.messagewrapper;challenge_requirederror code, a barechallenge-platformwithout thecdn-cgipath, and an empty body all stay unmatched;403and the ChatGPT WebSENTINEL_BLOCKED/Turnstile required403(fix: ChatGPT blocked the request (Sentinel/Turnstile required) #8813) both still classify asFORBIDDEN.Also run:
tests/unit/auth-terminal-status.test.ts11/11 pass,npm run typecheck:coreclean,eslintclean on both changed files.