Skip to content

fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban (#13157) - #13161

Closed
anhtran-ai wants to merge 1 commit into
cheaper-inference:release/v3.8.51from
azox-ai:fix/codex-cloudflare-managed-challenge
Closed

anhtran-ai wants to merge 1 commit into
cheaper-inference:release/v3.8.51from
azox-ai:fix/codex-cloudflare-managed-challenge

Conversation

@anhtran-ai

Copy link
Copy Markdown
Contributor

Closes #13157

Problem

A Cloudflare managed challenge served in front of an upstream provider is classified FORBIDDEN and persisted as the terminal banned connection state, taking the whole provider offline until an operator reconnects in the dashboard.

Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a healthy Codex OAuth account:

15:14:07  Trying model 2/2: cx/gpt-5.6-sol
15:14:10  Model cx/gpt-5.6-sol succeeded (5986ms, 0 fallbacks)      ← normal chat OK
15:14:38  POST /v1/responses/input_tokens | …
15:14:39  Successfully refreshed Codex token / CODEX | refreshed    ← credential healthy
15:14:39  [provider] Node e94ce3e6-… banned (403) — disabling permanently
15:14:39  codex | 1 accounts found but none active

The OAuth token refreshing successfully in the same second, and normal /responses traffic succeeding seconds before and after on the same connection, show the account was never banned upstream.

Root cause

isCloudflareFingerprintRejection() only recognises the Cloudflare 1010 family (error_code:1010, browser_signature_banned, fingerprint_rejection). A managed challenge is the same class of block — the edge refused the client's signature, not the credential — but it is a different product surface and carries none of those markers, so the guard at the top of the 403 branch misses it:

if (statusCode === 403 && isCloudflareFingerprintRejection(bodyStr)) {
  return PROVIDER_ERROR_TYPES.FINGERPRINT_REJECTION;   // ← not reached
}

Execution then falls through the rest of the ladder — not geo-blocked, not accountDeactivated, not a Cloud Code project error, not SENTINEL_BLOCKED/Turnstile required, codex is neither an apikey-category provider nor authType: "none" — and lands on return PROVIDER_ERROR_TYPES.FORBIDDEN, which chatCore persists through writeTerminalStatus().

Measured response, captured against the live account:

status 403
cf-mitigated: challenge
server: cloudflare
content-type: text/html; charset=UTF-8

_cf_chl_opt                                  present
cdn-cgi/challenge-platform                   present
id="challenge-error-text"                    present
cType: 'managed'   cZone: 'chatgpt.com'

error code: 1010                             absent
error_code                                   absent
browser_signature_banned                     absent
fingerprint_rejection                        absent

The ~12KB HTML body is also why the operator-visible error degrades to [403]: <html>.

Fix

Recognise the challenge interstitial and classify it as FINGERPRINT_REJECTION, reusing the existing non-terminal precedent from #9929 — authTerminalStatus.isNonTerminalProviderError() already treats that type as non-terminal, so the failing request falls through to the next combo target and the account state stays untouched.

Markers are matched as full, distinctive Cloudflare-internal strings — _cf_chl_opt, cdn-cgi/challenge-platform, and the challenge-error-text span id including its escaped-quote nested form — never as the loose word "challenge", which a provider body may legitimately use in prose.

This is an additive branch on an existing carve-out; the genuine-403 path is untouched.

Note for reviewers

ENABLE_TLS_FINGERPRINT=true is not a workaround for this. On the affected deployment all 2516 rows in proxy_logs — both codex and claude, successes included — carry tls_fingerprint = 0 even though the flag is set and wreq-js is installed, so the fingerprint transport is never actually taken there. That looks like a separate defect (possibly related to #12491) and is out of scope for this PR.

Tests

tests/unit/error-classifier.test.ts — 39/39 pass (6 new):

  • the full managed-challenge interstitial classifies as FINGERPRINT_REJECTION, not FORBIDDEN;
  • the same body nested inside the gateway's error.message wrapper;
  • each marker recognised individually, including the escaped-quote nested form;
  • false-positive guards: prose mentioning a "challenge", a challenge_required error code, a bare challenge-platform without the cdn-cgi path, and an empty body all stay unmatched;
  • regression guards: a genuine permission 403 and the ChatGPT Web SENTINEL_BLOCKED / Turnstile required 403 (fix: ChatGPT blocked the request (Sentinel/Turnstile required) #8813) both still classify as FORBIDDEN.

Also run: tests/unit/auth-terminal-status.test.ts 11/11 pass, npm run typecheck:core clean, eslint clean on both changed files.

…t rejection, not a ban

A Cloudflare managed/JS challenge served in front of an upstream provider is the
same class of block as a Cloudflare 1010 — the edge refused the CLIENT's
signature — but it is a different product surface and carries none of the 1010
markers isCloudflareFingerprintRejection() looks for.

It therefore fell through the entire 403 ladder in classifyProviderError() to the
terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as
testStatus=banned / isActive=false. That state never auto-recovers, so a single
challenge takes the whole provider offline until an operator reconnects in the
dashboard.

Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a
healthy Codex OAuth account: the response carried cf-mitigated: challenge,
server: cloudflare and a ~12KB text/html interstitial with
window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same
connection refreshed its OAuth token successfully in the same second and served
normal /responses traffic seconds before and after, so the account was never
banned upstream.

Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing
non-terminal precedent from cheaper-inference#9929: authTerminalStatus already treats that type as
non-terminal, so the request falls through to the next combo target and the
account state stays untouched.

The markers are matched as full, distinctive Cloudflare-internal strings
(_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id
including its escaped-quote nested form) and never as the loose word
"challenge", so provider bodies discussing a challenge in prose are unaffected.

Tests cover the full interstitial, the gateway-nested error.message form, each
marker individually, prose false-positive guards, and regression guards proving
a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (cheaper-inference#8813) both
still classify as FORBIDDEN.
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for tracking down the exact interstitial markers — the distinction between the
1010 signature and the managed-challenge HTML is a real gap, and the "healthy account,
single bad response" repro in the PR description matches the code path exactly. Probed
the new test file locally: 39/39 pass, including the false-positive guards for prose
mentioning "challenge" and the three 403 paths that must stay terminal. One small ask
before merge: this PR and #12864 both touch tests/unit/error-classifier.test.ts —
whichever lands second will need a small manual reconciliation of the appended test
blocks (no logic conflict, just two additions at the same spot). Also feel free to add
a changelog.d/fixes/ fragment like your sibling PRs in this batch.

diegosouzapw added a commit that referenced this pull request Sep 18, 2026
…#14090)

* fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban

A Cloudflare managed/JS challenge served in front of an upstream provider is the
same class of block as a Cloudflare 1010 — the edge refused the CLIENT's
signature — but it is a different product surface and carries none of the 1010
markers isCloudflareFingerprintRejection() looks for.

It therefore fell through the entire 403 ladder in classifyProviderError() to the
terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as
testStatus=banned / isActive=false. That state never auto-recovers, so a single
challenge takes the whole provider offline until an operator reconnects in the
dashboard.

Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a
healthy Codex OAuth account: the response carried cf-mitigated: challenge,
server: cloudflare and a ~12KB text/html interstitial with
window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same
connection refreshed its OAuth token successfully in the same second and served
normal /responses traffic seconds before and after, so the account was never
banned upstream.

Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing
non-terminal precedent from #9929: authTerminalStatus already treats that type as
non-terminal, so the request falls through to the next combo target and the
account state stays untouched.

The markers are matched as full, distinctive Cloudflare-internal strings
(_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id
including its escaped-quote nested form) and never as the loose word
"challenge", so provider bodies discussing a challenge in prose are unaffected.

Tests cover the full interstitial, the gateway-nested error.message form, each
marker individually, prose false-positive guards, and regression guards proving
a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (#8813) both
still classify as FORBIDDEN.

(cherry picked from commit 8204da6)

* fix(translator): skip replayed web_search_call metadata in Responses-to-Chat

OmniRoute's web-search fallback emits a native web_search_call output item
alongside function_call/function_call_output. Responses clients keep that item
in conversation history and replay it in the next request's input. When the
follow-up turn routes to a Chat Completions target (Claude), the translator hit
its default unsupported-feature branch and returned a deterministic HTTP 400:

  Unsupported Responses API feature: input item type 'web_search_call'
  cannot be represented in Chat Completions

Skip the replayed metadata next to tool_search_call/tool_search_result. The
paired function_call_output still carries the search results, so no context is
lost and the sources are not duplicated into assistant history.

(cherry picked from commit 2b3e63a)

* chore(changelog): credit the locked-fork landings of #13161 and #13304

Both PRs come from an organization fork (azox-ai) that refuses maintainer pushes
even with maintainerCanModify=true, so they cannot be re-synced in place and are
landed here by cherry-pick with the contributor's authorship preserved
(merge-gates §6). GitHub may not mark the PRs Merged, hence the explicit
credit in the fragments.

---------

Co-authored-by: anhth2 <anhth2@vng.com.vn>
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thank you @anhtran-ai — your fix has landed on release/v3.8.51 through #14090, cherry-picked with your authorship preserved. This branch lives on an organization fork (azox-ai) that refuses maintainer pushes even with "allow edits by maintainers" on, so GitHub could not re-sync it in place or mark it Merged automatically; the changelog fragment credits you explicitly (#13161 — thanks @anhtran-ai). Leaving this open for you to close (or for the maintainer) — nothing else is needed on your side.

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thank you @anhtran-ai! Your fork belongs to an organization that doesn't allow maintainer pushes, so we couldn't merge this PR directly. Your change landed in release/v3.8.51 through #14090, which cherry-picked your commits together with #13161/#13304. The squash carries your Co-authored-by trailer, and the changelog credits you.

Closing this as landed via #14090. Thanks again for the fix.

muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ry-pick (locked organization fork) (cheaper-inference#14090)

* fix(resilience): treat a Cloudflare managed challenge as a fingerprint rejection, not a ban

A Cloudflare managed/JS challenge served in front of an upstream provider is the
same class of block as a Cloudflare 1010 — the edge refused the CLIENT's
signature — but it is a different product surface and carries none of the 1010
markers isCloudflareFingerprintRejection() looks for.

It therefore fell through the entire 403 ladder in classifyProviderError() to the
terminal default FORBIDDEN, which chatCore persists via writeTerminalStatus() as
testStatus=banned / isActive=false. That state never auto-recovers, so a single
challenge takes the whole provider offline until an operator reconnects in the
dashboard.

Observed on POST chatgpt.com/backend-api/codex/responses/input_tokens for a
healthy Codex OAuth account: the response carried cf-mitigated: challenge,
server: cloudflare and a ~12KB text/html interstitial with
window._cf_chl_opt = {... cType: 'managed', cZone: 'chatgpt.com' ...}. The same
connection refreshed its OAuth token successfully in the same second and served
normal /responses traffic seconds before and after, so the account was never
banned upstream.

Classify the interstitial as FINGERPRINT_REJECTION, reusing the existing
non-terminal precedent from cheaper-inference#9929: authTerminalStatus already treats that type as
non-terminal, so the request falls through to the next combo target and the
account state stays untouched.

The markers are matched as full, distinctive Cloudflare-internal strings
(_cf_chl_opt, cdn-cgi/challenge-platform, the challenge-error-text span id
including its escaped-quote nested form) and never as the loose word
"challenge", so provider bodies discussing a challenge in prose are unaffected.

Tests cover the full interstitial, the gateway-nested error.message form, each
marker individually, prose false-positive guards, and regression guards proving
a genuine permission 403 and the ChatGPT Web Sentinel/Turnstile 403 (cheaper-inference#8813) both
still classify as FORBIDDEN.

(cherry picked from commit 8204da6)

* fix(translator): skip replayed web_search_call metadata in Responses-to-Chat

OmniRoute's web-search fallback emits a native web_search_call output item
alongside function_call/function_call_output. Responses clients keep that item
in conversation history and replay it in the next request's input. When the
follow-up turn routes to a Chat Completions target (Claude), the translator hit
its default unsupported-feature branch and returned a deterministic HTTP 400:

  Unsupported Responses API feature: input item type 'web_search_call'
  cannot be represented in Chat Completions

Skip the replayed metadata next to tool_search_call/tool_search_result. The
paired function_call_output still carries the search results, so no context is
lost and the sources are not duplicated into assistant history.

(cherry picked from commit 2b3e63a)

* chore(changelog): credit the locked-fork landings of cheaper-inference#13161 and cheaper-inference#13304

Both PRs come from an organization fork (azox-ai) that refuses maintainer pushes
even with maintainerCanModify=true, so they cannot be re-synced in place and are
landed here by cherry-pick with the contributor's authorship preserved
(merge-gates §6). GitHub may not mark the PRs Merged, hence the explicit
credit in the fragments.

---------

Co-authored-by: anhth2 <anhth2@vng.com.vn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(resilience): errorClassifier: a Cloudflare managed challenge on Codex /responses/input_tokens permanently bans a healthy Codex OAuth connection

3 participants