feat(resilience): surface TLS-fingerprint remediation when Cloudflare 1010 hits - #12531
Closed
alvinveroy wants to merge 16 commits into
Closed
alvinveroy wants to merge 16 commits into
alvinveroy wants to merge 16 commits into
Conversation
… 1010 hits A Cloudflare 1010 / browser-signature rejection (diegosouzapw#9929) means the CDN banned the CLIENT's TLS/UA signature — not the account. OmniRoute already ships the fix for exactly this: a Chrome-124 impersonation transport (open-sse/utils/tlsClient.ts via the optional wreq-js dependency), gated behind ENABLE_TLS_FINGERPRINT. But the gate is off by default, the dependency is optional, and nothing tells the operator any of this — so repeated 1010s look like flaky upstreams while a one-line env change would fix them. Emit a throttled (once-per-process) remediation hint when a response is classified FINGERPRINT_REJECTION and ENABLE_TLS_FINGERPRINT is not enabled, naming the exact env vars and the optional dependency. No behavior change otherwise; the hint is suppressed entirely once the transport is enabled.
…-fingerprint hint Maintainer request on diegosouzapw#12531: the classifier-only unit passes both before and after the change because classifyProviderError already returns FINGERPRINT_REJECTION — the defect is that resolveTerminalConnectionStatus silently dropped it. Drive markAccountUnavailable end-to-end and assert the account stays healthy AND the hint fires exactly once from that path (throttled; suppressed when ENABLE_TLS_FINGERPRINT=true). Mirrors the 8200 auth-path test setup (isolated DATA_DIR + real DB rows).
Resolves the PR's dirty merge state: keeps upstream's Kiro IDC profileArn PROJECT_ROUTE_ERROR block (diegosouzapw#10725) ahead of the Sentinel/Turnstile check and retains this branch's diegosouzapw#8813 comment refinement on the Sentinel block.
…codeProtocol allowlist to L313 Fast Quality Gates carried two reds on this PR: 1. mutation-test-coverage (ours): the new auth-path regression test covers a mutated module but was absent from stryker.conf.json tap.testFiles, so its mutant kills stopped counting (--strict drift gate). Registered. 2. public-creds (inherited base-red diegosouzapw#12581): upstream diegosouzapw#12179 moved the per-process ZCode handshake literal from L302 to L313, leaving the frozen file:line:value allowlist pin stale. Re-pin to L313 — same justification (local per-process handshake ID, not an upstream credential). Identical to the fix upstream's fix/release-v3.8.51-basereds-public-creds must carry.
…ode background identity headers + staged resilience work # Conflicts: # open-sse/executors/base.ts # open-sse/handlers/embeddings.ts # open-sse/services/combo.ts # open-sse/services/opencodeQuotaFetcher.ts # src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx # src/i18n/messages/in.json # src/lib/usage/callLogArtifacts.ts # src/shared/constants/visionModels.ts # src/shared/utils/httpClientAbortGuard.mjs # tests/unit/call-log-cap.test.ts # tests/unit/httpClientAbortGuard.test.mjs
…10, with a first-byte watchdog Root cause of the 2026-09-03/04 outage: with ENABLE_TLS_FINGERPRINT=true every direct inference dispatch rode the wreq-js (browser-JA3) transport, whose responses resolved but streamed ZERO bytes for 90-600s on opencode-go / command-code / bailian (agents aborted -> 499 storms; the then-live 110s headers-wait default turned the hangs into 504s). Plain node fetch streamed the same requests in seconds and control-plane calls were unaffected — the stall is transport+streaming-specific, not DNS/CF slow-walking. - wreq is no longer eager by default: plain direct fetch is the fast path; a Cloudflare fingerprint rejection (403 + 1010 / browser_signature_banned) arms a 1h per-provider cache and retries once via wreq (replay-safe requests only). TLS_FINGERPRINT_PROVIDERS keeps its legacy eager semantics, now guarded. - Every wreq response (direct, ladder retry, proxied overlay) carries a first-byte watchdog (TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS, default 10s): no first byte -> cancel and fall back to the direct dispatcher instead of hanging the caller. Verified live: fingerprint off + ladder armed-on-demand restores TTFB 3-6s on opencode-go (was 90-600s of zero bytes), zero 1010 recurrence in the window.
…10, with a first-byte watchdog; per-provider fetch-start cap override Root cause of the 2026-09-03/04 outage this PR was filed against: with ENABLE_TLS_FINGERPRINT=true every direct inference dispatch rode the wreq-js (browser-JA3) transport, whose responses resolved but streamed ZERO bytes for 90-600s on opencode-go / command-code / bailian (agents aborted -> 499 storms; the streaming headers-wait default turned the hangs into exact-110000ms 504s). Plain node fetch streamed the same requests in seconds. - wreq is no longer eager by default: plain direct fetch is the fast path; a Cloudflare fingerprint rejection arms a 1h per-provider cache and retries once via wreq (replay-safe requests only). TLS_FINGERPRINT_PROVIDERS keeps its legacy eager semantics, now watchdog-guarded. - Every wreq response carries a first-byte watchdog (TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS, default 10s): no first byte -> cancel and fall back to the direct dispatcher instead of hanging the caller. - fetchStartTimeoutCapMs: the per-provider headers-wait override now actually reaches the executor (generateLegacyProviders() dropped the field, so the 110s default fired even with the registry override present); opencode-go and command-code raise it to 600s (Console Go gateways buffer entire generations — production showed TTFB up to 493s). Verified live: TTFB 3-6s on opencode-go (was 90-600s of zero bytes); zero 1010 recurrence with the transport off; zero 110000ms timeouts after the cap fix.
Contributor
Author
|
Updated (head
New tests: |
… watchdog 2026-09-05 wedge incident: opencode h2 stream resets (ERR_HTTP2_STREAM_ERROR, UND_ERR_SOCKET) fanned out through every combo target (8 fallbacks / 18 decisions per request), then a client abort during combo cleanup left the event loop wedged — listener accepting, nothing served, nothing logged, for hours. The existing supervisor only restarts on process exit. - Bounded transport abort (both dispatchers): consecutive transport-class network failures share the same egress — after DEFAULT_TRANSPORT_ABORT_BOUND (3), abort the combo with an explicit 502 instead of burning the remaining fan-out. Non-transport failures reset the counter. New predicates isTransportClassNetworkError / shouldAbortComboForTransportFailures in comboPredicates.ts. - Self-probe wedge watchdog (scripts/dev/self-probe-watchdog.mjs, wired in standalone-server-ws.mjs after the listener accepts): loopback probe of /api/status on the same event loop — a wedged loop cannot answer it. N consecutive connect/timeout failures (any HTTP response counts as alive, incl. 401) exit(1) so launchd KeepAlive / systemd / Docker relaunch. Config: OMNIROUTE_SELF_PROBE(_INTERVAL_MS|_TIMEOUT_MS|_THRESHOLD), default 60s/15s/3; OMNIROUTE_SELF_PROBE=off disables. - file-size rebaseline for the touched frozen files with justification.
…in TLS watchdog; stryker tap registration - dashboard/combos/page.tsx: defer the post-hydration localStorage correction one microtask so setShowUsageGuide is not a synchronous setState inside the effect body (react-hooks/set-state-in-effect base red on release/v3.8.51 from diegosouzapw#12448); eslint:json gate green. - proxyFetch.ts withTlsFirstByteWatchdog: the watchdog race consumed the first body chunk from the reader but never re-emitted it — the first SSE frame was silently dropped. The rest stream now re-emits the captured chunk before continuing the original stream. - stryker.conf.json: register 10281-combo-reasoning-probe + transport-bounded-abort in tap.testFiles (they cover mutated modules; --strict gate). - callLogArtifacts.ts / earlier dashboard stray: restore pristine remote state for files untouched by this branch (resolves unused-var lint).
- take upstream useSyncExternalStore combos usage-guide (diegosouzapw#12671) over our queueMicrotask lint defuse (supersedes it; suppression already dropped) - file-size baseline: combos/page.tsx 5066 (upstream shape); keep both _rebaseline entries (bounded-abort/tls-ladder + 12671 external store) - upstream additions folded in: diegosouzapw#12682 sqlite cleanup, diegosouzapw#12691 tool_calls guard, diegosouzapw#12834 codex compression re-enable, diegosouzapw#12707 continuation, diegosouzapw#12710 video-transcript redaction, diegosouzapw#12699 npmrc shipping, docker/deps/docs
- docs counts: 169 -> 170 migrations (README.md, AGENTS.md, llm.txt + 41 i18n llm.txt mirrors) - env/docs contract: document OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS and TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS in .env.example + ENVIRONMENT.md - changelog: normalize upstream reset-aware-model-family.md fragment to bullet form - agent skills: regenerate cli-tunnel SKILL.md (catalog drift from merged CLI-tool commits) - drop stray changelog.d/.DS_Store
This was referenced Sep 6, 2026
Open
Contributor
Author
|
Closing this consolidated PR. Following the maintainer-friendly stacked-PR pattern The 17 commits consolidated here are split into 10 groups; each new PR links back to the Replacement stack (track these new PRs instead of this one):
Each PR body will include the focused-loop commands run on the VPS, the changed/added test Closing this entry to keep the review queue clean. The corresponding tracking issue at the |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12529
Closes #12657
Closes #12658
Closes #12659
Supersedes #12660, #12661, #12662 (their head branches carried the same cumulative content with an older base; this PR is the consolidated, updated head on current
release/v3.8.51).Summary
One production batch for the 3.8.51 fork, grouped by area. All pieces were deployed and validated on the production instance (see Deployment evidence per area) before this PR was assembled.
1. TLS-fingerprint remediation on Cloudflare 1010 (fixes #12529)
A Cloudflare
403/error_code 1010rejection means the CDN banned the CLIENT's TLS/UA signature — not the account. #9929 already classifies itFINGERPRINT_REJECTIONso it no longer poisons account state, and the repo ships the fix (Chrome-124 TLS impersonation via optionalwreq-js), but nothing connected a 1010 to that one-line remediation.open-sse/services/errorClassifier.ts: throttled (once-per-process) remediation hint when a response classifiesFINGERPRINT_REJECTIONwhileENABLE_TLS_FINGERPRINTis off; suppressed when the transport is enabled.open-sse/utils/proxyFetch.ts: adaptive TLS ladder — plain fetch fast path; wreq-js only on the 1010 arm (1h TTL) with a first-byte watchdog (TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS, default 10s) and direct-dispatcher fallback, fixing the wreq-js zero-byte streaming stall (TTFB 90–600s → 3–6s). New envTLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MSdocumented.open-sse/services/errorClassifier.ts+authpath keep the connection active/non-terminal on 1010.tests/unit/12529-fingerprint-remediation-auth-path.test.ts,tests/unit/error-classifier.test.ts,tests/unit/transport-bounded-abort.test.ts,tests/unit/self-probe-watchdog.test.mjs.2. opencode family: background-call identity + shape fixes (closes #12657)
Starting 2026-09-06 opencode.ai enforces
x-opencode-sessionon background calls (model discovery, quota fetches went out as UA "Bun fetch" with no session header).open-sse/utils/opencodeHeaders.ts:buildOpencodeBackgroundHeaders()attaches the conversation-stable session identity to discovery (src/app/api/providers/[id]/models/discovery/providerModelsConfig.tsopencode/zen/go entries) andopen-sse/services/opencodeQuotaFetcher.ts.open-sse/translator/index.ts+toResponses.ts: reasoning-presence sentinel + same-format user-turn handling for GLM[1214]; explicit-replay missing-reasoning branch.tests/unit/opencode-background-identity-headers.test.ts,tests/unit/opencode-reasoning-presence-responses.test.ts,tests/unit/glm-1214-same-format-user-turn.test.ts.3. Combo dispatch: per-target skip diagnostics + tiny-budget reasoning probe (closes #12659)
open-sse/services/combo/decisionTrace.ts+combo.ts: persisted per-target skip reasons surface in the ALL_TARGETS_SKIPPED diagnostics instead of a bare 503.max_tokens256) that get truncated return a truncated 200 viabuildReasoningProbeTruncatedResponsein both dispatchers (was a quality-validation 502 "reasoning consumed N/N tokens").tests/unit/12294-skipped-target-diagnostics.test.ts,tests/unit/10281-combo-reasoning-probe.test.ts.4. Ops robustness (closes #12658)
open-sse/handlers/embeddings.ts,src/lib/memory/embedding/remote.ts; testembeddings-input-clamp.test.ts.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS(none/off/false/0 disables; CSV replaces the default opencode free-tier family) —open-sse/config/providerErrorRules.ts; testegress-ip-lock-env-override.test.ts.src/shared/utils/httpClientAbortGuard.mjs+ sse handler) and bounded transport abort inside combos.fetchStartTimeoutCapMsactually reaches the runtime registry entries (open-sse/config/providerRegistry.ts +legacyprojection), raising the streaming cap for buffered gateways (opencode-go / command-code).x-opencode-sessionsynthesis on the executor client leg — complementary to item 2 (background leg); both touchopen-sse/utils/opencodeHeaders.tsbut different functions; merge order is safe.Testing
tsx/esm): fetch-start-cap projection, opencode background identity headers (6/6), opencode reasoning presence, glm-1214 same-format, embeddings clamp, egress-IP env override, skipped-target diagnostics (5/5), combo reasoning probe, transport-bounded abort, self-probe watchdog, error classifier (35/35) — all green.release/v3.8.51(17 upstream fixes); local gate run: changelog-integrity OK, api-docs-refs OK, docs-all OK (migration counts 169→170, env/docs contract), agent-skills sync OK, file-size gate OK.Deployment evidence
On the affected production instance:
ja3_hash 24a38702…h2 vs Node defaultd67b0948…); 1010 rejections stopped. With the adaptive ladder in place,ENABLE_TLS_FINGERPRINTmay stay off until 1010s recur.