Repository navigation
fix(combo): per-target skip diagnostics for ALL_TARGETS_SKIPPED + tiny-budget reasoning probe returns truncated 200 - #12661
Closed
alvinveroy wants to merge 10 commits into
Closed
alvinveroy wants to merge 10 commits into
alvinveroy wants to merge 10 commits into
Conversation
…ode background identity headers + staged resilience work # Conflicts: # open-sse/executors/base.ts # open-sse/handlers/embeddings.ts # open-sse/services/combo.ts # open-sse/services/opencodeQuotaFetcher.ts # src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx # src/i18n/messages/in.json # src/lib/usage/callLogArtifacts.ts # src/shared/constants/visionModels.ts # src/shared/utils/httpClientAbortGuard.mjs # tests/unit/call-log-cap.test.ts # tests/unit/httpClientAbortGuard.test.mjs
…10, with a first-byte watchdog Root cause of the 2026-09-03/04 outage: with ENABLE_TLS_FINGERPRINT=true every direct inference dispatch rode the wreq-js (browser-JA3) transport, whose responses resolved but streamed ZERO bytes for 90-600s on opencode-go / command-code / bailian (agents aborted -> 499 storms; the then-live 110s headers-wait default turned the hangs into 504s). Plain node fetch streamed the same requests in seconds and control-plane calls were unaffected — the stall is transport+streaming-specific, not DNS/CF slow-walking. - wreq is no longer eager by default: plain direct fetch is the fast path; a Cloudflare fingerprint rejection (403 + 1010 / browser_signature_banned) arms a 1h per-provider cache and retries once via wreq (replay-safe requests only). TLS_FINGERPRINT_PROVIDERS keeps its legacy eager semantics, now guarded. - Every wreq response (direct, ladder retry, proxied overlay) carries a first-byte watchdog (TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS, default 10s): no first byte -> cancel and fall back to the direct dispatcher instead of hanging the caller. Verified live: fingerprint off + ladder armed-on-demand restores TTFB 3-6s on opencode-go (was 90-600s of zero bytes), zero 1010 recurrence in the window.
…y-budget reasoning probe (cheaper-inference#12294, cheaper-inference#10281) Production evidence (2026-09-03): smart/yano-free/yano-openweights 503'd 1419x in one hour with diagnostics.attempted=0, excluded=[] — operators could not see WHY every target was pre-dispatch filtered (cheaper-inference#12294). - decisionTrace: persisted_cooldown skip reason + entry detail + pure summarizeSkippedTargets() aggregator. - combo.ts (both dispatchers): persisted-cooldown skips record a trace decision; buildComboDiag() emits skippedTargets[] + nextRetryAt (earliest cooldown reset) in the 503 body. - chat.ts: API-key allowlist rejections during combo target screening log at info with the key name (was debug-only; an allowlist mismatch silently emptied the pool). - validateQuality + combo dispatchers: a tiny-budget reasoning probe (max_tokens < 256) whose upstream 200 burned the whole budget on thinking returns a truncated 200 (finish_reason length) instead of 502 + model lockout (cheaper-inference#10281 combo-path parity). - glm [1214] / reasoning-presence fixes live in the sibling opencode-identity-headers branch.
… watchdog 2026-09-05 wedge incident: opencode h2 stream resets (ERR_HTTP2_STREAM_ERROR, UND_ERR_SOCKET) fanned out through every combo target (8 fallbacks / 18 decisions per request), then a client abort during combo cleanup left the event loop wedged — listener accepting, nothing served, nothing logged, for hours. The existing supervisor only restarts on process exit. - Bounded transport abort (both dispatchers): consecutive transport-class network failures share the same egress — after DEFAULT_TRANSPORT_ABORT_BOUND (3), abort the combo with an explicit 502 instead of burning the remaining fan-out. Non-transport failures reset the counter. New predicates isTransportClassNetworkError / shouldAbortComboForTransportFailures in comboPredicates.ts. - Self-probe wedge watchdog (scripts/dev/self-probe-watchdog.mjs, wired in standalone-server-ws.mjs after the listener accepts): loopback probe of /api/status on the same event loop — a wedged loop cannot answer it. N consecutive connect/timeout failures (any HTTP response counts as alive, incl. 401) exit(1) so launchd KeepAlive / systemd / Docker relaunch. Config: OMNIROUTE_SELF_PROBE(_INTERVAL_MS|_TIMEOUT_MS|_THRESHOLD), default 60s/15s/3; OMNIROUTE_SELF_PROBE=off disables. - file-size rebaseline for the touched frozen files with justification.
…in TLS watchdog; stryker tap registration - dashboard/combos/page.tsx: defer the post-hydration localStorage correction one microtask so setShowUsageGuide is not a synchronous setState inside the effect body (react-hooks/set-state-in-effect base red on release/v3.8.51 from cheaper-inference#12448); eslint:json gate green. - proxyFetch.ts withTlsFirstByteWatchdog: the watchdog race consumed the first body chunk from the reader but never re-emitted it — the first SSE frame was silently dropped. The rest stream now re-emits the captured chunk before continuing the original stream. - stryker.conf.json: register 10281-combo-reasoning-probe + transport-bounded-abort in tap.testFiles (they cover mutated modules; --strict gate). - callLogArtifacts.ts / earlier dashboard stray: restore pristine remote state for files untouched by this branch (resolves unused-var lint).
…-diagnostics # Conflicts: # config/quality/file-size-baseline.json
Contributor
Author
|
Superseded by #12531 — consolidating the 3.8.51 fork batch. This head branch carried the same cumulative content (this PR's topic + the TLS-fingerprint, opencode-identity, and ops work) on an older base and is CONFLICTING with current release/v3.8.51. #12531 now holds the full updated batch merged onto the latest release/v3.8.51 (17 upstream fixes), all gates green locally (changelog integrity, docs-all, agent-skills sync, file-size), with per-area change notes and tests. Closing as duplicate; reopen only if you want this topic split out separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12659. Implements the diagnostics ask in #12294; #10281 combo-path parity.
Production trigger
2026-09-03: 1419× 503
ALL_TARGETS_SKIPPEDin one hour (diagnostics.attempted=0, excluded=[]) — after the storm was over, the only actionable causes (an API-key allowlist mismatch silently skipping the one healthy target; multi-day persisted cooldowns) were invisible at the API surface and lived in debug-only logs.Change
decisionTrace.ts:persisted_cooldownskip reason + optionaldetail+ puresummarizeSkippedTargets().buildComboDiag()emitsskippedTargets[](target, reason, detail) +nextRetryAt(earliest future cooldown reset parsed from the detail) so clients can back off instead of hammering.chat.ts: allowlist rejections during target screening log at info with the key name (the mismatch class behind this incident: alias wildcardbailian/*vs dynamicopenai-compatible-chat-<uuid>provider id).finish_reason: "length") viabuildReasoningProbeTruncatedResponse— fix(api): answer tiny-budget reasoning probes (max_tokens:1) with truncated 200 — no 500 empty response, no cooldown poisoning #10281 direct-path parity; stops fallback burn + model-lockout poisoning (68× 502 in one day).Testing
tests/unit/12294-skipped-target-diagnostics.test.ts(5) +tests/unit/10281-combo-reasoning-probe.test.ts(3) + fulltests/unit/combo/*and reasoning suites green.