From 229cc15ddf41b6dfe7d678d07502dae76ee0a24c Mon Sep 17 00:00:00 2001 From: Dizzle <112548150+maxmad64bis@users.noreply.github.com> Date: Tue, 22 Sep 2026 22:57:16 +0200 Subject: [PATCH 1/2] feat(opencode): bound Responses headers wait with opt-in rotation budget No default change, off-by-default: without OPENCODE_RESPONSES_HEADERS_WAIT_MS the dispatch path is untouched (window 0, no timer). When an operator sets a positive window below the effective fetch-start ceiling, a streamed Responses attempt stops waiting for upstream headers after the window and rotates to the next account (budget OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS, default 2); the last remaining account always keeps the full headers window. Expiry reuses the stall settle (slot release, cooldown, tried-set); client aborts propagate and never rotate. Only shortens, never extends, the fetch-start ceiling. --- .env.example | 2 + .../features/14558-opencode-headers-wait.md | 1 + config/quality/file-size-baseline.json | 9 +- docs/reference/ENVIRONMENT.md | 4 + open-sse/executors/opencode.ts | 53 ++- open-sse/executors/opencodeHeadersWait.ts | 209 +++++++++ src/shared/utils/runtimeTimeouts.ts | 29 ++ .../opencode-headers-wait-rotation.test.ts | 405 ++++++++++++++++++ 8 files changed, 700 insertions(+), 12 deletions(-) create mode 100644 changelog.d/features/14558-opencode-headers-wait.md create mode 100644 open-sse/executors/opencodeHeadersWait.ts create mode 100644 tests/unit/opencode-headers-wait-rotation.test.ts diff --git a/.env.example b/.env.example index 8de94b6c4c9..9dc7987dfd0 100644 --- a/.env.example +++ b/.env.example @@ -1818,6 +1818,8 @@ CURSOR_USER_AGENT="Cursor/3.4" # OPENCODE_PARK_AND_RESUME=false # #13924 feature flag (Settings → Feature Flags wins): park the request with a heartbeat after repeated transient 429s, then replay one capped leg of up to 3 accounts #OPENCODE_POOL_STRAIN_MARKER_PATH=/tmp/opencode-pool-strain.json # #13924: pool-strain marker path (JSON {since, reason, ttl_s}); fresh marker parks without recounting # RESPONSES_FIRST_BYTE_TIMEOUT_MS=15000 # #13484: OpenCode Responses first-byte window, only used when the OPENCODE_RESPONSES_STALL_ROTATION flag is on (0 disables) +# OPENCODE_RESPONSES_HEADERS_WAIT_MS=30000 # opt-in bound on waiting for upstream response headers on streamed Responses calls before moving to the next account (0 = off, suggested 30000) +# OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS=2 # rotation budget for the headers-wait bound above # FLUSH_EMPTY_RETRY_ENABLED=false # #14213 feature flag (Settings → Feature Flags wins): retry empty translated streaming turns through the normal credential path (up to STREAM_RECOVERY.EMPTY_TURN_RETRY_MAX retries) # ── API Bridge (/v1 proxy server) ── diff --git a/changelog.d/features/14558-opencode-headers-wait.md b/changelog.d/features/14558-opencode-headers-wait.md new file mode 100644 index 00000000000..e32db3796a5 --- /dev/null +++ b/changelog.d/features/14558-opencode-headers-wait.md @@ -0,0 +1 @@ +- **feat(opencode):** bound the Responses headers wait with an opt-in rotation budget ([#14558](https://github.com/diegosouzapw/OmniRoute/pull/14558)) — thanks @maxmad64bis diff --git a/config/quality/file-size-baseline.json b/config/quality/file-size-baseline.json index c93bfb88d5d..c605058aa2f 100644 --- a/config/quality/file-size-baseline.json +++ b/config/quality/file-size-baseline.json @@ -1,6 +1,5 @@ { - "_rebaseline_2026_09_22_14069_model_not_in_catalog": "PR #14069 (@RaviTharuma) own growth: a live-catalog miss is now recorded as the model_not_in_catalog skip reason instead of collapsing into the generic availability bucket, so an unknown alias stops being reported as \"no credentials available\" (#14068). Measured on the tree reconciled with release/v3.8.51 @ea3c1226: src/sse/handlers/chat.ts 2559->2560 (+1 = the single modelInfo.errorType === \"model_not_found\" early return inside the existing isModelAvailable callback) and open-sse/services/combo/roundRobinCombo.ts 1261->1263 (+2 = the strict `available !== true` pre-check plus the sticky-target expression, which Prettier printWidth 100 reflows over three lines once it becomes `(await isModelAvailable(...)) === true`). Both files were already frozen exactly at their measured size with zero headroom (chat.ts was tightened to 2559 by #14223 on 2026-09-20), so the growth cannot be absorbed. These are call-site lines threading the new model_not_in_catalog skip reason through the two availability chokepoints and nothing else; the reason itself lives outside the frozen files, all under cap: the ModelAvailabilityResult union and modelAvailabilitySkipReason in open-sse/services/combo/types.ts, the COMBO_SKIP_REASONS entry in decisionTrace.ts and the threading in executeTargetGates.ts. Irreducible. Covered by tests/unit/combo/combo-skipped-targets-summary.test.ts. Structural shrink of both god-files stays tracked in #3501.", - "_rebaseline_2026_09_22_11725_cpa_auth_index": "PR for #11725 own growth: open-sse/handlers/chatCore.ts 6400->6402 (+2). Prettier printWidth 100 keeps the failure-usage cpaAuthIndex property and the readCpaAuthIndex import on their own lines; the streaming and non-streaming call sites stay on the existing endpoint line. The parser, stamp, and label join live in open-sse/handlers/chatCore/cpaTraceAuthIndex.ts (under cap). Covered by tests/unit/cpa-trace-auth-index.test.ts, tests/unit/cpa-auth-index-usage.test.ts, and tests/unit/db/migration-185-cpa-auth-index.test.ts.", + "_rebaseline_2026_09_22_headers_wait_optin_seam": "Opt-in Responses headers-wait rotation budget (off by default): open-sse/executors/opencode.ts 1318->1355 (+37 irreducible seam: policy-once-per-request state call + headersWaitDispatch branch with outcome ok/expired/aborted, abort rethrow guard, settleStalledDispatch reuse; bulk logic in new leaf open-sse/executors/opencodeHeadersWait.ts 210 lines under cap). Covered by tests/unit/opencode-headers-wait-rotation.test.ts (17/17) + stall/timeouts/transient/park/throttle neighbors (67/67).", "_rebaseline_2026_09_20_14223_rotation_attribution_growth": "PR #14223 own growth: rotation attribution diagnostics (masked serving-account id + request correlation id on proxy log rows, per-account rotation state, skip lines). Re-measured on the tree reconciled with release/v3.8.51 @59de50e4 (which brought #14149 -- the MuseSpark block and the per-request format/session context moved out of opencode.ts -- #14226 and #14464): open-sse/executors/opencode.ts 1251->1318 gate count (re-measured again after #14353 moved the member list off the instance) (snapshotEntries + logSkippedCooldownAccounts + attribution wiring at the nine existing rotation exits incl. the two park-and-replay returns; logic kept at the rotation seam), src/sse/handlers/chat.ts 2547->2559 (attribution sink type + flag read + two forwarded fields at the existing log call-site), src/sse/handlers/chatHelpers.ts 1253->1257 (+4, two additive optional params forwarded to the journal row), open-sse/utils/proxyFetch.ts 1268->1287 (+19, AppliedProxySink rotationAccount field + noteRotationAccount helper). Irreducible spec wiring at existing chokepoints, additive and flag-off inert. Covered by 6 new test files (17 tests).", "_rebaseline_2026_09_20_prettier_frozen": "Prettier-only pass over frozen proxy files (verified green on the base with split(\"\\n\").length counting): open-sse/utils/proxyFetch.ts 1276->1268 (inherited shrink ratchet, re-measured on the reconciled release/v3.8.51 tip @373c31f3 -- not this PR's growth); src/sse/handlers/chat.ts 2547= ; src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477= (iso-LOC format).", "_rebaseline_2026_09_21_14250_member_egress_lines": "PR #14250 own growth: src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477->1479 (+2 = the PoolMemberEgressLines import and its one-line mount under the pool members label, next to PoolEgressObservation). The member-egress observation itself lives outside the frozen file, all under cap: PoolMemberEgressLines.tsx, the dedicated GET /api/settings/proxies/pool/member-egress route, readPoolMemberEgressObservation in src/lib/proxyPoolEgressObservation.ts and getRecentEgressIpForProxy in src/lib/db/proxyLogs.ts. Only the mount point is irreducible. Covered by tests/unit/proxy-pool-member-egress-route.test.ts and tests/unit/ui/PoolMemberEgressLines.test.tsx.", @@ -535,7 +534,7 @@ "open-sse/executors/deepseek-web.ts": 1224, "open-sse/executors/default.ts": 1205, "open-sse/services/rateLimitManager.ts": 1329, - "open-sse/executors/opencode.ts": 1318 + "open-sse/executors/opencode.ts": 1355 }, "_rebaseline_2026_09_15_roundrobin_dashboard_events": "Fix #13089 (Combo Studio Live dashboard shows an empty backlog for round-robin combos): open-sse/services/combo/roundRobinCombo.ts 1205->1213. Round-robin is the only combo strategy that bypasses handleComboChat/executeTargetAttempt.ts, the path that publishes the combo.target.attempt/succeeded/failed EventBus events the Live dashboard listens for — so round-robin completions never showed up. The new call-site wiring (createRRDashboardEvents(...) instantiated once per target, one-line .attempt()/.succeeded()/.failed() calls at the 6 existing dispatch/outcome points) is the emitter logic actually extracted into a new module, open-sse/services/combo/rrDashboardEvents.ts — this is the minimum irreducible footprint for wiring 6 required call sites into 6 fixed control-flow points of the frozen file. Covered by tests/unit/issue-13089-roundrobin-live-ws-events.test.ts (2 tests: success + failure paths).", "_rebaseline_base_2026_08_10_proxyfetch": "Base-red fix (green-prs sweep, issue #9985): open-sse/utils/proxyFetch.ts 1207 > cap 1000 — new proxied-TLS fetch helper introduced by the Fal reference-image work. Owner-authorized quick rebaseline to green; structural slim tracked for v3.9.0.", @@ -733,5 +732,7 @@ "_rebaseline_2026_09_17c_chatcore_translation_paths_test": "tests/unit/chatcore-translation-paths.test.ts 3447->3449 (#13173, prefixos de cache de meio de conversa do Fable — as assercoes novas do caso). Ultimo teto remanescente da leva de merges de 2026-09-17; os outros dois (chatHelpers.ts e chatCore.ts) foram absorvidos pelos rebaselines das proprias PRs que mergearam depois. Medido no tip limpo.", "_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it.", "_rebaseline_2026_09_19_14162_native_codex_auto_resume": "PR #14162 (re-land of #13180, @mdigitalbh81 via @diegosouzapw): native Codex turn auto-resume. open-sse/services/combo/executeTargetAttempt.ts 1258->1273 (+15). Growth is 100% the PR's own, measured against the clean tip (1258 there, gate green): the pin step now advances the logical turn generation and logs the resumed provider/model when the attempt is an auto-resume dispatch, and the generation is passed into pinNativeCodexTurn — the branch has to sit at the pin site because that is the only place the winning target and effective connection are known. Covered by tests/unit/native-codex-auto-resume.test.ts + native-codex-auto-resume-guards.test.ts (15/15) and #13564's native-codex-turn-pin-model-scoped-fallback.test.ts (7/7).", - "_rebaseline_2026_09_22_14405_freetier_observed_tools_retry": "Issue #14405 own growth: open-sse/executors/opencode.ts 1251->1303 (measured after merging release/v3.8.51, whose own drift entry _rebaseline_2026_09_22_opencode_train10c_drift had already moved the ceiling 1247->1251; the extra 5 lines over the original 1247->1294 measurement are the #14148 reconciliation: freeTierRetryCtx now reads the contract attempt back from the request body via attemptFor() instead of the removed shared _contractAttempt field) (+47 irreducible call-site wiring: freeTierRetryCtx builder + direct fast-path retry call + rotation-loop refusal arm delegating to handleLoopFreeTierRefusal + api-typecheck fixes (unknown-cast on stall-guard dispatch, callback-shaped loop handler keeping private finalize methods); the retry logic itself lives in the new module open-sse/executors/opencodeFreeTierRetry.ts (131 lines, under cap) and the merge helper in opencodeFreeTierContract.ts, so no logic was added to the frozen file beyond wiring two existing dispatch arms. Compaction attempts measured and reverted: loop-call wrapper (+22 net), fast-path/loop fusion (fragile: retry-403 vs original-403 share status). Covered by tests/unit/opencode-free-tier-refusal-rotation.test.ts (3 retry tests: union success, original refusal + store untouched, no retry without names) + tests/unit/opencode-free-tier-request-contract.test.ts (2 merge tests)." + "_rebaseline_2026_09_22_14405_freetier_observed_tools_retry": "Issue #14405 own growth: open-sse/executors/opencode.ts 1251->1303 (measured after merging release/v3.8.51, whose own drift entry _rebaseline_2026_09_22_opencode_train10c_drift had already moved the ceiling 1247->1251; the extra 5 lines over the original 1247->1294 measurement are the #14148 reconciliation: freeTierRetryCtx now reads the contract attempt back from the request body via attemptFor() instead of the removed shared _contractAttempt field) (+47 irreducible call-site wiring: freeTierRetryCtx builder + direct fast-path retry call + rotation-loop refusal arm delegating to handleLoopFreeTierRefusal + api-typecheck fixes (unknown-cast on stall-guard dispatch, callback-shaped loop handler keeping private finalize methods); the retry logic itself lives in the new module open-sse/executors/opencodeFreeTierRetry.ts (131 lines, under cap) and the merge helper in opencodeFreeTierContract.ts, so no logic was added to the frozen file beyond wiring two existing dispatch arms. Compaction attempts measured and reverted: loop-call wrapper (+22 net), fast-path/loop fusion (fragile: retry-403 vs original-403 share status). Covered by tests/unit/opencode-free-tier-refusal-rotation.test.ts (3 retry tests: union success, original refusal + store untouched, no retry without names) + tests/unit/opencode-free-tier-request-contract.test.ts (2 merge tests).", + "_rebaseline_2026_09_22_14069_model_not_in_catalog": "PR #14069 (@RaviTharuma) own growth: a live-catalog miss is now recorded as the model_not_in_catalog skip reason instead of collapsing into the generic availability bucket, so an unknown alias stops being reported as \"no credentials available\" (#14068). Measured on the tree reconciled with release/v3.8.51 @ea3c1226: src/sse/handlers/chat.ts 2559->2560 (+1 = the single modelInfo.errorType === \"model_not_found\" early return inside the existing isModelAvailable callback) and open-sse/services/combo/roundRobinCombo.ts 1261->1263 (+2 = the strict `available !== true` pre-check plus the sticky-target expression, which Prettier printWidth 100 reflows over three lines once it becomes `(await isModelAvailable(...)) === true`). Both files were already frozen exactly at their measured size with zero headroom (chat.ts was tightened to 2559 by #14223 on 2026-09-20), so the growth cannot be absorbed. These are call-site lines threading the new model_not_in_catalog skip reason through the two availability chokepoints and nothing else; the reason itself lives outside the frozen files, all under cap: the ModelAvailabilityResult union and modelAvailabilitySkipReason in open-sse/services/combo/types.ts, the COMBO_SKIP_REASONS entry in decisionTrace.ts and the threading in executeTargetGates.ts. Irreducible. Covered by tests/unit/combo/combo-skipped-targets-summary.test.ts. Structural shrink of both god-files stays tracked in #3501.", + "_rebaseline_2026_09_22_11725_cpa_auth_index": "PR for #11725 own growth: open-sse/handlers/chatCore.ts 6400->6402 (+2). Prettier printWidth 100 keeps the failure-usage cpaAuthIndex property and the readCpaAuthIndex import on their own lines; the streaming and non-streaming call sites stay on the existing endpoint line. The parser, stamp, and label join live in open-sse/handlers/chatCore/cpaTraceAuthIndex.ts (under cap). Covered by tests/unit/cpa-trace-auth-index.test.ts, tests/unit/cpa-auth-index-usage.test.ts, and tests/unit/db/migration-185-cpa-auth-index.test.ts." } diff --git a/docs/reference/ENVIRONMENT.md b/docs/reference/ENVIRONMENT.md index dff6cfe841b..44a19314f49 100644 --- a/docs/reference/ENVIRONMENT.md +++ b/docs/reference/ENVIRONMENT.md @@ -756,6 +756,8 @@ REQUEST_TIMEOUT_MS (global override) │ ├─→ TLS_CLIENT_TIMEOUT_MS (inherits from FETCH_TIMEOUT_MS) │ │ └── TLS_FIRST_BYTE_WATCHDOG_MS (independent, default: 10000) │ ├── RESPONSES_FIRST_BYTE_TIMEOUT_MS (independent, default: 15000) +│ ├── OPENCODE_RESPONSES_HEADERS_WAIT_MS (independent, default: 0 = off, suggested: 30000) +│ ├── OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS (independent, default: 2) │ ├── FETCH_CONNECT_TIMEOUT_MS (independent, default: 30000) │ └── FETCH_KEEPALIVE_TIMEOUT_MS (independent, default: 4000) ├─→ STREAM_IDLE_TIMEOUT_MS (inherits from REQUEST_TIMEOUT_MS, default: 600000) @@ -798,6 +800,8 @@ REQUEST_TIMEOUT_MS (global override) | `TLS_CLIENT_TIMEOUT_MS` | = `FETCH_TIMEOUT_MS` | TLS fingerprint proxy (wreq-js) timeout. | | `TLS_FIRST_BYTE_WATCHDOG_MS` | `10000` | Bounds time-to-first-byte on the wreq-js TLS-fingerprint transport's body specifically; `TLS_CLIENT_TIMEOUT_MS` alone cannot catch a stalled body since it resolves as soon as headers arrive (#12656). A timeout cancels the wreq reader and falls back to the direct/proxy dispatcher; `0` disables the watchdog. | | `RESPONSES_FIRST_BYTE_TIMEOUT_MS` | `15000` | OpenCode executor only, and only while the `OPENCODE_RESPONSES_STALL_ROTATION` feature flag is on (default off): bounds the wait for the first body byte of a streamed Responses reply after its headers (#13484). A Responses stream opens with `response.created`, so silence past this window is a stall: the account is cooled down and the request rotates to the next account once; a second stall fails fast. `0` disables the guard even with the flag on. | +| `OPENCODE_RESPONSES_HEADERS_WAIT_MS` | `0` (= off) | OpenCode executor only: bounds the wait for upstream response headers on a streamed Responses call when another account is still available — a queued request stops waiting and moves on instead of holding the full headers window. Only streamed Responses calls are affected (the Responses stream opens with `response.created`, so silence there is a queue, not generation); chat completions and non-streamed calls are untouched, and the window only shortens the effective fetch-start ceiling, never extends it. Suggested value when enabling: `30000`. `0` disables. | +| `OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS` | `2` | OpenCode executor only: how many times per request the headers-wait bound above may move to the next account. The last remaining account always keeps the full headers window. | | `OPENCODE_PARK_AND_RESUME` | `false` | OpenCode executor only: park the request with a heartbeat after repeated transient 429s (or a fresh pool-strain marker), then replay one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet (#13924). Off by default: every 429 rotates to the next account exactly as before. | | `OPENCODE_POOL_STRAIN_MARKER_PATH` | _(unset)_ | OpenCode executor only: override path of the pool-strain marker read before parking (`{since, reason, ttl_s}`, default `/tmp/opencode-pool-strain.json`, #13924). A fresh marker parks without recounting; absent or stale falls back to the burst counter. | | `API_BRIDGE_PROXY_TIMEOUT_MS` | `30000` | Proxy hop timeout for `/v1` bridge requests. | diff --git a/open-sse/executors/opencode.ts b/open-sse/executors/opencode.ts index acf27a447d4..78071cf662e 100644 --- a/open-sse/executors/opencode.ts +++ b/open-sse/executors/opencode.ts @@ -80,6 +80,7 @@ import { resolveResponsesStallWindowMs, } from "./opencodeResponsesStall.ts"; import { discardResponseBody } from "./opencodeResponseBody.ts"; +import { headersWaitDispatch, headersWaitState } from "./opencodeHeadersWait.ts"; import { isRetriableUpstreamFailure, releaseResponseBody, @@ -531,6 +532,12 @@ export class OpencodeExecutor extends BaseExecutor { // Opt-in Responses first-byte stall guard (#13484); a no-op when the window is 0. const stallWindowMs = resolveResponsesStallWindowMs(input.stream, this._requestFormat); const guardStall = (r: T) => guardResponsesStall(r, stallWindowMs, input.signal); + const headersWait = headersWaitState( + input, + this._requestFormat, + this.getTimeoutMs(), + this.config?.fetchStartTimeoutCapMs + ); // Fast path: no multi-account proxy wiring configured → original behavior, // plus exactly ONE bounded retry when the upstream answers a 400 empty // rejection (same predicate and logging as the rotation loop). Everything @@ -622,8 +629,7 @@ export class OpencodeExecutor extends BaseExecutor { // (received refusal or refused TCP probe) are skipped. Off = plain rotation. const skipRecentlyFailed = isProxySkipRecentlyFailedEnabled(); let directTried = false; - // Stalls before the first Responses byte: one rotation, then fail fast. - const stallCounter = { attempts: 0 }; + const stallCounter = { attempts: 0 }; // first-byte stalls: one rotation, then fail fast // A response an opt-in branch rotated away from. It stays lastResult (and // intact) until a newer attempt replaces it, then its body is cancelled. let abandonedResponse: Response | null = null; @@ -747,13 +753,44 @@ export class OpencodeExecutor extends BaseExecutor { account = paced.account; let result: HttpExecuteResult; try { - // super.execute() dispatches the HTTP path (never the web/scraping arm). - result = (await guardStall( - await runWithProxyContext(account.proxy, () => - super.execute({ ...input, skipUpstreamRetry: true }) - ) - )) as HttpExecuteResult; + const { outcome, waitMs } = await headersWaitDispatch( + // opt-in bound on the guarded dispatch (stall guard inside the race) + headersWait, + account, + accounts, + isProxiedCandidate, + (attemptSignal) => + (async () => + guardStall( + await runWithProxyContext(account.proxy, () => + super.execute({ + ...input, + skipUpstreamRetry: true, + signal: attemptSignal ?? input.signal, + }) + ) + ) as Promise)(), + input.signal + ); + if (outcome.kind !== "ok") { + if (outcome.kind === "aborted") + egressPacing.throwPacedError(egressRelease, outcome.reason); + egressPacing.settleStalledDispatch(egressRelease, account, { + tried: geoTriedProxyKeys, + stalled: headersWait.spent, + cooldown: markCooldown, + markDirect: () => (directTried = true), + }); // same settle as the stall arm + log?.warn?.( + "OPENCODE", + `${cid}no response headers within ${waitMs}ms on account ${masked}, rotating to next…` + ); + continue; + } + result = outcome.result; } catch (err) { + if (headersWait.policy.windowMs > 0 && input.signal?.aborted) + egressPacing.throwPacedError(egressRelease, err); // client abort never rotates, slot released const reason = err instanceof Error ? err.message : String(err); // Stall guard: headers arrived, so the egress works — never a shared-egress // outage; proxied and proxy-less accounts rotate alike. A client abort never rotates. diff --git a/open-sse/executors/opencodeHeadersWait.ts b/open-sse/executors/opencodeHeadersWait.ts new file mode 100644 index 00000000000..757fb47c4a2 --- /dev/null +++ b/open-sse/executors/opencodeHeadersWait.ts @@ -0,0 +1,209 @@ +/** + * opencodeHeadersWait.ts — opt-in bound on waiting for upstream response + * headers on streamed Responses calls in the opencode executor. + * + * A streamed Responses reply opens with `response.created` before any + * generation, so when headers take longer than this window while another + * account is still available, the request stops waiting and moves on instead + * of holding the full headers window on a queued request. Chat Completions + * calls (which may buffer the whole answer behind the gateway) and + * non-streamed calls are never bounded: the policy resolves to 0 and the + * dispatch runs untouched. + * + * Activation is the window value alone (default 0 = off). The window only + * ever shortens the effective fetch-start ceiling, never extends it. + */ + +import { resolveFetchStartTimeout } from "../utils/fetchStartTimeoutPolicy.ts"; +import { + getOpencodeResponsesHeadersWaitMaxRotations, + getOpencodeResponsesHeadersWaitMs, +} from "@/shared/utils/runtimeTimeouts"; + +export const HEADERS_WAIT_TIMEOUT_NAME = "OpencodeHeadersWaitTimeout"; +export const HEADERS_WAIT_TIMEOUT_CODE = "OPENCODE_HEADERS_WAIT_TIMEOUT"; + +export type HeadersWaitPolicy = { + /** Effective per-attempt bound, or 0 when the mechanism is off. */ + windowMs: number; + maxRotations: number; +}; + +/** Minimal account shape for the per-attempt eligibility check (identity only). */ +export type ScopedAccountLike = { fingerprint: string }; + +/** Minimal dispatch result: the race only forwards it, never inspects it. */ +export type DispatchResult = { response: Response } & Record; + +export type HeadersWaitWindowInput = { + stream: boolean | undefined; + requestFormat: string | null; + windowMs: number; + capTimeoutMs: number; +}; + +/** + * Per-request policy, computed once before the account loop. Returns 0 unless + * the call is a streamed Responses call with a positive window below the + * effective fetch-start ceiling. + */ +export function resolveHeadersWaitWindowMs(input: HeadersWaitWindowInput): number { + if (!input.stream || input.requestFormat !== "openai-responses") return 0; + if (!(input.windowMs > 0)) return 0; + if (input.windowMs >= input.capTimeoutMs) return 0; + return input.windowMs; +} + +/** Request state for one executeOnce call: policy computed once, rotations spent. */ +export function headersWaitState( + input: { stream?: boolean }, + requestFormat: string | null, + baseTimeoutMs: number, + fetchStartCapMs: number | undefined, + env: Record = process.env +): HeadersWaitRequestState { + const windowMs = getOpencodeResponsesHeadersWaitMs(env); + const maxRotations = getOpencodeResponsesHeadersWaitMaxRotations(env); + if (!input.stream || requestFormat !== "openai-responses" || windowMs <= 0) { + return { policy: { windowMs: 0, maxRotations }, spent: { attempts: 0 } }; + } + const cap = resolveFetchStartTimeout({ + baseTimeoutMs, + stream: input.stream, + capMs: fetchStartCapMs, + }).timeoutMs; + const policy = { + windowMs: resolveHeadersWaitWindowMs({ + stream: input.stream, + requestFormat, + windowMs, + capTimeoutMs: cap, + }), + maxRotations, + }; + return { policy, spent: { attempts: 0 } }; +} + +export type HeadersWaitRequestState = { + policy: HeadersWaitPolicy; + /** Rotations spent, shared by reference with the settle call (same shape as the stall counter). */ + spent: { attempts: number }; +}; + +/** Per-attempt wait: the policy window only while budget and another account remain. */ +export function headersWaitWaitMs( + state: HeadersWaitRequestState, + account: T, + accounts: T[], + isCandidate: (a: T) => boolean +): number { + if (state.policy.windowMs <= 0 || state.spent.attempts >= state.policy.maxRotations) return 0; + const another = accounts.some((a) => a !== account && isCandidate(a)); + return another ? state.policy.windowMs : 0; +} + +/** + * One rotation-loop attempt under the headers-wait bound: resolve the + * per-attempt wait from the request state and race the dispatch against it. + * Returns the race outcome; the caller owns rotate/return/throw. + */ +export async function headersWaitDispatch( + state: HeadersWaitRequestState, + account: ScopedAccountLike, + accounts: ScopedAccountLike[], + isCandidate: (a: ScopedAccountLike) => boolean, + dispatch: (signal: AbortSignal | null | undefined) => Promise, + signal?: AbortSignal | null +): Promise<{ outcome: HeadersWaitOutcome; waitMs: number }> { + const waitMs = headersWaitWaitMs(state, account, accounts, isCandidate); + if (!(waitMs > 0)) { + return { + outcome: await raceDispatchWithHeadersWait(() => dispatch(signal), 0, signal), + waitMs, + }; + } + if (signal?.aborted) return { outcome: { kind: "aborted", reason: signal.reason }, waitMs }; + const attempt = new AbortController(); + const onClientAbort = () => attempt.abort(signal?.reason); + signal?.addEventListener("abort", onClientAbort, { once: true }); + try { + const outcome = await raceDispatchWithHeadersWait( + () => dispatch(attempt.signal), + waitMs, + signal + ); + if (outcome.kind === "expired") attempt.abort(headersWaitExpiryError(waitMs)); + return { outcome, waitMs }; + } finally { + signal?.removeEventListener("abort", onClientAbort); + } +} + +export type HeadersWaitOutcome = + { kind: "ok"; result: T } | { kind: "expired" } | { kind: "aborted"; reason: unknown }; + +/** + * Race one dispatch against the headers-wait window. `waitMs <= 0` runs the + * dispatch untouched (no timer). Otherwise the dispatch races a window timer: + * expiry yields `expired` (never throws), a client abort yields `aborted`, + * anything else `ok` — dispatch rejections propagate untouched. + */ +export async function raceDispatchWithHeadersWait( + dispatch: () => Promise, + waitMs: number, + signal?: AbortSignal | null +): Promise> { + if (!(waitMs > 0)) { + return { kind: "ok", result: await dispatch() }; + } + if (signal?.aborted) { + return { kind: "aborted", reason: signal.reason }; + } + let timer: ReturnType | null = null; + try { + return await new Promise>((resolve, reject) => { + const cleanup = () => { + if (timer !== null) { + clearTimeout(timer); + timer = null; + } + signal?.removeEventListener("abort", onAbort); + }; + const onAbort = () => { + cleanup(); + resolve({ kind: "aborted", reason: signal?.reason }); + }; + signal?.addEventListener("abort", onAbort, { once: true }); + if (signal?.aborted) { + onAbort(); + return; + } + timer = setTimeout(() => { + cleanup(); + if (signal?.aborted) resolve({ kind: "aborted", reason: signal.reason }); + else resolve({ kind: "expired" }); + }, waitMs); + void dispatch().then( + (result) => { + cleanup(); + resolve({ kind: "ok", result }); + }, + (reason) => { + cleanup(); + if (signal?.aborted) resolve({ kind: "aborted", reason }); + else reject(reason); + } + ); + }); + } finally { + if (timer !== null) clearTimeout(timer); + } +} + +/** Build the expiry error (distinct from TimeoutError — never logged as a fetch-start TIMEOUT). */ +export function headersWaitExpiryError(waitMs: number): Error { + const err = new Error(`No response headers within ${waitMs}ms, rotating to next account`); + err.name = HEADERS_WAIT_TIMEOUT_NAME; + (err as { code?: string }).code = HEADERS_WAIT_TIMEOUT_CODE; + return err; +} diff --git a/src/shared/utils/runtimeTimeouts.ts b/src/shared/utils/runtimeTimeouts.ts index 8ef930050a6..4ca4020d386 100644 --- a/src/shared/utils/runtimeTimeouts.ts +++ b/src/shared/utils/runtimeTimeouts.ts @@ -59,6 +59,10 @@ export const DEFAULT_TLS_FIRST_BYTE_WATCHDOG_MS = 10_000; // thinking. Executors that can rotate accounts use it to move on instead of waiting for the // readiness timeout. Set to 0 to disable. export const DEFAULT_RESPONSES_FIRST_BYTE_TIMEOUT_MS = 15_000; +// Suggested operator value when enabling the Responses headers-wait bound below. +// Not an active default: the getters read 0 (off) unless the operator sets the env var. +export const SUGGESTED_OPENCODE_RESPONSES_HEADERS_WAIT_MS = 30_000; +export const DEFAULT_OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS = 2; function hasEnvValue(env: EnvSource, name: string): boolean { const raw = env[name]; @@ -269,6 +273,31 @@ export function getResponsesFirstByteTimeoutMs( ); } +// Bound on waiting for upstream response headers on streamed Responses calls +// before moving to the next account. Off by default: the read default is 0 +// (unchanged behavior) — SUGGESTED_… is only the documented starting value. +export function getOpencodeResponsesHeadersWaitMs( + env: EnvSource = process.env, + logger?: TimeoutLogger +): number { + return readTimeoutMs(env, "OPENCODE_RESPONSES_HEADERS_WAIT_MS", 0, { + allowZero: true, + logger, + }); +} + +export function getOpencodeResponsesHeadersWaitMaxRotations( + env: EnvSource = process.env, + logger?: TimeoutLogger +): number { + return readTimeoutMs( + env, + "OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS", + DEFAULT_OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS, + { allowZero: true, logger } + ); +} + export function getApiBridgeTimeoutConfig( env: EnvSource = process.env, logger?: TimeoutLogger diff --git a/tests/unit/opencode-headers-wait-rotation.test.ts b/tests/unit/opencode-headers-wait-rotation.test.ts new file mode 100644 index 00000000000..ef075d3dda6 --- /dev/null +++ b/tests/unit/opencode-headers-wait-rotation.test.ts @@ -0,0 +1,405 @@ +import { describe, it, beforeEach, afterEach, before, after } from "node:test"; +import assert from "node:assert/strict"; +import net from "node:net"; +import { + HEADERS_WAIT_TIMEOUT_CODE, + HEADERS_WAIT_TIMEOUT_NAME, + headersWaitWaitMs, + raceDispatchWithHeadersWait, + resolveHeadersWaitWindowMs, +} from "../../open-sse/executors/opencodeHeadersWait.ts"; +import { OpencodeExecutor } from "../../open-sse/executors/opencode.ts"; +import type { ProviderCredentials } from "../../open-sse/executors/base.ts"; +import { resolveProxyForRequest } from "../../open-sse/utils/proxyFetch.ts"; +import { resetDbInstance } from "../../src/lib/db/core.ts"; +import { + getOpencodeResponsesHeadersWaitMs, + getOpencodeResponsesHeadersWaitMaxRotations, + SUGGESTED_OPENCODE_RESPONSES_HEADERS_WAIT_MS, +} from "../../src/shared/utils/runtimeTimeouts.ts"; + +const MS = "OPENCODE_RESPONSES_HEADERS_WAIT_MS"; +const MAX = "OPENCODE_RESPONSES_HEADERS_WAIT_MAX_ROTATIONS"; + +describe("Responses headers-wait bound (RED first)", () => { + let priorMs: string | undefined; + let priorMax: string | undefined; + + beforeEach(() => { + priorMs = process.env[MS]; + priorMax = process.env[MAX]; + delete process.env[MS]; + delete process.env[MAX]; + }); + + afterEach(() => { + if (priorMs === undefined) delete process.env[MS]; + else process.env[MS] = priorMs; + if (priorMax === undefined) delete process.env[MAX]; + else process.env[MAX] = priorMax; + }); + + it("stays off without env: getters read 0", () => { + assert.equal(getOpencodeResponsesHeadersWaitMs(), 0); + assert.equal(SUGGESTED_OPENCODE_RESPONSES_HEADERS_WAIT_MS, 30_000); + }); + + it("reads the operator values when set", () => { + process.env[MS] = "30000"; + process.env[MAX] = "3"; + assert.equal(getOpencodeResponsesHeadersWaitMs(), 30_000); + assert.equal(getOpencodeResponsesHeadersWaitMaxRotations(), 3); + }); + + it("default rotation budget is 2", () => { + assert.equal(getOpencodeResponsesHeadersWaitMaxRotations(), 2); + }); + + it("window applies to streamed Responses only", () => { + const cap = 110_000; + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai-responses", + windowMs: 30_000, + capTimeoutMs: cap, + }), + 30_000 + ); + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai", + windowMs: 30_000, + capTimeoutMs: cap, + }), + 0 + ); + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "claude", + windowMs: 30_000, + capTimeoutMs: cap, + }), + 0 + ); + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "gemini", + windowMs: 30_000, + capTimeoutMs: cap, + }), + 0 + ); + assert.equal( + resolveHeadersWaitWindowMs({ + stream: false, + requestFormat: "openai-responses", + windowMs: 30_000, + capTimeoutMs: cap, + }), + 0 + ); + }); + + it("window is neutralized when it would not shorten the fetch-start cap", () => { + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai-responses", + windowMs: 110_000, + capTimeoutMs: 110_000, + }), + 0 + ); + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai-responses", + windowMs: 600_000, + capTimeoutMs: 110_000, + }), + 0 + ); + }); + + it("no env means no window (off-by-default)", () => { + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai-responses", + windowMs: 0, + capTimeoutMs: 110_000, + }), + 0 + ); + }); + + it("no host scoping: same window for any provider once the format matches", () => { + assert.equal( + resolveHeadersWaitWindowMs({ + stream: true, + requestFormat: "openai-responses", + windowMs: 30_000, + capTimeoutMs: 600_000, + }), + 30_000 + ); + }); + + it("ok dispatch resolves before the window", { timeout: 5000 }, async () => { + const outcome = await raceDispatchWithHeadersWait( + () => new Promise((resolve) => setTimeout(() => resolve("done"), 5)), + 1000, + null + ); + assert.equal(outcome.kind, "ok"); + if (outcome.kind === "ok") assert.equal(outcome.result, "done"); + }); + + it("slow dispatch expires without throwing TimeoutError", { timeout: 5000 }, async () => { + const outcome = await raceDispatchWithHeadersWait(() => new Promise(() => {}), 20, null); + assert.equal(outcome.kind, "expired"); + assert.equal(HEADERS_WAIT_TIMEOUT_NAME, "OpencodeHeadersWaitTimeout"); + assert.equal(HEADERS_WAIT_TIMEOUT_CODE, "OPENCODE_HEADERS_WAIT_TIMEOUT"); + assert.notEqual(HEADERS_WAIT_TIMEOUT_NAME, "TimeoutError"); + }); + + it("client abort surfaces as aborted, never expired", { timeout: 5000 }, async () => { + const controller = new AbortController(); + setTimeout(() => controller.abort(), 10); + const outcome = await raceDispatchWithHeadersWait( + () => new Promise(() => {}), + 1000, + controller.signal + ); + assert.equal(outcome.kind, "aborted"); + }); + + it("waitMs 0 dispatches untouched with no timer", { timeout: 5000 }, async () => { + let calls = 0; + const outcome = await raceDispatchWithHeadersWait( + () => { + calls++; + return Promise.resolve("direct"); + }, + 0, + null + ); + assert.equal(outcome.kind, "ok"); + assert.equal(calls, 1); + }); + + it("dispatch rejections propagate (never mapped to expired)", { timeout: 5000 }, async () => { + const failure = new TypeError("fetch failed"); + await assert.rejects( + raceDispatchWithHeadersWait(() => Promise.reject(failure), 1000, null), + (err: unknown) => err === failure + ); + }); + + it("per-attempt wait needs budget and another candidate", () => { + const policy = { windowMs: 30_000, maxRotations: 2 }; + const a = { id: "a" }; + const b = { id: "b" }; + const accounts = [a, b]; + assert.equal( + headersWaitWaitMs({ policy, spent: { attempts: 0 } }, a, accounts, () => true), + 30_000 + ); + assert.equal( + headersWaitWaitMs({ policy, spent: { attempts: 2 } }, a, accounts, () => true), + 0, + "budget spent keeps the full window" + ); + assert.equal( + headersWaitWaitMs({ policy, spent: { attempts: 0 } }, a, [a], () => true), + 0, + "last account keeps the full window" + ); + assert.equal( + headersWaitWaitMs( + { policy: { windowMs: 0, maxRotations: 2 }, spent: { attempts: 0 } }, + a, + accounts, + () => true + ), + 0 + ); + }); +}); + +const HW_LOG = { debug() {}, info() {}, warn() {}, error() {} }; +const HW_RESPONSES_MODEL = "muse-spark-1.2-contributor-free"; +const HW_CHAT_MODEL = "deepseek-v4-flash-free"; +const HW_FPS = ["a".repeat(32), "b".repeat(32), "c".repeat(32)]; +const hwServers: net.Server[] = []; +const hwPorts: number[] = []; + +function hwListen(server: net.Server): Promise { + return new Promise((resolve) => { + server.listen(0, "127.0.0.1", () => resolve((server.address() as net.AddressInfo).port)); + }); +} + +function hwProxiedCredentials(count: number): ProviderCredentials { + const fingerprints = HW_FPS.slice(0, count); + return { + apiKey: null, + accessToken: null, + connectionId: "noauth", + providerSpecificData: { + fingerprints, + accountProxies: fingerprints.map((fp, i) => ({ + fingerprint: fp, + proxy: { type: "http", host: "127.0.0.1", port: hwPorts[i] }, + })), + }, + }; +} + +function hwSseBody(): ReadableStream { + const text = "event: message\ndata: {}\n\n"; + return new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(text)); + controller.close(); + }, + }); +} + +describe("OpencodeExecutor headers-wait rotation (seam)", () => { + let originalFetch: typeof globalThis.fetch; + let priorMs: string | undefined; + let priorMax: string | undefined; + let calls: string[]; + + before(async () => { + for (let i = 0; i < HW_FPS.length; i++) { + const server = net.createServer((s) => s.destroy()); + hwServers.push(server); + hwPorts.push(await hwListen(server)); + } + }); + + after(() => { + hwServers.forEach((s) => s.close()); + resetDbInstance(); + }); + + beforeEach(() => { + originalFetch = globalThis.fetch; + priorMs = process.env[MS]; + priorMax = process.env[MAX]; + process.env[MS] = "40"; + delete process.env[MAX]; + calls = []; + }); + + afterEach(() => { + globalThis.fetch = originalFetch; + if (priorMs === undefined) delete process.env[MS]; + else process.env[MS] = priorMs; + if (priorMax === undefined) delete process.env[MAX]; + else process.env[MAX] = priorMax; + }); + + // A queued first account delays its headers past the window; the second answers at once. + function installHangThenOk() { + let call = 0; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = + typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url; + const resolved = resolveProxyForRequest(url); + calls.push(resolved.proxyUrl ? new URL(resolved.proxyUrl).port : "direct"); + call++; + if (call === 1) { + await new Promise((resolve, reject) => { + const timer = setTimeout(resolve, 5000); + init?.signal?.addEventListener("abort", () => { + clearTimeout(timer); + reject(init.signal?.reason ?? new Error("aborted")); + }); + }); + } + return new Response(hwSseBody(), { + status: 200, + headers: { "Content-Type": "text/event-stream" }, + }); + }) as typeof globalThis.fetch; + } + + function run(exec: OpencodeExecutor, model: string, creds: ProviderCredentials, stream = true) { + return exec.execute({ + model, + body: { input: [{ role: "user", content: "hi" }], stream }, + stream, + signal: null, + credentials: creds, + log: HW_LOG, + }) as Promise<{ response: Response }>; + } + + it( + "rotates past a queued Responses dispatch to a healthy account", + { timeout: 10000 }, + async () => { + const exec = new OpencodeExecutor("opencode-zen"); + installHangThenOk(); + const result = await run(exec, HW_RESPONSES_MODEL, hwProxiedCredentials(2)); + assert.equal(result.response.status, 200); + assert.deepEqual(calls, [String(hwPorts[0]), String(hwPorts[1])]); + await result.response.body?.cancel(); + } + ); + + it("leaves chat/completions dispatches unbounded", { timeout: 10000 }, async () => { + const exec = new OpencodeExecutor("opencode-zen"); + installHangThenOk(); + let settled = false; + globalThis.fetch = (async () => { + calls.push("chat"); + settled = true; + return new Response(hwSseBody(), { + status: 200, + headers: { "Content-Type": "text/event-stream" }, + }); + }) as typeof globalThis.fetch; + const result = await run(exec, HW_CHAT_MODEL, hwProxiedCredentials(2)); + assert.equal(result.response.status, 200); + assert.ok(settled); + assert.equal(calls.length, 1); + await result.response.body?.cancel(); + }); + + it("off by default: a queued dispatch is returned untouched", { timeout: 10000 }, async () => { + delete process.env[MS]; + const exec = new OpencodeExecutor("opencode-zen"); + installHangThenOk(); + const result = await run(exec, HW_RESPONSES_MODEL, hwProxiedCredentials(2)); + assert.equal(result.response.status, 200); + assert.equal(calls.length, 1, "no second account is dispatched"); + await result.response.body?.cancel(); + }); + + it("a client abort during the bound window never rotates", { timeout: 10000 }, async () => { + process.env[MS] = "5000"; + const exec = new OpencodeExecutor("opencode-zen"); + installHangThenOk(); + const controller = new AbortController(); + setTimeout(() => controller.abort(), 30); + await assert.rejects( + exec.execute({ + model: HW_RESPONSES_MODEL, + body: { input: [{ role: "user", content: "hi" }], stream: true }, + stream: true, + signal: controller.signal, + credentials: hwProxiedCredentials(2), + log: HW_LOG, + }) + ); + assert.equal(calls.length, 1, "no dispatch after the client went away"); + }); +}); From 853413812eaf04b088147baa51c99ca9777c2dd1 Mon Sep 17 00:00:00 2001 From: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Date: Thu, 24 Sep 2026 02:53:59 -0300 Subject: [PATCH 2/2] chore(quality): re-measure opencode.ts ceiling on the reconciled release tip The tip had shrunk open-sse/executors/opencode.ts to 1301 lines under the old 1318 ceiling, so the headers-wait seam (+37) lands at 1338, not 1355. Set the frozen ceiling to the measured 1338 and restore the original key order of the neighbouring rebaseline annotations (the branch had moved two of them to the end of the file as a merge artifact). --- config/quality/file-size-baseline.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/config/quality/file-size-baseline.json b/config/quality/file-size-baseline.json index 2988f1f052f..1f4845a8471 100644 --- a/config/quality/file-size-baseline.json +++ b/config/quality/file-size-baseline.json @@ -1,5 +1,7 @@ { - "_rebaseline_2026_09_22_headers_wait_optin_seam": "Opt-in Responses headers-wait rotation budget (off by default): open-sse/executors/opencode.ts 1318->1355 (+37 irreducible seam: policy-once-per-request state call + headersWaitDispatch branch with outcome ok/expired/aborted, abort rethrow guard, settleStalledDispatch reuse; bulk logic in new leaf open-sse/executors/opencodeHeadersWait.ts 210 lines under cap). Covered by tests/unit/opencode-headers-wait-rotation.test.ts (17/17) + stall/timeouts/transient/park/throttle neighbors (67/67).", + "_rebaseline_2026_09_22_headers_wait_optin_seam": "Opt-in Responses headers-wait rotation budget (off by default): open-sse/executors/opencode.ts 1301->1338 gate count (+37 irreducible seam: policy-once-per-request state call + headersWaitDispatch branch with outcome ok/expired/aborted, abort rethrow guard, settleStalledDispatch reuse; bulk logic in new leaf open-sse/executors/opencodeHeadersWait.ts 210 lines under cap). Re-measured 2026-09-24 on the tree reconciled with release/v3.8.51 @6b8c5df6 (the tip had shrunk opencode.ts to 1301 under the old 1318 cap, so the ceiling is set to the measured 1338 instead of 1318+37). Covered by tests/unit/opencode-headers-wait-rotation.test.ts (17/17) + stall/timeouts/transient/park/throttle neighbors.", + "_rebaseline_2026_09_22_14069_model_not_in_catalog": "PR #14069 (@RaviTharuma) own growth: a live-catalog miss is now recorded as the model_not_in_catalog skip reason instead of collapsing into the generic availability bucket, so an unknown alias stops being reported as \"no credentials available\" (#14068). Measured on the tree reconciled with release/v3.8.51 @ea3c1226: src/sse/handlers/chat.ts 2559->2560 (+1 = the single modelInfo.errorType === \"model_not_found\" early return inside the existing isModelAvailable callback) and open-sse/services/combo/roundRobinCombo.ts 1261->1263 (+2 = the strict `available !== true` pre-check plus the sticky-target expression, which Prettier printWidth 100 reflows over three lines once it becomes `(await isModelAvailable(...)) === true`). Both files were already frozen exactly at their measured size with zero headroom (chat.ts was tightened to 2559 by #14223 on 2026-09-20), so the growth cannot be absorbed. These are call-site lines threading the new model_not_in_catalog skip reason through the two availability chokepoints and nothing else; the reason itself lives outside the frozen files, all under cap: the ModelAvailabilityResult union and modelAvailabilitySkipReason in open-sse/services/combo/types.ts, the COMBO_SKIP_REASONS entry in decisionTrace.ts and the threading in executeTargetGates.ts. Irreducible. Covered by tests/unit/combo/combo-skipped-targets-summary.test.ts. Structural shrink of both god-files stays tracked in #3501.", + "_rebaseline_2026_09_22_11725_cpa_auth_index": "PR for #11725 own growth: open-sse/handlers/chatCore.ts 6400->6402 (+2). Prettier printWidth 100 keeps the failure-usage cpaAuthIndex property and the readCpaAuthIndex import on their own lines; the streaming and non-streaming call sites stay on the existing endpoint line. The parser, stamp, and label join live in open-sse/handlers/chatCore/cpaTraceAuthIndex.ts (under cap). Covered by tests/unit/cpa-trace-auth-index.test.ts, tests/unit/cpa-auth-index-usage.test.ts, and tests/unit/db/migration-185-cpa-auth-index.test.ts.", "_rebaseline_2026_09_20_14223_rotation_attribution_growth": "PR #14223 own growth: rotation attribution diagnostics (masked serving-account id + request correlation id on proxy log rows, per-account rotation state, skip lines). Re-measured on the tree reconciled with release/v3.8.51 @59de50e4 (which brought #14149 -- the MuseSpark block and the per-request format/session context moved out of opencode.ts -- #14226 and #14464): open-sse/executors/opencode.ts 1251->1318 gate count (re-measured again after #14353 moved the member list off the instance) (snapshotEntries + logSkippedCooldownAccounts + attribution wiring at the nine existing rotation exits incl. the two park-and-replay returns; logic kept at the rotation seam), src/sse/handlers/chat.ts 2547->2559 (attribution sink type + flag read + two forwarded fields at the existing log call-site), src/sse/handlers/chatHelpers.ts 1253->1257 (+4, two additive optional params forwarded to the journal row), open-sse/utils/proxyFetch.ts 1268->1287 (+19, AppliedProxySink rotationAccount field + noteRotationAccount helper). Irreducible spec wiring at existing chokepoints, additive and flag-off inert. Covered by 6 new test files (17 tests).", "_rebaseline_2026_09_20_prettier_frozen": "Prettier-only pass over frozen proxy files (verified green on the base with split(\"\\n\").length counting): open-sse/utils/proxyFetch.ts 1276->1268 (inherited shrink ratchet, re-measured on the reconciled release/v3.8.51 tip @373c31f3 -- not this PR's growth); src/sse/handlers/chat.ts 2547= ; src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477= (iso-LOC format).", "_rebaseline_2026_09_21_14250_member_egress_lines": "PR #14250 own growth: src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477->1479 (+2 = the PoolMemberEgressLines import and its one-line mount under the pool members label, next to PoolEgressObservation). The member-egress observation itself lives outside the frozen file, all under cap: PoolMemberEgressLines.tsx, the dedicated GET /api/settings/proxies/pool/member-egress route, readPoolMemberEgressObservation in src/lib/proxyPoolEgressObservation.ts and getRecentEgressIpForProxy in src/lib/db/proxyLogs.ts. Only the mount point is irreducible. Covered by tests/unit/proxy-pool-member-egress-route.test.ts and tests/unit/ui/PoolMemberEgressLines.test.tsx.", @@ -535,7 +537,7 @@ "open-sse/executors/deepseek-web.ts": 1224, "open-sse/executors/default.ts": 1205, "open-sse/services/rateLimitManager.ts": 1329, - "open-sse/executors/opencode.ts": 1355 + "open-sse/executors/opencode.ts": 1338 }, "_rebaseline_2026_09_15_roundrobin_dashboard_events": "Fix #13089 (Combo Studio Live dashboard shows an empty backlog for round-robin combos): open-sse/services/combo/roundRobinCombo.ts 1205->1213. Round-robin is the only combo strategy that bypasses handleComboChat/executeTargetAttempt.ts, the path that publishes the combo.target.attempt/succeeded/failed EventBus events the Live dashboard listens for — so round-robin completions never showed up. The new call-site wiring (createRRDashboardEvents(...) instantiated once per target, one-line .attempt()/.succeeded()/.failed() calls at the 6 existing dispatch/outcome points) is the emitter logic actually extracted into a new module, open-sse/services/combo/rrDashboardEvents.ts — this is the minimum irreducible footprint for wiring 6 required call sites into 6 fixed control-flow points of the frozen file. Covered by tests/unit/issue-13089-roundrobin-live-ws-events.test.ts (2 tests: success + failure paths).", "_rebaseline_base_2026_08_10_proxyfetch": "Base-red fix (green-prs sweep, issue #9985): open-sse/utils/proxyFetch.ts 1207 > cap 1000 — new proxied-TLS fetch helper introduced by the Fal reference-image work. Owner-authorized quick rebaseline to green; structural slim tracked for v3.9.0.", @@ -733,7 +735,5 @@ "_rebaseline_2026_09_17c_chatcore_translation_paths_test": "tests/unit/chatcore-translation-paths.test.ts 3447->3449 (#13173, prefixos de cache de meio de conversa do Fable — as assercoes novas do caso). Ultimo teto remanescente da leva de merges de 2026-09-17; os outros dois (chatHelpers.ts e chatCore.ts) foram absorvidos pelos rebaselines das proprias PRs que mergearam depois. Medido no tip limpo.", "_rebaseline_2026_09_18_13929_antigravity_account_lease_merge": "PR #13929 (Re-land of #10011, @Ardem2025 via @diegosouzapw): the Antigravity account lease, merged onto the current release/v3.8.51 tip (which had independently moved chat.ts to 2520 and auth.ts to 3557 via unrelated PRs). Combined ceiling after merge: src/sse/handlers/chat.ts->2541, src/sse/services/auth.ts->3577. The lease registry, its lifecycle glue and its selection glue were extracted into three NEW modules (src/sse/services/antigravityRoutingState.ts, antigravityLeaseLifecycle.ts, antigravityLeaseSelection.ts) precisely to keep this growth to the call sites; what remains in chat.ts/auth.ts is the wiring itself, which cannot be moved out of the selection loop and the dispatch path. Every added hunk is inert unless ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (default false) is on. Covered by tests/unit/antigravity-routing-state.test.ts, antigravity-lease-lifecycle.test.ts and antigravity-account-lease-flag.test.ts. UPDATE (re-sync 2026-09-18 after trains 3b/4d moved the tip): auth.ts 3577->3582 (same +29 own growth over a tip now at 3552). open-sse/executors/base.ts 1753->1754 is NOT this PR's growth — it is release-tip drift from train 3b (#13002 +5 / #13705 -4 net +1, both merged without a baseline entry); absorbed here by the captain session under the owner-approved train-rebaseline policy so the tip stops failing check:file-size for every PR boarding after it.", "_rebaseline_2026_09_19_14162_native_codex_auto_resume": "PR #14162 (re-land of #13180, @mdigitalbh81 via @diegosouzapw): native Codex turn auto-resume. open-sse/services/combo/executeTargetAttempt.ts 1258->1273 (+15). Growth is 100% the PR's own, measured against the clean tip (1258 there, gate green): the pin step now advances the logical turn generation and logs the resumed provider/model when the attempt is an auto-resume dispatch, and the generation is passed into pinNativeCodexTurn — the branch has to sit at the pin site because that is the only place the winning target and effective connection are known. Covered by tests/unit/native-codex-auto-resume.test.ts + native-codex-auto-resume-guards.test.ts (15/15) and #13564's native-codex-turn-pin-model-scoped-fallback.test.ts (7/7).", - "_rebaseline_2026_09_22_14405_freetier_observed_tools_retry": "Issue #14405 own growth: open-sse/executors/opencode.ts 1251->1303 (measured after merging release/v3.8.51, whose own drift entry _rebaseline_2026_09_22_opencode_train10c_drift had already moved the ceiling 1247->1251; the extra 5 lines over the original 1247->1294 measurement are the #14148 reconciliation: freeTierRetryCtx now reads the contract attempt back from the request body via attemptFor() instead of the removed shared _contractAttempt field) (+47 irreducible call-site wiring: freeTierRetryCtx builder + direct fast-path retry call + rotation-loop refusal arm delegating to handleLoopFreeTierRefusal + api-typecheck fixes (unknown-cast on stall-guard dispatch, callback-shaped loop handler keeping private finalize methods); the retry logic itself lives in the new module open-sse/executors/opencodeFreeTierRetry.ts (131 lines, under cap) and the merge helper in opencodeFreeTierContract.ts, so no logic was added to the frozen file beyond wiring two existing dispatch arms. Compaction attempts measured and reverted: loop-call wrapper (+22 net), fast-path/loop fusion (fragile: retry-403 vs original-403 share status). Covered by tests/unit/opencode-free-tier-refusal-rotation.test.ts (3 retry tests: union success, original refusal + store untouched, no retry without names) + tests/unit/opencode-free-tier-request-contract.test.ts (2 merge tests).", - "_rebaseline_2026_09_22_14069_model_not_in_catalog": "PR #14069 (@RaviTharuma) own growth: a live-catalog miss is now recorded as the model_not_in_catalog skip reason instead of collapsing into the generic availability bucket, so an unknown alias stops being reported as \"no credentials available\" (#14068). Measured on the tree reconciled with release/v3.8.51 @ea3c1226: src/sse/handlers/chat.ts 2559->2560 (+1 = the single modelInfo.errorType === \"model_not_found\" early return inside the existing isModelAvailable callback) and open-sse/services/combo/roundRobinCombo.ts 1261->1263 (+2 = the strict `available !== true` pre-check plus the sticky-target expression, which Prettier printWidth 100 reflows over three lines once it becomes `(await isModelAvailable(...)) === true`). Both files were already frozen exactly at their measured size with zero headroom (chat.ts was tightened to 2559 by #14223 on 2026-09-20), so the growth cannot be absorbed. These are call-site lines threading the new model_not_in_catalog skip reason through the two availability chokepoints and nothing else; the reason itself lives outside the frozen files, all under cap: the ModelAvailabilityResult union and modelAvailabilitySkipReason in open-sse/services/combo/types.ts, the COMBO_SKIP_REASONS entry in decisionTrace.ts and the threading in executeTargetGates.ts. Irreducible. Covered by tests/unit/combo/combo-skipped-targets-summary.test.ts. Structural shrink of both god-files stays tracked in #3501.", - "_rebaseline_2026_09_22_11725_cpa_auth_index": "PR for #11725 own growth: open-sse/handlers/chatCore.ts 6400->6402 (+2). Prettier printWidth 100 keeps the failure-usage cpaAuthIndex property and the readCpaAuthIndex import on their own lines; the streaming and non-streaming call sites stay on the existing endpoint line. The parser, stamp, and label join live in open-sse/handlers/chatCore/cpaTraceAuthIndex.ts (under cap). Covered by tests/unit/cpa-trace-auth-index.test.ts, tests/unit/cpa-auth-index-usage.test.ts, and tests/unit/db/migration-185-cpa-auth-index.test.ts." + "_rebaseline_2026_09_22_14405_freetier_observed_tools_retry": "Issue #14405 own growth: open-sse/executors/opencode.ts 1251->1303 (measured after merging release/v3.8.51, whose own drift entry _rebaseline_2026_09_22_opencode_train10c_drift had already moved the ceiling 1247->1251; the extra 5 lines over the original 1247->1294 measurement are the #14148 reconciliation: freeTierRetryCtx now reads the contract attempt back from the request body via attemptFor() instead of the removed shared _contractAttempt field) (+47 irreducible call-site wiring: freeTierRetryCtx builder + direct fast-path retry call + rotation-loop refusal arm delegating to handleLoopFreeTierRefusal + api-typecheck fixes (unknown-cast on stall-guard dispatch, callback-shaped loop handler keeping private finalize methods); the retry logic itself lives in the new module open-sse/executors/opencodeFreeTierRetry.ts (131 lines, under cap) and the merge helper in opencodeFreeTierContract.ts, so no logic was added to the frozen file beyond wiring two existing dispatch arms. Compaction attempts measured and reverted: loop-call wrapper (+22 net), fast-path/loop fusion (fragile: retry-403 vs original-403 share status). Covered by tests/unit/opencode-free-tier-refusal-rotation.test.ts (3 retry tests: union success, original refusal + store untouched, no retry without names) + tests/unit/opencode-free-tier-request-contract.test.ts (2 merge tests)." }