Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -1801,6 +1801,8 @@ CURSOR_USER_AGENT="Cursor/3.4"
# TLS_CLIENT_TIMEOUT_MS=600000 # Inherits from FETCH_TIMEOUT_MS by default
# TLS_FIRST_BYTE_WATCHDOG_MS=10000 # #12656: bounds time-to-first-byte on the wreq body (0 disables)
# OPENCODE_RESPONSES_STALL_ROTATION=false # #13484 feature flag (Settings → Feature Flags wins): rotate once when a streamed Responses reply stalls before its first byte
# OPENCODE_PARK_AND_RESUME=false # #13924 feature flag (Settings → Feature Flags wins): park the request with a heartbeat after repeated transient 429s, then replay one capped leg of up to 3 accounts
#OPENCODE_POOL_STRAIN_MARKER_PATH=/tmp/opencode-pool-strain.json # #13924: pool-strain marker path (JSON {since, reason, ttl_s}); fresh marker parks without recounting
# RESPONSES_FIRST_BYTE_TIMEOUT_MS=15000 # #13484: OpenCode Responses first-byte window, only used when the OPENCODE_RESPONSES_STALL_ROTATION flag is on (0 disables)

# ── API Bridge (/v1 proxy server) ──
Expand Down
1 change: 1 addition & 0 deletions changelog.d/fixes/13924-park-and-resume-429-burst.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **fix(sse):** opt-in `OPENCODE_PARK_AND_RESUME` flag (default off): after repeated transient 429s the opencode rotation parks the request with a heartbeat and replays one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet; with the flag off every 429 rotates as before ([#13924](https://github.com/diegosouzapw/OmniRoute/pull/13924)) — thanks @maxmad64bis
4 changes: 3 additions & 1 deletion config/quality/file-size-baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"_rebaseline_2026_09_03_12648_xkiro_provider": "PR #12648 (feat/provider-xkiro) own growth: src/shared/constants/providers/apikey/gateways.ts +18 lines on top of #12649 (the xkiro APIKEY_PROVIDERS_GATEWAYS catalog entry with hasFree/freeNote/authHint/apiHint documenting the 5M tokens/day free plan, plus the Prettier reflow of two pre-existing >100-col authHint lines (oneminai, freebuff) that lint-staged enforces on any touch of the file; additive data at the existing registry chokepoint, same god-file no-split rationale as prior gateways.ts rebaselines: #11786 seekai, #10987 logfare, #10531 freebuff). Covered by tests/unit/free-provider-xkiro.test.ts (4/4).",
"_rebaseline_2026_09_15_12643_messages_entry_guard": "#12643 own growth: src/sse/handlers/chat.ts +10 (2490->2500 after syncing the 09-16 base, which itself moved the frozen value). A `messages` array containing a null/non-object entry (e.g. `[null]`) passed every existing entry guard (#5110/#6402/#6407/#6412) and crashed downstream translators/session helpers (openai-to-claude.ts, sessionManager.ts, contextManager.ts's fixToolPairs) reading `.role`/`.content` off the raw entry, surfacing as an HTTP 500 instead of a clean 400. Adds one more entry-shape guard clause to the same chokepoint, extending the existing guard family — same pattern, irreducible call-site wiring (the check itself is a one-line `.some()` predicate, not extractable into its own leaf without hiding the chokepoint). Covered by tests/unit/chat-messages-entry-objects-12643.test.ts (3/3) plus the sibling guard suites (chat-messages-validation-6402.test.ts, chat-non-string-model-6407.test.ts, 22/22, no regression). ATUALIZADO 2026-09-17: o teto foi refixado em 2519 ao mergear o tip atual. O tip sozinho ja esta em 2509 (acima do teto 2500 que esta PR havia fixado contra um tip anterior); o +10 desta PR e o proprio guard de entrada. O excedente do tip (2509>2500) e base-red herdado, nao introduzido aqui.",
"_rebaseline_2026_09_17_13185_claude_oauth_sticky_refresh": "PR #13185 (@RaviTharuma): soft-fail do refresh do Claude para CredentialHealth nao ficar sticky-dead. src/lib/tokenHealthCheck.ts 1214 (tip) -> 1220 na branch e 1221 na arvore combinada com #13426; teto fixado em 1221. O teto anterior (1218) tinha apenas 4 linhas de folga. O crescimento e o proprio fix: preservar o refresh_token e distinguir falha transitoria de credencial morta exige estado extra no caminho de sweep, que nao pode sair do modulo sem quebrar a API interna. Coberto por tests/unit/tokenHealthCheck-claude-refresh-token-preserved.test.ts; os 14 arquivos irmaos de tokenHealthCheck/credentialHealth foram rodados juntos (72/72).",
"_rebaseline_2026_09_19_13924_park_resume_growth": "PR #13924 park-and-resume transient 429 burst (rebased on release/v3.8.51 @7a921299c5): open-sse/executors/opencode.ts 1192->1233 (+41 irreducible chokepoint: imports + parkSleep + burstStreak/parked locals + RAZ + 429-branch park/replay block; leaf module opencodeParkResume.ts 257 lines holds all park/replay logic) + open-sse/executors/opencodeParkResume.ts new 257 (under 1200 cap). Covered by tests/unit/opencode-429-park-resume.test.ts (7 cases).",
"_rebaseline_2026_09_16_jxnlexn_wave_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chat.ts->2498; open-sse/handlers/chatCore.ts->6181. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
"_rebaseline_2026_09_16_wave22_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): src/sse/handlers/chatHelpers.ts->1231; open-sse/executors/cursor.ts->1808. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
"_rebaseline_2026_09_15_13572_combined_growth": "Combined growth of the 2026-09-15 maxmad64bis uplift batch (each PR rebaselined its own growth; the merged sum is larger): open-sse/executors/base.ts->1754. Every hunk is flag-gated or a verified fix covered by that PR's tests; see the batch report.",
Expand Down Expand Up @@ -523,7 +524,8 @@
"src/shared/middleware/chatBodyAdmission.ts": 1206,
"open-sse/executors/deepseek-web.ts": 1224,
"open-sse/executors/default.ts": 1205,
"open-sse/services/rateLimitManager.ts": 1329
"open-sse/services/rateLimitManager.ts": 1329,
"open-sse/executors/opencode.ts": 1233
},
"_rebaseline_2026_09_15_roundrobin_dashboard_events": "Fix #13089 (Combo Studio Live dashboard shows an empty backlog for round-robin combos): open-sse/services/combo/roundRobinCombo.ts 1205->1213. Round-robin is the only combo strategy that bypasses handleComboChat/executeTargetAttempt.ts, the path that publishes the combo.target.attempt/succeeded/failed EventBus events the Live dashboard listens for — so round-robin completions never showed up. The new call-site wiring (createRRDashboardEvents(...) instantiated once per target, one-line .attempt()/.succeeded()/.failed() calls at the 6 existing dispatch/outcome points) is the emitter logic actually extracted into a new module, open-sse/services/combo/rrDashboardEvents.ts — this is the minimum irreducible footprint for wiring 6 required call sites into 6 fixed control-flow points of the frozen file. Covered by tests/unit/issue-13089-roundrobin-live-ws-events.test.ts (2 tests: success + failure paths).",
"_rebaseline_base_2026_08_10_proxyfetch": "Base-red fix (green-prs sweep, issue #9985): open-sse/utils/proxyFetch.ts 1207 > cap 1000 — new proxied-TLS fetch helper introduced by the Fal reference-image work. Owner-authorized quick rebaseline to green; structural slim tracked for v3.9.0.",
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/ENVIRONMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -795,6 +795,8 @@ REQUEST_TIMEOUT_MS (global override)
| `TLS_CLIENT_TIMEOUT_MS` | = `FETCH_TIMEOUT_MS` | TLS fingerprint proxy (wreq-js) timeout. |
| `TLS_FIRST_BYTE_WATCHDOG_MS` | `10000` | Bounds time-to-first-byte on the wreq-js TLS-fingerprint transport's body specifically; `TLS_CLIENT_TIMEOUT_MS` alone cannot catch a stalled body since it resolves as soon as headers arrive (#12656). A timeout cancels the wreq reader and falls back to the direct/proxy dispatcher; `0` disables the watchdog. |
| `RESPONSES_FIRST_BYTE_TIMEOUT_MS` | `15000` | OpenCode executor only, and only while the `OPENCODE_RESPONSES_STALL_ROTATION` feature flag is on (default off): bounds the wait for the first body byte of a streamed Responses reply after its headers (#13484). A Responses stream opens with `response.created`, so silence past this window is a stall: the account is cooled down and the request rotates to the next account once; a second stall fails fast. `0` disables the guard even with the flag on. |
| `OPENCODE_PARK_AND_RESUME` | `false` | OpenCode executor only: park the request with a heartbeat after repeated transient 429s (or a fresh pool-strain marker), then replay one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet (#13924). Off by default: every 429 rotates to the next account exactly as before. |
| `OPENCODE_POOL_STRAIN_MARKER_PATH` | _(unset)_ | OpenCode executor only: override path of the pool-strain marker read before parking (`{since, reason, ttl_s}`, default `/tmp/opencode-pool-strain.json`, #13924). A fresh marker parks without recounting; absent or stale falls back to the burst counter. |
| `API_BRIDGE_PROXY_TIMEOUT_MS` | `30000` | Proxy hop timeout for `/v1` bridge requests. |
| `FIRECRAWL_BASE_URL` | `https://api.firecrawl.dev` | Point the Firecrawl web-fetch executor at a self-hosted instance (API key optional off-cloud). |
| `FIRECRAWL_TIMEOUT_MS` | `30000` | Per-request timeout for the Firecrawl web-fetch executor. |
Expand Down
7 changes: 4 additions & 3 deletions docs/reference/FEATURE_FLAGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,

## Flag Catalog

74 flags across 6 categories. **Default** is the definition default — the value
75 flags across 6 categories. **Default** is the definition default — the value
used when neither a DB override nor an environment variable is present.

### Security (10)
Expand All @@ -64,7 +64,7 @@ used when neither a DB override nor an environment variable is present.
| `AUTH_LOG_INCLUDE_ACCOUNT_ID` | boolean | `false` | Include account prefix in AUTH log lines (e.g. "Using <provider> account: abc12345..."). Disabled by default so account identifiers are redacted from shared/multi-tenant process logs. Independent from Debug Mode; flipping Debug Mode does not reveal this. |
| `OMNIROUTE_OIDC_DISABLE_PASSWORD_LOGIN` | boolean | `false` | When OIDC is enabled, disable password login so users can only authenticate via OIDC Single Sign-On. When disabled (default), both password login and OIDC are available. |

### Network (16)
### Network (17)

| Key | Type | Default | Restart | Description |
| ----------------------------------------------- | ------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand All @@ -79,6 +79,7 @@ used when neither a DB override nor an environment variable is present.
| `OPENCODE_RESPONSES_STALL_ROTATION` | boolean | `false` | | For the OpenCode executor, watch the first body byte of a streamed Responses reply (window: `RESPONSES_FIRST_BYTE_TIMEOUT_MS`, default `15000`). A 2xx Responses stream that stays silent past the window is treated as stalled: the account is cooled down and the request rotates to the next account once; a second stall fails fast. Off by default: stalled streams keep today's wait until the stream readiness timeout. |
| `OPENCODE_USER_BLOCKED_ROTATION` | boolean | `false` | | OpenCode executor: on a 403/451 carrying a `user_blocked` refusal (not geo, not a Cloudflare fingerprint rejection), cool the refused account down and rotate to the next account at most once per request; a second refusal is returned as-is, without a success mark. Off by default: routing around an upstream user block can look like evasion and spread the flag across the fleet. |
| `OPENCODE_TRANSIENT_FAILOVER_BACKOFF` | boolean | `false` | | OpenCode rotation: after two consecutive transient upstream failures (5xx or an empty 400), pause before the next account — 1.5s doubling per further failure, capped at 6s per pause and 10s per request, skipped on client disconnect; the failed body is released before waiting. Off by default: failover stays immediate. |
| `OPENCODE_PARK_AND_RESUME` | boolean | `false` | | OpenCode rotation: park the request after repeated transient 429s (or a fresh pool-strain marker) with a heartbeat, then replay one capped leg of up to 3 sequential accounts instead of fanning out the whole fleet. Off by default: every 429 rotates to the next account exactly as before. |
| `OPENCODE_RATE_LIMITED_429_EARLY_STOP` | boolean | `false` | | OpenCode rotation: stop the account wave at the first 429 classified as a real rate limit (parseable `Retry-After`, or a body naming a rate/usage limit) and return that upstream 429 unchanged. Unclassified 429s keep rotating. Off by default: the free tier is limited per egress IP (#9611), so every 429 rotates and an exhausted wave returns the last upstream 429. |
| `MITM_DISABLE_TLS_VERIFY` | boolean | `false` | ✓ | Disable TLS certificate verification for the MITM proxy. **Danger.** |
| `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` | boolean | `false` | | Allow provider URLs pointing to private/internal networks. |
Expand Down Expand Up @@ -214,7 +215,7 @@ Returns every flag with its effective value, source, and a summary.
"requiresRestart": false,
"warningLevel": "caution",
},
// ... all 74 flags
// ... all 75 flags
],
"summary": {
"total": 56,
Expand Down
43 changes: 42 additions & 1 deletion open-sse/executors/opencode.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import {
isEmptyUpstreamRejection,
extractChatcmplId,
} from "./accountRotation.ts";
import { markCooldown, markOutcome, noteResponseServed } from "./opencodeAccountHealth.ts";
import { markCooldown, markOutcome, markSuccess, noteResponseServed } from "./opencodeAccountHealth.ts";
import {
isOpencodeFreeTierRefusal,
isOpencodeGeoBlocked,
Expand Down Expand Up @@ -68,8 +68,15 @@ import {
isOpencodeUserBlockedRotationEnabled,
isOpencodeTransientFailoverBackoffEnabled,
isOpencodeRateLimited429EarlyStopEnabled,
isOpencodeParkAndResumeEnabled,
} from "@/shared/utils/featureFlags";
import { classifyUpstream429 } from "./opencodeRateLimited.ts";
import {
BURST_PARK_THRESHOLD,
parkWaitMs,
readPoolStrainMarker,
runParkAndReplay,
} from "./opencodeParkResume.ts";

/**
* The main OpenCode Zen host, shared by the `opencode` and `opencode-zen`
Expand Down Expand Up @@ -297,6 +304,7 @@ export class OpencodeExecutor extends BaseExecutor {
// tests swap in a recording fake instead of waiting on real timers.
transientPauseSleep: (ms: number, signal?: AbortSignal | null) => Promise<boolean> =
sleepAbortable;
parkSleep: (ms: number, signal?: AbortSignal | null) => Promise<boolean> = sleepAbortable;

constructor(provider: string) {
super(provider, PROVIDERS[provider] || PROVIDERS.openai);
Expand Down Expand Up @@ -617,6 +625,8 @@ export class OpencodeExecutor extends BaseExecutor {
// them this request — only acted on when OPENCODE_TRANSIENT_FAILOVER_BACKOFF is on.
let transientStreak = 0;
let transientPausedMs = 0;
let burstStreak = 0,
parked = false;

for (let attempt = 0; attempt < this.accounts.length + emptyRejectionBudget; attempt++) {
const isProxiedCandidate = (a: OpencodeAccountState): boolean => {
Expand Down Expand Up @@ -762,6 +772,7 @@ export class OpencodeExecutor extends BaseExecutor {
lastResult = result;
const priorTransientStreak = transientStreak;
transientStreak = 0;
if (result.response.status !== 429) burstStreak = 0;

const status = result.response.status;
if (status === 429) {
Expand Down Expand Up @@ -790,6 +801,36 @@ export class OpencodeExecutor extends BaseExecutor {
(setAsideMs ? `, member set aside for ${Math.round(setAsideMs / 1000)}s` : "") +
", rotating to next…"
);
burstStreak += 1;
if (!parked && isOpencodeParkAndResumeEnabled()) {
const marker = await readPoolStrainMarker();
if (burstStreak >= BURST_PARK_THRESHOLD || marker.fresh) {
parked = true;
log?.warn?.(
"OPENCODE",
`${cid}burstStreak=${burstStreak} freshD2=${marker.fresh} park`
);
const p = await runParkAndReplay(
{
execute: (i: ExecuteInput) =>
super.execute(i) as Promise<ExecutorExecuteResult & { response: Response }>,
markSuccess: (a: OpencodeAccountState) => markSuccess(a),
sleep: this.parkSleep,
accounts: this.accounts,
},
input,
parkWaitMs(marker.fresh ? marker.ttlLeftMs : null),
result,
log,
cid
);
if (p && p !== result) return this.normalizeMuseSparkResponse(input, p);
if (p) {
discardResponseBody(abandonedResponse);
return this.normalizeMuseSparkResponse(input, result);
}
}
}
continue;
}

Expand Down
Loading
Loading