feat(proxies): stop re-serving a proxy that just failed - #13578
Merged
diegosouzapw merged 93 commits intoSep 15, 2026
Merged
diegosouzapw merged 93 commits into
diegosouzapw merged 93 commits into
Conversation
maxmad64bis
force-pushed
the
feat/proxy-skip-recently-failed
branch
2 times, most recently
from
September 13, 2026 18:39
d35089f to
05a26ea
Compare
M0Abdullah
approved these changes
Sep 13, 2026
This was referenced Sep 13, 2026
OmniRoute picks a proxy in two places, a registry pool (diegosouzapw#6365) and opencode's per-account rotation, and neither remembers a failure past the request that saw it. The TCP reachability probe already fails a request on a dead proxy, then the next pick can land on it again; a member that stays reachable yet keeps refusing requests is served again on the next pick, and every request pays the same failure before moving on. A small per-process memory now sets such a member aside, keyed by its entry point (scheme, username, host, port; never the password): a refused probe for 1 minute, a refusal received through it for 2 minutes, doubling on each repeat up to 10 and 60 minutes. A response through the member clears it. Pools skip set-aside members under every rotation strategy (the round-robin cursor moves past the member actually served, sticky writes nothing more), the per-connection cache stops re-serving such a member, and the opencode rotation skips set-aside proxied accounts. Accounts without a proxy are never concerned. No proxy status is written and nothing is persisted. When every candidate is set aside, selection is exactly what it was, so the diegosouzapw#6246 fail-closed contract holds. No request is retried more than before and no traffic is added: the pool just stops re-serving a route it already knows is down. PROXY_SKIP_RECENTLY_FAILED=false (or 0, no, off) restores plain selection.
maxmad64bis
force-pushed
the
feat/proxy-skip-recently-failed
branch
from
September 13, 2026 20:24
05a26ea to
9543e9c
Compare
4 of 5 tasks
…iegosouzapw#13204) (diegosouzapw#13233) Merged after reconciling against the tip. `docs/architecture/QUALITY_GATES.md` conflicted: this PR adds the `check:vitest-exclusions` row while the tip had meanwhile rewritten the `check:model-lifecycle` description (diegosouzapw#12535). Resolved to the tip's table plus the new row. - `check:vitest-exclusions`: OK, 11 excluded files, each tracked and referenced - ESLint over the changed files: exit 0 - `npm run test:vitest` (the config this PR edits): 472 of 473. The one red, `tests/unit/autoCombo/provider-family-combos.test.ts`, is not touched by this PR and passes 11/11 in isolation on both this branch and the pure tip — a load flake from the full run, not a regression.⚠️ base-red inherited: diegosouzapw#12732
…ouzapw#12849) (diegosouzapw#13248) Merged after renumbering. `176_provider_connection_synced_models_at.sql` collided with `176_xp_action_counts.sql` (diegosouzapw#12651), which made the migration runner abort on every DB open. Renamed to **177**; the doc count moves 173 → 174 across README.md, AGENTS.md, llm.txt and the i18n mirrors (operator-approved, 206 numeric substitutions and nothing else). - `check:migration-numbering`: OK, 174 migrations, no duplicates - `check:docs-counts` migrations: ✓ - 84/84 across the NVIDIA suite plus the seven DB-touching suites the collision had taken down - ESLint and `typecheck:core`: exit 0 Heads-up for whoever lands next: **177 is claimed by eight other open PRs** (diegosouzapw#13610, diegosouzapw#13602, diegosouzapw#13580, diegosouzapw#13554, diegosouzapw#13405, diegosouzapw#13331, diegosouzapw#13177, diegosouzapw#13116) and 176 by diegosouzapw#13373 and diegosouzapw#13102. With this merged, all of them need to renumber at merge time — `check:migration-numbering` forbids new gaps, so the next free number is always the only valid one.⚠️ base-red inherited: diegosouzapw#12732
…k, docs provider count, agent-skills sync (diegosouzapw#13216) Merged after reconciling against the tip. The one conflict was `docs/reference/PROVIDER_REFERENCE.md`, a generated file — regenerated with `npm run gen:provider-reference` (358 unique IDs) rather than hand-merged. The three base-reds this PR targets, re-checked on the reconciled tree: - `check:docs-counts`: **exit 0** — provider count 356 → 358 in AGENTS.md, llm.txt, the i18n mirrors and the SVG diagrams was its last red - `check:open-sse-typecheck`: OK, 0 errors - `check:agent-skills-sync`: exit 0 (`skills/cli-tunnel/SKILL.md` → `tunnel create [type]`) - `autoCombo.test.ts` under the mcp vitest config: 63/63 - ESLint over the changed files: exit 0 AGENTS.md, llm.txt and SKILL.md are agent-instruction surfaces; the operator approved them for this PR explicitly.
…, Sinhala, Burmese, Khmer (59 locales) (diegosouzapw#13660) Batch 2 of the locale expansion across the dashboard catalog, docs mirrors, CLI catalog, README, locale index and the site. 51 → 59 locales. Also: the translator restores the ICU literal escape around angle placeholders, and the docs chunker splits oversized sections before translating.⚠️ base-red inherited: diegosouzapw#12732 — the eight red checks fail identically on unrelated PRs cut from the same base (e.g. diegosouzapw#13197); every gate is green locally after merging the base.
…xt mirrors (diegosouzapw#13674) Unblocks the pre-commit docs-sync gate on the release tip.
…zapw#12867 pipeline extraction, a legacy-DB boot abort and the catalog readers (diegosouzapw#13349) Merged after a real reconciliation — this PR and diegosouzapw#13069 fixed the same diegosouzapw#12867 regression (the non-streaming leg losing failure classification and credential refresh) with different strategies, and diegosouzapw#13069 landed first. Rather than stacking two implementations, the tip's was kept and this PR was trimmed to what the tip still lacked. **Dropped, already on the tip:** - non-streaming credential refresh and failure-state persistence → diegosouzapw#13069 (`applyProviderFailureClassification`); the `persistProviderFailureState` hook this PR added would have been dead code - body-derived rate-limit lock and non-JSON body message → already in the tip's pipeline (`chat-rate-limit-body-lock` passes there) - codex image-generation stringify of a sanitized error body → diegosouzapw#12945 (`stringifyImageErrorForLog`, which would otherwise be declared twice — TS2393) - the two test realignments (`hard-session-lease` inventory wording, kiro stream reader) → diegosouzapw#12945 **Kept, missing on the tip:** - **upstream error `code`/`type` in the pipeline error outcome** — ported onto the tip's implementation. Running this PR's own test against the tip returned `errorCode: undefined` instead of `missing_project_id`, so a config-class Antigravity 422 still degraded into an account cooldown. - legacy `call_logs` boot abort (index created after column healing) - malformed operator custom-models row no longer kills every `auto/*` pool (`virtualFactory.ts` 1219 → 1230, annotated) - realigned guards **Evidence on the reconciled tree** - 104 of 106 focused assertions. The 2 red (`models-catalog-route`, `provider-node-reserved-prefix`) fail identically on the pure tip. Against the tip, this PR turns 7 previously red cases green. - ESLint, `typecheck:core`, `check:open-sse-typecheck` (0 errors), `check:changelog-integrity`: all clean - changelog fragment rewritten to claim only what this PR still delivers Unblocking this also surfaced a repo-wide red: the eight llm.txt mirrors added by diegosouzapw#13660 kept the pre-diegosouzapw#13216/diegosouzapw#13248 counts, which failed the pre-commit `docs-sync` gate for everyone. Fixed separately in diegosouzapw#13674. `skills/cli-tunnel/SKILL.md` needed no change — diegosouzapw#13216 already landed the identical edit.⚠️ base-red inherited: diegosouzapw#12732
… test flake, pack provenance + pack policy (diegosouzapw#12959) Merged after salvaging what still applies. The doc-count half of this PR is superseded — it moved migrations 169 → 171 while the tip is already at 174, and the provider count landed via diegosouzapw#13216 — so every docs, diagram and llm.txt mirror change was reset to the tip's version and its changelog fragment dropped. Merging it as it stood would have regressed the counts. Kept, none of it on the tip: - `scripts/build/pack-artifact-policy.ts`: `@omniroute/opencode-plugin-v2/` added to the allowlist — diegosouzapw#12870 shipped the v2 plugin without widening it, so every packed file under it read as unexpected - `scripts/quality/validate-release-green.mjs`: points the pack provenance guard at `HEAD` instead of `origin/main`, which is structurally unreachable from a release branch mid-cycle - `12058-models-catalog-canonical-self-aliased.test.ts`: pins `CATALOG_BUILD_TIMEOUT_MS` out of the way of a cold catalog build on a loaded runner; assertions unchanged 54/54 across those three suites, ESLint exit 0, changelog integrity OK.⚠️ base-red inherited: diegosouzapw#12732
…w#12969) Merged after reconciling the whole stack onto the tip, operator-reviewed before merge. The core ownership fix for GHSA-wvxc-jp3v-5mg5 had already landed via diegosouzapw#13211 with a different implementation of the same endpoint. This branch carries a stricter one, built up in layers: this PR, diegosouzapw#13262 (explicit scope, audit log, atomic sweep), diegosouzapw#13297 (revoked, deactivated, banned or expired keys rejected) and diegosouzapw#13374 (owner-scoped file half, chunked instance sweep — SEC-C/SEC-D). The stack's implementation was kept over diegosouzapw#13211's because it is stricter on every point: - **route:** with diegosouzapw#13211, a request carrying BOTH a dashboard session cookie and an API key swept the whole instance. Here a presented key always scopes the sweep to that key; only a session without a key sweeps all tenants; neither returns 401. Instance-wide and row-deleting sweeps log at warn as an audit trail, and a failing sweep returns a sanitized 500. - **`deleteCompletedBatches`:** takes an explicit `{ apiKeyId } | { allTenants: true }`. An omitted or blank key throws instead of widening, and passing both throws. - **files:** a key sweep only soft-deletes files the caller owns (`deleteFileOwnedBy`), so a batch referencing another tenant's or an unowned file never nulls its content. - **transactions:** key mode is all-or-nothing across chunks; instance mode commits per 200-id chunk so a large sweep never holds one write lock on the table. diegosouzapw#13211's own test was aligned to the explicit-scope API with its assertions unchanged. Its seed now creates the file with the batch's owner, as an upload through that key does in production — without that, SEC-C correctly leaves the unowned file intact. - 51/51 across the six batch suites (diegosouzapw#13211's test, this stack's ownership and route-scope tests, `batch-deletion`, `batch-deletion-route-logic`, `files-delete-owned-by`) - ESLint, `typecheck:core`, complexity, cognitive-complexity, changelog integrity: clean⚠️ base-red inherited: diegosouzapw#12732
…apw#13286) (diegosouzapw#13418) Chains `npm run test:unit:serial` onto the plain `npm test` script, so the quarantined serial suite is no longer skipped when contributors run `npm test` (diegosouzapw#13286). The existing `test-serial-quarantine` guard now covers `test` too. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…uzapw#13273) (diegosouzapw#13401) Masks every `*-api-key` header spelling (`x-goog-api-key`, `api-key`, `xi-api-key`) in the request-log pipeline by matching on the hyphen-compacted key, and adds `xi-api-key` to the payload redaction set (diegosouzapw#13273). The new test drives the real `createRequestLogger` / `protectPayloadForLog` paths. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…iegosouzapw#13403) Fixes the stale `jina-ai/` prefix in the second Jina assertion of `models-catalog-route.test.ts`. The catalog emits `jina/…` (the first Jina test already expected it), so this clears a base red on the release branch. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…iegosouzapw#13308) (diegosouzapw#13404) Prunes `db_backups/` after the health-check-repair `VACUUM INTO` snapshot (diegosouzapw#13308). That path never ran retention, so every restart of a healthy database added another full-size copy. It now uses the same `DB_BACKUP_MAX_FILES` / `DB_BACKUP_RETENTION_DAYS` limits as `backup.ts`, importing `backupRetention` directly to avoid the import cycle. Maintainer additions: merged the current release branch and rebaselined `src/lib/db/core.ts` 1745→1767 in `file-size-baseline.json` with a dated annotation (legitimate growth). Chosen over diegosouzapw#13540, which did the same with a fire-and-forget dynamic import. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…ction (diegosouzapw#12776) (diegosouzapw#13406) Carries `context_length` through the MCP `list_models_catalog` projection when the upstream model record has it, and omits it otherwise (diegosouzapw#12776). Covered by the new case in `mcp-model-catalog.test.ts`. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…#13409) "Test all models" no longer sends image/music/video generation-only models through a chat completion, which triggered real billable generations (diegosouzapw#13376). `detectTestKind` flags them via `isNonChatGeneration` and `runSingleModelTest` skips them; chat+image models, embeddings and models without metadata are unaffected (8 new cases plus updated `model-test-runner` shapes). Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…iegosouzapw#13412) Adds CJK quota-exhaustion messages (GLM/z.ai 5-hour window, Kimi, Qwen/DashScope, MiniMax, plus Japanese and Korean phrasings) to the 429 quota classifier. They are now `quota_exhausted` (cooldown + failover) instead of a transient `rate_limit` retry loop (diegosouzapw#13194). The patterns go into the recoverable `QUOTA_PATTERNS` only, not the terminal set, and a guard case keeps the Chinese transient "请求过于频繁" as `rate_limit`. Chosen over diegosouzapw#13536. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…otAll regex (diegosouzapw#13413) Eval runner: a case whose upstream call failed can no longer score as passed when the grading regex happens to match the error text (diegosouzapw#13137), and regex grading compiles with dotAll so `.` spans newlines in multi-line answers, for both string and `RegExp` patterns (diegosouzapw#13138). Chosen over diegosouzapw#13542 / diegosouzapw#13530, which each covered half. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…on (diegosouzapw#13414) The eval runner sends `x-omniroute-compression: off` and `x-omniroute-no-memory: true`, so cases measure the model rather than injected output styles or retrieved memory (diegosouzapw#13139). Both headers are the existing per-request opt-outs honored by `chatCore` (`open-sse/handlers/chatCore/headers.ts`). Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
…ouzapw#13439) Behind the new `PROTECTED_PRIORITY_INFRA_502_ENABLED` flag (default off), protected-priority combo stops caused by provably non-quota infrastructure (provider circuit open, predictive-TTFT latency) surface as 502 instead of a quota-looking 503. Maintainer rework before merge (kept the idea, no default behavior change): - The original branch made 502 the default for every stop, including model lockouts and cooldowns, and removed the diegosouzapw#8133/diegosouzapw#1731 provider-wide skip for 401/5xx without a connection id; both are restored with their regression tests untouched. - Nineteen cases cover eight gate causes plus predictive latency, flag off and on. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
…rs plus fire-and-forget async (diegosouzapw#13614) Stale provider-pin (`clearStaleLKGP`) clears are no longer silent: the fire-and-forget promise carries a `.catch` that warns with combo, comboId and executionKey, and a `check:routing-error-guard` npm script keeps the inventory of swallowed catches in the routing hot path from growing. Maintainer rework before merge (kept the idea, no default behavior change): - The awaited DB writes in the fallback loop were reverted (they added latency and SQLite lock exposure on every skip); the clear stays non-blocking. - The guard keys its allowlist by file + normalized catch body instead of line numbers (the PR's version broke on any edit) and is wired as an npm script only, not in CI; the unused stats counters were dropped. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
…d off-by-default flag (diegosouzapw#13633) Behind `STREAM_RECOVERY_TOOLCALL_ORDER_FIX` (default off), mid-stream continuation becomes tool-call safe: any tool call seen in the stream — in flight or finished — blocks a continuation, and an empty continuation stops after one attempt. Maintainer rework before merge (kept the idea, no default behavior change): - The empty-continuation short-circuit also ran with the flag off; it is now gated, so the flag-off path uses the whole budget exactly as before (regression test added). - The latch re-arm that let a continuation fire after a completed `finish_reason: tool_calls` is gone; index-less tool calls on multi-choice payloads are now blocked too; ~150 lines of dead trace plumbing removed. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
…iegosouzapw#13650) Recovery traces for mid-stream continuation: one `onContinueOutcome` hook reports suffix stitched, overlap rejected, terminal, empty, no-stream and refused (with reason), logged through `chatCore` at debug level; warn is reserved for the cases where recovery gives up. Maintainer rework before merge (kept the idea, no default behavior change): - The original logged a warn-level latch line on every streamed tool call; nominal and tool-call streams are now silent, and the existing `mid-stream continuation attempt N/4` line keeps its format. - `chatCore.ts` ends 3 lines shorter than the tip, so the baseline bump the PR carried was removed; the wiring is tested through a real `handleChatCore` continuation. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
…only on partial estimates (diegosouzapw#13686) Estimated token usage is now visible to operators: usage a provider marks as `estimated` carries an internal marker through extraction and the call log records `_omniroute.usageEstimated: true` on the logged response. Maintainer rework before merge (kept the idea, no default behavior change): - Billing is unchanged: the original skipped cost/budget/quota-share for estimated usage, which would have let streams without upstream usage and eight web executors spend $0 against API-key budgets; that part is reverted and no opt-in flag was added because it cannot be made budget-safe. - Both open-sse TS2345 errors, the client-visible `estimated_prompt_tokens` field and the `as unknown as` casts are gone; four real `handleChatCore` cases assert the marker and unchanged spend. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
…ouzapw#13153) Behind `STREAM_EARLY_EOF_SIBLING_FAILOVER_ENABLED` (default off): after the bounded same-connection retry is spent, a stream that closed early fails over exactly once to a sibling connection. Maintainer rework before merge (kept the idea, no default behavior change): - The PR's own failover test was red on its head: the `/v1/chat/completions` route's early-stream keepalive dropped the `X-OmniRoute-Selected-Connection-Id` header on the first cold request. Tests now drive `handleChat()` directly; the assertion was kept. - "One hop" was one hop per connection (a 3-connection pool made 4 dispatches); it is now a single sibling hop per request, and when the pool runs out the original `STREAM_EARLY_EOF` 502 is returned instead of a generic `bad_gateway`, so combo-level detection keeps working. The source-regex timeout test became a behavioral one; flag description in all 59 locales. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
… obsidian always-protected, DATA_DIR vault refusal (diegosouzapw#13791) GHSA-7pq4-8pvv-rx7r (critical). Every link of the reported chain held on the release tip: 1. First boot without JWT_SECRET generates one and writes it in cleartext to $DATA_DIR/server.env. 2. With no password configured, isAuthRequired() returned false for POST /api/settings/require-login unconditionally — before the loopback check — so any network peer could switch requireLogin off. 3. With requireLogin off, POST /api/settings/obsidian/webdav accepted an arbitrary vault root and echoed freshly minted Basic credentials. 4. The WebDAV file service is served by the custom Node layer before Next.js, outside the authz pipeline. 5. Pointing it at DATA_DIR reads server.env, and JWT_SECRET forges an `{"authenticated":true}` admin session. A second, worse problem surfaced while verifying: isLoopbackRequest() decided "loopback" from nextUrl.hostname / the Host header, which the client controls. `Host: localhost` from a remote address made the whole fresh-install bootstrap reachable, not just the write path. Three cuts, plus the root cause: - isLoopbackRequest() now reads the trusted peer: the token-stamped real TCP peer the custom server writes (peerStamp), then the pipeline's own locality verdict once a stamp token exists, then a real socket peer. The bootstrap write path honours the same constraint instead of returning false, and managementPolicy hands down the peerContext verdict explicitly, because at policy time the original request still carries client-supplied headers. - Host is consulted only when the process has no stamp token at all — no stamping server in front, which in practice means route handlers invoked directly by the unit-test harness. Every supported runtime (run-next dev and start, standalone-server-ws for Docker, the npm CLI and Electron) calls ensurePeerStampToken() at boot, so there a signal-less request fails closed. Without this fallback ~340 route tests that call handlers with `new Request("http://localhost/…")` turned into 401s. - /api/settings/obsidian joins ALWAYS_PROTECTED_API_PATHS: issuing and rotating reusable WebDAV credentials is credential export, the same rationale as the GHSA-62vw entry for the password reveal. - enableObsidianVaultSync() refuses a vault that is, sits inside, or contains DATA_DIR, comparing realpath-resolved paths so a symlink cannot dodge it. Tests are red-first: remote stamped peer → auth required on the bootstrap write; Host: localhost plus a forged locality header from a non-loopback stamped peer → 401 through the full pipeline; the local operator keeps the first-password flow; obsidian inventory and DATA_DIR overlap cases.
# Conflicts: # docs/reference/FEATURE_FLAGS.md # src/i18n/messages/ar.json # src/i18n/messages/az.json # src/i18n/messages/bg.json # src/i18n/messages/bn.json # src/i18n/messages/cs.json # src/i18n/messages/da.json # src/i18n/messages/de.json # src/i18n/messages/el.json # src/i18n/messages/en.json # src/i18n/messages/es.json # src/i18n/messages/et.json # src/i18n/messages/fa.json # src/i18n/messages/fi.json # src/i18n/messages/fr.json # src/i18n/messages/ga.json # src/i18n/messages/gu.json # src/i18n/messages/he.json # src/i18n/messages/hi.json # src/i18n/messages/hr.json # src/i18n/messages/hu.json # src/i18n/messages/id.json # src/i18n/messages/it.json # src/i18n/messages/ja.json # src/i18n/messages/km.json # src/i18n/messages/kn.json # src/i18n/messages/ko.json # src/i18n/messages/lt.json # src/i18n/messages/lv.json # src/i18n/messages/ml.json # src/i18n/messages/mr.json # src/i18n/messages/ms.json # src/i18n/messages/mt.json # src/i18n/messages/my.json # src/i18n/messages/ne.json # src/i18n/messages/nl.json # src/i18n/messages/no.json # src/i18n/messages/or.json # src/i18n/messages/pa.json # src/i18n/messages/phi.json # src/i18n/messages/pl.json # src/i18n/messages/pt-BR.json # src/i18n/messages/pt.json # src/i18n/messages/ro.json # src/i18n/messages/ru.json # src/i18n/messages/si.json # src/i18n/messages/sk.json # src/i18n/messages/sl.json # src/i18n/messages/sr.json # src/i18n/messages/sv.json # src/i18n/messages/sw.json # src/i18n/messages/ta.json # src/i18n/messages/te.json # src/i18n/messages/th.json # src/i18n/messages/tr.json # src/i18n/messages/uk-UA.json # src/i18n/messages/ur.json # src/i18n/messages/vi.json # src/i18n/messages/zh-CN.json # src/i18n/messages/zh-TW.json # tests/unit/feature-flags-settings.test.ts # tests/unit/server-owned-tool-loop-flag.test.ts
…, stale suppressions)
diegosouzapw
merged commit Sep 15, 2026
29d66cb
into
diegosouzapw:release/v3.8.51
0 of 3 checks passed
diegosouzapw
added a commit
to maxmad64bis/OmniRoute
that referenced
this pull request
Sep 15, 2026
…OXY_SKIP_RECENTLY_FAILED Follow-up to the reworked diegosouzapw#13578/diegosouzapw#13580 base: - noteProxyOutcome ran inside the fire-and-forget safeLogEvents, after two awaited dynamic imports, so the set-aside landed late and requests picking from the pool meanwhile still got the refused member. It now runs first in safeLogEvents, ahead of any await, i.e. synchronously at the chat.ts call site as soon as the upstream status is known (no chat.ts change needed). - The outcome is gated by the opt-in PROXY_SKIP_RECENTLY_FAILED flag (default off = pool selection unchanged). The flag is read only to write a refusal; a success only clears what an earlier opt-in wrote. Edge relays are left out through their null egress key, so the dispatcher import goes away. - Drop the duplicate 177_proxy_logs_upstream_status.sql this stack carried next to the renumbered 179 migration. - Tests: flag on/off for the outcome memory and the chat log path, plus a check that the member is already set aside when the log call returns.
diegosouzapw
pushed a commit
that referenced
this pull request
Sep 15, 2026
#13602) Behind `PROXY_SKIP_RECENTLY_FAILED` (from #13578): a provider 429 received through a pool member sets that member aside and a 2xx clears it, for opencode providers. Maintainer rework before merge (kept the idea, no default behavior change): - `noteProxyOutcome` ran inside the fire-and-forget `safeLogEvents` after awaited dynamic imports, so a concurrent request could still pick the member; it now runs first, synchronously, before any `await`. - The duplicate `177_proxy_logs_upstream_status.sql` the stack still carried alongside the renamed 179 was removed; the regression test the PR body named exists as `pool-ip-quota-429-path.test.ts`. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
5 tasks done
diegosouzapw
added a commit
that referenced
this pull request
Sep 18, 2026
…13575) (#14044) resolveProxyForConnection cached a scope pool's first resolution result for the life of the per-connection cache, so a chat-path request never saw the pool's round-robin/sticky/random strategy advance again — only the narrow #13578 set-aside escape hatch could break the freeze. resolveProxyForScopeFromRegistry (used directly by every existing rotation test) always re-ran the strategy and rotated correctly. The cache now treats a registry-sourced pool result as due for re-resolution on every call (falling through to the same cascade the direct registry callers use), except for the two populations that need a stable egress across requests: EGRESS_BUCKETED_LOCK_PROVIDERS (opencode's quota is bucketed by egress IP) and grok-web (its cf_clearance cookie is pinned to the IP/UA/TLS fingerprint that earned it). Regression test: tests/unit/proxy-pool-chat-path-rotation-13575.test.ts, RED before the fix (resolveProxyForConnection returned the same host 6/6 times for a 3-member pool), GREEN after. Updated tests/unit/proxy-pool-skips-refused-member.test.ts's three assertions that encoded the frozen-cache contract to the corrected always-rotates-except-pinned contract; all other cases in that file and in tests/unit/proxy-pool-rotation-6365.test.ts pass unchanged.
This was referenced Sep 19, 2026
Merged
This was referenced Sep 23, 2026
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…#13578) Behind the new `PROXY_SKIP_RECENTLY_FAILED` flag (default off), pool rotation and the opencode account rotation remember a proxy that just failed (refused probe or 429) and skip it for a doubling cooldown instead of re-serving it immediately. Maintainer rework before merge (kept the idea, no default behavior change): - The original was on by default and re-queried the DB on every request while a member was set aside; selection now caches a refusal sequence number and re-runs the cascade once per set-aside event. - `src/lib/db` no longer imports the heavy dispatcher for key normalization (a parity test guarantees the same key as `proxyConfigToUrl()`); `.env.example` and `ENVIRONMENT.md` document the default as false. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 d61b804: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
diegosouzapw#13602) Behind `PROXY_SKIP_RECENTLY_FAILED` (from diegosouzapw#13578): a provider 429 received through a pool member sets that member aside and a 2xx clears it, for opencode providers. Maintainer rework before merge (kept the idea, no default behavior change): - `noteProxyOutcome` ran inside the fire-and-forget `safeLogEvents` after awaited dynamic imports, so a concurrent request could still pick the member; it now runs first, synchronously, before any `await`. - The duplicate `177_proxy_logs_upstream_status.sql` the stack still carried alongside the renamed 179 was removed; the regression test the PR body named exists as `pool-ip-quota-429-path.test.ts`. Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 d61b804: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests. Thanks @maxmad64bis!
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…iegosouzapw#13575) (diegosouzapw#14044) resolveProxyForConnection cached a scope pool's first resolution result for the life of the per-connection cache, so a chat-path request never saw the pool's round-robin/sticky/random strategy advance again — only the narrow diegosouzapw#13578 set-aside escape hatch could break the freeze. resolveProxyForScopeFromRegistry (used directly by every existing rotation test) always re-ran the strategy and rotated correctly. The cache now treats a registry-sourced pool result as due for re-resolution on every call (falling through to the same cascade the direct registry callers use), except for the two populations that need a stable egress across requests: EGRESS_BUCKETED_LOCK_PROVIDERS (opencode's quota is bucketed by egress IP) and grok-web (its cf_clearance cookie is pinned to the IP/UA/TLS fingerprint that earned it). Regression test: tests/unit/proxy-pool-chat-path-rotation-13575.test.ts, RED before the fix (resolveProxyForConnection returned the same host 6/6 times for a 3-member pool), GREEN after. Updated tests/unit/proxy-pool-skips-refused-member.test.ts's three assertions that encoded the frozen-cache contract to the corrected always-rotates-except-pinned contract; all other cases in that file and in tests/unit/proxy-pool-rotation-6365.test.ts pass unchanged.
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Set
PROXY_SKIP_RECENTLY_FAILED=false(or0,no,off) to keep today's selection.OmniRoute picks a proxy in two places — a registry pool and opencode's per-account rotation — and neither remembers a failure past the request that saw it. A dead proxy keeps getting picked on the next request, and a member that stays reachable yet keeps refusing requests is served again on the next pick, each request paying the same failure before moving on. This PR adds a small per-process memory that sets such a member aside for a period that doubles on each repeat (1 min, up to 10 min, for a refused TCP probe; 2 min, up to 60 min, for a refusal received through it), and clears it as soon as a response comes back through it. No request is retried more than before and no traffic is added — the pool just stops re-serving a route it already knows is down. When every candidate is set aside, selection is exactly what it was.
Related Issues
Validation
npm run lint— ESLint on the touched files is clean; the full run is red on the baseTests Added Or Updated
tests/unit/proxy-refusal-memory.test.ts(new, 12 tests: canonical key, backoff caps, purge, LRU, switch)tests/unit/proxy-health-refusal-memory.test.ts(new, 4 tests: refused probe sets aside, recovery, switch off, malformed URL)tests/unit/opencode-proxy-refusal-memory.test.ts(new, 6 tests: received refusal sets aside, switch off, expiry, all-set-aside fallback, direct accounts untouched, no-accounts path)tests/unit/proxy-pool-skips-refused-member.test.ts(new, 9 tests: all four rotation strategies, cursor advance, sticky without write, pool-member cache guard, legacy level)tests/unit/opencode-transient-rotation.test.ts,tests/unit/opencode-geo-block-rotation.test.ts(reset the process-wide memory inbeforeEach)exit=0with zeronot okcheck-file-size,check-api-typecheck,typecheck:core,any-budget,tracked-artifacts,check-docs-sync,prettier --check,eslinton touched files,check-complexity-ratchetsall clean;check-env-doc-syncred only on six inheritedOMNIROUTE_*entries;check-open-sse-typecheckred only on the inheritedautoCombo.test.tsTS2739Coverage Notes
open-sse/utils/proxyRefusalMemory.tsis covered by its 12 unit tests (key normalization, doubling periods, caps, per-kind purge, eviction, switch values)src/lib/proxyHealth.tschange is covered by the 4 probe tests plus the existingproxy-health-6246andproxy-health-ipv6suitesopen-sse/executors/opencode.tschange is covered by the 6 rotation tests plus the existing transient/geo-block/empty-rejection/proxy-rotation/account-rotation suitessrc/lib/db/proxies/rotation.tsandsrc/lib/db/settings.tschanges are covered by the 9 pool tests plus the existingproxy-pool-rotation-6365andproxy-pool-route-6365suitesReviewer Notes
randomdraws uniformly among eligible members, so one set-aside member slightly raises the draw rate of the restPROXY_SKIP_RECENTLY_FAILEDdefaults to on; set it tofalse(or0,no,off) to restore plain selectionchangelog.d/features/13578-proxy-skip-recently-failed.mduses the real number.env.exampleanddocs/reference/ENVIRONMENT.md, at a different spot; no textual conflict expected.