Skip to content

feat(proxy-logs): keep the HTTP status the provider actually returned - #13580

Merged
diegosouzapw merged 92 commits into
diegosouzapw:release/v3.8.51from
maxmad64bis:feat/proxy-log-upstream-status
Sep 15, 2026
Merged

diegosouzapw merged 92 commits into
diegosouzapw:release/v3.8.51from
maxmad64bis:feat/proxy-log-upstream-status

Conversation

@maxmad64bis

@maxmad64bis maxmad64bis commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #12732

Summary

The proxy log collapsed every outcome into success/timeout/error, so a throttled egress IP (429), a refused one (403) and a provider outage (500) read as the same line. result.status can't fill the gap: several 429s are generated locally (rate limiter, quota sharing, backpressure) before any request leaves.

Each proxy_logs row now keeps upstream_status, the status of the last response the patched fetch received while a provider request was dispatched, or null when no response arrived. The textual status column is untouched, so existing filters return the same rows.

Related Issues

Validation

  • Change type: DB
  • Focused tests and category gates from the golden path
  • npm run lint — ESLint clean on the touched files; the full run is red on the base
  • Reconciled with the current active release base; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR

Tests Added Or Updated

  • tests/unit/upstream-status-capture.test.ts (8 tests): wrapper alone — dispatched call recorded, response untouched; outside dispatch ignored; throw clears; late response after dispatch end ignored; args forwarded; no sink harmless; isProviderRequestCaptureActive true only until runWithCapture settles; second dispatch without fetch leaves nothing stale.
  • tests/unit/proxyfetch-upstream-status-capture.test.ts (9 tests): real patched fetch over local HTTP — 429 captured; last response wins; background fetch still running at dispatch return keeps 429; side fetch keeps provider status; network error on next dispatch clears; explicit-dispatcher path; direct-context path; no sink harmless; second dispatch without fetch leaves nothing stale.
  • tests/unit/proxy-logs-upstream-status.test.ts (4 tests): column after migration + reconciler restore; memory/SQLite/export; absent → null never 0; textual filters unchanged.
  • tests/unit/merge-applied-proxy-sink.test.ts (8 tests): withUpstreamStatus/mergeAppliedProxySink; safeLogEvents forwards; local 429 without fetch → null.
  • Regression: 224 files, 1716 pass; only inherited failures (chatcore-hierarchical-admission, error-public-boundaries-hardening, proxyfetch-direct-response-start-timeout-10214 in batch, green alone).

Coverage Notes

  • open-sse/utils/upstreamStatusCapture.ts (new): covered by the 8 wrapper tests, including the second-dispatch invalidation and the settled-dispatch exclusion.
  • open-sse/utils/providerRequestLogging.ts (isProviderRequestCaptureActive, dispatch entry hook): covered by the capture tests plus transitively by every proxyfetch test running through the real runWithCapture.
  • open-sse/utils/proxyFetch.ts (wrapper install only, +4): covered by the 9 real-fetch tests (sentinel, explicit dispatcher) plus the untouched proxyfetch-* suites, green.
  • src/lib/db/migrations/177_* / migrationRunner.ts / schemaColumns.ts / src/lib/proxyLogger.ts: covered by the 4 journal tests (migration, reconciler, memory/SQLite/export, null, filters).
  • src/sse/handlers/chatHelpers.ts (withUpstreamStatus, mergeAppliedProxySink, safeLogEvents) / chat.ts (call site, 0 net): covered by the 8 transport tests, including the local-429-no-fetch case.
  • Untested by choice: relay/TLS/native-fallback/Bun return paths (single wrapper covers them by construction, grep-verified one install).

Reviewer Notes

  • Contract: upstream_status is the status received; it can differ from the status served to the client after applyStatusRestatement (403 received, 429 served).
  • Request capture wraps status capture with no dependency between them.
  • A chat POST via the TLS layer whose first byte expires gives null (never replayable).
  • A fetch left running in the background by an executor after its return no longer writes (dispatch token); a side fetch awaited before the return counts.
  • Relay rows carry the relay's code, identifiable by proxy type.
  • Cloud mode: null.
  • proxyFetch.ts rebaselined +4 (single install-site plumbing: one import, one blank, sink getter, wrapper const) with note; 50 llm.txt copies regenerated via sync-llm-mirrors.
  • Migration count 173 → 174.
  • Expected merge conflict with feat(proxies): show how many egress IPs actually served a proxy pool #13581, in config/quality/file-size-baseline.json only: both PRs insert their _rebaseline_… note right after the opening {. Whichever lands second needs both keys and both caps kept (open-sse/utils/proxyFetch.ts 1275 here, ProxyRegistryManager.tsx 1477 there). I can update the branch if you'd rather not resolve it on merge.
  • Follow-up: feat(proxies): stop re-serving a pool member the provider just refused #13595 (stacked on this PR and feat(proxies): stop re-serving a proxy that just failed #13578) uses the recorded upstream status to set aside a pool member that got a real provider 429, while a 429 generated locally never is.

The proxy log status collapses every outcome into success/timeout/error, so a
429 (the provider limits this egress IP), a 403 (it refuses it) and a 500 (an
outage) are the same line. result.status cannot fill the gap: several 429s are
generated locally (rate limiter, quota sharing, backpressure) before any
request leaves.

The status is now captured where it is known for sure: the patched global fetch
is wrapped once, and while a provider request is being dispatched (inside
runWithCapture) the received status is written to the applied-proxy sink added
in diegosouzapw#5217. A new dispatch invalidates the earlier status at entry, so a retry
that never reaches the network (local refusal, start timeout) leaves no stale
code behind; a throw during dispatch clears it, so a network error after a 429
on another proxy leaves no stale code either. Side fetches of the same request
(usage sync, dashboard events) and background calls an executor leaves running
after it returns (settled dispatch token) do not overwrite it. The status then
travels with the applied proxy into safeLogEvents and lands in a nullable
proxy_logs.upstream_status column: null means no response was received.

The textual status column is untouched, so existing filters return the same
rows. proxyFetch.ts grows by four lines at its single install site; its cap is
raised with a note. chat.ts does not grow.
diegosouzapw and others added 22 commits September 14, 2026 13:40
…iegosouzapw#13204) (diegosouzapw#13233)

Merged after reconciling against the tip. `docs/architecture/QUALITY_GATES.md` conflicted: this PR adds the `check:vitest-exclusions` row while the tip had meanwhile rewritten the `check:model-lifecycle` description (diegosouzapw#12535). Resolved to the tip's table plus the new row.

- `check:vitest-exclusions`: OK, 11 excluded files, each tracked and referenced
- ESLint over the changed files: exit 0
- `npm run test:vitest` (the config this PR edits): 472 of 473. The one red, `tests/unit/autoCombo/provider-family-combos.test.ts`, is not touched by this PR and passes 11/11 in isolation on both this branch and the pure tip — a load flake from the full run, not a regression.

⚠️ base-red inherited: diegosouzapw#12732
…ouzapw#12849) (diegosouzapw#13248)

Merged after renumbering. `176_provider_connection_synced_models_at.sql` collided with `176_xp_action_counts.sql` (diegosouzapw#12651), which made the migration runner abort on every DB open. Renamed to **177**; the doc count moves 173 → 174 across README.md, AGENTS.md, llm.txt and the i18n mirrors (operator-approved, 206 numeric substitutions and nothing else).

- `check:migration-numbering`: OK, 174 migrations, no duplicates
- `check:docs-counts` migrations: ✓
- 84/84 across the NVIDIA suite plus the seven DB-touching suites the collision had taken down
- ESLint and `typecheck:core`: exit 0

Heads-up for whoever lands next: **177 is claimed by eight other open PRs** (diegosouzapw#13610, diegosouzapw#13602, diegosouzapw#13580, diegosouzapw#13554, diegosouzapw#13405, diegosouzapw#13331, diegosouzapw#13177, diegosouzapw#13116) and 176 by diegosouzapw#13373 and diegosouzapw#13102. With this merged, all of them need to renumber at merge time — `check:migration-numbering` forbids new gaps, so the next free number is always the only valid one.

⚠️ base-red inherited: diegosouzapw#12732
…k, docs provider count, agent-skills sync (diegosouzapw#13216)

Merged after reconciling against the tip. The one conflict was `docs/reference/PROVIDER_REFERENCE.md`, a generated file — regenerated with `npm run gen:provider-reference` (358 unique IDs) rather than hand-merged.

The three base-reds this PR targets, re-checked on the reconciled tree:

- `check:docs-counts`: **exit 0** — provider count 356 → 358 in AGENTS.md, llm.txt, the i18n mirrors and the SVG diagrams was its last red
- `check:open-sse-typecheck`: OK, 0 errors
- `check:agent-skills-sync`: exit 0 (`skills/cli-tunnel/SKILL.md` → `tunnel create [type]`)
- `autoCombo.test.ts` under the mcp vitest config: 63/63
- ESLint over the changed files: exit 0

AGENTS.md, llm.txt and SKILL.md are agent-instruction surfaces; the operator approved them for this PR explicitly.
…, Sinhala, Burmese, Khmer (59 locales) (diegosouzapw#13660)

Batch 2 of the locale expansion across the dashboard catalog, docs mirrors, CLI catalog, README, locale index and the site. 51 → 59 locales.

Also: the translator restores the ICU literal escape around angle placeholders, and the docs chunker splits oversized sections before translating.

⚠️ base-red inherited: diegosouzapw#12732 — the eight red checks fail identically on unrelated PRs cut from the same base (e.g. diegosouzapw#13197); every gate is green locally after merging the base.
…xt mirrors (diegosouzapw#13674)

Unblocks the pre-commit docs-sync gate on the release tip.
…zapw#12867 pipeline extraction, a legacy-DB boot abort and the catalog readers (diegosouzapw#13349)

Merged after a real reconciliation — this PR and diegosouzapw#13069 fixed the same diegosouzapw#12867 regression (the non-streaming leg losing failure classification and credential refresh) with different strategies, and diegosouzapw#13069 landed first. Rather than stacking two implementations, the tip's was kept and this PR was trimmed to what the tip still lacked.

**Dropped, already on the tip:**
- non-streaming credential refresh and failure-state persistence → diegosouzapw#13069 (`applyProviderFailureClassification`); the `persistProviderFailureState` hook this PR added would have been dead code
- body-derived rate-limit lock and non-JSON body message → already in the tip's pipeline (`chat-rate-limit-body-lock` passes there)
- codex image-generation stringify of a sanitized error body → diegosouzapw#12945 (`stringifyImageErrorForLog`, which would otherwise be declared twice — TS2393)
- the two test realignments (`hard-session-lease` inventory wording, kiro stream reader) → diegosouzapw#12945

**Kept, missing on the tip:**
- **upstream error `code`/`type` in the pipeline error outcome** — ported onto the tip's implementation. Running this PR's own test against the tip returned `errorCode: undefined` instead of `missing_project_id`, so a config-class Antigravity 422 still degraded into an account cooldown.
- legacy `call_logs` boot abort (index created after column healing)
- malformed operator custom-models row no longer kills every `auto/*` pool (`virtualFactory.ts` 1219 → 1230, annotated)
- realigned guards

**Evidence on the reconciled tree**
- 104 of 106 focused assertions. The 2 red (`models-catalog-route`, `provider-node-reserved-prefix`) fail identically on the pure tip. Against the tip, this PR turns 7 previously red cases green.
- ESLint, `typecheck:core`, `check:open-sse-typecheck` (0 errors), `check:changelog-integrity`: all clean
- changelog fragment rewritten to claim only what this PR still delivers

Unblocking this also surfaced a repo-wide red: the eight llm.txt mirrors added by diegosouzapw#13660 kept the pre-diegosouzapw#13216/diegosouzapw#13248 counts, which failed the pre-commit `docs-sync` gate for everyone. Fixed separately in diegosouzapw#13674.

`skills/cli-tunnel/SKILL.md` needed no change — diegosouzapw#13216 already landed the identical edit.

⚠️ base-red inherited: diegosouzapw#12732
… test flake, pack provenance + pack policy (diegosouzapw#12959)

Merged after salvaging what still applies. The doc-count half of this PR is superseded — it moved migrations 169 → 171 while the tip is already at 174, and the provider count landed via diegosouzapw#13216 — so every docs, diagram and llm.txt mirror change was reset to the tip's version and its changelog fragment dropped. Merging it as it stood would have regressed the counts.

Kept, none of it on the tip:

- `scripts/build/pack-artifact-policy.ts`: `@omniroute/opencode-plugin-v2/` added to the allowlist — diegosouzapw#12870 shipped the v2 plugin without widening it, so every packed file under it read as unexpected
- `scripts/quality/validate-release-green.mjs`: points the pack provenance guard at `HEAD` instead of `origin/main`, which is structurally unreachable from a release branch mid-cycle
- `12058-models-catalog-canonical-self-aliased.test.ts`: pins `CATALOG_BUILD_TIMEOUT_MS` out of the way of a cold catalog build on a loaded runner; assertions unchanged

54/54 across those three suites, ESLint exit 0, changelog integrity OK.

⚠️ base-red inherited: diegosouzapw#12732
…w#12969)

Merged after reconciling the whole stack onto the tip, operator-reviewed before merge.

The core ownership fix for GHSA-wvxc-jp3v-5mg5 had already landed via diegosouzapw#13211 with a different implementation of the same endpoint. This branch carries a stricter one, built up in layers: this PR, diegosouzapw#13262 (explicit scope, audit log, atomic sweep), diegosouzapw#13297 (revoked, deactivated, banned or expired keys rejected) and diegosouzapw#13374 (owner-scoped file half, chunked instance sweep — SEC-C/SEC-D). The stack's implementation was kept over diegosouzapw#13211's because it is stricter on every point:

- **route:** with diegosouzapw#13211, a request carrying BOTH a dashboard session cookie and an API key swept the whole instance. Here a presented key always scopes the sweep to that key; only a session without a key sweeps all tenants; neither returns 401. Instance-wide and row-deleting sweeps log at warn as an audit trail, and a failing sweep returns a sanitized 500.
- **`deleteCompletedBatches`:** takes an explicit `{ apiKeyId } | { allTenants: true }`. An omitted or blank key throws instead of widening, and passing both throws.
- **files:** a key sweep only soft-deletes files the caller owns (`deleteFileOwnedBy`), so a batch referencing another tenant's or an unowned file never nulls its content.
- **transactions:** key mode is all-or-nothing across chunks; instance mode commits per 200-id chunk so a large sweep never holds one write lock on the table.

diegosouzapw#13211's own test was aligned to the explicit-scope API with its assertions unchanged. Its seed now creates the file with the batch's owner, as an upload through that key does in production — without that, SEC-C correctly leaves the unowned file intact.

- 51/51 across the six batch suites (diegosouzapw#13211's test, this stack's ownership and route-scope tests, `batch-deletion`, `batch-deletion-route-logic`, `files-delete-owned-by`)
- ESLint, `typecheck:core`, complexity, cognitive-complexity, changelog integrity: clean

⚠️ base-red inherited: diegosouzapw#12732
…apw#13286) (diegosouzapw#13418)

Chains `npm run test:unit:serial` onto the plain `npm test` script, so the quarantined serial suite is no longer skipped when contributors run `npm test` (diegosouzapw#13286). The existing `test-serial-quarantine` guard now covers `test` too.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…uzapw#13273) (diegosouzapw#13401)

Masks every `*-api-key` header spelling (`x-goog-api-key`, `api-key`, `xi-api-key`) in the request-log pipeline by matching on the hyphen-compacted key, and adds `xi-api-key` to the payload redaction set (diegosouzapw#13273). The new test drives the real `createRequestLogger` / `protectPayloadForLog` paths.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…iegosouzapw#13403)

Fixes the stale `jina-ai/` prefix in the second Jina assertion of `models-catalog-route.test.ts`. The catalog emits `jina/…` (the first Jina test already expected it), so this clears a base red on the release branch.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…iegosouzapw#13308) (diegosouzapw#13404)

Prunes `db_backups/` after the health-check-repair `VACUUM INTO` snapshot (diegosouzapw#13308). That path never ran retention, so every restart of a healthy database added another full-size copy. It now uses the same `DB_BACKUP_MAX_FILES` / `DB_BACKUP_RETENTION_DAYS` limits as `backup.ts`, importing `backupRetention` directly to avoid the import cycle.

Maintainer additions: merged the current release branch and rebaselined `src/lib/db/core.ts` 1745→1767 in `file-size-baseline.json` with a dated annotation (legitimate growth). Chosen over diegosouzapw#13540, which did the same with a fire-and-forget dynamic import.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…ction (diegosouzapw#12776) (diegosouzapw#13406)

Carries `context_length` through the MCP `list_models_catalog` projection when the upstream model record has it, and omits it otherwise (diegosouzapw#12776). Covered by the new case in `mcp-model-catalog.test.ts`.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…#13409)

"Test all models" no longer sends image/music/video generation-only models through a chat completion, which triggered real billable generations (diegosouzapw#13376). `detectTestKind` flags them via `isNonChatGeneration` and `runSingleModelTest` skips them; chat+image models, embeddings and models without metadata are unaffected (8 new cases plus updated `model-test-runner` shapes).

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…iegosouzapw#13412)

Adds CJK quota-exhaustion messages (GLM/z.ai 5-hour window, Kimi, Qwen/DashScope, MiniMax, plus Japanese and Korean phrasings) to the 429 quota classifier. They are now `quota_exhausted` (cooldown + failover) instead of a transient `rate_limit` retry loop (diegosouzapw#13194). The patterns go into the recoverable `QUOTA_PATTERNS` only, not the terminal set, and a guard case keeps the Chinese transient "请求过于频繁" as `rate_limit`. Chosen over diegosouzapw#13536.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…otAll regex (diegosouzapw#13413)

Eval runner: a case whose upstream call failed can no longer score as passed when the grading regex happens to match the error text (diegosouzapw#13137), and regex grading compiles with dotAll so `.` spans newlines in multi-line answers, for both string and `RegExp` patterns (diegosouzapw#13138). Chosen over diegosouzapw#13542 / diegosouzapw#13530, which each covered half.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…on (diegosouzapw#13414)

The eval runner sends `x-omniroute-compression: off` and `x-omniroute-no-memory: true`, so cases measure the model rather than injected output styles or retrieved memory (diegosouzapw#13139). Both headers are the existing per-request opt-outs honored by `chatCore` (`open-sse/handlers/chatCore/headers.ts`).

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…iegosouzapw#13411)

The media-providers web-search example card now sends `provider` in the request body. `POST /v1/search` selects the provider from `body.provider`, so the card was silently hitting the default provider (diegosouzapw#13245).

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…osouzapw#13410)

De-duplicates the `agy` and `antigravity` model catalogs into `antigravitySharedModels.ts`, with an explicit per-surface add/remove delta (diegosouzapw#12724). Verified behavior-neutral: every exported catalog constant of both modules serializes byte-identically before and after (201-line JSON dump).

Maintainer fix: `buildSurfaceCatalog` returned `readonly { id: string; [k: string]: unknown }[]`, which erased the element type. `name` became `unknown`, and the `RegistryEntry.models` assignments failed with 4× TS2322 under `check:open-sse-typecheck` (not covered by `typecheck:core`). It is now generic (`<T extends { id: string }>`), so the literal model shape flows through.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…ifting at messages[0] (diegosouzapw#13425) (diegosouzapw#13427)

Memory injection merges into an Anthropic-shaped top-level `system` field (string or text-block array) instead of prepending a `role: "system"` message at `messages[0]`, which Anthropic rejects with a 400 (diegosouzapw#13425). Handled on both the system-first path (xiaomi-mimo) and the general path. Chosen over diegosouzapw#13549, which covered only the string case.

Maintainer fix: widened `ChatRequest.system` to `string | Array<{ type; text?; … }>`. Assigning the block array to the `string`-typed field raised 2× TS2322 under `check:open-sse-typecheck`.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…3369) (diegosouzapw#13433)

The CLI readiness budget is configurable through `OMNIROUTE_READY_TIMEOUT_MS` or `omniroute serve --ready-timeout <ms>` (default unchanged at 60s). The timeout warning prints the budget it actually used and suggests a larger value, for slow cold starts such as Windows (diegosouzapw#13369). Documented in `ENVIRONMENT.md` and `TROUBLESHOOTING.md`, with 11 resolver cases.

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
…iegosouzapw#12779) (diegosouzapw#13340)

Adds the Microsoft 365 Copilot (BizChat) provider guide for the existing `copilot-m365-web` provider (alias `m365copilot`, `open-sse/config/providers/registry/copilot-m365-web`), including the WebSocket credential capture flow, and links it from the docs index and the web-cookie guide (diegosouzapw#12779).

Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green.

Thanks @KooshaPari!
diegosouzapw and others added 20 commits September 15, 2026 13:24
Dependabot diegosouzapw#214 (GHSA-vwc7-r8mq-g2x9 / CVE-2026-76845, moderate): adm-zip
0.5.9–0.6.0 follows a symlink that already exists inside the extraction root
and writes through it, outside the root. The advisory still reports
`first_patched_version: null`, but 0.6.1 (published after the advisory) is the
fix — `util/utils.js` gains `assertPathSafe`, which walks every path component
below the root with `lstat` and throws on a symlink; `extractAllTo` calls it
before every write. Verified by diffing the two tarballs.

Reach in this repo: adm-zip is pulled only by `onnxruntime-node` (an
optionalDependency, itself pinned by override) and used only in its install
script to unpack the vendor's own runtime binary. No request path touches it.

The override already existed at ^0.6.0 (PR diegosouzapw#7732, the previous adm-zip CVE);
this just raises the floor. Lockfile moves 0.6.0 → 0.6.1, nothing else.
…credential catalog (diegosouzapw#13744)

GHSA-r4q7-7f24-m29p. `CREDENTIAL_PATTERNS` (open-sse/utils/credentialPatterns.ts)
is the single catalog iterated in order by both the opt-in credential-masker
guardrail and the public error sanitizer. It had no entry for Groq (`gsk_`) or
xAI (`xai-`), and only knew the exact 48-char OpenAI `sk-` form.

Measured on the release tip before this change:

| Shape                        | public sanitizer | guardrail |
|------------------------------|------------------|-----------|
| Groq  gsk_ + 52              | LEAK             | LEAK      |
| xAI   xai- + 80              | LEAK             | LEAK      |
| DeepSeek sk- + 32 hex        | redacted         | LEAK      |
| sk- + 20/36/40/51 (not 48)   | redacted         | LEAK      |

The public path already caught every `sk-` shape through STRONG_CREDENTIAL_TOKEN,
so the advisory's "both layers" framing only holds for gsk_/xai-; for the sk-
family the exposure was the guardrail.

Adds `groq` and `xai` after `anthropic_alt`, and a generic `openai_compatible`
`sk-` fallback as the LAST entry. Ordering matters: both consumers replace as
they iterate, so `openai_proj`, `openai` and `anthropic*` stamp their specific
label first and the fallback only sees shapes nothing else claimed. The
lookbehind mirrors STRONG_CREDENTIAL_TOKEN so `risk-…`-style words do not match.
All three regexes are a fixed prefix plus one bounded character class — linear,
no nested quantifiers.

Tests are red-first: the new guardrail cases (bare / sentence / JSON-body
contexts per shape, plus label-ordering and negative cases) and the catalog
coverage array in error-sensitive-redaction both failed on the tip.

Follow-ups deliberately left out of scope: `tskey-auth-` (Tailscale) was never in
the catalog, and the guardrail does not decode `\uXXXX` escapes the way the
public path does.
diegosouzapw#13748)

GHSA-34rg-3pqj-35g9. `fetchRemoteImage()` defaults to
`getProviderOutboundGuard()` — the OPERATOR outbound policy, local-first by
design so self-hosted providers on loopback/LAN keep working. Since diegosouzapw#11062 added
the `block-metadata` middle tier, a default install resolves to that mode: the
string check only rejects 169.254/16 and the IMDS hostnames, and the DNS
validation step is skipped entirely (it only runs under `public-only`).

Three sinks feed that default with CALLER input, so a request body could make
the server fetch `http://127.0.0.1:…` or any RFC-1918 host and forward the bytes
upstream:

- imageGeneration.ts `resolveImageSource()` — `image_url`, `mask_url`, message parts
- imageUpscale/shared.ts `resolveUpscaleImageSource()` — 14 body aliases,
  `provider_options.*`, message parts (Stability, Topaz)
- visionBridgeHelpers.ts `fetchRemoteImageAsDataUri()` — chat `image_url` parts
  inlined into the vision self-call

plus the NanoBanana result-URL download, which is upstream-supplied rather than
OmniRoute-controlled.

Same trust confusion as GHSA-3f8g / GHSA-j7j4 on the search base URL: operator
config and caller input must not share a guard. Each site now passes
`guard: "public-only"` (string check + DNS validation of every answer), matching
the siblings that already did it right — embeddings, the audio bridge and the
AI Horde result download.

`pinDns` is set only on the vision bridge. The other three sites use
`globalThis.fetch`, and connection pinning would swap that for a raw undici
fetch — the same reason the AI Horde site leaves it off. On the vision bridge a
`fetchImpl` is injected, so `pinDns` there validates every DNS answer but cannot
pin the connection; commented in place.

Blind SSRF rather than full read: the bytes go upstream or into the vision
self-call, not back to the caller — but the status oracle and upstream
exfiltration are real.

Tests are red-first — per sink, `http://127.0.0.1:1/x.png` and
`http://192.168.1.50/x.png` are rejected with the injected fetch never called,
and a public host whose DNS resolves to a public IP still downloads.
… records and anonymous listing (diegosouzapw#13749)

GHSA-2jm2-mpx8-6523 and GHSA-m3hp-hq9g-fpmv, one root cause.

`getApiKeyRequestScope()` never rejects: with the default REQUIRE_API_KEY=false
the client-api policy admits both a missing and an invalid bearer as anonymous,
and the scope comes back `{ apiKeyId: null, isSessionAuth: false }`. The
`/v1/files` and `/v1/batches` routes then treated "null" as permissive in two
different ways:

- GHSA-m3hp — the list routes coerced `apiKeyId || undefined`, and the DB layer
  reads `undefined` as "no owner filter", so an anonymous or invalid-bearer
  caller got every tenant's file and batch metadata, the same unfiltered view as
  the operator's dashboard.
- GHSA-2jm2 — the single-record checks were `record.apiKeyId !== null && …`, so
  a record with no owner short-circuited to "allowed" for any caller: read,
  download raw content, delete, cancel, or use as a batch input. Null-owner
  records are common — every dashboard-session upload, and every batch output
  file inheriting a session batch's owner, which carries model responses.

`api_key_id` has existed since the table was created (migration 028), so a null
owner is not a legacy row; it is an unattributable write. No doc described it
as shared — API_REFERENCE says files are scoped per key — and batch_api.test.ts
pinned the by-id exposure as expected behaviour.

One rule now, in `_helpers/apiKeyScope.ts`:

- `canAccessOwnedRecord(scope, owner)`: a dashboard session is the instance
  operator and may act on any record; an API key acts on its own records only;
  a null owner is denied to every non-session caller. Applied to files
  GET / DELETE / content, batches GET / DELETE / cancel, and the batch-create
  input-file check.
- `resolveListScope(scope)`: an explicit union for list/count reads — scoped to
  the presented key (a key wins even alongside a session cookie), instance-wide
  only for a session without a key, and 401 otherwise, including for a bearer
  that does not resolve to a key. There is no default that widens a read.

This follows the GHSA-wvxc shape already used by the delete-completed sweep.

Behaviour change: the anonymous upload → batch → download flow no longer works
without an API key, because a null owner cannot be attributed.

Subsumes diegosouzapw#13683: it moved `scopeCheck` into the shared helper so a session can
cancel any batch — kept, and its test ported — but it also kept null-owner
records open on the premise they predate ownership tracking, which migration
028 contradicts.

Tests are red-first. batch_api's by-id case is flipped to 404 with a negative
assertion; batch-deletion-route-logic now imports the real helper instead of a
local copy that had silently diverged from production; the two integration
tests present a real key, since their subject is limits and rate logging, not
auth.

Co-authored-by: Markus Hartung <mail@hartmark.se>
…LOCAL_ONLY (diegosouzapw#13745)

GHSA-35fw-cv32-2373 and GHSA-jx89-f37j-pq89 — the same defect class as
/api/acp/agents (GHSA-hf57): a route whose handler chain spawns a host process
was classified Tier 3 MANAGEMENT only, and requireManagementAuth() waives auth
when requireLogin=false. Hard Rules diegosouzapw#15/diegosouzapw#17 require the LOCAL_ONLY gate, which
runs on the stamped real peer before any auth check.

cli-tools (GHSA-35fw): 14 routes reach
getCliRuntimeStatus() -> locateCommand() -> runProcess("sh", ["-c",
'command -v -- "$1"']) -> spawn(), exactly like their six gated siblings
(forge/grok-build/jcode/qwen/omp/letta-settings):
all-statuses, status, and the claude/cline/codewhale/codex/crush/deepseek-tui/
droid/kilo/openclaw/pi/smelt-settings routes. The advisory counted 13; it
missed /api/cli-tools/detect, which is heavier — detectAllTools() runs
execFile(binary, ["--version"]) and execFile("which") per tool.

skills (GHSA-jx89): POST /api/skills/install stores the request's handlerCode
verbatim as the skill handler with no allowlist, so a value equal to a built-in
name (execute_command / eval_code) aliases the real sandboxed built-in;
POST /api/skills/executions then runs it. The sandbox is a real container, but
the spawn is transitive, which is why the 6A.8 source scan never flagged it.

Entries are exact paths, not a /api/cli-tools/ blanket prefix: apply, backups,
config, guide-settings, hermes-agent-settings, keys, logs, openclaw/auto-order
and codex-profiles do not spawn and remote dashboards use them. All 16 are
mirrored into SPAWN_CAPABLE_PREFIXES (no manage-scope bypass) and added to the
route-guard-membership roots so the gate enforces them from now on.

Functional trade-off, same one already accepted for grok/forge/jcode/qwen: a
dashboard served through a tunnel no longer shows the CLI Tools status badges.

Tests are red-first. Two existing negative controls pointed at routes that turn
out to spawn (/api/cli-tools/all-statuses, /api/skills/install); they now point
at routes that genuinely do not (/api/cli-tools/config, /api/skills/marketplace,
/api/skills/skillssh/install), so the non-over-gating assertions are kept.
…ouzapw#13645)

The provider-page Free badge can require a free tier the provider actually honors: behind the new `FREE_BADGE_REQUIRES_PROVIDER_FREE_TIER` flag (default off) the display-name "free" heuristic, non-boolean `free` fields and `:free` suffixes on registered providers without a documented free tier no longer light the badge. With the flag off the historical rule is unchanged.

Maintainer rework before merge (kept the idea, no default behavior change):
- `:free` models on free-tier providers and on compatible nodes (OpenRouter-style endpoints) keep the badge in both modes — the original change dropped them.
- Test D derives its provider set from `FREE_MODEL_BUDGETS` instead of a hard-coded allowlist; a vitest render covers both sections with the flag endpoint on, off and erroring.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…apw#13646)

Both settings routes use the shared `isSocks5ProxyEnabled()` reader instead of a copied check (identical logic, no behavior change).

Maintainer rework before merge (kept the idea, no default behavior change):
- The source-grep tests were replaced by behavioral tests of both routes across the flag on/off matrix (`GET /api/settings/proxies` reports `socks5Enabled`; `PUT /api/settings/proxy` accepts socks5 or returns 400).

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…owns (diegosouzapw#13440)

Daily-quota lockouts on the non-TPD path honor the provider's configured daily-reset clock (`dailyQuotaResetTimezone`/hour) in combo routing instead of the host's midnight.

Maintainer rework before merge (kept the idea, no default behavior change):
- The process-lifetime clock cache is gone: the clock is resolved on each failure through the already-TTL'd `getCachedProviderNodes`, so a timezone change takes effect without a restart and a DB error is never cached as `{}` forever.
- Round-robin combos are threaded too (the PR left them out); an option on `recordModelLockoutFailure` that could never run was removed; tests prove both call sites pass the configured clock.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…ons (diegosouzapw#13671)

Fixes the DST-gap bug in `nextDailyResetAtMs`: a reset hour that does not exist on the transition day landed one hour early (New York 02:00 came out as 01:00; Havana/Santiago midnight as 23:00 the day before). The walk across the gap is bounded to one day and uses a cached formatter.

Maintainer rework before merge (kept the idea, no default behavior change):
- Dropped the 24h clamp in `getMsUntilTomorrow` (on a 25h fall-back day 24.5h is the correct wait; clamping expired the lock 30 minutes early) and the unreachable `ms <= 0` branch, with their tests; characterization tests pin ordinary and fall-back days.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…in path (diegosouzapw#13672)

Behind the new `RETRY_AFTER_PROVENANCE_ENABLED` flag (default off): `unavailableResponse` omits the synthetic `Retry-After: 1` when there is no real retry signal, marks `retry_after_provenance` on its bodies, and both combo drain readers parse prose retry hints from plain-text bodies too. With the flag off, headers and bodies are exactly as before.

Maintainer rework before merge (kept the idea, no default behavior change):
- A past `Retry-After` date is no longer labelled as an upstream signal with `Retry-After: 1`; non-JSON bodies (HTML 502 pages) log at debug instead of warning on every request.
- The provenance claim is narrowed to responses built by `unavailableResponse`, documented in the flag row.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…ouzapw#13439)

Behind the new `PROTECTED_PRIORITY_INFRA_502_ENABLED` flag (default off), protected-priority combo stops caused by provably non-quota infrastructure (provider circuit open, predictive-TTFT latency) surface as 502 instead of a quota-looking 503.

Maintainer rework before merge (kept the idea, no default behavior change):
- The original branch made 502 the default for every stop, including model lockouts and cooldowns, and removed the diegosouzapw#8133/diegosouzapw#1731 provider-wide skip for 401/5xx without a connection id; both are restored with their regression tests untouched.
- Nineteen cases cover eight gate causes plus predictive latency, flag off and on.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…rs plus fire-and-forget async (diegosouzapw#13614)

Stale provider-pin (`clearStaleLKGP`) clears are no longer silent: the fire-and-forget promise carries a `.catch` that warns with combo, comboId and executionKey, and a `check:routing-error-guard` npm script keeps the inventory of swallowed catches in the routing hot path from growing.

Maintainer rework before merge (kept the idea, no default behavior change):
- The awaited DB writes in the fallback loop were reverted (they added latency and SQLite lock exposure on every skip); the clear stays non-blocking.
- The guard keys its allowlist by file + normalized catch body instead of line numbers (the PR's version broke on any edit) and is wired as an npm script only, not in CI; the unused stats counters were dropped.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…d off-by-default flag (diegosouzapw#13633)

Behind `STREAM_RECOVERY_TOOLCALL_ORDER_FIX` (default off), mid-stream continuation becomes tool-call safe: any tool call seen in the stream — in flight or finished — blocks a continuation, and an empty continuation stops after one attempt.

Maintainer rework before merge (kept the idea, no default behavior change):
- The empty-continuation short-circuit also ran with the flag off; it is now gated, so the flag-off path uses the whole budget exactly as before (regression test added).
- The latch re-arm that let a continuation fire after a completed `finish_reason: tool_calls` is gone; index-less tool calls on multi-choice payloads are now blocked too; ~150 lines of dead trace plumbing removed.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…iegosouzapw#13650)

Recovery traces for mid-stream continuation: one `onContinueOutcome` hook reports suffix stitched, overlap rejected, terminal, empty, no-stream and refused (with reason), logged through `chatCore` at debug level; warn is reserved for the cases where recovery gives up.

Maintainer rework before merge (kept the idea, no default behavior change):
- The original logged a warn-level latch line on every streamed tool call; nominal and tool-call streams are now silent, and the existing `mid-stream continuation attempt N/4` line keeps its format.
- `chatCore.ts` ends 3 lines shorter than the tip, so the baseline bump the PR carried was removed; the wiring is tested through a real `handleChatCore` continuation.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…only on partial estimates (diegosouzapw#13686)

Estimated token usage is now visible to operators: usage a provider marks as `estimated` carries an internal marker through extraction and the call log records `_omniroute.usageEstimated: true` on the logged response.

Maintainer rework before merge (kept the idea, no default behavior change):
- Billing is unchanged: the original skipped cost/budget/quota-share for estimated usage, which would have let streams without upstream usage and eight web executors spend $0 against API-key budgets; that part is reverted and no opt-in flag was added because it cannot be made budget-safe.
- Both open-sse TS2345 errors, the client-visible `estimated_prompt_tokens` field and the `as unknown as` casts are gone; four real `handleChatCore` cases assert the marker and unchanged spend.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
…ouzapw#13153)

Behind `STREAM_EARLY_EOF_SIBLING_FAILOVER_ENABLED` (default off): after the bounded same-connection retry is spent, a stream that closed early fails over exactly once to a sibling connection.

Maintainer rework before merge (kept the idea, no default behavior change):
- The PR's own failover test was red on its head: the `/v1/chat/completions` route's early-stream keepalive dropped the `X-OmniRoute-Selected-Connection-Id` header on the first cold request. Tests now drive `handleChat()` directly; the assertion was kept.
- "One hop" was one hop per connection (a 3-connection pool made 4 dispatches); it is now a single sibling hop per request, and when the pool runs out the original `STREAM_EARLY_EOF` 502 is returned instead of a generic `bad_gateway`, so combo-level detection keeps working. The source-regex timeout test became a behavioral one; flag description in all 59 locales.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
… obsidian always-protected, DATA_DIR vault refusal (diegosouzapw#13791)

GHSA-7pq4-8pvv-rx7r (critical). Every link of the reported chain held on the
release tip:

1. First boot without JWT_SECRET generates one and writes it in cleartext to
   $DATA_DIR/server.env.
2. With no password configured, isAuthRequired() returned false for
   POST /api/settings/require-login unconditionally — before the loopback
   check — so any network peer could switch requireLogin off.
3. With requireLogin off, POST /api/settings/obsidian/webdav accepted an
   arbitrary vault root and echoed freshly minted Basic credentials.
4. The WebDAV file service is served by the custom Node layer before Next.js,
   outside the authz pipeline.
5. Pointing it at DATA_DIR reads server.env, and JWT_SECRET forges an
   `{"authenticated":true}` admin session.

A second, worse problem surfaced while verifying: isLoopbackRequest() decided
"loopback" from nextUrl.hostname / the Host header, which the client controls.
`Host: localhost` from a remote address made the whole fresh-install bootstrap
reachable, not just the write path.

Three cuts, plus the root cause:

- isLoopbackRequest() now reads the trusted peer: the token-stamped real TCP
  peer the custom server writes (peerStamp), then the pipeline's own locality
  verdict once a stamp token exists, then a real socket peer. The bootstrap
  write path honours the same constraint instead of returning false, and
  managementPolicy hands down the peerContext verdict explicitly, because at
  policy time the original request still carries client-supplied headers.
- Host is consulted only when the process has no stamp token at all — no
  stamping server in front, which in practice means route handlers invoked
  directly by the unit-test harness. Every supported runtime (run-next dev and
  start, standalone-server-ws for Docker, the npm CLI and Electron) calls
  ensurePeerStampToken() at boot, so there a signal-less request fails closed.
  Without this fallback ~340 route tests that call handlers with
  `new Request("http://localhost/…")` turned into 401s.
- /api/settings/obsidian joins ALWAYS_PROTECTED_API_PATHS: issuing and rotating
  reusable WebDAV credentials is credential export, the same rationale as the
  GHSA-62vw entry for the password reveal.
- enableObsidianVaultSync() refuses a vault that is, sits inside, or contains
  DATA_DIR, comparing realpath-resolved paths so a symlink cannot dodge it.

Tests are red-first: remote stamped peer → auth required on the bootstrap
write; Host: localhost plus a forged locality header from a non-loopback
stamped peer → 401 through the full pipeline; the local operator keeps the
first-password flow; obsidian inventory and DATA_DIR overlap cases.
…#13578)

Behind the new `PROXY_SKIP_RECENTLY_FAILED` flag (default off), pool rotation and the opencode account rotation remember a proxy that just failed (refused probe or 429) and skip it for a doubling cooldown instead of re-serving it immediately.

Maintainer rework before merge (kept the idea, no default behavior change):
- The original was on by default and re-queried the DB on every request while a member was set aside; selection now caches a refusal sequence number and re-runs the cascade once per set-aside event.
- `src/lib/db` no longer imports the heavy dispatcher for key normalization (a parity test guarantees the same key as `proxyConfigToUrl()`); `.env.example` and `ENVIRONMENT.md` document the default as false.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 c0f92ec: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
# Conflicts:
#	config/quality/file-size-baseline.json
@diegosouzapw
diegosouzapw merged commit 238cb1b into diegosouzapw:release/v3.8.51 Sep 15, 2026
3 of 7 checks passed
diegosouzapw added a commit to maxmad64bis/OmniRoute that referenced this pull request Sep 15, 2026
…OXY_SKIP_RECENTLY_FAILED

Follow-up to the reworked diegosouzapw#13578/diegosouzapw#13580 base:

- noteProxyOutcome ran inside the fire-and-forget safeLogEvents, after two
  awaited dynamic imports, so the set-aside landed late and requests picking
  from the pool meanwhile still got the refused member. It now runs first in
  safeLogEvents, ahead of any await, i.e. synchronously at the chat.ts call
  site as soon as the upstream status is known (no chat.ts change needed).
- The outcome is gated by the opt-in PROXY_SKIP_RECENTLY_FAILED flag (default
  off = pool selection unchanged). The flag is read only to write a refusal;
  a success only clears what an earlier opt-in wrote. Edge relays are left
  out through their null egress key, so the dispatcher import goes away.
- Drop the duplicate 177_proxy_logs_upstream_status.sql this stack carried
  next to the renumbered 179 migration.
- Tests: flag on/off for the outcome memory and the chat log path, plus a
  check that the member is already set aside when the log call returns.
@diegosouzapw diegosouzapw mentioned this pull request Sep 21, 2026
@maxmad64bis
maxmad64bis deleted the feat/proxy-log-upstream-status branch September 24, 2026 21:11
SCys pushed a commit to SCys/OmniRoute that referenced this pull request Sep 25, 2026
… the loopback exemption

The diegosouzapw#13580 guard "a fetch still running when the dispatch returns does not
change the status" started a background fetch to the refused port
127.0.0.1:1 and assumed it would still be in flight when the dispatch
settled. That only held because loopback targets used to go through the
direct undici bound-and-replay path: ECONNREFUSED on attempt 0, a
RETRY_BACKOFF_MS wait, attempt 1, then the native fallback.

diegosouzapw#14311 intentionally sends loopback targets straight to native fetch
(open-sse/utils/proxyFetch.ts, the isLoopbackTarget early return), so the
refused call now rejects within about a millisecond, while the dispatch is
still reading the provider body. withUpstreamStatusCapture then does what
it documents for a call that throws during an active dispatch: it clears
upstreamStatus, and the test read undefined instead of 429. A probe showed
":1 rejected dispatching=true" before "dispatch settled". The capture code
is unchanged and correct; the test's timing assumption broke.

The failing background call is now a delayed connection drop on the test
server (drop=1, delay=50), which still fails after the dispatch settles, as
the test intends. The 429 assertion is unchanged. A mutation that ignores
the settled dispatch flag still fails the test.

Refs diegosouzapw#14547
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ouzapw#12849) (diegosouzapw#13248)

Merged after renumbering. `176_provider_connection_synced_models_at.sql` collided with `176_xp_action_counts.sql` (diegosouzapw#12651), which made the migration runner abort on every DB open. Renamed to **177**; the doc count moves 173 → 174 across README.md, AGENTS.md, llm.txt and the i18n mirrors (operator-approved, 206 numeric substitutions and nothing else).

- `check:migration-numbering`: OK, 174 migrations, no duplicates
- `check:docs-counts` migrations: ✓
- 84/84 across the NVIDIA suite plus the seven DB-touching suites the collision had taken down
- ESLint and `typecheck:core`: exit 0

Heads-up for whoever lands next: **177 is claimed by eight other open PRs** (diegosouzapw#13610, diegosouzapw#13602, diegosouzapw#13580, diegosouzapw#13554, diegosouzapw#13405, diegosouzapw#13331, diegosouzapw#13177, diegosouzapw#13116) and 176 by diegosouzapw#13373 and diegosouzapw#13102. With this merged, all of them need to renumber at merge time — `check:migration-numbering` forbids new gaps, so the next free number is always the only valid one.

⚠️ base-red inherited: diegosouzapw#12732
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…diegosouzapw#13580)

`proxy_logs` records `upstream_status`, the HTTP status the provider actually returned through the proxy, instead of only success/timeout/error.

Maintainer rework before merge (kept the idea, no default behavior change):
- The migration collided with the tip (177 was already taken): renumbered to `179_proxy_logs_upstream_status.sql`, the runner's already-applied check moved to `case "179"` (the old `"177"` would have skipped the tip's own 177), migration count bumped to 176 in README, AGENTS.md, llm.txt and its mirrors (operator-approved).
- A new test runs the real migration runner on the real SQL files and fails with the old case number.

Validated first on the combined board of all 38 PRs of this batch (10 merged as-is, 28 after the maintainer rework) on top of release/v3.8.51 d61b804: typecheck:core, check:open-sse-typecheck and check:dashboard-typecheck clean; ESLint clean on every changed file; file-size (rebaselined for the combined growth), complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync, migration-numbering and i18n new-key gates green; 735 focused node:test cases with the only batch-caused failure (a flag-count assertion) fixed. Then re-validated alone on the fresh release tip right before this merge: ESLint on the changed files, typecheck:core, check:open-sse-typecheck, the file-size/complexity/changelog gates and this PR's own tests.

Thanks @maxmad64bis!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants