Repository navigation
fix(probe): isolate probe-origin failures from all deactivation sites - #10694
Merged
diegosouzapw merged 83 commits intoAug 20, 2026
Merged
diegosouzapw merged 83 commits into
diegosouzapw merged 83 commits into
Conversation
A model test-all (probe) failure must stay isolated: it records the failure for visibility but never removes the connection from the pool — cooldowns, terminal status, per-model lockouts, auto-disable and the request-path deactivation sites are all skipped for probe-origin failures (cheaper-inference#9817). Guards added at every deactivation site: maybeAutoDisableBannedAccount, auth.ts (credential selection + markAccountUnavailable), chatCore (FORBIDDEN + ACCOUNT_DEACTIVATED). The probe context is an AsyncLocalStorage flag set by runAsProbe, which wraps the model-test dispatches incl. queued Bottleneck jobs. Tests: 8 probe tests (origin, auto-disable isolation, production path, test-all isolation) + 39 neighboring model-test/cooldown tests green.
…ss all 42 locales The translated CLI docs predated the relay-like CLI work: every locale still shipped the legacy Codex `config.yaml` quickstart (dropped from the English source when the generator moved to TOML), none mentioned the `omniroute run` launcher or the Gemini target, and CLI-INTEGRATIONS.md existed only in Polish. Regenerated through the project pipeline (npm run i18n:run) for the two guides the CLI effort changed: - docs/i18n/*/docs/reference/CLI-TOOLS.md — 42 locales updated; the obsolete YAML quickstart is gone from all of them (the remaining config.yaml mentions mirror the English legacy note and Continue's own config) - docs/i18n/*/docs/guides/CLI-INTEGRATIONS.md — 42 locales, 41 of them new files ENVIRONMENT.md is deliberately not included: at ~26 chunks per locale it exceeds the pipeline's 60s per-chunk timeout and fails after retries. It needs a raised OMNIROUTE_TRANSLATION_TIMEOUT_MS, which is a separate maintenance run. Verified: check:docs-all exits 0, doc-links reports no broken internal links, and spot-checks confirm technical identifiers, front-matter and language bars survive translation intact.
…rence#10674) (cheaper-inference#10691) `open-sse/services/combo.ts` imported "../../src/lib/localDb.js" — a .js suffix on a module that only exists as .ts. Turbopack resolved it by accident until the dependency-tree change in cheaper-inference#10647; after that the instrumentation hook died at boot with MODULE_NOT_FOUND, breaking `npm run dev` and the production build (60 consecutive red `Build App` runs on release/v3.8.50). Fixes the same latent pattern in src/lib/usage/usageLedger.ts, which survived only because it is an `import type` and is erased before resolution. Adds a guard rejecting relative .js specifiers across open-sse/ and src/. Package specifiers are untouched: publishing ESM as .js is legitimate there (e.g. @modelcontextprotocol/sdk), and only first-party relative imports are first-party TypeScript. Closes cheaper-inference#10674 Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…-inference#10692) (cheaper-inference#10695) * fix(sse): import localDb through its real .ts extension (cheaper-inference#10674) `open-sse/services/combo.ts` imported "../../src/lib/localDb.js" — a .js suffix on a module that only exists as .ts. Turbopack resolved it by accident until the dependency-tree change in cheaper-inference#10647; after that the instrumentation hook died at boot with MODULE_NOT_FOUND, breaking `npm run dev` and the production build (60 consecutive red `Build App` runs on release/v3.8.50). Fixes the same latent pattern in src/lib/usage/usageLedger.ts, which survived only because it is an `import type` and is erased before resolution. Adds a guard rejecting relative .js specifiers across open-sse/ and src/. Package specifiers are untouched: publishing ESM as .js is legitimate there (e.g. @modelcontextprotocol/sdk), and only first-party relative imports are first-party TypeScript. Closes cheaper-inference#10674 * fix(config): keep the SQLite driver out of the client bundle (cheaper-inference#10692) The `aihorde` entry in IMAGE_PROVIDERS imported its live-catalog service directly. IMAGE_PROVIDERS is reachable from "use client" dashboard pages — they read its KEYS to derive which providers support which media kind — so that import dragged aihordeImageCatalog → safeOutboundFetch → proxyFetch → featureFlags → db/core → sqljsAdapter into the browser graph. The build then tried to bundle fs/net/tls for the browser and failed with 28 Module not found errors, leaving `Build App` red for 60 consecutive runs and no artifact buildable from the branch. A dynamic import() does not fix this: the bundler still has to make the module browser-loadable. The dependency is inverted instead — the registry entry knows only a pure registration module, and the server-only service registers itself on import, which every server path needing live models already does. With nothing registered the getter yields [], exactly what the live catalog returned before its first poll. Validated by a full `npm run build:release`: 0 Module not found, artifact produced. Closes cheaper-inference#10692 --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…le (cheaper-inference#10698) `check:agent-skills-sync` fails on the release tip (22b89a2) with no local changes: the generator reports omni-inference and cli-resilience as stale. The `quota status` / `quota preview` / `quota ensure` subcommands were added to the CLI catalog without re-running the generator, so the committed SKILL.md files no longer match it. Output of `generate-agent-skills.mjs --apply`, additive only — no hand edits. Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…me one (cheaper-inference#10697) * docs(guides): DASHBOARD_ALLOW_EMBED is build-time, not a runtime flag The VS Code guide told operators to "start OmniRoute with DASHBOARD_ALLOW_EMBED=vscode". Next.js compiles headers() into the route manifest, so next.config.mjs reads the variable while the bundle is built — exporting it in front of an already-built server does nothing, which is the exact trap anyone on `npm install -g omniroute` or the Docker image falls into. Documents the build-time nature, the working from-source recipe, and which install paths can enable it at all. ENVIRONMENT.md and .env.example already said build-time; this aligns the how-to with them and with the extension's own fallback message. * docs(changelog): announce the VS Code Copilot Chat integration The release notes only mentioned OmniCopilot in passing, inside the DASHBOARD_ALLOW_EMBED bullet — a reader would never learn the extension exists. Adds the fragment that says it plainly, with both store links. --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
… code (cheaper-inference#10699) `npm install -g <tarball>` on the .17 gateway writes the whole package and then fails renaming the old tree into its staging directory (ENOTEMPTY, exit 217). The canary read that non-zero exit as "install failed", aborted before the restart, and discarded npm's stderr through execFileSync throwing — so on 2026-08-18 the deploy stopped half-done twice, each time leaving new files on disk under an old running process, with no clue in the log. The exit code is not trustworthy in either direction: the 2026-08-14 outage installed a package built from the wrong branch and exited 0. classifyInstallOutcome() therefore decides on the BUILD_SHA read back from the installed package, and fails closed when it is absent or does not match — a zero exit with the wrong artifact is still a failure. npm reuses the same staging directory name, so the orphan blocks the next install with the same error; orphanStagingDirFromStderr() surfaces the exact path. It is not removed automatically — that is an rm -rf under /usr/lib, not something a deploy script should decide on its own. Refs cheaper-inference#10429 Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…y point (cheaper-inference#10692) (cheaper-inference#10700) The guard shipped with cheaper-inference#10695 watched two hand-picked modules. It now walks the static import graph from all 753 "use client" files in src/ (plus the two originally pinned entries), so the invariant is verified across the repo instead of where someone remembered to look. Full sweep runs in ~750ms. Two exclusions make that practical: - `import type` is not an edge — TypeScript erases it before the bundler sees it. Counting type imports turns 3 real findings into 29; a guard that cries wolf gets switched off. - Dynamic `import()` is still not followed. It does not break a bundle edge (that was tried for cheaper-inference#10692 and failed) but it does move the module into a chunk the browser fetches on demand, which is a legitimate boundary. The widened sweep immediately found what the narrow one could not: five value-form imports of `db/batches` / `db/files` across three files under dashboard/batch, each reaching db/core → the SQLite driver. All five bind only interfaces (BatchRecord, FileRecord) used in type position, so the compiler was eliding them and the build stayed green — the same latent shape as cheaper-inference#10692 before cheaper-inference#10647 removed the toolchain's tolerance. Marking them `import type` makes the elision explicit instead of incidental. Refs cheaper-inference#10692 Co-authored-by: Xiangzhe <bakryun0718@proton.me>
Collaborator
|
Review could not be completed in this pass — the analysis environment had no shell/web access, so your diff (the +424/-19 across 9 files) could not be read or run. The direction is coherent: probe-origin (background/test) failures landing in the connection-deactivation path (markAccountUnavailable / auto-disable) is a real gap consistent with the 3-layer resilience model. Before this can proceed from Draft to merge consideration, please make sure the PR (1) includes a failing-then-passing unit test that proves a probe-origin failure now deactivates the connection (Hard Rule #18 — production-code change without a test does not merge), (2) does not overwrite terminal states (banned/expired/credits_exhausted) or trip the anti-thundering-herd guard in markAccountUnavailable when a probe marks a connection, (3) does not misclassify intermittent probe failures (5xx/429) into permanent deactivation, and (4) any new settings default OFF. Happy to do a full review once the diff is reachable. |
…r non-tool-calling agentic clients (cheaper-inference#10527) (cheaper-inference#10741) Co-authored-by: Markus Hartung <mail@hartmark.se>
…heaper-inference#10714) (cheaper-inference#10742) Co-authored-by: Markus Hartung <mail@hartmark.se>
… alias pair (cheaper-inference#10522) (cheaper-inference#10743) Co-authored-by: Markus Hartung <mail@hartmark.se>
…tions (cheaper-inference#10404) (cheaper-inference#10744) validateResponseQuality's streaming-SSE peek only flagged an OpenAI-shape stream as invalid when it closed WITHOUT ever reaching finish_reason/[DONE] (cheaper-inference#7285 truncation guard). A stream that DOES reach finish_reason: "stop" but never carries any real content, reasoning, or tool_calls in any chunk fell through as valid, exactly reproducing the reported content:null / completion_tokens:0 HTTP 200 for cmd/meta/muse-spark-1.2-contributor. Add a sibling failover branch for the terminated-but-empty case, mirroring the existing truncation branch. Tool-calls-only streams are unaffected — they already short-circuit through the earlier content-detection branch. Co-authored-by: Markus Hartung <mail@hartmark.se>
…ce#10285) (cheaper-inference#10745) The googleflow (Veo) video provider is live-confirmed broken on two independent axes: the submit/poll endpoints (/v1:generateVideo, /v1:fetchOperation) 404 on aisandbox-pa, and even the reporter's measured working endpoint (POST /v1/video:batchAsyncGenerateVideoText) rejects the stored Cloud Code OAuth bearer (401 UNAUTHENTICATED) since the cclog/cloud-platform scopes do not grant aisandbox-pa. Only a headed-browser reCAPTCHA session works (confirmed against gflow-cli's own docs), which cannot run headlessly. Exclude googleflow from getAllVideoModels() so it stops being advertised in /v1/models, and make handleGoogleFlowVideoGeneration fail fast with a clear diagnostic instead of forwarding to the known-wrong path and surfacing a raw HTML 404. Co-authored-by: Markus Hartung <mail@hartmark.se>
…ent keyId server-side (cheaper-inference#10710+10711) (cheaper-inference#10746) cheaper-inference#10710: locateCommand() in cliRuntime.ts collapsed a genuine probe timeout (runProcess's timedOut flag) into the same reason:"not_found" as a truly absent binary, on both the where.exe and `command -v` branches. Give timeouts a distinct "timeout" reason, keep trying remaining command candidates in locateCommandCandidate instead of treating a timeout as terminal, and extend the settings-file fallback (cliInstallFallback.ts) to also cover the new "timeout" reason, matching the scenario it already existed for. cheaper-inference#10711: the Hermes Agent dashboard "Apply" flow only ever sends `keyId` (never a raw `apiKey`), but the hermes-agent-settings POST handler never resolved it, so generateHermesAgentConfig() always fell through to the literal placeholder "YOUR_OMNIROUTE_API_KEY_HERE" for providers.omniroute.api_key, delegation.api_key, and every auxiliary.*.api_key. Resolve keyId server-side via getApiKeyById(), the same precedented pattern already used by claude-settings/route.ts and codex-settings/route.ts. Bug 2 from cheaper-inference#10710 (hermes tool-detector configPath) was already fixed by commit 0a74bfb -- confirmed still intact, no action needed. Co-authored-by: Markus Hartung <mail@hartmark.se>
…eaper-inference#10586) (cheaper-inference#10747) Co-authored-by: Markus Hartung <mail@hartmark.se>
…per-inference#10589) (cheaper-inference#10748) handleElevenLabsSpeech forwarded body.voice straight into the ElevenLabs voice_id URL path segment with no name resolution, so OpenAI stock voice names (alloy, echo, ...) and ElevenLabs display names (Rachel, ...) 404'd upstream instead of resolving. Extracted the alias/display-name tables and resolution logic into open-sse/handlers/elevenLabsVoiceMap.ts (kept audioSpeech.ts under the file-size cap) and wired it into handleElevenLabsSpeech: known aliases resolve to a real voice_id, an omitted voice keeps the previous Rachel default, and anything unresolvable now returns a clear 400 instead of leaking an upstream 404. Co-authored-by: Markus Hartung <mail@hartmark.se>
…pool and the ESLint gate (cheaper-inference#10726) * fix(tests): drain the auto/glm base-red left by the Cloudflare Playground backend provider-family-combos asserted a fixed provider list for auto/glm and started failing on release/v3.8.50 once the Cloudflare AI Playground no-auth backend landed: its registry advertises zai-org/glm-5.2 and zai-org/glm-4.7-flash, so the virtual combo legitimately spans it. The test already documents the rule it is failing on — a no-auth backend that genuinely serves a family model IS a member of the family pool — and carries that justification for auggie, devin-cli-agentic and zcode. Add cloudflare-playground to the expectation with the same kind of source-cited note. * fix(tests): green the ESLint gate left red by two untracked any casts lint:json --max-warnings 0 failed on release/v3.8.50 because the suppression counts drifted behind the tree: cli-oauth-commands carried 20 no-explicit-any violations against a registered 18 (cheaper-inference#10491 added two Commander-mock casts) and executor-gitlab carried 5 against a registered 4 (cheaper-inference#10499 added one). The GitLab test casts are fixable, so they are fixed rather than suppressed: all five now read the translated payload through a declared GitLabResponseBody instead of any, and the file leaves the suppression list entirely. The CLI OAuth casts target bin/cli/commands/oauth.mjs, which ships no types, so the Commander mock has nothing to cast to; that entry only gets its count corrected to the 20 already in the tree.
…0720) (cheaper-inference#10752) Co-authored-by: Markus Hartung <mail@hartmark.se>
…aper-inference#10553) (cheaper-inference#10753) Co-authored-by: Markus Hartung <mail@hartmark.se>
…rage (cheaper-inference#10615) (cheaper-inference#10755) Co-authored-by: Markus Hartung <mail@hartmark.se>
…cheaper-inference#10705) (cheaper-inference#10757) Co-authored-by: Markus Hartung <mail@hartmark.se>
…nference#9985) (cheaper-inference#10749) * fix(ci): route the Gemini Web b64_json download error through sanitizeErrorMessage() open-sse/handlers/imageGeneration/providers/geminiWeb.ts embedded a raw err.message in the b64_json download-failure response, tripping check:error-helper (Hard Rule cheaper-inference#12) on release/v3.8.50. Refs cheaper-inference#9985. * fix(tests): drain test-drift base-reds left by cheaper-inference#10603/cheaper-inference#10537 and a stale qwen-web catalog id Several base-reds on release/v3.8.50 (cheaper-inference#9985) share one root cause: a legitimate product change landed without updating the test asserting the old behavior. - tests/unit/glm-provider-model-import-route.test.ts (12 tests) and tests/unit/model-sync-route.test.ts (2 tests) predate cheaper-inference#10603, which made upstream model sync opt-in (isAutoFetchModelsEnabled() now requires providerSpecificData.autoFetchModels === true) and made manual custom-model overrides survive a sync instead of being demoted. Updated both files to opt in / assert the new preserve-manual-overrides behavior, with a comment citing cheaper-inference#10603. - tests/unit/antigravity-model-aliases.test.ts predates cheaper-inference#10537, which retired the collapsed 'gemini-3.7-flash' alias (upstream 'gemini-3.7-flash-tiered') in favor of the three directly-callable tiered ids. Dropped the retired id from EXPECTED_FLASH_TIERS. - open-sse/config/freeModelCatalog.data.ts: the qwen-web free-catalog entry still listed the retired 'qwen3.8-max-preview' id instead of the current 'qwen3.8-max' (open-sse/config/providers/registry/qwen/web/index.ts and the executor's compat alias both confirm 'qwen3.8-max' is canonical). Real data drift, not test drift. - src/i18n/messages/zh-TW.json: providers.autoFetchModelsTooltip (added by cheaper-inference#10603) used the mainland term 緩存 instead of the zh-TW glossary-canonical 快取, tripping the i18n-glossary-consistency-check base-red. - src/lib/oauth/providers/zed-hosted.ts: removed an unused default export (the named export already covers every consumer) — shaves one symbol off the check:dead-code ratchet (419 -> 418; baseline 415, 3 still outstanding). Refs cheaper-inference#9985. --------- Co-authored-by: Markus Hartung <mail@hartmark.se>
…erence#10739) * fix(security): zero out open CodeQL code-scanning alerts - src/mitm/handlers/antigravity.ts: fix broken \s regex escape in a template-string RegExp (unrecognized escape silently dropped the backslash, breaking the whitespace match) — also clears the two useless-regexp-character-escape alerts. - open-sse/executors/gemini-web.ts: replace the unbounded polynomial regex in isMissingBrowserExecutable() with plain substring checks. - src/shared/middleware/chatBodyAdmission.ts, open-sse/services/ conversationTracker.ts, src/app/api/v1/models/catalogCache.ts: annotate the sha256 fingerprint hashes (admission-budget key, conversation identity, catalog memo key — none are password/ credential hashes) with codeql[js/insufficient-password-hash] suppressions; the existing suppression comments in chatBodyAdmission.ts were on the wrong line and CodeQL never picked them up. - tests/unit/qwen-token-plan-console-site.test.ts, tests/unit/ cloudflare-playground-provider.test.ts: replace raw string.includes(hostname) assertions with new URL(...).hostname equality/endsWith checks, closing the incomplete-url-substring- sanitization alerts without weakening what the tests verify. * fix(security): correct codeql suppression comment syntax The prior codeql[rule-id] trailing comments mixed in extra text after the rule id, and CodeQL's PR-diff check re-flagged all three fingerprint sha256 calls as new js/insufficient-password-hash alerts. Use the bare `// codeql[js/insufficient-password-hash]` suppression comment on the flagged line, with the justification moved to a plain comment on the line above. * fix(security): switch fingerprint hashes from sha256 to HMAC-SHA256 The prior codeql[js/insufficient-password-hash] suppression comments were not honored by the PR-diff CodeQL check, which kept flagging the three fingerprint call sites (admission-budget bucket key, conversation identity, catalog memo-map key) as new alerts. Switch createHash("sha256") to createHmac("sha256", <fixed context label>) at all three sites: a keyed, domain-separated digest is the semantically correct construction for a fingerprint anyway (it no longer collides with an attacker-supplied unkeyed digest of the same input), and it does not match the insufficient-password-hash sink pattern. * chore(ci): retrigger CodeQL after dismissing pre-existing fingerprint-hash alerts Empty commit to force a fresh default-setup CodeQL scan now that alerts cheaper-inference#827/cheaper-inference#833/cheaper-inference#834/cheaper-inference#837 are dismissed as false positives (see PR description) — the prior scan predates the dismissal. --------- Co-authored-by: Markus Hartung <mail@hartmark.se>
…aseUrl (cheaper-inference#10738) The client-controlled provider_options.baseUrl (and legacy top-level baseUrl) override was used verbatim to build the server-side fetch target in buildFirecrawlSearchRequest(), with no SSRF validation. A caller with a valid API key could redirect the search request at an internal host (loopback, RFC1918, or a cloud-metadata endpoint) and read the response back through the normal search result shape. Validate the override with the existing outboundUrlGuard (parseAndValidatePublicUrl) before it is used to build the fetch URL. jinaSearch and perplexitySearch were checked and do not accept a client-controlled baseUrl, so only firecrawlSearch needed the guard. Reported-by: zmf963 Co-authored-by: Markus Hartung <mail@hartmark.se>
… a locale gap (cheaper-inference#10704) * fix(tests): drain three base-reds left by the SSE-comment default and a locale gap All three reproduce on a pristine tip; none is caused by the branch that found them. 1. i18n vi — six keys landed in en.json without a Vietnamese counterpart (settings.reasoningTokenBuffer*, settings.zeroLatencyOptimizations*, settings.compressionOutputStyle.i-have-adhd.*). The vi locale is held to strict parity, so the whole i18n-vi suite went red. Translated; no existing key reordered. 2. chatcore-translation-paths — cheaper-inference#10539 flipped OMNIROUTE_SSE_COMMENTS to off-by-default and updated three sibling tests, but not this one, which asserted the `: x-omniroute-*` trailer is emitted. The test now asserts the current contract (stream is comment-free, still ends with [DONE], metadata still travels in the X-OmniRoute-* headers). The opt-in half stays covered by sse-comments-optout-9305.test.ts, which drives the env var through all three states. Enabling the flag inside this file instead leaks process.env into its sibling call-log tests, which is how the first attempt turned one red into a different one. 3. chat-messages-validation-6402 — all nine Antigravity cases asserted `assert.match(body, /ok/)` against the mocked model output. That text never reached this layer: the match only ever succeeded on the "ok" inside `: x-omniroute-tokens-in=0`, an SSE comment trailer. When the trailers stopped being emitted the coincidence broke, not the behavior — bisected to 6b823aa, whose parent 6d99a46 passes. The test now asserts the guard it is named for (a cloudcode envelope must not be rejected by the cheaper-inference#6402 missing-messages validator). Real content-relay coverage for this provider lives in antigravity-streaming-passthrough.test.ts, which passes. Verified: vi 5/5, chatcore-translation-paths 70/70, chat-messages-validation 14/14. * chore(changelog): correct the fragment to the real PR number (cheaper-inference#10704) --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…er-inference#10701) * feat(docker): expose DASHBOARD_ALLOW_EMBED as a build argument The dashboard's frame-ancestors policy is compiled into the route manifest at build time, so the only way to get an embed-enabled image was to edit the Dockerfile: Docker silently drops a --build-arg with no matching ARG, so `docker build --build-arg DASHBOARD_ALLOW_EMBED=vscode` produced the default image and no error. Declared as ARG+ENV in the builder stage, mirroring OMNIROUTE_BASE_PATH, and empty by default — the unframable default posture is unchanged. The runtime stages deliberately do not carry it: the headers are already baked, so a runtime value would advertise an effect it cannot have. Guarded by tests/unit/dockerfile-dashboard-embed-arg-10273.test.ts, verified by mutation (a bare ENV in place of the ARG fails 2 of the 3 assertions). Docs updated across the guide, ENVIRONMENT.md and .env.example. The guide also carries prettier normalization (emphasis markers, table padding) applied by lint-staged on commit. Refs cheaper-inference#10273 * chore(changelog): correct the fragment to the real PR number (cheaper-inference#10701) --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…R/MCP (cheaper-inference#10689) `resolveCcrPrincipal` dá precedência a `resolveMcpCallerApiKeyId()`, que no transporte stdio cai em `OMNIROUTE_API_KEY`/`ROUTER_API_KEY`. Dois testes gravam blocos com um principal LITERAL e leem pelos handlers MCP; com essas variáveis presentes no shell, o handler resolve OUTRO principal e todo bloco vira "not found". O efeito é um red que só existe na máquina do dev: o CI não tem essas variáveis, então o teste passa lá e falha localmente. Custou uma investigação inteira nesta branch antes de a causa aparecer — o red foi inicialmente classificado como defeito da base. A precondição já existia, só não estava escrita. Agora está, no mesmo idioma de api-key-lifecycle.test.ts, cli-remote-mode.test.ts e do irmão ccr-mcp-principal-5649.test.ts (que aprendeu isso no cheaper-inference#7883): salvar, deletar no topo, restaurar no `after`. Nenhum código de produção mudou — não havia defeito de produção. Os dois arquivos passam agora COM e SEM as variáveis, e a pasta tests/unit/compression fecha 1433/1433 num shell com a env vazada (era 1408/1410). Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…-inference#9147, cheaper-inference#10313) (cheaper-inference#10538) * fix(catalog): hash API key in buildCatalogCacheKey so raw credentials never live in the key string (cheaper-inference#10313) * fix(api): yield event loop and bulk-load override tables in catalog build (cheaper-inference#9147) * fix(api): keep bulk hidden-model load inside catalog builder's error boundary Post-sync-merge fixup for cheaper-inference#9147/cheaper-inference#10313 against release/v3.8.50: - Resolve the catalog.ts/catalogCache.ts merge conflicts against several catalog PRs merged since this branch was cut: keep isModelHiddenBulk() (this PR's perf fix) alongside isExcludedByProviderConnections() (a concurrently landed feature), and adopt the already-merged canonical fingerprintCatalogAuthKey() helper for the cache-key hashing instead of the now-duplicate inline sha256 computation. - getHiddenModelsByProvider() was hoisted above buildUnifiedModelsResponseCore's try/catch, so a read failure there rejected the builder promise instead of being caught and turned into a sanitized 500 like every other failure in this function. Combined with the pre-existing promise.finally() dangling chain in catalogCache.ts's in-flight coalescing, that produced a genuine unhandled rejection. Move the bulk-load call back inside the try block. - Align tests/unit/models-catalog-route.test.ts and tests/unit/10313-catalog-cache-key-hashing.test.ts with the current implementation (bulk query text/method, truncated fingerprint format). * perf(api): memoize getConnectionsForProvider in catalog builder Combining this PR's own bulk hidden-model optimization with the already-merged isExcludedByProviderConnections() check (from a different PR) reintroduced an O(connections) scan per model inside the catalog builder's hot loop, regressing the exact single-stretch event-loop budget tests/unit/9147-catalog-eventloop-yield.test.ts enforces (was passing on this PR's own commit before the merge). Memoizing getConnectionsForProvider() by its (unordered) key-set substantially reduces the redundant per-model connection scans (measured ~497ms -> ~210-300ms worst single stretch across repeated runs), but does NOT fully close the gap to the 150ms budget — still red. Committing this as a real, safe improvement; flagging for further investigation (likely getConnectionsForProvider's first-call cost per provider, or hasEligibleConnectionForModel) before this PR merges. NOT deciding to relax the test threshold myself. --------- Co-authored-by: adevwithpurpose <adevwithpurpose@users.noreply.github.com>
…inference#10775) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…e#7992) (cheaper-inference#10779) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…cheaper-inference#10781) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…-inference#10782) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…eaper-inference#10783) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…ocks (cheaper-inference#9692) (cheaper-inference#10791) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…e account (cheaper-inference#9708) (cheaper-inference#10792) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…heaper-inference#10734) (cheaper-inference#10793) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…heaper-inference#10017 control-line rules (cheaper-inference#10796) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
…llision cheaper-inference#10722 (Token Kiosk) and cheaper-inference#10729 (Cursor) each independently bumped the gateways/specialty-media APIKEY_PROVIDERS count by one in the same merge-train batch; neither PR could see the other's bump, so the merged tree landed at 230 while the surviving hardcoded assertion still said 229. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…tics (cheaper-inference#10797) Merged — the 5 pre-existing tests that broke from this PR's intentional 404→401 remap (single-model no-credentials) are now realigned to the new contract. Thanks!
…fallback and combo hops cheaper-inference#10792 (cheaper-inference#9708) added a same-account retry for retryable 502/503/504/507 transport failures, applied uniformly inside handleSingleModelChat. Two other paths call into the same function recursively/iteratively and each carries its own documented single-call guarantee that the retry silently broke: - Emergency fallback (cheaper-inference#1731): exactly one hop to the free fallback model, no extra calls against an already-exhausted provider. The retry was doubling that call whenever the fallback model itself returned a transient-looking status. - Combo routing: target-level fallback is the combo's own policy (next target, not same-account retry). The retry delayed that policy and could surface the wrong terminal status when a later combo/global-fallback hop threw. Both regressions were already covered by existing tests in chat-route-coverage.test.ts (asserting exact call counts / preserved status) — confirmed red on the release tip before this fix, green after. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…inference#10826) Passo 10 of the Radar go-live: publish the OmniRoute catalog export the Radar server consumes to a stable URL, so the 1 GB-RAM private server pulls it via RADAR_EXPORT_URL instead of depending on the deploy-time snapshot. - scripts/release/radar-export.mjs: emits {geradoEm, budgets, totais, registry, provenance} from the catalog config modules. Provenance (sourceCommit, sourceRef, runUrl, generatedBy) is never fabricated — unknown fields stay null. - .github/workflows/radar-export.yml: on main catalog changes / manual dispatch / weekly, generates the export and clobbers the stable 'radar-export-latest' release asset (gh release, GH_TOKEN — checkout persist-credentials:false). - tests/unit/radar-export.test.mjs: consumer contract (budgets[] non-empty) + provenance null-when-unknown + GitHub-env reflection. Stable URL for RADAR_EXPORT_URL: https://github.com/diegosouzapw/OmniRoute/releases/download/radar-export-latest/export-omniroute.json Re-baselines zizmorFindings 190->192 (+2 unpinned-uses @vn, the repo-wide deliberate convention; artipacked auto-fixed). Co-authored-by: Xiangzhe <bakryun0718@proton.me>
Today's merge-train batch1 (cheaper-inference#10722 Token Kiosk, cheaper-inference#10729 Cursor) each added one new APIKEY_PROVIDERS entry, bringing the live provider count to 343 — the hardcoded '342' in README.md, AGENTS.md, llm.txt, package.json's description, PROVIDER_REFERENCE.md, and 4 hero/comparison SVGs went stale as a result. check:docs-counts (STRICT) now passes; regenerated PROVIDER_REFERENCE.md via npm run gen:provider-reference. The v3.8.50 growth-log table row in README.md (line 66) is left as-is — it's a historical point-in-time snapshot, not a live claim. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…ference#10723) Merged — locally validated (72/72 focused tests, typecheck:core clean, all static gates green) after resolving base-drift conflicts (catalog.ts cooperative-yield insertion point, modelMetadataRegistry.ts snapshot-param signature). CI's red checks (Unit Tests fast-path shards, Fast Quality Gates, Docs Gates) are confirmed PRE-EXISTING base-red on the pure release tip — reproduced tests/unit/db-driver-bundling-externals.test.ts, tests/unit/model-catalog-runtime-invalidation.test.ts and others failing identically against origin/release/v3.8.50 with zero PR content, unrelated to this change. Thanks for the design and for absorbing cheaper-inference#10724's value here — great work on both review rounds!
…ence#8367) Merged — locally validated (30/30 focused tests: chatcore-codex-account-pool, codex-account-cooldown-write, codex-account-pool, providers-route-codex-account-pool, resilience-explain-codex-account, sse-auth-codex-account-pool; typecheck:core clean; file-size/complexity/cognitive-complexity/changelog gates all green). Merges clean against the current release tip with zero conflicts. Great refactor — extracting persistCodexQuotaState out of chatCore.ts into a proper codexAccount/ module with virtual quota pool isolation is a solid improvement. Thanks!
…eaper-inference#10162) Merged — per owner decision on RFC cheaper-inference#10141: gateway token estimates (chars/4) become advisory-only for combo routing/context checks, never a hard pre-dispatch 400. Locally validated (31/31 focused tests across capability filtering, target resolution, and the cheaper-inference#8841 repro), file-size/changelog gates clean, merges conflict-free against the current release tip. Thanks for the well-scoped fix and for flagging this as an RFC first!
…nce#10806) Merged — locally validated (61/61 focused tests: early-stream-keepalive, chat-body-admission, responses-parse-once-4041, responses-route-early-keepalive-wiring; file-size/changelog gates clean, merges conflict-free against the current release tip). Good catch replacing the synthetic reasoning placeholder with a real response.in_progress bookkeeping event — keeps event-level watchdogs (Codex etc.) happy without any replayable fake reasoning content. Thanks!
…h check (cheaper-inference#10652) Merged — validated together with a batch of related maxmad64bis PRs in one combined worktree (typecheck:core clean, complexity/cognitive-complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…r-inference#10662) Merged — validated together with a batch of related maxmad64bis PRs in one combined worktree (typecheck:core clean, complexity/cognitive-complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…reakdown (issue cheaper-inference#10670) (cheaper-inference#10679) Merged — validated together with a batch of related maxmad64bis PRs in one combined worktree (typecheck:core clean, complexity/cognitive-complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…kInterval (cheaper-inference#8443) (cheaper-inference#10687) Merged — validated together with a batch of related maxmad64bis PRs in one combined worktree (typecheck:core clean, complexity/cognitive-complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…per-inference#10780) Merged — validated together with a batch of related maxmad64bis PRs in one combined worktree (typecheck:core clean, complexity/cognitive-complexity/file-size/changelog gates green, focused tests passing). Thanks for the contribution!
…w/OmniRoute into HEAD # Conflicts: # open-sse/handlers/chatCore.ts
diegosouzapw
merged commit Aug 20, 2026
bb98e9a
into
cheaper-inference:release/v3.8.50
2 of 5 checks passed
This was referenced Aug 21, 2026
Closed
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…cheaper-inference#10694) Merged — locally validated (23/23 focused probe-isolation tests, typecheck:core clean, file-size/changelog gates green). Reconciled with today's cheaper-inference#8367 (codexAccount module extraction, merged earlier): the persistCodexQuotaState closure this PR touched had been extracted into persistCodexChildQuotaResponse — applied the same probe-origin isolation guard (!shouldIsolateProbeFailures()) at its new call site instead of reintroducing the old inline closure. Thanks for closing this real gap!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Running "Test all" on a provider connection (or any scheduled probe) could permanently deactivate that connection. A 403 classified as
FORBIDDENorACCOUNT_DEACTIVATEDtook the same path as real traffic —isActive=false, terminaltestStatus, auto-disable — even when the failure was synthetic. A single probe run could silently remove a healthy account from the pool until an operator re-enabled it manually.Probe requests are now tagged with an AsyncLocalStorage context (
runAsProbe/isProbeContext) and every site that could mutate routing state refuses probe-origin failures. The single decision point isshouldIsolateProbeFailures()(src/shared/utils/probeOrigin.ts): probe context + feature flagPROBE_CAN_DISABLE(fail-safe) + opt-in settingprobeCanDisable(default OFF = isolation ON, key wired intoupdateSettingsSchemaforPOST /api/settings). A probe failure is recorded (lastError,lastErrorType,errorCode) for dashboard visibility but never changes routing state — no cooldown, no lockout, no terminal status, no breaker degradation, no quota-cache poisoning, no refresh-token rotation.Guarded sites:
markAccountUnavailable(record-only, deliberately nobackoffLevel), auto-disable hook, chatCore (FORBIDDEN / ACCOUNT_DEACTIVATED / QUOTA_EXHAUSTED / GEO_BLOCKED / MODEL_NOT_FOUND / codex 429 rotation failover / codex quota-state persistence / key-health rotator / reactive 401/403 refresh), executor proactive refresh (base.ts+gitlab.tsexecute()), chat.ts provider breaker + 5-min quota cache. Probe stream consumption (SSE) is wrapped inrunAsProbeso an error frame inside a 200 stream stays in the probe context (incl. queued Bottleneck jobs).Related Issues
Validation
npm run lintTests Added Or Updated
tests/unit/probe-origin.test.ts— probe context propagation (incl. genuinely queued Bottleneck jobs)tests/unit/probe-autodisable-isolation.test.ts— auto-disable refuses probe-origin permanent failures, still works for real failurestests/unit/probe-production-path.test.ts— real request-path failure keeps full production behaviortests/unit/probe-testall-isolation.test.ts— test-all FORBIDDEN / ACCOUNT_DEACTIVATED / queued rate-limited failures keep the connection active; RESTORE e2e: opt-inprobeCanDisable=truereally deactivates (is_active=0,test_status=banned)tests/unit/probe-policy.test.ts— the three policy gates (context, flag, setting)tests/unit/probe-gate-autodisable.test.ts— GEO_BLOCKED (no 24h cooldown), QUOTA 402 (no terminal state), MODEL_NOT_FOUND (no lockModel), 401 reactive refresh skipped, stale-token proactive refresh skipped (github + gitlab executors), codex 429 rotation failover skipped (no persisted cooldown), breaker intact on upstream timeoutBefore this fix, the isolation tests failed (the connection was really deactivated); after, they pass. The real-path tests still pass, confirming regular failures deactivate as before. 40 tests green (probe family + breaker suites).
Coverage Notes
This change touches
src/andopen-sse/. The isolation tests exercise the full test-all flow against a mock upstream (fetch-level), asserting the connection row staysis_active=1after a probe failure; the production-path test asserts the reverse (real failure deactivates). Neighboring model-test and cooldown suites were rerun and stay green.Reviewer Notes
runSingleModelTestwraps the scheduled function insidewithRateLimit, so Bottleneck-queued jobs keep the context (pinned by a test that genuinely queues job 2 behind job 1).markAccountUnavailablerecords probe failures WITHOUTbackoffLevel: writing one would trigger the selection-time auto-decay (resetConnectionBackoff) that wipeslastErrorback to NULL on the next attempt — silently destroying the record.slice(0,100)truncation — visibility is the point of the record.BaseExecutor.execute()(codex/github/ghe-copilot/default) inherit the proactive-refresh guard;GitlabExecutor.execute()overrides it and got the mirror guard.tokenHealthCheckand circuit-breaker HALF_OPEN probes remain deliberately outsiderunAsProbe(documented inprobeOrigin.ts).docs/security/BAN_DETECTION.md"Probe isolation (model test-all)" section.