Repository navigation
perf(proxy): implement non-blocking async proxy log batching and performance optimizations (A, B, C, D) - #11182
Merged
diegosouzapw merged 23 commits intoAug 23, 2026
Conversation
…ormance optimizations (A, B, C, D)
… confirmed) (cheaper-inference#11114) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Companion to cheaper-inference#11113 (consumer side): tokenrouter joins BUILTIN_PROVIDERS_SYSTEM_MUST_BE_FIRST — memory-system-first-6135 suite green. Live-confirmed 400 class documented in the body. Thank you @ggdayup!
…inference#11162) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Combo without models is now refused at the schema boundary (API 400), the CLI flags it, and openapi.yaml matches the real contract (phantom props removed). combo-* suites + cli-combo-create-models green on the board. Closes cheaper-inference#10954. Thank you @maxmad64bis!
…e) (cheaper-inference#11158) Validated on the combined batch board over tip 92ef3c7: static gates clean (changelog, file-size, complexity 2624<=2774, cognitive 1182<=1223, dead-code 411<=416), typecheck:core clean, focused tests green. Empty-envelope 400 (no error field, empty content, finish_reason null) now rotates/retries instead of propagating as success; 200/streaming path never buffered; real-error 400s untouched. account-rotation + new rotation suite 34/34 on the board. Thank you @maxmad64bis!
…stem message (cheaper-inference#11113) Validated on the combined batch board: purify-system-first suite 4/4, typecheck clean. Pre-merge: file-size baseline gained a frozen entry for contextManager.ts at 1001 (+1, this PR's merge-into-leading-system branch) with a dated annotation — the gate caps unlisted files at 1000. Producer side of the live-confirmed TokenRouter 400 class: no internal path emits a mid-array system message anymore. Thank you @ggdayup — the call-log evidence made this airtight!
…ly output (cheaper-inference#11151) Merged after conflict resolution against the tip's cheaper-inference#11109 (per-call tool_call tracking): scanOpenAiSseText keeps the per-call finish_reason special-case AND gains reasoningText + literal finishReason; canContinue uses the in-flight predicate with the new reasoning-only-clean-stop escape. One integration fix on the branch: the PR's hallucinatedEmptyStop referenced emittedToolCall, which cheaper-inference#11109 had renamed — the branch now tracks emittedSawToolCall at the emitted level (any tool_call delta, complete or not), preserving the PR's don't-recover-after-tool-calls intent. Chain suites green: stream-continuation-wiring + stream-continuation + stream-recovery-toolcall 29/29. Thank you @maxmad64bis!
…f concatenating it raw (cheaper-inference#11152) Merged after sibling cheaper-inference#11151 landed: streamRecovery.ts auto-merged byte-identical to the validated combined board; the test-file conflict (both PRs added suites at the same anchor) resolved keeping all 11 tests — cheaper-inference#11151's four clean-stop cases plus this PR's three threshold cases, with the PR's updated partial-tail fixture for the pre-existing overlap test. Full chain green: 32/32 (wiring + continuation + toolcall regression). The documented 8-char overlap threshold ends the silent mid-word gluing. Thank you @maxmad64bis!
cheaper-inference#11190) * feat(api): structured ?format=json for the self-service usage endpoint GET /api/usage/om-usage already let any key read its own usage — personal daily/weekly USD limits and the provider quota snapshot — but only as text/plain, which a UI cannot parse safely. OmniCopilot issue cheaper-inference#8 asks exactly for this surface. Adds ?format=json, returning the ApiKeyUsageLimitStatus + UsageSnapshot the text is rendered from. Text and JSON share the same collectors (collectUsageSnapshots, getApiKeyUsageLimitStatus), so the two can never disagree about a number. The response is a discriminated union: a key without allowUsageCommand (403) or an invalid key (401) returns { allowed:false, error:{message} }, distinct from allowed:true with empty sections — the state a panel must render as "nothing learned yet", not a refusal. Text form unchanged; without ?format the contract is untouched. The endpoint was previously missing from API_REFERENCE.md; it now has a section documenting both forms, the allowUsageCommand gate, and the self-service auth model (caller's own key, not requireManagementAuth). Regression guards in tests/unit/usage-command-json-format.test.ts (4 tests: json shape, text default preserved, structured 403, sanitized 401 with no stack trace). Existing internal-usage-command suite still 12/12. * chore(changelog): correct the fragment to the real PR number (cheaper-inference#11190) --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
…-usage json (cheaper-inference#11192) * feat(api): structured ?format=json for the self-service usage endpoint GET /api/usage/om-usage already let any key read its own usage — personal daily/weekly USD limits and the provider quota snapshot — but only as text/plain, which a UI cannot parse safely. OmniCopilot issue cheaper-inference#8 asks exactly for this surface. Adds ?format=json, returning the ApiKeyUsageLimitStatus + UsageSnapshot the text is rendered from. Text and JSON share the same collectors (collectUsageSnapshots, getApiKeyUsageLimitStatus), so the two can never disagree about a number. The response is a discriminated union: a key without allowUsageCommand (403) or an invalid key (401) returns { allowed:false, error:{message} }, distinct from allowed:true with empty sections — the state a panel must render as "nothing learned yet", not a refusal. Text form unchanged; without ?format the contract is untouched. The endpoint was previously missing from API_REFERENCE.md; it now has a section documenting both forms, the allowUsageCommand gate, and the self-service auth model (caller's own key, not requireManagementAuth). Regression guards in tests/unit/usage-command-json-format.test.ts (4 tests: json shape, text default preserved, structured 403, sanitized 401 with no stack trace). Existing internal-usage-command suite still 12/12. * chore(changelog): correct the fragment to the real PR number (cheaper-inference#11190) * feat(api): return every connection's snapshot under providers[] in om-usage json Closes cheaper-inference#11191. buildUsageCommandJson picked a single snapshot via selectUsageSnapshot, so a panel could only ever show one provider. The collector already had them all — the single-pick is a presentation choice for a terminal. The JSON form now also returns the full UsageSnapshot[] alongside the selected provider, so a UI can render Codex / Claude / OpenCode side by side. The text form is untouched. --------- Co-authored-by: Xiangzhe <bakryun0718@proton.me>
… links through the shortener (cheaper-inference#11196) - New CheaperInferenceSponsorBanner on the dashboard home, same size/shape as KimiSponsorBanner, no version gate (durable partnership). Uses the cheaperinference ProviderIcon and the brand green (#31f889) with the dark ink CTA (contrast, per colors.ts token). - CTA points at https://link.omniroute.online/cheaper — the branded short link — so clicks land in our Kutt metrics. - VscodeCopilotBanner CTA now points at https://link.omniroute.online/vsx instead of the raw Marketplace URL, for the same reason. - i18n strings in en + pt (en is the namespace-level fallback for the other 41 locales). - Tests: new cheaperInferenceSponsorBanner.test.tsx (render, CTA href, dismiss persistence); vscodeCopilotBanner.test.tsx updated to the new CTA URL. Co-authored-by: Xiangzhe <bakryun0718@proton.me>
… response quality validation (cheaper-inference#11036) SSE comment lines (OpenRouter keep-alives) and leading whitespace no longer fail the combo quality gate's JSON fallback. First contribution — clean minimal fix with test. Thank you @asorourx, welcome aboard!
…rence#11041) Compaction-V2 output now counts as real model output (no synthetic response.failed after response.completed), the Codex SSE filter handles CRLF framing, and terminal detection runs before scan-state bounding. 88/88 stream/readiness suites on the board. Thank you @jackjinke!
…11020) Validated on the combined batch board + this branch alone: chat-body-admission + authz/pipeline 65/65, file-size gate green with a dated frozen entry (chatBodyAdmission 1005→1009 — the +4 lease/drain wiring lines, owner-authorized rebaseline). trackRequest was never called, so SIGTERM waitForDrain saw zero in-flight and killed live SSE; leases now hold the drain counter for the stream's lifetime, and the 503 carries Retry-After. Closes cheaper-inference#11015. Thank you @RaviTharuma!
… retrieve tool (cheaper-inference#11084) Validated on the combined batch board + this branch: ccr-non-mcp-full-prompt-loss + ccr-retrieval-ramp 20/20, file-size gate green with the ccr/index listing (1024, dated annotation — owner-authorized). The callerSupportsCcrRetrieve gate now skips the whole engine for callers whose tools[] cannot reach omniroute_ccr_retrieve — no more 15KB prompt arriving upstream as 112 tokens. Production-measured root cause, textbook TDD. Thank you @HouMinXi!
…0963) Merged after conflict resolution in modelMetadataRegistry.ts: the tip's effortTiers chain (declared efforts → declared tiers → undefined-if-thinking-declared → codex extension) now carries this PR's GLM guard as the final-fallback override — GLM-family models without a provider-declared contract get the authoritative empty tier list instead of generic OpenAI tiers. GLM/ZCode suites 40/40 on the resolved branch. Closes cheaper-inference#10962. Thank you @xz-dev!
…heaper-inference#10949, cheaper-inference#10959) (cheaper-inference#10961) Validated on the combined batch board + this branch: 231/231 across chatcore-translation-paths, reasoning-cache, strip-reasoning-blobs, and both Responses translator suites. Pre-merge: propagated the cheaper-inference#11110/cheaper-inference#11129 summary:[] defaults into five assertions here (each commented with its PR) — without it this branch red against the tip, and as a bonus the merge drains the 4 reasoning reds that were live on the tip from those merges. Plaintext now wins over a coexisting opaque companion; opaque-only drops cleanly for plaintext targets; combos keep explicit Skip. Fixes cheaper-inference#10949 and cheaper-inference#10959. Thank you @jackjinke!
…eaper-inference#10644) (cheaper-inference#10987) Merged after count reconciliation: the branch's regenerated docs claimed 57 free forever / 157 migrations from its older base; gate-verified values on the current tip are 56 free forever (Logfare carries a Free badge via gateways.ts freeNote but has no freeModelCatalog per-model entries, so the live-code counter stays at 56) and 159 migrations — the README/SVGs now match the check:docs-counts output exactly. provider-consistency OK at 267 registry / 349 canonical. logfare-registry 4/4, icon + KNOWN_PNGS + discovery-set membership all verified present. Closes cheaper-inference#10644. Thank you @jonlwheat2-gif!
…eaper-inference#10964) Merged after conflict triage: the six base-red repair files (vi.json, opencode.ts JSDoc, context-manager test, the three webhook dispatcher tests, the uncloseai orphan-test rename) were already drained on the tip by today's cheaper-inference#11130/cheaper-inference#11157/cheaper-inference#11160/cheaper-inference#11113 — those hunks resolved to the tip shape. What lands is the production-fix set: GLM transport-aware Anthropic headers, Claude Code-compatible model-listing rejection, combo live-test single-probe, zero-cost Auto-Combo interval normalization, recovery-clearing union handling, LLMLingua real-path compare, macOS netstat PID discovery, AI Horde R2 strict public-host validation. Sweep of every touched test file: 243/243 green; typecheck + file-size clean. (guide-settings-route's 4 reds reproduce on the pure tip — pre-existing drift from cheaper-inference#11079, not from here.) Thank you @backryun!
…ocess-spawning endpoints (cheaper-inference#11189) Validated on the combined batch board (gates + typecheck clean) and this branch: security-route-guard-tiers green. Regression coverage for the Hard Rule cheaper-inference#15/cheaper-inference#17 contract — Tier 1 process-spawning prefixes (/api/services/, /api/mcp/, /api/cli-tools/runtime/) must stay LOCAL_ONLY before any auth check. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!
…ed labels (cheaper-inference#11188) Validated on the combined batch board + this branch: dashboard-ux-operability green. Unmapped custom quota keys now render as title-cased labels instead of raw snake_case. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!
…login (cheaper-inference#11143) (cheaper-inference#11175) Validated on the combined batch board + this branch: login-11143 green. Full document navigation after login guarantees the auth_token cookie is committed before any RSC prefetch fires — no more 307 back to /login. Conflict with the tip was only stale provider-count docs. Fixes cheaper-inference#11143. Thank you @rqzbeh!
…fication (cheaper-inference#10443) (cheaper-inference#11177) Validated on the combined batch board + this branch: antigravity-dynamic-session-id + proxy-fetch-dns-retry green; file-size gate green with the proxyFetch 1244 frozen entry (dated annotation for the +5 retry-classification lines, owner-authorized). Static per-account sessionId unpinning ends the concurrent-turn 429s and EmptyStreamError drops on the Hermes→Antigravity path; EAI_AGAIN/ENOTFOUND/ETIMEDOUT now classified retryable. Conflict with the tip was only stale provider-count docs. Resolves the remaining cheaper-inference#10443 root causes. Thank you @rqzbeh!
diegosouzapw
merged commit Aug 23, 2026
c92bd40
into
cheaper-inference:release/v3.8.50
4 of 7 checks passed
diegosouzapw
pushed a commit
that referenced
this pull request
Aug 25, 2026
…e2e assertion and 10 integration reds Electron Package Smoke — a packaging defect that had been hidden behind another packaging defect for nine days. Once the loginHeaderCapture fix let the main process start, the server underneath died on 'Cannot find module next': resources/app/server.js shipped without resources/app/node_modules. electron-builder discards the ROOT node_modules in code, not by configuration — app-builder-lib/out/util/filter.js:42 has a hard-coded `if (relative === "node_modules") return false` that runs before any filter pattern. The second extraResources entry pointing INTO ../.build/electron-standalone/node_modules is what sidesteps it, because those relative paths are never equal to "node_modules". #10325 removed that entry as an apparent duplicate and flipped the test to assert "exactly once", freezing the regression as if it were the contract. Restored, and the unit guard now pins both entries — proven by mutation: reverting package.json to the post-#10325 shape fails the guard 3/4, restoring it passes 4/4. group-b-quota-plans-config — the assertion was impossible to satisfy on ANY route, and the page was never broken. layout.tsx hands the whole message catalogue to NextIntlClientProvider, React serialises that prop into the RSC payload, and en.json carries "Internal Server Error" twice, so page.content() always contains it: probing /dashboard, /dashboard/costs, /dashboard/settings and /login showed the string present with every page rendering fine, and a pageerror probe on the failing run captured zero client exceptions. This is the same trap that killed the sibling not.toContain("500") in fc77100 ("raw HTML is unreliable") — that one was removed, this one was kept. Now asserts on rendered text, which still catches a real error boundary. The pageerror capture stays: the CI failure carried no stack trace, which is why it was misread twice. Integration — 10 of the 14 shard-2 reds, all sibling-test gaps behind security fixes: monitoring health now takes a Request and requires management auth (GHSA-mvf8-qc78-5mxm); the OAuth import routes moved to requireManagementAuth (GHSA-mg76) — the test accepts both guard shapes and gained a stronger anchor that every exported handler awaits a guard on its own request, mutation-verified; skill tool names are derived from encodeSkillToolName() and the fake upstream now returns the encoded name so decodeSkillToolName() is exercised too; previous_response_id now fails closed (#10262); proxy_logs persist as an async batch (#11182) so the test flushes first; providerQuotaOverrides joined GET /api/resilience (#9871); the reasoning fixture used a model that stopped being thinking-incompatible, replaced and pinned with a premise assert so it cannot rot silently again. A vacuous assert.ok(true, "all 10 streams completed without hanging") was replaced with real anchors — content must arrive on every stream and the active Timeout count must not grow. Four are deliberately left red rather than aligned, each now tracked: #11551 (the /v1/models after() wiring is dead — the route passes a third argument to a two-parameter function and catalogCache never imports after, so the #8728 contract is unimplemented), #11552 (~27% of requests emit an extra discarded upstream call; the delivered distribution is exactly 0.70, so weighted routing is correct and the waste is the real finding), the fixed-account combo pin (aligning it would destroy the per-step attribution the test exists for), and the web_search fallback already tracked as #11524. Package Artifact — the provenance stamp I added last round used git rev-parse HEAD, which under pull_request is the ephemeral merge commit and therefore never an ancestor of the release branch. Now takes the PR head sha. Refs #10692
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ormance optimizations (A, B, C, D) (cheaper-inference#11182) Validated on the combined batch board + this branch: perf-a-b-c-d + account-fallback-service 93/93, gates + typecheck clean. Owner approved the full bundle including item A (async proxy-log batching, 1s/100-item flush with an unref'd timer — reviewed the implementation: flush helper exists for shutdown wiring, buffered logs are the accepted tradeoff). B (lazy modals), C (O(1) alias maps), D (pre-compiled regex — this is also the entire content of cheaper-inference#11187, being closed as subsumed) ride along. Conflict with the tip was only stale provider-count docs. Thank you @rqzbeh!
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…e2e assertion and 10 integration reds Electron Package Smoke — a packaging defect that had been hidden behind another packaging defect for nine days. Once the loginHeaderCapture fix let the main process start, the server underneath died on 'Cannot find module next': resources/app/server.js shipped without resources/app/node_modules. electron-builder discards the ROOT node_modules in code, not by configuration — app-builder-lib/out/util/filter.js:42 has a hard-coded `if (relative === "node_modules") return false` that runs before any filter pattern. The second extraResources entry pointing INTO ../.build/electron-standalone/node_modules is what sidesteps it, because those relative paths are never equal to "node_modules". cheaper-inference#10325 removed that entry as an apparent duplicate and flipped the test to assert "exactly once", freezing the regression as if it were the contract. Restored, and the unit guard now pins both entries — proven by mutation: reverting package.json to the post-cheaper-inference#10325 shape fails the guard 3/4, restoring it passes 4/4. group-b-quota-plans-config — the assertion was impossible to satisfy on ANY route, and the page was never broken. layout.tsx hands the whole message catalogue to NextIntlClientProvider, React serialises that prop into the RSC payload, and en.json carries "Internal Server Error" twice, so page.content() always contains it: probing /dashboard, /dashboard/costs, /dashboard/settings and /login showed the string present with every page rendering fine, and a pageerror probe on the failing run captured zero client exceptions. This is the same trap that killed the sibling not.toContain("500") in feb0709 ("raw HTML is unreliable") — that one was removed, this one was kept. Now asserts on rendered text, which still catches a real error boundary. The pageerror capture stays: the CI failure carried no stack trace, which is why it was misread twice. Integration — 10 of the 14 shard-2 reds, all sibling-test gaps behind security fixes: monitoring health now takes a Request and requires management auth (GHSA-mvf8-qc78-5mxm); the OAuth import routes moved to requireManagementAuth (GHSA-mg76) — the test accepts both guard shapes and gained a stronger anchor that every exported handler awaits a guard on its own request, mutation-verified; skill tool names are derived from encodeSkillToolName() and the fake upstream now returns the encoded name so decodeSkillToolName() is exercised too; previous_response_id now fails closed (cheaper-inference#10262); proxy_logs persist as an async batch (cheaper-inference#11182) so the test flushes first; providerQuotaOverrides joined GET /api/resilience (cheaper-inference#9871); the reasoning fixture used a model that stopped being thinking-incompatible, replaced and pinned with a premise assert so it cannot rot silently again. A vacuous assert.ok(true, "all 10 streams completed without hanging") was replaced with real anchors — content must arrive on every stream and the active Timeout count must not grow. Four are deliberately left red rather than aligned, each now tracked: cheaper-inference#11551 (the /v1/models after() wiring is dead — the route passes a third argument to a two-parameter function and catalogCache never imports after, so the cheaper-inference#8728 contract is unimplemented), cheaper-inference#11552 (~27% of requests emit an extra discarded upstream call; the delivered distribution is exactly 0.70, so weighted routing is correct and the waste is the real finding), the fixed-account combo pin (aligning it would destroy the per-step attribution the test exists for), and the web_search fallback already tracked as cheaper-inference#11524. Package Artifact — the provenance stamp I added last round used git rev-parse HEAD, which under pull_request is the ephemeral merge commit and therefore never an ancestor of the release branch. Now takes the PR head sha. Refs cheaper-inference#10692
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements comprehensive core proxy performance & architecture optimizations (A, B, C, D):
A — Non-blocking Async Proxy Log Batching (
src/lib/proxyLogger.ts):QUOTA_STORE_DRIVER=redisorREDIS_URLis set.flushProxyLogsSync()flush helper.B — Next.js App Router Dynamic Imports (
src/app/(dashboard)/dashboard/providers/page.tsx):AddCompatibleProviderModal,ImportProvidersFromFileModal) vianext/dynamic(ssr: false), improving initial bundle size and page hydration TTI.C —$O(1)$ Fast Map Lookups (
open-sse/services/accountFallback.ts):canonicalProviderCache) forD — Pre-compiled Safe Regex Transforms (
open-sse/services/cloudCodeThinking.ts):PREFIX_TRIM_RE) to minimize per-request string allocations.Verification
node --import tsx/esm --test tests/unit/perf-a-b-c-d.test.ts(PASS)npm run typecheck:core(0 errors)npm run check:docs-all(PASS)