Skip to content

feat(api): wire bifrost sidecar fast-path into /v1/chat/completions and /v1/messages with native fallback - #12953

Open
rqzbeh wants to merge 1 commit into
diegosouzapw:release/v3.8.52from
rqzbeh:feat/v3.8.51-bifrost-fastpath
Open

rqzbeh wants to merge 1 commit into
diegosouzapw:release/v3.8.52from
rqzbeh:feat/v3.8.51-bifrost-fastpath

Conversation

@rqzbeh

@rqzbeh rqzbeh commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR connects the supervised Go Bifrost sidecar (@maximhq/bifrost) directly into OmniRoute's primary high-traffic ingress endpoints (POST /v1/chat/completions and POST /v1/messages) based on release/v3.8.51.

  • Zero-Copy Fast-Path: When Bifrost is active and healthy on loopback (127.0.0.1:8080), eligible requests bypass the heavy V8 handler stack and stream tokens directly through Go's net/http connection pool.
  • Graceful Auto-Fallback: If Bifrost is offline, times out, or returns 5xx errors, requests automatically trip failure cooldowns and transparently fall through to native TypeScript handleChat() with X-Routing-Fallback headers attached.
  • Shared Subsystem: Consolidates Bifrost routing, circuit breaker cooldowns, and client dispatching into src/shared/services/bifrost/, deduplicating logic across the relay proxy and primary routes.
  • Test Suite: Added comprehensive unit and integration test suites covering unary dispatch, SSE streaming finalization, failure cooldowns, and fallback mechanics.

Verification

  • npm run typecheck:core: Passed (0 errors).
  • npm run check:route-guard-membership: Passed.
  • npm run check:deps: Passed (0 new dependencies).
  • node --import tsx/esm --test tests/unit/services/bifrost/*.test.ts tests/unit/api/v1/*bifrost*.test.ts: 22/22 tests passed (100% green).

⚠️ base-red inherited: #13866

@rqzbeh
rqzbeh requested a review from diegosouzapw as a code owner September 7, 2026 12:54
@rqzbeh
rqzbeh force-pushed the feat/v3.8.51-bifrost-fastpath branch from 32b43c1 to ec22686 Compare September 7, 2026 16:38
Raudbjorn added a commit to Raudbjorn/OmniRoute that referenced this pull request Sep 8, 2026
* feat(services): integrate LLMLingua and Bifrost from upstream PRs

Port diegosouzapw#12967 and diegosouzapw#12953 onto dev. Run Bifrost
inside the native executor pipeline to preserve authentication, policy,
guardrails and accounting, with cancellation and native fallback.
Replace the placeholder compression server with the existing ONNX worker
and expose the LLMLingua service lifecycle through the dashboard.

Co-authored-by: Rouzbeh <78313022+rqzbeh@users.noreply.github.com>

* fix(services): address PR 5 review findings

Wrap native fallback responses with copied headers instead of mutating
immutable fetch responses. Preserve the body stream, status and executor
metadata for both return shapes. Cover error and cooldown fallback with
real immutable responses for streaming and nonstreaming requests.

Remove the unused changelog fragment and retain upstream attribution in
the embedded-services guide. Explain the worker callback concurrency guard.

Review: #5
Base: Raudbjorn/OmniRoute feat/llmlingua-bifrost
793302f

Validation:
- New regression fails before the fix with TypeError: immutable.
- 26 focused Bifrost, response-header and LLMLingua installer tests pass.
- Focused ESLint, lint-staged --no-stash and check:any-budget:t11 pass.
- typecheck:core reports duplicate provider identifiers/properties;
  reproduced on the original PR head.
- check-docs-sync.mjs fails on removed CHANGELOG.md;
  check-tracked-artifacts.mjs flags two existing docs/superpowers files.
  Both reproduce on the original PR head.
- Pre-commit checks run manually with --no-stash; hook automation disabled
  for this commit because the unchanged baseline gates fail.
- Full suite, build and live provider/model behavior were not exercised.

* fix(llmlingua): recover missing host and handle unavailable worker

Remove obsolete changelog validation and untrack ignored planning artifacts so retained commit gates pass.

---------

Co-authored-by: Rouzbeh <78313022+rqzbeh@users.noreply.github.com>
@diegosouzapw

Copy link
Copy Markdown
Owner

This one's solid — real fetch-based dispatch with proper timeout, fallback to the native
handler on any 5xx/network error, and a cooldown so a downed sidecar doesn't get hammered.
Ran your 4 new/changed test files live against this branch's head: 12/12 pass. Two things
before merge: (1) this branch shares ~19 duplicated "fix(security)"/"fix(ci)" commits with
your other two PRs (#12967, #12962) that touch errorPathRedaction.ts — which has since
diverged on the release tip with a different fix — so a clean rebase dropping those is needed;
(2) dispatchToBifrost passes the sidecar's response straight through on any status < 500,
bypassing sanitizeErrorMessage() for 4xx bodies — worth confirming the bifrost binary never
puts internal paths/stack traces in an error body, or routing that through the same
sanitization as native errors.

@diegosouzapw diegosouzapw added the protected-surface Touches an agent-instruction surface (AGENTS/CLAUDE/llm.txt/SKILL.md) — per-PR operator OK to merge label Sep 15, 2026
diegosouzapw added a commit to rqzbeh/OmniRoute that referenced this pull request Sep 16, 2026
…mits

Merges the release tip into this branch and resolves every conflict by
taking the tip's version. All ~54 conflicting files (docs/i18n/*/llm.txt,
errorPathRedaction.ts helpers, chatCore.ts, inventory/redaction tests,
eslint-suppressions.json, etc.) carry generic security/docs/CI commits
that this branch duplicated from its sibling PRs (diegosouzapw#12967, diegosouzapw#12953) and
that have since diverged from the tip's own independent fix — none of
them touch sing-box code. This discards the duplicated generic changes
(equivalent to the requested clean rebase, done as a fast-forwardable
merge instead of a history rewrite) while keeping every sing-box-specific
file (installer, bootstrap.ts wiring, tproxy setup, tests, env docs)
untouched, since none of those files conflicted with the tip.

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@rqzbeh
rqzbeh force-pushed the feat/v3.8.51-bifrost-fastpath branch from 8cb9290 to ad0f871 Compare September 18, 2026 22:48
@rqzbeh

rqzbeh commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author
  1. Rebased cleanly onto release/v3.8.51 (4d1282be31), dropping all diverged commits from main and leaving open-sse/utils/errorPathRedaction.ts untouched so the diff contains only the 10 Bifrost domain files.
  2. In src/shared/services/bifrost/bifrostClient.ts, intercepted sidecar 4xx responses to run them through sanitizeErrorMessage() and buildErrorBody() per Hard Rule fix(ui): fix Select dropdown dark theme inconsistency #12 rather than passing them through verbatim, preventing path/credential leaks from third-party binary error payloads. Verified with unit test bifrostClient: dispatchToBifrost sanitizes a 4xx sidecar body instead of passing it through.

@rqzbeh
rqzbeh force-pushed the feat/v3.8.51-bifrost-fastpath branch from ad0f871 to c6d62cc Compare September 19, 2026 12:51
@diegosouzapw diegosouzapw added the deferred-v3.8.52 Grande demais / suspeito para o lote atual; precisa de sessão dedicada no ciclo v3.8.52 label Sep 25, 2026
@diegosouzapw diegosouzapw changed the title feat(api): wire bifrost sidecar fast-path into /v1/chat/completions and /v1/messages with native fallback [defer] feat(api): wire bifrost sidecar fast-path into /v1/chat/completions and /v1/messages with native fallback Sep 25, 2026
@diegosouzapw
diegosouzapw changed the base branch from release/v3.8.51 to release/v3.8.52 September 29, 2026 11:27
@diegosouzapw

Copy link
Copy Markdown
Owner

Re-homed to release/v3.8.52: v3.8.51 entered its release freeze, so the branch now belongs to the release captain and development continues on the next cycle. Nothing is wrong with this PR — it just needed a live base. No action needed from you; CI will re-run against the new base.

@diegosouzapw diegosouzapw removed the deferred-v3.8.52 Grande demais / suspeito para o lote atual; precisa de sessão dedicada no ciclo v3.8.52 label Oct 1, 2026
@diegosouzapw diegosouzapw changed the title [defer] feat(api): wire bifrost sidecar fast-path into /v1/chat/completions and /v1/messages with native fallback feat(api): wire bifrost sidecar fast-path into /v1/chat/completions and /v1/messages with native fallback Oct 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

protected-surface Touches an agent-instruction surface (AGENTS/CLAUDE/llm.txt/SKILL.md) — per-PR operator OK to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants