feat: routing info headers for streaming by TejasGhatte · Pull Request #5495 · maximhq/bifrost

TejasGhatte · 2026-07-23T11:20:21Z

Summary

Streaming responses in Bifrost write HTTP response headers before the first SSE chunk arrives, but routing identity (provider, model, key, fallback status) was only available on per-chunk ExtraFields. This meant routed-identity x-bifrost-* headers were missing entirely from streaming responses. This PR fixes that by stashing a RoutingInfo snapshot into the BifrostContext at stream setup time, then reading it in the transport layer to emit the correct headers before the first chunk is written.

Changes

Added BifrostContextKeyRoutingInfo as a new reserved context key. Core writes a RoutingInfo snapshot into the context at each stream attempt (overwritten on retry, so the winning attempt's snapshot survives). On successful fallback, the snapshot is updated to reflect IsFallback, PrimaryProvider, and PrimaryModel, mirroring the existing SetFallbackRoutingInfo logic.
Added RoutingInfo.ToExtraFields(requestType) to build a BifrostResponseExtraFields from a finalized RoutingInfo, using the same deprecated-triplet sync rules as the non-streaming response path.
Added ApplyBifrostStreamResponseHeaders in the transport lib, which reads the context snapshot and calls the existing ApplyBifrostResponseHeaders before any SSE write. When no snapshot is present (e.g. a plugin short-circuited the stream), only the request-type header is emitted.
All streaming handler entry points (handleStreamingTextCompletion, handleStreamingChatCompletion, handleStreamingResponses, handleStreamingSpeech, handleStreamingTranscriptionRequest, handleStreamingImageGeneration, handleStreamingImageEditRequest) now pass their RequestType through to handleStreamingResponse, which calls ApplyBifrostStreamResponseHeaders after the stream channel is obtained but before any SSE headers are flushed.
The generic router's handleStreamingRequest follows the same pattern, tracking requestType per branch and calling ApplyBifrostStreamResponseHeaders at the same point.
Tests cover the normal identity case, the fallback-layered case (including deprecated header derivation), and the missing-snapshot fallback.

Type of change

Affected areas

How to test

go test ./core/... ./transports/bifrost-http/...

To validate end-to-end: send a streaming chat completion request through the HTTP transport and inspect the response headers. Expect x-bifrost-routing-info-provider, x-bifrost-routing-info-model, x-bifrost-routing-info-key, and x-bifrost-request-type to be present on the response before any SSE data is received. For a fallback scenario, also expect x-bifrost-routing-info-is-fallback: true, x-bifrost-routing-info-primary-provider, and x-bifrost-routing-info-primary-model.

Screenshots/Recordings

N/A

Breaking changes

Yes
No

Related issues

N/A

Security considerations

The RoutingInfo snapshot stored in context contains provider key identifiers (alias names, not secret values). This is consistent with what was already emitted on non-streaming responses via ExtraFields.

Checklist

I read docs/contributing/README.md and followed the guidelines
I added/updated tests where appropriate
I updated documentation where needed
I verified builds succeed (Go and UI)
I verified the CI pipeline passes locally if applicable

CLAassistant · 2026-07-23T11:20:27Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

coderabbitai · 2026-07-23T11:20:27Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 64f32209-571a-4230-af38-d91a5384fbf6

📥 Commits

Reviewing files that changed from the base of the PR and between 91e2e15 and f551da7.

📒 Files selected for processing (7)

core/bifrost.go
core/schemas/bifrost.go
core/schemas/context.go
transports/bifrost-http/handlers/inference.go
transports/bifrost-http/integrations/router.go
transports/bifrost-http/lib/responseheaders.go
transports/bifrost-http/lib/responseheaders_test.go

🚧 Files skipped from review as they are similar to previous changes (7)

transports/bifrost-http/lib/responseheaders.go
core/schemas/bifrost.go
core/schemas/context.go
transports/bifrost-http/integrations/router.go
core/bifrost.go
transports/bifrost-http/handlers/inference.go
transports/bifrost-http/lib/responseheaders_test.go

📝 Walkthrough

Summary by CodeRabbit

Bug Fixes
- Streaming responses now include routing and provider identity headers before the first streamed content arrives.
- Fallback attempts accurately report fallback status and primary provider/model details.
- Routing headers are now consistent across text, chat, speech, transcription, image, and response streaming endpoints.
Tests
- Added coverage for streaming routing, fallback, and missing-routing scenarios.

Walkthrough

Streaming routing snapshots are added to request context, updated during fallback and retries, and used to apply routed identity headers before provider headers or SSE chunks across inference and image streaming endpoints.

Changes

Streaming routing header flow

Layer / File(s)	Summary
Routing context contract `core/schemas/bifrost.go`, `core/schemas/context.go`	Adds the reserved routing-info context key and converts `RoutingInfo` into response extra fields with synchronized legacy provider/model fields.
Per-attempt routing capture `core/bifrost.go`	Stores fallback and retry attempt routing information in the request context, leaving the active attempt snapshot available to transports.
Streaming header application `transports/bifrost-http/lib/*`, `transports/bifrost-http/handlers/inference.go`, `transports/bifrost-http/integrations/router.go`	Passes request types through streaming handlers and applies routed identity, fallback, and legacy headers before streamed data; tests cover snapshot, fallback, and missing-snapshot cases.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant handleStreamingRequest
  participant BifrostCore
  participant ApplyBifrostStreamResponseHeaders
  Client->>handleStreamingRequest: request streaming response
  handleStreamingRequest->>BifrostCore: start request or retry attempt
  BifrostCore->>handleStreamingRequest: return stream and RoutingInfo snapshot
  handleStreamingRequest->>ApplyBifrostStreamResponseHeaders: apply request-type and routing headers
  ApplyBifrostStreamResponseHeaders->>Client: response headers before chunks

Possibly related PRs

maximhq/bifrost#5475: Both changes modify shared streaming SSE handling in transports/bifrost-http/handlers/inference.go.
maximhq/bifrost#5485: Both changes extend routed identity response-header emission in the Bifrost HTTP transport.

Suggested reviewers: pratham-mishra04, roroghost17, akshaydeo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: streaming routing info headers.
Description check	✅ Passed	The PR description follows the template well and includes the required summary, changes, testing, impacts, and checklist sections.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch 07-23-feat_routing_info_headers_for_streaming

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

_{Comment @coderabbitai help to get the list of available commands.}

TejasGhatte · 2026-07-23T11:20:37Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@transports/bifrost-http/handlers/inference.go`:
- Around line 1936-1938: Move ApplyBifrostStreamResponseHeaders below the
provider-header forwarding block in both
transports/bifrost-http/handlers/inference.go (lines 1936-1938) and
transports/bifrost-http/integrations/router.go (lines 2609-2611), ensuring
routed-identity headers remain authoritative in both streaming paths.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a120ac19-7609-4752-aefc-8db3d19e46bc

📥 Commits

Reviewing files that changed from the base of the PR and between 03a862a and 0a42665.

📒 Files selected for processing (7)

core/bifrost.go
core/schemas/bifrost.go
core/schemas/context.go
transports/bifrost-http/handlers/inference.go
transports/bifrost-http/integrations/router.go
transports/bifrost-http/lib/responseheaders.go
transports/bifrost-http/lib/responseheaders_test.go

akshaydeo · 2026-07-23T15:53:41Z

Merge activity

Jul 23, 3:53 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Jul 23, 3:55 PM UTC: Graphite rebased this pull request as part of a merge.
Jul 23, 3:56 PM UTC: Graphite couldn't merge this PR because it was not satisfying all requirements (PR does not have required approvals).
Jul 23, 4:06 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Jul 23, 4:06 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.

## Summary Streaming responses in Bifrost write HTTP response headers before the first SSE chunk arrives, but routing identity (provider, model, key, fallback status) was only available on per-chunk `ExtraFields`. This meant routed-identity `x-bifrost-*` headers were missing entirely from streaming responses. This PR fixes that by stashing a `RoutingInfo` snapshot into the `BifrostContext` at stream setup time, then reading it in the transport layer to emit the correct headers before the first chunk is written. ## Changes - Added `BifrostContextKeyRoutingInfo` as a new reserved context key. Core writes a `RoutingInfo` snapshot into the context at each stream attempt (overwritten on retry, so the winning attempt's snapshot survives). On successful fallback, the snapshot is updated to reflect `IsFallback`, `PrimaryProvider`, and `PrimaryModel`, mirroring the existing `SetFallbackRoutingInfo` logic. - Added `RoutingInfo.ToExtraFields(requestType)` to build a `BifrostResponseExtraFields` from a finalized `RoutingInfo`, using the same deprecated-triplet sync rules as the non-streaming response path. - Added `ApplyBifrostStreamResponseHeaders` in the transport lib, which reads the context snapshot and calls the existing `ApplyBifrostResponseHeaders` before any SSE write. When no snapshot is present (e.g. a plugin short-circuited the stream), only the request-type header is emitted. - All streaming handler entry points (`handleStreamingTextCompletion`, `handleStreamingChatCompletion`, `handleStreamingResponses`, `handleStreamingSpeech`, `handleStreamingTranscriptionRequest`, `handleStreamingImageGeneration`, `handleStreamingImageEditRequest`) now pass their `RequestType` through to `handleStreamingResponse`, which calls `ApplyBifrostStreamResponseHeaders` after the stream channel is obtained but before any SSE headers are flushed. - The generic router's `handleStreamingRequest` follows the same pattern, tracking `requestType` per branch and calling `ApplyBifrostStreamResponseHeaders` at the same point. - Tests cover the normal identity case, the fallback-layered case (including deprecated header derivation), and the missing-snapshot fallback. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... ./transports/bifrost-http/... ``` To validate end-to-end: send a streaming chat completion request through the HTTP transport and inspect the response headers. Expect `x-bifrost-routing-info-provider`, `x-bifrost-routing-info-model`, `x-bifrost-routing-info-key`, and `x-bifrost-request-type` to be present on the response before any SSE data is received. For a fallback scenario, also expect `x-bifrost-routing-info-is-fallback: true`, `x-bifrost-routing-info-primary-provider`, and `x-bifrost-routing-info-primary-model`. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations The `RoutingInfo` snapshot stored in context contains provider key identifiers (alias names, not secret values). This is consistent with what was already emitted on non-streaming responses via `ExtraFields`. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable

TejasGhatte mentioned this pull request Jul 23, 2026

fix: send routing info headers for inference and integration apis #5485

Merged

18 tasks

TejasGhatte marked this pull request as ready for review July 23, 2026 11:25

coderabbitai Bot requested review from Pratham-Mishra04, akshaydeo and roroghost17 July 23, 2026 11:26

coderabbitai Bot requested changes Jul 23, 2026

View reviewed changes

Comment thread transports/bifrost-http/handlers/inference.go Outdated

TejasGhatte force-pushed the 07-23-fix_send_routing_info_headers_for_inference_and_integration_apis branch from 03a862a to 7fd4fce Compare July 23, 2026 11:54

TejasGhatte force-pushed the 07-23-feat_routing_info_headers_for_streaming branch from 0a42665 to 91e2e15 Compare July 23, 2026 11:54

coderabbitai Bot previously approved these changes Jul 23, 2026

View reviewed changes

TejasGhatte mentioned this pull request Jul 23, 2026

feat: responses retrieve stream method #5498

Merged

17 tasks

akshaydeo previously approved these changes Jul 23, 2026

View reviewed changes

akshaydeo changed the base branch from 07-23-fix_send_routing_info_headers_for_inference_and_integration_apis to graphite-base/5495 July 23, 2026 15:54

akshaydeo changed the base branch from graphite-base/5495 to dev July 23, 2026 15:54

akshaydeo force-pushed the 07-23-feat_routing_info_headers_for_streaming branch from 91e2e15 to 82795cf Compare July 23, 2026 15:54

feat: routing info headers for streaming

f551da7

TejasGhatte force-pushed the 07-23-feat_routing_info_headers_for_streaming branch from 82795cf to f551da7 Compare July 23, 2026 15:57

coderabbitai Bot requested a review from akshaydeo July 23, 2026 15:59

coderabbitai Bot approved these changes Jul 23, 2026

View reviewed changes

akshaydeo merged commit ac5bc9a into dev Jul 23, 2026
15 of 16 checks passed

akshaydeo deleted the 07-23-feat_routing_info_headers_for_streaming branch July 23, 2026 16:06

This was referenced Jul 24, 2026

fix: pass routing info headers in error/passthrough paths #5528

Merged

tests: harness, response headers validation #5519

Merged

fix: routing info headers on streaming fallbacks #5542

Merged

coderabbitai Bot mentioned this pull request Aug 5, 2026

moves sse hearbeats to a common structure to reuse #5850

Merged

10 tasks

Navigation Menu

feat: routing info headers for streaming - #5495

feat: routing info headers for streaming#5495
akshaydeo merged 1 commit into
devfrom
07-23-feat_routing_info_headers_for_streaming

TejasGhatte commented Jul 23, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Jul 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

TejasGhatte commented Jul 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

akshaydeo commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

TejasGhatte commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Type of change

Affected areas

How to test

Screenshots/Recordings

Breaking changes

Related issues

Security considerations

Checklist

Uh oh!

CLAassistant commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

TejasGhatte commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

akshaydeo commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TejasGhatte commented Jul 23, 2026 •

edited

Loading

CLAassistant commented Jul 23, 2026 •

edited

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

TejasGhatte commented Jul 23, 2026 •

edited

Loading

akshaydeo commented Jul 23, 2026 •

edited

Loading