Skip to content

feat: add Cerebras provider support and update provider count to 12+ #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akshaydeo merged 1 commit into from
Aug 13, 2025
Merged

feat: add Cerebras provider support and update provider count to 12+ #267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

## The fastest way to build AI applications that never go down

Bifrost is a high-performance AI gateway that connects you to 10+ providers (OpenAI, Anthropic, Bedrock, and more) through a single API. Get automatic failover, load balancing, and zero-downtime deployments in under 30 seconds.
Bifrost is a high-performance AI gateway that connects you to 12+ providers (OpenAI, Anthropic, Bedrock, and more) through a single API. Get automatic failover, load balancing, and zero-downtime deployments in under 30 seconds.
Comment thread
Pratham-Mishra04 marked this conversation as resolved.

🚀 **Just launched:** Native MCP (Model Context Protocol) support for seamless tool integration
⚡ **Performance:** Adds only 11µs latency while handling 5,000+ RPS
Expand Down Expand Up @@ -75,6 +75,7 @@ Your AI gateway is now running with a beautiful web interface. You can:
## 📑 Table of Contents

- [Bifrost](#bifrost)
- [The fastest way to build AI applications that never go down](#the-fastest-way-to-build-ai-applications-that-never-go-down)
- [⚡ Quickstart (30 seconds)](#-quickstart-30-seconds)
- [Using Bifrost HTTP Transport](#using-bifrost-http-transport)
- [📑 Table of Contents](#-table-of-contents)
Expand Down Expand Up @@ -247,7 +248,7 @@ Choose higher settings (like the t3.xlarge profile above) for raw speed, or lowe
<details>
<summary><strong>🎯 I want to understand what Bifrost can do</strong></summary>

- **[🔗 Multi-Provider Support](./docs/usage/providers.md)** - Connect to 10+ AI providers with one API
- **[🔗 Multi-Provider Support](./docs/usage/providers.md)** - Connect to 12+ AI providers with one API
- **[🛡️ Fallback & Reliability](./docs/usage/providers.md#fallback-mechanisms)** - Never lose a request with automatic failover
- **[🛠️ MCP Tool Integration](./docs/usage/http-transport/configuration/mcp.md)** - Give your AI external capabilities
- **[🔌 Plugin Ecosystem](./docs/usage/http-transport/configuration/plugins.md)** - Extend Bifrost with custom middleware
Expand Down
2 changes: 1 addition & 1 deletion ci/npx/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@maximhq/bifrost",
"version": "1.0.4",
"description": "High-performance AI gateway CLI - connect to 10+ providers through a single API",
"description": "High-performance AI gateway CLI - connect to 12+ providers through a single API",

@coderabbitai coderabbitai Bot Aug 13, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Minor copy edit for the CLI description.

Stylistic: em dash and “via” read cleaner.

-  "description": "High-performance AI gateway CLI - connect to 12+ providers through a single API",
+  "description": "High-performance AI gateway CLI — connect to 12+ providers via a single API",

Optional: add “cerebras” to keywords for discoverability (no diff shown since it’s outside the changed line).

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"description": "High-performance AI gateway CLI - connect to 12+ providers through a single API",
"description": "High-performance AI gateway CLI — connect to 12+ providers via a single API",
🤖 Prompt for AI Agents 
In ci/npx/package.json around line 4, the CLI description should be copy-edited:
replace "connect to 12+ providers through a single API" with "connect to 12+
providers — via a single API" (use an em dash and "via" for cleaner style). Also
optionally add "cerebras" to the keywords array for discoverability by adding it
to the existing keyword list.

"keywords": ["ai", "gateway", "openai", "anthropic", "cli", "bifrost"],
"homepage": "https://github.com/maximhq/bifrost",
"repository": {
Expand Down
2 changes: 2 additions & 0 deletions core/bifrost.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -687,6 +687,8 @@ func (bifrost *Bifrost) createProviderFromProviderKey(providerKey schemas.ModelP
return providers.NewSGLProvider(config, bifrost.logger)
case schemas.Parasail:
return providers.NewParasailProvider(config, bifrost.logger)
case schemas.Cerebras:
return providers.NewCerebrasProvider(config, bifrost.logger)
default:
return nil, fmt.Errorf("unsupported provider: %s", providerKey)
}
Expand Down
54 changes: 29 additions & 25 deletions core/providers/azure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,26 +50,26 @@ var azureTextCompletionResponsePool = sync.Pool{
},
}

// azureChatResponsePool provides a pool for Azure chat response objects.
var azureChatResponsePool = sync.Pool{
New: func() interface{} {
return &schemas.BifrostResponse{}
},
}

// acquireAzureChatResponse gets an Azure chat response from the pool and resets it.
func acquireAzureChatResponse() *schemas.BifrostResponse {
resp := azureChatResponsePool.Get().(*schemas.BifrostResponse)
*resp = schemas.BifrostResponse{} // Reset the struct
return resp
}

// releaseAzureChatResponse returns an Azure chat response to the pool.
func releaseAzureChatResponse(resp *schemas.BifrostResponse) {
if resp != nil {
azureChatResponsePool.Put(resp)
}
}
// // azureChatResponsePool provides a pool for Azure chat response objects.
// var azureChatResponsePool = sync.Pool{
// New: func() interface{} {
// return &schemas.BifrostResponse{}
// },
// }

// // acquireAzureChatResponse gets an Azure chat response from the pool and resets it.
// func acquireAzureChatResponse() *schemas.BifrostResponse {
// resp := azureChatResponsePool.Get().(*schemas.BifrostResponse)
// *resp = schemas.BifrostResponse{} // Reset the struct
// return resp
// }

// // releaseAzureChatResponse returns an Azure chat response to the pool.
// func releaseAzureChatResponse(resp *schemas.BifrostResponse) {
// if resp != nil {
// azureChatResponsePool.Put(resp)
// }
// }

Comment on lines +53 to 73

@coderabbitai coderabbitai Bot Aug 13, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick (assertive)

Optional: Add a brief TODO above the commented chat pooling to document intent.

Consistent with other providers, a short note helps explain the temporary state.

-// // azureChatResponsePool provides a pool for Azure chat response objects.
+// TODO: Revisit Azure chat response pooling after safe lifetime management (e.g., ref-counting).
+// Disabled now to avoid correctness risks during migration away from pooling.
+// // azureChatResponsePool provides a pool for Azure chat response objects.
 // var azureChatResponsePool = sync.Pool{
 // 	New: func() interface{} {
 // 		return &schemas.BifrostResponse{}
 // 	},
 // }
 
 // // acquireAzureChatResponse gets an Azure chat response from the pool and resets it.
 // func acquireAzureChatResponse() *schemas.BifrostResponse {
 // 	resp := azureChatResponsePool.Get().(*schemas.BifrostResponse)
 // 	*resp = schemas.BifrostResponse{} // Reset the struct
 // 	return resp
 // }
 
 // // releaseAzureChatResponse returns an Azure chat response to the pool.
 // func releaseAzureChatResponse(resp *schemas.BifrostResponse) {
 // 	if resp != nil {
 // 		azureChatResponsePool.Put(resp)
 // 	}
 // }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// // azureChatResponsePool provides a pool for Azure chat response objects.
// var azureChatResponsePool = sync.Pool{
// New: func() interface{} {
// return &schemas.BifrostResponse{}
// },
// }
// // acquireAzureChatResponse gets an Azure chat response from the pool and resets it.
// func acquireAzureChatResponse() *schemas.BifrostResponse {
// resp := azureChatResponsePool.Get().(*schemas.BifrostResponse)
// *resp = schemas.BifrostResponse{} // Reset the struct
// return resp
// }
// // releaseAzureChatResponse returns an Azure chat response to the pool.
// func releaseAzureChatResponse(resp *schemas.BifrostResponse) {
// if resp != nil {
// azureChatResponsePool.Put(resp)
// }
// }
// TODO: Revisit Azure chat response pooling after safe lifetime management (e.g., ref-counting).
// Disabled now to avoid correctness risks during migration away from pooling.
// // azureChatResponsePool provides a pool for Azure chat response objects.
// var azureChatResponsePool = sync.Pool{
// New: func() interface{} {
// return &schemas.BifrostResponse{}
// },
// }
// // acquireAzureChatResponse gets an Azure chat response from the pool and resets it.
// func acquireAzureChatResponse() *schemas.BifrostResponse {
// resp := azureChatResponsePool.Get().(*schemas.BifrostResponse)
// *resp = schemas.BifrostResponse{} // Reset the struct
// return resp
// }
// // releaseAzureChatResponse returns an Azure chat response to the pool.
// func releaseAzureChatResponse(resp *schemas.BifrostResponse) {
// if resp != nil {
// azureChatResponsePool.Put(resp)
// }
// }
🤖 Prompt for AI Agents 
In core/providers/azure.go around lines 53 to 73, the commented-out chat
response pooling code lacks a brief TODO explaining why it’s commented and the
intended future behavior; add a single-line TODO comment above the block noting
that pooling is temporarily disabled, why (e.g., stability/bug/perf reason), and
what should be done to re-enable it (e.g., revisit thread-safety or response
struct changes), keeping the note short and consistent with other providers'
TODO style.

// acquireAzureTextResponse gets an Azure text completion response from the pool and resets it.
func acquireAzureTextResponse() *AzureTextResponse {
Expand Down Expand Up @@ -113,7 +113,7 @@ func NewAzureProvider(config *schemas.ProviderConfig, logger schemas.Logger) (*A

// Pre-warm response pools
for range config.ConcurrencyAndBufferSize.Concurrency {
azureChatResponsePool.Put(&schemas.BifrostResponse{})
// azureChatResponsePool.Put(&schemas.BifrostResponse{})
azureTextCompletionResponsePool.Put(&AzureTextResponse{})

}
Comment thread
Pratham-Mishra04 marked this conversation as resolved.
Expand Down Expand Up @@ -308,8 +308,10 @@ func (provider *AzureProvider) ChatCompletion(ctx context.Context, model string,
}

// Create response object from pool
response := acquireAzureChatResponse()
defer releaseAzureChatResponse(response)
// response := acquireAzureChatResponse()
// defer releaseAzureChatResponse(response)

response := &schemas.BifrostResponse{}

Comment thread
Pratham-Mishra04 marked this conversation as resolved.
rawResponse, bifrostErr := handleProviderResponse(responseBody, response, provider.sendBackRawResponse)
if bifrostErr != nil {
Expand Down Expand Up @@ -359,8 +361,10 @@ func (provider *AzureProvider) Embedding(ctx context.Context, model string, key
}

// Pre-allocate response structs from pools
response := acquireAzureChatResponse()
defer releaseAzureChatResponse(response)
// response := acquireAzureChatResponse()
// defer releaseAzureChatResponse(response)

response := &schemas.BifrostResponse{}

Comment thread
Pratham-Mishra04 marked this conversation as resolved.
// Use enhanced response handler with pre-allocated response
rawResponse, bifrostErr := handleProviderResponse(responseBody, response, provider.sendBackRawResponse)
Expand Down
Loading
Loading