diff --git a/README.md b/README.md index e227739c01..bd48a86f9c 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,179 @@ Bifrost is an open-source middleware that serves as a unified gateway to various AI model providers, enabling seamless integration and fallback mechanisms for your AI-powered applications. +## ⚡ Quickstart + +### Prerequisites + +- Go 1.23 or higher (not needed if using Docker) +- Access to at least one AI model provider (OpenAI, Anthropic, etc.) +- API keys for the providers you wish to use + +### A. Using Bifrost as an HTTP Server + +1. **Create `config.json`**: This file should contain your provider settings and API keys. + ```json + [ + "openai": { + "keys": [{ + "value": "env.OPENAI_API_KEY", + "models": ["gpt-4o-mini"], + "weight": 1.0 + }], + }, + ] + ``` +2. **Create `.env`**: Set up your environment variables here. + ```env + OPENAI_API_KEY=your_openai_api_key; + ``` +3. **Start the Bifrost HTTP Server**: + + You have two options to run the server, either using Go Binary or a Docker setup if go is not installed. + + #### i) Using Go Binary + + - Install the transport package: + ```bash + go install github.com/maximhq/bifrost/transports/bifrost-http@latest + ``` + - Run the server: + + - If it's in your PATH: + + ```bash + bifrost-http -config config.json -env .env -port 8080 -pool-size 300 + ``` + + - Otherwise: + + ```bash + ./bifrost-http -config config.json -env .env -port 8080 -pool-size 300 + ``` + + #### ii) OR Using Docker + + - Download the Dockerfile: + + ```bash + curl -L -o Dockerfile https://raw.githubusercontent.com/maximhq/bifrost/main/transports/Dockerfile + ``` + + - Build the Docker image: + + ```bash + docker build \ + --build-arg CONFIG_PATH=./config.example.json \ + --build-arg ENV_PATH=./.env.sample \ + --build-arg PORT=8080 \ + --build-arg POOL_SIZE=300 \ + -t bifrost-transports . + ``` + + - Run the Docker container: + ```bash + docker run -p 8080:8080 bifrost-transports + ``` + +4. **Using the API**: Once the server is running, you can send requests to the HTTP endpoints. + + ```bash + curl -X POST http://localhost:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "provider": "openai", + "model": "gpt-4o-mini", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Tell me about Bifrost in Norse mythology."} + ] + }' + ``` + +For additional configurations in HTTP server setup, please read [this](https://github.com/maximhq/bifrost/blob/main/transports/README.md). + +### B. Using Bifrost as a Go Package + +1. **Implement Your Account Interface**: You first need to create your account which follows [Bifrost's account interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/account.go). + + ```golang + type BaseAccount struct{} + + func (baseAccount *BaseAccount) GetConfiguredProviders() ([]schemas.ModelProvider, error) { + return []schemas.ModelProvider{schemas.OpenAI}, nil + } + + func (baseAccount *BaseAccount) GetKeysForProvider(providerKey schemas.ModelProvider) ([]schemas.Key, error) { + return []schemas.Key{ + { + Value: os.Getenv("OPENAI_API_KEY"), + Models: []string{"gpt-4o-mini"}, + }, + }, nil + } + + func (baseAccount *BaseAccount) GetConfigForProvider(providerKey schemas.ModelProvider) (*schemas.ProviderConfig, error) { + return &schemas.ProviderConfig{ + ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ + Concurrency: 3, + BufferSize: 10, + }, + }, nil + } + ``` + + Bifrost uses these methods to get all the keys and configurations it needs to call the providers. You can check the [Additional Configurations](#additional-configurations) section for further customizations. + +2. **Initialize Bifrost**: Set up the Bifrost instance by providing your account implementation. + + ```golang + account := BaseAccount{} + + client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &account, + }) + ``` + +3. **Use Bifrost**: Make your First LLM Call! + + ```golang + bifrostResult, bifrostErr := bifrost.ChatCompletionRequest( + schemas.OpenAI, &schemas.BifrostRequest{ + Model: "gpt-4o-mini", // make sure you have configured gpt-4o-mini in your account interface + Input: schemas.RequestInput{ + ChatCompletionInput: bifrost.Ptr([]schemas.Message{{ + Role: schemas.RoleUser, + Content: bifrost.Ptr("What is a LLM gateway?"), + }}), + }, + }, context.Background(), + ) + ``` + + you can add model parameters by passing them in `Params: &schemas.ModelParameters{...yourParams}` in ChatCompletionRequest. + ## 📑 Table of Contents - [Bifrost](#bifrost) + - [⚡ Quickstart](#-quickstart) + - [Prerequisites](#prerequisites) + - [A. Using Bifrost as an HTTP Server](#a-using-bifrost-as-an-http-server) + - [i) Using Go Binary](#i-using-go-binary) + - [ii) OR Using Docker](#ii-or-using-docker) + - [B. Using Bifrost as a Go Package](#b-using-bifrost-as-a-go-package) - [📑 Table of Contents](#-table-of-contents) - [🔍 Overview](#-overview) - [✨ Features](#-features) - [🏗️ Repository Structure](#️-repository-structure) + - [🚀 Getting Started](#-getting-started) + - [Package Structure](#package-structure) + - [Additional Configurations](#additional-configurations) - [📊 Benchmarks](#-benchmarks) - [Test Environment](#test-environment) - [t3.medium Instance](#t3medium-instance) - [t3.xlarge Instance](#t3xlarge-instance) - [Performance Metrics](#performance-metrics) - [Key Performance Highlights](#key-performance-highlights) - - [🚀 Getting Started](#-getting-started) - - [Package Structure](#package-structure) - - [Prerequisites](#prerequisites) - - [Setting up Bifrost](#setting-up-bifrost) - - [Additional Configurations](#additional-configurations) - [🤝 Contributing](#-contributing) - [📄 License](#-license) @@ -62,9 +217,9 @@ bifrost/ │ ├── schemas/ # Interfaces and structs used in bifrost │ ├── tests/ # Tests to make sure everything is in place │ ├── bifrost.go # Main Bifrost implementation -│ +│ ├── transports/ # Interface layers (HTTP, gRPC, etc.) -│ ├── http/ # HTTP transport implementation +│ ├── bifrost-http/ # HTTP transport implementation │ └── ... │ └── plugins/ # Plugin Implementations @@ -76,88 +231,6 @@ The system uses a provider-agnostic approach with well-defined interfaces to eas --- -## 📊 Benchmarks - -Bifrost has been tested under high load conditions to ensure optimal performance. The following results were obtained from benchmark tests running at 5000 requests per second (RPS) on different AWS EC2 instances, with Bifrost running inside Docker containers. - -### Test Environment - -#### t3.medium Instance -- **Instance**: AWS EC2 t3.medium -- **vCPUs**: 2 -- **Memory**: 4GB RAM -- **Container**: Docker container with resource limits matching instance specs -- **Bifrost Configurations**: - - Buffer Size: 15,000 - - Initial Pool Size: 10,000 - -#### t3.xlarge Instance -- **Instance**: AWS EC2 t3.xlarge -- **vCPUs**: 4 -- **Memory**: 16GB RAM -- **Container**: Docker container with resource limits matching instance specs -- **Bifrost Configurations**: - - Buffer Size: 20,000 - - Initial Pool Size: 15,000 - -### Performance Metrics - -| Metric | t3.medium | t3.xlarge | -|--------|-----------|-----------| -| Success Rate | 100.00% | 100.00% | -| Average Request Size | 0.13 KB | 0.13 KB | -| **Average Response Size** | **`1.37 KB`** | **`10.32 KB`** | -| Average Latency | 2.12s | 1.61s | -| Peak Memory Usage | 1312.79 MB | 3340.44 MB | -| Queue Wait Time | 47.13 µs | 1.67 µs | -| Key Selection Time | 16 ns | 10 ns | -| Message Formatting | 2.19 µs | 2.11 µs | -| Params Preparation | 436 ns | 417 ns | -| Request Body Preparation | 2.65 µs | 2.36 µs | -| JSON Marshaling | 63.47 µs | 26.80 µs | -| Request Setup | 6.59 µs | 7.17 µs | -| HTTP Request | 1.56s | 1.50s | -| Error Handling | 189 ns | 162 ns | -| Response Parsing | 11.30 ms | 2.11 ms | - -### Key Performance Highlights - -- **Perfect Success Rate**: 100% request success rate under high load on both instances -- **Efficient Queue Management**: Minimal queue wait time (1.67 µs on t3.xlarge) -- **Fast Key Selection**: Near-instantaneous key selection (10 ns on t3.xlarge) -- **Optimized Memory Usage**: - - t3.medium: ~1.3GB at 5000 RPS - - t3.xlarge: ~3.3GB at 5000 RPS -- **Efficient Request Processing**: Most operations complete in microseconds -- **Network Efficiency**: - - Consistent small request sizes (0.13 KB) across instances - - Larger response sizes on t3.xlarge (10.32 KB vs 1.37 KB) due to more detailed responses -- **Improved Performance on t3.xlarge**: - - 24% faster average latency - - 81% faster response parsing - - 58% faster JSON marshaling - - Significantly reduced queue wait times - - Higher buffer and pool sizes enabled by increased resources - -These benchmarks demonstrate Bifrost's ability to handle high-throughput scenarios while maintaining reliability and performance, even when containerized. The t3.xlarge instance shows improved performance across most metrics, particularly in processing times and latency, while maintaining the same high reliability and success rate. The larger response sizes on t3.xlarge indicate its ability to handle more detailed responses without compromising performance. - -One of Bifrost's key strengths is its flexibility in configuration. You can freely decide the tradeoff between memory usage and processing speed by adjusting Bifrost's configurations: - -- **Memory vs Speed Tradeoff**: - - Higher buffer and pool sizes (like in t3.xlarge) improve speed but use more memory - - Lower configurations (like in t3.medium) use less memory but may have slightly higher latencies - - You can fine-tune these parameters based on your specific needs and available resources - -- **Customizable Parameters**: - - Buffer Size: Controls the maximum number of concurrent requests - - Initial Pool Size: Determines the initial allocation of resources - - Concurrency Settings: Adjustable per provider - - Retry and Timeout Configurations: Customizable based on your requirements - -This flexibility allows you to optimize Bifrost for your specific use case, whether you prioritize speed, memory efficiency, or a balance between the two. - ---- - ## 🚀 Getting Started If you want to **set up the Bifrost API quickly**, [check the transports documentation](https://github.com/maximhq/bifrost/tree/main/transports/README.md). @@ -167,108 +240,28 @@ If you want to **set up the Bifrost API quickly**, [check the transports documen Bifrost is divided into three Go packages: core, plugins, and transports. 1. **core**: This package contains the core implementation of Bifrost as a Go package. - 2. **plugins**: This package serves as an extension to core. You can download this package using `go get github.com/maximhq/bifrost/plugins` and pass the plugins while initializing Bifrost. - - ```golang - plugin, err := plugins.NewMaximLoggerPlugin(os.Getenv("MAXIM_API_KEY"), os.Getenv("MAXIM_LOGGER_ID")) - if err != nil { - return nil, err - } - - // Initialize Bifrost - client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &account, - Plugins: []schemas.Plugin{plugin}, - }) - ``` - -3. **transports**: This package contains transport clients like HTTP to expose your Bifrost client. You can either `go get` this package or directly use the independent Dockerfile to quickly spin up your Bifrost API interface ([Click here](https://github.com/maximhq/bifrost/tree/main/transports/README.md) to read more on this). - -### Prerequisites - -- Go 1.23 or higher -- Access to at least one AI model provider (OpenAI, Anthropic, etc.) -- API keys for the providers you wish to use - -### Setting up Bifrost - -1. Setting up your account: You first need to create your account which follows [Bifrost's account interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/account.go). - -Example: - ```golang - type BaseAccount struct{} - func (baseAccount *BaseAccount) GetConfiguredProviders() ([]schemas.ModelProvider, error) { - return []schemas.ModelProvider{schemas.OpenAI}, nil - } - - func (baseAccount *BaseAccount) GetKeysForProvider(providerKey schemas.ModelProvider) ([]schemas.Key, error) { - switch providerKey { - case schemas.OpenAI: - return []schemas.Key{ - { - Value: os.Getenv("OPENAI_API_KEY"), - Models: []string{"gpt-4o-mini"}, - }, - }, nil - default: - return nil, fmt.Errorf("unsupported provider: %s", providerKey) - } - } - - func (baseAccount *BaseAccount) GetConfigForProvider(providerKey schemas.ModelProvider) (*schemas.ProviderConfig, error) { - switch providerKey { - case schemas.OpenAI: - return &schemas.ProviderConfig{ - ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ - Concurrency: 3, - BufferSize: 10, - }, - }, nil - default: - return nil, fmt.Errorf("unsupported provider: %s", providerKey) - } - } - ``` - -Bifrost uses these methods to get all the keys and configurations it needs to call the providers. You can check the [Additional Configurations](#additional-configurations) section for further customizations. - -2. Get bifrost core package: Simply run `go get github.com/maximhq/bifrost/core` to download bifrost/core package. - -3. Initialising Bifrost: Initialise bifrost by providing your account implementation +```golang +plugin, err := plugins.NewMaximLoggerPlugin(os.Getenv("MAXIM_API_KEY"), os.Getenv("MAXIM_LOGGER_ID")) +if err != nil { + return nil, err +} -```golang +// Initialize Bifrost client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &yourAccount, + Account: &account, + Plugins: []schemas.Plugin{plugin}, }) ``` -4. Make your First LLM Call! - -```golang - msg = "What is a LLM gateway?" - messages := []schemas.Message{ - { Role: schemas.RoleUser, Content: &msg }, - } - - bifrostResult, bifrostErr := bifrost.ChatCompletionRequest( - schemas.OpenAI, &schemas.BifrostRequest{ - Model: "gpt-4o", // make sure you have configured gpt-4o in your account interface - Input: schemas.RequestInput{ - ChatCompletionInput: &messages, - }, - }, context.Background() - ) -``` - -you can add model parameters by passing them in `Params:&schemas.ModelParameters{...yourParams}` ChatCompletionRequest. +3. **transports**: This package contains transport clients like HTTP to expose your Bifrost client. You can either `go get` this package or directly use the independent Dockerfile to quickly spin up your Bifrost API interface ([Click here](https://github.com/maximhq/bifrost/tree/main/transports/README.md) to read more on this). ### Additional Configurations 1. InitalPoolSize and DropExcessRequests: You can customise the initial pool size of the structs and channels bifrost creates on `bifrost.Init()`. A higher value would mean lesser run time allocations and lower latency but at the cost of more memory usage. Takes the defined default value if not provided. -```golang +```golang client, err := bifrost.Init(schemas.BifrostConfig{ Account: &yourAccount, InitialPoolSize: 500, @@ -280,16 +273,16 @@ When `DropExcessRequests` is set to true, in cases where the queue is full, requ 2. Logger: Like account interface, bifrost also allows you to pass your custom logger if it follows [bifrost's logger interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/logger.go). Takes in the [default logger](https://github.com/maximhq/bifrost/blob/main/core/logger.go) if not provided. -```golang +```golang client, err := bifrost.Init(schemas.BifrostConfig{ Account: &yourAccount, Logger: &yourLogger, }) ``` -The default logger is set to level info by default. If you wish to use it but with a different log level, you can do it like this - +The default logger is set to level info by default. If you wish to use it but with a different log level, you can do it like this - -```golang +```golang client, err := bifrost.Init(schemas.BifrostConfig{ Account: &yourAccount, Logger: bifrost.NewDefaultLogger(schemas.LogLevelDebug), @@ -298,7 +291,7 @@ The default logger is set to level info by default. If you wish to use it but wi 3. Plugins: You can create and pass your custom pre-hook and post-hook plugins to bifrost as long as they follow [bifrost's plugin interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/plugin.go). -```golang +```golang client, err := bifrost.Init(schemas.BifrostConfig{ Account: &yourAccount, Plugins: []schemas.Plugin{yourPlugin1, yourPlugin2, ...}, @@ -308,6 +301,7 @@ The default logger is set to level info by default. If you wish to use it but wi 4. Customise your provider settings: You can customise proxy config, timeouts, retry settings, concurrency buffer sizes for each of your provider in your account interface's GetConfigForProvider() method. exmaple: + ```golang schemas.ProviderConfig{ NetworkConfig: schemas.NetworkConfig{ @@ -362,7 +356,7 @@ You can check [this](https://github.com/maximhq/bifrost/blob/main/core/tests/acc ```golang result, err := bifrost.ChatCompletionRequest( schemas.OpenAI, &schemas.BifrostRequest{ - Model: "gpt-4o", + Model: "gpt-4o-mini", Input: schemas.RequestInput{ ChatCompletionInput: &messages, }, @@ -378,6 +372,89 @@ You can check [this](https://github.com/maximhq/bifrost/blob/main/core/tests/acc --- +## 📊 Benchmarks + +Bifrost has been tested under high load conditions to ensure optimal performance. The following results were obtained from benchmark tests running at 5000 requests per second (RPS) on different AWS EC2 instances, with Bifrost running inside Docker containers. + +### Test Environment + +#### t3.medium Instance + +- **Instance**: AWS EC2 t3.medium +- **vCPUs**: 2 +- **Memory**: 4GB RAM +- **Container**: Docker container with resource limits matching instance specs +- **Bifrost Configurations**: + - Buffer Size: 15,000 + - Initial Pool Size: 10,000 + +#### t3.xlarge Instance + +- **Instance**: AWS EC2 t3.xlarge +- **vCPUs**: 4 +- **Memory**: 16GB RAM +- **Container**: Docker container with resource limits matching instance specs +- **Bifrost Configurations**: + - Buffer Size: 20,000 + - Initial Pool Size: 15,000 + +### Performance Metrics + +| Metric | t3.medium | t3.xlarge | +| ------------------------- | ------------- | -------------- | +| Success Rate | 100.00% | 100.00% | +| Average Request Size | 0.13 KB | 0.13 KB | +| **Average Response Size** | **`1.37 KB`** | **`10.32 KB`** | +| Average Latency | 2.12s | 1.61s | +| Peak Memory Usage | 1312.79 MB | 3340.44 MB | +| Queue Wait Time | 47.13 µs | 1.67 µs | +| Key Selection Time | 16 ns | 10 ns | +| Message Formatting | 2.19 µs | 2.11 µs | +| Params Preparation | 436 ns | 417 ns | +| Request Body Preparation | 2.65 µs | 2.36 µs | +| JSON Marshaling | 63.47 µs | 26.80 µs | +| Request Setup | 6.59 µs | 7.17 µs | +| HTTP Request | 1.56s | 1.50s | +| Error Handling | 189 ns | 162 ns | +| Response Parsing | 11.30 ms | 2.11 ms | + +### Key Performance Highlights + +- **Perfect Success Rate**: 100% request success rate under high load on both instances +- **Efficient Queue Management**: Minimal queue wait time (1.67 µs on t3.xlarge) +- **Fast Key Selection**: Near-instantaneous key selection (10 ns on t3.xlarge) +- **Optimized Memory Usage**: + - t3.medium: ~1.3GB at 5000 RPS + - t3.xlarge: ~3.3GB at 5000 RPS +- **Efficient Request Processing**: Most operations complete in microseconds +- **Network Efficiency**: + - Consistent small request sizes (0.13 KB) across instances + - Larger response sizes on t3.xlarge (10.32 KB vs 1.37 KB) due to more detailed responses +- **Improved Performance on t3.xlarge**: + - 24% faster average latency + - 81% faster response parsing + - 58% faster JSON marshaling + - Significantly reduced queue wait times + - Higher buffer and pool sizes enabled by increased resources + +One of Bifrost's key strengths is its flexibility in configuration. You can freely decide the tradeoff between memory usage and processing speed by adjusting Bifrost's configurations: + +- **Memory vs Speed Tradeoff**: + + - Higher buffer and pool sizes (like in t3.xlarge) improve speed but use more memory + - Lower configurations (like in t3.medium) use less memory but may have slightly higher latencies + - You can fine-tune these parameters based on your specific needs and available resources + +- **Customizable Parameters**: + - Buffer Size: Controls the maximum number of concurrent requests + - Initial Pool Size: Determines the initial allocation of resources + - Concurrency Settings: Adjustable per provider + - Retry and Timeout Configurations: Customizable based on your requirements + +This flexibility allows you to optimize Bifrost for your specific use case, whether you prioritize speed, memory efficiency, or a balance between the two. + +--- + ## 🤝 Contributing Contributions are welcome! We welcome all kinds of contributions — bug fixes, features, docs, and ideas. Please feel free to submit a Pull Request. diff --git a/core/go.mod b/core/go.mod index af649c745f..b35f8bf61b 100644 --- a/core/go.mod +++ b/core/go.mod @@ -7,6 +7,7 @@ require github.com/joho/godotenv v1.5.1 require ( github.com/aws/aws-sdk-go-v2 v1.36.3 github.com/aws/aws-sdk-go-v2/config v1.29.14 + github.com/goccy/go-json v0.10.5 github.com/maximhq/bifrost/plugins v1.0.0 github.com/valyala/fasthttp v1.60.0 ) @@ -24,7 +25,6 @@ require ( github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.33.19 // indirect github.com/aws/smithy-go v1.22.3 // indirect - github.com/goccy/go-json v0.10.5 // indirect github.com/klauspost/compress v1.18.0 // indirect github.com/maximhq/maxim-go v0.1.1 // indirect github.com/valyala/bytebufferpool v1.0.0 // indirect diff --git a/core/tests/account.go b/core/tests/account.go index 53278b7ce3..b5a86a57be 100644 --- a/core/tests/account.go +++ b/core/tests/account.go @@ -8,6 +8,7 @@ import ( "os" "time" + bifrost "github.com/maximhq/bifrost/core" schemas "github.com/maximhq/bifrost/core/schemas" "github.com/maximhq/bifrost/core/schemas/meta" ) @@ -156,7 +157,7 @@ func (baseAccount *BaseAccount) GetConfigForProvider(providerKey schemas.ModelPr }, MetaConfig: &meta.BedrockMetaConfig{ SecretAccessKey: os.Getenv("BEDROCK_ACCESS_KEY"), - Region: StrPtr("us-east-1"), + Region: bifrost.Ptr("us-east-1"), }, ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ Concurrency: 3, @@ -189,7 +190,7 @@ func (baseAccount *BaseAccount) GetConfigForProvider(providerKey schemas.ModelPr Deployments: map[string]string{ "gpt-4o": "gpt-4o-aug", }, - APIVersion: StrPtr("2024-08-01-preview"), + APIVersion: bifrost.Ptr("2024-08-01-preview"), }, ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ Concurrency: 3, diff --git a/core/tests/setup.go b/core/tests/setup.go index 9af47664f3..0059685ec6 100644 --- a/core/tests/setup.go +++ b/core/tests/setup.go @@ -100,7 +100,3 @@ func getBifrost() (*bifrost.Bifrost, error) { return b, nil } - -func StrPtr(s string) *string { - return &s -} diff --git a/core/tests/tests.go b/core/tests/tests.go index 69b62c7f97..183af51e2c 100644 --- a/core/tests/tests.go +++ b/core/tests/tests.go @@ -82,7 +82,7 @@ var WeatherToolParams = schemas.ModelParameters{ // - bifrost: The Bifrost instance to use for the request // - config: Test configuration containing model and parameters // - ctx: Context for the request -func setupTextCompletionRequest(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Context) { +func setupTextCompletionRequest(bifrostClient *bifrost.Bifrost, config TestConfig, ctx context.Context) { text := "Hello world!" if config.CustomTextCompletion != nil { text = *config.CustomTextCompletion @@ -94,7 +94,7 @@ func setupTextCompletionRequest(bifrost *bifrost.Bifrost, config TestConfig, ctx } go func() { - result, err := bifrost.TextCompletionRequest(config.Provider, &schemas.BifrostRequest{ + result, err := bifrostClient.TextCompletionRequest(config.Provider, &schemas.BifrostRequest{ Model: config.TextModel, Input: schemas.RequestInput{ TextCompletionInput: &text, @@ -117,7 +117,7 @@ func setupTextCompletionRequest(bifrost *bifrost.Bifrost, config TestConfig, ctx // - bifrost: The Bifrost instance to use for the requests // - config: Test configuration containing model and parameters // - ctx: Context for the requests -func setupChatCompletionRequests(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Context) { +func setupChatCompletionRequests(bifrostClient *bifrost.Bifrost, config TestConfig, ctx context.Context) { messages := config.Messages if len(messages) == 0 { messages = CommonTestMessages @@ -138,7 +138,7 @@ func setupChatCompletionRequests(bifrost *bifrost.Bifrost, config TestConfig, ct Content: &msg, }, } - result, err := bifrost.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ + result, err := bifrostClient.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ Model: config.ChatModel, Input: schemas.RequestInput{ ChatCompletionInput: &messages, @@ -162,7 +162,7 @@ func setupChatCompletionRequests(bifrost *bifrost.Bifrost, config TestConfig, ct // - bifrost: The Bifrost instance to use for the requests // - config: Test configuration containing model and parameters // - ctx: Context for the requests -func setupImageTests(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Context) { +func setupImageTests(bifrostClient *bifrost.Bifrost, config TestConfig, ctx context.Context) { params := schemas.ModelParameters{} if config.CustomParams != nil { params = *config.CustomParams @@ -172,20 +172,20 @@ func setupImageTests(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Co urlImageMessages := []schemas.Message{ { Role: schemas.RoleUser, - Content: StrPtr("What is Happening in this picture?"), + Content: bifrost.Ptr("What is Happening in this picture?"), ImageContent: &schemas.ImageContent{ - Type: StrPtr("url"), + Type: bifrost.Ptr("url"), URL: "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg", }, }, } if config.Provider == schemas.Anthropic { - urlImageMessages[0].ImageContent.Type = StrPtr("url") + urlImageMessages[0].ImageContent.Type = bifrost.Ptr("url") } go func() { - result, err := bifrost.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ + result, err := bifrostClient.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ Model: config.ChatModel, Input: schemas.RequestInput{ ChatCompletionInput: &urlImageMessages, @@ -205,17 +205,17 @@ func setupImageTests(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Co base64ImageMessages := []schemas.Message{ { Role: schemas.RoleUser, - Content: StrPtr("What is this image about?"), + Content: bifrost.Ptr("What is this image about?"), ImageContent: &schemas.ImageContent{ - Type: StrPtr("base64"), + Type: bifrost.Ptr("base64"), URL: "/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAAIAAoDASIAAhEBAxEB/8QAFQABAQAAAAAAAAAAAAAAAAAAAAb/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/8QAFQEBAQAAAAAAAAAAAAAAAAAAAAX/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIRAxEAPwCdABmX/9k=", - MediaType: StrPtr("image/jpeg"), + MediaType: bifrost.Ptr("image/jpeg"), }, }, } go func() { - result, err := bifrost.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ + result, err := bifrostClient.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ Model: config.ChatModel, Input: schemas.RequestInput{ ChatCompletionInput: &base64ImageMessages, @@ -239,7 +239,7 @@ func setupImageTests(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Co // - bifrost: The Bifrost instance to use for the requests // - config: Test configuration containing model and parameters // - ctx: Context for the requests -func setupToolCalls(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Context) { +func setupToolCalls(bifrostClient *bifrost.Bifrost, config TestConfig, ctx context.Context) { messages := []string{"What's the weather like in Mumbai?"} params := WeatherToolParams @@ -263,7 +263,7 @@ func setupToolCalls(bifrost *bifrost.Bifrost, config TestConfig, ctx context.Con Content: &msg, }, } - result, err := bifrost.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ + result, err := bifrostClient.ChatCompletionRequest(config.Provider, &schemas.BifrostRequest{ Model: config.ChatModel, Input: schemas.RequestInput{ ChatCompletionInput: &messages, diff --git a/core/utils.go b/core/utils.go new file mode 100644 index 0000000000..be284fbddf --- /dev/null +++ b/core/utils.go @@ -0,0 +1,5 @@ +package bifrost + +func Ptr[T any](v T) *T { + return &v +} diff --git a/transports/Dockerfile b/transports/Dockerfile index df9ac9901c..51e57f9012 100644 --- a/transports/Dockerfile +++ b/transports/Dockerfile @@ -12,7 +12,7 @@ ARG TRANSPORT_TYPE=http # Initialize Go module and fetch the bifrost transport package RUN go mod init bifrost-transports && \ - go get github.com/maximhq/bifrost/transports/${TRANSPORT_TYPE}@latest + go get github.com/maximhq/bifrost/transports/bifrost-${TRANSPORT_TYPE}@latest # Build the binary from the fetched package with static linking RUN go build -ldflags="-w -s" -o /app/main github.com/maximhq/bifrost/transports/${TRANSPORT_TYPE} && \ diff --git a/transports/README.md b/transports/README.md index 0f0670a03b..b6ce6c84d9 100644 --- a/transports/README.md +++ b/transports/README.md @@ -22,6 +22,7 @@ This package contains clients for various transports that can be used to spin up ## 🚀 Setting Up Transports ### Prerequisites + - Go 1.23 or higher (if not using Docker) - Access to at least one AI model provider (OpenAI, Anthropic, etc.) - API keys for the providers you wish to use @@ -31,16 +32,17 @@ This package contains clients for various transports that can be used to spin up Bifrost uses a combination of a JSON configuration file and environment variables: 1. **JSON Configuration File**: Bifrost requires a configuration file to set up the gateway. This includes all your provider-level settings, keys, and meta configs for each of your providers. - -2. **Environment Variables**: If you don't want to include your keys in your config file, you can provide a `.env` file and add a prefix of `env.` followed by its key in your `.env` file. +2. **Environment Variables**: If you don't want to include your keys in your config file, you can provide a `.env` file and add a prefix of `env.` followed by its key in your `.env` file. ```json { - "keys": [{ - "value": "env.OPENAI_API_KEY", - "models": ["gpt-4o-mini", "gpt-4-turbo"], - "weight": 1.0 - }] + "keys": [ + { + "value": "env.OPENAI_API_KEY", + "models": ["gpt-4o-mini", "gpt-4-turbo"], + "weight": 1.0 + } + ] } ``` @@ -63,7 +65,13 @@ Please refer to `config.example.json` and `.env.sample` for examples. ### Docker Setup -You can run Bifrost using our **independent Dockerfile**. Just copy our Dockerfile and run these commands to get your Bifrost instance up and running: +1. Download the Dockerfile: + +```bash +curl -L -o Dockerfile https://raw.githubusercontent.com/maximhq/bifrost/main/transports/Dockerfile +``` + +2. Build the Docker image: ```bash docker build \ @@ -72,7 +80,11 @@ docker build \ --build-arg PORT=8080 \ --build-arg POOL_SIZE=300 \ -t bifrost-transports . +``` + +3. Run the Docker container: +```bash docker run -p 8080:8080 bifrost-transports ``` @@ -87,19 +99,21 @@ If you wish to run Bifrost in your Go environment, follow these steps: 1. Install your binary: ```bash -go install github.com/maximhq/bifrost/transports/http@latest +go install github.com/maximhq/bifrost/transports/bifrost-http@latest ``` 2. Run your binary: - If it's in your PATH: + ```bash -http -config config.json -env .env -port 8080 -pool-size 300 +bifrost-http -config config.json -env .env -port 8080 -pool-size 300 ``` - Otherwise: + ```bash -./http -config config.json -env .env -port 8080 -pool-size 300 +./bifrost-http -config config.json -env .env -port 8080 -pool-size 300 ``` You can also add a flag for `-drop-excess-requests=false` in your command to drop excess requests when the buffer is full. Read more about `DROP_EXCESS_REQUESTS` and `POOL_SIZE` [here](https://github.com/maximhq/bifrost/tree/main?tab=README-ov-file#additional-configurations). @@ -107,17 +121,19 @@ You can also add a flag for `-drop-excess-requests=false` in your command to dro ## 🧰 Usage Ensure that: + - Bifrost's HTTP server is running - The providers/models you use are configured in your JSON config file ### Text Completions ```bash +# Make sure to setup anthropic and claude-2.1 in your config.json curl -X POST http://localhost:8080/v1/text/completions \ -H "Content-Type: application/json" \ -d '{ - "provider": "openai", - "model": "gpt-4o-mini", + "provider": "anthropic", + "model": "claude-2.1", "text": "Once upon a time in the land of AI,", "params": { "temperature": 0.7, @@ -175,4 +191,4 @@ Read more about fallbacks and other additional configurations [here](https://git --- -Built with ❤️ by [Maxim](https://github.com/maximhq) \ No newline at end of file +Built with ❤️ by [Maxim](https://github.com/maximhq) diff --git a/transports/http/main.go b/transports/bifrost-http/main.go similarity index 100% rename from transports/http/main.go rename to transports/bifrost-http/main.go