diff --git a/README.md b/README.md index bd48a86f9c..1d4462ecce 100644 --- a/README.md +++ b/README.md @@ -218,12 +218,15 @@ bifrost/ │ ├── tests/ # Tests to make sure everything is in place │ ├── bifrost.go # Main Bifrost implementation │ +├── docs/ # Documentations for Bifrost's configurations and contribution guides +│ └── ... +│ ├── transports/ # Interface layers (HTTP, gRPC, etc.) │ ├── bifrost-http/ # HTTP transport implementation │ └── ... │ └── plugins/ # Plugin Implementations - ├── maxim-logger.go + ├── maxim/ └── ... ``` @@ -240,7 +243,7 @@ If you want to **set up the Bifrost API quickly**, [check the transports documen Bifrost is divided into three Go packages: core, plugins, and transports. 1. **core**: This package contains the core implementation of Bifrost as a Go package. -2. **plugins**: This package serves as an extension to core. You can download this package using `go get github.com/maximhq/bifrost/plugins` and pass the plugins while initializing Bifrost. +2. **plugins**: This package serves as an extension to core. You can download individual packages using `go get github.com/maximhq/bifrost/plugins/{plugin-name}` and pass the plugins while initializing Bifrost. ```golang plugin, err := plugins.NewMaximLoggerPlugin(os.Getenv("MAXIM_API_KEY"), os.Getenv("MAXIM_LOGGER_ID")) @@ -259,116 +262,11 @@ client, err := bifrost.Init(schemas.BifrostConfig{ ### Additional Configurations -1. InitalPoolSize and DropExcessRequests: You can customise the initial pool size of the structs and channels bifrost creates on `bifrost.Init()`. A higher value would mean lesser run time allocations and lower latency but at the cost of more memory usage. Takes the defined default value if not provided. - -```golang - client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &yourAccount, - InitialPoolSize: 500, - DropExcessRequests: true, - }) -``` - -When `DropExcessRequests` is set to true, in cases where the queue is full, requests will not wait for the queue to be empty and will be dropped instead. By default it is set to false. - -2. Logger: Like account interface, bifrost also allows you to pass your custom logger if it follows [bifrost's logger interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/logger.go). Takes in the [default logger](https://github.com/maximhq/bifrost/blob/main/core/logger.go) if not provided. - -```golang - client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &yourAccount, - Logger: &yourLogger, - }) -``` - -The default logger is set to level info by default. If you wish to use it but with a different log level, you can do it like this - - -```golang - client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &yourAccount, - Logger: bifrost.NewDefaultLogger(schemas.LogLevelDebug), - }) -``` - -3. Plugins: You can create and pass your custom pre-hook and post-hook plugins to bifrost as long as they follow [bifrost's plugin interface](https://github.com/maximhq/bifrost/blob/main/core/schemas/plugin.go). - -```golang - client, err := bifrost.Init(schemas.BifrostConfig{ - Account: &yourAccount, - Plugins: []schemas.Plugin{yourPlugin1, yourPlugin2, ...}, - }) -``` - -4. Customise your provider settings: You can customise proxy config, timeouts, retry settings, concurrency buffer sizes for each of your provider in your account interface's GetConfigForProvider() method. - -exmaple: - -```golang - schemas.ProviderConfig{ - NetworkConfig: schemas.NetworkConfig{ - DefaultRequestTimeoutInSeconds: 30, - MaxRetries: 2, - RetryBackoffInitial: 100 * time.Millisecond, - RetryBackoffMax: 2 * time.Second, - }, - MetaConfig: &meta.BedrockMetaConfig{ - SecretAccessKey: os.Getenv("BEDROCK_ACCESS_KEY"), - Region: StrPtr("us-east-1"), - }, - ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ - Concurrency: 3, - BufferSize: 10, - }, - ProxyConfig: &schemas.ProxyConfig{ - Type: schemas.HttpProxy, - URL: yourProxyURL, - }, - } -``` - -You can manage buffer size (maximum number of requests you want to hold in the system) concurrency (maximum number of requests you want to be made concurrently) for each provider. You can manage user usage and provider limits by providing these custom provider settings Default values are taken for network config, concurrecy and buffer sizes if not provided. - -Bifrost also supports multiple API keys per provider, enabling both load balancing and redundancy. You can assign weights to each key to control how frequently they are selected for requests. By default, all keys are treated with equal weight unless specified otherwise. - -```golang - []schemas.Key{ - { - Value: os.Getenv("OPEN_AI_API_KEY1"), - Models: []string{"gpt-4o-mini", "gpt-4-turbo"}, - Weight: 0.6, - }, - { - Value: os.Getenv("OPEN_AI_API_KEY2"), - Models: []string{"gpt-4-turbo"}, - Weight: 0.3, - }, - { - Value: os.Getenv("OPEN_AI_API_KEY3"), - Models: []string{"gpt-4o-mini"}, - Weight: 0.1, - }, - } -``` - -You can check [this](https://github.com/maximhq/bifrost/blob/main/core/tests/account.go) file to refer all the customisation settings. - -5. Fallbacks: You can define fallback providers for each request, which will be used if all retry attempts with your primary provider fail. These fallback providers are attempted in the order you specify, provided they are configured in your account at runtime. Once a fallback is triggered, its own retry settings will apply, rather than those of the original provider. - -```golang - result, err := bifrost.ChatCompletionRequest( - schemas.OpenAI, &schemas.BifrostRequest{ - Model: "gpt-4o-mini", - Input: schemas.RequestInput{ - ChatCompletionInput: &messages, - }, - Fallbacks: []schemas.Fallback{ - { - Provider: schemas.Anthropic, - Model: "claude-3-5-sonnet-20240620", // make sure you have configured this - }, - }, - }, context.Background() - ) -``` +- [Memory Management](https://github.com/maximhq/bifrost/blob/main/docs/memory-management.md) +- [Logger](https://github.com/maximhq/bifrost/blob/main/docs/logger.md)) +- [Plugins](https://github.com/maximhq/bifrost/blob/main/docs/plugins.md) +- [Provider Configurations](https://github.com/maximhq/bifrost/blob/main/docs/providers.md) +- [Fallbacks](https://github.com/maximhq/bifrost/blob/main/docs/fallbacks.md) --- @@ -457,7 +355,9 @@ This flexibility allows you to optimize Bifrost for your specific use case, whet ## 🤝 Contributing -Contributions are welcome! We welcome all kinds of contributions — bug fixes, features, docs, and ideas. Please feel free to submit a Pull Request. +We welcome contributions of all kinds—whether it's bug fixes, features, documentation improvements, or new ideas. Feel free to open an issue, and once its’ assigned, submit a Pull Request. + +Here's how to get started (after picking up an issue): 1. Fork the repository 2. Create your feature branch (`git checkout -b feature/amazing-feature`) diff --git a/docs/fallbacks.md b/docs/fallbacks.md new file mode 100644 index 0000000000..34b4ffd6df --- /dev/null +++ b/docs/fallbacks.md @@ -0,0 +1,205 @@ +# Bifrost Fallback System + +Bifrost provides a robust fallback mechanism that allows you to define alternative providers and models to use when the primary provider fails. This ensures high availability and reliability for your AI-powered applications. + +## 1. How Fallbacks Work + +1. When a request is made to a primary provider, Bifrost first attempts to complete the request using that provider +2. If the primary provider fails after all retry attempts, Bifrost automatically tries the fallback providers in the order specified +3. Each fallback provider uses its own retry settings and configuration set in your account implementation +4. The first successful fallback response is returned to the client + +## 2. Configuring Fallbacks + +### Basic Fallback Configuration + +```golang +result, err := bifrost.ChatCompletionRequest( + context.Background(), &schemas.BifrostRequest{ + Provider: schemas.OpenAI, + Model: "gpt-4", + Input: schemas.RequestInput{ + ChatCompletionInput: &messages, + }, + Fallbacks: []schemas.Fallback{ + { + Provider: schemas.Anthropic, + Model: "claude-3-sonnet", + }, + }, + }, +) +``` + +### Multiple Fallbacks + +```golang +result, err := bifrost.ChatCompletionRequest( + context.Background(), &schemas.BifrostRequest{ + Provider: schemas.OpenAI, + Model: "gpt-4", + Input: schemas.RequestInput{ + ChatCompletionInput: &messages, + }, + Fallbacks: []schemas.Fallback{ + { + Provider: schemas.Anthropic, + Model: "claude-3-sonnet", + }, + { + Provider: schemas.Bedrock, + Model: "anthropic.claude-3-sonnet", + }, + { + Provider: schemas.Azure, + Model: "gpt-4", + }, + }, + }, +) +``` + +## 3. Important Considerations + +### Provider Configuration + +- Each fallback provider must be properly configured in your account +- If a fallback provider is not configured, it will be skipped +- Each provider's configuration (retries, timeouts, etc.) is independent + +### Model Compatibility + +- Ensure that the fallback models support the same capabilities as your primary model +- Consider model-specific parameters and limitations +- Verify that the fallback models are available in your account + +### Performance Impact + +- Fallbacks add latency when the primary provider fails +- Consider the order of fallbacks based on: + - Provider reliability + - Model performance + - Cost considerations + - Geographic location + +## 4. Best Practices + +1. **Provider Selection** + + - Choose fallback providers with different infrastructure + - Consider geographic distribution for high availability + - Balance cost and performance in fallback order + +2. **Model Selection** + + - Use models with similar capabilities + - Consider model-specific features (e.g., function calling, streaming) + - Account for different token limits and pricing + +3. **Error Handling** + + - Monitor fallback usage to identify provider issues + - Set up alerts for frequent fallback activations (can be done using bifrost's plugin interface) + - Regularly review and update fallback configurations + +4. **Testing** + - Test fallback scenarios in development + - Verify all fallback providers are properly configured + - Simulate provider failures to ensure smooth fallback + +## 5. HTTP Transport Examples + +### Basic HTTP Fallback Request + +```json +POST /v1/chat/completions +{ + "provider": "openai", + "model": "gpt-4", + "input": { + "chat_completion_input": [ + { + "role": "user", + "content": "Hello, how are you?" + } + ] + }, + "fallbacks": [ + { + "provider": "anthropic", + "model": "claude-3-sonnet" + } + ] +} +``` + +### HTTP Request with Multiple Fallbacks + +```json +POST /v1/chat/completions +{ + "provider": "openai", + "model": "gpt-4", + "input": { + "chat_completion_input": [ + { + "role": "user", + "content": "Explain quantum computing" + } + ] + }, + "fallbacks": [ + { + "provider": "anthropic", + "model": "claude-3-sonnet" + }, + { + "provider": "bedrock", + "model": "anthropic.claude-3-sonnet" + }, + { + "provider": "azure", + "model": "gpt-4" + } + ], + "params": { + "temperature": 0.7, + "max_tokens": 1000 + } +} +``` + +### HTTP Response Example + +```json +{ + "id": "chatcmpl-123", + "object": "chat.completion", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Quantum computing is a type of computing..." + }, + "finish_reason": "stop" + } + ], + "model": "claude-3-sonnet", + "usage": { + "prompt_tokens": 10, + "completion_tokens": 100, + "total_tokens": 110 + }, + "extra_fields": { + "provider": "anthropic", + "latency": 1.234, + "billed_usage": { + "prompt_tokens": 10.0, + "completion_tokens": 100.0 + } + } +} +``` + +Note: The response includes metadata about which provider was used (in this case, the fallback provider "anthropic") and its performance metrics. diff --git a/docs/logger.md b/docs/logger.md new file mode 100644 index 0000000000..05df5808a2 --- /dev/null +++ b/docs/logger.md @@ -0,0 +1,123 @@ +# Bifrost Logging System + +Bifrost provides a flexible logging system that allows you to either use the built-in logger or implement your own custom logger. + +## 1. Log Levels + +Bifrost supports four log levels: + +- `debug`: Detailed debugging information, typically only needed during development +- `info`: General informational messages about normal operation +- `warn`: Potentially harmful situations that don't prevent normal operation +- `error`: Serious problems that need attention and may prevent normal operation + +## 2. Using the Default Logger + +Bifrost comes with a built-in logger that writes to stdout/stderr with formatted timestamps and log levels. It's used by default if no custom logger is provided. + +### Default Configuration + +```golang +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + // Logger not specified, will use default logger with info level +}) +``` + +### Customizing Default Logger Level + +```golang +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + Logger: bifrost.NewDefaultLogger(schemas.LogLevelDebug), // Set to debug level +}) +``` + +### Default Logger Output Format + +The default logger formats messages as: + +``` +[BIFROST-TIMESTAMP] LEVEL: message +[BIFROST-TIMESTAMP] ERROR: (error: error_message) +``` + +Example outputs: + +``` +[BIFROST-2024-03-20T10:15:30Z] INFO: Initializing provider OpenAI +[BIFROST-2024-03-20T10:15:31Z] ERROR: (error: failed to connect to provider) +``` + +## 3. Implementing a Custom Logger + +You can implement your own logger by following the `Logger` interface: + +```golang +type Logger interface { + // Debug logs a debug-level message + Debug(msg string) + + // Info logs an info-level message + Info(msg string) + + // Warn logs a warning-level message + Warn(msg string) + + // Error logs an error-level message + Error(err error) +} +``` + +### Example Custom Logger Implementation + +```golang +type CustomLogger struct { + // Your logger fields +} + +func (l *CustomLogger) Debug(msg string) { + // Implement debug logging +} + +func (l *CustomLogger) Info(msg string) { + // Implement info logging +} + +func (l *CustomLogger) Warn(msg string) { + // Implement warning logging +} + +func (l *CustomLogger) Error(err error) { + // Implement error logging +} + +// Using the custom logger +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + Logger: &CustomLogger{}, +}) +``` + +## 4. Best Practices + +1. **Log Level Selection** + + - Use `debug` for development and troubleshooting + - Use `info` for production monitoring + - Use `warn` for potential issues that don't affect functionality + - Use `error` for critical issues that need immediate attention + +2. **Custom Logger Implementation** + + - Ensure thread safety if your logger is used concurrently + - Consider implementing log rotation for production environments + - Include relevant context in log messages + - Handle errors appropriately in your logging implementation + +3. **Performance Considerations** + - Avoid expensive operations in logging methods + - Consider using async logging for high-throughput scenarios + - Be mindful of log volume in production environments + +Remember that logging is crucial for monitoring and debugging your Bifrost implementation. Choose the appropriate logging strategy based on your environment and requirements. diff --git a/docs/memory-management.md b/docs/memory-management.md new file mode 100644 index 0000000000..67da19e824 --- /dev/null +++ b/docs/memory-management.md @@ -0,0 +1,95 @@ +# Bifrost Memory and Concurrency Management + +This document outlines the key configurations for managing memory usage and concurrency in Bifrost. + +## 1. Initial Pool Size + +The `InitialPoolSize` configuration determines the initial size of object pools that Bifrost creates during initialization. These pools are used to reduce runtime allocations and improve performance. + +### Default Value + +- Default: `100` + +### Configuration + +```golang +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + InitialPoolSize: 500, // Custom pool size + DropExcessRequests: true, +}) +``` + +### Impact + +- Higher values reduce runtime allocations and latency +- Higher values increase memory usage +- Recommended to set based on your expected concurrent request volume + +## 2. Drop Excess Requests + +The `DropExcessRequests` flag controls how Bifrost handles requests when queues are full. + +### Default Value + +- Default: `false` + +### Configuration + +```golang +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + InitialPoolSize: 500, + DropExcessRequests: true, // Enable dropping excess requests +}) +``` + +### Behavior + +- When `true`: Requests are dropped immediately if the queue is full +- When `false`: Requests wait for queue space to become available + +## 3. Provider Concurrency and Buffer Size + +Each provider can be configured with specific concurrency and buffer size settings. + +### Default Values + +- Default Concurrency: `10` workers +- Default Buffer Size: `100` requests + +### Configuration + +```json +{ + "openai": { + "concurrency_and_buffer_size": { + "concurrency": 20, // Number of concurrent workers + "buffer_size": 200 // Size of the request queue + } + } +} +``` + +### Impact + +- **Concurrency**: Controls the number of parallel workers processing requests + + - Higher values increase throughput but also increase resource usage + - Should be set based on your provider's rate limits and server capacity + +- **Buffer Size**: Controls the size of the request queue + - Higher values allow more requests to be queued + - Should be set based on your expected request volume and latency requirements + +### Best Practices + +1. Set `InitialPoolSize` to match your expected concurrent request volume +2. Enable `DropExcessRequests` if you want to fail fast when the system is overloaded +3. Configure provider concurrency based on: + - Provider's rate limits + - Available system resources + - Expected request patterns +4. Set buffer size to handle expected request spikes while considering memory constraints + +Remember that these configurations have direct impact on your system's performance and resource usage. It's recommended to test your configuration under expected load conditions to find the optimal settings for your use case. diff --git a/docs/plugins.md b/docs/plugins.md new file mode 100644 index 0000000000..9f22085714 --- /dev/null +++ b/docs/plugins.md @@ -0,0 +1,268 @@ +# Bifrost Plugin System + +Bifrost provides a powerful plugin system that allows you to extend and customize the request/response pipeline. Plugins can be used to implement various functionalities like rate limiting, caching, logging, monitoring, and more. + +## 1. How Plugins Work + +Plugins in Bifrost follow a simple but powerful interface that allows them to intercept and modify requests and responses at different stages of processing: + +1. **PreHook**: Executed before a request is sent to a provider + + - Can modify the request + - Can add custom headers or parameters + - Can implement rate limiting or validation + - Executed in the order they are registered + +2. **PostHook**: Executed after receiving a response from a provider + - Can modify the response + - Can implement caching + - Can add monitoring or logging + - Executed in reverse order of PreHooks + +## 2. Plugin Interface + +```golang +type Plugin interface { + // PreHook is called before a request is processed by a provider + PreHook(ctx *context.Context, req *BifrostRequest) (*BifrostRequest, error) + + // PostHook is called after a response is received from a provider + PostHook(ctx *context.Context, result *BifrostResponse) (*BifrostResponse, error) +} +``` + +## 3. Building Custom Plugins + +### Basic Plugin Structure + +```golang +type CustomPlugin struct { + // Your plugin fields +} + +func (p *CustomPlugin) PreHook(ctx *context.Context, req *BifrostRequest) (*BifrostRequest, error) { + // Modify request or add custom logic + return req, nil +} + +func (p *CustomPlugin) PostHook(ctx *context.Context, result *BifrostResponse) (*BifrostResponse, error) { + // Modify response or add custom logic + return result, nil +} +``` + +### Example: Rate Limiting Plugin + +```golang +type RateLimitPlugin struct { + limiter *rate.Limiter +} + +func NewRateLimitPlugin(rps float64) *RateLimitPlugin { + return &RateLimitPlugin{ + limiter: rate.NewLimiter(rate.Limit(rps), 1), + } +} + +func (p *RateLimitPlugin) PreHook(ctx *context.Context, req *BifrostRequest) (*BifrostRequest, error) { + if err := p.limiter.Wait(*ctx); err != nil { + return nil, err + } + return req, nil +} + +func (p *RateLimitPlugin) PostHook(ctx *context.Context, result *BifrostResponse) (*BifrostResponse, error) { + return result, nil +} +``` + +### Example: Logging Plugin + +```golang +type LoggingPlugin struct { + logger schemas.Logger +} + +func NewLoggingPlugin(logger schemas.Logger) *LoggingPlugin { + return &LoggingPlugin{logger: logger} +} + +func (p *LoggingPlugin) PreHook(ctx *context.Context, req *BifrostRequest) (*BifrostRequest, error) { + p.logger.Info(fmt.Sprintf("Request to %s with model %s", req.Provider, req.Model)) + return req, nil +} + +func (p *LoggingPlugin) PostHook(ctx *context.Context, result *BifrostResponse) (*BifrostResponse, error) { + p.logger.Info(fmt.Sprintf("Response from %s with %d tokens", result.Model, result.Usage.TotalTokens)) + return result, nil +} +``` + +## 4. Using Plugins + +### Initializing Bifrost with Plugins + +```golang +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + Plugins: []schemas.Plugin{ + NewRateLimitPlugin(10.0), // 10 requests per second + NewLoggingPlugin(logger), // Custom logging + // Add more plugins as needed + }, +}) +``` + +## 5. Available Plugins + +Bifrost comes with several built-in plugins that you can use out of the box. Each plugin has its own documentation in its respective folder: + +- **Rate Limiting**: `plugins/rate-limiting/` +- **Caching**: `plugins/caching/` +- **Monitoring**: `plugins/monitoring/` +- **Logging**: `plugins/logging/` + +To use these plugins, you can import them from their respective packages: + +```golang +import ( + "github.com/maximhq/bifrost/plugins/rate-limiting" + "github.com/maximhq/bifrost/plugins/caching" + // ... other plugin imports +) + +// Initialize with built-in plugins +client, err := bifrost.Init(schemas.BifrostConfig{ + Account: &yourAccount, + Plugins: []schemas.Plugin{ + rate_limiting.New(10.0), + caching.New(cacheConfig), + // ... other plugins + }, +}) +``` + +## 6. Best Practices + +1. **Plugin Order** + + - Consider the order of plugins carefully + - Rate limiting plugins should typically be first + - Logging plugins should be last to capture all modifications + +2. **Error Handling** + + - Always handle errors in both PreHook and PostHook + - Return meaningful error messages + - Consider the impact of errors on the request pipeline + +3. **Performance** + + - Keep plugin logic lightweight + - Avoid blocking operations in hooks + - Use context for cancellation + +4. **State Management** + + - Be careful with shared state between hooks + - Use context for passing data between hooks + - Consider thread safety for concurrent requests + +5. **Testing** + - Write unit tests for your plugins + - Test error scenarios + - Verify plugin order and execution + +## 7. Plugin Development Guidelines + +1. **Documentation** + + - Document your plugin's purpose and configuration + - Provide usage examples + - Include performance considerations + +2. **Configuration** + + - Make plugins configurable + - Use sensible defaults + - Validate configuration + +3. **Error Handling** + + - Use custom error types + - Provide detailed error messages + - Handle edge cases gracefully + +4. **Contribution Process** + + - Open an issue first to discuss the plugin's use case and design + - Create a pull request with a clear explanation of the plugin's purpose + - Follow the plugin structure requirements below + +5. **Plugin Structure** + Each plugin should be organized as follows: + + ``` + plugins/ + └── your-plugin-name/ + ├── main.go # Plugin implementation + ├── README.md # Documentation + └── go.mod # Module definition + ``` + + Example `main.go`: + + ```golang + package your_plugin_name + + import ( + "context" + "github.com/maximhq/bifrost/core/schemas" + ) + + type YourPlugin struct { + // Plugin fields + } + + func New(config YourPluginConfig) *YourPlugin { + return &YourPlugin{ + // Initialize plugin + } + } + + func (p *YourPlugin) PreHook(ctx *context.Context, req *schemas.BifrostRequest) (*schemas.BifrostRequest, error) { + // Implementation + return req, nil + } + + func (p *YourPlugin) PostHook(ctx *context.Context, result *schemas.BifrostResponse) (*schemas.BifrostResponse, error) { + // Implementation + return result, nil + } + ``` + + Example `README.md`: + + ````markdown + # Your Plugin Name + + Brief description of what the plugin does. + + ## Installation + + ```bash + go get github.com/maximhq/bifrost/plugins/your-plugin-name + ``` + + ## Usage + + Explain plugin usage. + + ## Configuration + + Describe configuration options. + + ## Examples + + Show usage examples. + ```` diff --git a/docs/providers.md b/docs/providers.md new file mode 100644 index 0000000000..6e82476c8f --- /dev/null +++ b/docs/providers.md @@ -0,0 +1,273 @@ +# Bifrost Provider System + +Bifrost supports multiple AI model providers, each with its own configuration options and capabilities. This document explains how to configure providers and develop new ones. + +## 1. Supported Providers + +Bifrost currently supports the following providers: + +- OpenAI +- Anthropic +- Azure +- Bedrock +- Cohere +- Vertex + +## 2. Provider Configuration + +### Basic Configuration Structure + +```golang +schemas.ProviderConfig{ + NetworkConfig: schemas.NetworkConfig{ + DefaultRequestTimeoutInSeconds: 30, + MaxRetries: 2, + RetryBackoffInitial: 100 * time.Millisecond, + RetryBackoffMax: 2 * time.Second, + }, + ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{ + Concurrency: 3, // Number of concurrent requests + BufferSize: 10, // Maximum requests in queue + }, + ProxyConfig: &schemas.ProxyConfig{ + Type: schemas.HttpProxy, + URL: "http://your-proxy:port", + }, +} +``` + +### Default Values + +```golang +const ( + DefaultMaxRetries = 0 + DefaultRetryBackoffInitial = 500 * time.Millisecond + DefaultRetryBackoffMax = 5 * time.Second + DefaultRequestTimeoutInSeconds = 30 + DefaultBufferSize = 100 + DefaultConcurrency = 10 +) +``` + +## 3. Provider-Specific Meta Configurations + +Few providers new meta configs for their setup. + +### Azure + +```golang +meta.AzureMetaConfig{ + Endpoint: "https://your-resource.openai.azure.com", + APIVersion: "2024-02-15-preview", + Deployments: map[string]string{ + "gpt-4": "gpt-4-deployment", + "gpt-35-turbo": "gpt-35-turbo-deployment", + }, +} +``` + +### Bedrock + +```golang +meta.BedrockMetaConfig{ + SecretAccessKey: os.Getenv("BEDROCK_ACCESS_KEY"), + Region: "us-east-1", + SessionToken: os.Getenv("BEDROCK_SESSION_TOKEN"), // Optional + ARN: os.Getenv("BEDROCK_ARN"), // Optional + InferenceProfiles: map[string]string{ + "gpt-4": "gpt-4-deployment-profile", + } +} +``` + +### Vertex + +```golang +meta.VertexMetaConfig{ + ProjectID: os.Getenv("VERTEX_PROJECT_ID"), + Location: "us-central1", + AuthCredentials: os.Getenv("VERTEX_AUTH_CREDENTIALS"), // GCP Auth creds +} +``` + +## 4. API Key Management + +### Key Weights + +Bifrost supports weighted distribution of requests across multiple API keys. The weight determines the relative frequency of key usage: + +- Weights are normalized (sum to 1.0) +- Higher weight = more frequent usage +- Equal weights if not specified +- Model-specific key assignment + +Example with weights: + +```golang +[]schemas.Key{ + { + Value: os.Getenv("OPEN_AI_API_KEY1"), + Models: []string{"gpt-4", "gpt-4-turbo"}, + Weight: 0.6, // 60% of requests for these models + }, + { + Value: os.Getenv("OPEN_AI_API_KEY2"), + Models: []string{"gpt-4-turbo"}, + Weight: 0.3, // 30% of requests for gpt-4-turbo + }, + { + Value: os.Getenv("OPEN_AI_API_KEY3"), + Models: []string{"gpt-4"}, + Weight: 0.1, // 10% of requests for gpt-4 + }, +} +``` + +### Key Selection Logic + +1. Filters keys that support the requested model +2. Normalizes weights of available keys +3. Uses weighted random selection +4. Falls back to first available key if selection fails + +## 5. Proxy Configuration + +Bifrost supports various proxy types for provider connections: + +### HTTP Proxy + +```golang +schemas.ProxyConfig{ + Type: schemas.HttpProxy, + URL: "http://proxy.example.com:8080", + Username: "user", // Optional + Password: "pass", // Optional +} +``` + +### SOCKS5 Proxy + +```golang +schemas.ProxyConfig{ + Type: schemas.Socks5Proxy, + URL: "socks5://proxy.example.com:1080", + Username: "user", // Optional + Password: "pass", // Optional +} +``` + +### Environment Proxy + +```golang +schemas.ProxyConfig{ + Type: schemas.EnvProxy, + // Uses HTTP_PROXY, HTTPS_PROXY environment variables +} +``` + +### Proxy Best Practices + +1. **Security** + + - Use HTTPS proxies when possible + - Rotate proxy credentials regularly + - Monitor proxy performance + +2. **Performance** + + - Choose geographically close proxies + - Monitor proxy latency + - Implement proxy fallbacks + +3. **Configuration** + + - Set appropriate timeouts + - Configure retry policies + - Monitor proxy errors + +## 6. Provider Development Guidelines + +### 1. Provider Structure + +All providers should be implemented in the `core/providers` directory. The structure should be: + +``` +core/ +├── providers/ +│ ├── your_provider.go # Provider implementation +│ └── ... # Other provider implementations +└── schemas/ + └── meta/ + └── your_provider.go # Provider-specific meta configuration +``` + +### 2. Provider Interface + +```golang +type Provider interface { + // GetProviderKey returns the provider's identifier + GetProviderKey() ModelProvider + + // TextCompletion performs a text completion request + TextCompletion(model, key, text string, params *ModelParameters) (*BifrostResponse, *BifrostError) + + // ChatCompletion performs a chat completion request + ChatCompletion(model, key string, messages []Message, params *ModelParameters) (*BifrostResponse, *BifrostError) +} +``` + +### 3. Meta Configuration + +If your provider requires additional configuration beyond the standard `ProviderConfig`, implement a meta configuration in `core/schemas/meta/your_provider.go`: + +```golang +// YourProviderMetaConfig implements the MetaConfig interface +type YourProviderMetaConfig struct { + // Add your provider-specific fields here + Endpoint string `json:"endpoint"` + APIVersion string `json:"api_version"` + // ... other fields +} + +// Implement all required methods from the MetaConfig interface +func (c *YourProviderMetaConfig) GetSecretAccessKey() *string { /* ... */ } +func (c *YourProviderMetaConfig) GetRegion() *string { /* ... */ } +// ... implement other interface methods +``` + +The meta configuration must implement all methods from the `MetaConfig` interface defined in `core/schemas/provider.go`. Return `nil` for methods that don't apply to your provider. + +### 4. Development Process + +1. Open an issue to discuss the new provider +2. Create a pull request with: + - Provider implementation in `core/providers/` + - Addition of provider key in `ModelProvider` in `/core/schemas/bifrost.go` + - Meta configuration in `core/schemas/meta/` (if needed) + - Tests in `core/tests` with `Test{ProviderName}` function name. + - Documentation update in `docs/providers.go` + +### 5. Implementation Requirements + +1. **Error Handling** + + - Use standard Bifrost error types + - Gracefully handling and logging (using bifrost logger) all runtime errors + +2. **Configuration** + + - Support provider-specific settings through meta configuration (if needed) + - Implement default values + - Validate configuration + - Implement sync pools for optimized resource allocations + +3. **Testing** + + - Unit tests for all methods (using `core/tests/setup.go` file) + - Integration tests + - Error case coverage + +4. **Documentation** + - Provider capabilities + - Configuration options + - Meta configuration usage