diff --git a/README.md b/README.md index 85bba84470..4447b524a3 100644 --- a/README.md +++ b/README.md @@ -26,43 +26,47 @@ Bifrost is a high-performance AI gateway that connects you to 8+ providers (Open ๐Ÿ“– For detailed setup guides with multiple providers, advanced configuration, and language examples, see [Quick Start Documentation](./docs/quickstart/README.md) -**Step 1:** Create your config (copy & paste this) - -```json -{ - "providers": { - "openai": { - "keys": [ - { - "value": "env.OPENAI_API_KEY", - "models": ["gpt-4o-mini"], - "weight": 1.0 - } - ] - } - } -} +**Step 1:** Start Bifrost (choose one) + +```bash +# ๐Ÿณ Docker (easiest - zero config needed!) +docker pull maximhq/bifrost +docker run -p 8080:8080 maximhq/bifrost + +# ๐Ÿ”ง Or install Go binary (Make sure Go is in your PATH) +go install github.com/maximhq/bifrost/transports/bifrost-http@latest +bifrost-http -port 8080 ``` -**Step 2:** Add your API key +**Step 2:** Open the built-in web interface ```bash -export OPENAI_API_KEY=your_openai_api_key +# ๐Ÿ–ฅ๏ธ Configure visually - no config files needed! +# macOS: +open http://localhost:8080 + +# Linux: +xdg-open http://localhost:8080 + +# Windows: +start http://localhost:8080 + +# Or simply open http://localhost:8080 manually in your browser ``` -**Step 3:** Start Bifrost (choose one) +**Step 3:** Add your provider via the web UI or API ```bash -# ๐Ÿณ Docker -docker pull maximhq/bifrost -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY \ - maximhq/bifrost +# Via Web UI: Just click "Add Provider" and enter your OpenAI API key +# Or via API: +curl -X POST http://localhost:8080/providers \ + -H "Content-Type: application/json" \ + -d '{ + "provider": "openai", + "keys": [{"value": "env.OPENAI_API_KEY", "models": ["gpt-4o-mini"], "weight": 1.0}] + }' -# ๐Ÿ”ง Or install Go binary (Make sure Go is in your PATH) -go install github.com/maximhq/bifrost/transports/bifrost-http@latest -bifrost-http -config config.json -port 8080 +# Make sure to set the environment variable OPENAI_API_KEY in bifrost's session, or pass it as a flag in Docker (docker run -e OPENAI_API_KEY maximhq/bifrost). ``` **Step 4:** Test it works @@ -81,11 +85,12 @@ curl -X POST http://localhost:8080/v1/chat/completions \ **๐ŸŽ‰ Boom! You're done!** -Your AI gateway is now running and ready for production. You can: +Your AI gateway is now running with a beautiful web interface. You can: -- Add more providers for automatic failover -- Scale to thousands of requests per second -- Drop this into existing OpenAI/Anthropic code with zero changes +- **๐Ÿ–ฅ๏ธ Configure everything visually** - No more JSON files! +- **๐Ÿ“Š Monitor requests in real-time** - See logs, analytics, and metrics +- **๐Ÿ”„ Add providers and MCP clients on-the-fly** - Scale and failover without restarts +- **๐Ÿš€ Drop into existing code** - Zero changes to your OpenAI/Anthropic apps > **Want more?** See our [Complete Setup Guide](./docs/quickstart/http-transport.md) for multi-provider configuration, failover strategies, and production deployment. @@ -112,18 +117,19 @@ Your AI gateway is now running and ready for production. You can: ## โœจ Features -- **Multi-Provider Support**: Integrate with OpenAI, Anthropic, Amazon Bedrock, Mistral, Ollama, and more through a single API -- **Fallback Mechanisms**: Automatically retry failed requests with alternative models or providers -- **Dynamic Key Management**: Rotate and manage API keys efficiently with weighted distribution -- **Connection Pooling**: Optimize network resources for better performance -- **Concurrency Control**: Manage rate limits and parallel requests effectively -- **Flexible Transports**: Multiple transports for easy integration into your infra -- **Plugin First Architecture**: No callback hell, simple addition/creation of custom plugins -- **MCP Integration**: Built-in Model Context Protocol (MCP) support for external tool integration and execution -- **Custom Configuration**: Offers granular control over pool sizes, network retry settings, fallback providers, and network proxy configurations -- **Built-in Observability**: Native Prometheus metrics out of the box, no wrappers, no sidecars, just drop it in and scrape -- **SDK Support**: Bifrost is available as a Go package, so you can use it directly in your own applications. -- **Seamless Integration with Generative AI SDKs**: Effortlessly transition to Bifrost by simply updating the `base_url` in your existing SDKs, such as OpenAI, Anthropic, GenAI, and more. Just one line of code is all it takes to make the switch. +- **๐Ÿ–ฅ๏ธ Built-in Web UI**: Visual configuration, real-time monitoring, and analytics dashboard - no config files needed +- **๐Ÿš€ Zero-Config Startup & Easy Integration**: Start immediately with dynamic provider configuration, or integrate existing SDKs by simply updating the `base_url` - one line of code to get running +- **๐Ÿ”„ Multi-Provider Support**: Integrate with OpenAI, Anthropic, Amazon Bedrock, Mistral, Ollama, and more through a single API +- **๐Ÿ›ก๏ธ Fallback Mechanisms**: Automatically retry failed requests with alternative models or providers +- **๐Ÿ”‘ Dynamic Key Management**: Rotate and manage API keys efficiently with weighted distribution +- **โšก Connection Pooling**: Optimize network resources for better performance +- **๐ŸŽฏ Concurrency Control**: Manage rate limits and parallel requests effectively +- **๐Ÿ”Œ Flexible Transports**: Multiple transports for easy integration into your infra +- **๐Ÿ—๏ธ Plugin First Architecture**: No callback hell, simple addition/creation of custom plugins +- **๐Ÿ› ๏ธ MCP Integration**: Built-in Model Context Protocol (MCP) support for external tool integration and execution +- **โš™๏ธ Custom Configuration**: Offers granular control over pool sizes, network retry settings, fallback providers, and network proxy configurations +- **๐Ÿ“Š Built-in Observability**: Native Prometheus metrics out of the box, no wrappers, no sidecars, just drop it in and scrape +- **๐Ÿ”ง SDK Support**: Bifrost is available as a Go package, so you can use it directly in your own applications --- diff --git a/docs/README.md b/docs/README.md index 8fd80383f4..0026572399 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,7 +11,7 @@ Choose your preferred way to use Bifrost: | **๐Ÿ”ง Go Package** | Direct integration, maximum control | 2 minutes | [๐Ÿ“– Go Package Guide](quickstart/go-package.md) | | **๐ŸŒ HTTP Transport** | Language-agnostic, microservices | 30 seconds | [๐Ÿ“– HTTP Transport Guide](quickstart/http-transport.md) | -**New to Bifrost?** Start with [โšก Quick Start](quickstart/) to get running in under 30 seconds. +**New to Bifrost?** Start with [โšก Quick Start](quickstart/) to get running with zero configuration in under 30 seconds. --- diff --git a/docs/contributing/README.md b/docs/contributing/README.md index d0e912fd61..85b3a21786 100644 --- a/docs/contributing/README.md +++ b/docs/contributing/README.md @@ -13,14 +13,17 @@ Ready to contribute? Here's your fastest path to making an impact: ```bash # 1. Fork and clone git clone https://github.com/YOUR_USERNAME/bifrost.git -cd bifrost +cd bifrost/core # or bifrost/transports # 2. Install dependencies go mod download # 3. Verify setup -go test ./core/... -cd transports && go build -o bifrost-http +cd ../tests/core-providers/ +go test -run TestOpenAI # or any provider you want to test + +cd ../transports-integrations/ +# read the README.md file in the transports-integrations directory for testing instructions # 4. You're ready! ๐ŸŽ‰ ``` diff --git a/docs/media/cover.png b/docs/media/cover.png index 7b5f4c52a4..b19c328ca8 100644 Binary files a/docs/media/cover.png and b/docs/media/cover.png differ diff --git a/docs/quickstart/README.md b/docs/quickstart/README.md index a3b5e19d31..202df6b563 100644 --- a/docs/quickstart/README.md +++ b/docs/quickstart/README.md @@ -25,11 +25,13 @@ Get up and running with Bifrost in under 30 seconds. Choose your preferred integ ## ๐ŸŒ **HTTP Transport** - Choose if you: +- โœ… Want a clean UI for configuration and monitoring - โœ… Use any programming language (Python, Node.js, etc.) - โœ… Want to keep AI logic separate from your application - โœ… Need a centralized AI gateway for multiple services - โœ… Prefer REST API integration patterns - โœ… Want drop-in compatibility with existing provider SDKs +- โœ… Want to **add providers & MCP clients on-the-fly** without restarts **โ†’ [Start with HTTP Transport](http-transport.md)** diff --git a/docs/quickstart/http-transport.md b/docs/quickstart/http-transport.md index 5f8e163a0d..6c296af59e 100644 --- a/docs/quickstart/http-transport.md +++ b/docs/quickstart/http-transport.md @@ -1,12 +1,43 @@ # ๐ŸŒ HTTP Transport Quick Start -Get Bifrost running as an HTTP API in 30 seconds using Docker. Perfect for any programming language. +Get Bifrost running as an HTTP API in **15 seconds** with **zero configuration**! Perfect for any programming language. -## โšก 30-Second Setup +## ๐Ÿš€ Zero-Config Setup (15 seconds!) -### 1. Create `config.json` +### 1. Start Bifrost (No config needed!) -This file should contain your provider settings and API keys. +```bash +# ๐Ÿณ Docker (fastest) +docker pull maximhq/bifrost +docker run -p 8080:8080 maximhq/bifrost + +# ๐Ÿ”ง OR Go Binary (Make sure Go is in your PATH) +go install github.com/maximhq/bifrost/transports/bifrost-http@latest +bifrost-http -port 8080 +``` + +### 2. Open the Web Interface + +```bash +# ๐Ÿ–ฅ๏ธ Beautiful web UI for zero-config setup +# macOS: +open http://localhost:8080 +# Linux: +xdg-open http://localhost:8080 +# Windows: +start http://localhost:8080 +# Or simply open http://localhost:8080 manually in your browser +``` + +**๐ŸŽ‰ That's it!** Configure providers visually, monitor requests in real-time, and get analytics - all through the web interface! + +--- + +## ๐Ÿ“‚ File-Based Configuration (Optional) + +Want to use a config file instead? Bifrost automatically looks for `config.json` in your app directory: + +### 1. Create `config.json` in your app directory ```json { @@ -24,32 +55,77 @@ This file should contain your provider settings and API keys. } ``` -### 2. Set Up Your Environment - -Add your environment variable to the session. +### 2. Set environment variables and start ```bash export OPENAI_API_KEY="your-openai-api-key" + +# Docker with volume mount for persistence +docker run -p 8080:8080 \ + -v $(pwd):/app/data \ + -e OPENAI_API_KEY \ + maximhq/bifrost + +# OR Go Binary with app directory +bifrost-http -app-dir . -port 8080 ``` -### 3. Start the Bifrost HTTP Server +--- + +## ๐Ÿ“ Understanding App Directory & Docker Volumes -You can run using Docker or Go binary. +### **How the `-app-dir` Flag Works** + +The `-app-dir` flag tells Bifrost where to store and look for data: ```bash -# Docker -docker pull maximhq/bifrost -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY \ - maximhq/bifrost +# Use current directory as app directory +bifrost-http -app-dir . -# OR Go Binary (Make sure Go in your PATH) -go install github.com/maximhq/bifrost/transports/bifrost-http@latest -bifrost-http -config config.json -port 8080 +# Use specific directory as app directory +bifrost-http -app-dir /path/to/bifrost-data + +# Default: current directory if no flag specified +bifrost-http -port 8080 ``` -### 4. Test the API +**What Bifrost stores in the app directory:** + +- `config.json` - Configuration file (if using file-based config) +- `logs/` - Database logs and request history +- Any other persistent data + +### **How Docker Volumes Work with App Directory** + +Docker volumes map your host directory to Bifrost's app directory: + +```bash +# Map current host directory โ†’ /app/data inside container +docker run -p 8080:8080 -v $(pwd):/app/data maximhq/bifrost + +# Map specific host directory โ†’ /app/data inside container +docker run -p 8080:8080 -v /host/path/bifrost-data:/app/data maximhq/bifrost + +# No volume = ephemeral storage (lost when container stops) +docker run -p 8080:8080 maximhq/bifrost +``` + +### **Persistence Scenarios** + +| Scenario | Command | Result | +| ---------------------------- | ------------------------------------------------------------- | --------------------------------------- | +| **Ephemeral (testing)** | `docker run -p 8080:8080 maximhq/bifrost` | No persistence, configure via web UI | +| **Persistent (recommended)** | `docker run -p 8080:8080 -v $(pwd):/app/data maximhq/bifrost` | Saves config & logs to host directory | +| **Pre-configured** | Create `config.json`, then run with volume | Starts with your existing configuration | + +### **Best Practices** + +- **๐Ÿ”ง Development**: Use `-v $(pwd):/app/data` to persist config between restarts +- **๐Ÿš€ Production**: Mount dedicated volume for data persistence +- **๐Ÿงช Testing**: Run without volume for clean ephemeral instances +- **๐Ÿ‘ฅ Teams**: Share `config.json` in version control, mount directory with volume + +### 3. Test the API ```bash # Make your first request @@ -131,31 +207,33 @@ client = genai.Client( --- -## ๐Ÿš€ Next Steps (2 minutes each) +## ๐Ÿš€ Next Steps (30 seconds each) + +### **๐Ÿ–ฅ๏ธ Add Multiple Providers via Web UI** -### **๐Ÿ”— Add Multiple Providers** +1. Open `http://localhost:8080` in your browser +2. Click **"Add Provider"** +3. Select **OpenAI**, enter your API key, choose models +4. Click **"Add Provider"** again +5. Select **Anthropic**, enter your API key, choose models +6. **Done!** Your providers are now load-balanced automatically + +### **๐Ÿ“ก Or Add Multiple Providers via API** ```bash -# Create config.json -echo '{ - "providers": { - "openai": { - "keys": [{"value": "env.OPENAI_API_KEY", "models": ["gpt-4o-mini"], "weight": 1.0}] - }, - "anthropic": { - "keys": [{"value": "env.ANTHROPIC_API_KEY", "models": ["claude-3-sonnet-20240229"], "weight": 1.0}] - } - } -}' > config.json +# Add OpenAI +curl -X POST http://localhost:8080/providers \ + -H "Content-Type: application/json" \ + -d '{"provider": "openai", "keys": [{"value": "env.OPENAI_API_KEY", "models": ["gpt-4o-mini"], "weight": 1.0}]}' + +# Add Anthropic +curl -X POST http://localhost:8080/providers \ + -H "Content-Type: application/json" \ + -d '{"provider": "anthropic", "keys": [{"value": "env.ANTHROPIC_API_KEY", "models": ["claude-3-sonnet-20240229"], "weight": 1.0}]}' # Set environment variables +export OPENAI_API_KEY="your-openai-key" export ANTHROPIC_API_KEY="your-anthropic-key" - -# Start with config -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY -e ANTHROPIC_API_KEY \ - maximhq/bifrost ``` ### **โšก Test Different Providers** @@ -239,12 +317,14 @@ response, err := http.Post( ## ๐Ÿ”ง Setup Methods Comparison -| Method | Pros | Use When | -| ------------- | ----------------------------------------------- | -------------------------------- | -| **Docker** | No Go installation needed, isolated environment | Production, CI/CD, quick testing | -| **Go Binary** | Direct execution, easier debugging | Development, custom builds | +| Method | Pros | Use When | +| --------------- | ---------------------------------------------------- | -------------------------------- | +| **Zero Config** | No files needed, visual setup, instant start | Quick testing, demos, new users | +| **File-Based** | Version control, automation, reproducible deployment | Production, CI/CD, team setups | +| **Docker** | No Go installation needed, isolated environment | Production, CI/CD, quick testing | +| **Go Binary** | Direct execution, easier debugging | Development, custom builds | -Both methods require the same `config.json` file and environment variables. +**Note:** When using file-based config, Bifrost only looks for `config.json` in your specified app directory. --- @@ -274,10 +354,12 @@ If you're building a Go application and want direct integration, try the **[Go P ## ๐Ÿ’ก Why HTTP Transport? -- โœ… **Language agnostic** - Use from Python, Node.js, PHP, etc. -- โœ… **Drop-in replacement** - Zero code changes for existing apps -- โœ… **OpenAI compatible** - All responses follow OpenAI structure -- โœ… **Microservices ready** - Centralized AI gateway -- โœ… **Production features** - Health checks, metrics, monitoring +- **๐Ÿ–ฅ๏ธ Built-in Web UI** - Visual configuration, monitoring, and analytics +- **๐Ÿš€ Zero configuration** - Start instantly, configure dynamically +- **๐ŸŒ Language agnostic** - Use from Python, Node.js, PHP, etc. +- **๐Ÿ”„ Drop-in replacement** - Zero code changes for existing apps +- **๐Ÿ”— OpenAI compatible** - All responses follow OpenAI structure +- **โš™๏ธ Microservices ready** - Centralized AI gateway +- **๐Ÿ“Š Production features** - Health checks, metrics, monitoring **๐ŸŽฏ Ready for production? Check out [Complete HTTP Usage Guide](../usage/http-transport/) โ†’** diff --git a/docs/usage/http-transport/README.md b/docs/usage/http-transport/README.md index ea2f470ae7..ce10f8a9dc 100644 --- a/docs/usage/http-transport/README.md +++ b/docs/usage/http-transport/README.md @@ -1,27 +1,29 @@ # ๐ŸŒ HTTP Transport -Complete guide to using Bifrost as an HTTP API service for multi-provider AI access, drop-in integrations, and production deployment. +Complete guide to using Bifrost as an HTTP API service with **built-in web UI**, zero-configuration startup, multi-provider AI access, drop-in integrations, and production deployment. -> **๐Ÿ’ก Quick Start:** See the [30-second setup](../../quickstart/http-transport.md) to get the HTTP service running quickly. +> **๐Ÿ’ก Quick Start:** See the [15-second zero-config setup](../../quickstart/http-transport.md) to get the HTTP service running with web UI instantly. --- ## ๐Ÿ“‹ HTTP Transport Overview -Bifrost HTTP transport provides a REST API service for: +Bifrost HTTP transport provides a REST API service with **built-in web UI** for: -- **Multi-provider access** through unified endpoints -- **Drop-in replacements** for OpenAI, Anthropic, Google GenAI APIs -- **Language-agnostic integration** with any HTTP client -- **Production-ready deployment** with monitoring and scaling -- **MCP tool execution** via HTTP endpoints +- **๐Ÿ–ฅ๏ธ Visual configuration** with real-time monitoring and analytics +- **๐Ÿš€ Zero-configuration startup** - begin immediately, configure dynamically +- **๐Ÿ”„ Multi-provider access** through unified endpoints +- **๐Ÿ”— Drop-in replacements** for OpenAI, Anthropic, Google GenAI APIs +- **๐ŸŒ Language-agnostic integration** with any HTTP client +- **๐Ÿ“Š Production-ready deployment** with monitoring and scaling +- **๐Ÿ› ๏ธ MCP tool execution** via HTTP endpoints ```bash -# Start Bifrost HTTP service -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY \ - maximhq/bifrost +# Start Bifrost HTTP service (zero config!) +docker run -p 8080:8080 maximhq/bifrost + +# Open web interface for visual configuration +open http://localhost:8080 # Make requests to any provider curl -X POST http://localhost:8080/v1/chat/completions \ @@ -191,25 +193,23 @@ curl -X POST http://localhost:8080/openai/v1/chat/completions \ ## ๐Ÿš€ Deployment Options -### **Docker (Recommended)** +### **Zero-Config Docker (Recommended)** ```bash -# Quick start -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY \ - -e ANTHROPIC_API_KEY \ - maximhq/bifrost +# Start instantly with web UI +docker run -p 8080:8080 maximhq/bifrost +# Configure via http://localhost:8080 +``` -# Production with custom settings +### **File-Based Docker** + +```bash +# With persistent config.json in app directory docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -v $(pwd)/logs:/app/logs \ + -v $(pwd):/app/data \ -e OPENAI_API_KEY \ -e ANTHROPIC_API_KEY \ - maximhq/bifrost \ - -pool-size 500 \ - -drop-excess-requests + maximhq/bifrost ``` ### **Binary Deployment** @@ -218,50 +218,14 @@ docker run -p 8080:8080 \ # Install go install github.com/maximhq/bifrost/transports/bifrost-http@latest -# Run -bifrost-http \ - -config config.json \ - -port 8080 \ - -pool-size 300 \ - -plugins maxim +# Zero config startup (uses current directory) +bifrost-http -port 8080 ``` -### **Kubernetes** - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: bifrost -spec: - replicas: 3 - selector: - matchLabels: - app: bifrost - template: - metadata: - labels: - app: bifrost - spec: - containers: - - name: bifrost - image: maximhq/bifrost:latest - ports: - - containerPort: 8080 - env: - - name: OPENAI_API_KEY - valueFrom: - secretKeyRef: - name: ai-keys - key: openai - volumeMounts: - - name: config - mountPath: /app/config - volumes: - - name: config - configMap: - name: bifrost-config -``` +For detailed deployment instructions including app directory setup, Docker volumes, and production best practices, see: + +- [Understanding App Directory & Docker Volumes](../../quickstart/http-transport.md#understanding-app-directory--docker-volumes) +- [Production Deployment Guide](../../quickstart/http-transport.md#production-deployment) --- @@ -321,3 +285,12 @@ curl http://localhost:8080/v1/chat/completions \ 4. **[๐Ÿš€ Deploy to Production](../../quickstart/http-transport.md#production-deployment)** - Scale for production workloads > **๐Ÿ›๏ธ Architecture:** For HTTP transport design and performance details, see [Architecture Documentation](../../architecture/README.md). + +--- + +## ๐Ÿ“š Additional Resources + +- [Configuration Guide](./configuration/providers.md) +- [API Endpoints](./endpoints.md) +- [Error Handling](../errors.md) +- [Monitoring & Metrics](./configuration/plugins.md) diff --git a/docs/usage/memory-management.md b/docs/usage/memory-management.md index a5d087e737..27acfc0041 100644 --- a/docs/usage/memory-management.md +++ b/docs/usage/memory-management.md @@ -112,10 +112,13 @@ bifrost, err := bifrost.Init(schemas.BifrostConfig{
๐ŸŒ HTTP Transport - Buffer Configuration -The buffer size is set in your `config.json`. Note that `dropExcessRequests` is not configurable for the HTTP transport and defaults to `false` (blocking). +The buffer size is set in your `config.json`. The `drop_excess_requests` setting can be configured in the `client` section and defaults to `false` (blocking). ```json { + "client": { + "drop_excess_requests": false + }, "providers": { "openai": { // ... @@ -141,7 +144,7 @@ Bifrost uses object pools to reuse request and response objects, reducing the lo - **Trade-offs**: - **Larger Pool**: Improves performance under heavy load by minimizing allocations. Increases the initial memory footprint of Bifrost. - **Smaller Pool**: Lower initial memory usage, but may lead to more GC activity and higher latency under load. -- **Configuration**: This is a global setting. For the Go package, it is set in `BifrostConfig`. For the HTTP transport, it's configured via command-line flags or environment variables, not in `config.json`. +- **Configuration**: This is a global setting. For the Go package, it is set in `BifrostConfig`. For the HTTP transport, it's configured in the `client` section of `config.json` or via the web UI.
๐Ÿ”ง Go Package - Object Pool Configuration @@ -162,27 +165,29 @@ bifrost, err := bifrost.Init(schemas.BifrostConfig{
๐ŸŒ HTTP Transport - Object Pool Configuration -The pool size for the HTTP transport is set at startup. +The pool size for the HTTP transport is configured in the `config.json` file under the `client` section or via the web UI. -**Using Go Binary** +**Using Config File** -Use the `-pool-size` command-line flag. - -```bash -bifrost-http -config config.json -port 8080 -pool-size 1000 +```json +{ + "client": { + "initial_pool_size": 1000, + "drop_excess_requests": false + }, + "providers": { + // ... provider configurations + } +} ``` -**Using Docker** +**Using Web UI** -Use the `APP_POOL_SIZE` environment variable. - -```bash -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e APP_POOL_SIZE=1000 \ - -e OPENAI_API_KEY \ - maximhq/bifrost -``` +1. Start Bifrost: `docker run -p 8080:8080 maximhq/bifrost` +2. Open `http://localhost:8080` +3. Navigate to the "Configuration" section +4. Set "Initial Pool Size" and other client settings +5. Save configuration
diff --git a/transports/Dockerfile b/transports/Dockerfile index 624ae6aff5..96b0dccea6 100644 --- a/transports/Dockerfile +++ b/transports/Dockerfile @@ -33,19 +33,26 @@ RUN test -f /app/main || (echo "Build failed" && exit 1) FROM alpine:3.19 WORKDIR /app -# Copy necessary files and create structure in one layer +# Create data directory and set up user COPY --from=builder /app/main . -RUN mkdir -p /app/config && \ +COPY docker-entrypoint.sh . +RUN mkdir -p /app/data/logs && \ adduser -D -s /bin/sh appuser && \ - chown -R appuser:appuser /app + chown -R appuser:appuser /app && \ + chmod +x /app/docker-entrypoint.sh USER appuser # Environment variables with defaults ENV APP_PORT=8080 \ - APP_POOL_SIZE=300 \ APP_PLUGINS="" +# Declare volume for data persistence +VOLUME ["/app/data"] EXPOSE 8080 -# Direct entrypoint with environment variable expansion -ENTRYPOINT ["/bin/sh", "-c", "exec /app/main -config /app/config/config.json -port \"${APP_PORT}\" -pool-size \"${APP_POOL_SIZE}\" -plugins \"${APP_PLUGINS}\""] \ No newline at end of file +# Health check for container status monitoring +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:${APP_PORT}/metrics || exit 1 + +# Use entrypoint script that handles volume permissions +ENTRYPOINT ["/bin/sh", "-c", "/app/docker-entrypoint.sh -app-dir /app/data -port ${APP_PORT} -plugins \"${APP_PLUGINS}\""] \ No newline at end of file diff --git a/transports/README.md b/transports/README.md index b8bc37943b..c4660607d8 100644 --- a/transports/README.md +++ b/transports/README.md @@ -13,18 +13,35 @@ Bifrost Transports let you run Bifrost as a blazing-fast HTTP API or integrate i ```bash # Pull and run Bifrost HTTP API docker pull maximhq/bifrost -docker run -p 8080:8080 \ - -v $(pwd)/config.json:/app/config/config.json \ - -e OPENAI_API_KEY \ - maximhq/bifrost +docker run -p 8080:8080 maximhq/bifrost + +# ๐Ÿ–ฅ๏ธ Open web interface for visual configuration +# macOS: +open http://localhost:8080 +# Linux: +xdg-open http://localhost:8080 +# Windows: +start http://localhost:8080 +# Or simply open http://localhost:8080 manually in your browser ``` +**๐ŸŽ‰ That's it!** No config files needed. Configure providers, monitor requests, and manage everything through the **built-in web interface**. + ### Go Binary ```bash # Install and run locally (Make sure Go is in your PATH) go install github.com/maximhq/bifrost/transports/bifrost-http@latest -bifrost-http -config config.json -port 8080 +bifrost-http -port 8080 + +# Open web interface at http://localhost:8080 +``` + +### Volume Mount (Optional) + +```bash +# For configuration persistence across restarts +docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost ``` **Ready in 30 seconds!** See [HTTP Transport Quickstart](../docs/quickstart/http-transport.md) for detailed setup. @@ -35,6 +52,7 @@ bifrost-http -config config.json -port 8080 | Feature | Description | Learn More | | ----------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------- | +| **๐Ÿ–ฅ๏ธ Built-in Web UI** | Visual configuration, live monitoring, request logs, and analytics | Open `http://localhost:8080` after startup | | **๐Ÿ”„ Multi-Provider Support** | OpenAI, Anthropic, Bedrock, Vertex, Cohere, Mistral, Ollama | [Provider Setup](../docs/usage/providers.md) | | **๐Ÿ”Œ Drop-in Compatibility** | Replace OpenAI/Anthropic/GenAI APIs with zero code changes | [Integrations](../docs/usage/http-transport/integrations/) | | **๐Ÿ› ๏ธ MCP Tool Calling** | Enable AI models to use external tools (filesystem, web, databases) | [MCP Guide](../docs/mcp.md) | @@ -97,10 +115,59 @@ curl -X POST http://localhost:8080/v1/chat/completions \ ## โš™๏ธ Configuration -### Minimal Config +Bifrost supports **two configuration modes**: + +### 1. ๐Ÿš€ Dynamic Configuration (Recommended) + +**No config file needed!** Start the container and configure via **Web UI** or **API**: + +```bash +# Start with zero configuration +docker run -p 8080:8080 maximhq/bifrost + +# Open web interface for easy configuration +# macOS: open http://localhost:8080 +# Linux: xdg-open http://localhost:8080 +# Windows: start http://localhost:8080 +# Or simply open http://localhost:8080 manually in your browser +``` + +๐Ÿ–ฅ๏ธ **Web UI Features:** + +- **Visual provider setup** - Add OpenAI, Anthropic, Bedrock, etc. +- **Real-time configuration** - Changes apply immediately +- **Live monitoring** - Request logs, metrics, and analytics +- **Export/Import** - Save configurations as JSON + +๐Ÿ“ก **Or configure via API:** + +```bash +# Add providers programmatically +curl -X POST http://localhost:8080/providers \ + -H "Content-Type: application/json" \ + -d '{ + "provider": "openai", + "keys": [{"value": "env.OPENAI_API_KEY", "models": ["gpt-4o-mini"], "weight": 1.0}], + "network_config": {"default_request_timeout_in_seconds": 30}, + "concurrency_and_buffer_size": {"concurrency": 3, "buffer_size": 10} + }' + +# Save configuration to file for persistence +curl -X POST http://localhost:8080/config/save +``` + +**Benefits**: Perfect for containerized deployments, GitOps workflows, and both visual and API-first configuration management. + +### 2. ๐Ÿ“„ File-based Configuration + +Traditional config file approach _(Volume mount needed when using docker)_: ```json { + "client": { + "drop_excess_requests": false, + "prometheus_labels": ["model", "provider"] + }, "providers": { "openai": { "keys": [ @@ -109,12 +176,22 @@ curl -X POST http://localhost:8080/v1/chat/completions \ "models": ["gpt-4o-mini"], "weight": 1.0 } - ] + ], + "concurrency_and_buffer_size": { + "concurrency": 3, + "buffer_size": 10 + } } } } ``` +**Client Configuration Options:** + +- `drop_excess_requests`: Whether to drop requests when queues are full (default: `false`) +- `initial_pool_size`: Initial connection pool size (default: `300`) +- `prometheus_labels`: Custom labels for Prometheus metrics + **Learn More:** - [Provider Setup Guide](../docs/usage/http-transport/configuration/providers.md) @@ -145,7 +222,7 @@ Add custom middleware for analytics, caching, rate limiting: ```bash # Run with plugins -bifrost-http -config config.json -plugins "maxim,redis" +bifrost-http -app-dir . -plugins "maxim,redis" ``` **Available plugins**: [Plugin Repository](https://github.com/maximhq/bifrost/tree/main/plugins) | [Plugin Guide](../docs/plugins.md) @@ -158,7 +235,7 @@ Built-in metrics collection at `/metrics`: curl http://localhost:8080/metrics ``` -**Custom labels**: `-prometheus-labels team-id,task-id` +**Custom labels**: Configure via `prometheus_labels` in config file or web UI --- @@ -166,22 +243,30 @@ curl http://localhost:8080/metrics ### For Go Binary -| Flag | Default | Description | -| -------------------- | ------- | --------------------------- | -| `-config` | - | Path to config.json file | -| `-port` | 8080 | HTTP server port | -| `-pool-size` | 300 | Connection pool size | -| `-plugins` | - | Comma-separated plugin list | -| `-prometheus-labels` | - | Custom Prometheus labels | +| Flag | Default | Description | +| ---------- | ------- | ---------------------------------------- | +| `-app-dir` | . | Application data directory (config+logs) | +| `-port` | 8080 | HTTP server port | +| `-plugins` | - | Comma-separated plugin list | + +### Understanding App Directory & Docker Volumes + +> **๐Ÿ“– Detailed Guide:** See [Understanding App Directory & Docker Volumes](../docs/quickstart/http-transport.md#understanding-app-directory--docker-volumes) for complete details on data persistence and Docker deployment. + +**Quick Reference:** + +- Default app directory: Current working directory (`.`) +- Docker volume mount: `-v $(pwd)/data:/app/data` +- Key files: `config.json`, `logs/` + +For complete setup instructions, deployment scenarios, and best practices, see the [detailed guide](../docs/quickstart/http-transport.md#understanding-app-directory--docker-volumes). ### For Docker -| Variable | Description | -| -------------------------- | ------------------------------------- | -| `APP_PORT` | Server port override | -| `APP_POOL_SIZE` | Pool size override | -| `APP_PLUGINS` | Plugin list override | -| `APP_DROP_EXCESS_REQUESTS` | Drop excess requests when buffer full | +| Variable | Description | +| ------------- | -------------------- | +| `APP_PORT` | Server port override | +| `APP_PLUGINS` | Plugin list override | --- diff --git a/transports/bifrost-http/lib/config.go b/transports/bifrost-http/lib/config.go index 80d0fe000a..b441618d38 100644 --- a/transports/bifrost-http/lib/config.go +++ b/transports/bifrost-http/lib/config.go @@ -10,8 +10,8 @@ import ( // It includes settings for excess request handling, Prometheus metrics, and initial pool size. type ClientConfig struct { DropExcessRequests bool `json:"drop_excess_requests"` - PrometheusLabels []string `json:"prometheus_labels"` InitialPoolSize int `json:"initial_pool_size"` + PrometheusLabels []string `json:"prometheus_labels"` } // ProviderConfig represents the configuration for a specific AI model provider. diff --git a/transports/bifrost-http/lib/store.go b/transports/bifrost-http/lib/store.go index ac27a14126..dda9dda50a 100644 --- a/transports/bifrost-http/lib/store.go +++ b/transports/bifrost-http/lib/store.go @@ -62,12 +62,16 @@ func NewConfigStore(logger schemas.Logger) (*ConfigStore, error) { // with full preprocessing including environment variable resolution and meta config parsing. // All processing is done upfront to ensure zero latency when retrieving data. // +// If the config file doesn't exist, the system starts with default configuration +// and users can add providers dynamically via the HTTP API. +// // This method handles: // - JSON config file parsing // - Environment variable substitution for API keys (env.VARIABLE_NAME) // - Provider-specific meta config processing (Azure, Bedrock, Vertex) // - Case conversion for provider names (e.g., "OpenAI" -> "openai") // - In-memory storage for ultra-fast access during request processing +// - Graceful handling of missing config files func (s *ConfigStore) LoadFromConfig(configPath string) error { s.mu.Lock() defer s.mu.Unlock() @@ -75,9 +79,27 @@ func (s *ConfigStore) LoadFromConfig(configPath string) error { s.configPath = configPath s.logger.Info(fmt.Sprintf("Loading configuration from: %s", configPath)) - // Read and parse the JSON config file + // Check if config file exists data, err := os.ReadFile(configPath) if err != nil { + if os.IsNotExist(err) { + s.logger.Info(fmt.Sprintf("Config file %s not found, starting with default configuration. Providers can be added dynamically via API.", configPath)) + + // Initialize with default configuration + s.ClientConfig = ClientConfig{ + DropExcessRequests: false, + PrometheusLabels: []string{}, + InitialPoolSize: 300, + } + s.Providers = make(map[schemas.ModelProvider]ProviderConfig) + s.MCPConfig = nil + + // Auto-detect and configure providers from common environment variables + s.autoDetectProviders() + + s.logger.Info("Successfully initialized with default configuration.") + return nil + } return fmt.Errorf("failed to read config file: %w", err) } @@ -92,92 +114,103 @@ func (s *ConfigStore) LoadFromConfig(configPath string) error { return fmt.Errorf("failed to unmarshal config: %w", err) } - // Process core configuration if present + // Process core configuration if present, otherwise use defaults if len(configData.Client) > 0 { var clientConfig ClientConfig if err := json.Unmarshal(configData.Client, &clientConfig); err != nil { return fmt.Errorf("failed to unmarshal client config: %w", err) } s.ClientConfig = clientConfig + } else { + // Default client configuration + s.ClientConfig = ClientConfig{ + DropExcessRequests: false, + PrometheusLabels: []string{}, + InitialPoolSize: 300, + } } // Process provider configurations processedProviders := make(map[schemas.ModelProvider]ProviderConfig) - // First unmarshal providers into a map with string keys to handle case conversion - var rawProviders map[string]ProviderConfig - if providersBytes, err := json.Marshal(configData.Providers); err != nil { - return fmt.Errorf("failed to marshal providers: %w", err) - } else if err := json.Unmarshal(providersBytes, &rawProviders); err != nil { - return fmt.Errorf("failed to unmarshal providers: %w", err) - } + if len(configData.Providers) > 0 { + // First unmarshal providers into a map with string keys to handle case conversion + var rawProviders map[string]ProviderConfig + if providersBytes, err := json.Marshal(configData.Providers); err != nil { + return fmt.Errorf("failed to marshal providers: %w", err) + } else if err := json.Unmarshal(providersBytes, &rawProviders); err != nil { + return fmt.Errorf("failed to unmarshal providers: %w", err) + } - // Create a temporary structure to unmarshal the full JSON with proper meta configs - var tempConfig struct { - Providers map[string]struct { - MetaConfig json.RawMessage `json:"meta_config"` - } `json:"providers"` - } + // Create a temporary structure to unmarshal the full JSON with proper meta configs + var tempConfig struct { + Providers map[string]struct { + MetaConfig json.RawMessage `json:"meta_config"` + } `json:"providers"` + } - if err := json.Unmarshal(data, &tempConfig); err != nil { - return fmt.Errorf("failed to unmarshal configuration file: %w", err) - } + if err := json.Unmarshal(data, &tempConfig); err != nil { + return fmt.Errorf("failed to unmarshal configuration file: %w", err) + } - // Process each provider configuration - for rawProviderName, cfg := range rawProviders { - newEnvKeys := make(map[string]struct{}) + // Process each provider configuration + for rawProviderName, cfg := range rawProviders { + newEnvKeys := make(map[string]struct{}) - provider := schemas.ModelProvider(strings.ToLower(rawProviderName)) + provider := schemas.ModelProvider(strings.ToLower(rawProviderName)) - // Process meta config if it exists - if tempProvider, exists := tempConfig.Providers[rawProviderName]; exists && len(tempProvider.MetaConfig) > 0 { - processedMetaConfig, envKeys, err := s.processMetaConfigEnvVars(tempProvider.MetaConfig, provider) + // Process meta config if it exists + if tempProvider, exists := tempConfig.Providers[rawProviderName]; exists && len(tempProvider.MetaConfig) > 0 { + processedMetaConfig, envKeys, err := s.processMetaConfigEnvVars(tempProvider.MetaConfig, provider) - if err != nil { - s.cleanupEnvKeys(string(provider), "", envKeys) - s.logger.Warn(fmt.Sprintf("failed to process env vars in meta config for %s: %v", provider, err)) - continue - } + if err != nil { + s.cleanupEnvKeys(string(provider), "", envKeys) + s.logger.Warn(fmt.Sprintf("failed to process env vars in meta config for %s: %v", provider, err)) + continue + } - // Parse and set the meta config - metaConfig, err := s.parseMetaConfig(processedMetaConfig, provider) - if err != nil { - s.cleanupEnvKeys(string(provider), "", envKeys) - s.logger.Warn(fmt.Sprintf("failed to process meta config for %s: %v", provider, err)) - continue - } else { - cfg.MetaConfig = metaConfig + // Parse and set the meta config + metaConfig, err := s.parseMetaConfig(processedMetaConfig, provider) + if err != nil { + s.cleanupEnvKeys(string(provider), "", envKeys) + s.logger.Warn(fmt.Sprintf("failed to process meta config for %s: %v", provider, err)) + continue + } else { + cfg.MetaConfig = metaConfig + } } - } - // Process environment variables in keys - for i, key := range cfg.Keys { - processedValue, envVar, err := s.processEnvValue(key.Value) - if err != nil { - s.cleanupEnvKeys(string(provider), "", newEnvKeys) - s.logger.Warn(fmt.Sprintf("failed to process env vars in keys for %s: %v", provider, err)) - continue + // Process environment variables in keys + for i, key := range cfg.Keys { + processedValue, envVar, err := s.processEnvValue(key.Value) + if err != nil { + s.cleanupEnvKeys(string(provider), "", newEnvKeys) + s.logger.Warn(fmt.Sprintf("failed to process env vars in keys for %s: %v", provider, err)) + continue + } + cfg.Keys[i].Value = processedValue + + // Track environment key if it came from env + if envVar != "" { + newEnvKeys[envVar] = struct{}{} + s.EnvKeys[envVar] = append(s.EnvKeys[envVar], EnvKeyInfo{ + EnvVar: envVar, + Provider: string(provider), + KeyType: "api_key", + ConfigPath: fmt.Sprintf("providers.%s.keys[%d]", provider, i), + }) + } } - cfg.Keys[i].Value = processedValue - // Track environment key if it came from env - if envVar != "" { - newEnvKeys[envVar] = struct{}{} - s.EnvKeys[envVar] = append(s.EnvKeys[envVar], EnvKeyInfo{ - EnvVar: envVar, - Provider: string(provider), - KeyType: "api_key", - ConfigPath: fmt.Sprintf("providers.%s.keys[%d]", provider, i), - }) - } + processedProviders[provider] = cfg } - processedProviders[provider] = cfg + // Store processed configurations in memory + s.Providers = processedProviders + } else { + s.autoDetectProviders() } - // Store processed configurations in memory - s.Providers = processedProviders - // Parse MCP config if present if len(configData.MCP) > 0 { var mcpConfig schemas.MCPConfig @@ -1171,3 +1204,66 @@ func (s *ConfigStore) cleanupEnvVar(envVar, provider, mcpClientName string) { s.EnvKeys[envVar] = filteredInfos } } + +// autoDetectProviders automatically detects common environment variables and sets up providers +// when no configuration file exists. This enables zero-config startup when users have set +// standard environment variables like OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. +// +// Supported environment variables: +// - OpenAI: OPENAI_API_KEY, OPENAI_KEY +// - Anthropic: ANTHROPIC_API_KEY, ANTHROPIC_KEY +// - Mistral: MISTRAL_API_KEY, MISTRAL_KEY +// +// For each detected provider, it creates a default configuration with: +// - The detected API key with weight 1.0 +// - Empty models list (provider will use default models) +// - Default concurrency and buffer size settings +func (s *ConfigStore) autoDetectProviders() { + // Define common environment variable patterns for each provider + providerEnvVars := map[schemas.ModelProvider][]string{ + schemas.OpenAI: {"OPENAI_API_KEY", "OPENAI_KEY"}, + schemas.Anthropic: {"ANTHROPIC_API_KEY", "ANTHROPIC_KEY"}, + schemas.Mistral: {"MISTRAL_API_KEY", "MISTRAL_KEY"}, + } + + detectedCount := 0 + + for provider, envVars := range providerEnvVars { + for _, envVar := range envVars { + if apiKey := os.Getenv(envVar); apiKey != "" { + // Create default provider configuration + providerConfig := ProviderConfig{ + Keys: []schemas.Key{ + { + Value: apiKey, + Models: []string{}, // Empty means all supported models + Weight: 1.0, + }, + }, + ConcurrencyAndBufferSize: &schemas.DefaultConcurrencyAndBufferSize, + } + + // Add to providers map + s.Providers[provider] = providerConfig + + // Track the environment variable + s.EnvKeys[envVar] = append(s.EnvKeys[envVar], EnvKeyInfo{ + EnvVar: envVar, + Provider: string(provider), + KeyType: "api_key", + ConfigPath: fmt.Sprintf("providers.%s.keys[0]", provider), + }) + + s.logger.Info(fmt.Sprintf("Auto-detected %s provider from environment variable %s", provider, envVar)) + detectedCount++ + break // Only use the first found env var for each provider + } + } + } + + if detectedCount > 0 { + s.logger.Info(fmt.Sprintf("Auto-configured %d provider(s) from environment variables", detectedCount)) + } else { + s.logger.Info("No common provider environment variables detected. Use the web UI or configuration file to add providers.") + } +} diff --git a/transports/bifrost-http/main.go b/transports/bifrost-http/main.go index b1dc930b7f..b8105f4dce 100644 --- a/transports/bifrost-http/main.go +++ b/transports/bifrost-http/main.go @@ -8,9 +8,9 @@ // - /providers/*: For provider configuration management // // Configuration is handled through a JSON config file, high-performance ConfigStore, and environment variables: -// - Use -config flag to specify the config file location +// - Use -app-dir flag to specify the application data directory (contains config.json and logs) // - Use -port flag to specify the server port (default: 8080) -// - Use -pool-size flag to specify the initial connection pool size (default: 300) +// - When no config file exists, common environment variables are auto-detected (OPENAI_API_KEY, ANTHROPIC_API_KEY, MISTRAL_API_KEY) // // ConfigStore Features: // - Pure in-memory storage for ultra-fast config access @@ -29,8 +29,8 @@ // // Example usage: // -// go run main.go -config config.example.json -port 8080 -pool-size 300 -// after setting the environment variables present in config.example.json in the environment. +// go run main.go -app-dir ./data -port 8080 +// after setting provider API keys like OPENAI_API_KEY in the environment. // // Integration Support: // Bifrost supports multiple AI provider integrations through dedicated HTTP endpoints. @@ -52,7 +52,6 @@ package main import ( "embed" "flag" - "fmt" "log" "mime" "os" @@ -81,28 +80,24 @@ var uiContent embed.FS // Command line flags var ( port string // Port to run the server on - configPath string // Path to the config file - pluginsToLoad []string // Path to the plugins + appDir string // Application data directory + pluginsToLoad []string // Plugins to load ) // init initializes command line flags and validates required configuration. // It sets up the following flags: -// - pool-size: Initial connection pool size (default: 300) // - port: Server port (default: 8080) -// - config: Path to config file (required) +// - app-dir: Application data directory (default: current directory) +// - plugins: Comma-separated list of plugins to load func init() { pluginString := "" flag.StringVar(&port, "port", "8080", "Port to run the server on") - flag.StringVar(&configPath, "config", "", "Path to the config file") + flag.StringVar(&appDir, "app-dir", ".", "Application data directory (contains config.json and logs)") flag.StringVar(&pluginString, "plugins", "", "Comma separated list of plugins to load") flag.Parse() pluginsToLoad = strings.Split(pluginString, ",") - - if configPath == "" { - log.Fatalf("config path is required") - } } // registerCollectorSafely attempts to register a Prometheus collector, @@ -238,6 +233,15 @@ func uiHandler(ctx *fasthttp.RequestCtx) { // - POST /v1/chat/completions: For chat completion requests // - GET /metrics: For Prometheus metrics func main() { + // Ensure app directory exists + if err := os.MkdirAll(appDir, 0755); err != nil { + log.Fatalf("failed to create app directory %s: %v", appDir, err) + } + + // Derive paths from app-dir + configPath := filepath.Join(appDir, "config.json") + logDir := filepath.Join(appDir, "logs") + // Register Prometheus collectors registerCollectorSafely(collectors.NewGoCollector()) registerCollectorSafely(collectors.NewProcessCollector(collectors.ProcessCollectorOpts{})) @@ -253,7 +257,7 @@ func main() { // Load configuration from JSON file into the store with full preprocessing // This processes environment variables and stores all configurations in memory for ultra-fast access if err := store.LoadFromConfig(configPath); err != nil { - log.Fatalf("failed to load config into store: %v", err) + log.Fatalf("failed to load config: %v", err) } // Create account backed by the high-performance store (all processing is done in LoadFromConfig) @@ -288,7 +292,14 @@ func main() { log.Println("Prometheus Go/Process collectors registered.") promPlugin := telemetry.NewPrometheusPlugin() - loggingPlugin, err := logging.NewLoggerPlugin(nil) + + // Initialize logging plugin with app-dir based path + loggingConfig := &logging.Config{ + DatabasePath: logDir, + LogQueueSize: 1000, + } + + loggingPlugin, err := logging.NewLoggerPlugin(loggingConfig, logger) if err != nil { log.Fatalf("failed to initialize logging plugin: %v", err) } @@ -346,22 +357,12 @@ func main() { handlers.SendError(ctx, fasthttp.StatusNotFound, "Route not found: "+string(ctx.Path()), logger) } - server := &fasthttp.Server{ - // A custom handler that applies CORS to all routes - Handler: func(ctx *fasthttp.RequestCtx) { - if string(ctx.Path()) == "/metrics" { - // Apply CORS even to metrics endpoint for monitoring dashboards - corsMiddleware(r.Handler)(ctx) - return - } - // Apply CORS middleware for all API routes - corsMiddleware(telemetry.PrometheusMiddleware(r.Handler))(ctx) - }, - } + // Apply CORS middleware to all routes + corsHandler := corsMiddleware(r.Handler) - log.Println("Started Bifrost HTTP server on port", port) - if err := server.ListenAndServe(fmt.Sprintf(":%s", port)); err != nil { - log.Fatalf("failed to start server: %v", err) + log.Printf("Starting server on port %s", port) + if err := fasthttp.ListenAndServe(":"+port, corsHandler); err != nil { + log.Fatalf("Error starting server: %v", err) } wsHandler.Stop() diff --git a/transports/bifrost-http/plugins/logging/main.go b/transports/bifrost-http/plugins/logging/main.go index 58320fdf39..f20ef61b8b 100644 --- a/transports/bifrost-http/plugins/logging/main.go +++ b/transports/bifrost-http/plugins/logging/main.go @@ -111,6 +111,7 @@ type LogStats struct { // Config represents the configuration for the logging plugin type Config struct { DatabasePath string `json:"database_path"` + LogQueueSize int `json:"log_queue_size"` } // LogCallback is a function that gets called when a new log entry is created @@ -125,14 +126,16 @@ type LoggerPlugin struct { logQueue chan *LogEntry done chan struct{} wg sync.WaitGroup + logger schemas.Logger logCallback LogCallback // Callback for real-time log updates } // NewLoggerPlugin creates a new logging plugin -func NewLoggerPlugin(config *Config) (*LoggerPlugin, error) { +func NewLoggerPlugin(config *Config, logger schemas.Logger) (*LoggerPlugin, error) { if config == nil { config = &Config{ - DatabasePath: "./badger_logs", + DatabasePath: "./bifrost-logs", + LogQueueSize: 1000, } } @@ -148,8 +151,9 @@ func NewLoggerPlugin(config *Config) (*LoggerPlugin, error) { plugin := &LoggerPlugin{ config: config, db: db, - logQueue: make(chan *LogEntry, 1000), // Buffer for 1000 log entries + logQueue: make(chan *LogEntry, config.LogQueueSize), // Buffer for 1000 log entries done: make(chan struct{}), + logger: logger, stats: &LogStats{ ProviderStats: make(map[string]int64), ModelStats: make(map[string]int64), @@ -175,9 +179,9 @@ func (p *LoggerPlugin) processLogEntry(entry *LogEntry, isShutdown bool) { // Store the log entry if err := p.storeLogEntry(entry); err != nil { if isShutdown { - fmt.Printf("BadgerDB Logger: failed to store log entry during shutdown: %v\n", err) + p.logger.Error(fmt.Errorf("failed to store log entry during shutdown: %w", err)) } else { - fmt.Printf("BadgerDB Logger: failed to store log entry: %v\n", err) + p.logger.Error(fmt.Errorf("failed to store log entry: %w", err)) } } @@ -366,7 +370,7 @@ func (p *LoggerPlugin) PostHook(ctx *context.Context, result *schemas.BifrostRes // Successfully queued default: // Queue is full, log warning but don't block the request - fmt.Printf("Logger: log queue is full, dropping log entry\n") + p.logger.Warn("log queue is full, dropping log entry") } return result, err, nil diff --git a/transports/config.example.json b/transports/config.example.json index 51967ff6e5..1c17c62529 100644 --- a/transports/config.example.json +++ b/transports/config.example.json @@ -1,6 +1,7 @@ { "client": { "drop_excess_requests": false, + "initial_pool_size": 500, "prometheus_labels": ["model", "provider"] }, "providers": { diff --git a/transports/docker-entrypoint.sh b/transports/docker-entrypoint.sh new file mode 100644 index 0000000000..be2c679b20 --- /dev/null +++ b/transports/docker-entrypoint.sh @@ -0,0 +1,40 @@ +#!/bin/sh +set -e + +# Function to fix permissions on mounted volumes +fix_permissions() { + # Check if /app/data exists and fix ownership if needed + if [ -d "/app/data" ]; then + # Get current user info + CURRENT_UID=$(id -u) + CURRENT_GID=$(id -g) + + # Get directory ownership + DATA_UID=$(stat -c %u /app/data 2>/dev/null || echo "0") + DATA_GID=$(stat -c %g /app/data 2>/dev/null || echo "0") + + # If ownership doesn't match current user, try to fix it + if [ "$DATA_UID" != "$CURRENT_UID" ] || [ "$DATA_GID" != "$CURRENT_GID" ]; then + echo "Fixing permissions on /app/data (was $DATA_UID:$DATA_GID, setting to $CURRENT_UID:$CURRENT_GID)" + + # Try to change ownership (will work if running as root or if user has permission) + if chown -R "$CURRENT_UID:$CURRENT_GID" /app/data 2>/dev/null; then + echo "Successfully updated permissions on /app/data" + else + echo "Warning: Could not change ownership of /app/data. You may need to run:" + echo " docker run --user \$(id -u):\$(id -g) ..." + echo " or ensure the host directory is owned by UID:GID $CURRENT_UID:$CURRENT_GID" + fi + fi + + # Ensure logs subdirectory exists with correct permissions + mkdir -p /app/data/logs + chmod 755 /app/data/logs 2>/dev/null || true + fi +} + +# Fix permissions before starting the application +fix_permissions + +# Execute the main application with all passed arguments +exec /app/main "$@" \ No newline at end of file