Skip to content

Latest commit

 

History

History
169 lines (146 loc) · 7.52 KB

README.md

File metadata and controls

169 lines (146 loc) · 7.52 KB
Raw

BricksLLM: AI Gateway For Putting LLMs In Production

YCombinator S22 PRs Welcome Join BricksLLM on Discord License

Tip

A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:

  • Set LLM usage limits for users on different pricing tiers
  • Track LLM usage on a per user and per organization basis
  • Block or redact requests containing PIIs
  • Improve LLM reliability with failovers, retries and caching
  • Distribute API keys with rate limits and cost limits for internal development/production use cases
  • Distribute API keys with rate limits and cost limits for students

Features

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker compose up

You can run this in detach mode use the -d flag: docker compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name type description default
POSTGRESQL_HOSTS required Hosts for Postgresql DB. Separated by , localhost
POSTGRESQL_DB_NAME optional Name for Postgresql DB.
POSTGRESQL_USERNAME required Postgresql DB username
POSTGRESQL_PASSWORD required Postgresql DB password
POSTGRESQL_SSL_MODE optional Postgresql SSL mode disable
POSTGRESQL_PORT optional The port that Postgresql DB runs on 5432
POSTGRESQL_READ_TIME_OUT optional Timeout for Postgresql read operations 2m
POSTGRESQL_WRITE_TIME_OUT optional Timeout for Postgresql write operations 5s
REDIS_HOSTS required Host for Redis. Separated by , localhost
REDIS_PASSWORD optional Redis Password
REDIS_PORT optional The port that Redis DB runs on 6379
REDIS_READ_TIME_OUT optional Timeout for Redis read operations 1s
REDIS_WRITE_TIME_OUT optional Timeout for Redis write operations 500ms
IN_MEMORY_DB_UPDATE_INTERVAL optional The interval BricksLLM API gateway polls Postgresql DB for latest key configurations 1s
STATS_PROVIDER optional "datadog" or Host:Port(127.0.0.1:8125) for statsd.
PROXY_TIMEOUT optional Timeout for proxy HTTP requests. 600s
NUMBER_OF_EVENT_MESSAGE_CONSUMERS optional Number of event message consumers that help handle counting tokens and inserting event into db. 3
AWS_SECRET_ACCESS_KEY optional It is for PII detection feature. 5s
AWS_ACCESS_KEY_ID optional It is for using PII detection feature. 5s
AMAZON_REGION optional Region for AWS. us-west-2
AMAZON_REQUEST_TIMEOUT optional Timeout for amazon requests. 5s
AMAZON_CONNECTION_TIMEOUT optional Timeout for amazon connection. 10s
ADMIN_PASS optional Simple password for the admin server.

Admin Server

Swagger Doc

Proxy Server

Swagger Doc