BricksLLM: AI Gateway For Putting LLM In Production

Tip

A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:

Set LLM usage limits for users on different pricing tiers
Track LLM usage on a per user and per organization basis
Block or redact requests containing PIIs
Improve LLM reliability with failovers, retries and caching
Distribute API keys with rate limits and cost limits for internal development/production use cases
Distribute API keys with rate limits and cost limits for students

Features

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker-compose up

You can run this in detach mode use the -d flag: docker-compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name type description default

POSTGRESQL_HOSTS required Hosts for Postgresql DB. Separated by , localhost

POSTGRESQL_DB_NAME optional Name for Postgresql DB.

POSTGRESQL_USERNAME required Postgresql DB username

POSTGRESQL_PASSWORD required Postgresql DB password

POSTGRESQL_SSL_MODE optional Postgresql SSL mode disable

POSTGRESQL_PORT optional The port that Postgresql DB runs on 5432

POSTGRESQL_READ_TIME_OUT optional Timeout for Postgresql read operations 2m

POSTGRESQL_WRITE_TIME_OUT optional Timeout for Postgresql write operations 5s

REDIS_HOSTS required Host for Redis. Separated by , localhost

REDIS_PASSWORD optional Redis Password

REDIS_PORT optional The port that Redis DB runs on 6379

REDIS_READ_TIME_OUT optional Timeout for Redis read operations 1s

REDIS_WRITE_TIME_OUT optional Timeout for Redis write operations 500ms

IN_MEMORY_DB_UPDATE_INTERVAL optional The interval BricksLLM API gateway polls Postgresql DB for latest key configurations 1s

STATS_PROVIDER optional This value can only be datadog. Required for integration with Datadog.

PROXY_TIMEOUT optional Timeout for proxy HTTP requests. 600s

NUMBER_OF_EVENT_MESSAGE_CONSUMERS optional Number of event message consumers that help handle counting tokens and inserting event into db. 3

AWS_SECRET_ACCESS_KEY optional It is for PII detection feature. 5s

AWS_ACCESS_KEY_ID optional It is for using PII detection feature. 5s

AMAZON_REGION optional Region for AWS. us-west-2

AMAZON_REQUEST_TIMEOUT optional Timeout for amazon requests. 5s

AMAZON_CONNECTION_TIMEOUT optional Timeout for amazon connection. 10s

AWS_SECRET_ACCESS_KEY optional Required for PII detection features.

AWS_ACCESS_KEY_ID optional Required for PII detection features.

Configuration Endpoints

The configuration server runs on Port 8001.

Headers

name type data type description

X-API-KEY optional string Key authentication header.

Get keys: GET /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Query Parameters

name type data type description

tag optional string Identifier attached to a key configuration

tags optional []string Identifiers attached to a key configuration

provider optional string Provider attached to a key provider configuration. Its value can only be openai.

keyIds optional []string Unique identifiers for keys.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/key-management/keys

Response

Response Body

[]KeyConfiguration

KeyConfiguration

Field type example description

name string spike's developer key Name of the API key.

createdAt int64 1257894000 Key configuration creation time in unix.

updatedAt int64 1257894000 Key configuration update time in unix.

revoked boolean true Indicator for whether the key is revoked.

revokedReason string The key has expired Reason for why the key is revoked.

tags []string ["org-tag-12345"] Identifiers associated with the key.

keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.

costLimitInUsd float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] Allowed paths that can be accessed using the key.

settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.

shouldLogRequest bool false Should request be stored.

shouldLogResponse bool true Should response be stored.

rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed bool false Flag controls whether or not the key should be hashed.

Get keys V2: POST /api/v2/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request

Field required type example description

keyIds optional []string ["98daa3ae-961d-4253-bf6a-322a32fdca3d"] Array of key IDs.

tags optional []string ["org-tag-12345"] Identifiers associated with keys.

name optional string key Filter keys by the substring match on the name field.

revoked optional bool false Filters keys by revoked status.

limit optional int 5 Pagination limit.

offset optional int 5 Pagination offset.

order optional string asc asc and desc are the two available values.

returnCount optional bool true Flag that controls whether the response should contain total key counts.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/v2/key-management/keys

Response

Field type example description

keys []key Array of keys.

count int 123 Total number of keys.

key

Field type example description

name string spike's developer key Name of the API key.

createdAt int64 1257894000 Key configuration creation time in unix.

updatedAt int64 1257894000 Key configuration update time in unix.

revoked boolean true Indicator for whether the key is revoked.

revokedReason string The key has expired Reason for why the key is revoked.

tags []string ["org-tag-12345"] Identifiers associated with the key.

keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.

costLimitInUsd float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.

settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.

shouldLogRequest bool false Should request be stored.

shouldLogResponse bool true Should response be stored.

rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed bool false Flag controls whether or not the key should be hashed.

Create key: PUT /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request

PathConfig

Field required type example description

path required string /api/providers/openai/v1/chat/completion Allowed path

method required string POST HTTP Method

Field required type example description

name required string spike's developer key Name of the API key.

tags optional []string ["org-tag-12345"] Identifiers associated with the key.

key required string abcdef12345 API key.

settingId depercated string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds required string 98daa3ae-961d-4253-bf6a-322a32fdca3d Setting ids associated with the key.

costLimitInUsd optional float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime optional int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit optional enum m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl optional string 24h time to live. Available units are [s, m, h].

allowedPaths optional []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Paths allowed for access.

shouldLogRequest optional bool false Should request be stored.

shouldLogResponse optional bool true Should response be stored.

rotationEnabled optional bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed optional bool false Flag controls whether or not the key should be hashed.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/key-management/keys

Responses

Field type example description

name string spike's developer key Name of the API key.

createdAt int64 1257894000 Key configuration creation time in unix.

updatedAt int64 1257894000 Key configuration update time in unix.

revoked boolean true Indicator for whether the key is revoked.

revokedReason string The key has expired Reason for why the key is revoked.

tags []string ["org-tag-12345"] Identifiers associated with the key.

keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.

costLimitInUsd float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d].

ttl string 2d time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.

settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.

shouldLogRequest bool false Should request be stored.

shouldLogResponse bool true Should response be stored.

rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed bool false Flag controls whether or not the key should be hashed.

Update key: PATCH /api/key-management/keys/{keyId}

Description

This endpoint is set up for updating key configurations using key id.

Parameters

name type data type description

keyId required string Unique key configuration identifier.

Request

PathConfig

Field required type example description

path required string /api/providers/openai/v1/chat/completion Allowed path

method required string POST HTTP Method

Field required type example description

settingId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Setting ids associated with the key.

name optional string spike's developer key Name of the API key.

tags optional []string ["org-tag-12345"] Identifiers associated with the key.

revoked optional boolean true Indicator for whether the key is revoked.

revokedReason optional string The key has expired Reason for why the key is revoked.

costLimitInUsd optional float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime optional int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit optional string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

allowedPaths optional [{ "path": "/api/providers/openai/v1/chat/completions", "method": "POST"}] `` Paths allowed for access.

shouldLogRequest optional bool false Should request be stored.

shouldLogResponse optional bool true Should response be stored.

rotationEnabled optional bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed optional bool false Flag controls whether or not the key should be hashed.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/key-management/keys

Response

Field type example description

name string spike's developer key Name of the API key.

createdAt int64 1257894000 Key configuration creation time in unix.

updatedAt int64 1257894000 Key configuration update time in unix.

revoked boolean true Indicator for whether the key is revoked.

revokedReason string The key has expired Reason for why the key is revoked.

tags []string ["org-tag-12345"] Identifiers associated with the key.

keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.

costLimitInUsd float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 2d time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.

settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.

shouldLogRequest bool false Should request be stored.

shouldLogResponse bool true Should response be stored.

rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed bool false Flag controls whether or not the key should be hashed.

Get top spending key Ids: POST /api/reporting/top-keys

Description

This endpoint is set up for getting a list of key ids ordered by spend.

Request

Field required type example description

start required int64 1257894000 Start unix timestamp.

end required int64 1257894000 End unix timestamp

keyIds optional []string ["98daa3ae-961d-4253-bf6a-322a32fdca3d"] Array of key IDs.

tags optional []string ["org-tag-12345"] Identifiers associated with keys.

name optional string key Filter keys by the substring match on the name field.

revoked optional bool false Filters keys by revoked status.

limit optional int 5 Pagination limit.

offset optional int 5 Pagination offset.

order optional string asc asc and desc are the two available values.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/reporting/top-keys

Response

Field type example description

name string spike's developer key Name of the API key.

createdAt int64 1257894000 Key configuration creation time in unix.

updatedAt int64 1257894000 Key configuration update time in unix.

revoked boolean true Indicator for whether the key is revoked.

revokedReason string The key has expired Reason for why the key is revoked.

tags []string ["org-tag-12345"] Identifiers associated with the key.

keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.

costLimitInUsd float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 2d time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.

settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.

settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.

shouldLogRequest bool false Should request be stored.

shouldLogResponse bool true Should response be stored.

rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.

policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.

isKeyNotHashed bool false Flag controls whether or not the key should be hashed.

Create a provider setting: POST /api/provider-settings

Description

This endpoint is creating a provider setting.

Request

Field required type example description

provider required enum openai This value can only be openai, anthropic and azure as for now.

setting required Setting { "apikey": "YOUR_OPENAI_KEY" } A map of values used for authenticating with the selected provider.

name optional string YOUR_PROVIDER_SETTING_NAME This field is used for giving a name to provider setting

allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Setting

Field required type example description

apiKey required string xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx This value is required.

url required string https://your.deployment.url This value is required when the provider is vllm

resourceName required string YOUR_AZURE_RESOURCE_NAME This value is required when the provider is azure.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/provider-settings

Response

Field type example description

createdAt int64 1699933571 Unix timestamp for creation time.

updatedAt int64 1699933571 Unix timestamp for update time.

provider enum openai This value can only be openai as for now.

id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier.

name string YOUR_PROVIDER_SETTING_NAME Provider setting name.

allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Get provider settings: GET /api/provider-settings

Description

This endpoint is getting provider settings.

Query Parameters

name type data type description

ids optional []string Provider setting ids

Error Response

http code content-type

500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/provider-settings

Response

[]ProviderSetting

ProviderSetting

Field type example description

createdAt int64 1699933571 Unix timestamp for creation time.

updatedAt int64 1699933571 Unix timestamp for update time.

provider enum openai This value can only be openai as for now.

id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier.

name string YOUR_PROVIDER_SETTING_NAME Provider setting name.

allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Update a provider setting: PATCH /api/provider-settings/:id

Description

This endpoint is updating a provider setting .

Parameters

name type data type description

id required string Unique identifier for the provider setting that you want to update.

Request

Field required type example description

setting required Setting { "apikey": "YOUR_OPENAI_KEY" } A map of values used for authenticating with the selected provider.

name optional string YOUR_PROVIDER_SETTING_NAME This field is used for giving a name to provider setting

allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Setting

Field required type example description

apiKey required string xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx This value is required.

url required string https://your.deployment.url This value is required when the provider is vllm

resourceName required string YOUR_AZURE_RESOURCE_NAME This value is required when the provider is azure.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/provider-settings

Response

Field type example description

createdAt int64 1699933571 Unix timestamp for creation time.

updatedAt int64 1699933571 Unix timestamp for update time.

provider enum openai This value can only be openai as for now.

id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier

name string YOUR_PROVIDER_SETTING_NAME Provider setting name.

allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Retrieve Metrics: POST /api/reporting/events

Description

This endpoint is retrieving aggregated metrics given an array of key ids and tags.

Request

Field required type example description

keyIds required []string ["key-1", "key-2", "key-3" ] Array of ids that specicify the keys that you want to aggregate stats from.

tags required []string ["tag-1", "tag-2"] Array of tags that specicify the key tags that you want to aggregate stats from.

customIds required []string ["customId-1", "customId-2"] A list of custom IDs that you want to aggregate stats from.

filters required []string ["model", "keyId"] Group by data points through different filters(model,keyId, userId or customId).

start required int64 1699933571 Start timestamp for the requested timeseries data.

end required int64 1699933571 End timestamp for the requested timeseries data.

increment required int 60 This field is the increment in seconds for the requested timeseries data.

Error Response

http code content-type

500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/provider-settings

Response

Field type example description

dataPoints []dataPoint [{ "timeStamp": 1699933571, "numberOfRequests": 1, "costInUsd": 0.8, "latencyInMs": 600, "promptTokenCount": 0, "completionTokenCount": 0, "successCount": 1 }] Unix timestamp for creation time.

latencyInMsMedian float64 656.7 Median latency for the given time period.

latencyInMs99th float64 555.7 99th percentile latency for the given time period.

Datapoint

Field type example description

timeStamp int64 1702504746 Unix timestamp for the data point

numberOfRequests int64 100 Aggregated number of http requests over the given time increment.

costInUsd float64 1.7 Aggregated cost of proxied requests in USD over the given time increment.

latencyInMs int 555 Aggregated latency in milliseconds of http requests over the given time increment.

promptTokenCount int 25 Aggregated prompt token counts over the given time increment.

completionTokenCount int 4000 Aggregated completion token counts over the given time increment.

successCount int 555 Aggregated number of successful http requests over the given time increment.

keyId int 555.7 key Id associated with the event.

model string gpt-3.5-turbo model associated with the event.

customId string customId customId associated with the event.

Get events: GET /api/events

Description

This endpoint is for getting events.

Query Parameters

name type data type description

customId optional string Custom identifier attached to an event.

keyIds optional []string A list of key IDs.

start required if keyIds is specified int64 Start timestamp.

end required if keyIds is specified int64 End timestamp.

Error Response

http code content-type

500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/provider-settings

Response

[]Event

Event

Field type example description

id int64 1699933571 Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

tags int64 ["YOUR_TAG"] Tags of the key.

key_id string YOUR_KEY_ID Key Id associated with the proxy request.

cost_in_usd float64 0.0004 Cost incured by the proxy request.

model string gpt-4-1105-preview Model used in the proxy request.

provider string openai Provider for the proxy request.

status int 200 Http status.

prompt_token_count int 8 Prompt token count of the proxy request.

completion_token_count int 16 Completion token counts of the proxy request.

latency_in_ms int 160 Provider setting name.

path string /api/v1/chat/completion Provider setting name.

method string POST Http method for the assoicated proxu request.

custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.

request []byte {} Request in bytes to providers.

custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.

response []byte {} Response in bytes from providers.

Get events V2: GET /api/v2/events

Description

This endpoint retrieves a list of events based on provided filters.

Request

name type data type example description

userIds optional []string ["user1", "user2"] List of user IDs to filter events by specific users.

customIds optional []string ["id1", "id2"] List of custom identifiers for filtering events.

keyIds optional []string ["key1", "key2"] List of key IDs used to filter events.

tags optional []string ["tag1", "tag2"] List of tags associated with events for filtering.

start required int64 1609459200 Start timestamp for filtering events.

end required int64 1612137600 End timestamp for filtering events.

limit optional int 100 Maximum number of events to return.

offset optional int 0 Offset for pagination.

policyIds optional []string ["policy1", "policy2"] List of policy IDs to filter events related to specific policies.

actions optional []string ["allowed"] Values can include allowed, redacted, and blocked.

costOrder optional string asc Order by cost, specify asc or desc.

dateOrder optional string desc Order by date, specify 'asc' or 'desc'.

returnCount optional bool true Whether to return the count of events instead of event details.

Response

Field type example description

events []Event [] List of events.

count int 20 Total number of events returned.

Event

Field type example description

id int64 1699933571 Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

tags int64 ["YOUR_TAG"] Tags of the key.

key_id string YOUR_KEY_ID Key Id associated with the proxy request.

cost_in_usd float64 0.0004 Cost incured by the proxy request.

model string gpt-4-1105-preview Model used in the proxy request.

provider string openai Provider for the proxy request.

status int 200 Http status.

prompt_token_count int 8 Prompt token count of the proxy request.

completion_token_count int 16 Completion token counts of the proxy request.

latency_in_ms int 160 Provider setting name.

path string /api/v1/chat/completion Provider setting name.

method string POST Http method for the assoicated proxu request.

custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.

request []byte {} Request in bytes to providers.

custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.

response []byte {} Response in bytes from providers.

Create custom provider: POST /api/custom/providers

Description

This endpoint is creating custom providers.

RouteConfig

Field required type example description

path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.

target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.

model_location required string model JSON field for the model in the HTTP request.

request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.

response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.

stream_location required string stream JSON field for the stream boolean in the HTTP request.

stream_end_word required string [DONE] End word for the stream.

stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.

stream_max_empty_messages required int 10 Number of max empty messages in stream.

Request

Field required type example description

provider required string bricks Unique identifier associated with the route config.

route_configs required []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.

authentication_param optional string apikey The authentication parameter required for.

Error Response

http code content-type

500, 400 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/custom/providers

Response

Field type example description

id int64 1699933571 Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

provider string bricks Unique identifier associated with the route config.

route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.

authentication_param string apikey The authentication parameter required for.

Update custom provider: PATCH /api/custom/providers/:id

Description

This endpoint is updating a custom provider.

RouteConfig

Field required type example description

path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.

target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.

model_location required string model JSON field for the model in the HTTP request.

request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.

response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.

stream_location required string stream JSON field for the stream boolean in the HTTP request.

stream_end_word required string [DONE] End word for the stream.

stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.

stream_max_empty_messages required int 10 Number of max empty messages in stream.

Request

Field required type example description

route_configs optional []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.

authentication_param optional string apikey The authentication parameter required for.

Error Response

http code content-type

500, 404, 400 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/custom/providers

Response

Field type example description

id int64 1699933571 Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

provider string bricks Unique identifier associated with the route config.

route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.

authentication_param string apikey The authentication parameter required for.

Create a policy: POST /api/policies

Description

This endpoint is for creating a policy.

Request

Field required type example description

name optional string Pivacy Policy #1 Name for the policy.

tags required []string ["orgId-1"] Associated tags.

config optional config {"rules": { "address": "block" }} PII detection rules.

regexConfig optional regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.

Config

Field required type example description

rules required map[Rule]Action { "address": "block" } PII entities mapped to their associated actions.

RegexConfig

Field required type example description

definition required string [2-9]|[12]\d|3[0-6] Regex definition.

action required Action block Action associated with the regex definition.

Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response

http code content-type

500 application/json

Field type example

status int 400, 500

title string request body reader error

type string /errors/policies

detail string something is wrong

instance string /api/policies

Response

Field type example description

id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

tags []string ["org-111"] Tags attached to policies.

config config {"rules": { "address": "block" }} PII detection rules.

regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.

Update a policy: PATCH /api/policies/:id

Description

This endpoint is for updating a policy.

Request

Field required type example description

name optional string Pivacy Policy #1 Name for the policy.

tags required []string ["orgId-1"] Associated tags.

config optional config {"rules": { "address": "block" }} PII detection rules.

regexConfig optional regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.

Config

Field required type example description

rules required map[Rule]Action { "address": "block" } PII entities mapped to their associated actions.

RegexConfig

Field required type example description

definition required string [2-9]|[12]\d|3[0-6] Regex definition.

action required Action block Action associated with the regex definition.

Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response

http code content-type

500 application/json

Field type example

status int 400, 500

title string request body reader error

type string /errors/policies

detail string something is wrong

instance string /api/policies

Response

Field type example description

id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

tags []string ["org-111"] Tags attached to policies.

config config {"rules": { "address": "block" }} PII detection rules.

regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.

Get policies by tags: GET /api/policies

Description

This endpoint is for retrieving policies by tags.

Query Parameters

| name | type | data type | description | |--------|------------|----------------|------------------------------------------------------|s | tags | required | []string | Tags attached to the policies. |

Response

[]Policy

Policy

Field type example description

id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

tags []string ["org-111"] Tags attached to policies.

config config {"rules": { "address": "block" }} PII detection rules.

regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.

Get custom providers: GET /api/custom/providers

Description

This endpoint is for getting custom providers.

RouteConfig

Field required type example description

path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.

target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.

model_location required string model JSON field for the model in the HTTP request.

request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.

response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.

stream_location required string stream JSON field for the stream boolean in the HTTP request.

stream_end_word required string [DONE] End word for the stream.

stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.

stream_max_empty_messages required int 10 Number of max empty messages in stream.

Request

Field required type example description

route_configs optional []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.

authentication_param optional string apikey The authentication parameter required for.

Error Response

http code content-type

500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/custom/providers

Response

[]Provider

Provider

Field type example description

id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.

created_at int64 1699933571 Unix timestamp for creation time.

updated_at int64 1699933571 Unix timestamp for update time.

provider string bricks Unique identifier associated with the route config.

route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.

authentication_param string apikey The authentication parameter required for.

Create routes: POST /api/routes

Description

This endpoint is for creating routes.

StepConfig

Field required type example description

provider required enum azure Provider for the step. Can only be either azure or openai.

model required string gpt-3.5-turbo Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.

retries optional int 2 Number of retries.

params optional object { deploymentId: "ada-test",apiVersion: "2022-12-01" } Params required for maing API requests to desired modela and provider combo. Required if the provider is azure

timeout optional string 5s Timeout desired for each request. Default value is 5m.

CacheConfig

Field required type example description

enabled required bool false Boolean flag indicating whether caching is enabled.

ttl optional string 5s TTL for the cache. Default value is 168h.

Request

Field required type example description

name required string staging-openai-azure-completion-route Name for the route.

path required string / Unique identifier for.

steps required []StepConfig apikey The authentication parameter required for.

keyIds required []string [] The authentication parameter required for.

cacheConfig required CacheConfig [] The authentication parameter required for.

Error Response

http code content-type

500, 400 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/routes

Response

Field type example description

id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb

createdAt required string 1699933571

updatedAt required string 1699933571

name required string staging-openai-azure-completion-route

path required string /production/chat/completion

steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]

keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]

cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }

Retrieve a route: GET /api/routes/:id

Description

This endpoint is for retrieving a route.

Parameters

name type data type description

id required string Unique identifier for the route.

Error Response

http code content-type

500, 404 application/json

Field type example

status int 404

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/routes/:id

Response

Field type example description

id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb

createdAt required string 1699933571

updatedAt required string 1699933571

name required string staging-openai-azure-completion-route

path required string /production/chat/completion

steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]

keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]

cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }

Retrieve routes: GET /api/routes

Description

This endpoint is for retrieving routes.

Error Response

http code content-type

500 application/json

Field type example

status int 404

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/routes/:id

CacheConfig

Field required type example description

enabled required bool false Boolean flag indicating whether caching is enabled.

ttl optional string 5s TTL for the cache. Default value is 168h.

StepConfig

Field required type example description

provider required enum azure Provider for the step. Can only be either azure or openai.

model required string gpt-3.5-turbo Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.

retries optional int 2 Number of retries.

params optional object { deploymentId: "ada-test",apiVersion: "2022-12-01" } Params required for maing API requests to desired modela and provider combo. Required if the provider is azure

timeout optional string 5s Timeout desired for each request. Default value is 5m.

RouteConfig

Field type example description

id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb

createdAt required string 1699933571

updatedAt required string 1699933571

name required string staging-openai-azure-completion-route

path required string /production/chat/completion

steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]

keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]

cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }

Response

[]RouteConfig

Get user ids: GET /api/reporting/users-ids

Description

This endpoint is for getting user ids given a key id.

Query Parameters

name type data type description

keyId required string Key Id

Error Response

http code content-type

500 application/json

Field type example

status int 500

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/reporting/user-ids

Response

[]string

Get custom ids: GET /api/reporting/custom-ids

Description

This endpoint is for getting custom ids given a key id.

Query Parameters

name type data type description

keyId required string Key Id

Error Response

http code content-type

500 application/json

Field type example

status int 500

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/reporting/custom-ids

Response

[]string

Create a user: POST /api/users

Description

This endpoint is creating a user.

Query Parameters

name type data type description

tag optional string Identifier attached to a key configuration

tags optional []string Identifiers attached to a key configuration

provider optional string Provider attached to a key provider configuration. Its value can only be openai.

keyIds optional []string Unique identifiers for keys.

Request

PathConfig

Field required type example description

path required string /api/providers/openai/v1/chat/completion Allowed path

method required string POST HTTP Method

Field required type example description

name required string spike's developer key Name of the API key.

tags optional []string ["org-tag-12345"] Identifiers associated with the key.

keyIds optional []string ["98daa3ae-961d-4253-bf6a-322a32fdca3d"] Setting ids associated with the key.

costLimitInUsd optional float64 5.5 Total spend limit of the API key.

costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime optional int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit optional enum m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl optional string 2d time to live. Available units are [s, m, h].

allowedPaths optional []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels optional []string ["gpt-4"] List of models that can be accessed by the user.

userId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Client defined user ID.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/users

Response

Field type example description

id string 550e8400-e29b-41d4-a716-446655440000 Id of the user object.

name string spike's developer key Name of the user.

createdAt int64 1257894000 UserKey configuration creation time in unix.

updatedAt int64 1257894000 User configuration update time in unix.

tags []string ["org-tag-12345"] Identifiers associated with the user.

keyIds []string ["550e8400-e29b-41d4-a716-446655440000"] Array of key IDs.

revoked boolean true Indicator for whether the user is revoked.

revokedReason string The key has expired Reason for why the user is revoked.

costLimitInUsd float64 5.5 Total spend limit of the user.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels []string ["gpt-4"] List of models that can be accessed by the user.

userId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Client defined user id.

Update a user via id: PATCH /api/users/:id

Description

This endpoint is updating a user via id.

Request

PathConfig

Field required type example description

path required string /api/providers/openai/v1/chat/completion Allowed path

method required string POST HTTP Method

Field required type example description

name required string spike Name of the user.

keyIds optional []string ["98daa3ae-961d-4253-bf6a-322a32fdca3d"] Keys ids associated with the user.

costLimitInUsd optional float64 5.5 Total spend limit of the user.

revoked optional boolean true Indicator for whether the user is revoked.

revokedReason optional string expired Reason for why the user is revoked.

costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime optional int 2 Rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit optional enum m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl optional string 24h time to live. Available units are [s, m, h].

allowedPaths optional []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels optional []string ["gpt-4"] List of models that can be accessed by the user.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/users

Response

Field type example description

id string 550e8400-e29b-41d4-a716-446655440000 Id of the user object.

name string spike Name of the user.

createdAt int64 1257894000 User configuration creation time in unix.

updatedAt int64 1257894000 User configuration update time in unix.

tags []string ["org-tag-12345"] Identifiers associated with the user.

keyIds []string ["550e8400-e29b-41d4-a716-446655440000"] Array of key IDs.

revoked boolean true Indicator for whether the user is revoked.

revokedReason string expired Reason for why the user is revoked.

costLimitInUsd float64 5.5 Total spend limit of the user.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels []string ["gpt-4"] List of models that can be accessed by the user.

userId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Client defined user id.

Update a user via tags and user id: PATCH /api/users

Description

This endpoint is updating a user via tags and id.

Query Parameters

name type data type description

userId optional string User identifier.

tags optional []string Array of identifiers associated with a user.

Request

PathConfig

Field required type example description

path required string /api/providers/openai/v1/chat/completion Allowed path

method required string POST HTTP Method

Field required type example description

name required string spike Name of the user.

keyIds optional []string ["98daa3ae-961d-4253-bf6a-322a32fdca3d"] Keys ids associated with the user.

costLimitInUsd optional float64 5.5 Total spend limit of the user.

revoked optional boolean true Indicator for whether the user is revoked.

revokedReason optional string expired Reason for why the user is revoked.

costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime optional int 2 Rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit optional enum m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl optional string 24h time to live. Available units are [s, m, h].

allowedPaths optional []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels optional []string ["gpt-4"] List of models that can be accessed by the user.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/users

Response

Field type example description

id string 550e8400-e29b-41d4-a716-446655440000 Id of the user object.

name string spike Name of the user.

createdAt int64 1257894000 User configuration creation time in unix.

updatedAt int64 1257894000 User configuration update time in unix.

tags []string ["org-tag-12345"] Identifiers associated with the user.

keyIds []string ["550e8400-e29b-41d4-a716-446655440000"] Array of key IDs.

revoked boolean true Indicator for whether the user is revoked.

revokedReason string expired Reason for why the user is revoked.

costLimitInUsd float64 5.5 Total spend limit of the user.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels []string ["gpt-4"] List of models that can be accessed by the user.

userId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Client defined user id.

Get users: GET /api/users

Description

This endpoint is retrieving users.

Query Parameters

name type data type description

userIds optional []string Array of user identifiers.

tags optional []string Array of identifiers associated with a user.

offset optional int Pagination offset.

limit optional int Pagination limit.

keyIds optional []string Array of key IDs associated with a user.

Error Response

http code content-type

400, 500 application/json

Field type example

status int 400

title string request body reader error

type string /errors/request-body-read

detail string something is wrong

instance string /api/users

Response

Response Body

[]User

User

Field type example description

id string 550e8400-e29b-41d4-a716-446655440000 Id of the user object.

name string spike Name of the user.

createdAt int64 1257894000 User configuration creation time in unix.

updatedAt int64 1257894000 User configuration update time in unix.

tags []string ["org-tag-12345"] Identifiers associated with the user.

keyIds []string ["550e8400-e29b-41d4-a716-446655440000"] Array of key IDs.

revoked boolean true Indicator for whether the user is revoked.

revokedReason string expired Reason for why the user is revoked.

costLimitInUsd float64 5.5 Total spend limit of the user.

costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.

costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].

rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.

rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]

ttl string 24h time to live. Available units are [s, m, h]

allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] List of paths that can be accessed by the user.

allowedModels []string ["gpt-4"] List of models that can be accessed by the user.

userId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Client defined user id.

OpenAI Proxy

The OpenAI proxy runs on Port 8002.

Headers

name type data type description

x-custom-event-id optional string Custom Id that can be used to retrieve an event associated with each proxy request.

Chat Completion

Call OpenAI chat completions: POST /api/providers/openai/v1/chat/completions

Description

This endpoint is set up for proxying OpenAI chat completion requests. Documentation for this endpoint can be found here.

Embeddings

Call OpenAI embeddings: POST /api/providers/openai/v1/embeddings

Description

This endpoint is set up for proxying OpenAI embedding requests. Documentation for this endpoint can be found here.

Moderations

Call OpenAI moderations: POST /api/providers/openai/v1/moderations

Description

This endpoint is set up for proxying OpenAI moderation requests. Documentation for this endpoint can be found here.

Models

Get OpenAI models: GET /api/providers/openai/v1/models

Description

This endpoint is set up for retrieving OpenAI models. Documentation for this endpoint can be found here.

Retrieve an OpenAI model: GET /api/providers/openai/v1/models/:model

Description

This endpoint is set up for retrieving an OpenAI model. Documentation for this endpoint can be found here.

Files

List files: GET /api/providers/openai/v1/files

Description

This endpoint is set up for list OpenAI files. Documentation for this endpoint can be found here.

Upload a file: POST /api/providers/openai/v1/files

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Delete a file: POST /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Retrieve a file: GET /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI file. Documentation for this endpoint can be found here.

Retrieve file content: GET /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for retrieving an OpenAI file content. Documentation for this endpoint can be found here.

Images

Generate images: POST /api/providers/openai/v1/images/generations

Description

This endpoint is set up for generating OpenAI images. Documentation for this endpoint can be found here.

Edit images: POST /api/providers/openai/v1/images/edits

Description

This endpoint is set up for editting OpenAI generated images. Documentation for this endpoint can be found here.

Create image variations: POST /api/providers/openai/v1/images/variations

Description

This endpoint is set up for creating OpenAI image variations. Documentation for this endpoint can be found here.

Voices

Create speech: POST /api/providers/openai/v1/audio/speech

Description

This endpoint is set up for creating speeches. Documentation for this endpoint can be found here.

Create transcriptions: POST /api/providers/openai/v1/audio/transcriptions

Description

This endpoint is set up for editting generated images. Documentation for this endpoint can be found here.

Create translations: POST /api/providers/openai/v1/audios/translations

Description

This endpoint is set up for creating translations. Documentation for this endpoint can be found here.

Assistants

Create assistant: POST /api/providers/openai/v1/assistants

Description

This endpoint is set up for creating an OpenAI assistant. Documentation for this endpoint can be found here.

Retrieve assistant: GET /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for retrieving an OpenAI assistant. Documentation for this endpoint can be found here.

Modify assistant: POST /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for modifying an OpenAI assistant. Documentation for this endpoint can be found here.

Delete assistant: DELETE /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for deleting an OpenAI assistant. Documentation for this endpoint can be found here.

List assistants: GET /api/providers/openai/v1/assistants

Description

This endpoint is set up for listing OpenAI assistants. Documentation for this endpoint can be found here.

Create assistant file: POST /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for creating an OpenAI assistant file. Documentation for this endpoint can be found here.

Retrieve assistant file: GET /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI assistant file. Documentation for this endpoint can be found here.

Delete assistant file: DELETE /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for deleting an OpenAI assistant file. Documentation for this endpoint can be found here.

List assistant files: GET /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for retrieving OpenAI assistant files. Documentation for this endpoint can be found here.

Create thread: POST /api/providers/openai/v1/threads

Description

This endpoint is set up for creating an OpenAI thread. Documentation for this endpoint can be found here.

Retrieve thread: GET /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for retrieving an OpenAI thread. Documentation for this endpoint can be found here.

Modify thread: POST /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for modifying an OpenAI thread. Documentation for this endpoint can be found here.

Delete thread: DELETE /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for deleting an OpenAI thread. Documentation for this endpoint can be found here.

Create message: POST /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for creating an OpenAI message. Documentation for this endpoint can be found here.

Retrieve message: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id

Description

This endpoint is set up for retrieving an OpenAI message. Documentation for this endpoint can be found here.

Modify message: POST /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for modifying an OpenAI message. Documentation for this endpoint can be found here.

List messages: GET /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for listing OpenAI messages. Documentation for this endpoint can be found here.

Retrieve message file: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI message file. Documentation for this endpoint can be found here.

List message files: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files

Description

This endpoint is set up for retrieving OpenAI message files. Documentation for this endpoint can be found here.

Create run: POST /api/providers/openai/v1/threads/:thread_id/runs

Description

This endpoint is set up for creating an OpenAI run. Documentation for this endpoint can be found here.

Retrieve run: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for retrieving an OpenAI run. Documentation for this endpoint can be found here.

Modify run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for modifying an OpenAI run. Documentation for this endpoint can be found here.

List runs: GET /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for retrieving OpenAI runs. Documentation for this endpoint can be found here.

Submit tool outputs to run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for submitting tool outputs to an OpenAI run. Documentation for this endpoint can be found here.

Cancel a run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id/cancel

Description

This endpoint is set up for cancellling an OpenAI run. Documentation for this endpoint can be found here.

Create thread and run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for creating an OpenAI thread and run. Documentation for this endpoint can be found here.

Retrieve run step: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps/:step_id

Description

This endpoint is set up for retrieving an OpenAI run step. Documentation for this endpoint can be found here.

List run steps: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps

Description

This endpoint is set up for listing OpenAI run steps. Documentation for this endpoint can be found here.

Azure OpenAI Proxy

The custom provider proxy runs on Port 8002.

Create Azure OpenAI chat completion: POST /api/providers/azure/openai/deployments/:deployment_id/chat/completions?api-version={API_VERSION}

Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Create Azure OpenAI embeddings: POST /api/providers/azure/openai/deployments/:deployment_id/embeddings?api-version={API_VERSION}

Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Anthropic Proxy

The custom provider proxy runs on Port 8002.

Create Anthropic completion: POST /api/providers/anthropic/v1/complete

Description

This endpoint is set up for proxying Anthropic completion requests. Documentation for this endpoint can be found here.

Create Anthropic messages: POST /api/providers/anthropic/v1/messages

Description

This endpoint is set up for proxying Anthropic messages requests. Documentation for this endpoint can be found here.

vllm Provider Proxy

The vllm provider proxy runs on Port 8002.

Create chat completions: POST /api/providers/vllm/v1/chat/completions

Description

This endpoint is set up for proxying vllm chat completions requests. Documentation for this endpoint can be found here.

Create completions: POST /api/providers/vllm/v1/completions

Description

This endpoint is set up for proxying vllm completions requests. Documentation for this endpoint can be found here.

Deepinfra Provider Proxy

The deepinfra provider proxy runs on Port 8002.

Create chat completions: POST /api/providers/deepinfra/v1/chat/completions

Description

This endpoint is set up for proxying deepinfra chat completions requests. Documentation for this endpoint can be found here.

Create completions: POST /api/providers/deepinfra/v1/completions

Description

This endpoint is set up for proxying deepinfra completions requests. Documentation for this endpoint can be found here.

Create embeddings: POST /api/providers/deepinfra/v1/embeddings

Description

This endpoint is set up for proxying deepinfra embeddings requests. Documentation for this endpoint can be found here.

Custom Provider Proxy

The custom provider proxy runs on Port 8002.

Call custom providers: POST /api/custom/providers/:provider/*

Description

First you need to use create custom providers endpoint to create custom providers. Then create corresponding provider setting for the newly created custom provider. Afterward, you can start creating keys associated with the custom provider, and use the keys to access this endpoint by placing the created key in Authorization: Bearer YOUR_BRICKSLLM_KEY as part of your HTTP request headers.

Route Proxy

The custom provider proxy runs on Port 8002.

Call a route: POST /api/route/*

Description

Route helps you interpolate different models (embeddings or chat completion models) and providers (OpenAI or Azure OpenAI) to gurantee API responses.

First you need to use create route endpoint to create routes. If the route uses both Azure and OpenAI, you need to create API keys with corresponding provider settings as well. If the route is for chat completion, just call the route using the OpenAI chat completion format. On the other hand, if the route is for embeddings, just call the route using the embeddings format.

Name		Name	Last commit message	Last commit date
Latest commit History 603 Commits
.github/workflows		.github/workflows
assets		assets
blog		blog
cmd/bricksllm		cmd/bricksllm
cookbook		cookbook
internal		internal
scripts		scripts
.gitignore		.gitignore
.goreleaser.yml		.goreleaser.yml
CHANGELOG.md		CHANGELOG.md
Dockerfile.datadog		Dockerfile.datadog
Dockerfile.dev		Dockerfile.dev
Dockerfile.prod		Dockerfile.prod
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum

Name	type	description	default
`POSTGRESQL_HOSTS`	required	Hosts for Postgresql DB. Separated by ,	`localhost`
`POSTGRESQL_DB_NAME`	optional	Name for Postgresql DB.
`POSTGRESQL_USERNAME`	required	Postgresql DB username
`POSTGRESQL_PASSWORD`	required	Postgresql DB password
`POSTGRESQL_SSL_MODE`	optional	Postgresql SSL mode	`disable`
`POSTGRESQL_PORT`	optional	The port that Postgresql DB runs on	`5432`
`POSTGRESQL_READ_TIME_OUT`	optional	Timeout for Postgresql read operations	`2m`
`POSTGRESQL_WRITE_TIME_OUT`	optional	Timeout for Postgresql write operations	`5s`
`REDIS_HOSTS`	required	Host for Redis. Separated by ,	`localhost`
`REDIS_PASSWORD`	optional	Redis Password
`REDIS_PORT`	optional	The port that Redis DB runs on	`6379`
`REDIS_READ_TIME_OUT`	optional	Timeout for Redis read operations	`1s`
`REDIS_WRITE_TIME_OUT`	optional	Timeout for Redis write operations	`500ms`
`IN_MEMORY_DB_UPDATE_INTERVAL`	optional	The interval BricksLLM API gateway polls Postgresql DB for latest key configurations	`1s`
`STATS_PROVIDER`	optional	This value can only be datadog. Required for integration with Datadog.
`PROXY_TIMEOUT`	optional	Timeout for proxy HTTP requests.	`600s`
`NUMBER_OF_EVENT_MESSAGE_CONSUMERS`	optional	Number of event message consumers that help handle counting tokens and inserting event into db.	`3`
`AWS_SECRET_ACCESS_KEY`	optional	It is for PII detection feature.	`5s`
`AWS_ACCESS_KEY_ID`	optional	It is for using PII detection feature.	`5s`
`AMAZON_REGION`	optional	Region for AWS.	`us-west-2`
`AMAZON_REQUEST_TIMEOUT`	optional	Timeout for amazon requests.	`5s`
`AMAZON_CONNECTION_TIMEOUT`	optional	Timeout for amazon connection.	`10s`
`AWS_SECRET_ACCESS_KEY`	optional	Required for PII detection features.
`AWS_ACCESS_KEY_ID`	optional	Required for PII detection features.

name	type	data type	description
`tag`	optional	`string`	Identifier attached to a key configuration
`tags`	optional	`[]string`	Identifiers attached to a key configuration
`provider`	optional	`string`	Provider attached to a key provider configuration. Its value can only be `openai`.
`keyIds`	optional	`[]string`	Unique identifiers for keys.

Field	type	example
status	`int`	400
title	`string`	request body reader error
type	`string`	/errors/request-body-read
detail	`string`	something is wrong
instance	`string`	/api/key-management/keys

Field	type	example	description
name	`string`	spike's developer key	Name of the API key.
createdAt	`int64`	`1257894000`	Key configuration creation time in unix.
updatedAt	`int64`	`1257894000`	Key configuration update time in unix.
revoked	`boolean`	true	Indicator for whether the key is revoked.
revokedReason	`string`	The key has expired	Reason for why the key is revoked.
tags	`[]string`	["org-tag-12345"]	Identifiers associated with the key.
keyId	`string`	`550e8400-e29b-41d4-a716-446655440000`	Unique identifier for the key.
costLimitInUsd	`float64`	`5.5`	Total spend limit of the API key.
costLimitInUsdOverTime	`float64`	`2`	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	`enum`	`d`	Time unit for costLimitInUsdOverTime. Possible values are [`m`, `h`, `d`, `mo`].
rateLimitOverTime	`int`	`2`	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	`string`	`m`	Time unit for rateLimitOverTime. Possible values are [`h`, `m`, `s`, `d`]
ttl	`string`	`24h`	time to live. Available units are [`s`, `m`, `h`]
allowedPaths	`[]PathConfig`	`[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]`	Allowed paths that can be accessed using the key.
settingId	`string`	`98daa3ae-961d-4253-bf6a-322a32fdca3d`	This field is DEPERCATED. Use `settingIds` field instead.
settingIds	`string`	`[98daa3ae-961d-4253-bf6a-322a32fdca3d]`	Setting ids associated with the key.
shouldLogRequest	`bool`	`false`	Should request be stored.
shouldLogResponse	`bool`	`true`	Should response be stored.
rotationEnabled	`bool`	`false`	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	`string`	`98daa3ae-961d-4253-bf6a-322a32fdca3d`	Policy id associated with the key.
isKeyNotHashed	`bool`	`false`	Flag controls whether or not the key should be hashed.

Field	required	type	example	description
keyIds	optional	`[]string`	`["98daa3ae-961d-4253-bf6a-322a32fdca3d"]`	Array of key IDs.
tags	optional	`[]string`	`["org-tag-12345"]`	Identifiers associated with keys.
name	optional	`string`	`key`	Filter keys by the substring match on the `name` field.
revoked	optional	`bool`	`false`	Filters keys by `revoked` status.
limit	optional	`int`	`5`	Pagination limit.
offset	optional	`int`	`5`	Pagination offset.
order	optional	`string`	`asc`	`asc` and `desc` are the two available values.
returnCount	optional	`bool`	`true`	Flag that controls whether the response should contain total key counts.

Field	required	type	example	description
path	required	`string`	/api/providers/openai/v1/chat/completion	Allowed path
method	required	`string`	POST	HTTP Method

Field	type	example
status	`int`	`400`
title	`string`	request body reader error
type	`string`	/errors/request-body-read
detail	`string`	something is wrong
instance	`string`	/api/key-management/keys

Field	required	type	example	description
start	required	`int64`	`1257894000`	Start unix timestamp.
end	required	`int64`	`1257894000`	End unix timestamp
keyIds	optional	`[]string`	`["98daa3ae-961d-4253-bf6a-322a32fdca3d"]`	Array of key IDs.
tags	optional	`[]string`	`["org-tag-12345"]`	Identifiers associated with keys.
name	optional	`string`	`key`	Filter keys by the substring match on the `name` field.
revoked	optional	`bool`	`false`	Filters keys by `revoked` status.
limit	optional	`int`	`5`	Pagination limit.
offset	optional	`int`	`5`	Pagination offset.
order	optional	`string`	`asc`	`asc` and `desc` are the two available values.

Field	required	type	example	description
apiKey	required	`string`	`xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`	This value is required.
url	required	`string`	`https://your.deployment.url`	This value is required when the provider is `vllm`
resourceName	required	`string`	`YOUR_AZURE_RESOURCE_NAME`	This value is required when the provider is `azure`.

Field	type	example	description
createdAt	`int64`	`1699933571`	Unix timestamp for creation time.
updatedAt	`int64`	`1699933571`	Unix timestamp for update time.
provider	`enum`	`openai`	This value can only be `openai` as for now.
id	`string`	`98daa3ae-961d-4253-bf6a-322a32fdca3d`	This value is a unique identifier.
name	`string`	`YOUR_PROVIDER_SETTING_NAME`	Provider setting name.
allowedModels	`[]string`	`["text-embedding-ada-002"]`	Allowed models for this provider setting.

Field	required	type	example	description
setting	required	`Setting`	`{ "apikey": "YOUR_OPENAI_KEY" }`	A map of values used for authenticating with the selected provider.
name	optional	`string`	`YOUR_PROVIDER_SETTING_NAME`	This field is used for giving a name to provider setting
allowedModels	`[]string`	`["text-embedding-ada-002"]`	Allowed models for this provider setting.

Field	type	example	description
dataPoints	`[]dataPoint`	`[{ "timeStamp": 1699933571, "numberOfRequests": 1, "costInUsd": 0.8, "latencyInMs": 600, "promptTokenCount": 0, "completionTokenCount": 0, "successCount": 1 }]`	Unix timestamp for creation time.
latencyInMsMedian	`float64`	`656.7`	Median latency for the given time period.
latencyInMs99th	`float64`	`555.7`	99th percentile latency for the given time period.

Field	type	example	description
timeStamp	`int64`	`1702504746`	Unix timestamp for the data point
numberOfRequests	`int64`	`100`	Aggregated number of http requests over the given time increment.
costInUsd	`float64`	`1.7`	Aggregated cost of proxied requests in USD over the given time increment.
latencyInMs	`int`	`555`	Aggregated latency in milliseconds of http requests over the given time increment.
promptTokenCount	`int`	`25`	Aggregated prompt token counts over the given time increment.
completionTokenCount	`int`	`4000`	Aggregated completion token counts over the given time increment.
successCount	`int`	`555`	Aggregated number of successful http requests over the given time increment.
keyId	`int`	`555.7`	key Id associated with the event.
model	`string`	`gpt-3.5-turbo`	model associated with the event.
customId	`string`	`customId`	customId associated with the event.

Field	type	example	description
id	`int64`	`1699933571`	Unique identifier associated with the event.
created_at	`int64`	`1699933571`	Unix timestamp for creation time.
tags	`int64`	`["YOUR_TAG"]`	Tags of the key.
key_id	`string`	`YOUR_KEY_ID`	Key Id associated with the proxy request.
cost_in_usd	`float64`	`0.0004`	Cost incured by the proxy request.
model	`string`	`gpt-4-1105-preview`	Model used in the proxy request.
provider	`string`	`openai`	Provider for the proxy request.
status	`int`	`200`	Http status.
prompt_token_count	`int`	`8`	Prompt token count of the proxy request.
completion_token_count	`int`	`16`	Completion token counts of the proxy request.
latency_in_ms	`int`	`160`	Provider setting name.
path	`string`	`/api/v1/chat/completion`	Provider setting name.
method	`string`	`POST`	Http method for the assoicated proxu request.
custom_id	`string`	`YOUR_CUSTOM_ID`	Custom Id passed by the user in the headers of proxy requests.
request	`[]byte`	`{}`	Request in bytes to providers.
custom_id	`string`	`YOUR_CUSTOM_ID`	Custom Id passed by the user in the headers of proxy requests.
response	`[]byte`	`{}`	Response in bytes from providers.

name	type	data type	example	description
`userIds`	optional	`[]string`	`["user1", "user2"]`	List of user IDs to filter events by specific users.
`customIds`	optional	`[]string`	`["id1", "id2"]`	List of custom identifiers for filtering events.
`keyIds`	optional	`[]string`	`["key1", "key2"]`	List of key IDs used to filter events.
`tags`	optional	`[]string`	`["tag1", "tag2"]`	List of tags associated with events for filtering.
`start`	required	`int64`	`1609459200`	Start timestamp for filtering events.
`end`	required	`int64`	`1612137600`	End timestamp for filtering events.
`limit`	optional	`int`	`100`	Maximum number of events to return.
`offset`	optional	`int`	`0`	Offset for pagination.
`policyIds`	optional	`[]string`	`["policy1", "policy2"]`	List of policy IDs to filter events related to specific policies.
`actions`	optional	`[]string`	`["allowed"]`	Values can include `allowed`, `redacted`, and `blocked`.
`costOrder`	optional	`string`	`asc`	Order by cost, specify `asc` or `desc`.
`dateOrder`	optional	`string`	`desc`	Order by date, specify 'asc' or 'desc'.
`returnCount`	optional	`bool`	`true`	Whether to return the count of events instead of event details.

Field	type	example	description
events	`[]Event`	`[]`	List of events.
count	`int`	`20`	Total number of events returned.

Field	required	type	example	description
route_configs	optional	`[]RouteConfig`	`{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}`	Route configurations for the custom provider.
authentication_param	optional	`string`	`apikey`	The authentication parameter required for.

Field	required	type	example	description
definition	required	`string`	`[2-9]\|[12]\d\|3[0-6]`	Regex definition.
action	required	`Action`	`block`	Action associated with the regex definition.

Field	type	example
status	`int`	`400`, `500`
title	`string`	`request body reader error`
type	`string`	`/errors/policies`
detail	`string`	`something is wrong`
instance	`string`	`/api/policies`

Field	type	example	description
id	`int64`	`9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb`	Unique identifier associated with the event.
created_at	`int64`	`1699933571`	Unix timestamp for creation time.
updated_at	`int64`	`1699933571`	Unix timestamp for update time.
tags	`[]string`	`["org-111"]`	Tags attached to policies.
config	`config`	`{"rules": { "address": "block" }}`	PII detection rules.
regexConfig	`regexConfig`	`{"rules": [{"definition": "[2-9]\|[12]\d\|3[0-6]", "action": "block"}]}`	Regular expression rules.

License

justoy/BricksLLM

Folders and files

Latest commit

History

Repository files navigation

BricksLLM: AI Gateway For Putting LLM In Production

Features

Getting Started

Step 1 - Clone BricksLLM-Docker repository

Step 2 - Change to BricksLLM-Docker directory

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

Step 4 - Create a provider setting

Step 5 - Create a Bricks API key

Congratulations you are done!!!

How to Update?

Documentation

Environment variables

Configuration Endpoints

Headers

Description

Query Parameters

Error Response

Response

Description

Request

Error Response

Response

key

Description

Request

Error Response

Responses

Description

Parameters

Request

Error Response

Response

Description

Request

Error Response

Response

Description

Request

Error Response

Response

Description

Query Parameters

Error Response

Response

Description

Parameters

Request

Error Response

Response

Description

Request

Error Response

Response

Description

Query Parameters

Error Response

Response

Description

Request

Response

Description

RouteConfig

Request

Error Response

Response

Description

RouteConfig

Request

Error Response

Response

Description

Request

Config

RegexConfig

Field	required	type	example	description
enabled	required	`bool`	`false`	Boolean flag indicating whether caching is enabled.
ttl	optional	`string`	`5s`	TTL for the cache. Default value is `168h`.

Field	type	example
status	`int`	`404`
title	`string`	`request body reader error`
type	`string`	`/errors/request-body-read`
detail	`string`	`something is wrong`
instance	`string`	`/api/routes/:id`

Field	type	example
status	`int`	`500`
title	`string`	request body reader error
type	`string`	/errors/request-body-read
detail	`string`	something is wrong
instance	`string`	/api/reporting/user-ids

Field	type	example	description
id	`string`	`550e8400-e29b-41d4-a716-446655440000`	Id of the user object.
name	`string`	spike's developer key	Name of the user.
createdAt	`int64`	1257894000	UserKey configuration creation time in unix.
updatedAt	`int64`	1257894000	User configuration update time in unix.
tags	`[]string`	["org-tag-12345"]	Identifiers associated with the user.
keyIds	`[]string`	`["550e8400-e29b-41d4-a716-446655440000"]`	Array of key IDs.
revoked	`boolean`	true	Indicator for whether the user is revoked.
revokedReason	`string`	The key has expired	Reason for why the user is revoked.
costLimitInUsd	`float64`	`5.5`	Total spend limit of the user.
costLimitInUsdOverTime	`float64`	`2`	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	`enum`	d	Time unit for costLimitInUsdOverTime. Possible values are [`m`, `h`, `d`, `mo`].
rateLimitOverTime	`int`	`2`	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	`string`	m	Time unit for rateLimitOverTime. Possible values are [`h`, `m`, `s`, `d`]
ttl	`string`	`24h`	time to live. Available units are [`s`, `m`, `h`]
allowedPaths	`[]PathConfig`	`[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]`	List of paths that can be accessed by the user.
allowedModels	`[]string`	`["gpt-4"]`	List of models that can be accessed by the user.
userId	`string`	`98daa3ae-961d-4253-bf6a-322a32fdca3d`	Client defined user id.

name	type	data type	description
`userId`	optional	`string`	User identifier.
`tags`	optional	`[]string`	Array of identifiers associated with a user.