BricksLLM: AI Gateway For Putting LLM In Production

BricksLLM is a cloud native AI gateway written in Go. Currently, it serves as a proxy to OpenAI. We let you create API keys that have rate limits, cost limits and TTLs. The API keys can be used in both development and production to achieve fine-grained access control that is not provided by OpenAI at the moment. The proxy is compatible with OpenAI API and its SDKs.

The vision of BricksLLM is to support many more large language models such as LLama2, Claude, PaLM2 etc, and streamline LLM operations.

Roadmap

• Access control via API key with rate limit, cost limit and ttl
• Logging integration
• Statsd integration
• Custom Provider Integration
• PII detection and masking 🚧

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker-compose up

You can run this in detach mode use the -d flag: docker-compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'   

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingId": "ID_FROM_STEP_FOUR",
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'   

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name	type	description	default
POSTGRESQL_HOSTS	required	Hosts for Postgresql DB. Seperated by ,	localhost
POSTGRESQL_DB_NAME	optional	Name for Postgresql DB.
POSTGRESQL_USERNAME	required	Postgresql DB username
POSTGRESQL_PASSWORD	required	Postgresql DB password
POSTGRESQL_SSL_MODE	optional	Postgresql SSL mode	disable
POSTGRESQL_PORT	optional	The port that Postgresql DB runs on	5432
POSTGRESQL_READ_TIME_OUT	optional	Timeout for Postgresql read operations	2s
POSTGRESQL_WRITE_TIME_OUT	optional	Timeout for Postgresql write operations	1s
REDIS_HOSTS	required	Host for Redis. Seperated by ,	localhost
REDIS_PASSWORD	optional	Redis Password
REDIS_PORT	optional	The port that Redis DB runs on	6379
REDIS_READ_TIME_OUT	optional	Timeout for Redis read operations	1s
REDIS_WRITE_TIME_OUT	optional	Timeout for Redis write operations	500ms
IN_MEMORY_DB_UPDATE_INTERVAL	optional	The interval BricksLLM API gateway polls Postgresql DB for latest key configurations	1s
STATS_PROVIDER	optional	This value can only be datadog. Required for integration with Datadog.
PROXY_TIMEOUT	optional	This value can only be datadog. Required for integration with Datadog.

Configuration Endpoints

The configuration server runs on Port 8001.

Get keys: GET /api/key-management/keysCreate key: PUT /api/key-management/keysUpdate key: PATCH /api/key-management/keys/{keyId}Create a provider setting: POST /api/provider-settingsGet all provider settings: GET /api/provider-settingsUpdate a provider setting: PATCH /api/provider-settings/:idRetrieve Metrics: POST /api/reporting/eventsGet events: GET /api/eventsCreate custom provider: POST /api/custom/providersUpdate custom provider: PATCH /api/custom/providers/:idGet custom providers: GET /api/custom/providers

OpenAI Proxy

The OpenAI proxy runs on Port 8002.

Headers

name	type	data type	description
x-custom-event-id	optional	string	Custom Id that can be used to retrieve an event associated with each proxy request.

Chat Completion

Call OpenAI chat completions: POST /api/providers/openai/v1/chat/completions

Embeddings

Call OpenAI embeddings: POST /api/providers/openai/v1/embeddings

Moderations

Call OpenAI moderations: POST /api/providers/openai/v1/moderations

Models

Get OpenAI models: GET /api/providers/openai/v1/modelsRetrieve an OpenAI model: GET /api/providers/openai/v1/models/:model

Files

List files: GET /api/providers/openai/v1/filesUpload a file: POST /api/providers/openai/v1/filesDelete a file: POST /api/providers/openai/v1/files/:file_idRetrieve a file: GET /api/providers/openai/v1/files/:file_idRetrieve file content: GET /api/providers/openai/v1/files/:file_id/content

Assistants

Create assistant: POST /api/providers/openai/v1/assistantsRetrieve assistant: GET /api/providers/openai/v1/assistants/:assistant_idModify assistant: POST /api/providers/openai/v1/assistants/:assistant_idDelete assistant: DELETE /api/providers/openai/v1/assistants/:assistant_idList assistants: GET /api/providers/openai/v1/assistantsCreate assistant file: POST /api/providers/openai/v1/assistants/:assistant_id/filesRetrieve assistant file: GET /api/providers/openai/v1/assistants/:assistant_id/files/:file_idDelete assistant file: DELETE /api/providers/openai/v1/assistants/:assistant_id/files/:file_idList assistant files: GET /api/providers/openai/v1/assistants/:assistant_id/filesCreate thread: POST /api/providers/openai/v1/threadsRetrieve thread: GET /api/providers/openai/v1/threads/:thread_idModify thread: POST /api/providers/openai/v1/threads/:thread_idDelete thread: DELETE /api/providers/openai/v1/threads/:thread_idCreate message: POST /api/providers/openai/v1/threads/:thread_id/messagesRetrieve message: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_idModify message: POST /api/providers/openai/v1/files/:file_id/contentList messages: GET /api/providers/openai/v1/threads/:thread_id/messagesRetrieve message file: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files/:file_idList message files: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/filesCreate run: POST /api/providers/openai/v1/threads/:thread_id/runsRetrieve run: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_idModify run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_idList runs: GET /api/providers/openai/v1/threads/runsSubmit tool outputs to run: POST /api/providers/openai/v1/threads/runsCancel a run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id/cancelCreate thread and run: POST /api/providers/openai/v1/threads/runsRetrieve run step: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps/:step_idList run steps: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps

Anthropic Proxy

The custom provider proxy runs on Port 8002.

Create Anthropic completion: POST /api/providers/anthropic/v1/complete

Custom Provider Proxy

The custom provider proxy runs on Port 8002.

Call custom providers: POST /api/custom/providers/:provider/*