bricksllm

module
v1.15.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 9, 2024 License: MIT

README

BricksLLM: AI Gateway For Putting LLM In Production

YCombinator S22 PRs Welcome Join BricksLLM on Discord License

[!TIP] A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:

  • Set LLM usage limits for users on different pricing tiers
  • Track LLM usage on a per user and per organization basis
  • Block or redact requests containing PIIs
  • Improve LLM reliability with failovers, retries and caching
  • Distribute API keys with rate limits and cost limits for internal development/production use cases
  • Distribute API keys with rate limits and cost limits for students

Features

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository
git clone https://github.com/bricks-cloud/BricksLLM-Docker
Step 2 - Change to BricksLLM-Docker directory
cd BricksLLM-Docker
Step 3 - Deploy BricksLLM locally with Postgresql and Redis
docker-compose up

You can run this in detach mode use the -d flag: docker-compose up -d

Step 4 - Create a provider setting
curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'
Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name type description default
POSTGRESQL_HOSTS required Hosts for Postgresql DB. Separated by , localhost
POSTGRESQL_DB_NAME optional Name for Postgresql DB.
POSTGRESQL_USERNAME required Postgresql DB username
POSTGRESQL_PASSWORD required Postgresql DB password
POSTGRESQL_SSL_MODE optional Postgresql SSL mode disable
POSTGRESQL_PORT optional The port that Postgresql DB runs on 5432
POSTGRESQL_READ_TIME_OUT optional Timeout for Postgresql read operations 2m
POSTGRESQL_WRITE_TIME_OUT optional Timeout for Postgresql write operations 5s
REDIS_HOSTS required Host for Redis. Separated by , localhost
REDIS_PASSWORD optional Redis Password
REDIS_PORT optional The port that Redis DB runs on 6379
REDIS_READ_TIME_OUT optional Timeout for Redis read operations 1s
REDIS_WRITE_TIME_OUT optional Timeout for Redis write operations 500ms
IN_MEMORY_DB_UPDATE_INTERVAL optional The interval BricksLLM API gateway polls Postgresql DB for latest key configurations 1s
STATS_PROVIDER optional This value can only be datadog. Required for integration with Datadog.
PROXY_TIMEOUT optional Timeout for proxy HTTP requests. 600s
NUMBER_OF_EVENT_MESSAGE_CONSUMERS optional Number of event message consumers that help handle counting tokens and inserting event into db. 3
AMAZON_REQUEST_TIMEOUT optional Timeout for amazon requests. 5s
AMAZON_CONNECTION_TIMEOUT optional Timeout for amazon connection. 10s
AWS_SECRET_ACCESS_KEY optional Required for PII detection features.
AWS_ACCESS_KEY_ID optional Required for PII detection features.

Configuration Endpoints

The configuration server runs on Port 8001.

Headers
name type data type description
X-API-KEY optional string Key authentication header.
Get keys: GET /api/key-management/keys
Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Query Parameters
name type data type description
tag optional string Identifier attached to a key configuration
tags optional []string Identifiers attached to a key configuration
provider optional string Provider attached to a key provider configuration. Its value can only be openai.
keyIds optional []string Unique identifiers for keys.
Error Response
http code content-type
400, 500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/key-management/keys
Response
Response Body
[]KeyConfiguration

Fields of KeyConfiguration

Field type example description
name string spike's developer key Name of the API key.
createdAt int64 1257894000 Key configuration creation time in unix.
updatedAt int64 1257894000 Key configuration update time in unix.
revoked boolean true Indicator for whether the key is revoked.
revokedReason string The key has expired Reason for why the key is revoked.
tags []string ["org-tag-12345"] Identifiers associated with the key.
keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.
costLimitInUsd float64 5.5 Total spend limit of the API key.
costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl string 2d time to live. Available units are [s, m, h]
allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}] Allowed paths that can be accessed using the key.
settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.
settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.
shouldLogRequest bool false Should request be stored.
shouldLogResponse bool true Should response be stored.
rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.
Create key: PUT /api/key-management/keys
Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request
PathConfig
Field required type example description
path required string /api/providers/openai/v1/chat/completion Allowed path
method required string POST HTTP Method
Field required type example description
name required string spike's developer key Name of the API key.
tags optional []string ["org-tag-12345"] Identifiers associated with the key.
key required string abcdef12345 API key.
settingId depercated string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.
settingIds required string 98daa3ae-961d-4253-bf6a-322a32fdca3d Setting ids associated with the key.
costLimitInUsd optional float64 5.5 Total spend limit of the API key.
costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime optional int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit optional enum m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl optional string 2d time to live. Available units are [s, m, h].
allowedPaths optional []PathConfig 2d Pathes allowed for access.
shouldLogRequest optional bool false Should request be stored.
shouldLogResponse optional bool true Should response be stored.
rotationEnabled optional bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.
Error Response
http code content-type
400, 500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/key-management/keys
Responses
Field type example description
name string spike's developer key Name of the API key.
createdAt int64 1257894000 Key configuration creation time in unix.
updatedAt int64 1257894000 Key configuration update time in unix.
revoked boolean true Indicator for whether the key is revoked.
revokedReason string The key has expired Reason for why the key is revoked.
tags []string ["org-tag-12345"] Identifiers associated with the key.
keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.
costLimitInUsd float64 5.5 Total spend limit of the API key.
costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d].
ttl string 2d time to live. Available units are [s, m, h]
allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.
settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.
settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.
shouldLogRequest bool false Should request be stored.
shouldLogResponse bool true Should response be stored.
rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.
Update key: PATCH /api/key-management/keys/{keyId}
Description

This endpoint is set up for updating key configurations using key id.

Parameters
name type data type description
keyId required string Unique key configuration identifier.
Request
PathConfig
Field required type example description
path required string /api/providers/openai/v1/chat/completion Allowed path
method required string POST HTTP Method
Field required type example description
settingId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.
settingIds optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Setting ids associated with the key.
name optional string spike's developer key Name of the API key.
tags optional []string ["org-tag-12345"] Identifiers associated with the key.
revoked optional boolean true Indicator for whether the key is revoked.
revokedReason optional string The key has expired Reason for why the key is revoked.
costLimitInUsd optional float64 5.5 Total spend limit of the API key.
costLimitInUsdOverTime optional float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit optional enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime optional int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit optional string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
allowedPaths optional [{ "path": "/api/providers/openai/v1/chat/completions", "method": "POST"}] `` Pathes allowed for access.
shouldLogRequest optional bool false Should request be stored.
shouldLogResponse optional bool true Should response be stored.
rotationEnabled optional bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId optional string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.
Error Response
http code content-type
400, 500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/key-management/keys
Response
Field type example description
name string spike's developer key Name of the API key.
createdAt int64 1257894000 Key configuration creation time in unix.
updatedAt int64 1257894000 Key configuration update time in unix.
revoked boolean true Indicator for whether the key is revoked.
revokedReason string The key has expired Reason for why the key is revoked.
tags []string ["org-tag-12345"] Identifiers associated with the key.
keyId string 550e8400-e29b-41d4-a716-446655440000 Unique identifier for the key.
costLimitInUsd float64 5.5 Total spend limit of the API key.
costLimitInUsdOverTime float64 2 Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit enum d Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime int 2 rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit string m Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl string 2d time to live. Available units are [s, m, h]
allowedPaths []PathConfig [{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}] Allowed paths that can be accessed using the key.
settingId string 98daa3ae-961d-4253-bf6a-322a32fdca3d This field is DEPERCATED. Use settingIds field instead.
settingIds string [98daa3ae-961d-4253-bf6a-322a32fdca3d] Setting ids associated with the key.
shouldLogRequest bool false Should request be stored.
shouldLogResponse bool true Should response be stored.
rotationEnabled bool false Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId string 98daa3ae-961d-4253-bf6a-322a32fdca3d Policy id associated with the key.
Create a provider setting: POST /api/provider-settings
Description

This endpoint is creating a provider setting.

Request
Field required type example description
provider required enum openai This value can only be openai, anthropic and azure as for now.
setting required Setting { "apikey": "YOUR_OPENAI_KEY" } A map of values used for authenticating with the selected provider.
name optional string YOUR_PROVIDER_SETTING_NAME This field is used for giving a name to provider setting
allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Setting

Field required type example description
apiKey required string xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx This value is required.
url required string https://your.deployment.url This value is required when the provider is vllm
resourceName required string YOUR_AZURE_RESOURCE_NAME This value is required when the provider is azure.
Error Response
http code content-type
400, 500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/provider-settings
Response
Field type example description
createdAt int64 1699933571 Unix timestamp for creation time.
updatedAt int64 1699933571 Unix timestamp for update time.
provider enum openai This value can only be openai as for now.
id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier.
name string YOUR_PROVIDER_SETTING_NAME Provider setting name.
allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.
Get provider settings: GET /api/provider-settings
Description

This endpoint is getting provider settings.

Query Parameters
name type data type description
ids optional []string Provider setting ids
Error Response
http code content-type
500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/provider-settings
Response
[]ProviderSetting

ProviderSetting

Field type example description
createdAt int64 1699933571 Unix timestamp for creation time.
updatedAt int64 1699933571 Unix timestamp for update time.
provider enum openai This value can only be openai as for now.
id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier.
name string YOUR_PROVIDER_SETTING_NAME Provider setting name.
allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.
Update a provider setting: PATCH /api/provider-settings/:id
Description

This endpoint is updating a provider setting .

Parameters
name type data type description
id required string Unique identifier for the provider setting that you want to update.
Request
Field required type example description
setting required Setting { "apikey": "YOUR_OPENAI_KEY" } A map of values used for authenticating with the selected provider.
name optional string YOUR_PROVIDER_SETTING_NAME This field is used for giving a name to provider setting
allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.

Setting

Field required type example description
apiKey required string xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx This value is required.
url required string https://your.deployment.url This value is required when the provider is vllm
resourceName required string YOUR_AZURE_RESOURCE_NAME This value is required when the provider is azure.
Error Response
http code content-type
400, 500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/provider-settings
Response
Field type example description
createdAt int64 1699933571 Unix timestamp for creation time.
updatedAt int64 1699933571 Unix timestamp for update time.
provider enum openai This value can only be openai as for now.
id string 98daa3ae-961d-4253-bf6a-322a32fdca3d This value is a unique identifier
name string YOUR_PROVIDER_SETTING_NAME Provider setting name.
allowedModels []string ["text-embedding-ada-002"] Allowed models for this provider setting.
Retrieve Metrics: POST /api/reporting/events
Description

This endpoint is retrieving aggregated metrics given an array of key ids and tags.

Request
Field required type example description
keyIds required []string ["key-1", "key-2", "key-3" ] Array of ids that specicify the keys that you want to aggregate stats from.
tags required []string ["tag-1", "tag-2"] Array of tags that specicify the key tags that you want to aggregate stats from.
customIds required []string ["customId-1", "customId-2"] A list of custom IDs that you want to aggregate stats from.
filters required []string ["model", "keyId"] Group by data points through different filters(model,keyId, userId or customId).
start required int64 1699933571 Start timestamp for the requested timeseries data.
end required int64 1699933571 End timestamp for the requested timeseries data.
increment required int 60 This field is the increment in seconds for the requested timeseries data.
Error Response
http code content-type
500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/provider-settings
Response
Field type example description
dataPoints []dataPoint [{ "timeStamp": 1699933571, "numberOfRequests": 1, "costInUsd": 0.8, "latencyInMs": 600, "promptTokenCount": 0, "completionTokenCount": 0, "successCount": 1 }] Unix timestamp for creation time.
latencyInMsMedian float64 656.7 Median latency for the given time period.
latencyInMs99th float64 555.7 99th percentile latency for the given time period.

Datapoint

Field type example description
timeStamp int64 1702504746 Unix timestamp for the data point
numberOfRequests int64 100 Aggregated number of http requests over the given time increment.
costInUsd float64 1.7 Aggregated cost of proxied requests in USD over the given time increment.
latencyInMs int 555 Aggregated latency in milliseconds of http requests over the given time increment.
promptTokenCount int 25 Aggregated prompt token counts over the given time increment.
completionTokenCount int 4000 Aggregated completion token counts over the given time increment.
successCount int 555 Aggregated number of successful http requests over the given time increment.
keyId int 555.7 key Id associated with the event.
model string gpt-3.5-turbo model associated with the event.
customId string customId customId associated with the event.
Get events: GET /api/events
Description

This endpoint is for getting events.

Query Parameters
name type data type description
customId optional string Custom identifier attached to an event.
keyIds optional []string A list of key IDs.
start required if keyIds is specified int64 Start timestamp.
end required if keyIds is specified int64 End timestamp.
Error Response
http code content-type
500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/provider-settings
Response
[]Event

Event

Field type example description
id int64 1699933571 Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
tags int64 ["YOUR_TAG"] Tags of the key.
key_id string YOUR_KEY_ID Key Id associated with the proxy request.
cost_in_usd float64 0.0004 Cost incured by the proxy request.
model string gpt-4-1105-preview Model used in the proxy request.
provider string openai Provider for the proxy request.
status int 200 Http status.
prompt_token_count int 8 Prompt token count of the proxy request.
completion_token_count int 16 Completion token counts of the proxy request.
latency_in_ms int 160 Provider setting name.
path string /api/v1/chat/completion Provider setting name.
method string POST Http method for the assoicated proxu request.
custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.
request []byte {} Request in bytes to providers.
custom_id string YOUR_CUSTOM_ID Custom Id passed by the user in the headers of proxy requests.
response []byte {} Response in bytes from providers.
Create custom provider: POST /api/custom/providers
Description

This endpoint is creating custom providers.

RouteConfig
Field required type example description
path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.
target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.
model_location required string model JSON field for the model in the HTTP request.
request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.
response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.
stream_location required string stream JSON field for the stream boolean in the HTTP request.
stream_end_word required string [DONE] End word for the stream.
stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.
stream_max_empty_messages required int 10 Number of max empty messages in stream.
Request
Field required type example description
provider required string bricks Unique identifier associated with the route config.
route_configs required []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.
authentication_param optional string apikey The authentication parameter required for.
Error Response
http code content-type
500, 400 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/custom/providers
Response
Field type example description
id int64 1699933571 Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
provider string bricks Unique identifier associated with the route config.
route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.
authentication_param string apikey The authentication parameter required for.
Update custom provider: PATCH /api/custom/providers/:id
Description

This endpoint is updating a custom provider.

RouteConfig
Field required type example description
path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.
target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.
model_location required string model JSON field for the model in the HTTP request.
request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.
response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.
stream_location required string stream JSON field for the stream boolean in the HTTP request.
stream_end_word required string [DONE] End word for the stream.
stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.
stream_max_empty_messages required int 10 Number of max empty messages in stream.
Request
Field required type example description
route_configs optional []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.
authentication_param optional string apikey The authentication parameter required for.
Error Response
http code content-type
500, 404, 400 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/custom/providers
Response
Field type example description
id int64 1699933571 Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
provider string bricks Unique identifier associated with the route config.
route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.
authentication_param string apikey The authentication parameter required for.
Create a policy: POST /api/policies
Description

This endpoint is for creating a policy.

Request
Field required type example description
name optional string Pivacy Policy #1 Name for the policy.
tags required []string ["orgId-1"] Associated tags.
config optional config {"rules": { "address": "block" }} PII detection rules.
regexConfig optional regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.
Config
Field required type example description
rules required map[Rule]Action { "address": "block" } PII entities mapped to their associated actions.
RegexConfig
Field required type example description
definition required string [2-9]|[12]\d|3[0-6] Regex definition.
action required Action block Action associated with the regex definition.
Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response
http code content-type
500 application/json
Field type example
status int 400, 500
title string request body reader error
type string /errors/policies
detail string something is wrong
instance string /api/policies
Response
Field type example description
id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
tags []string ["org-111"] Tags attached to policies.
config config {"rules": { "address": "block" }} PII detection rules.
regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.
Update a policy: PATCH /api/policies/:id
Description

This endpoint is for updating a policy.

Request
Field required type example description
name optional string Pivacy Policy #1 Name for the policy.
tags required []string ["orgId-1"] Associated tags.
config optional config {"rules": { "address": "block" }} PII detection rules.
regexConfig optional regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.
Config
Field required type example description
rules required map[Rule]Action { "address": "block" } PII entities mapped to their associated actions.
RegexConfig
Field required type example description
definition required string [2-9]|[12]\d|3[0-6] Regex definition.
action required Action block Action associated with the regex definition.
Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response
http code content-type
500 application/json
Field type example
status int 400, 500
title string request body reader error
type string /errors/policies
detail string something is wrong
instance string /api/policies
Response
Field type example description
id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
tags []string ["org-111"] Tags attached to policies.
config config {"rules": { "address": "block" }} PII detection rules.
regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.
Get policies by tags: GET /api/policies
Description

This endpoint is for retrieving policies by tags.

Query Parameters

| name | type | data type | description | |--------|------------|----------------|------------------------------------------------------|s | tags | required | []string | Tags attached to the policies. |

Response
[]Policy

Policy

Field type example description
id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
tags []string ["org-111"] Tags attached to policies.
config config {"rules": { "address": "block" }} PII detection rules.
regexConfig regexConfig {"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]} Regular expression rules.
Get custom providers: GET /api/custom/providers
Description

This endpoint is for getting custom providers.

RouteConfig
Field required type example description
path required string /chat/completion Path associated with the custom provider route. It must be unique within the custom provider.
target_url required string https://api.openai.com/v1/chat/completions Proxy destination URL for the custom provider route.
model_location required string model JSON field for the model in the HTTP request.
request_prompt_location required string messages.#.content JSON field for the prompt request in the HTTP request.
response_completion_location required string choices.#.message.content JSON field for the completion content in the HTTP response.
stream_location required string stream JSON field for the stream boolean in the HTTP request.
stream_end_word required string [DONE] End word for the stream.
stream_response_completion_location required string choices.#.delta.content JSON field for the completion content in the streaming response.
stream_max_empty_messages required int 10 Number of max empty messages in stream.
Request
Field required type example description
route_configs optional []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Route configurations for the custom provider.
authentication_param optional string apikey The authentication parameter required for.
Error Response
http code content-type
500 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/custom/providers
Response
[]Provider

Provider

Field type example description
id int64 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb Unique identifier associated with the event.
created_at int64 1699933571 Unix timestamp for creation time.
updated_at int64 1699933571 Unix timestamp for update time.
provider string bricks Unique identifier associated with the route config.
route_configs []RouteConfig {{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }} Start timestamp for the requested timeseries data.
authentication_param string apikey The authentication parameter required for.
Create routes: POST /api/routes
Description

This endpoint is for creating routes.

StepConfig
Field required type example description
provider required enum azure Provider for the step. Can only be either azure or openai.
model required string gpt-3.5-turbo Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries optional int 2 Number of retries.
params optional object { deploymentId: "ada-test",apiVersion: "2022-12-01" } Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout optional string 5s Timeout desired for each request. Default value is 5m.
CacheConfig
Field required type example description
enabled required bool false Boolean flag indicating whether caching is enabled.
ttl optional string 5s TTL for the cache. Default value is 168h.
Request
Field required type example description
name required string staging-openai-azure-completion-route Name for the route.
path required string / Unique identifier for.
steps required []StepConfig apikey The authentication parameter required for.
keyIds required []string [] The authentication parameter required for.
cacheConfig required CacheConfig [] The authentication parameter required for.
Error Response
http code content-type
500, 400 application/json
Field type example
status int 400
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/routes
Response
Field type example description
id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt required string 1699933571
updatedAt required string 1699933571
name required string staging-openai-azure-completion-route
path required string /production/chat/completion
steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }
Retrieve a route: GET /api/routes/:id
Description

This endpoint is for retrieving a route.

Parameters
name type data type description
id required string Unique identifier for the route.
Error Response
http code content-type
500, 404 application/json
Field type example
status int 404
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/routes/:id
Response
Field type example description
id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt required string 1699933571
updatedAt required string 1699933571
name required string staging-openai-azure-completion-route
path required string /production/chat/completion
steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }
Retrieve routes: GET /api/routes
Description

This endpoint is for retrieving routes.

Error Response
http code content-type
500 application/json
Field type example
status int 404
title string request body reader error
type string /errors/request-body-read
detail string something is wrong
instance string /api/routes/:id
CacheConfig
Field required type example description
enabled required bool false Boolean flag indicating whether caching is enabled.
ttl optional string 5s TTL for the cache. Default value is 168h.
StepConfig
Field required type example description
provider required enum azure Provider for the step. Can only be either azure or openai.
model required string gpt-3.5-turbo Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries optional int 2 Number of retries.
params optional object { deploymentId: "ada-test",apiVersion: "2022-12-01" } Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout optional string 5s Timeout desired for each request. Default value is 5m.
RouteConfig
Field type example description
id required string 9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt required string 1699933571
updatedAt required string 1699933571
name required string staging-openai-azure-completion-route
path required string /production/chat/completion
steps required []StepConfig [{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds required []string ["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig required CacheConfig { "enabled": false, "ttl": "5s" }
Response
[]RouteConfig

OpenAI Proxy

The OpenAI proxy runs on Port 8002.

Headers
name type data type description
x-custom-event-id optional string Custom Id that can be used to retrieve an event associated with each proxy request.
Chat Completion
Call OpenAI chat completions: POST /api/providers/openai/v1/chat/completions
Description

This endpoint is set up for proxying OpenAI chat completion requests. Documentation for this endpoint can be found here.

Embeddings
Call OpenAI embeddings: POST /api/providers/openai/v1/embeddings
Description

This endpoint is set up for proxying OpenAI embedding requests. Documentation for this endpoint can be found here.

Moderations
Call OpenAI moderations: POST /api/providers/openai/v1/moderations
Description

This endpoint is set up for proxying OpenAI moderation requests. Documentation for this endpoint can be found here.

Models
Get OpenAI models: GET /api/providers/openai/v1/models
Description

This endpoint is set up for retrieving OpenAI models. Documentation for this endpoint can be found here.

Retrieve an OpenAI model: GET /api/providers/openai/v1/models/:model
Description

This endpoint is set up for retrieving an OpenAI model. Documentation for this endpoint can be found here.

Files
List files: GET /api/providers/openai/v1/files
Description

This endpoint is set up for list OpenAI files. Documentation for this endpoint can be found here.

Upload a file: POST /api/providers/openai/v1/files
Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Delete a file: POST /api/providers/openai/v1/files/:file_id
Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Retrieve a file: GET /api/providers/openai/v1/files/:file_id
Description

This endpoint is set up for retrieving an OpenAI file. Documentation for this endpoint can be found here.

Retrieve file content: GET /api/providers/openai/v1/files/:file_id/content
Description

This endpoint is set up for retrieving an OpenAI file content. Documentation for this endpoint can be found here.

Images
Generate images: POST /api/providers/openai/v1/images/generations
Description

This endpoint is set up for generating OpenAI images. Documentation for this endpoint can be found here.

Edit images: POST /api/providers/openai/v1/images/edits
Description

This endpoint is set up for editting OpenAI generated images. Documentation for this endpoint can be found here.

Create image variations: POST /api/providers/openai/v1/images/variations
Description

This endpoint is set up for creating OpenAI image variations. Documentation for this endpoint can be found here.

Voices
Create speech: POST /api/providers/openai/v1/audio/speech
Description

This endpoint is set up for creating speeches. Documentation for this endpoint can be found here.

Create transcriptions: POST /api/providers/openai/v1/audio/transcriptions
Description

This endpoint is set up for editting generated images. Documentation for this endpoint can be found here.

Create translations: POST /api/providers/openai/v1/audios/translations
Description

This endpoint is set up for creating translations. Documentation for this endpoint can be found here.

Assistants
Create assistant: POST /api/providers/openai/v1/assistants
Description

This endpoint is set up for creating an OpenAI assistant. Documentation for this endpoint can be found here.

Retrieve assistant: GET /api/providers/openai/v1/assistants/:assistant_id
Description

This endpoint is set up for retrieving an OpenAI assistant. Documentation for this endpoint can be found here.

Modify assistant: POST /api/providers/openai/v1/assistants/:assistant_id
Description

This endpoint is set up for modifying an OpenAI assistant. Documentation for this endpoint can be found here.

Delete assistant: DELETE /api/providers/openai/v1/assistants/:assistant_id
Description

This endpoint is set up for deleting an OpenAI assistant. Documentation for this endpoint can be found here.

List assistants: GET /api/providers/openai/v1/assistants
Description

This endpoint is set up for listing OpenAI assistants. Documentation for this endpoint can be found here.

Create assistant file: POST /api/providers/openai/v1/assistants/:assistant_id/files
Description

This endpoint is set up for creating an OpenAI assistant file. Documentation for this endpoint can be found here.

Retrieve assistant file: GET /api/providers/openai/v1/assistants/:assistant_id/files/:file_id
Description

This endpoint is set up for retrieving an OpenAI assistant file. Documentation for this endpoint can be found here.

Delete assistant file: DELETE /api/providers/openai/v1/assistants/:assistant_id/files/:file_id
Description

This endpoint is set up for deleting an OpenAI assistant file. Documentation for this endpoint can be found here.

List assistant files: GET /api/providers/openai/v1/assistants/:assistant_id/files
Description

This endpoint is set up for retrieving OpenAI assistant files. Documentation for this endpoint can be found here.

Create thread: POST /api/providers/openai/v1/threads
Description

This endpoint is set up for creating an OpenAI thread. Documentation for this endpoint can be found here.

Retrieve thread: GET /api/providers/openai/v1/threads/:thread_id
Description

This endpoint is set up for retrieving an OpenAI thread. Documentation for this endpoint can be found here.

Modify thread: POST /api/providers/openai/v1/threads/:thread_id
Description

This endpoint is set up for modifying an OpenAI thread. Documentation for this endpoint can be found here.

Delete thread: DELETE /api/providers/openai/v1/threads/:thread_id
Description

This endpoint is set up for deleting an OpenAI thread. Documentation for this endpoint can be found here.

Create message: POST /api/providers/openai/v1/threads/:thread_id/messages
Description

This endpoint is set up for creating an OpenAI message. Documentation for this endpoint can be found here.

Retrieve message: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id
Description

This endpoint is set up for retrieving an OpenAI message. Documentation for this endpoint can be found here.

Modify message: POST /api/providers/openai/v1/files/:file_id/content
Description

This endpoint is set up for modifying an OpenAI message. Documentation for this endpoint can be found here.

List messages: GET /api/providers/openai/v1/threads/:thread_id/messages
Description

This endpoint is set up for listing OpenAI messages. Documentation for this endpoint can be found here.

Retrieve message file: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files/:file_id
Description

This endpoint is set up for retrieving an OpenAI message file. Documentation for this endpoint can be found here.

List message files: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files
Description

This endpoint is set up for retrieving OpenAI message files. Documentation for this endpoint can be found here.

Create run: POST /api/providers/openai/v1/threads/:thread_id/runs
Description

This endpoint is set up for creating an OpenAI run. Documentation for this endpoint can be found here.

Retrieve run: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id
Description

This endpoint is set up for retrieving an OpenAI run. Documentation for this endpoint can be found here.

Modify run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id
Description

This endpoint is set up for modifying an OpenAI run. Documentation for this endpoint can be found here.

List runs: GET /api/providers/openai/v1/threads/runs
Description

This endpoint is set up for retrieving OpenAI runs. Documentation for this endpoint can be found here.

Submit tool outputs to run: POST /api/providers/openai/v1/threads/runs
Description

This endpoint is set up for submitting tool outputs to an OpenAI run. Documentation for this endpoint can be found here.

Cancel a run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id/cancel
Description

This endpoint is set up for cancellling an OpenAI run. Documentation for this endpoint can be found here.

Create thread and run: POST /api/providers/openai/v1/threads/runs
Description

This endpoint is set up for creating an OpenAI thread and run. Documentation for this endpoint can be found here.

Retrieve run step: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps/:step_id
Description

This endpoint is set up for retrieving an OpenAI run step. Documentation for this endpoint can be found here.

List run steps: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps
Description

This endpoint is set up for listing OpenAI run steps. Documentation for this endpoint can be found here.

Azure OpenAI Proxy

The custom provider proxy runs on Port 8002.

Create Azure OpenAI chat completion: POST /api/providers/azure/openai/deployments/:deployment_id/chat/completions?api-version={API_VERSION}
Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Create Azure OpenAI embeddings: POST /api/providers/azure/openai/deployments/:deployment_id/embeddings?api-version={API_VERSION}
Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Anthropic Proxy

The custom provider proxy runs on Port 8002.

Create Anthropic completion: POST /api/providers/anthropic/v1/complete
Description

This endpoint is set up for proxying Anthropic completion requests. Documentation for this endpoint can be found here.

Create Anthropic messages: POST /api/providers/anthropic/v1/messages
Description

This endpoint is set up for proxying Anthropic messages requests. Documentation for this endpoint can be found here.

vllm Provider Proxy

The vllm provider proxy runs on Port 8002.

Create chat completions: POST /api/providers/vllm/v1/chat/completions
Description

This endpoint is set up for proxying vllm chat completions requests. Documentation for this endpoint can be found here.

Create completions: POST /api/providers/vllm/v1/completions
Description

This endpoint is set up for proxying vllm completions requests. Documentation for this endpoint can be found here.

Custom Provider Proxy

The custom provider proxy runs on Port 8002.

Call custom providers: POST /api/custom/providers/:provider/*
Description

First you need to use create custom providers endpoint to create custom providers. Then create corresponding provider setting for the newly created custom provider. Afterward, you can start creating keys associated with the custom provider, and use the keys to access this endpoint by placing the created key in Authorization: Bearer YOUR_BRICKSLLM_KEY as part of your HTTP request headers.

Route Proxy

The custom provider proxy runs on Port 8002.

Call a route: POST /api/route/*
Description

Route helps you interpolate different models (embeddings or chat completion models) and providers (OpenAI or Azure OpenAI) to gurantee API responses.

First you need to use create route endpoint to create routes. If the route uses both Azure and OpenAI, you need to create API keys with corresponding provider settings as well. If the route is for chat completion, just call the route using the OpenAI chat completion format. On the other hand, if the route is for embeddings, just call the route using the embeddings format.

Directories

Path Synopsis
cmd
bricksllm command
internal
authenticator
cache
config
errors
event
hasher
key
logger
logger/zap
manager
message
pii
pii/amazon
policy
policy/custom
provider
provider/anthropic
provider/anthropic/assets
provider/azure
provider/custom
provider/openai
provider/vllm
recorder
route
server/web/admin
server/web/proxy
stats
storage/memdb
storage/postgresql
storage/redis
testing
util
validator

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.