olp

module
v0.0.0-...-e5a87b8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: AGPL-3.0

README

OpenLLMProxy

OpenLLMProxy is a self-hosted AI gateway and control plane built with Go, SvelteKit, PostgreSQL, and Valkey. It routes OpenAI, Anthropic, Gemini, and Bedrock client requests across certified provider models. Providers include OpenAI, Anthropic, Gemini, Vertex AI, Amazon Bedrock, Azure OpenAI, and reviewed OpenAI-compatible endpoints.

OpenLLMProxy 0.1.0 is a work in progress. During 0.x it makes no compatibility, upgrade, mixed-version or rollback promises: any release may change the management API, configuration and storage.

Develop locally

Install the development prerequisites, then run:

make setup
make dev

Open http://127.0.0.1:5173 and use .local/secrets/bootstrap.token to create the first owner. See Contributing for the development workflow and Testing for checks and suite selection.

Install

Build the image locally with Compose:

cp .env.example .env
./scripts/prepare-compose-secrets.sh
docker compose --env-file .env \
  -f deploy/compose.yaml -f deploy/compose.build.yaml \
  -f deploy/compose.bootstrap.yaml up --build -d

Visit the configured OLP_PUBLIC_ORIGIN and use deploy/secrets/olp_bootstrap_token for first-owner setup. After setup, recreate the application without the bootstrap overlay and retire the token. To use a published 0.x image, set OLP_IMAGE and omit the build overlay and --build.

In the console, connect a provider, discover and certify models, and activate it. Create and publish a route targeting those models, then issue an API key in the same project with the required scopes and route access. A route is strict unless you declare it transformed; declare it transformed when it translates between dialects, redacts content or uses a provider without a provider profile. See provider routing for route fidelity, credential pools, bulk workflows, and price, latency, throughput, and privacy preferences.

For production, follow deployment and configuration. Gateway, control, and worker modes can run separately; workers persist accounting and reconcile media jobs and budgets.

Make an SDK request

Install the openai JavaScript package and set OLP_API_KEY to a key with inference scope. The model is a published route slug:

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OLP_API_KEY,
  baseURL: 'http://127.0.0.1:5173/v1'
});
const response = await client.responses.create({
  model: 'assistant',
  input: 'Hello'
});
console.log(response.output_text);

Use /anthropic as the Anthropic SDK base URL and /gemini as the Gemini SDK base URL. Bedrock uses /bedrock and separate gateway authentication; see the Bedrock guide, including proxy requirements.

Interface Path
Management API and OpenAPI /api/v1 and /api/v1/openapi.json
OpenAI /v1
Anthropic /anthropic/v1
Gemini /gemini/v1beta and /gemini/v1
Bedrock /bedrock
Private health and metrics Separate observability listener on port 9090

The compatibility matrix lists generation, token counting, embeddings, rerank, moderation, media, and qualified file, batch, realtime, and stored-response operations. Support depends on the provider, model, and certified capability. Persisted telemetry excludes prompts, outputs, credentials, and uploaded content; opt-in provider state may retain content upstream.

Documentation

Guide Covers
Concepts Projects, routes, revisions, keys, budgets, and privacy
Provider routing Onboarding, credential pools, model facts, and selection policy
Provider profiles API dialects, defaults, bindings, and connection security
Provider plugins Installing, approving, and confining operator-installed plugins
Plugin authoring The Go SDK, reference plugin, and plugin ABI
Compatibility Endpoints, supported providers, and translation limits
Client compatibility Tested coding agents, agent SDKs and frameworks, their pinned releases and configuration
Deployment Database setup, topology, capacity, qualification limits, and edge routing
Configuration Variables, secret files, CLI settings, and configuration promotion
Access control Identity, projects, management tokens, and account recovery
Management clients Contract-generated CLI commands, stale-safe configuration promotion, client setup, and scoped management MCP
Configuration Actions Reusable pull-request planning, review comments and merged configuration promotion
Terraform and OpenTofu Individual resource automation and acceptance qualification
Named guardrails Reusable bounded policies, immutable revisions, and portable definitions
Upstream MCP servers Project-owned certified catalogs, immutable revisions, and sealed bearer bindings
Export sinks Signed content-free facts, durable retries, scope isolation, and promotion
Operator console Installation identity, independent-installation bookmarks, bulk member editing, and session-scoped saved views
Model catalog Scoped route discovery, SDK samples, and owner-controlled public catalogs
External secrets Wrapped key rings, workload identities, pinned provider credentials, and safe rotation
Security architecture Trust boundaries, authorization, secrets, egress, and response headers
Gateway execution Admission, attempts, content policies, and durable media
Operations Monitoring, spend reconciliation, backups, key rotation, and versions
Performance Benchmark method, scenarios and targets, the LiteLLM comparison, regression gates, and how to reproduce them
Roadmap Milestones to LiteLLM feature parity and beyond, with the parity matrix
Contributing Setup, tests, architecture, and releases

OpenLLMProxy is licensed under AGPL-3.0-only. Report vulnerabilities through the security policy; community participation follows the Code of Conduct.

Directories

Path Synopsis
cmd
olp command
internal
access
Package access owns installation identity and management transactions.
 Package access owns installation identity and management transactions.
attribution
bodylimit
Package bodylimit defines the operator's optional per-route ingress bound.
 Package bodylimit defines the operator's optional per-route ingress bound.
branding
Package branding manages the installation's authenticated console identity.
 Package branding manages the installation's authenticated console identity.
budgetcalendar
Package budgetcalendar derives civil budget boundaries independently of the machine timezone.
 Package budgetcalendar derives civil budget boundaries independently of the machine timezone.
catalog
Package catalog is the signed reference catalog: model facts and list prices by vendor, transcribed from each vendor's own documentation with the page and time every entry was observed.
 Package catalog is the signed reference catalog: model facts and list prices by vendor, transcribed from each vendor's own documentation with the page and time every entry was observed.
catalog/cmd/catalogdrift command
Command catalogdrift compares the embedded reference catalog with the models each vendor lists today and prints the drift as Markdown.
 Command catalogdrift compares the embedded reference catalog with the models each vendor lists today and prints the drift as Markdown.
catalog/drift
Package drift compares the reference catalog with what vendors list today, so maintainers can see where the catalog has fallen behind.
 Package drift compares the reference catalog with what vendors list today, so maintainers can see where the catalog has fallen behind.
codeadapter
Package codeadapter is the table of code-mode adapters: the plugin profiles each subscription family enrolls through, the client paths and wire protocols it serves, the header that carries its upstream credential, and the coding clients OLP generates configuration for.
 Package codeadapter is the table of code-mode adapters: the plugin profiles each subscription family enrolls through, the client paths and wire protocols it serves, the header that carries its upstream credential, and the coding clients OLP generates configuration for.
codemode
Package codemode defines the observation and management contracts of coding traffic.
 Package codemode defines the observation and management contracts of coding traffic.
codeplans
Package codeplans implements enrollment for API-key coding plans used by OLP code-mode.
 Package codeplans implements enrollment for API-key coding plans used by OLP code-mode.
codewire
Package codewire observes code-mode wire messages without rewriting them: bounded body decoding, header forwarding, SSE event splitting, and the Anthropic Messages and Chat Completions observers and client identities.
 Package codewire observes code-mode wire messages without rewriting them: bounded body decoding, header forwarding, SSE event splitting, and the Anthropic Messages and Chat Completions observers and client identities.
codexauth
Package codexauth implements the Codex device authorization contract.
 Package codexauth implements the Codex device authorization contract.
codexwire
Package codexwire observes Codex wire messages without rewriting them.
 Package codexwire observes Codex wire messages without rewriting them.
config
Package config parses process configuration without opening network connections.
 Package config parses process configuration without opening network connections.
configuration
connectors
Package connectors owns provider addressing and the preparation of upstream requests: hosting, authentication and signing.
 Package connectors owns provider addressing and the preparation of upstream requests: hosting, authentication and signing.
console
Package console serves the separately built client-only SvelteKit console.
 Package console serves the separately built client-only SvelteKit console.
contentpolicy
coordination
Package coordination owns the concrete GLIDE client lifecycle and transport outcome classification.
 Package coordination owns the concrete GLIDE client lifecycle and transport outcome classification.
database
Package database owns PostgreSQL connection lifecycle.
 Package database owns PostgreSQL connection lifecycle.
durablecontract
Package durablecontract owns the strict native file/batch contract.
 Package durablecontract owns the strict native file/batch contract.
egress
Package egress validates provider destinations and builds bounded outbound HTTP clients whose dials are pinned to validated addresses.
 Package egress validates provider destinations and builds bounded outbound HTTP clients whose dials are pinned to validated addresses.
export
gateway
Package gateway serves the native OpenAI inference surface (/v1) and the console playground on top of a pinned runtime release: bounded ingress, API key authentication, route selection, attempt failover, streaming commitment, circuit health, and the terminal request envelope.
 Package gateway serves the native OpenAI inference surface (/v1) and the console playground on top of a pinned runtime release: bounded ingress, API key authentication, route selection, attempt failover, streaming commitment, circuit health, and the terminal request envelope.
geminilifecycle
Package geminilifecycle validates the two distinct native Gemini lifecycle dialects.
 Package geminilifecycle validates the two distinct native Gemini lifecycle dialects.
grants
Package grants holds grants beneath immutable credential versions and runs grant enrollment through provider plugins.
 Package grants holds grants beneath immutable credential versions and runs grant enrollment through provider plugins.
guardrails
Package guardrails manages reusable, project-scoped definitions for the existing bounded content-policy engine.
 Package guardrails manages reusable, project-scoped definitions for the existing bounded content-policy engine.
interaction
Package interaction compiles and binds strict operation contracts.
 Package interaction compiles and binds strict operation contracts.
limits
Package limits admits requests against the request, token, concurrency and cost budgets that every replica shares through Valkey.
 Package limits admits requests against the request, token, concurrency and cost budgets that every replica shares through Valkey.
management
management/contract
Package contract provides primitives to interact with the openapi HTTP API.
 Package contract provides primitives to interact with the openapi HTTP API.
mcpservers
Package mcpservers registers and certifies project-owned upstream MCP servers.
 Package mcpservers registers and certifies project-owned upstream MCP servers.
media
Package media provides the bounded private filesystem spool for staged request and response media, the durable media-job records behind the asynchronous video surface, the reconciliation worker that keeps those records converged with providers, and the media management API.
 Package media provides the bounded private filesystem spool for staged request and response media, the durable media-job records behind the asynchronous video surface, the reconciliation worker that keeps those records converged with providers, and the media management API.
mediacontract
Package mediacontract owns the strict native media contract.
 Package mediacontract owns the strict native media contract.
mfa
Package mfa contains local second-factor cryptographic helpers.
 Package mfa contains local second-factor cryptographic helpers.
modelcatalog
Package modelcatalog projects published routes into a consumer-facing catalog.
 Package modelcatalog projects published routes into a consumer-facing catalog.
notifications
observability
oif
Package oif owns source-preserving, in-process operation contracts.
 Package oif owns source-preserving, in-process operation contracts.
operationplan
Package operationplan compiles strict unary contracts without a generation representation.
 Package operationplan compiles strict unary contracts without a generation representation.
operationregistry
Package operationregistry is the trusted composition root for unary operation contracts.
 Package operationregistry is the trusted composition root for unary operation contracts.
operations
Package operations links trusted operation-owned unary codecs.
 Package operations links trusted operation-owned unary codecs.
operations/classification
Package classification retains operation-owned moderation, classification and similarity views.
 Package classification retains operation-owned moderation, classification and similarity views.
operations/embeddings
Package embeddings retains native vector storage and its logical shape.
 Package embeddings retains native vector storage and its logical shape.
operations/generation
Package generation owns ordered generation views without depending on a wire dialect, provider, transport, or unrelated operation family.
 Package generation owns ordered generation views without depending on a wire dialect, provider, transport, or unrelated operation family.
operations/rerank
Package rerank preserves candidate identity, exact native scores and order.
 Package rerank preserves candidate identity, exact native scores and order.
operations/tokenization
Package tokenization preserves native counting and tokenizer results.
 Package tokenization preserves native counting and tokenizer results.
operations/tokenization/estimate
Package estimate counts the tokens of a request before the provider reports them, which is what admission reserves, what the planner compares with a model's context window, and what budgets charge.
 Package estimate counts the tokens of a request before the provider reports them, which is what admission reserves, what the planner compares with a model's context window, and what budgets charge.
operatorcli
Package operatorcli implements management commands over the public SDK.
 Package operatorcli implements management commands over the public SDK.
pluginindex
Package pluginindex is the signed index of reviewed provider plugins: for each plugin, its source and every reviewed release's module digest, ABI, origins and profiles.
 Package pluginindex is the signed index of reviewed provider plugins: for each plugin, its source and every reviewed release's module digest, ABI, origins and profiles.
pluginindex/cmd/pluginindex command
Command pluginindex builds the plugins the reviewed plugin index lists from this repository, reproducibly, and compares them with the index.
 Command pluginindex builds the plugins the reviewed plugin index lists from this repository, reproducibly, and compares them with the index.
plugins
Package plugins installs provider plugins and runs them confined.
 Package plugins installs provider plugins and runs them confined.
process
Package process explicitly composes the resources owned by each process mode.
 Package process explicitly composes the resources owned by each process mode.
protocols
Package protocols connects native wire codecs through the shared operation model.
 Package protocols connects native wire codecs through the shared operation model.
protocols/awsframe
Package awsframe validates native AWS event framing before the SDK decoder can normalize duplicate headers or allocate from untrusted header lengths.
 Package awsframe validates native AWS event framing before the SDK decoder can normalize duplicate headers or allocate from untrusted header lengths.
protocols/openai
Package openai implements the OpenAI Chat Completions and Responses codecs: gateway envelope validation, upstream encoding with model rewriting, and unary/streaming response decoding with usage extraction.
 Package openai implements the OpenAI Chat Completions and Responses codecs: gateway envelope validation, upstream encoding with model rewriting, and unary/streaming response decoding with usage extraction.
protocols/sse
Package sse decodes and encodes bounded server-sent events independently of provider codecs.
 Package sse decodes and encodes bounded server-sent events independently of provider codecs.
providerinvoke
Package providerinvoke links semantic preparation to a validated hosting profile.
 Package providerinvoke links semantic preparation to a validated hosting profile.
providers
Package providers owns provider connections: configuration drafts, model discovery and certification, credential pools, activation into immutable revisions, and history for the connector matrix.
 Package providers owns provider connections: configuration drafts, model discovery and certification, credential pools, activation into immutable revisions, and history for the connector matrix.
realtimecontract
Package realtimecontract compiles the narrow same-dialect WebSocket contract admitted by strict OpenAI Realtime routes.
 Package realtimecontract compiles the narrow same-dialect WebSocket contract admitted by strict OpenAI Realtime routes.
resources
routes
Package routes serves route drafts, published routes, revision history, and routing simulation on the management surface.
 Package routes serves route drafts, published routes, revision history, and routing simulation on the management surface.
runtime
Package runtime owns the immutable serving snapshot: its schema, digest, validation, deterministic target selection, publication, and the gateway manager that installs releases and refreshes key authority.
 Package runtime owns the immutable serving snapshot: its schema, digest, validation, deterministic target selection, publication, and the gateway manager that installs releases and refreshes key authority.
samlidentity
Package samlidentity implements OLP's bounded, solicited SAML POST profile.
 Package samlidentity implements OLP's bounded, solicited SAML POST profile.
scim
Package scim implements bounded SCIM 2.0 protocol documents and queries.
 Package scim implements bounded SCIM 2.0 protocol documents and queries.
secrets
Package secrets owns secret input, domain-separated digests, and record-bound encryption.
 Package secrets owns secret input, domain-separated digests, and record-bound encryption.
secretstore
Package secretstore resolves version-pinned external secrets and unwraps data keys.
 Package secretstore resolves version-pinned external secrets and unwraps data keys.
signing
Package signing verifies the detached Ed25519 signatures of the documents a release ships: the reference catalog and the plugin index.
 Package signing verifies the detached Ed25519 signatures of the documents a release ships: the reference catalog and the plugin index.
signing/cmd/olpsign command
Command olpsign signs and verifies the documents a release ships: the reference catalog and the plugin index.
 Command olpsign signs and verifies the documents a release ships: the reference catalog and the plugin index.
sinks
Package sinks exports durable, content-free facts through bounded workers.
 Package sinks exports durable, content-free facts through bounded workers.
surface
Package surface classifies public request paths.
 Package surface classifies public request paths.
telemetry
Package telemetry owns optional OTLP/HTTP trace export: exporter construction, endpoint and secret validation, bounded batching, and the allowlisted request and attempt spans the gateway records.
 Package telemetry owns optional OTLP/HTTP trace export: exporter construction, endpoint and secret validation, bounded batching, and the allowlisted request and attempt spans the gateway records.
testutil
Package testutil provides behavioral fixtures without constructing application composition.
 Package testutil provides behavioral fixtures without constructing application composition.
upstream
Package upstream classifies how an exchange with a provider ended: the failure class that governs failover and cooldown, and the upstream acceptance that records what is known about the provider's work.
 Package upstream classifies how an exchange with a provider ended: the failure class that governs failover and cooldown, and the upstream acceptance that records what is known about the provider's work.
usage
Package usage carries the request metadata contract: the content-free event the gateway emits for every request, the wire envelope it travels in, the validation the consumer applies before any of it becomes money, and the health rows the recovery workers checkpoint into.
 Package usage carries the request metadata contract: the content-free event the gateway emits for every request, the wire envelope it travels in, the validation the consumer applies before any of it becomes money, and the health rows the recovery workers checkpoint into.
vendors
Package vendors holds the reviewed vendor contracts: the documented facts the onboarding catalogue, the connector matrix and the request codecs share.
 Package vendors holds the reviewed vendor contracts: the documented facts the onboarding catalogue, the connector matrix and the request codecs share.
workload
Package workload verifies bounded workload JWTs against operator-declared issuers.
 Package workload verifies bounded workload JWTs against operator-declared issuers.
Package openapi embeds the checked-in management contract, independently of application composition and generated Go transport types.
 Package openapi embeds the checked-in management contract, independently of application composition and generated Go transport types.
plugins
codex command
opencode-go command
zai-coding command
sdk
management
Package management provides contract-generated operations and a bounded HTTP client for OLP's management API.
 Package management provides contract-generated operations and a bounded HTTP client for OLP's management API.
plugin
Package plugin is the Go SDK for OpenLLMProxy provider plugins.
 Package plugin is the Go SDK for OpenLLMProxy provider plugins.
plugin/abi
Package abi defines the OpenLLMProxy provider plugin ABI: the functions a plugin module exports, the host function it may import, and the JSON messages that cross between them.
 Package abi defines the OpenLLMProxy provider plugin ABI: the functions a plugin module exports, the host function it may import, and the JSON messages that cross between them.
plugin/reference command
Command reference is the reference provider plugin built on the Go SDK.
 Command reference is the reference provider plugin built on the Go SDK.
plugin/templates/oauth-client-credentials command
Command oauth-client-credentials is a template for a provider plugin whose upstream authenticates API calls with short-lived access tokens issued to OAuth 2.0 clients by the client credentials grant (RFC 6749, section 4.4).
 Command oauth-client-credentials is a template for a provider plugin whose upstream authenticates API calls with short-lived access tokens issued to OAuth 2.0 clients by the client credentials grant (RFC 6749, section 4.4).
plugin/templates/signed-request command
Command signed-request is a template for a provider plugin whose upstream authenticates each request by a signature rather than a bearer token: the secret never travels, so a captured request cannot be replayed against another path, body or time.
 Command signed-request is a template for a provider plugin whose upstream authenticates each request by a signature rather than a bearer token: the secret never travels, so a captured request cannot be replayed against another path, body or time.
plugin/templates/token-exchange command
Command token-exchange is a template for a provider plugin whose upstream takes short-lived bearer tokens that an authorization server issues in exchange for a long-lived API key, as OAuth 2.0 token exchange (RFC 8693) describes and cloud IAM services commonly do.
 Command token-exchange is a template for a provider plugin whose upstream takes short-lived bearer tokens that an authorization server issues in exchange for a long-lived API key, as OAuth 2.0 token exchange (RFC 8693) describes and cloud IAM services commonly do.
tests
clientfixture command
clientfixture serves the OLP inference surface in front of the scripted upstream of tests/clients, so real client releases run against the real gateway: bounded ingress, key authentication, route selection, credential injection, model rewriting and native streaming.
 clientfixture serves the OLP inference surface in front of the scripted upstream of tests/clients, so real client releases run against the real gateway: bounded ingress, key authentication, route selection, credential injection, model rewriting and native streaming.
clientfixture/scripted
Package scripted serves deterministic upstream vendors for client qualification: OpenAI Chat Completions, Responses and embeddings, Anthropic Messages, and Gemini generation, counting and embeddings.
 Package scripted serves deterministic upstream vendors for client qualification: OpenAI Chat Completions, Responses and embeddings, Anthropic Messages, and Gemini generation, counting and embeddings.
codecli
Package codecli drives the pinned, unmodified official Codex executable.
 Package codecli drives the pinned, unmodified official Codex executable.
fidelity
Package fidelity compares externally observable protocol documents without importing production codecs.
 Package fidelity compares externally observable protocol documents without importing production codecs.
fixtures
Package fixtures preserves the language-neutral reference corpus used by protocol, routing, and egress conformance tests.
 Package fixtures preserves the language-neutral reference corpus used by protocol, routing, and egress conformance tests.
fixtures/codex-qualified
Package codexfixture is a controlled wire peer, never evidence of subscription eligibility.
 Package codexfixture is a controlled wire peer, never evidence of subscription eligibility.
fixtures/fidelity
Package fidelity contains independently authored, versioned reference inputs.
 Package fidelity contains independently authored, versioned reference inputs.
samlfixture command
The loopback-only SAML identity provider used by Chromium journeys.
 The loopback-only SAML identity provider used by Chromium journeys.
sdkfixture command
sdkfixture serves the OLP inference surface against an in-process mock upstream so the official SDKs exercise the real gateway: bounded ingress, key authentication, route selection, credential injection, model rewriting, and native streaming.
 sdkfixture serves the OLP inference surface against an in-process mock upstream so the official SDKs exercise the real gateway: bounded ingress, key authentication, route selection, credential injection, model rewriting, and native streaming.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.