llm-proxy

❤️ Sponsor

Atlas Cloud Atlas Cloud is a full-modal AI inference platform that gives developers a single AI API to access video generation, image generation, and LLM APIs. Instead of managing multiple vendor integrations, you connect once and get unified access to 300+ curated models across all modalities. Check out Atlas Cloud's new coding plan promotion for more budget-friendly API access.

Krill AI Thank you to Krill AI for sponsoring this project. Krill provides official, stable, high-speed API relay services for GPT, Claude, Gemini, and a wide range of Chinese models, with enterprise customization, invoicing support, and dedicated technical support 16 hours a day, 7 days a week. Its optimized WebSocket connection can deliver a faster time to first token. Register through this link and enter the llm-proxy coupon code when topping up to receive 23% off your first Codex plan purchase.

Sponsor	Description
Bloome	Try llm-proxy with zero local setup on Bloome: Quick start, one click from your browser or phone, easy to share with your team.

📖 Project Introduction

All-in-one AI Development Platform

llm-proxy is the AI gateway that lets you switch between model providers without changing a single line of code.

Whether you're using OpenAI SDK, Anthropic SDK, or any AI SDK, llm-proxy transparently translates your requests to work with any supported model provider. No refactoring, no SDK swaps—just change a configuration and you're done.

What it solves:

• 🔒 Vendor lock-in - Switch from GPT-4 to Claude or Gemini instantly
• 🔧 Integration complexity - One API format for 10+ providers
• 📊 Observability gap - Complete request tracing out of the box
• 💸 Cost control - Real-time usage tracking and budget management

Core Features

📚 Documentation

For detailed technical documentation, API references, architecture design, and more:

• 📑 Documentation Index - Complete documentation navigation
• 🧭 Backend Architecture - Backend modules, request pipeline, Adapter runtime and troubleshooting
• 🖥️ Frontend Architecture - Frontend pages, API boundaries, UI state and verification
• 
• 

🎯 Demo

Try llm-proxy live at our demo instance!

Note：The demo instance currently configures Zhipu and OpenRouter free models.

Demo Account

• Email: demo@example.com
• Password: 12345678

⭐ Features

📸 Screenshots

Here are some screenshots of llm-proxy in action:

System Dashboard	Channel Management	Model Price
Models	Trace Viewer	Request Monitoring

🚀 API Types

🤖 Supported Providers

🚀 Quick Start

30-Second Local Start

# Download and extract (macOS ARM64 example)
curl -sSL https://github.com/mutallipp/llm-proxy/releases/latest/download/llm-proxy_darwin_arm64.tar.gz | tar xz
cd llm-proxy_*

# Run with SQLite (default)
./llm-proxy

# Open http://localhost:8090
# First run: Follow the setup wizard to initialize the system (create admin account, password must be at least 6 characters)

That's it! Now configure your first AI channel and start calling models through llm-proxy.

Zero-Code Migration Example

Your existing code works without any changes. Just point your SDK to llm-proxy:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8090/v1",  # Point to llm-proxy
    api_key="your-llm-proxy-api-key"        # Use llm-proxy API key
)

# Call Claude using OpenAI SDK!
response = client.chat.completions.create(
    model="claude-3-5-sonnet",  # Or gpt-4, gemini-pro, deepseek-chat...
    messages=[{"role": "user", "content": "Hello!"}]
)

Switch models by changing one line: model="gpt-4" → model="claude-3-5-sonnet". No SDK changes needed.

1-click Deploy to Render

Deploy llm-proxy with 1-click on Render for free.

🚀 Deployment Guide

💻 Personal Computer Deployment

Perfect for individual developers and small teams. No complex configuration required.

Quick Download & Run

1. Download the latest release from GitHub Releases

   • Choose the appropriate version for your operating system:

2. Extract and run

   # Extract the downloaded file
   unzip llm-proxy_*.zip
   cd llm-proxy_*
   
   # Add execution permissions (only for Linux/macOS)
   chmod +x llm-proxy
   
   # Run directly - default SQLite database
   
   # Install llm-proxy to system
   sudo ./install.sh
   
   # Start llm-proxy service
   ./start.sh
   
   # Stop llm-proxy service
   ./stop.sh
   

3. Access the application

   http://localhost:8090
   

🖥️ Server Deployment

For production environments, high availability, and enterprise deployments.

Database Support

llm-proxy supports multiple databases to meet different scale deployment needs:

Configuration

llm-proxy uses YAML configuration files with environment variable override support:

# config.yml
server:
  port: 8090
  name: "llm-proxy"
  debug: false

db:
  dialect: "tidb"
  dsn: "<USER>.root:<PASSWORD>@tcp(gateway01.us-west-2.prod.aws.tidbcloud.com:4000)/llm-proxy?tls=true&parseTime=true&multiStatements=true&charset=utf8mb4"

log:
  level: "info"
  encoding: "json"

Environment variables:

LLM_PROXY_SERVER_PORT=8090
LLM_PROXY_DB_DIALECT="tidb"
LLM_PROXY_DB_DSN="<USER>.root:<PASSWORD>@tcp(gateway01.us-west-2.prod.aws.tidbcloud.com:4000)/llm-proxy?tls=true&parseTime=true&multiStatements=true&charset=utf8mb4"
LLM_PROXY_LOG_LEVEL=info

For detailed configuration instructions, please refer to configuration documentation.

Docker Compose Deployment

# Clone project
git clone https://github.com/mutallipp/llm-proxy.git
cd llm-proxy

# Set environment variables
export LLM_PROXY_DB_DIALECT="tidb"
export LLM_PROXY_DB_DSN="<USER>.root:<PASSWORD>@tcp(gateway01.us-west-2.prod.aws.tidbcloud.com:4000)/llm-proxy?tls=true&parseTime=true&multiStatements=true&charset=utf8mb4"

# Start services
docker compose up -d --build --force-recreate

# Check status
docker compose ps

Helm Kubernetes Deployment

Deploy llm-proxy on Kubernetes using the official Helm chart:

# Quick installation
git clone https://github.com/mutallipp/llm-proxy.git
cd llm-proxy
helm install llm-proxy ./deploy/helm

# Production deployment
helm install llm-proxy ./deploy/helm -f ./deploy/helm/values-production.yaml

# Access llm-proxy
kubectl port-forward svc/llm-proxy 8090:8090
# Visit http://localhost:8090

Key Configuration Options:

For detailed configuration and troubleshooting, see Helm Chart Documentation.

Virtual Machine Deployment

Download the latest release from GitHub Releases

# Extract and run
unzip llm-proxy_*.zip
cd llm-proxy_*

# Set environment variables
export LLM_PROXY_DB_DIALECT="tidb"
export LLM_PROXY_DB_DSN="<USER>.root:<PASSWORD>@tcp(gateway01.us-west-2.prod.aws.tidbcloud.com:4000)/llm-proxy?tls=true&parseTime=true&multiStatements=true&charset=utf8mb4"

sudo ./install.sh

# Configuration file check
llm-proxy config check

# Start service
#  For simplicity, we recommend managing llm-proxy with the helper scripts:

# Start
./start.sh

# Stop
./stop.sh

📖 Usage Guide

Unified API Overview

llm-proxy provides a unified API gateway that supports both OpenAI Chat Completions and Anthropic Messages APIs. This means you can:

• Use OpenAI API to call Anthropic models - Keep using your OpenAI SDK while accessing Claude models
• Use Anthropic API to call OpenAI models - Use Anthropic's native API format with GPT models
• Use Gemini API to call OpenAI models - Use Gemini's native API format with GPT models
• Automatic API translation - llm-proxy handles format conversion automatically
• Zero code changes - Your existing OpenAI or Anthropic client code continues to work

1. Initial Setup

1. Access Management Interface

   http://localhost:8090
   

2. Configure AI Providers

   • Add API keys in the management interface
   • Test connections to ensure correct configuration

3. Create Users and Roles

   • Set up permission management
   • Assign appropriate access permissions

2. Channel Configuration

Configure AI provider channels in the management interface. For detailed information on channel configuration, including model mappings, parameter overrides, and troubleshooting, see the Channel Configuration Guide.

3. Model Management

llm-proxy provides a flexible model management system that supports mapping abstract models to specific channels and model implementations through Model Associations. This enables:

• Unified Model Interface - Use abstract model IDs (e.g., gpt-4, claude-3-opus) instead of channel-specific names
• Intelligent Channel Selection - Automatically route requests to optimal channels based on association rules and load balancing
• Flexible Mapping Strategies - Support for precise channel-model matching, regex patterns, and tag-based selection
• Priority-based Fallback - Configure multiple associations with priorities for automatic failover

For comprehensive information on model management, including association types, configuration examples, and best practices, see the Model Management Guide.

4. Create API Keys

Create API keys to authenticate your applications with llm-proxy. Each API key can be configured with multiple profiles that define:

• Model Mappings - Transform user-requested models to actual available models using exact match or regex patterns
• Channel Restrictions - Limit which channels an API key can use by channel IDs or tags
• Model Access Control - Control which models are accessible through a specific profile
• Profile Switching - Change behavior on-the-fly by activating different profiles

For detailed information on API key profiles, including configuration examples, validation rules, and best practices, see the API Key Profile Guide.

5. AI Coding Tools Integration

See the dedicated guides for detailed setup steps, troubleshooting, and tips on combining these tools with llm-proxy model profiles:

• OpenCode Integration Guide
• Claude Code Integration Guide
• Codex Integration Guide

6. SDK Usage

For detailed SDK usage examples and code samples, please refer to the API documentation:

• OpenAI API
• Anthropic API
• Gemini API

🛠️ Development Guide

For detailed development instructions, architecture design, and contribution guidelines, please see docs/en/development/development.md.

本地开发 vs 生产部署

dev 与 prod 状态完全隔离：dev DB 库名为 llm-proxy-dev（独立 PostgreSQL 库），dev 容器名为 llm-proxy-dev，互不冲突。.env.dev 是可提交的默认配置；首次运行前，在被 Git 忽略的 .env.dev.local 写入完整 LLM_PROXY_DB_DSN。如需以线上库的最新表结构初始化空 dev 库，执行 make dev-db-sync-schema：它同步表、索引、约束与序列，但不会复制业务数据、API Key、OAuth token 或渠道 Cookie；dev 库非空时会拒绝执行，避免误覆盖开发数据。需要完整复刻线上数据时，执行 make dev-db-sync-full：它会先停止 dev、销毁并重建 dev DB，再导入完整 prod 快照，包括 API Key、OAuth token 与渠道 Cookie，且会覆盖现有 dev 数据。

环境变量已统一为 LLM_PROXY_* 前缀（旧 AXONHUB_* 已彻底移除）。常见字段：

LLM_PROXY_DB_DSN          # 数据库连接串
LLM_PROXY_DB_DIALECT      # postgres | sqlite | mysql | tidb
LLM_PROXY_SERVER_PORT     # 监听端口
LLM_PROXY_SERVER_HOST     # 监听地址
LLM_PROXY_LOG_LEVEL       # debug | info | warn | error
LLM_PROXY_HTTP_PROXY      # 出口代理（可选）

完整配置字段见 conf/conf.go 和 config.example.yml。

🤝 Acknowledgments

• 🙏 musistudio/llms - LLM transformation framework, source of inspiration
• 🎨 satnaing/shadcn-admin - Admin interface template
• 🔧 99designs/gqlgen - GraphQL code generation
• 🌐 gin-gonic/gin - HTTP framework
• 🗄️ ent/ent - ORM framework
• 🔧 air-verse/air - Auto reload Go service
• ☁️ Render - Free cloud deployment platform for hosting our demo
• 🗃️ TiDB Cloud - Serverless database platform for demo deployment

📄 License

This project is licensed under multiple licenses (Apache-2.0 and LGPL-3.0). See LICENSE file for the detailed licensing overview and terms.