llm-router

command module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: MIT Imports: 2 Imported by: 0

README

llm-router

Single-binary, bloat-free, extensible multi-provider OpenAI-compatible LLM gateway. Go on the backend, Svelte on the front, compiled into one file.

llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Zero external dependencies.


What It Does

Sits between your applications and LLM providers, exposing a single OpenAI-compatible /v1 API.

  • Load distribution: Split traffic across multiple providers and provider accounts.
  • Access control: Issue separate API keys with individual model, provider, and rate limits.
  • Built-in dashboard: Full admin UI embedded into the go binary.
  • Plugin system: Lua plugins allow you to add new provider integrations easily.

Architecture & Philosophy

  • Blazingly fast: Go + Svelte, no heavy JS runtime.
  • Modular: Core repo remains lightweight and maintainable. Easily extensible with Lua plugins.
  • Everything works: Zero dead code, all features are tested by real people.
  • Clean UI: Fully reviewed Svelte dashboard polished by a human.
  • UX with common sense: No unmoderated AI slop that nobody knows how to use.

Key Features

Admin Panel
  • Chat: Test models directly in the browser.
  • Metrics: Inspect model and provider usage.
  • Providers: Connect new providers and accounts.
  • Models: Browse available models across active providers.
  • Plugins: Install plugins from the online store.
  • Virtual models: Create a composite model out of many.
  • Tokens: Manage access tokens with fine-grained model permissions.
Zero-Config

No config files. Minimal set of optional CLI flags and you are done.

llm-router --help

llm-router — minimalist LLM routing gateway

Usage:
  llm-router [host] [flags]

Examples:
  llm-router
llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Flags:
      --api Port             port for /v1 OpenAI-compatible API (default 8081)
      --db string            path to the database file (default "llm-router.db")
  -h, --help                 help for llm-router
      --log-level LogLevel   log level: debug, info, warn, error (default info)
      --no-auth              disable all authorization (dev and AI debugging only)
  -v, --version              print version information and exit
      --web Port             port for dashboard UI (default 8080)

Quick Start

Development server
make start
make start HOST=localhost WEB_PORT=8080 API_PORT=8081
Production builds
make build
make build PLATFORMS="windows/amd64 linux/amd64 darwin/amd64 freebsd/amd64"
Release tags
make publish TYPE=fix
make publish VERSION=1.3.1
Dashboard setup
  1. Create the initial admin account.
  2. Go to Providers and authenticate in any of listed providers, or add a new one.
  3. Go to Tokens and issue a new token. In wizard, select Allow all if you don't care about limits.
  4. The token will be shown to you once, afterwards you won't be able see it again.
API usage

Pass your token and the endpoint URL to your harness or any other AI-powered software:

  • Base URL: http://localhost:8081/v1
  • Token: 10b7****************************************************4fd7

License

MIT

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Package cmd defines the llm-router CLI using cobra.
Package cmd defines the llm-router CLI using cobra.
internal
adapters/generic
Package generic provides a generic OpenAI-compatible backend for custom providers.
Package generic provides a generic OpenAI-compatible backend for custom providers.
api/v1
Package v1 implements the OpenAI-compatible /v1/...
Package v1 implements the OpenAI-compatible /v1/...
config
Package config holds the runtime configuration populated from environment variables with optional .env file support.
Package config holds the runtime configuration populated from environment variables with optional .env file support.
dashboard
Package dashboard serves the admin SPA and its backing JSON API.
Package dashboard serves the admin SPA and its backing JSON API.
db
Package db manages the embedded database handles.
Package db manages the embedded database handles.
db/migrate
Package migrate owns the forward-only SQLite migration registry and runner.
Package migrate owns the forward-only SQLite migration registry and runner.
db/migrations
Package migrations owns numbered SQLite schema changes.
Package migrations owns numbered SQLite schema changes.
db/record
Package record owns GORM persistence structs.
Package record owns GORM persistence structs.
errors
Package errors defines standard error types used across llm-router services.
Package errors defines standard error types used across llm-router services.
httpapi
Package httpapi owns the HTTP route tables shared by both listeners.
Package httpapi owns the HTTP route tables shared by both listeners.
httpkit
Package httpkit holds shared HTTP transport helpers for the API surfaces.
Package httpkit holds shared HTTP transport helpers for the API surfaces.
logging
Package logging provides the bracket log handler for llm-router plus the events handler that persists records at or above the configured level to the events database while rendering them to stderr.
Package logging provides the bracket log handler for llm-router plus the events handler that persists records at or above the configured level to the events database while rendering them to stderr.
models
Package models compatibility additions: OpenAI prior-generation and Anthropic-compatible wire types plus the router-side async stores backing responses, conversations, assistants, threads, runs and message batches.
Package models compatibility additions: OpenAI prior-generation and Anthropic-compatible wire types plus the router-side async stores backing responses, conversations, assistants, threads, runs and message batches.
requestsource
Package requestsource identifies which listener served a request.
Package requestsource identifies which listener served a request.
server
Package server wires all services together and starts the HTTP servers.
Package server wires all services together and starts the HTTP servers.
services/batches
SQLiteService persists batches in SQLite: one header row in message_batches plus ordered child rows in message_batch_entries and message_batch_results.
SQLiteService persists batches in SQLite: one header row in message_batches plus ordered child rows in message_batch_entries and message_batch_results.
services/config
Package config manages RouterConfiguration persisted in the instance_settings row: the configuration columns plus the bootstrapped flag.
Package config manages RouterConfiguration persisted in the instance_settings row: the configuration columns plus the bootstrapped flag.
services/datamanagement
GORM-backed data management.
GORM-backed data management.
services/doctor
GORM-backed database inspection and repair.
GORM-backed database inspection and repair.
services/events
Package events owns operator telemetry: append-only event records with free-form metadata keys, retention pruning, and filtered queries for GET /api/events.
Package events owns operator telemetry: append-only event records with free-form metadata keys, retention pruning, and filtered queries for GET /api/events.
services/healthcheck
Package healthcheck owns failure-triggered credential health verification.
Package healthcheck owns failure-triggered credential health verification.
services/luaplugin/statepool
Package statepool holds idle Lua states in per-key buckets.
Package statepool holds idle Lua states in per-key buckets.
services/maintenance
GORM-backed maintenance scheduling.
GORM-backed maintenance scheduling.
services/metrics
Package metrics implements the Metrics Service for tracking API usage.
Package metrics implements the Metrics Service for tracking API usage.
services/pluginrepo
Package pluginrepo manages external plugin repositories (the plugin store).
Package pluginrepo manages external plugin repositories (the plugin store).
services/proxypool
Package proxypool adapts the proxypool library to router policy and storage.
Package proxypool adapts the proxypool library to router policy and storage.
services/router
Package router implements the Router Service.
Package router implements the Router Service.
services/token
Package token implements the Internal Token Service.
Package token implements the Internal Token Service.
services/videojobs
Package videojobs persists router-side video generation jobs.
Package videojobs persists router-side video generation jobs.
testutil
Package testutil provides shared test utilities for llm-router tests.
Package testutil provides shared test utilities for llm-router tests.
testutil/mockplugin
Package mockplugin owns deterministic Lua plugin fixtures for Go tests.
Package mockplugin owns deterministic Lua plugin fixtures for Go tests.
util
Package util provides shared utility functions for the llm-router framework.
Package util provides shared utility functions for the llm-router framework.
version
Package version owns the Version struct plus Parse/String helpers.
Package version owns the Version struct plus Parse/String helpers.
providers
virtual
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL