llm-router

command module
v0.3.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 7, 2026 License: MIT Imports: 2 Imported by: 0

README

llm-router

Single-binary, bloat-free, extensible multi-provider OpenAI-compatible LLM gateway. Go on the backend, Svelte on the front, compiled into one file.

llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Zero external dependencies.


What It Does

Sits between your applications and LLM providers, exposing a single OpenAI-compatible /v1 API.

  • Load distribution: Split traffic across multiple providers and provider accounts.
  • Access control: Issue separate API keys with individual model, provider, and rate limits.
  • Built-in dashboard: Full admin UI embedded into the go binary.
  • Plugin system: Lua plugins allow you to add new provider integrations easily.

Architecture & Philosophy

  • Blazingly fast: Go + Svelte, no heavy JS runtime.
  • Modular: Core repo remains lightweight and maintainable. Easily extensible with Lua plugins.
  • Everything works: Zero dead code, all features are tested by real people.
  • Clean UI: Fully reviewed Svelte dashboard polished by a human.
  • UX with common sense: No unmoderated AI slop that nobody knows how to use.

Key Features

Admin Panel
  • Chat: Test models directly in the browser.
  • Metrics: Inspect model and provider usage.
  • Providers: Connect new providers and accounts.
  • Models: Browse available models across active providers.
  • Plugins: Install plugins from the online store.
  • Virtual models: Create a composite model out of many.
  • Tokens: Manage access tokens with fine-grained model permissions.
Zero-Config

No config files. Minimal set of optional CLI flags and you are done.

llm-router --help

llm-router — minimalist LLM routing gateway

Usage:
  llm-router [host] [flags]

Examples:
  llm-router
llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Flags:
      --api Port             port for /v1 OpenAI-compatible API (default 8081)
      --db string            path to the database file (default "llm-router.db")
  -h, --help                 help for llm-router
      --log-level LogLevel   log level: debug, info, warn, error (default info)
      --no-auth              disable all authorization (dev and AI debugging only)
  -v, --version              print version information and exit
      --web Port             port for dashboard UI (default 8080)

Quick Start

Development server
make start
make start HOST=localhost WEB_PORT=8080 API_PORT=8081
Production builds
make build
make build PLATFORMS="windows/amd64 linux/amd64 darwin/amd64 freebsd/amd64"
Release tags
make publish TYPE=fix
make publish VERSION=1.3.1
Dashboard setup
  1. Create the initial admin account.
  2. Go to Providers and authenticate in any of listed providers, or add a new one.
  3. Go to Tokens and issue a new token. In wizard, select Allow all if you don't care about limits.
  4. The token will be shown to you once, afterwards you won't be able see it again.
API usage

Pass your token and the endpoint URL to your harness or any other AI-powered software:

  • Base URL: http://localhost:8081/v1
  • Token: 10b7****************************************************4fd7

License

MIT

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Package cmd defines the llm-router CLI using cobra.
Package cmd defines the llm-router CLI using cobra.
internal
adapters/generic
Package generic provides a generic OpenAI-compatible backend for custom providers.
Package generic provides a generic OpenAI-compatible backend for custom providers.
api/v1
Package v1 implements the OpenAI-compatible /v1/...
Package v1 implements the OpenAI-compatible /v1/...
config
Package config holds the runtime configuration populated from CLI flags.
Package config holds the runtime configuration populated from CLI flags.
dashboard
Package dashboard serves the admin SPA and its backing JSON API.
Package dashboard serves the admin SPA and its backing JSON API.
db
Package db manages the embedded bbolt database and all bucket definitions.
Package db manages the embedded bbolt database and all bucket definitions.
errors
Package errors defines standard error types used across llm-router services.
Package errors defines standard error types used across llm-router services.
httpkit
Package httpkit holds shared HTTP transport helpers for the API surfaces.
Package httpkit holds shared HTTP transport helpers for the API surfaces.
logging
Package logging provides the bracket log handler for llm-router: [LEVEL][dd.mm.yyyy h:m:s.ms][key: value] message.
Package logging provides the bracket log handler for llm-router: [LEVEL][dd.mm.yyyy h:m:s.ms][key: value] message.
models
Package models compatibility additions: OpenAI prior-generation and Anthropic-compatible wire types plus the router-side async stores backing responses, conversations, assistants, threads, runs and message batches.
Package models compatibility additions: OpenAI prior-generation and Anthropic-compatible wire types plus the router-side async stores backing responses, conversations, assistants, threads, runs and message batches.
repository
Package repository provides generic type-safe database operations using Go generics.
Package repository provides generic type-safe database operations using Go generics.
server
Package server wires all services together and starts the HTTP servers.
Package server wires all services together and starts the HTTP servers.
services/admin
Package admin manages the bootstrap flow and dashboard authentication.
Package admin manages the bootstrap flow and dashboard authentication.
services/batches
Package batches persists router-side Anthropic message batches.
Package batches persists router-side Anthropic message batches.
services/config
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
services/credential
Package credential implements the Credential Pool Service.
Package credential implements the Credential Pool Service.
services/healthcheck
Package healthcheck owns failure-triggered credential health verification.
Package healthcheck owns failure-triggered credential health verification.
services/maintenance
Package maintenance runs plugin background jobs and provider upkeep.
Package maintenance runs plugin background jobs and provider upkeep.
services/metrics
Package metrics implements the Metrics Service for tracking API usage.
Package metrics implements the Metrics Service for tracking API usage.
services/modelinfo
Package modelinfo manages model metadata caching.
Package modelinfo manages model metadata caching.
services/pluginrepo
Package pluginrepo manages external plugin repositories (the plugin store).
Package pluginrepo manages external plugin repositories (the plugin store).
services/provider
Package provider owns the unified provider registry.
Package provider owns the unified provider registry.
services/proxypool
Package proxypool adapts the proxypool library to router policy and storage.
Package proxypool adapts the proxypool library to router policy and storage.
services/responses
Package responses persists router-side compatibility records.
Package responses persists router-side compatibility records.
services/router
Package router implements the Router Service.
Package router implements the Router Service.
services/token
Package token implements the Internal Token Service.
Package token implements the Internal Token Service.
services/videojobs
Package videojobs persists router-side video generation jobs.
Package videojobs persists router-side video generation jobs.
services/virtual
Package virtual implements the Virtual Models service.
Package virtual implements the Virtual Models service.
testutil
Package testutil provides shared test utilities for llm-router tests.
Package testutil provides shared test utilities for llm-router tests.
util
Package util provides shared utility functions for the llm-router framework.
Package util provides shared utility functions for the llm-router framework.
version
Package version owns the Version struct plus Parse/String helpers.
Package version owns the Version struct plus Parse/String helpers.
providers
virtual
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL