nexus

Nexus: AI gateway and LLM proxy library for Go

Nexus is an open-source AI gateway library for Go. You put it in front of OpenAI, Anthropic, Gemini, Bedrock, Ollama and 25 other LLM providers, and every request goes through one pipeline: provider routing, model aliases, response caching, PII and prompt-injection guardrails, multi-tenant gateway keys, rate limits and monthly budgets, and usage tracking with exact cost. It serves an OpenAI-compatible API with SSE, NDJSON and WebSocket streaming, and it runs inside your own process, either standalone or as a Forge extension.

Documentation · Quick start · Features · Providers · Pipeline · Forge · Examples · FAQ

Quick start

You need Go 1.26 or later and an OpenAI API key. Each provider is its own Go module, so you add the root and the providers you call:

go get github.com/xraph/nexus@latest
go get github.com/xraph/nexus/providers/openai@latest

Save this as main.go:

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"time"

	"github.com/xraph/nexus"
	"github.com/xraph/nexus/model"
	"github.com/xraph/nexus/providers/openai"
	"github.com/xraph/nexus/proxy"
)

func main() {
	ctx := context.Background()

	gw := nexus.New(
		nexus.WithProvider(openai.New(os.Getenv("OPENAI_API_KEY"))),
		nexus.WithAlias("fast", model.AliasTarget{Provider: "openai", Model: "gpt-4o-mini"}),
		// An open gateway, for trying it out locally. Remove this line
		// before anyone else can reach the port.
		nexus.WithRequireAPIKey(false),
	)
	if err := gw.Initialize(ctx); err != nil {
		log.Fatal(err)
	}
	defer gw.Shutdown(ctx)

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           proxy.New(gw.Engine()),
		ReadHeaderTimeout: 10 * time.Second,
	}
	log.Println("Nexus listening on http://localhost:8080/v1")
	if err := srv.ListenAndServe(); err != nil {
		log.Println(err)
	}
}

Run it, then call it the way you'd call OpenAI:

OPENAI_API_KEY=sk-... go run .

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "fast", "messages": [{"role": "user", "content": "Say hello"}]}'

fast is an alias, so the alias stage rewrites the model to gpt-4o-mini before the provider call. Any OpenAI SDK works too: set its base URL to http://localhost:8080/v1. Add "stream": true to the body and you get server-sent events in the chat.completion.chunk shape the SDKs expect.

The gateway keeps tenants, keys and usage in memory here, so they're gone when you stop it. Drop the WithRequireAPIKey(false) line and every /v1 request needs an nxs_ gateway key. The proxy example creates a tenant and a key at startup, and Persistence shows where they go in production.

Why Nexus?

You probably started by calling one provider's SDK straight from your handlers. That works until you need a second provider, a budget per customer or a record of who spent what, and then each of your services grows its own half-finished copy of the same routing, retry and metering code. Nexus is that layer as a Go library. You compose a nexus.Gateway from options and mount it on your own server, so you don't deploy anything new.

Providers

Each provider lives in providers/<name> as its own module, so you only download what you import. They all implement provider.Provider, and most of the OpenAI-compatible ones wrap providers/openai.

geminilive and openairealtime (realtime audio) are in the repo too, but they haven't been tagged yet, so you can't go get them at a released version.

The bedrock catalog is out of date. It still lists an old Claude id at its old price and wasn't updated in v1.9.0, so price the Bedrock models you call with nexus.WithPriceOverrides until it is.

Register as many as you like. A request goes to the provider it names in Provider, else to its alias target's provider, else to a provider whose catalog lists the model. Only a model nobody lists falls to the routing strategy, and without one Nexus uses the priority strategy, which goes by the order you registered providers in:

gw := nexus.New(
	nexus.WithProvider(openai.New(os.Getenv("OPENAI_API_KEY"))),
	nexus.WithProvider(anthropic.New(os.Getenv("ANTHROPIC_API_KEY"))),
	nexus.WithRouter(strategies.NewCostOptimized()),
)

Call it from Go

You don't need HTTP at all. gw.Engine() sends requests through the same pipeline, guardrails and cache included:

gw := nexus.New(
	nexus.WithProvider(openai.New(os.Getenv("OPENAI_API_KEY"))),
	nexus.WithCache(stores.NewMemory()),
	nexus.WithGuard(guards.NewPII(guard.ActionRedact)),
	nexus.WithGuard(guards.NewInjection()),
)
if err := gw.Initialize(ctx); err != nil {
	log.Fatal(err)
}
defer gw.Shutdown(ctx)

resp, err := gw.Engine().Complete(ctx, &provider.CompletionRequest{
	Model:     "gpt-4o-mini",
	Messages:  []provider.Message{{Role: "user", Content: "What is the capital of France?"}},
	MaxTokens: 100,
})
if err != nil {
	log.Fatal(err)
}
fmt.Println(resp.Choices[0].Message.Content, resp.Usage.TotalTokens)

Engine also has CompleteStream, Embed and ListModels. The streaming example reads a stream three ways: as an iterator, over a channel, and through an accumulator.

The request pipeline

Every request runs through the same chain of middleware, whether it came in over HTTP, over gRPC or through Engine. Stages are sorted by priority, lower first. A stage you didn't configure isn't in the chain at all, so you only pay for what you turn on:

WithMiddleware adds your own stage at whatever priority it reports. A stage that turns a request away returns *nexus.RefusalError, with an HTTP status, a code and, when a limit applies, Limit and RetryAfter.

Transforms live in transform: NewSystemPrompt, NewRAG for retrieval context, NewAnonymizer and NewAnonymizerOutput, and NewNormalizer. Put them in a transform.NewRegistry() and pass it with WithTransforms.

Tenants, keys and persistence

A tenant carries a quota (RPM, TPM, DailyRequests, MonthlyBudgetUSD, MaxTokensPerReq, and stream caps) and its own allowed or blocked models. Gateway keys belong to a tenant, look like nxs_ plus 64 hex characters, carry scopes, and are stored as hashes. You create them through gw.Tenants() and gw.Keys() in Go, or over the admin API once you have an admin key.

If you don't pass a store, everything lives in memory and is gone when the process exits. For anything you want to keep, pass a Grove-backed store:

Hand it to the gateway with nexus.WithDatabase(s) and call s.Migrate() once at startup. Money is money.USD, an exact decimal, so compare amounts with Equal and never with ==.

HTTP API

Two packages serve HTTP, and both are plain net/http handlers you can mount on any router:

The /admin routes always need a key with the admin scope, whatever RequireAPIKey says. Streaming responses pick their wire format from ?stream_format= (sse, nexus, ndjson) or the Accept header. grpcsrv is a separate module that serves completions over gRPC server-streaming, and the gRPC example shows it with key auth.

Use it with Forge

The extension package turns Nexus into a Forge extension. It reads config under extensions.nexus (or nexus), builds a store from a *grove.DB in the container when it finds one, migrates it on start, and registers *nexus.Gateway for the rest of your app.

import (
	"github.com/xraph/forge"
	nexusext "github.com/xraph/nexus/extension"
	"github.com/xraph/nexus/providers/openai"
)

app := forge.NewApp(forge.AppConfig{
	Name:        "ai-gateway",
	HTTPAddress: ":8080",
	Extensions: []forge.Extension{
		nexusext.New(
			nexusext.WithProvider(openai.New(os.Getenv("OPENAI_API_KEY"))),
		),
	},
})

The extension builds the gateway in Register and puts it in the container, so another extension can resolve it with vessel.Inject[*nexus.Gateway](app.Container()) from its own Register. Its engine answers nexus.ErrNotInitialized until Start has run. The extension doesn't register HTTP routes, so serve proxy.New(gw.Engine()) or api.New(gw).Handler() where you want the API to live. base_path and disable_routes are deprecated and ignored.

extensions:
  nexus:
    default_timeout: 30s
    default_max_retries: 2
    global_rate_limit: 1000   # requests per minute, 0 means unlimited
    require_api_key: true
    enable_usage: true
    enable_cache: false
    grove_database: ""        # a named grove.DB; empty uses the default one
    disable_migrate: false
    log_level: info

The admin API needs a first admin key, and no route can create one. Set NEXUS_BOOTSTRAP_ADMIN_KEY (or bootstrap_admin_key, though a secret in a committed YAML file stays there forever) to nxs_ followed by the output of openssl rand -hex 32. On start the extension stores it under a tenant with the slug operator, once, and never logs it.

WithDatabase doesn't stop Grove discovery. Register still looks for a *grove.DB (the named one from WithGroveDatabase, or the default one) and, when it finds it, builds a store from it after your options, so the container's store wins over the one you passed. If you want your own store, keep a *grove.DB out of the container.

The Forge extension guide covers how YAML and code options merge.

Documentation

The Go API is on pkg.go.dev, and each provider has its own page there under github.com/xraph/nexus/providers/<name>.

Upgrading? MIGRATION.md lists what changed in each release that can change how your code or your gateway behaves.

Examples

Most of them read OPENAI_API_KEY. The ones without a go.mod sit in the root module under _examples, which go build ./... skips, so run them by path: go run ./_examples/proxy.

FAQ

Can I use Nexus as an OpenAI-compatible proxy?

Yes. proxy.New(gw.Engine()) serves /v1/chat/completions, /v1/embeddings and /v1/models in OpenAI's request and response shapes. Point an OpenAI SDK's base URL at it and pass an nxs_ gateway key as the API key.

How do I set a per-customer budget or rate limit for LLM calls?

Create a tenant with a Quota: RPM, TPM, DailyRequests, MonthlyBudgetUSD and MaxTokensPerReq. Give the customer a key under that tenant. The quota stage refuses a request over any limit with a RefusalError that carries RetryAfter when waiting would help.

Does Nexus fall back to another provider when one fails?

The retry stage retries a failed call twice by default, with backoff (WithMaxRetries changes the count). Which provider answers is up to the routing strategy, and the priority strategy tries providers in registration order. A fallback package with a circuit breaker exists, but the default pipeline doesn't use it yet.

Can I use Nexus without Forge?

Yes. The root module, proxy, api and every provider work with plain net/http. The extension package is the only part that imports Forge.

Do I need an API key to run the tests?

No. Provider tests run against mock HTTP servers from testutil. Postgres and MongoDB store tests only run when you set NEXUS_TEST_POSTGRES_DSN or NEXUS_TEST_MONGO_URI.

Contributing

Issues and focused pull requests are welcome. Open an issue first for anything large.

git clone https://github.com/xraph/nexus.git
cd nexus
make test     # go test ./... in every module
make lint     # golangci-lint in every module
make fmt      # gofmt and goimports

The repo is many Go modules (the root, config, grpcsrv and one per provider), tied together by the committed go.work. Run a command with GOWORK=off when you want to see how one module resolves on its own. Skip make run, make dev and make build-bin, which point at a cmd/nexus that doesn't exist. AGENTS.md has the full command list, the lint rules and how releases are cut. Commits follow Conventional Commits with a scope, as in fix(sqlite): ....

License

MIT. See LICENSE.

Nexus is part of the forgery stack built on Forge, by Rex Raphael and contributors.