aibrix

module

v0.7.0 Latest Latest Go to latest Published: Jun 18, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/vllm-project/aibrix

Links

README ¶

AIBrix

Welcome to AIBrix, an open-source initiative designed to provide essential building blocks to construct scalable GenAI inference infrastructure. AIBrix delivers a cloud-native solution optimized for deploying, managing, and scaling large language model (LLM) inference, tailored specifically to enterprise needs.

Latest News

Releases

[2026-06-16] AIBrix v0.7.0 is released. Check out the release notes and Blog Post for more details.
[2026-03-05] AIBrix v0.6.0 is released. Check out the release notes and Blog Post for more details.
[2025-11-10] AIBrix v0.5.0 is released. Check out the release notes and Blog Post for more details.
[2025-08-05] AIBrix v0.4.0 is released. Check out the release notes and Blog Post for more details.
[2025-05-21] AIBrix v0.3.0 is released. Check out the release notes and Blog Post for more details.
[2025-03-09] AIBrix v0.2.1 is released. DeepSeek-R1 full weights deployment is supported and gateway stability has been improved! Check Blog Post for more details.
[2025-02-19] AIBrix v0.2.0 is released. Check out the release notes and Blog Post for more details.
[2024-11-13] AIBrix v0.1.0 is released. Check out the release notes and Blog Post for more details.

Talks and Presentations

[2025-11-12] AIBrix team co-delivered a keynote at KubeCon North America 2025 AIBrix: Kubernetes-native GenAI Inference Infrastructure, providing AIBrix overview.
[2025-06-10] AIBrix team delivered a talk at KubeCon China 2025 titled AIBrix: Cost-Effective and Scalable Kubernetes Control Plane for vLLM, discussing how the framework optimizes vLLM deployment via Kubernetes for cost efficiency and scalability.
[2025-04-04] AIBrix team co-delivered a keynote at KubeCon EU 2025 with Google on LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency, focusing on LLM specific routing solutions.
[2025-03-30] AIBrix was featured at the ASPLOS'25 workshop with the presentation AIBrix: An Open-Source, Large-Scale LLM Inference Infrastructure for System Research, showcasing its architecture for efficient LLM inference in system research scenarios.

Key Features

The initial release includes the following key features:

High-Density LoRA Management: Streamlined support for lightweight, low-rank adaptations of models.
LLM Gateway and Routing: Efficiently manage and direct traffic across multiple models and replicas.
LLM App-Tailored Autoscaler: Dynamically scale inference resources based on real-time demand.
Unified AI Runtime: A versatile sidecar enabling metric standardization, model downloading, and management.
Distributed Inference: Scalable architecture to handle large workloads across multiple nodes.
Distributed KV Cache: Enables high-capacity, cross-engine KV reuse.
Cost-efficient Heterogeneous Serving: Enables mixed GPU inference to reduce costs with SLO guarantees.
GPU Hardware Failure Detection: Proactive detection of GPU hardware issues.

Architecture

aibrix-architecture-v1

Quick Start

To get started with AIBrix, clone this repository and follow the setup instructions in the documentation. Our comprehensive guide will help you configure and deploy your first LLM infrastructure seamlessly.

# Local Testing
git clone https://github.com/vllm-project/aibrix.git
cd aibrix

# Install nightly aibrix dependencies
kubectl apply -k config/dependency --server-side

# Install nightly AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -k config/crd --server-side

# Install nightly aibrix components
kubectl apply -k config/default

Install stable distribution

# Install component dependencies
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-dependency-v0.7.0.yaml" --server-side

# Install AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-crds-v0.7.0.yaml" --server-side

# Install aibrix components
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-v0.7.0.yaml"

Documentation

For detailed documentation on installation, configuration, and usage, please visit our documentation page.

Contributing

We welcome contributions from the community! Check out our contributing guidelines to see how you can make a difference.

Slack Channel: #aibrix

License

AIBrix is licensed under the Apache 2.0 License.

Support

If you have any questions or encounter any issues, please submit an issue on our GitHub issues page.

Thank you for choosing AIBrix for your GenAI infrastructure needs!

Directories ¶

Path	Synopsis
api
autoscaling/v1alpha1 Package v1alpha1 contains API Schema definitions for the autoscaling v1alpha1 API group +kubebuilder:object:generate=true +groupName=autoscaling.aibrix.ai	Package v1alpha1 contains API Schema definitions for the autoscaling v1alpha1 API group +kubebuilder:object:generate=true +groupName=autoscaling.aibrix.ai
model/v1alpha1 Package v1alpha1 contains API Schema definitions for the model v1alpha1 API group +kubebuilder:object:generate=true +groupName=model.aibrix.ai	Package v1alpha1 contains API Schema definitions for the model v1alpha1 API group +kubebuilder:object:generate=true +groupName=model.aibrix.ai
orchestration/v1alpha1 Package v1alpha1 contains API Schema definitions for the orchestration v1alpha1 API group +kubebuilder:object:generate=true +groupName=orchestration.aibrix.ai	Package v1alpha1 contains API Schema definitions for the orchestration v1alpha1 API group +kubebuilder:object:generate=true +groupName=orchestration.aibrix.ai
apps
console/api/common
console/api/config
console/api/error_injection
console/api/gen/console/v1 Package consolev1 is a reverse proxy.	Package consolev1 is a reverse proxy.
console/api/handler JobHandler implements the Console BFF JobService:	JobHandler implements the Console BFF JobService:
console/api/middleware
console/api/planner/api Package plannerapi defines the Console-facing surface of the AIBrix planner: the Planner interface, the request/response shapes Enqueue / GetJob / ListJobs / GetQueueStats / GetProvisionResourceStats carry, and the typed errors Console handlers route on.	Package plannerapi defines the Console-facing surface of the AIBrix planner: the Planner interface, the request/response shapes Enqueue / GetJob / ListJobs / GetQueueStats / GetProvisionResourceStats carry, and the typed errors Console handlers route on.
console/api/planner/client Package client defines the planner -> Metadata Service adapter: the BatchClient interface, the request/response shapes its methods carry, and the AIBrixExtraBody fields that feed extra_body.aibrix.*.	Package client defines the planner -> Metadata Service adapter: the BatchClient interface, the request/response shapes its methods carry, and the AIBrixExtraBody fields that feed extra_body.aibrix.*.
console/api/planner/impl
console/api/planner/utils
console/api/resource_manager
console/api/resource_manager/catalog Package catalog provides instance type discovery, pricing information, and resource management.	Package catalog provides instance type discovery, pricing information, and resource management.
console/api/resource_manager/provider/kubernetes
console/api/resource_manager/provider/lambdacloud Package lambdacloud is a self-contained client for the Lambda Cloud public API (https://docs.lambda.ai/public-cloud/cloud-api/).	Package lambdacloud is a self-contained client for the Lambda Cloud public API (https://docs.lambda.ai/public-cloud/cloud-api/).
console/api/resource_manager/provider/runpod Package runpod is a self-contained client for the RunPod REST API (https://rest.runpod.io/v1).	Package runpod is a self-contained client for the RunPod REST API (https://rest.runpod.io/v1).
console/api/resource_manager/provisioner
console/api/resource_manager/types
console/api/server
console/api/store
console/api/store/models
console/api/utils
cmd
console command
controllers command
kvcache-watcher command
plugins command
pkg
cache
cache/discovery Package discovery provides service discovery backends for the gateway.	Package discovery provides service discovery backends for the gateway.
cache/kvcache
cert
client/applyconfiguration
client/applyconfiguration/autoscaling/v1alpha1
client/applyconfiguration/internal
client/applyconfiguration/model/v1alpha1
client/applyconfiguration/orchestration/v1alpha1
client/clientset/versioned
client/clientset/versioned/fake This package has the automatically generated fake clientset.	This package has the automatically generated fake clientset.
client/clientset/versioned/scheme This package contains the scheme of the automatically generated clientset.	This package contains the scheme of the automatically generated clientset.
client/clientset/versioned/typed/autoscaling/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/autoscaling/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/clientset/versioned/typed/model/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/model/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/clientset/versioned/typed/orchestration/v1alpha1 This package has the automatically generated typed clients.	This package has the automatically generated typed clients.
client/clientset/versioned/typed/orchestration/v1alpha1/fake Package fake has the automatically generated clients.	Package fake has the automatically generated clients.
client/informers/externalversions
client/informers/externalversions/autoscaling
client/informers/externalversions/autoscaling/v1alpha1
client/informers/externalversions/internalinterfaces
client/informers/externalversions/model
client/informers/externalversions/model/v1alpha1
client/informers/externalversions/orchestration
client/informers/externalversions/orchestration/v1alpha1
client/listers/autoscaling/v1alpha1
client/listers/model/v1alpha1
client/listers/orchestration/v1alpha1
config
constants
controller
controller/constants
controller/kvcache
controller/kvcache/backends
controller/modeladapter
controller/modeladapter/scheduling
controller/modelrouter
controller/podautoscaler Package podautoscaler provides controllers for managing PodAutoscaler resources.	Package podautoscaler provides controllers for managing PodAutoscaler resources.
controller/podautoscaler/aggregation
controller/podautoscaler/algorithm Package algorithm provides scaling algorithms for different autoscaling strategies.	Package algorithm provides scaling algorithms for different autoscaling strategies.
controller/podautoscaler/context
controller/podautoscaler/metrics
controller/podautoscaler/monitor
controller/podautoscaler/types
controller/podset
controller/rayclusterfleet
controller/rayclusterfleet/util
controller/rayclusterreplicaset
controller/roleset
controller/stormservice
controller/util
controller/util/expectation
controller/util/history
controller/util/orchestration
controller/util/patch
features
metrics
plugins/gateway
plugins/gateway/algorithms
plugins/gateway/algorithms/pd Package pd contains the scoring types used by the PD (prefill-decode) disaggregated-inference router.	Package pd contains the scoring types used by the PD (prefill-decode) disaggregated-inference router.
plugins/gateway/algorithms/pd/engine
plugins/gateway/algorithms/pd/prefill
plugins/gateway/algorithms/pd/selector Package selector defines the PodSelector contract for PD-disaggregated routing.	Package selector defines the PodSelector contract for PD-disaggregated routing.
plugins/gateway/algorithms/pd/transfer
plugins/gateway/algorithms/vtc Package vtc implements the Virtual Token Counter routing algorithms focused on fairness and utilization	Package vtc implements the Virtual Token Counter routing algorithms focused on fairness and utilization
plugins/gateway/configprofiles Package configprofiles parses the model.aibrix.ai/config annotation (or ConfigMap) and supports multiple named profiles selectable at runtime via config-profile header.	Package configprofiles parses the model.aibrix.ai/config annotation (or ConfigMap) and supports multiple named profiles selectable at runtime via config-profile header.
plugins/gateway/queue
plugins/gateway/ratelimiter
plugins/gateway/statesync Package statesync provides a generic Redis-backed state sync layer for use across multiple gateway (or other) replicas.	Package statesync provides a generic Redis-backed state sync layer for use across multiple gateway (or other) replicas.
types
utils
utils/hash
utils/lrustore
utils/prefixcacheindexer
utils/syncable Package syncable defines the interface for state that can be synced across replicas (e.g.	Package syncable defines the interface for state that can be synced across replicas (e.g.
utils/syncprefixcacheindexer
utils/tokenizer
webhook
test
e2e
utils
utils/validation
utils/wrapper

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL