README
¶
Envoy AI Gateway
Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to Generative AI services.
Usage
When using Envoy AI Gateway, we refer to a two-tier gateway pattern. The Tier One Gateway functions as a centralized entry point, and the Tier Two Gateway handles ingress traffic to a self-hosted model serving cluster.
- The Tier One Gateway handles authentication, top-level routing, and global rate limiting
- The Tier Two Gateway provides fine-grained control over self-hosted model access, with endpoint picker support for LLM inference optimization.
Supported AI Providers
Envoy AI Gateway supports a wide range of AI providers, making it easy to integrate with your preferred LLM services:
OpenAI |
Azure OpenAI |
Google Gemini |
Vertex AI |
AWS Bedrock |
Mistral |
Cohere |
Groq |
Together AI |
DeepInfra |
DeepSeek |
Hunyuan |
SambaNova |
Grok |
Documentation
- Blog introducing Envoy AI Gateway.
- Documentation for Envoy AI Gateway.
- Quickstart to use Envoy AI Gateway in a few simple steps.
- Concepts to understand the architecture and resources of Envoy AI Gateway.
Contact
- Slack: Join the Envoy Slack workspace if you're not already a member. Otherwise, use the Envoy AI Gateway channel to start collaborating with the community.
Get Involved
We adhere to the CNCF Code of conduct
The Envoy AI Gateway team and community members meet every Thursday. Please register for the meeting, add agenda points, and get involved. The meeting details are available in the public document.
To contribute to the project via pull requests, please read the CONTRIBUTING.md file which includes information on how to build and test the project.
Background
The proposal of using Envoy Gateway as a Cloud Native LLM Gateway inspired the initiation of this project.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
v1alpha1
Package v1alpha1 contains API schema definitions for the aigateway.envoyproxy.io API group.
|
Package v1alpha1 contains API schema definitions for the aigateway.envoyproxy.io API group. |
|
cmd
|
|
|
aigw
command
|
|
|
controller
command
|
|
|
extproc
command
|
|
|
Package filterapi provides the configuration for the AI Gateway-implemented filter which is currently an external processor (See https://github.com/envoyproxy/ai-gateway/issues/90).
|
Package filterapi provides the configuration for the AI Gateway-implemented filter which is currently an external processor (See https://github.com/envoyproxy/ai-gateway/issues/90). |
|
internal
|
|
|
apischema/anthropic
Package anthropic contains Anthropic API schema definitions using the official SDK types.
|
Package anthropic contains Anthropic API schema definitions using the official SDK types. |
|
apischema/openai
Package openai contains the following is the OpenAI API schema definitions.
|
Package openai contains the following is the OpenAI API schema definitions. |
|
controller/rotators
Package rotators provides credential rotation implementations.
|
Package rotators provides credential rotation implementations. |
|
internalapi
Package internalapi provides constants and functions used across the boundary among controller, extension server and extproc.
|
Package internalapi provides constants and functions used across the boundary among controller, extension server and extproc. |
|
llmcostcel
Package llmcostcel provides functions to create and evaluate CEL programs to calculate costs.
|
Package llmcostcel provides functions to create and evaluate CEL programs to calculate costs. |
|
testing/testotel
Package testotel provides test utilities for OpenTelemetry tracing tests.
|
Package testotel provides test utilities for OpenTelemetry tracing tests. |
|
tracing/api
Package api provides types for OpenTelemetry tracing support, notably to reduce chance of cyclic imports.
|
Package api provides types for OpenTelemetry tracing support, notably to reduce chance of cyclic imports. |
|
tracing/openinference
Package openinference provides OpenInference semantic conventions for OpenTelemetry tracing.
|
Package openinference provides OpenInference semantic conventions for OpenTelemetry tracing. |
|
tracing/openinference/openai
Package openai provides OpenInference semantic conventions hooks for OpenAI instrumentation used by the ExtProc router filter.
|
Package openai provides OpenInference semantic conventions hooks for OpenAI instrumentation used by the ExtProc router filter. |
|
tests
|
|
|
internal/testopenai
Package testopenai provides a test OpenAI API server for testing.
|
Package testopenai provides a test OpenAI API server for testing. |
|
internal/testopeninference
Package testopeninference provides OpenInference span recording and caching for testing AI Gateway's OpenTelemetry tracing implementation.
|
Package testopeninference provides OpenInference span recording and caching for testing AI Gateway's OpenTelemetry tracing implementation. |
|
tools
module
|
Click to show internal directories.
Click to hide internal directories.