aigw

Envoy AI Gateway CLI (aigw)

Quick Start

docker-compose.yml builds and runs aigw, targeting Ollama for OpenAI chat completion requests on port 1975.

• aigw (port 1975): Envoy AI Gateway CLI (standalone mode)
• chat-completion: curl command making a simple chat completion

The simplest way to get started is to have aigw generate a configuration for your OpenAI-compatible backend. This happens when there is no configuration file and at least the OPENAI_API_KEY environment variable is set.

Here are values we use for Ollama:

• OPENAI_API_KEY=unused (Ollama does not require an API key)
• OPENAI_BASE_URL=http://localhost:11434/v1 (host.docker.internal in Docker)

1. Start Ollama on your host machine:

   Start Ollama on all interfaces, with a large context. This allows it to be addressable by Docker and handle large tasks, such as from Goose.

   OLLAMA_CONTEXT_LENGTH=131072 OLLAMA_HOST=0.0.0.0 ollama serve
   

2. Run the example minimal stack:

   up builds aigw from source and starts the stack, awaiting health checks.

   docker compose up --wait -d
   

3. Make requests to Envoy AI Gateway:

   The following services use curl to send requests to the AI Gateway CLI (aigw) which routes them to Ollama:

   • Chat completion:

     docker compose run --rm chat-completion
     

   • Completion (legacy):

     docker compose run --rm completion
     

   • Embeddings:

     docker compose run --rm embeddings
     

   • MCP (Model Context Protocol) tool call:

     docker compose run --rm mcp
     

     This calls the kiwi MCP server through aigw's MCP Gateway at /mcp.

4. Shutdown the example stack:

   down stops the containers and removes the volumes used by the stack.

   docker compose down --remove-orphans
   

Quick Start with OpenTelemetry

docker-compose-otel.yaml includes OpenTelemetry metrics and tracing.

All profiles below use at least these Docker services:

• aigw (port 1975): Envoy AI Gateway CLI with OpenAI endpoints at /v1/* and MCP endpoint at /mcp
• chat-completion: OpenAI Python client for chat completions, instrumented with OpenTelemetry
• completion: OpenAI Python client for completions (legacy), instrumented with OpenTelemetry
• create-embeddings: OpenAI Python client for embeddings, instrumented with OpenTelemetry

Prerequisites

1. Start Ollama on your host machine:

   Start Ollama on all interfaces, with a large context. This allows it to be addressable by Docker and handle large tasks, such as from Goose.

   OLLAMA_CONTEXT_LENGTH=131072 OLLAMA_HOST=0.0.0.0 ollama serve
   

Configure OpenTelemetry Export

Choose how you want to export telemetry data (traces and metrics). We provide pre-configured .env files for common scenarios:

Run the Stack

1. Start the services:

   COMPOSE_PROFILES="<profile>" docker compose -f docker-compose-otel.yaml up --build --wait -d
   

   Where <profile> is:

   • console - Export to console for debugging (default if omitted)
   • otel-tui - Export to otel-tui Terminal UI (also starts otel-tui service)
   • phoenix - Export to Phoenix (also starts Phoenix service)

2. Send test requests:

   COMPOSE_PROFILES="<profile>" docker compose -f docker-compose-otel.yaml run --build --rm chat-completion
   COMPOSE_PROFILES="<profile>" docker compose -f docker-compose-otel.yaml run --build --rm create-embeddings
   COMPOSE_PROFILES="<profile>" docker compose -f docker-compose-otel.yaml run --build --rm completion
   COMPOSE_PROFILES="<profile>" docker compose -f docker-compose-otel.yaml run --build --rm mcp
   

3. Check telemetry output:

   For Console export

   # View traces and metrics in aigw logs
   docker compose -f docker-compose-otel.yaml logs aigw | grep -E "(SpanContext|gen_ai)"
   

   For Phoenix export

   If you configured Phoenix as your OTLP endpoint, you can view detailed traces showing the OpenAI CLI (Python) joining a trace with the Envoy AI Gateway CLI (aigw), including LLM inputs and outputs served by Ollama:

   

   # Verify Phoenix is receiving traces
   docker compose -f docker-compose-otel.yaml logs phoenix | grep "POST /v1/traces"
   
   # Open Phoenix UI
   open http://localhost:6006
   

   For otel-tui export

   # Show TUI in your current terminal session
   docker compose -f docker-compose-otel.yaml attach otel-tui
   
   # Detach by pressing Ctrl+p -> Ctrl+q
   

   Access logs with GenAI fields (always available):

   docker compose -f docker-compose-otel.yaml logs aigw | grep "genai_model_name"
   

Shutdown

Stop the services:

docker compose -f docker-compose-otel.yaml down --remove-orphans