Documentation
¶
Overview ¶
Example: Custom LLM Server (OpenAI-compatible)
This example demonstrates how to implement an OpenAI-compatible chat completions endpoint that Tavus can use as a custom LLM backend.
The server: - Exposes POST /chat/completions endpoint - Validates API key authentication - Uses omnillm-core for multi-provider LLM support (OpenAI, Anthropic, etc.) - Supports streaming responses
Environment variables: - PORT: Server port (default: 8000) - LLM_PROVIDER: Provider name (openai, anthropic, xai, ollama, etc.) - LLM_MODEL: Model to use (e.g., gpt-4o-mini, claude-3-5-haiku) - OPENAI_API_KEY: OpenAI API key (when using openai provider) - ANTHROPIC_API_KEY: Anthropic API key (when using anthropic provider) - XAI_API_KEY: X.AI API key (when using xai provider) - OLLAMA_BASE_URL: Ollama base URL (when using ollama provider)
Usage:
- Start this server: LLM_PROVIDER=openai OPENAI_API_KEY=... go run main.go
- Expose via ngrok: ngrok http 8000
- Configure your Tavus PAL to use the ngrok URL as the LLM base_url
Based on: github.com/Tavus-Engineering/tavus-examples/cvi-custom-llm-with-backend
Source Files