rtp-agent

RTP Agent

Golang-first effort to build a practical alternative to the LiveKit Agent SDK, focused on lower end-to-end latency and reduced hardware requirements.

Why This Project

rtp-agent explores a lean, production-friendly agent runtime in Go for real-time voice and multimodal workflows.

Core goals:

• Reduce latency across STT -> LLM -> TTS turn loops.
• Minimize CPU and memory requirements so agents can run on smaller instances.
• Keep architecture modular through ports and adapters, so providers can be swapped quickly.

Table of Contents

• Features
• Built With
• Getting Started
• Usage
• Roadmap
• Contributing
• License
• Acknowledgments

Features

• Go-based worker runtime and CLI entrypoint.
• Hexagonal architecture (adapter, core, interface, library) for clear separation of concerns.
• Architecture details: ARCHITECTURE.md.
• Broad provider coverage with 60+ adapter packages (LLM, STT, TTS, avatar, utilities).
• LiveKit-compatible worker transport and job lifecycle handling.
• Optional Agora RTC worker transport for joining Agora channels directly as an agent.

Built With

• Go
• LiveKit Server SDK for Go
• Pion WebRTC

Getting Started

Prerequisites

• Go 1.25+ (as defined in go.mod)
• Access to a LiveKit deployment (URL, API key, API secret)
• For Agora RTC transport: an Agora-Golang-Server-SDK checkout with native SDK assets and Agora project credentials.

Installation

git clone https://github.com/cavos-io/rtp-agent.git
cd rtp-agent
go mod tidy

Configure Worker Options

Current bootstrap is code-first. Update worker options in cmd/main.go before running:

opts := worker.WorkerOptions{
  AgentName:  "example-agent",
  WorkerType: worker.WorkerTypeRoom,
  WSRL:       "wss://<your-livekit-host>",
  APIKey:     "<your-api-key>",
  APISecret:  "<your-api-secret>",
}

Usage

Run the worker:

go run ./cmd/main.go start

Build with in-process FFmpeg recording to write browser-playable MP4/AAC instead of Ogg/Opus:

go build -tags ffmpeg ./examples/starter

The ffmpeg build tag requires CGO plus the libavcodec, libavformat, libavutil, and libswresample development libraries. The repository Nix shell provides them. It links the native libraries directly and does not execute an external ffmpeg command. LiveKit session-report upload is disabled in this build because that upload contract requires Ogg audio; local session reports remain available.

Development mode with autoreload:

go run ./cmd/main.go dev

Connect mode for local room testing:

go run ./cmd/main.go connect <room_name> [participant_identity]

Agora RTC Worker Transport

The Agora transport is optional and built behind the agora_sdk tag because it depends on Agora native runtime libraries.

Build an Agora-enabled binary:

AGORA_GO_SDK_DIR=/path/to/Agora-Golang-Server-SDK \
  OUT=.tmp/rtp-agent-agora \
  scripts/build-agora-sdk.sh

Run the worker in an Agora RTC channel:

export RTP_AGENT_TRANSPORT=agora
export AGORA_GO_SDK_DIR=/path/to/Agora-Golang-Server-SDK
export LD_LIBRARY_PATH="$AGORA_GO_SDK_DIR/agora_sdk:$LD_LIBRARY_PATH"
export AGORA_APP_ID=<agora-app-id>
export AGORA_APP_CERTIFICATE=<agora-app-certificate> # or set AGORA_TOKEN
export AGORA_CHANNEL=<channel-name>
export AGORA_UID=agent-0

.tmp/rtp-agent-agora start

Validate the join path with the smoke script:

AGORA_GO_SDK_DIR=/path/to/Agora-Golang-Server-SDK \
  AGORA_APP_ID=<agora-app-id> \
  AGORA_APP_CERTIFICATE=<agora-app-certificate> \
  AGORA_CHANNEL=<channel-name> \
  AGORA_UID=agent-0 \
  scripts/smoke-agora-rtc.sh

The smoke script succeeds only after the worker logs agora transport connected for a stable window and prints:

Agora RTC connected

Roadmap

• End-to-end latency profiling and optimization pass.
• Lower-memory execution mode for constrained environments.
• Compatibility layer and integrations for Pipecat.
• Compatibility layer and integrations for VisionAgent.
• Additional open-source integration target: Vocode.

Contributing

Contributions are welcome. Open an issue for bug reports or feature proposals, and submit a PR for improvements.

Repository: https://github.com/cavos-io/rtp-agent

License

This project is licensed under the MIT License.

See LICENSE for details.

Acknowledgments

• LiveKit for the reference agent model and ecosystem.
• Pion for core WebRTC building blocks in Go.
• The broader open-source realtime AI community.