Documentation
¶
Overview ¶
Package avatar provides lip-sync avatar support for omni-livekit voice agents.
This package enables voice agents to display real-time lip-synced video avatars that move their mouths in sync with generated speech. It supports multiple avatar providers (Tavus, Anam, Simli, etc.) through a pluggable architecture.
Architecture ¶
The avatar system consists of three main components:
- Session - Manages the avatar lifecycle (start, wait for join, close)
- AudioDestination - Streams TTS audio to the avatar provider
- RPC Handlers - Handle playback control (start, finish, clear buffer)
The avatar provider joins the LiveKit room as a separate participant and publishes video+audio tracks on behalf of the agent using the "lk.publish_on_behalf" participant attribute.
Basic Usage ¶
// Create avatar session with a provider
session, err := tavus.New(tavus.Config{
APIKey: os.Getenv("TAVUS_API_KEY"),
FaceID: "your-face-id",
})
// Start the avatar
err = session.Start(ctx, avatar.StartOptions{
Room: agent.Room(),
AgentIdentity: "meeting-pm",
LiveKitURL: os.Getenv("LIVEKIT_URL"),
// ...
})
// Wait for avatar to join
err = session.WaitForJoin(ctx, 10*time.Second)
// Get audio destination for TTS output
audioOut := session.AudioOutput()
// Stream TTS audio to avatar
audioOut.CaptureFrame(ctx, pcmFrame)
audioOut.Flush(ctx)
// Handle interruption
audioOut.ClearBuffer(ctx)
Providers ¶
Avatar providers are implemented in sub-packages:
- avatar/tavus - Tavus avatar provider
- avatar/anam - Anam avatar provider
- avatar/simli - Simli avatar provider
Each provider implements the Session interface and handles provider-specific API calls to create avatar sessions.
Audio Streaming ¶
Audio is streamed from the agent to the avatar using LiveKit's ByteStream API. The DataStreamAudioOutput implementation handles:
- Streaming PCM audio frames to the avatar
- Managing stream lifecycle (create, write, close)
- RPC-based playback control
Playback Control ¶
The avatar communicates playback state via RPC:
- lk.playback_started - Avatar started speaking
- lk.playback_finished - Avatar finished speaking (with position and interrupted flag)
- lk.clear_buffer - Agent requests playback interruption
Index ¶
- Constants
- Variables
- func GenerateAvatarToken(opts TokenOptions) (string, error)
- type AudioConfig
- type AudioDestination
- type AvatarMetadata
- type BaseSession
- func (s *BaseSession) AudioOutput() AudioDestination
- func (s *BaseSession) AvatarIdentity() string
- func (s *BaseSession) Callbacks() *SessionCallbacks
- func (s *BaseSession) EmitError(err error)
- func (s *BaseSession) EmitMetrics(metrics Metrics)
- func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)
- func (s *BaseSession) EmitPlaybackStarted()
- func (s *BaseSession) IsStarted() bool
- func (s *BaseSession) MarkStarted()
- func (s *BaseSession) Provider() string
- func (s *BaseSession) Room() *lksdk.Room
- func (s *BaseSession) SetAudioOutput(out AudioDestination)
- func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)
- func (s *BaseSession) SetRoom(room *lksdk.Room)
- func (s *BaseSession) StartTime() time.Time
- type DataStreamAudioOutput
- func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
- func (o *DataStreamAudioOutput) Channels() int
- func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error
- func (o *DataStreamAudioOutput) Close() error
- func (o *DataStreamAudioOutput) Flush(ctx context.Context) error
- func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)
- func (o *DataStreamAudioOutput) SampleRate() int
- type DataStreamConfig
- type Metrics
- type PlaybackCallback
- type PlaybackEvent
- type PlaybackEventType
- type ProviderError
- type QueueAudioOutput
- func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
- func (q *QueueAudioOutput) Channels() int
- func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error
- func (q *QueueAudioOutput) Close() error
- func (q *QueueAudioOutput) Flush(ctx context.Context) error
- func (q *QueueAudioOutput) FrameCount() int
- func (q *QueueAudioOutput) Frames() [][]byte
- func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)
- func (q *QueueAudioOutput) Reset()
- func (q *QueueAudioOutput) SampleRate() int
- func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)
- func (q *QueueAudioOutput) SimulatePlaybackStarted()
- func (q *QueueAudioOutput) TotalBytes() int
- func (q *QueueAudioOutput) WasCleared() bool
- func (q *QueueAudioOutput) WasFlushed() bool
- type Session
- type SessionCallbacks
- type StartOptions
- type TokenOptions
Constants ¶
const ( // RPCPlaybackStarted is sent by the avatar when it starts speaking. RPCPlaybackStarted = "lk.playback_started" // RPCPlaybackFinished is sent by the avatar when it finishes speaking. RPCPlaybackFinished = "lk.playback_finished" // RPCClearBuffer is sent by the agent to interrupt playback. RPCClearBuffer = "lk.clear_buffer" )
RPC method names for playback control. These must match the methods implemented by avatar workers.
const AudioStreamTopic = "lk.audio_stream"
AudioStreamTopic is the ByteStream topic for audio data.
Variables ¶
var ( // ErrSessionNotStarted indicates that Start() was not called before // attempting an operation that requires an active session. ErrSessionNotStarted = errors.New("avatar: session not started") // ErrSessionAlreadyStarted indicates that Start() was called on a // session that is already running. ErrSessionAlreadyStarted = errors.New("avatar: session already started") // ErrAvatarJoinTimeout indicates that the avatar participant did not // join the room within the specified timeout. ErrAvatarJoinTimeout = errors.New("avatar: join timeout") // ErrAvatarTrackTimeout indicates that the avatar did not publish // the expected track (video/audio) within the timeout. ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout") // is unreachable or returned an error. ErrProviderUnavailable = errors.New("avatar: provider unavailable") // ErrProviderAuthFailed indicates that authentication with the // avatar provider failed (invalid API key, etc.). ErrProviderAuthFailed = errors.New("avatar: provider authentication failed") // ErrProviderRateLimited indicates that the avatar provider has // rate-limited the request. ErrProviderRateLimited = errors.New("avatar: provider rate limited") // ErrInvalidConfig indicates that the avatar configuration is // invalid or incomplete. ErrInvalidConfig = errors.New("avatar: invalid configuration") // ErrRPCTimeout indicates that an RPC call to the avatar timed out. ErrRPCTimeout = errors.New("avatar: RPC timeout") // ErrRPCFailed indicates that an RPC call to the avatar failed. ErrRPCFailed = errors.New("avatar: RPC failed") // ErrStreamClosed indicates that the audio stream was closed // unexpectedly. ErrStreamClosed = errors.New("avatar: stream closed") // ErrAvatarDisconnected indicates that the avatar participant // disconnected from the room unexpectedly. ErrAvatarDisconnected = errors.New("avatar: disconnected") )
Sentinel errors for avatar operations.
Functions ¶
func GenerateAvatarToken ¶
func GenerateAvatarToken(opts TokenOptions) (string, error)
GenerateAvatarToken creates a JWT token for an avatar to join a room.
The token includes the special "lk.publish_on_behalf" attribute that allows the avatar participant to publish tracks that appear in the UI as if they came from the agent participant.
Types ¶
type AudioConfig ¶
type AudioConfig struct {
// SampleRate is the audio sample rate in Hz.
// Default: 24000 (common for avatar providers)
SampleRate int
// Channels is the number of audio channels.
// Default: 1 (mono)
Channels int
// Encoding is the audio encoding format.
// Default: "linear16" (PCM16 little-endian)
Encoding string
}
AudioConfig holds audio configuration for avatar sessions.
func DefaultAudioConfig ¶
func DefaultAudioConfig() AudioConfig
DefaultAudioConfig returns the default audio configuration.
func (AudioConfig) FrameDuration ¶
func (c AudioConfig) FrameDuration(frameSize int) int
FrameDuration returns the duration in milliseconds for the given frame size.
func (AudioConfig) FrameSize ¶
func (c AudioConfig) FrameSize(durationMs int) int
FrameSize returns the number of bytes per frame for the given duration in ms.
type AudioDestination ¶
type AudioDestination interface {
// CaptureFrame sends a PCM16 audio frame to the avatar.
//
// The frame should be little-endian PCM16 at the configured sample rate.
// Frames are typically 20ms of audio (e.g., 960 samples at 48kHz).
//
// This method may block if the underlying transport is congested.
CaptureFrame(ctx context.Context, frame []byte) error
// Flush marks the end of an audio segment (utterance).
//
// The avatar should speak all buffered audio before accepting more.
// This closes the current ByteStream and prepares for the next utterance.
Flush(ctx context.Context) error
// ClearBuffer interrupts current playback.
//
// Use this when the user interrupts the agent. The avatar should
// stop speaking immediately and discard any buffered audio.
//
// This sends an RPC to the avatar worker and waits for acknowledgment.
ClearBuffer(ctx context.Context) error
// SampleRate returns the expected input sample rate in Hz.
// Common values: 16000, 24000, 48000
SampleRate() int
// Channels returns the expected number of audio channels.
// Typically 1 (mono).
Channels() int
// OnPlayback registers a callback for playback events.
// Multiple callbacks can be registered.
OnPlayback(callback PlaybackCallback)
// Close releases resources associated with this audio destination.
// After Close(), no more frames should be sent.
Close() error
}
AudioDestination receives audio frames and forwards them to an avatar.
Implementations include:
- DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
- QueueAudioOutput: In-memory queue for testing
The typical usage flow is:
- Call CaptureFrame() for each PCM audio frame from TTS
- Call Flush() when the utterance is complete
- Call ClearBuffer() to interrupt playback (e.g., user interruption)
type AvatarMetadata ¶
type AvatarMetadata struct {
// Kind identifies this as an avatar participant.
Kind string `json:"kind"`
// Provider is the avatar provider name.
Provider string `json:"provider,omitempty"`
// AgentIdentity is the identity of the agent this avatar represents.
AgentIdentity string `json:"agent_identity,omitempty"`
}
AvatarMetadata is the standard metadata structure for avatar participants.
func DefaultAvatarMetadata ¶
func DefaultAvatarMetadata(provider, agentIdentity string) AvatarMetadata
DefaultAvatarMetadata returns the default metadata for avatar participants.
type BaseSession ¶
type BaseSession struct {
// contains filtered or unexported fields
}
BaseSession provides common functionality for avatar session implementations. Provider-specific sessions can embed this struct.
func NewBaseSession ¶
func NewBaseSession(provider, avatarIdentity string) *BaseSession
NewBaseSession creates a new BaseSession.
func (*BaseSession) AudioOutput ¶
func (s *BaseSession) AudioOutput() AudioDestination
AudioOutput returns the audio destination.
func (*BaseSession) AvatarIdentity ¶
func (s *BaseSession) AvatarIdentity() string
AvatarIdentity returns the avatar participant identity.
func (*BaseSession) Callbacks ¶
func (s *BaseSession) Callbacks() *SessionCallbacks
Callbacks returns the session callbacks.
func (*BaseSession) EmitError ¶
func (s *BaseSession) EmitError(err error)
EmitError emits an error event if a callback is registered.
func (*BaseSession) EmitMetrics ¶
func (s *BaseSession) EmitMetrics(metrics Metrics)
EmitMetrics emits metrics if a callback is registered.
func (*BaseSession) EmitPlaybackFinished ¶
func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)
EmitPlaybackFinished emits a playback finished event if a callback is registered.
func (*BaseSession) EmitPlaybackStarted ¶
func (s *BaseSession) EmitPlaybackStarted()
EmitPlaybackStarted emits a playback started event if a callback is registered.
func (*BaseSession) IsStarted ¶
func (s *BaseSession) IsStarted() bool
IsStarted returns true if the session has been started.
func (*BaseSession) MarkStarted ¶
func (s *BaseSession) MarkStarted()
MarkStarted marks the session as started.
func (*BaseSession) Provider ¶
func (s *BaseSession) Provider() string
Provider returns the provider name.
func (*BaseSession) Room ¶
func (s *BaseSession) Room() *lksdk.Room
Room returns the room reference.
func (*BaseSession) SetAudioOutput ¶
func (s *BaseSession) SetAudioOutput(out AudioDestination)
SetAudioOutput sets the audio destination.
func (*BaseSession) SetCallbacks ¶
func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)
SetCallbacks sets the session callbacks.
func (*BaseSession) SetRoom ¶
func (s *BaseSession) SetRoom(room *lksdk.Room)
SetRoom sets the room reference.
func (*BaseSession) StartTime ¶
func (s *BaseSession) StartTime() time.Time
StartTime returns when the session was started.
type DataStreamAudioOutput ¶
type DataStreamAudioOutput struct {
// contains filtered or unexported fields
}
DataStreamAudioOutput streams audio to a remote avatar via LiveKit ByteStream.
It handles:
- Creating ByteStream writers for each utterance
- Registering RPC handlers for playback events
- Sending clear buffer requests
func NewDataStreamAudioOutput ¶
func NewDataStreamAudioOutput(cfg DataStreamConfig) (*DataStreamAudioOutput, error)
NewDataStreamAudioOutput creates a new DataStreamAudioOutput.
func (*DataStreamAudioOutput) CaptureFrame ¶
func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
CaptureFrame sends a PCM16 audio frame to the avatar.
func (*DataStreamAudioOutput) Channels ¶
func (o *DataStreamAudioOutput) Channels() int
Channels returns the expected number of audio channels.
func (*DataStreamAudioOutput) ClearBuffer ¶
func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error
ClearBuffer interrupts current playback.
func (*DataStreamAudioOutput) Close ¶
func (o *DataStreamAudioOutput) Close() error
Close releases resources.
func (*DataStreamAudioOutput) Flush ¶
func (o *DataStreamAudioOutput) Flush(ctx context.Context) error
Flush marks the end of an audio segment.
func (*DataStreamAudioOutput) OnPlayback ¶
func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)
OnPlayback registers a callback for playback events.
func (*DataStreamAudioOutput) SampleRate ¶
func (o *DataStreamAudioOutput) SampleRate() int
SampleRate returns the expected input sample rate.
type DataStreamConfig ¶
type DataStreamConfig struct {
// Room is the LiveKit room.
// Required.
Room *lksdk.Room
// DestinationIdentity is the avatar participant identity.
// Required.
DestinationIdentity string
// Audio is the audio configuration.
// Optional, defaults to DefaultAudioConfig().
Audio AudioConfig
// RPCTimeout is the timeout for RPC calls.
// Default: 5 seconds
RPCTimeout time.Duration
}
DataStreamConfig configures the DataStreamAudioOutput.
type Metrics ¶
type Metrics struct {
// AvatarJoinLatency is the time from Start() to avatar joining the room.
AvatarJoinLatency time.Duration
// PlaybackLatency is the time from audio send to avatar speech start.
// This is measured per utterance.
PlaybackLatency time.Duration
// Provider is the avatar provider name.
Provider string
// Timestamp is when the metrics were collected.
Timestamp time.Time
}
Metrics contains avatar performance metrics.
type PlaybackCallback ¶
type PlaybackCallback func(event PlaybackEvent)
PlaybackCallback is called when playback events occur.
type PlaybackEvent ¶
type PlaybackEvent struct {
// Type is the type of playback event.
Type PlaybackEventType
// Position is the playback position in seconds when the event occurred.
// Only meaningful for PlaybackFinished events.
Position float64
// Interrupted indicates if playback was interrupted by ClearBuffer().
// Only meaningful for PlaybackFinished events.
Interrupted bool
}
PlaybackEvent represents a playback state change from the avatar.
type PlaybackEventType ¶
type PlaybackEventType string
PlaybackEventType identifies the type of playback event.
const ( // PlaybackStarted indicates the avatar started speaking. PlaybackStarted PlaybackEventType = "started" // PlaybackFinished indicates the avatar finished speaking. // Check Position and Interrupted for details. PlaybackFinished PlaybackEventType = "finished" )
type ProviderError ¶
type ProviderError struct {
Provider string // Provider name (e.g., "tavus", "anam")
Op string // Operation that failed
Err error // Underlying error
}
ProviderError wraps an error from an avatar provider with additional context.
func NewProviderError ¶
func NewProviderError(provider, op string, err error) *ProviderError
NewProviderError creates a new ProviderError.
func (*ProviderError) Error ¶
func (e *ProviderError) Error() string
Error implements the error interface.
func (*ProviderError) Unwrap ¶
func (e *ProviderError) Unwrap() error
Unwrap returns the underlying error.
type QueueAudioOutput ¶
type QueueAudioOutput struct {
// contains filtered or unexported fields
}
QueueAudioOutput is an in-memory AudioDestination for testing.
It captures all audio frames sent to it, allowing tests to verify that the correct audio data is being produced.
func NewQueueAudioOutput ¶
func NewQueueAudioOutput(config AudioConfig) *QueueAudioOutput
NewQueueAudioOutput creates a new QueueAudioOutput with the given config.
func (*QueueAudioOutput) CaptureFrame ¶
func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
CaptureFrame stores a PCM16 audio frame in the queue.
func (*QueueAudioOutput) Channels ¶
func (q *QueueAudioOutput) Channels() int
Channels returns the configured number of channels.
func (*QueueAudioOutput) ClearBuffer ¶
func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error
ClearBuffer simulates interrupting playback.
func (*QueueAudioOutput) Close ¶
func (q *QueueAudioOutput) Close() error
Close marks the queue as closed.
func (*QueueAudioOutput) Flush ¶
func (q *QueueAudioOutput) Flush(ctx context.Context) error
Flush marks the end of an audio segment.
func (*QueueAudioOutput) FrameCount ¶
func (q *QueueAudioOutput) FrameCount() int
FrameCount returns the number of frames captured.
func (*QueueAudioOutput) Frames ¶
func (q *QueueAudioOutput) Frames() [][]byte
Frames returns all captured frames.
func (*QueueAudioOutput) OnPlayback ¶
func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)
OnPlayback registers a callback for playback events.
func (*QueueAudioOutput) Reset ¶
func (q *QueueAudioOutput) Reset()
Reset clears all captured data and resets state.
func (*QueueAudioOutput) SampleRate ¶
func (q *QueueAudioOutput) SampleRate() int
SampleRate returns the configured sample rate.
func (*QueueAudioOutput) SimulatePlaybackFinished ¶
func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)
SimulatePlaybackFinished simulates a playback finished event.
func (*QueueAudioOutput) SimulatePlaybackStarted ¶
func (q *QueueAudioOutput) SimulatePlaybackStarted()
SimulatePlaybackStarted simulates a playback started event.
func (*QueueAudioOutput) TotalBytes ¶
func (q *QueueAudioOutput) TotalBytes() int
TotalBytes returns the total number of bytes captured.
func (*QueueAudioOutput) WasCleared ¶
func (q *QueueAudioOutput) WasCleared() bool
WasCleared returns true if ClearBuffer() was called.
func (*QueueAudioOutput) WasFlushed ¶
func (q *QueueAudioOutput) WasFlushed() bool
WasFlushed returns true if Flush() was called.
type Session ¶
type Session interface {
// AvatarIdentity returns the participant identity of the avatar worker.
// This is the identity that appears in the room and publishes video.
AvatarIdentity() string
// Provider returns the provider name (e.g., "tavus", "anam", "simli").
Provider() string
// Start initializes the avatar session.
//
// This method should:
// 1. Create a session with the avatar provider's API
// 2. Generate a LiveKit token for the avatar to join the room
// 3. Configure the audio output to stream to the avatar
//
// The avatar will join the room asynchronously. Use WaitForJoin()
// to wait for the avatar to be ready.
Start(ctx context.Context, opts StartOptions) error
// WaitForJoin blocks until the avatar participant joins the room
// and publishes the expected tracks (typically video).
//
// Returns an error if the timeout is exceeded or the context is cancelled.
WaitForJoin(ctx context.Context, timeout time.Duration) error
// AudioOutput returns the audio destination for streaming TTS audio
// to the avatar. Returns nil if the session is not started.
AudioOutput() AudioDestination
// Close disconnects the avatar and cleans up resources.
//
// This method should:
// 1. End the session with the avatar provider
// 2. Remove the avatar participant from the room
// 3. Clean up any registered RPC handlers
Close(ctx context.Context) error
}
Session manages a lip-sync avatar that publishes video to the room.
Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.
The typical flow is:
- Create a Session with provider-specific configuration
- Call Start() to initialize the avatar and connect it to the room
- Call WaitForJoin() to wait for the avatar to be ready
- Use AudioOutput() to stream TTS audio to the avatar
- Call Close() when done to clean up resources
type SessionCallbacks ¶
type SessionCallbacks struct {
// OnMetricsCollected is called when avatar metrics are available.
OnMetricsCollected func(metrics Metrics)
// OnPlaybackStarted is called when the avatar starts speaking.
OnPlaybackStarted func()
// OnPlaybackFinished is called when the avatar finishes speaking.
// The position is the playback position in seconds when stopped.
// The interrupted flag indicates if playback was interrupted by
// a ClearBuffer() call.
OnPlaybackFinished func(position float64, interrupted bool)
// OnAvatarJoined is called when the avatar participant joins the room.
OnAvatarJoined func(identity string)
// OnAvatarLeft is called when the avatar participant leaves the room.
OnAvatarLeft func(identity string)
// OnError is called when an error occurs.
// This is for non-fatal errors that don't cause the session to fail.
OnError func(err error)
}
SessionCallbacks defines optional event callbacks for avatar sessions.
type StartOptions ¶
type StartOptions struct {
// Room is the LiveKit room the agent has joined.
// Required.
Room *lksdk.Room
// AgentIdentity is the identity of the agent participant.
// The avatar will publish tracks on behalf of this identity
// using the lk.publish_on_behalf attribute.
// Required.
AgentIdentity string
// LiveKitURL is the LiveKit server URL for the avatar to connect to.
// This should match the URL the agent connected to.
// Required.
LiveKitURL string
// LiveKitAPIKey is used to generate tokens for the avatar.
// Required.
LiveKitAPIKey string
// LiveKitAPISecret is used to generate tokens for the avatar.
// Required.
LiveKitAPISecret string
// Callbacks configures optional event callbacks.
Callbacks *SessionCallbacks
}
StartOptions configures avatar session startup.
func (*StartOptions) Validate ¶
func (o *StartOptions) Validate() error
Validate checks that all required fields are set.
type TokenOptions ¶
type TokenOptions struct {
// APIKey is the LiveKit API key.
// Required.
APIKey string
// APISecret is the LiveKit API secret.
// Required.
APISecret string
// RoomName is the room the avatar will join.
// Required.
RoomName string
// AvatarIdentity is the participant identity for the avatar.
// Required.
AvatarIdentity string
// AvatarName is the display name for the avatar participant.
// Optional, defaults to AvatarIdentity.
AvatarName string
// PublishOnBehalf is the identity of the agent participant.
// The avatar will publish tracks that appear as if they're from this participant.
// Required.
PublishOnBehalf string
// TTL is the token validity duration.
// Default: 5 minutes
TTL time.Duration
// Metadata is optional participant metadata.
Metadata string
}
TokenOptions configures avatar token generation.