Documentation
¶
Overview ¶
Package avatar provides lip-sync avatar support for omni-livekit voice agents.
This package enables voice agents to display real-time lip-synced video avatars that move their mouths in sync with generated speech. It supports multiple avatar providers (Tavus, Anam, Simli, etc.) through a pluggable architecture.
Architecture ¶
The avatar system consists of three main components:
- Session - Manages the avatar lifecycle (start, wait for join, close)
- AudioDestination - Streams TTS audio to the avatar provider
- RPC Handlers - Handle playback control (start, finish, clear buffer)
The avatar provider joins the LiveKit room as a separate participant and publishes video+audio tracks on behalf of the agent using the "lk.publish_on_behalf" participant attribute.
Basic Usage ¶
// Create avatar session with a provider
session, err := tavus.New(tavus.Config{
APIKey: os.Getenv("TAVUS_API_KEY"),
FaceID: "your-face-id",
})
// Start the avatar
err = session.Start(ctx, avatar.StartOptions{
Room: agent.Room(),
AgentIdentity: "meeting-pm",
LiveKitURL: os.Getenv("LIVEKIT_URL"),
// ...
})
// Wait for avatar to join
err = session.WaitForJoin(ctx, 10*time.Second)
// Get audio destination for TTS output
audioOut := session.AudioOutput()
// Stream TTS audio to avatar
audioOut.CaptureFrame(ctx, pcmFrame)
audioOut.Flush(ctx)
// Handle interruption
audioOut.ClearBuffer(ctx)
Providers ¶
Avatar providers are implemented in sub-packages:
- avatar/tavus - Tavus avatar provider
- avatar/anam - Anam avatar provider
- avatar/simli - Simli avatar provider
Each provider implements the Session interface and handles provider-specific API calls to create avatar sessions.
Audio Streaming ¶
Audio is streamed from the agent to the avatar using LiveKit's ByteStream API. The DataStreamAudioOutput implementation handles:
- Streaming PCM audio frames to the avatar
- Managing stream lifecycle (create, write, close)
- RPC-based playback control
Playback Control ¶
The avatar communicates playback state via RPC:
- lk.playback_started - Avatar started speaking
- lk.playback_finished - Avatar finished speaking (with position and interrupted flag)
- lk.clear_buffer - Agent requests playback interruption
Index ¶
- Constants
- Variables
- func GenerateAvatarToken(opts TokenOptions) (string, error)
- func IsProviderRegistered(name string) bool
- func RegisterProvider(name string, constructor SessionConstructor)
- func RegisteredProviders() []string
- type AnamConfig
- type AudioConfig
- type AudioDestination
- type AvatarMetadata
- type BaseSession
- func (s *BaseSession) AudioOutput() AudioDestination
- func (s *BaseSession) AvatarIdentity() string
- func (s *BaseSession) Callbacks() *SessionCallbacks
- func (s *BaseSession) EmitError(err error)
- func (s *BaseSession) EmitMetrics(metrics Metrics)
- func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)
- func (s *BaseSession) EmitPlaybackStarted()
- func (s *BaseSession) IsStarted() bool
- func (s *BaseSession) MarkStarted()
- func (s *BaseSession) Provider() string
- func (s *BaseSession) Room() *lksdk.Room
- func (s *BaseSession) SetAudioOutput(out AudioDestination)
- func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)
- func (s *BaseSession) SetRoom(room *lksdk.Room)
- func (s *BaseSession) StartTime() time.Time
- type BitHumanConfig
- type Config
- type DataStreamAudioOutput
- func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
- func (o *DataStreamAudioOutput) Channels() int
- func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error
- func (o *DataStreamAudioOutput) Close() error
- func (o *DataStreamAudioOutput) Flush(ctx context.Context) error
- func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)
- func (o *DataStreamAudioOutput) SampleRate() int
- type DataStreamConfig
- type LiveAvatarHelper
- func (h *LiveAvatarHelper) AudioOutput() AudioDestination
- func (h *LiveAvatarHelper) Close(ctx context.Context) error
- func (h *LiveAvatarHelper) Session() Session
- func (h *LiveAvatarHelper) Start(ctx context.Context, room *lksdk.Room, agentIdentity string) error
- func (h *LiveAvatarHelper) WaitForJoin(ctx context.Context, timeout time.Duration) error
- type Metrics
- type PlaybackCallback
- type PlaybackEvent
- type PlaybackEventType
- type ProviderError
- type QueueAudioOutput
- func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
- func (q *QueueAudioOutput) Channels() int
- func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error
- func (q *QueueAudioOutput) Close() error
- func (q *QueueAudioOutput) Flush(ctx context.Context) error
- func (q *QueueAudioOutput) FrameCount() int
- func (q *QueueAudioOutput) Frames() [][]byte
- func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)
- func (q *QueueAudioOutput) Reset()
- func (q *QueueAudioOutput) SampleRate() int
- func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)
- func (q *QueueAudioOutput) SimulatePlaybackStarted()
- func (q *QueueAudioOutput) TotalBytes() int
- func (q *QueueAudioOutput) WasCleared() bool
- func (q *QueueAudioOutput) WasFlushed() bool
- type Session
- type SessionCallbacks
- type SessionConstructor
- type SetupConfig
- type SetupMode
- type SetupResult
- type SimliConfig
- type StartOptions
- type StaticImageConfig
- type StaticImageResult
- type TavusConfig
- type TokenOptions
Constants ¶
const ( // RPCPlaybackStarted is sent by the avatar when it starts speaking. RPCPlaybackStarted = "lk.playback_started" // RPCPlaybackFinished is sent by the avatar when it finishes speaking. RPCPlaybackFinished = "lk.playback_finished" // RPCClearBuffer is sent by the agent to interrupt playback. RPCClearBuffer = "lk.clear_buffer" )
RPC method names for playback control. These must match the methods implemented by avatar workers.
const ( ProviderNone = "" // No avatar (audio-only mode) ProviderTavus = "tavus" // Tavus conversational video ProviderBitHuman = "bithuman" // bitHuman real-time avatars ProviderAnam = "anam" // Anam avatar (future) ProviderSimli = "simli" // Simli avatar (future) ProviderStatic = "static" // Static image (not a Session, handled separately) )
Provider constants for avatar providers.
const AudioStreamTopic = "lk.audio_stream"
AudioStreamTopic is the ByteStream topic for audio data.
Variables ¶
var ( // ErrSessionNotStarted indicates that Start() was not called before // attempting an operation that requires an active session. ErrSessionNotStarted = errors.New("avatar: session not started") // ErrSessionAlreadyStarted indicates that Start() was called on a // session that is already running. ErrSessionAlreadyStarted = errors.New("avatar: session already started") // ErrAvatarJoinTimeout indicates that the avatar participant did not // join the room within the specified timeout. ErrAvatarJoinTimeout = errors.New("avatar: join timeout") // ErrAvatarTrackTimeout indicates that the avatar did not publish // the expected track (video/audio) within the timeout. ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout") // is unreachable or returned an error. ErrProviderUnavailable = errors.New("avatar: provider unavailable") // ErrProviderAuthFailed indicates that authentication with the // avatar provider failed (invalid API key, etc.). ErrProviderAuthFailed = errors.New("avatar: provider authentication failed") // ErrProviderRateLimited indicates that the avatar provider has // rate-limited the request. ErrProviderRateLimited = errors.New("avatar: provider rate limited") // ErrInvalidConfig indicates that the avatar configuration is // invalid or incomplete. ErrInvalidConfig = errors.New("avatar: invalid configuration") // ErrRPCTimeout indicates that an RPC call to the avatar timed out. ErrRPCTimeout = errors.New("avatar: RPC timeout") // ErrRPCFailed indicates that an RPC call to the avatar failed. ErrRPCFailed = errors.New("avatar: RPC failed") // ErrStreamClosed indicates that the audio stream was closed // unexpectedly. ErrStreamClosed = errors.New("avatar: stream closed") // ErrAvatarDisconnected indicates that the avatar participant // disconnected from the room unexpectedly. ErrAvatarDisconnected = errors.New("avatar: disconnected") )
Sentinel errors for avatar operations.
Functions ¶
func GenerateAvatarToken ¶
func GenerateAvatarToken(opts TokenOptions) (string, error)
GenerateAvatarToken creates a JWT token for an avatar to join a room.
The token includes the special "lk.publish_on_behalf" attribute that allows the avatar participant to publish tracks that appear in the UI as if they came from the agent participant.
func IsProviderRegistered ¶ added in v0.3.0
IsProviderRegistered returns true if the provider is registered.
func RegisterProvider ¶ added in v0.3.0
func RegisterProvider(name string, constructor SessionConstructor)
RegisterProvider registers a session constructor for a provider. This allows providers to be registered from their sub-packages without creating import cycles.
Example usage in avatar/tavus/init.go:
func init() {
avatar.RegisterProvider("tavus", func(cfg avatar.Config) (avatar.Session, error) {
return NewSession(SessionConfig{
APIKey: cfg.Tavus.APIKey,
BaseURL: cfg.Tavus.BaseURL,
PalID: cfg.Tavus.PalID,
FaceID: cfg.Tavus.FaceID,
AudioConfig: cfg.Tavus.AudioConfig,
})
})
}
func RegisteredProviders ¶ added in v0.3.0
func RegisteredProviders() []string
RegisteredProviders returns a list of registered provider names.
Types ¶
type AnamConfig ¶ added in v0.3.0
type AnamConfig struct {
// APIKey is the Anam API key.
APIKey string
// PersonaID is the Anam persona to use.
PersonaID string
// AudioConfig configures the audio format.
AudioConfig AudioConfig
}
AnamConfig contains configuration for the Anam avatar provider. Placeholder for future implementation.
type AudioConfig ¶
type AudioConfig struct {
// SampleRate is the audio sample rate in Hz.
// Default: 24000 (common for avatar providers)
SampleRate int
// Channels is the number of audio channels.
// Default: 1 (mono)
Channels int
// Encoding is the audio encoding format.
// Default: "linear16" (PCM16 little-endian)
Encoding string
}
AudioConfig holds audio configuration for avatar sessions.
func DefaultAudioConfig ¶
func DefaultAudioConfig() AudioConfig
DefaultAudioConfig returns the default audio configuration.
func (AudioConfig) FrameDuration ¶
func (c AudioConfig) FrameDuration(frameSize int) int
FrameDuration returns the duration in milliseconds for the given frame size.
func (AudioConfig) FrameSize ¶
func (c AudioConfig) FrameSize(durationMs int) int
FrameSize returns the number of bytes per frame for the given duration in ms.
type AudioDestination ¶
type AudioDestination interface {
// CaptureFrame sends a PCM16 audio frame to the avatar.
//
// The frame should be little-endian PCM16 at the configured sample rate.
// Frames are typically 20ms of audio (e.g., 960 samples at 48kHz).
//
// This method may block if the underlying transport is congested.
CaptureFrame(ctx context.Context, frame []byte) error
// Flush marks the end of an audio segment (utterance).
//
// The avatar should speak all buffered audio before accepting more.
// This closes the current ByteStream and prepares for the next utterance.
Flush(ctx context.Context) error
// ClearBuffer interrupts current playback.
//
// Use this when the user interrupts the agent. The avatar should
// stop speaking immediately and discard any buffered audio.
//
// This sends an RPC to the avatar worker and waits for acknowledgment.
ClearBuffer(ctx context.Context) error
// SampleRate returns the expected input sample rate in Hz.
// Common values: 16000, 24000, 48000
SampleRate() int
// Channels returns the expected number of audio channels.
// Typically 1 (mono).
Channels() int
// OnPlayback registers a callback for playback events.
// Multiple callbacks can be registered.
OnPlayback(callback PlaybackCallback)
// Close releases resources associated with this audio destination.
// After Close(), no more frames should be sent.
Close() error
}
AudioDestination receives audio frames and forwards them to an avatar.
Implementations include:
- DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
- QueueAudioOutput: In-memory queue for testing
The typical usage flow is:
- Call CaptureFrame() for each PCM audio frame from TTS
- Call Flush() when the utterance is complete
- Call ClearBuffer() to interrupt playback (e.g., user interruption)
type AvatarMetadata ¶
type AvatarMetadata struct {
// Kind identifies this as an avatar participant.
Kind string `json:"kind"`
// Provider is the avatar provider name.
Provider string `json:"provider,omitempty"`
// AgentIdentity is the identity of the agent this avatar represents.
AgentIdentity string `json:"agent_identity,omitempty"`
}
AvatarMetadata is the standard metadata structure for avatar participants.
func DefaultAvatarMetadata ¶
func DefaultAvatarMetadata(provider, agentIdentity string) AvatarMetadata
DefaultAvatarMetadata returns the default metadata for avatar participants.
type BaseSession ¶
type BaseSession struct {
// contains filtered or unexported fields
}
BaseSession provides common functionality for avatar session implementations. Provider-specific sessions can embed this struct.
func NewBaseSession ¶
func NewBaseSession(provider, avatarIdentity string) *BaseSession
NewBaseSession creates a new BaseSession.
func (*BaseSession) AudioOutput ¶
func (s *BaseSession) AudioOutput() AudioDestination
AudioOutput returns the audio destination.
func (*BaseSession) AvatarIdentity ¶
func (s *BaseSession) AvatarIdentity() string
AvatarIdentity returns the avatar participant identity.
func (*BaseSession) Callbacks ¶
func (s *BaseSession) Callbacks() *SessionCallbacks
Callbacks returns the session callbacks.
func (*BaseSession) EmitError ¶
func (s *BaseSession) EmitError(err error)
EmitError emits an error event if a callback is registered.
func (*BaseSession) EmitMetrics ¶
func (s *BaseSession) EmitMetrics(metrics Metrics)
EmitMetrics emits metrics if a callback is registered.
func (*BaseSession) EmitPlaybackFinished ¶
func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)
EmitPlaybackFinished emits a playback finished event if a callback is registered.
func (*BaseSession) EmitPlaybackStarted ¶
func (s *BaseSession) EmitPlaybackStarted()
EmitPlaybackStarted emits a playback started event if a callback is registered.
func (*BaseSession) IsStarted ¶
func (s *BaseSession) IsStarted() bool
IsStarted returns true if the session has been started.
func (*BaseSession) MarkStarted ¶
func (s *BaseSession) MarkStarted()
MarkStarted marks the session as started.
func (*BaseSession) Provider ¶
func (s *BaseSession) Provider() string
Provider returns the provider name.
func (*BaseSession) Room ¶
func (s *BaseSession) Room() *lksdk.Room
Room returns the room reference.
func (*BaseSession) SetAudioOutput ¶
func (s *BaseSession) SetAudioOutput(out AudioDestination)
SetAudioOutput sets the audio destination.
func (*BaseSession) SetCallbacks ¶
func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)
SetCallbacks sets the session callbacks.
func (*BaseSession) SetRoom ¶
func (s *BaseSession) SetRoom(room *lksdk.Room)
SetRoom sets the room reference.
func (*BaseSession) StartTime ¶
func (s *BaseSession) StartTime() time.Time
StartTime returns when the session was started.
type BitHumanConfig ¶ added in v0.3.0
type BitHumanConfig struct {
// APIKey is the bitHuman API key.
// Required.
APIKey string
// BaseURL is the bitHuman API base URL.
// Default: https://api.bithuman.ai
BaseURL string
// AgentID is the bitHuman agent to use for the session.
// Required.
AgentID string
// AudioConfig configures the audio format.
// Default: 24kHz mono PCM16
AudioConfig AudioConfig
}
BitHumanConfig contains configuration for the bitHuman avatar provider.
type Config ¶ added in v0.3.0
type Config struct {
// Provider selects the avatar provider.
// Supported values: "tavus", "bithuman", "anam", "simli", "" (none/audio-only)
// Note: "static" is not a Session provider; handle static images separately.
Provider string
// Tavus contains Tavus-specific configuration.
// Used when Provider is "tavus".
Tavus TavusConfig
// BitHuman contains bitHuman-specific configuration.
// Used when Provider is "bithuman".
BitHuman BitHumanConfig
// Anam contains Anam-specific configuration.
// Used when Provider is "anam".
Anam AnamConfig
// Simli contains Simli-specific configuration.
// Used when Provider is "simli".
Simli SimliConfig
}
Config is the unified configuration for creating avatar sessions. The Provider field determines which provider-specific config is used.
type DataStreamAudioOutput ¶
type DataStreamAudioOutput struct {
// contains filtered or unexported fields
}
DataStreamAudioOutput streams audio to a remote avatar via LiveKit ByteStream.
It handles:
- Creating ByteStream writers for each utterance
- Registering RPC handlers for playback events
- Sending clear buffer requests
func NewDataStreamAudioOutput ¶
func NewDataStreamAudioOutput(cfg DataStreamConfig) (*DataStreamAudioOutput, error)
NewDataStreamAudioOutput creates a new DataStreamAudioOutput.
func (*DataStreamAudioOutput) CaptureFrame ¶
func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
CaptureFrame sends a PCM16 audio frame to the avatar.
func (*DataStreamAudioOutput) Channels ¶
func (o *DataStreamAudioOutput) Channels() int
Channels returns the expected number of audio channels.
func (*DataStreamAudioOutput) ClearBuffer ¶
func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error
ClearBuffer interrupts current playback.
func (*DataStreamAudioOutput) Close ¶
func (o *DataStreamAudioOutput) Close() error
Close releases resources.
func (*DataStreamAudioOutput) Flush ¶
func (o *DataStreamAudioOutput) Flush(ctx context.Context) error
Flush marks the end of an audio segment.
func (*DataStreamAudioOutput) OnPlayback ¶
func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)
OnPlayback registers a callback for playback events.
func (*DataStreamAudioOutput) SampleRate ¶
func (o *DataStreamAudioOutput) SampleRate() int
SampleRate returns the expected input sample rate.
type DataStreamConfig ¶
type DataStreamConfig struct {
// Room is the LiveKit room.
// Required.
Room *lksdk.Room
// DestinationIdentity is the avatar participant identity.
// Required.
DestinationIdentity string
// Audio is the audio configuration.
// Optional, defaults to DefaultAudioConfig().
Audio AudioConfig
// RPCTimeout is the timeout for RPC calls.
// Default: 5 seconds
RPCTimeout time.Duration
}
DataStreamConfig configures the DataStreamAudioOutput.
type LiveAvatarHelper ¶ added in v0.3.0
type LiveAvatarHelper struct {
// contains filtered or unexported fields
}
LiveAvatarHelper manages a live avatar session lifecycle. This is a convenience wrapper for common live avatar operations.
func NewLiveAvatarHelper ¶ added in v0.3.0
func NewLiveAvatarHelper(cfg SetupConfig) (*LiveAvatarHelper, error)
NewLiveAvatarHelper creates a helper for managing a live avatar session.
func (*LiveAvatarHelper) AudioOutput ¶ added in v0.3.0
func (h *LiveAvatarHelper) AudioOutput() AudioDestination
AudioOutput returns the audio destination for streaming TTS audio.
func (*LiveAvatarHelper) Close ¶ added in v0.3.0
func (h *LiveAvatarHelper) Close(ctx context.Context) error
Close cleans up the avatar session.
func (*LiveAvatarHelper) Session ¶ added in v0.3.0
func (h *LiveAvatarHelper) Session() Session
Session returns the underlying avatar session.
func (*LiveAvatarHelper) Start ¶ added in v0.3.0
Start initializes the live avatar session. Call this after the agent has joined the room.
func (*LiveAvatarHelper) WaitForJoin ¶ added in v0.3.0
WaitForJoin waits for the avatar to join the room.
type Metrics ¶
type Metrics struct {
// AvatarJoinLatency is the time from Start() to avatar joining the room.
AvatarJoinLatency time.Duration
// PlaybackLatency is the time from audio send to avatar speech start.
// This is measured per utterance.
PlaybackLatency time.Duration
// Provider is the avatar provider name.
Provider string
// Timestamp is when the metrics were collected.
Timestamp time.Time
}
Metrics contains avatar performance metrics.
type PlaybackCallback ¶
type PlaybackCallback func(event PlaybackEvent)
PlaybackCallback is called when playback events occur.
type PlaybackEvent ¶
type PlaybackEvent struct {
// Type is the type of playback event.
Type PlaybackEventType
// Position is the playback position in seconds when the event occurred.
// Only meaningful for PlaybackFinished events.
Position float64
// Interrupted indicates if playback was interrupted by ClearBuffer().
// Only meaningful for PlaybackFinished events.
Interrupted bool
}
PlaybackEvent represents a playback state change from the avatar.
type PlaybackEventType ¶
type PlaybackEventType string
PlaybackEventType identifies the type of playback event.
const ( // PlaybackStarted indicates the avatar started speaking. PlaybackStarted PlaybackEventType = "started" // PlaybackFinished indicates the avatar finished speaking. // Check Position and Interrupted for details. PlaybackFinished PlaybackEventType = "finished" )
type ProviderError ¶
type ProviderError struct {
Provider string // Provider name (e.g., "tavus", "anam")
Op string // Operation that failed
Err error // Underlying error
}
ProviderError wraps an error from an avatar provider with additional context.
func NewProviderError ¶
func NewProviderError(provider, op string, err error) *ProviderError
NewProviderError creates a new ProviderError.
func (*ProviderError) Error ¶
func (e *ProviderError) Error() string
Error implements the error interface.
func (*ProviderError) Unwrap ¶
func (e *ProviderError) Unwrap() error
Unwrap returns the underlying error.
type QueueAudioOutput ¶
type QueueAudioOutput struct {
// contains filtered or unexported fields
}
QueueAudioOutput is an in-memory AudioDestination for testing.
It captures all audio frames sent to it, allowing tests to verify that the correct audio data is being produced.
func NewQueueAudioOutput ¶
func NewQueueAudioOutput(config AudioConfig) *QueueAudioOutput
NewQueueAudioOutput creates a new QueueAudioOutput with the given config.
func (*QueueAudioOutput) CaptureFrame ¶
func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error
CaptureFrame stores a PCM16 audio frame in the queue.
func (*QueueAudioOutput) Channels ¶
func (q *QueueAudioOutput) Channels() int
Channels returns the configured number of channels.
func (*QueueAudioOutput) ClearBuffer ¶
func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error
ClearBuffer simulates interrupting playback.
func (*QueueAudioOutput) Close ¶
func (q *QueueAudioOutput) Close() error
Close marks the queue as closed.
func (*QueueAudioOutput) Flush ¶
func (q *QueueAudioOutput) Flush(ctx context.Context) error
Flush marks the end of an audio segment.
func (*QueueAudioOutput) FrameCount ¶
func (q *QueueAudioOutput) FrameCount() int
FrameCount returns the number of frames captured.
func (*QueueAudioOutput) Frames ¶
func (q *QueueAudioOutput) Frames() [][]byte
Frames returns all captured frames.
func (*QueueAudioOutput) OnPlayback ¶
func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)
OnPlayback registers a callback for playback events.
func (*QueueAudioOutput) Reset ¶
func (q *QueueAudioOutput) Reset()
Reset clears all captured data and resets state.
func (*QueueAudioOutput) SampleRate ¶
func (q *QueueAudioOutput) SampleRate() int
SampleRate returns the configured sample rate.
func (*QueueAudioOutput) SimulatePlaybackFinished ¶
func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)
SimulatePlaybackFinished simulates a playback finished event.
func (*QueueAudioOutput) SimulatePlaybackStarted ¶
func (q *QueueAudioOutput) SimulatePlaybackStarted()
SimulatePlaybackStarted simulates a playback started event.
func (*QueueAudioOutput) TotalBytes ¶
func (q *QueueAudioOutput) TotalBytes() int
TotalBytes returns the total number of bytes captured.
func (*QueueAudioOutput) WasCleared ¶
func (q *QueueAudioOutput) WasCleared() bool
WasCleared returns true if ClearBuffer() was called.
func (*QueueAudioOutput) WasFlushed ¶
func (q *QueueAudioOutput) WasFlushed() bool
WasFlushed returns true if Flush() was called.
type Session ¶
type Session interface {
// AvatarIdentity returns the participant identity of the avatar worker.
// This is the identity that appears in the room and publishes video.
AvatarIdentity() string
// Provider returns the provider name (e.g., "tavus", "anam", "simli").
Provider() string
// Start initializes the avatar session.
//
// This method should:
// 1. Create a session with the avatar provider's API
// 2. Generate a LiveKit token for the avatar to join the room
// 3. Configure the audio output to stream to the avatar
//
// The avatar will join the room asynchronously. Use WaitForJoin()
// to wait for the avatar to be ready.
Start(ctx context.Context, opts StartOptions) error
// WaitForJoin blocks until the avatar participant joins the room
// and publishes the expected tracks (typically video).
//
// Returns an error if the timeout is exceeded or the context is cancelled.
WaitForJoin(ctx context.Context, timeout time.Duration) error
// AudioOutput returns the audio destination for streaming TTS audio
// to the avatar. Returns nil if the session is not started.
AudioOutput() AudioDestination
// Close disconnects the avatar and cleans up resources.
//
// This method should:
// 1. End the session with the avatar provider
// 2. Remove the avatar participant from the room
// 3. Clean up any registered RPC handlers
Close(ctx context.Context) error
}
Session manages a lip-sync avatar that publishes video to the room.
Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.
The typical flow is:
- Create a Session with provider-specific configuration
- Call Start() to initialize the avatar and connect it to the room
- Call WaitForJoin() to wait for the avatar to be ready
- Use AudioOutput() to stream TTS audio to the avatar
- Call Close() when done to clean up resources
func NewSession ¶ added in v0.3.0
NewSession creates an avatar session based on the provided configuration.
Returns nil, nil if Provider is empty (audio-only mode). Returns an error if the provider is not registered or configuration is invalid.
The provider must be registered via RegisterProvider before calling this function. The tavus package registers itself automatically when imported.
Example:
import _ "github.com/plexusone/omni-livekit/avatar/tavus" // Register tavus provider
session, err := avatar.NewSession(avatar.Config{
Provider: avatar.ProviderTavus,
Tavus: avatar.TavusConfig{
APIKey: os.Getenv("TAVUS_API_KEY"),
PalID: os.Getenv("TAVUS_PAL_ID"),
},
})
type SessionCallbacks ¶
type SessionCallbacks struct {
// OnMetricsCollected is called when avatar metrics are available.
OnMetricsCollected func(metrics Metrics)
// OnPlaybackStarted is called when the avatar starts speaking.
OnPlaybackStarted func()
// OnPlaybackFinished is called when the avatar finishes speaking.
// The position is the playback position in seconds when stopped.
// The interrupted flag indicates if playback was interrupted by
// a ClearBuffer() call.
OnPlaybackFinished func(position float64, interrupted bool)
// OnAvatarJoined is called when the avatar participant joins the room.
OnAvatarJoined func(identity string)
// OnAvatarLeft is called when the avatar participant leaves the room.
OnAvatarLeft func(identity string)
// OnError is called when an error occurs.
// This is for non-fatal errors that don't cause the session to fail.
OnError func(err error)
}
SessionCallbacks defines optional event callbacks for avatar sessions.
type SessionConstructor ¶ added in v0.3.0
SessionConstructor is a function that creates a Session from a Config.
type SetupConfig ¶ added in v0.3.0
type SetupConfig struct {
// Provider selects the avatar type:
// - "" or "none": Audio-only, no avatar
// - "static": Static image (configure StaticImage field)
// - "tavus": Live Tavus avatar (configure Tavus field)
// - "anam", "simli": Other live avatar providers
Provider string
// StaticImage configures a static image avatar.
// Used when Provider is "static".
StaticImage StaticImageConfig
// Tavus configures a Tavus live avatar.
// Used when Provider is "tavus".
Tavus TavusConfig
// Anam configures an Anam live avatar.
// Used when Provider is "anam".
Anam AnamConfig
// Simli configures a Simli live avatar.
// Used when Provider is "simli".
Simli SimliConfig
// LiveKit connection settings (required for live avatars)
LiveKitURL string
LiveKitAPIKey string
LiveKitAPISecret string
}
SetupConfig contains all configuration for avatar setup. This is the unified configuration that commands should use.
type SetupMode ¶ added in v0.3.0
type SetupMode string
SetupMode indicates the type of avatar setup.
const ( // SetupModeNone means no avatar (audio-only). SetupModeNone SetupMode = "none" // SetupModeStatic means a static image avatar. // The caller should configure the agent's Image options. SetupModeStatic SetupMode = "static" // SetupModeLive means a live lip-sync avatar. // The caller should use the returned Session for audio streaming. SetupModeLive SetupMode = "live" )
type SetupResult ¶ added in v0.3.0
type SetupResult struct {
// Mode indicates the avatar mode to use.
Mode SetupMode
// Session is the live avatar session (nil for static or none).
// The caller is responsible for calling Session.Close() when done.
Session Session
// StaticImage contains static image configuration for the agent.
// Only populated when Mode is SetupModeStatic.
StaticImage *StaticImageResult
}
SetupResult contains the result of avatar setup.
func Setup ¶ added in v0.3.0
func Setup(cfg SetupConfig) (*SetupResult, error)
Setup creates the appropriate avatar based on the configuration.
For static avatars, it returns configuration to apply to agent options. For live avatars, it creates a Session that must be started and closed.
Example usage:
result, err := avatar.Setup(avatar.SetupConfig{
Provider: "tavus",
Tavus: avatar.TavusConfig{
APIKey: os.Getenv("TAVUS_API_KEY"),
},
LiveKitURL: os.Getenv("LIVEKIT_URL"),
LiveKitAPIKey: os.Getenv("LIVEKIT_API_KEY"),
LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})
if err != nil {
log.Fatal(err)
}
switch result.Mode {
case avatar.SetupModeNone:
// Audio-only, no avatar configuration needed
case avatar.SetupModeStatic:
// Apply static image config to agent
agentOpts.MediaMode = agent.AudioWithImage
agentOpts.Image.H264Path = result.StaticImage.H264Path
case avatar.SetupModeLive:
// Use live avatar session
defer result.Session.Close(ctx)
err := result.Session.Start(ctx, avatar.StartOptions{...})
}
type SimliConfig ¶ added in v0.3.0
type SimliConfig struct {
// APIKey is the Simli API key.
APIKey string
// AvatarID is the Simli avatar to use.
AvatarID string
// AudioConfig configures the audio format.
AudioConfig AudioConfig
}
SimliConfig contains configuration for the Simli avatar provider. Placeholder for future implementation.
type StartOptions ¶
type StartOptions struct {
// Room is the LiveKit room the agent has joined.
// Required.
Room *lksdk.Room
// AgentIdentity is the identity of the agent participant.
// The avatar will publish tracks on behalf of this identity
// using the lk.publish_on_behalf attribute.
// Required.
AgentIdentity string
// LiveKitURL is the LiveKit server URL for the avatar to connect to.
// This should match the URL the agent connected to.
// Required.
LiveKitURL string
// LiveKitAPIKey is used to generate tokens for the avatar.
// Required.
LiveKitAPIKey string
// LiveKitAPISecret is used to generate tokens for the avatar.
// Required.
LiveKitAPISecret string
// Callbacks configures optional event callbacks.
Callbacks *SessionCallbacks
}
StartOptions configures avatar session startup.
func (*StartOptions) Validate ¶
func (o *StartOptions) Validate() error
Validate checks that all required fields are set.
type StaticImageConfig ¶ added in v0.3.0
type StaticImageConfig struct {
// H264Path is the path to a pre-encoded H.264 keyframe file.
// This is the recommended approach - no CGO required at runtime.
H264Path string
// H264Data is pre-encoded H.264 keyframe data (alternative to H264Path).
H264Data []byte
// UseDefault uses the embedded default OmniAgent avatar.
// If true, H264Path and H264Data are ignored.
UseDefault bool
}
StaticImageConfig configures a static image avatar.
type StaticImageResult ¶ added in v0.3.0
type StaticImageResult struct {
// H264Path is the path to the H.264 file (if using file).
H264Path string
// H264Data is the H.264 data (if using embedded data).
H264Data []byte
}
StaticImageResult contains static image configuration to apply to agent options.
type TavusConfig ¶ added in v0.3.0
type TavusConfig struct {
// APIKey is the Tavus API key.
// Required.
APIKey string
// BaseURL is the Tavus API base URL.
// Default: https://tavusapi.com
BaseURL string
// PalID is the PAL (Personalized AI Likeness) to use.
// Default: stock avatar (provider default)
PalID string
// FaceID is an optional face override.
FaceID string
// AudioConfig configures the audio format.
// Default: 24kHz mono PCM16 (Tavus requirement)
AudioConfig AudioConfig
}
TavusConfig contains configuration for the Tavus avatar provider.
type TokenOptions ¶
type TokenOptions struct {
// APIKey is the LiveKit API key.
// Required.
APIKey string
// APISecret is the LiveKit API secret.
// Required.
APISecret string
// RoomName is the room the avatar will join.
// Required.
RoomName string
// AvatarIdentity is the participant identity for the avatar.
// Required.
AvatarIdentity string
// AvatarName is the display name for the avatar participant.
// Optional, defaults to AvatarIdentity.
AvatarName string
// PublishOnBehalf is the identity of the agent participant.
// The avatar will publish tracks that appear as if they're from this participant.
// Required.
PublishOnBehalf string
// TTL is the token validity duration.
// Default: 5 minutes
TTL time.Duration
// Metadata is optional participant metadata.
Metadata string
}
TokenOptions configures avatar token generation.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bithuman provides bitHuman avatar integration for omni-livekit voice agents.
|
Package bithuman provides bitHuman avatar integration for omni-livekit voice agents. |
|
Package tavus provides Tavus avatar integration for omni-livekit voice agents.
|
Package tavus provides Tavus avatar integration for omni-livekit voice agents. |