avatar

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 12, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package avatar provides lip-sync avatar support for omni-livekit voice agents.

This package enables voice agents to display real-time lip-synced video avatars that move their mouths in sync with generated speech. It supports multiple avatar providers (Tavus, Anam, Simli, etc.) through a pluggable architecture.

Architecture

The avatar system consists of three main components:

  1. Session - Manages the avatar lifecycle (start, wait for join, close)
  2. AudioDestination - Streams TTS audio to the avatar provider
  3. RPC Handlers - Handle playback control (start, finish, clear buffer)

The avatar provider joins the LiveKit room as a separate participant and publishes video+audio tracks on behalf of the agent using the "lk.publish_on_behalf" participant attribute.

Basic Usage

// Create avatar session with a provider
session, err := tavus.New(tavus.Config{
    APIKey: os.Getenv("TAVUS_API_KEY"),
    FaceID: "your-face-id",
})

// Start the avatar
err = session.Start(ctx, avatar.StartOptions{
    Room:          agent.Room(),
    AgentIdentity: "meeting-pm",
    LiveKitURL:    os.Getenv("LIVEKIT_URL"),
    // ...
})

// Wait for avatar to join
err = session.WaitForJoin(ctx, 10*time.Second)

// Get audio destination for TTS output
audioOut := session.AudioOutput()

// Stream TTS audio to avatar
audioOut.CaptureFrame(ctx, pcmFrame)
audioOut.Flush(ctx)

// Handle interruption
audioOut.ClearBuffer(ctx)

Providers

Avatar providers are implemented in sub-packages:

  • avatar/tavus - Tavus avatar provider
  • avatar/anam - Anam avatar provider
  • avatar/simli - Simli avatar provider

Each provider implements the Session interface and handles provider-specific API calls to create avatar sessions.

Audio Streaming

Audio is streamed from the agent to the avatar using LiveKit's ByteStream API. The DataStreamAudioOutput implementation handles:

  • Streaming PCM audio frames to the avatar
  • Managing stream lifecycle (create, write, close)
  • RPC-based playback control

Playback Control

The avatar communicates playback state via RPC:

  • lk.playback_started - Avatar started speaking
  • lk.playback_finished - Avatar finished speaking (with position and interrupted flag)
  • lk.clear_buffer - Agent requests playback interruption

Index

Constants

View Source
const (
	// RPCPlaybackStarted is sent by the avatar when it starts speaking.
	RPCPlaybackStarted = "lk.playback_started"

	// RPCPlaybackFinished is sent by the avatar when it finishes speaking.
	RPCPlaybackFinished = "lk.playback_finished"

	// RPCClearBuffer is sent by the agent to interrupt playback.
	RPCClearBuffer = "lk.clear_buffer"
)

RPC method names for playback control. These must match the methods implemented by avatar workers.

View Source
const (
	ProviderNone     = ""         // No avatar (audio-only mode)
	ProviderTavus    = "tavus"    // Tavus conversational video
	ProviderBitHuman = "bithuman" // bitHuman real-time avatars
	ProviderHeyGen   = "heygen"   // HeyGen LiveAvatar real-time streaming
	ProviderAnam     = "anam"     // Anam avatar (future)
	ProviderSimli    = "simli"    // Simli avatar (future)
	ProviderStatic   = "static"   // Static image (not a Session, handled separately)
)

Provider constants for avatar providers.

View Source
const AudioStreamTopic = "lk.audio_stream"

AudioStreamTopic is the ByteStream topic for audio data.

Variables

View Source
var (
	// ErrSessionNotStarted indicates that Start() was not called before
	// attempting an operation that requires an active session.
	ErrSessionNotStarted = errors.New("avatar: session not started")

	// ErrSessionAlreadyStarted indicates that Start() was called on a
	// session that is already running.
	ErrSessionAlreadyStarted = errors.New("avatar: session already started")

	// ErrAvatarJoinTimeout indicates that the avatar participant did not
	// join the room within the specified timeout.
	ErrAvatarJoinTimeout = errors.New("avatar: join timeout")

	// ErrAvatarTrackTimeout indicates that the avatar did not publish
	// the expected track (video/audio) within the timeout.
	ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout")

	// ErrProviderUnavailable indicates that the avatar provider API
	// is unreachable or returned an error.
	ErrProviderUnavailable = errors.New("avatar: provider unavailable")

	// ErrProviderAuthFailed indicates that authentication with the
	// avatar provider failed (invalid API key, etc.).
	ErrProviderAuthFailed = errors.New("avatar: provider authentication failed")

	// ErrProviderRateLimited indicates that the avatar provider has
	// rate-limited the request.
	ErrProviderRateLimited = errors.New("avatar: provider rate limited")

	// ErrInvalidConfig indicates that the avatar configuration is
	// invalid or incomplete.
	ErrInvalidConfig = errors.New("avatar: invalid configuration")

	// ErrRPCTimeout indicates that an RPC call to the avatar timed out.
	ErrRPCTimeout = errors.New("avatar: RPC timeout")

	// ErrRPCFailed indicates that an RPC call to the avatar failed.
	ErrRPCFailed = errors.New("avatar: RPC failed")

	// ErrStreamClosed indicates that the audio stream was closed
	// unexpectedly.
	ErrStreamClosed = errors.New("avatar: stream closed")

	// ErrAvatarDisconnected indicates that the avatar participant
	// disconnected from the room unexpectedly.
	ErrAvatarDisconnected = errors.New("avatar: disconnected")
)

Sentinel errors for avatar operations.

Functions

func GenerateAvatarToken

func GenerateAvatarToken(opts TokenOptions) (string, error)

GenerateAvatarToken creates a JWT token for an avatar to join a room.

The token includes the special "lk.publish_on_behalf" attribute that allows the avatar participant to publish tracks that appear in the UI as if they came from the agent participant.

func IsProviderRegistered added in v0.3.0

func IsProviderRegistered(name string) bool

IsProviderRegistered returns true if the provider is registered.

func RegisterProvider added in v0.3.0

func RegisterProvider(name string, constructor SessionConstructor)

RegisterProvider registers a session constructor for a provider. This allows providers to be registered from their sub-packages without creating import cycles.

Example usage in avatar/tavus/init.go:

func init() {
    avatar.RegisterProvider("tavus", func(cfg avatar.Config) (avatar.Session, error) {
        return NewSession(SessionConfig{
            APIKey:      cfg.Tavus.APIKey,
            BaseURL:     cfg.Tavus.BaseURL,
            PalID:       cfg.Tavus.PalID,
            FaceID:      cfg.Tavus.FaceID,
            AudioConfig: cfg.Tavus.AudioConfig,
        })
    })
}

func RegisteredProviders added in v0.3.0

func RegisteredProviders() []string

RegisteredProviders returns a list of registered provider names.

Types

type AnamConfig added in v0.3.0

type AnamConfig struct {
	// APIKey is the Anam API key.
	APIKey string

	// PersonaID is the Anam persona to use.
	PersonaID string

	// AudioConfig configures the audio format.
	AudioConfig AudioConfig
}

AnamConfig contains configuration for the Anam avatar provider. Placeholder for future implementation.

type AudioConfig

type AudioConfig struct {
	// SampleRate is the audio sample rate in Hz.
	// Default: 24000 (common for avatar providers)
	SampleRate int

	// Channels is the number of audio channels.
	// Default: 1 (mono)
	Channels int

	// Encoding is the audio encoding format.
	// Default: "linear16" (PCM16 little-endian)
	Encoding string
}

AudioConfig holds audio configuration for avatar sessions.

func DefaultAudioConfig

func DefaultAudioConfig() AudioConfig

DefaultAudioConfig returns the default audio configuration.

func (AudioConfig) FrameDuration

func (c AudioConfig) FrameDuration(frameSize int) int

FrameDuration returns the duration in milliseconds for the given frame size.

func (AudioConfig) FrameSize

func (c AudioConfig) FrameSize(durationMs int) int

FrameSize returns the number of bytes per frame for the given duration in ms.

type AudioDestination

type AudioDestination interface {
	// CaptureFrame sends a PCM16 audio frame to the avatar.
	//
	// The frame should be little-endian PCM16 at the configured sample rate.
	// Frames are typically 20ms of audio (e.g., 960 samples at 48kHz).
	//
	// This method may block if the underlying transport is congested.
	CaptureFrame(ctx context.Context, frame []byte) error

	// Flush marks the end of an audio segment (utterance).
	//
	// The avatar should speak all buffered audio before accepting more.
	// This closes the current ByteStream and prepares for the next utterance.
	Flush(ctx context.Context) error

	// ClearBuffer interrupts current playback.
	//
	// Use this when the user interrupts the agent. The avatar should
	// stop speaking immediately and discard any buffered audio.
	//
	// This sends an RPC to the avatar worker and waits for acknowledgment.
	ClearBuffer(ctx context.Context) error

	// SampleRate returns the expected input sample rate in Hz.
	// Common values: 16000, 24000, 48000
	SampleRate() int

	// Channels returns the expected number of audio channels.
	// Typically 1 (mono).
	Channels() int

	// OnPlayback registers a callback for playback events.
	// Multiple callbacks can be registered.
	OnPlayback(callback PlaybackCallback)

	// Close releases resources associated with this audio destination.
	// After Close(), no more frames should be sent.
	Close() error
}

AudioDestination receives audio frames and forwards them to an avatar.

Implementations include:

  • DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
  • QueueAudioOutput: In-memory queue for testing

The typical usage flow is:

  1. Call CaptureFrame() for each PCM audio frame from TTS
  2. Call Flush() when the utterance is complete
  3. Call ClearBuffer() to interrupt playback (e.g., user interruption)

type AvatarMetadata

type AvatarMetadata struct {
	// Kind identifies this as an avatar participant.
	Kind string `json:"kind"`

	// Provider is the avatar provider name.
	Provider string `json:"provider,omitempty"`

	// AgentIdentity is the identity of the agent this avatar represents.
	AgentIdentity string `json:"agent_identity,omitempty"`
}

AvatarMetadata is the standard metadata structure for avatar participants.

func DefaultAvatarMetadata

func DefaultAvatarMetadata(provider, agentIdentity string) AvatarMetadata

DefaultAvatarMetadata returns the default metadata for avatar participants.

type BaseSession

type BaseSession struct {
	// contains filtered or unexported fields
}

BaseSession provides common functionality for avatar session implementations. Provider-specific sessions can embed this struct.

func NewBaseSession

func NewBaseSession(provider, avatarIdentity string) *BaseSession

NewBaseSession creates a new BaseSession.

func (*BaseSession) AudioOutput

func (s *BaseSession) AudioOutput() AudioDestination

AudioOutput returns the audio destination.

func (*BaseSession) AvatarIdentity

func (s *BaseSession) AvatarIdentity() string

AvatarIdentity returns the avatar participant identity.

func (*BaseSession) Callbacks

func (s *BaseSession) Callbacks() *SessionCallbacks

Callbacks returns the session callbacks.

func (*BaseSession) EmitError

func (s *BaseSession) EmitError(err error)

EmitError emits an error event if a callback is registered.

func (*BaseSession) EmitMetrics

func (s *BaseSession) EmitMetrics(metrics Metrics)

EmitMetrics emits metrics if a callback is registered.

func (*BaseSession) EmitPlaybackFinished

func (s *BaseSession) EmitPlaybackFinished(position float64, interrupted bool)

EmitPlaybackFinished emits a playback finished event if a callback is registered.

func (*BaseSession) EmitPlaybackStarted

func (s *BaseSession) EmitPlaybackStarted()

EmitPlaybackStarted emits a playback started event if a callback is registered.

func (*BaseSession) IsStarted

func (s *BaseSession) IsStarted() bool

IsStarted returns true if the session has been started.

func (*BaseSession) MarkStarted

func (s *BaseSession) MarkStarted()

MarkStarted marks the session as started.

func (*BaseSession) Provider

func (s *BaseSession) Provider() string

Provider returns the provider name.

func (*BaseSession) Room

func (s *BaseSession) Room() *lksdk.Room

Room returns the room reference.

func (*BaseSession) SetAudioOutput

func (s *BaseSession) SetAudioOutput(out AudioDestination)

SetAudioOutput sets the audio destination.

func (*BaseSession) SetCallbacks

func (s *BaseSession) SetCallbacks(callbacks *SessionCallbacks)

SetCallbacks sets the session callbacks.

func (*BaseSession) SetRoom

func (s *BaseSession) SetRoom(room *lksdk.Room)

SetRoom sets the room reference.

func (*BaseSession) StartTime

func (s *BaseSession) StartTime() time.Time

StartTime returns when the session was started.

type BitHumanConfig added in v0.3.0

type BitHumanConfig struct {
	// APIKey is the bitHuman API key.
	// Required.
	APIKey string

	// BaseURL is the bitHuman API base URL.
	// Default: https://api.bithuman.ai
	BaseURL string

	// AgentID is the bitHuman agent to use for the session.
	// Required.
	AgentID string

	// AudioConfig configures the audio format.
	// Default: 24kHz mono PCM16
	AudioConfig AudioConfig
}

BitHumanConfig contains configuration for the bitHuman avatar provider.

type Config added in v0.3.0

type Config struct {
	// Provider selects the avatar provider.
	// Supported values: "tavus", "bithuman", "heygen", "anam", "simli", "" (none/audio-only)
	// Note: "static" is not a Session provider; handle static images separately.
	Provider string

	// Tavus contains Tavus-specific configuration.
	// Used when Provider is "tavus".
	Tavus TavusConfig

	// BitHuman contains bitHuman-specific configuration.
	// Used when Provider is "bithuman".
	BitHuman BitHumanConfig

	// HeyGen contains HeyGen LiveAvatar configuration.
	// Used when Provider is "heygen".
	HeyGen HeyGenConfig

	// Anam contains Anam-specific configuration.
	// Used when Provider is "anam".
	Anam AnamConfig

	// Simli contains Simli-specific configuration.
	// Used when Provider is "simli".
	Simli SimliConfig
}

Config is the unified configuration for creating avatar sessions. The Provider field determines which provider-specific config is used.

type DataStreamAudioOutput

type DataStreamAudioOutput struct {
	// contains filtered or unexported fields
}

DataStreamAudioOutput streams audio to a remote avatar via LiveKit ByteStream.

It handles:

  • Creating ByteStream writers for each utterance
  • Registering RPC handlers for playback events
  • Sending clear buffer requests

func NewDataStreamAudioOutput

func NewDataStreamAudioOutput(cfg DataStreamConfig) (*DataStreamAudioOutput, error)

NewDataStreamAudioOutput creates a new DataStreamAudioOutput.

func (*DataStreamAudioOutput) CaptureFrame

func (o *DataStreamAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error

CaptureFrame sends a PCM16 audio frame to the avatar.

func (*DataStreamAudioOutput) Channels

func (o *DataStreamAudioOutput) Channels() int

Channels returns the expected number of audio channels.

func (*DataStreamAudioOutput) ClearBuffer

func (o *DataStreamAudioOutput) ClearBuffer(ctx context.Context) error

ClearBuffer interrupts current playback.

func (*DataStreamAudioOutput) Close

func (o *DataStreamAudioOutput) Close() error

Close releases resources.

func (*DataStreamAudioOutput) Flush

Flush marks the end of an audio segment.

func (*DataStreamAudioOutput) OnPlayback

func (o *DataStreamAudioOutput) OnPlayback(callback PlaybackCallback)

OnPlayback registers a callback for playback events.

func (*DataStreamAudioOutput) SampleRate

func (o *DataStreamAudioOutput) SampleRate() int

SampleRate returns the expected input sample rate.

type DataStreamConfig

type DataStreamConfig struct {
	// Room is the LiveKit room.
	// Required.
	Room *lksdk.Room

	// DestinationIdentity is the avatar participant identity.
	// Required.
	DestinationIdentity string

	// Audio is the audio configuration.
	// Optional, defaults to DefaultAudioConfig().
	Audio AudioConfig

	// RPCTimeout is the timeout for RPC calls.
	// Default: 5 seconds
	RPCTimeout time.Duration
}

DataStreamConfig configures the DataStreamAudioOutput.

type HeyGenConfig added in v0.4.0

type HeyGenConfig struct {
	// APIKey is the LiveAvatar API key.
	// Note: This is different from the HeyGen video generation API key.
	// Get your key from: https://app.liveavatar.com/developers
	// Required.
	APIKey string

	// BaseURL is the LiveAvatar API base URL.
	// Default: https://api.liveavatar.com
	BaseURL string

	// AvatarID is the UUID of the avatar to use.
	// Use liveavatar.SandboxAvatarID ("65f9e3c9-d48b-4118-b73a-4ae2e3cbb8f0") for testing.
	// Required.
	AvatarID string

	// Sandbox enables sandbox mode (60s limit, no credits).
	// Recommended for development and testing.
	Sandbox bool

	// VideoQuality sets the avatar video quality.
	// Options: "very_high", "high", "medium", "low"
	// Default: "high"
	VideoQuality string

	// AudioConfig configures the audio format.
	// Default: 24kHz mono PCM16
	AudioConfig AudioConfig
}

HeyGenConfig contains configuration for the HeyGen LiveAvatar provider.

type LiveAvatarHelper added in v0.3.0

type LiveAvatarHelper struct {
	// contains filtered or unexported fields
}

LiveAvatarHelper manages a live avatar session lifecycle. This is a convenience wrapper for common live avatar operations.

func NewLiveAvatarHelper added in v0.3.0

func NewLiveAvatarHelper(cfg SetupConfig) (*LiveAvatarHelper, error)

NewLiveAvatarHelper creates a helper for managing a live avatar session.

func (*LiveAvatarHelper) AudioOutput added in v0.3.0

func (h *LiveAvatarHelper) AudioOutput() AudioDestination

AudioOutput returns the audio destination for streaming TTS audio.

func (*LiveAvatarHelper) Close added in v0.3.0

func (h *LiveAvatarHelper) Close(ctx context.Context) error

Close cleans up the avatar session.

func (*LiveAvatarHelper) Session added in v0.3.0

func (h *LiveAvatarHelper) Session() Session

Session returns the underlying avatar session.

func (*LiveAvatarHelper) Start added in v0.3.0

func (h *LiveAvatarHelper) Start(ctx context.Context, room *lksdk.Room, agentIdentity string) error

Start initializes the live avatar session. Call this after the agent has joined the room.

func (*LiveAvatarHelper) WaitForJoin added in v0.3.0

func (h *LiveAvatarHelper) WaitForJoin(ctx context.Context, timeout time.Duration) error

WaitForJoin waits for the avatar to join the room.

type LiveKitSession added in v0.4.0

type LiveKitSession struct {
	// contains filtered or unexported fields
}

LiveKitSession wraps an omniavatar Session with LiveKit-specific functionality.

This adapter:

  • Creates DataStreamAudioOutput for audio streaming
  • Wires up playback callbacks to the underlying session
  • Provides a unified interface for LiveKit agent integration

func WrapSession added in v0.4.0

func WrapSession(session avcore.Session) *LiveKitSession

WrapSession wraps an omniavatar Session with LiveKit-specific functionality.

The session's Start() method expects *omniavatar.LiveKitStartOptions. The adapter handles DataStreamAudioOutput creation and callback wiring.

func (*LiveKitSession) AudioOutput added in v0.4.0

func (s *LiveKitSession) AudioOutput() AudioDestination

AudioOutput returns the audio destination for streaming TTS audio.

func (*LiveKitSession) AvatarIdentity added in v0.4.0

func (s *LiveKitSession) AvatarIdentity() string

AvatarIdentity returns the avatar participant identity. Alias for Identity() for backwards compatibility.

func (*LiveKitSession) Close added in v0.4.0

func (s *LiveKitSession) Close(ctx context.Context) error

Close disconnects the avatar and cleans up resources.

func (*LiveKitSession) Identity added in v0.4.0

func (s *LiveKitSession) Identity() string

Identity returns the avatar participant identity.

func (*LiveKitSession) Provider added in v0.4.0

func (s *LiveKitSession) Provider() string

Provider returns the provider name.

func (*LiveKitSession) Room added in v0.4.0

func (s *LiveKitSession) Room() *lksdk.Room

Room returns the LiveKit room reference.

func (*LiveKitSession) Session added in v0.4.0

func (s *LiveKitSession) Session() avcore.Session

Session returns the underlying omniavatar session.

func (*LiveKitSession) Start added in v0.4.0

func (s *LiveKitSession) Start(ctx context.Context, opts StartOptions) error

Start initializes the session with LiveKit-specific setup.

This method:

  1. Creates DataStreamAudioOutput for audio streaming
  2. Wires up playback callbacks
  3. Calls the underlying session's Start()

func (*LiveKitSession) WaitForJoin added in v0.4.0

func (s *LiveKitSession) WaitForJoin(ctx context.Context, timeout time.Duration) error

WaitForJoin blocks until the avatar participant joins the room.

type Metrics

type Metrics struct {
	// AvatarJoinLatency is the time from Start() to avatar joining the room.
	AvatarJoinLatency time.Duration

	// PlaybackLatency is the time from audio send to avatar speech start.
	// This is measured per utterance.
	PlaybackLatency time.Duration

	// Provider is the avatar provider name.
	Provider string

	// Timestamp is when the metrics were collected.
	Timestamp time.Time
}

Metrics contains avatar performance metrics.

type PlaybackCallback

type PlaybackCallback func(event PlaybackEvent)

PlaybackCallback is called when playback events occur.

type PlaybackEvent

type PlaybackEvent struct {
	// Type is the type of playback event.
	Type PlaybackEventType

	// Position is the playback position in seconds when the event occurred.
	// Only meaningful for PlaybackFinished events.
	Position float64

	// Interrupted indicates if playback was interrupted by ClearBuffer().
	// Only meaningful for PlaybackFinished events.
	Interrupted bool
}

PlaybackEvent represents a playback state change from the avatar.

type PlaybackEventType

type PlaybackEventType string

PlaybackEventType identifies the type of playback event.

const (
	// PlaybackStarted indicates the avatar started speaking.
	PlaybackStarted PlaybackEventType = "started"

	// PlaybackFinished indicates the avatar finished speaking.
	// Check Position and Interrupted for details.
	PlaybackFinished PlaybackEventType = "finished"
)

type ProviderError

type ProviderError struct {
	Provider string // Provider name (e.g., "tavus", "anam")
	Op       string // Operation that failed
	Err      error  // Underlying error
}

ProviderError wraps an error from an avatar provider with additional context.

func NewProviderError

func NewProviderError(provider, op string, err error) *ProviderError

NewProviderError creates a new ProviderError.

func (*ProviderError) Error

func (e *ProviderError) Error() string

Error implements the error interface.

func (*ProviderError) Unwrap

func (e *ProviderError) Unwrap() error

Unwrap returns the underlying error.

type QueueAudioOutput

type QueueAudioOutput struct {
	// contains filtered or unexported fields
}

QueueAudioOutput is an in-memory AudioDestination for testing.

It captures all audio frames sent to it, allowing tests to verify that the correct audio data is being produced.

func NewQueueAudioOutput

func NewQueueAudioOutput(config AudioConfig) *QueueAudioOutput

NewQueueAudioOutput creates a new QueueAudioOutput with the given config.

func (*QueueAudioOutput) CaptureFrame

func (q *QueueAudioOutput) CaptureFrame(ctx context.Context, frame []byte) error

CaptureFrame stores a PCM16 audio frame in the queue.

func (*QueueAudioOutput) Channels

func (q *QueueAudioOutput) Channels() int

Channels returns the configured number of channels.

func (*QueueAudioOutput) ClearBuffer

func (q *QueueAudioOutput) ClearBuffer(ctx context.Context) error

ClearBuffer simulates interrupting playback.

func (*QueueAudioOutput) Close

func (q *QueueAudioOutput) Close() error

Close marks the queue as closed.

func (*QueueAudioOutput) Flush

func (q *QueueAudioOutput) Flush(ctx context.Context) error

Flush marks the end of an audio segment.

func (*QueueAudioOutput) FrameCount

func (q *QueueAudioOutput) FrameCount() int

FrameCount returns the number of frames captured.

func (*QueueAudioOutput) Frames

func (q *QueueAudioOutput) Frames() [][]byte

Frames returns all captured frames.

func (*QueueAudioOutput) OnPlayback

func (q *QueueAudioOutput) OnPlayback(callback PlaybackCallback)

OnPlayback registers a callback for playback events.

func (*QueueAudioOutput) Reset

func (q *QueueAudioOutput) Reset()

Reset clears all captured data and resets state.

func (*QueueAudioOutput) SampleRate

func (q *QueueAudioOutput) SampleRate() int

SampleRate returns the configured sample rate.

func (*QueueAudioOutput) SimulatePlaybackFinished

func (q *QueueAudioOutput) SimulatePlaybackFinished(position float64, interrupted bool)

SimulatePlaybackFinished simulates a playback finished event.

func (*QueueAudioOutput) SimulatePlaybackStarted

func (q *QueueAudioOutput) SimulatePlaybackStarted()

SimulatePlaybackStarted simulates a playback started event.

func (*QueueAudioOutput) TotalBytes

func (q *QueueAudioOutput) TotalBytes() int

TotalBytes returns the total number of bytes captured.

func (*QueueAudioOutput) WasCleared

func (q *QueueAudioOutput) WasCleared() bool

WasCleared returns true if ClearBuffer() was called.

func (*QueueAudioOutput) WasFlushed

func (q *QueueAudioOutput) WasFlushed() bool

WasFlushed returns true if Flush() was called.

type Session

type Session interface {
	// AvatarIdentity returns the participant identity of the avatar worker.
	// This is the identity that appears in the room and publishes video.
	AvatarIdentity() string

	// Provider returns the provider name (e.g., "tavus", "anam", "simli").
	Provider() string

	// Start initializes the avatar session.
	//
	// This method should:
	//  1. Create a session with the avatar provider's API
	//  2. Generate a LiveKit token for the avatar to join the room
	//  3. Configure the audio output to stream to the avatar
	//
	// The avatar will join the room asynchronously. Use WaitForJoin()
	// to wait for the avatar to be ready.
	Start(ctx context.Context, opts StartOptions) error

	// WaitForJoin blocks until the avatar participant joins the room
	// and publishes the expected tracks (typically video).
	//
	// Returns an error if the timeout is exceeded or the context is cancelled.
	WaitForJoin(ctx context.Context, timeout time.Duration) error

	// AudioOutput returns the audio destination for streaming TTS audio
	// to the avatar. Returns nil if the session is not started.
	AudioOutput() AudioDestination

	// Close disconnects the avatar and cleans up resources.
	//
	// This method should:
	//  1. End the session with the avatar provider
	//  2. Remove the avatar participant from the room
	//  3. Clean up any registered RPC handlers
	Close(ctx context.Context) error
}

Session manages a lip-sync avatar that publishes video to the room.

Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.

The typical flow is:

  1. Create a Session with provider-specific configuration
  2. Call Start() to initialize the avatar and connect it to the room
  3. Call WaitForJoin() to wait for the avatar to be ready
  4. Use AudioOutput() to stream TTS audio to the avatar
  5. Call Close() when done to clean up resources

func NewSession added in v0.3.0

func NewSession(cfg Config) (Session, error)

NewSession creates an avatar session based on the provided configuration.

Returns nil, nil if Provider is empty (audio-only mode). Returns an error if the provider is not registered or configuration is invalid.

The provider must be registered via RegisterProvider before calling this function. The tavus package registers itself automatically when imported.

Example:

import _ "github.com/plexusone/omni-livekit/avatar/tavus" // Register tavus provider

session, err := avatar.NewSession(avatar.Config{
    Provider: avatar.ProviderTavus,
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
        PalID:  os.Getenv("TAVUS_PAL_ID"),
    },
})

type SessionCallbacks

type SessionCallbacks struct {
	// OnMetricsCollected is called when avatar metrics are available.
	OnMetricsCollected func(metrics Metrics)

	// OnPlaybackStarted is called when the avatar starts speaking.
	OnPlaybackStarted func()

	// OnPlaybackFinished is called when the avatar finishes speaking.
	// The position is the playback position in seconds when stopped.
	// The interrupted flag indicates if playback was interrupted by
	// a ClearBuffer() call.
	OnPlaybackFinished func(position float64, interrupted bool)

	// OnAvatarJoined is called when the avatar participant joins the room.
	OnAvatarJoined func(identity string)

	// OnAvatarLeft is called when the avatar participant leaves the room.
	OnAvatarLeft func(identity string)

	// OnError is called when an error occurs.
	// This is for non-fatal errors that don't cause the session to fail.
	OnError func(err error)
}

SessionCallbacks defines optional event callbacks for avatar sessions.

type SessionConstructor added in v0.3.0

type SessionConstructor func(cfg Config) (Session, error)

SessionConstructor is a function that creates a Session from a Config.

type SetupConfig added in v0.3.0

type SetupConfig struct {
	// Provider selects the avatar type:
	//   - "" or "none": Audio-only, no avatar
	//   - "static": Static image (configure StaticImage field)
	//   - "tavus": Live Tavus avatar (configure Tavus field)
	//   - "anam", "simli": Other live avatar providers
	Provider string

	// StaticImage configures a static image avatar.
	// Used when Provider is "static".
	StaticImage StaticImageConfig

	// Tavus configures a Tavus live avatar.
	// Used when Provider is "tavus".
	Tavus TavusConfig

	// Anam configures an Anam live avatar.
	// Used when Provider is "anam".
	Anam AnamConfig

	// Simli configures a Simli live avatar.
	// Used when Provider is "simli".
	Simli SimliConfig

	// LiveKit connection settings (required for live avatars)
	LiveKitURL       string
	LiveKitAPIKey    string
	LiveKitAPISecret string
}

SetupConfig contains all configuration for avatar setup. This is the unified configuration that commands should use.

type SetupMode added in v0.3.0

type SetupMode string

SetupMode indicates the type of avatar setup.

const (
	// SetupModeNone means no avatar (audio-only).
	SetupModeNone SetupMode = "none"

	// SetupModeStatic means a static image avatar.
	// The caller should configure the agent's Image options.
	SetupModeStatic SetupMode = "static"

	// SetupModeLive means a live lip-sync avatar.
	// The caller should use the returned Session for audio streaming.
	SetupModeLive SetupMode = "live"
)

type SetupResult added in v0.3.0

type SetupResult struct {
	// Mode indicates the avatar mode to use.
	Mode SetupMode

	// Session is the live avatar session (nil for static or none).
	// The caller is responsible for calling Session.Close() when done.
	Session Session

	// StaticImage contains static image configuration for the agent.
	// Only populated when Mode is SetupModeStatic.
	StaticImage *StaticImageResult
}

SetupResult contains the result of avatar setup.

func Setup added in v0.3.0

func Setup(cfg SetupConfig) (*SetupResult, error)

Setup creates the appropriate avatar based on the configuration.

For static avatars, it returns configuration to apply to agent options. For live avatars, it creates a Session that must be started and closed.

Example usage:

result, err := avatar.Setup(avatar.SetupConfig{
    Provider: "tavus",
    Tavus: avatar.TavusConfig{
        APIKey: os.Getenv("TAVUS_API_KEY"),
    },
    LiveKitURL:       os.Getenv("LIVEKIT_URL"),
    LiveKitAPIKey:    os.Getenv("LIVEKIT_API_KEY"),
    LiveKitAPISecret: os.Getenv("LIVEKIT_API_SECRET"),
})
if err != nil {
    log.Fatal(err)
}

switch result.Mode {
case avatar.SetupModeNone:
    // Audio-only, no avatar configuration needed
case avatar.SetupModeStatic:
    // Apply static image config to agent
    agentOpts.MediaMode = agent.AudioWithImage
    agentOpts.Image.H264Path = result.StaticImage.H264Path
case avatar.SetupModeLive:
    // Use live avatar session
    defer result.Session.Close(ctx)
    err := result.Session.Start(ctx, avatar.StartOptions{...})
}

type SimliConfig added in v0.3.0

type SimliConfig struct {
	// APIKey is the Simli API key.
	APIKey string

	// AvatarID is the Simli avatar to use.
	AvatarID string

	// AudioConfig configures the audio format.
	AudioConfig AudioConfig
}

SimliConfig contains configuration for the Simli avatar provider. Placeholder for future implementation.

type StartOptions

type StartOptions struct {
	// Room is the LiveKit room the agent has joined.
	// Required.
	Room *lksdk.Room

	// AgentIdentity is the identity of the agent participant.
	// The avatar will publish tracks on behalf of this identity
	// using the lk.publish_on_behalf attribute.
	// Required.
	AgentIdentity string

	// LiveKitURL is the LiveKit server URL for the avatar to connect to.
	// This should match the URL the agent connected to.
	// Required.
	LiveKitURL string

	// LiveKitAPIKey is used to generate tokens for the avatar.
	// Required.
	LiveKitAPIKey string

	// LiveKitAPISecret is used to generate tokens for the avatar.
	// Required.
	LiveKitAPISecret string

	// Callbacks configures optional event callbacks.
	Callbacks *SessionCallbacks
}

StartOptions configures avatar session startup.

func (*StartOptions) Validate

func (o *StartOptions) Validate() error

Validate checks that all required fields are set.

type StaticImageConfig added in v0.3.0

type StaticImageConfig struct {
	// H264Path is the path to a pre-encoded H.264 keyframe file.
	// This is the recommended approach - no CGO required at runtime.
	H264Path string

	// H264Data is pre-encoded H.264 keyframe data (alternative to H264Path).
	H264Data []byte

	// UseDefault uses the embedded default OmniAgent avatar.
	// If true, H264Path and H264Data are ignored.
	UseDefault bool
}

StaticImageConfig configures a static image avatar.

type StaticImageResult added in v0.3.0

type StaticImageResult struct {
	// H264Path is the path to the H.264 file (if using file).
	H264Path string

	// H264Data is the H.264 data (if using embedded data).
	H264Data []byte
}

StaticImageResult contains static image configuration to apply to agent options.

type TavusConfig added in v0.3.0

type TavusConfig struct {
	// APIKey is the Tavus API key.
	// Required.
	APIKey string

	// BaseURL is the Tavus API base URL.
	// Default: https://tavusapi.com
	BaseURL string

	// PalID is the PAL (Personalized AI Likeness) to use.
	// Default: stock avatar (provider default)
	PalID string

	// FaceID is an optional face override.
	FaceID string

	// AudioConfig configures the audio format.
	// Default: 24kHz mono PCM16 (Tavus requirement)
	AudioConfig AudioConfig
}

TavusConfig contains configuration for the Tavus avatar provider.

type TokenOptions

type TokenOptions struct {
	// APIKey is the LiveKit API key.
	// Required.
	APIKey string

	// APISecret is the LiveKit API secret.
	// Required.
	APISecret string

	// RoomName is the room the avatar will join.
	// Required.
	RoomName string

	// AvatarIdentity is the participant identity for the avatar.
	// Required.
	AvatarIdentity string

	// AvatarName is the display name for the avatar participant.
	// Optional, defaults to AvatarIdentity.
	AvatarName string

	// PublishOnBehalf is the identity of the agent participant.
	// The avatar will publish tracks that appear as if they're from this participant.
	// Required.
	PublishOnBehalf string

	// TTL is the token validity duration.
	// Default: 5 minutes
	TTL time.Duration

	// Metadata is optional participant metadata.
	Metadata string
}

TokenOptions configures avatar token generation.

Directories

Path Synopsis
Package bithuman provides bitHuman avatar integration for omni-livekit voice agents.
Package bithuman provides bitHuman avatar integration for omni-livekit voice agents.
Package heygen provides HeyGen LiveAvatar integration for omni-livekit.
Package heygen provides HeyGen LiveAvatar integration for omni-livekit.
Package tavus provides Tavus avatar integration for omni-livekit voice agents.
Package tavus provides Tavus avatar integration for omni-livekit voice agents.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL