Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrSessionNotStarted indicates that Start() was not called before // attempting an operation that requires an active session. ErrSessionNotStarted = errors.New("avatar: session not started") // ErrSessionAlreadyStarted indicates that Start() was called on a // session that is already running. ErrSessionAlreadyStarted = errors.New("avatar: session already started") // ErrAvatarJoinTimeout indicates that the avatar participant did not // join the room within the specified timeout. ErrAvatarJoinTimeout = errors.New("avatar: join timeout") // ErrAvatarTrackTimeout indicates that the avatar did not publish // the expected track (video/audio) within the timeout. ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout") // is unreachable or returned an error. ErrProviderUnavailable = errors.New("avatar: provider unavailable") // ErrProviderAuthFailed indicates that authentication with the // avatar provider failed (invalid API key, etc.). ErrProviderAuthFailed = errors.New("avatar: provider authentication failed") // ErrProviderRateLimited indicates that the avatar provider has // rate-limited the request. ErrProviderRateLimited = errors.New("avatar: provider rate limited") // ErrInvalidConfig indicates that the avatar configuration is // invalid or incomplete. ErrInvalidConfig = errors.New("avatar: invalid configuration") // ErrRPCTimeout indicates that an RPC call to the avatar timed out. ErrRPCTimeout = errors.New("avatar: RPC timeout") // ErrRPCFailed indicates that an RPC call to the avatar failed. ErrRPCFailed = errors.New("avatar: RPC failed") // ErrStreamClosed indicates that the audio stream was closed // unexpectedly. ErrStreamClosed = errors.New("avatar: stream closed") // ErrAvatarDisconnected indicates that the avatar participant // disconnected from the room unexpectedly. ErrAvatarDisconnected = errors.New("avatar: disconnected") )
Sentinel errors for avatar operations.
Functions ¶
This section is empty.
Types ¶
type AudioConfig ¶
type AudioConfig struct {
// SampleRate is the audio sample rate in Hz.
// Default: 24000 (common for avatar providers)
SampleRate int
// Channels is the number of audio channels.
// Default: 1 (mono)
Channels int
// Encoding is the audio encoding format.
// Default: "linear16" (PCM16 little-endian)
Encoding string
}
AudioConfig holds audio configuration for avatar sessions.
func DefaultAudioConfig ¶
func DefaultAudioConfig() AudioConfig
DefaultAudioConfig returns the default audio configuration.
func (AudioConfig) FrameDuration ¶
func (c AudioConfig) FrameDuration(frameSize int) int
FrameDuration returns the duration in milliseconds for the given frame size.
func (AudioConfig) FrameSize ¶
func (c AudioConfig) FrameSize(durationMs int) int
FrameSize returns the number of bytes per frame for the given duration in ms.
type AudioDestination ¶
type AudioDestination interface {
// CaptureFrame sends a PCM16 audio frame to the avatar.
//
// The frame should be little-endian PCM16 at the configured sample rate.
// Frames are typically 20ms of audio (e.g., 480 samples at 24kHz).
//
// This method may block if the underlying transport is congested.
CaptureFrame(ctx context.Context, frame []byte) error
// Flush marks the end of an audio segment (utterance).
//
// The avatar should speak all buffered audio before accepting more.
// This closes the current stream and prepares for the next utterance.
Flush(ctx context.Context) error
// ClearBuffer interrupts current playback.
//
// Use this when the user interrupts the agent. The avatar should
// stop speaking immediately and discard any buffered audio.
ClearBuffer(ctx context.Context) error
// SampleRate returns the expected input sample rate in Hz.
// Common values: 16000, 24000, 48000
SampleRate() int
// Channels returns the expected number of audio channels.
// Typically 1 (mono).
Channels() int
// Close releases resources associated with this audio destination.
// After Close(), no more frames should be sent.
Close() error
}
AudioDestination receives audio frames and forwards them to an avatar.
Implementations include:
- DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
- QueueAudioOutput: In-memory queue for testing
The typical usage flow is:
- Call CaptureFrame() for each PCM audio frame from TTS
- Call Flush() when the utterance is complete
- Call ClearBuffer() to interrupt playback (e.g., user interruption)
type Metrics ¶
type Metrics struct {
// AvatarJoinLatency is the time from Start() to avatar joining the room.
AvatarJoinLatency time.Duration
// PlaybackLatency is the time from audio send to avatar speech start.
// This is measured per utterance.
PlaybackLatency time.Duration
// Provider is the avatar provider name.
Provider string
// Timestamp is when the metrics were collected.
Timestamp time.Time
}
Metrics contains avatar performance metrics.
type PlaybackCallback ¶
type PlaybackCallback func(event PlaybackEvent)
PlaybackCallback is called when playback events occur.
type PlaybackEvent ¶
type PlaybackEvent struct {
// Type is the type of playback event.
Type PlaybackEventType
// Position is the playback position in seconds when the event occurred.
// Only meaningful for PlaybackFinished events.
Position float64
// Interrupted indicates if playback was interrupted by ClearBuffer().
// Only meaningful for PlaybackFinished events.
Interrupted bool
}
PlaybackEvent represents a playback state change from the avatar.
type PlaybackEventType ¶
type PlaybackEventType string
PlaybackEventType identifies the type of playback event.
const ( // PlaybackStarted indicates the avatar started speaking. PlaybackStarted PlaybackEventType = "started" // PlaybackFinished indicates the avatar finished speaking. // Check Position and Interrupted for details. PlaybackFinished PlaybackEventType = "finished" )
type Provider ¶
type Provider interface {
// Name returns the provider name (e.g., "heygen", "tavus", "bithuman").
Name() string
// CreateSession creates a new avatar session with the given config.
// The session is not started until Session.Start() is called.
CreateSession(cfg SessionConfig) (Session, error)
}
Provider creates avatar sessions.
Implementations of this interface handle provider-specific configuration and session creation. Each provider (HeyGen, Tavus, bitHuman, etc.) has its own Provider implementation.
Example usage:
provider, err := heygen.NewProvider(heygen.Config{
APIKey: os.Getenv("HEYGEN_API_KEY"),
AvatarID: os.Getenv("HEYGEN_AVATAR_ID"),
})
if err != nil {
return err
}
session, err := provider.CreateSession(avatar.SessionConfig{
AudioConfig: avatar.DefaultAudioConfig(),
})
type ProviderError ¶
type ProviderError struct {
Provider string // Provider name (e.g., "heygen", "tavus")
Op string // Operation that failed
Err error // Underlying error
}
ProviderError wraps an error from an avatar provider with additional context.
func NewProviderError ¶
func NewProviderError(provider, op string, err error) *ProviderError
NewProviderError creates a new ProviderError.
func (*ProviderError) Error ¶
func (e *ProviderError) Error() string
Error implements the error interface.
func (*ProviderError) Unwrap ¶
func (e *ProviderError) Unwrap() error
Unwrap returns the underlying error.
type Session ¶
type Session interface {
// Identity returns the participant identity of the avatar worker.
// This is the identity that appears in the room and publishes video.
Identity() string
// Provider returns the provider name (e.g., "heygen", "tavus", "bithuman").
Provider() string
// Start initializes the avatar session with platform-specific options.
//
// The opts parameter is platform-specific. For LiveKit integration,
// pass a *LiveKitStartOptions. Other platforms may define their own
// options type.
//
// This method should:
// 1. Create a session with the avatar provider's API
// 2. Generate credentials for the avatar to join the room
// 3. Configure the audio output to stream to the avatar
//
// The avatar will join the room asynchronously. Use WaitForJoin()
// to wait for the avatar to be ready.
Start(ctx context.Context, opts any) error
// WaitForJoin blocks until the avatar participant joins the room
// and publishes the expected tracks (typically video).
//
// Returns an error if the timeout is exceeded or the context is cancelled.
WaitForJoin(ctx context.Context, timeout time.Duration) error
// AudioOutput returns the audio destination for streaming TTS audio
// to the avatar. Returns nil if the session is not started.
AudioOutput() AudioDestination
// Close disconnects the avatar and cleans up resources.
//
// This method should:
// 1. End the session with the avatar provider
// 2. Remove the avatar participant from the room
// 3. Clean up any registered handlers
Close(ctx context.Context) error
// SetCallbacks registers event callbacks for the session.
SetCallbacks(callbacks *SessionCallbacks)
}
Session manages a lip-sync avatar that publishes video to a room.
Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.
The typical flow is:
- Create a Session with provider-specific configuration
- Call Start() to initialize the avatar and connect it to the room
- Call WaitForJoin() to wait for the avatar to be ready
- Use AudioOutput() to stream TTS audio to the avatar
- Call Close() when done to clean up resources
type SessionCallbacks ¶
type SessionCallbacks struct {
// OnAvatarJoined is called when the avatar participant joins the room.
OnAvatarJoined func(identity string)
// OnAvatarLeft is called when the avatar participant leaves the room.
OnAvatarLeft func(identity string)
// OnPlaybackStarted is called when the avatar starts speaking.
OnPlaybackStarted func()
// OnPlaybackFinished is called when the avatar finishes speaking.
// The position is the playback position in seconds when stopped.
// The interrupted flag indicates if playback was interrupted by
// a ClearBuffer() call.
OnPlaybackFinished func(position float64, interrupted bool)
// OnError is called when an error occurs.
// This is for non-fatal errors that don't cause the session to fail.
OnError func(err error)
}
SessionCallbacks defines optional event callbacks for avatar sessions.
type SessionConfig ¶
type SessionConfig struct {
// AudioConfig specifies the audio format requirements.
// Default: 24kHz mono PCM16
AudioConfig AudioConfig
// Extensions holds provider-specific configuration.
// Keys and values depend on the provider.
Extensions map[string]any
}
SessionConfig contains configuration for creating an avatar session.