avatar

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 12, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrSessionNotStarted indicates that Start() was not called before
	// attempting an operation that requires an active session.
	ErrSessionNotStarted = errors.New("avatar: session not started")

	// ErrSessionAlreadyStarted indicates that Start() was called on a
	// session that is already running.
	ErrSessionAlreadyStarted = errors.New("avatar: session already started")

	// ErrAvatarJoinTimeout indicates that the avatar participant did not
	// join the room within the specified timeout.
	ErrAvatarJoinTimeout = errors.New("avatar: join timeout")

	// ErrAvatarTrackTimeout indicates that the avatar did not publish
	// the expected track (video/audio) within the timeout.
	ErrAvatarTrackTimeout = errors.New("avatar: track publish timeout")

	// ErrProviderUnavailable indicates that the avatar provider API
	// is unreachable or returned an error.
	ErrProviderUnavailable = errors.New("avatar: provider unavailable")

	// ErrProviderAuthFailed indicates that authentication with the
	// avatar provider failed (invalid API key, etc.).
	ErrProviderAuthFailed = errors.New("avatar: provider authentication failed")

	// ErrProviderRateLimited indicates that the avatar provider has
	// rate-limited the request.
	ErrProviderRateLimited = errors.New("avatar: provider rate limited")

	// ErrInvalidConfig indicates that the avatar configuration is
	// invalid or incomplete.
	ErrInvalidConfig = errors.New("avatar: invalid configuration")

	// ErrRPCTimeout indicates that an RPC call to the avatar timed out.
	ErrRPCTimeout = errors.New("avatar: RPC timeout")

	// ErrRPCFailed indicates that an RPC call to the avatar failed.
	ErrRPCFailed = errors.New("avatar: RPC failed")

	// ErrStreamClosed indicates that the audio stream was closed
	// unexpectedly.
	ErrStreamClosed = errors.New("avatar: stream closed")

	// ErrAvatarDisconnected indicates that the avatar participant
	// disconnected from the room unexpectedly.
	ErrAvatarDisconnected = errors.New("avatar: disconnected")
)

Sentinel errors for avatar operations.

Functions

This section is empty.

Types

type AudioConfig

type AudioConfig struct {
	// SampleRate is the audio sample rate in Hz.
	// Default: 24000 (common for avatar providers)
	SampleRate int

	// Channels is the number of audio channels.
	// Default: 1 (mono)
	Channels int

	// Encoding is the audio encoding format.
	// Default: "linear16" (PCM16 little-endian)
	Encoding string
}

AudioConfig holds audio configuration for avatar sessions.

func DefaultAudioConfig

func DefaultAudioConfig() AudioConfig

DefaultAudioConfig returns the default audio configuration.

func (AudioConfig) FrameDuration

func (c AudioConfig) FrameDuration(frameSize int) int

FrameDuration returns the duration in milliseconds for the given frame size.

func (AudioConfig) FrameSize

func (c AudioConfig) FrameSize(durationMs int) int

FrameSize returns the number of bytes per frame for the given duration in ms.

type AudioDestination

type AudioDestination interface {
	// CaptureFrame sends a PCM16 audio frame to the avatar.
	//
	// The frame should be little-endian PCM16 at the configured sample rate.
	// Frames are typically 20ms of audio (e.g., 480 samples at 24kHz).
	//
	// This method may block if the underlying transport is congested.
	CaptureFrame(ctx context.Context, frame []byte) error

	// Flush marks the end of an audio segment (utterance).
	//
	// The avatar should speak all buffered audio before accepting more.
	// This closes the current stream and prepares for the next utterance.
	Flush(ctx context.Context) error

	// ClearBuffer interrupts current playback.
	//
	// Use this when the user interrupts the agent. The avatar should
	// stop speaking immediately and discard any buffered audio.
	ClearBuffer(ctx context.Context) error

	// SampleRate returns the expected input sample rate in Hz.
	// Common values: 16000, 24000, 48000
	SampleRate() int

	// Channels returns the expected number of audio channels.
	// Typically 1 (mono).
	Channels() int

	// Close releases resources associated with this audio destination.
	// After Close(), no more frames should be sent.
	Close() error
}

AudioDestination receives audio frames and forwards them to an avatar.

Implementations include:

  • DataStreamAudioOutput: Streams audio to a remote avatar via LiveKit ByteStream
  • QueueAudioOutput: In-memory queue for testing

The typical usage flow is:

  1. Call CaptureFrame() for each PCM audio frame from TTS
  2. Call Flush() when the utterance is complete
  3. Call ClearBuffer() to interrupt playback (e.g., user interruption)

type Metrics

type Metrics struct {
	// AvatarJoinLatency is the time from Start() to avatar joining the room.
	AvatarJoinLatency time.Duration

	// PlaybackLatency is the time from audio send to avatar speech start.
	// This is measured per utterance.
	PlaybackLatency time.Duration

	// Provider is the avatar provider name.
	Provider string

	// Timestamp is when the metrics were collected.
	Timestamp time.Time
}

Metrics contains avatar performance metrics.

type PlaybackCallback

type PlaybackCallback func(event PlaybackEvent)

PlaybackCallback is called when playback events occur.

type PlaybackEvent

type PlaybackEvent struct {
	// Type is the type of playback event.
	Type PlaybackEventType

	// Position is the playback position in seconds when the event occurred.
	// Only meaningful for PlaybackFinished events.
	Position float64

	// Interrupted indicates if playback was interrupted by ClearBuffer().
	// Only meaningful for PlaybackFinished events.
	Interrupted bool
}

PlaybackEvent represents a playback state change from the avatar.

type PlaybackEventType

type PlaybackEventType string

PlaybackEventType identifies the type of playback event.

const (
	// PlaybackStarted indicates the avatar started speaking.
	PlaybackStarted PlaybackEventType = "started"

	// PlaybackFinished indicates the avatar finished speaking.
	// Check Position and Interrupted for details.
	PlaybackFinished PlaybackEventType = "finished"
)

type Provider

type Provider interface {
	// Name returns the provider name (e.g., "heygen", "tavus", "bithuman").
	Name() string

	// CreateSession creates a new avatar session with the given config.
	// The session is not started until Session.Start() is called.
	CreateSession(cfg SessionConfig) (Session, error)
}

Provider creates avatar sessions.

Implementations of this interface handle provider-specific configuration and session creation. Each provider (HeyGen, Tavus, bitHuman, etc.) has its own Provider implementation.

Example usage:

provider, err := heygen.NewProvider(heygen.Config{
    APIKey:   os.Getenv("HEYGEN_API_KEY"),
    AvatarID: os.Getenv("HEYGEN_AVATAR_ID"),
})
if err != nil {
    return err
}

session, err := provider.CreateSession(avatar.SessionConfig{
    AudioConfig: avatar.DefaultAudioConfig(),
})

type ProviderError

type ProviderError struct {
	Provider string // Provider name (e.g., "heygen", "tavus")
	Op       string // Operation that failed
	Err      error  // Underlying error
}

ProviderError wraps an error from an avatar provider with additional context.

func NewProviderError

func NewProviderError(provider, op string, err error) *ProviderError

NewProviderError creates a new ProviderError.

func (*ProviderError) Error

func (e *ProviderError) Error() string

Error implements the error interface.

func (*ProviderError) Unwrap

func (e *ProviderError) Unwrap() error

Unwrap returns the underlying error.

type Session

type Session interface {
	// Identity returns the participant identity of the avatar worker.
	// This is the identity that appears in the room and publishes video.
	Identity() string

	// Provider returns the provider name (e.g., "heygen", "tavus", "bithuman").
	Provider() string

	// Start initializes the avatar session with platform-specific options.
	//
	// The opts parameter is platform-specific. For LiveKit integration,
	// pass a *LiveKitStartOptions. Other platforms may define their own
	// options type.
	//
	// This method should:
	//  1. Create a session with the avatar provider's API
	//  2. Generate credentials for the avatar to join the room
	//  3. Configure the audio output to stream to the avatar
	//
	// The avatar will join the room asynchronously. Use WaitForJoin()
	// to wait for the avatar to be ready.
	Start(ctx context.Context, opts any) error

	// WaitForJoin blocks until the avatar participant joins the room
	// and publishes the expected tracks (typically video).
	//
	// Returns an error if the timeout is exceeded or the context is cancelled.
	WaitForJoin(ctx context.Context, timeout time.Duration) error

	// AudioOutput returns the audio destination for streaming TTS audio
	// to the avatar. Returns nil if the session is not started.
	AudioOutput() AudioDestination

	// Close disconnects the avatar and cleans up resources.
	//
	// This method should:
	//  1. End the session with the avatar provider
	//  2. Remove the avatar participant from the room
	//  3. Clean up any registered handlers
	Close(ctx context.Context) error

	// SetCallbacks registers event callbacks for the session.
	SetCallbacks(callbacks *SessionCallbacks)
}

Session manages a lip-sync avatar that publishes video to a room.

Implementations of this interface handle provider-specific API calls to create avatar sessions and manage their lifecycle.

The typical flow is:

  1. Create a Session with provider-specific configuration
  2. Call Start() to initialize the avatar and connect it to the room
  3. Call WaitForJoin() to wait for the avatar to be ready
  4. Use AudioOutput() to stream TTS audio to the avatar
  5. Call Close() when done to clean up resources

type SessionCallbacks

type SessionCallbacks struct {
	// OnAvatarJoined is called when the avatar participant joins the room.
	OnAvatarJoined func(identity string)

	// OnAvatarLeft is called when the avatar participant leaves the room.
	OnAvatarLeft func(identity string)

	// OnPlaybackStarted is called when the avatar starts speaking.
	OnPlaybackStarted func()

	// OnPlaybackFinished is called when the avatar finishes speaking.
	// The position is the playback position in seconds when stopped.
	// The interrupted flag indicates if playback was interrupted by
	// a ClearBuffer() call.
	OnPlaybackFinished func(position float64, interrupted bool)

	// OnError is called when an error occurs.
	// This is for non-fatal errors that don't cause the session to fail.
	OnError func(err error)
}

SessionCallbacks defines optional event callbacks for avatar sessions.

type SessionConfig

type SessionConfig struct {
	// AudioConfig specifies the audio format requirements.
	// Default: 24kHz mono PCM16
	AudioConfig AudioConfig

	// Extensions holds provider-specific configuration.
	// Keys and values depend on the provider.
	Extensions map[string]any
}

SessionConfig contains configuration for creating an avatar session.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL