live

package
v0.46.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 21, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package live exposes the low-level Voice Agent realtime-protocol types.

This is the public-API surface for building custom realtime providers (Gemini Live, OpenAI Realtime, or a third-party WebSocket model) and for libraries that drive a SpeechKit session directly without going through the higher-level [Service] in the parent package.

Most embedders use the parent [voiceagent] package or github.com/kombifyio/SpeechKit/pkg/speechkit/agentkit instead; reach for this package only when you need to plug in your own provider or read the raw LiveMessage stream.

Index

Constants

View Source
const DefaultOpenAIRealtimeModel = defaultOpenAIRealtimeModel

DefaultOpenAIRealtimeModel is the public runtime default for OpenAI-backed Voice Agent sessions.

Variables

This section is empty.

Functions

func RenderHostInstructionUpdate

func RenderHostInstructionUpdate(cfg LiveConfig) string

RenderHostInstructionUpdate turns a workflow step change into a provider text update. Providers that implement LiveInstructionUpdater receive the structured LiveConfig instead.

Types

type ActivityDetectionPolicy

type ActivityDetectionPolicy struct {
	Automatic         bool
	StartSensitivity  StartSensitivity
	EndSensitivity    EndSensitivity
	PrefixPaddingMs   int32
	SilenceDurationMs int32
	ActivityHandling  ActivityHandling
	TurnCoverage      TurnCoverage
}

ActivityDetectionPolicy defines server-side VAD/session turn behavior.

type ActivityHandling

type ActivityHandling string

ActivityHandling controls what Gemini Live should do when new activity starts.

const (
	ActivityHandlingUnspecified               ActivityHandling = ""
	ActivityHandlingNoInterrupt               ActivityHandling = "no_interrupt"
	ActivityHandlingStartOfActivityInterrupts ActivityHandling = "start_of_activity_interrupts"
)

type Callbacks

type Callbacks struct {
	OnStateChange          func(state State)
	OnAudio                func(audio []byte) // Audio chunk to play
	OnText                 func(text string)  // Text for display (speech bubble)
	OnError                func(err error)
	OnInputTranscript      func(text string, done bool) // User speech transcribed
	OnOutputTranscript     func(text string, done bool) // Model speech transcribed
	OnToolCall             func(call ToolCall)
	OnToolCallCancellation func(ids []string)
	OnInterrupted          func() // User interrupted model (barge-in)
	OnSessionEnd           func() // Session ended (error, GoAway failure, or deactivation)
}

Callbacks are event handlers for UI integration with a low-level Session.

This is the rich Callbacks struct used by the low-level Session runtime. The higher-level [voiceagent.Callbacks] in the parent package carries only the three most-common handlers (OnAudio, OnText, OnError) and is the right shape for the embedded [voiceagent.Service]; reach for this struct only when wiring a custom realtime host.

type ContextCompressionPolicy

type ContextCompressionPolicy struct {
	Enabled       bool
	TriggerTokens int64
	TargetTokens  int64
}

ContextCompressionPolicy defines how the live API should compress long sessions.

type DeepgramLive added in v0.46.0

type DeepgramLive struct {
	// Optional overrides; zero values fall back to the package defaults.
	ListenModel   string
	SpeakModel    string
	ThinkProvider string
	ThinkModel    string

	// ThinkEndpointURL + ThinkAPIKey switch the think leg to a bring-your-own
	// LLM deployment. When ThinkEndpointURL is set, the Settings message carries
	// an agent.think.endpoint block so Deepgram calls the operator's own LLM
	// instead of a Deepgram-managed model; ThinkAPIKey (when set) is sent as an
	// "Authorization: Bearer <key>" header on that endpoint. Leave both empty to
	// use Deepgram's managed LLM for ThinkProvider/ThinkModel (no key needed).
	ThinkEndpointURL string
	ThinkAPIKey      string
	// contains filtered or unexported fields
}

DeepgramLive implements LiveProvider against the Deepgram Voice Agent API (WebSocket). It mirrors GeminiLive/OpenAILive's surface so callers don't need to know which backend is active.

The think (LLM) leg is configurable: Deepgram drives the LLM server-side, so ThinkProvider/ThinkModel select which model reasons over the transcript. Defaults target a widely-available option; the wiring layer overrides them from deployment config. Listen is always Deepgram Nova-3 and speak is always a Deepgram Aura-2 voice — that is the point of routing a session here.

func NewDeepgramLive added in v0.46.0

func NewDeepgramLive() *DeepgramLive

NewDeepgramLive returns a fresh Deepgram Voice Agent provider.

func (*DeepgramLive) Close added in v0.46.0

func (p *DeepgramLive) Close() error

Close terminates the WebSocket and stops the keepalive loop. Idempotent.

func (*DeepgramLive) ConfigureThink added in v0.46.0

func (p *DeepgramLive) ConfigureThink(provider, model, endpointURL, apiKey string)

ConfigureThink applies the deployment's think-LLM selection to the provider. Non-empty provider/model override the package defaults; empty values keep the Deepgram-managed default. endpointURL/apiKey select a bring-your-own think LLM (see ThinkEndpointURL/ThinkAPIKey) and are cleared when empty. Both Targets call this from their Voice Agent wiring with values resolved from config.

func (*DeepgramLive) Connect added in v0.46.0

func (p *DeepgramLive) Connect(ctx context.Context, cfg LiveConfig) error

Connect dials the Deepgram Voice Agent WebSocket and sends the initial Settings message describing listen/think/speak and the audio formats. The SettingsApplied acknowledgement is consumed asynchronously by Receive().

func (*DeepgramLive) Name added in v0.46.0

func (p *DeepgramLive) Name() string

Name identifies the provider in Voice Agent logs.

func (*DeepgramLive) Receive added in v0.46.0

func (p *DeepgramLive) Receive(ctx context.Context) (*LiveMessage, error)

Receive translates the next server frame into a LiveMessage. Binary frames are agent audio; text frames are JSON control events. Events that don't map to LiveMessage fields are swallowed and the loop fetches the next frame.

func (*DeepgramLive) SendAudio added in v0.46.0

func (p *DeepgramLive) SendAudio(chunk []byte) error

SendAudio forwards a 16 kHz PCM16 mic chunk as a binary frame. Deepgram accepts the mic rate directly (declared in the Settings input config), so no resample is needed. Empty chunks are no-ops.

func (*DeepgramLive) SendAudioStreamEnd added in v0.46.0

func (p *DeepgramLive) SendAudioStreamEnd() error

SendAudioStreamEnd is a no-op for Deepgram: the Voice Agent performs its own endpointing/turn-detection server-side and responds when the user stops speaking. There is no client-side commit in the protocol.

func (*DeepgramLive) SendText added in v0.46.0

func (p *DeepgramLive) SendText(text string) error

SendText injects a text user turn (e.g. an idle reminder) and lets the agent respond. Empty text is a no-op.

func (*DeepgramLive) SendToolResponse added in v0.46.0

func (p *DeepgramLive) SendToolResponse(response ToolResponse) error

SendToolResponse returns a host-side function result to the agent.

func (*DeepgramLive) UpdateInstructions added in v0.46.0

func (p *DeepgramLive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error

UpdateInstructions refreshes the agent's system prompt without a reconnect. Implements LiveInstructionUpdater.

type EndSensitivity

type EndSensitivity string

EndSensitivity controls how aggressively automatic activity detection commits speech end.

const (
	EndSensitivityLow    EndSensitivity = "low"
	EndSensitivityMedium EndSensitivity = "medium"
	EndSensitivityHigh   EndSensitivity = "high"
)

type GeminiLive added in v0.46.0

type GeminiLive struct {
	// contains filtered or unexported fields
}

GeminiLive implements LiveProvider using the Google GenAI Live API.

func NewGeminiLive added in v0.46.0

func NewGeminiLive() *GeminiLive

NewGeminiLive creates a Gemini Live provider.

func (*GeminiLive) Close added in v0.46.0

func (g *GeminiLive) Close() error

func (*GeminiLive) Connect added in v0.46.0

func (g *GeminiLive) Connect(ctx context.Context, cfg LiveConfig) error

func (*GeminiLive) Name added in v0.46.0

func (g *GeminiLive) Name() string

func (*GeminiLive) Receive added in v0.46.0

func (g *GeminiLive) Receive(ctx context.Context) (*LiveMessage, error)

func (*GeminiLive) Reconnect added in v0.46.0

func (g *GeminiLive) Reconnect(ctx context.Context) error

Reconnect re-establishes the session using the stored resumption handle. If the handle has expired (TTL) or been cleared, a fresh session is opened.

func (*GeminiLive) SendAudio added in v0.46.0

func (g *GeminiLive) SendAudio(chunk []byte) error

func (*GeminiLive) SendAudioStreamEnd added in v0.46.0

func (g *GeminiLive) SendAudioStreamEnd() error

func (*GeminiLive) SendText added in v0.46.0

func (g *GeminiLive) SendText(text string) error

func (*GeminiLive) SendToolResponse added in v0.46.0

func (g *GeminiLive) SendToolResponse(response ToolResponse) error

type IdleConfig

type IdleConfig struct {
	ReminderAfter   time.Duration // Default: 5 minutes
	DeactivateAfter time.Duration // Default: 15 minutes
}

IdleConfig configures the idle timer behavior.

func DefaultIdleConfig

func DefaultIdleConfig() IdleConfig

DefaultIdleConfig returns sensible defaults.

type IdleTimer

type IdleTimer struct {
	// contains filtered or unexported fields
}

IdleTimer manages reminder and auto-deactivation for Voice Agent.

func NewIdleTimer

func NewIdleTimer(cfg IdleConfig, session *Session) *IdleTimer

NewIdleTimer creates an idle timer bound to a session.

func (*IdleTimer) Reset

func (t *IdleTimer) Reset()

Reset restarts the idle countdown. Call after each user interaction.

func (*IdleTimer) Stop

func (t *IdleTimer) Stop()

Stop cancels all timers.

type LiveConfig

type LiveConfig struct {
	Model string // e.g. "gemini-3.1-flash-live-preview"
	// FallbackModel is tried when the primary Model's Connect fails. Empty
	// disables the fallback. Typical pairing in 2026: a preview model as
	// Model + the last GA model as FallbackModel, so transient preview
	// outages don't take down a Voice Agent deployment.
	FallbackModel    string
	APIKey           string
	Voice            string // Voice name
	FrameworkPrompt  string
	RefinementPrompt string
	VocabularyHint   string
	Locale           string
	// Region is the Google Cloud region the caller's API key / project is
	// pinned to (e.g. "europe-west3", "us-central1"). Used by providers that
	// support regional endpoints. For Gemini Live (as of May 2026) the API
	// exposes a single global WebSocket endpoint, so this field does NOT
	// redirect traffic — it is logged at connect time for compliance evidence
	// (byok.key_updated audit event) and reserved for future regional routing
	// once Google publishes per-region hostnames. Data residency is controlled
	// at the Google Cloud project level; both the project region AND this field
	// must agree for audit records to be accurate.
	// See docs/compliance/byok-gemini-region-pinning.md.
	Region          string
	Policies        LivePolicies
	Tools           []ToolDefinition
	Workflow        *WorkflowConfig
	Speaker         speaker.Options
	Options         provideropts.Values
	ProviderOptions provideropts.Values
}

LiveConfig configures a real-time session.

type LiveInstructionUpdater

type LiveInstructionUpdater interface {
	UpdateInstructions(ctx context.Context, cfg LiveConfig) error
}

LiveInstructionUpdater is optionally implemented by providers that can refresh active host instructions without treating the update as a user turn.

type LiveMessage

type LiveMessage struct {
	Audio []byte // PCM audio chunk (24kHz 16-bit mono)
	Text  string // Text transcript (may be partial or empty)
	Done  bool   // True when the model's turn is complete

	// Transcription fields (populated when transcription is enabled).
	InputTranscript        string  // User speech transcribed by server
	InputTranscriptDone    bool    // True when input transcription segment is final
	InputSpeakerLabel      string  // Optional diarized speaker label for input transcript segments
	InputPersonID          string  // Optional known person id for input transcript segments
	InputDisplayName       string  // Optional known display name for input transcript segments
	InputSpeakerConfidence float64 // Optional speaker-label confidence
	OutputTranscript       string  // Model speech transcribed by server
	OutputTranscriptDone   bool    // True when output transcription segment is final

	ToolCalls               []ToolCall
	ToolCallCancellationIDs []string
	Interrupted             bool // True when user interrupted model (barge-in)
	GoAway                  bool // True when server signals imminent session end
}

LiveMessage is a message received from the real-time model.

type LivePolicies

type LivePolicies struct {
	EnableInputAudioTranscription  bool
	EnableOutputAudioTranscription bool
	EnableAffectiveDialog          bool
	Thinking                       ThinkingPolicy
	ContextCompression             ContextCompressionPolicy
	ActivityDetection              ActivityDetectionPolicy
}

LivePolicies configures Google Live API features that shape Voice Agent behavior.

type LiveProvider

type LiveProvider interface {
	// Connect establishes a WebSocket session to the real-time model.
	Connect(ctx context.Context, cfg LiveConfig) error

	// SendAudio streams PCM audio chunks to the model.
	// Format: 16-bit signed int, little-endian, mono, 16kHz.
	SendAudio(chunk []byte) error

	// SendAudioStreamEnd signals that microphone input for the current turn ended.
	SendAudioStreamEnd() error

	// Receive blocks until the next server message arrives.
	// Returns audio chunks and/or text from the model.
	Receive(ctx context.Context) (*LiveMessage, error)

	// SendText injects a text prompt into the session (for idle reminders).
	SendText(text string) error

	// SendToolResponse sends the result of a host-side tool invocation back to the model.
	SendToolResponse(response ToolResponse) error

	// Close terminates the WebSocket session.
	Close() error

	// Name returns the provider identifier.
	Name() string
}

LiveProvider abstracts a real-time audio-to-audio model connection.

type LiveReconnector

type LiveReconnector interface {
	Reconnect(ctx context.Context) error
}

LiveReconnector is an optional interface for providers that support session reconnection.

type OpenAILive added in v0.46.0

type OpenAILive struct {
	// contains filtered or unexported fields
}

OpenAILive implements LiveProvider against the OpenAI Realtime API (WebSocket). It mirrors GeminiLive's surface so callers don't need to know which backend is active.

func NewOpenAILive added in v0.46.0

func NewOpenAILive() *OpenAILive

NewOpenAILive returns a fresh OpenAI Realtime provider.

func (*OpenAILive) Close added in v0.46.0

func (p *OpenAILive) Close() error

Close terminates the WebSocket. Idempotent.

func (*OpenAILive) Connect added in v0.46.0

func (p *OpenAILive) Connect(ctx context.Context, cfg LiveConfig) error

Connect dials the OpenAI Realtime WebSocket, sends the configured instructions/voice/tools as a session.update, and waits for the session.updated acknowledgement before returning.

func (*OpenAILive) Name added in v0.46.0

func (p *OpenAILive) Name() string

Name identifies the provider in Voice Agent logs.

func (*OpenAILive) Receive added in v0.46.0

func (p *OpenAILive) Receive(ctx context.Context) (*LiveMessage, error)

Receive translates the next server event into a LiveMessage. Server events that don't map to LiveMessage fields (session.created/updated, rate-limit telemetry, etc.) are swallowed and the loop fetches the next frame so callers see an aligned event stream.

func (*OpenAILive) SendAudio added in v0.46.0

func (p *OpenAILive) SendAudio(chunk []byte) error

SendAudio resamples a 16 kHz mic chunk to 24 kHz and forwards it as a base64-encoded input_audio_buffer.append event. Empty chunks are no-ops.

func (*OpenAILive) SendAudioStreamEnd added in v0.46.0

func (p *OpenAILive) SendAudioStreamEnd() error

SendAudioStreamEnd flushes the input audio buffer and triggers a model response. With server VAD enabled OpenAI commits the buffer automatically at end-of-speech; this explicit commit covers push-to-talk style turns where the kernel decides when the user stopped speaking.

func (*OpenAILive) SendText added in v0.46.0

func (p *OpenAILive) SendText(text string) error

SendText injects a text-only user turn and triggers a response.

func (*OpenAILive) SendToolResponse added in v0.46.0

func (p *OpenAILive) SendToolResponse(response ToolResponse) error

SendToolResponse delivers the host-side tool result back to the model and triggers a follow-up response.

func (*OpenAILive) UpdateInstructions added in v0.46.0

func (p *OpenAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error

UpdateInstructions sends a fresh session.update with new instructions/tools. Implements LiveInstructionUpdater so the kernel can refresh persona prompts without forcing a reconnect.

type ResolvedLiveOptions added in v0.45.0

type ResolvedLiveOptions struct {
	Locale              string
	Voice               string
	Keyterms            []string
	TurnDetection       bool
	TurnDetectionSource provideropts.ValueSource
	EndpointingMs       int
	EndpointingSource   provideropts.ValueSource
	Effective           provideropts.EffectiveOptions
}

func ResolveLiveOptions added in v0.45.0

func ResolveLiveOptions(provider, profileID string, cfg LiveConfig, providerDefaults, providerOverrides provideropts.Values) ResolvedLiveOptions

func (ResolvedLiveOptions) HasEndpointingOverride added in v0.45.0

func (r ResolvedLiveOptions) HasEndpointingOverride() bool

func (ResolvedLiveOptions) HasTurnDetectionOverride added in v0.45.0

func (r ResolvedLiveOptions) HasTurnDetectionOverride() bool

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session manages a Voice Agent conversation.

func NewSession

func NewSession(provider LiveProvider, callbacks Callbacks) *Session

NewSession creates a Voice Agent session with the given provider.

func (*Session) AdvanceWorkflowStep

func (s *Session) AdvanceWorkflowStep(ctx context.Context, reason string) error

AdvanceWorkflowStep moves a configured local workflow to its next step and updates the active provider instructions. It returns nil when no workflow is active or the workflow has already completed.

func (*Session) CurrentState

func (s *Session) CurrentState() State

State returns the current session state.

func (*Session) EndAudioStream

func (s *Session) EndAudioStream() error

EndAudioStream tells the live provider that the current microphone stream ended.

func (*Session) ProviderName

func (s *Session) ProviderName() string

func (*Session) SendAudio

func (s *Session) SendAudio(chunk []byte) error

SendAudio forwards a PCM audio chunk to the real-time model.

func (*Session) SendText

func (s *Session) SendText(text string) error

SendText injects a user text turn into the live session.

func (*Session) SendToolResponse

func (s *Session) SendToolResponse(response ToolResponse) error

SendToolResponse forwards the result of a host-side tool invocation to the model.

func (*Session) Start

func (s *Session) Start(ctx context.Context, cfg LiveConfig, idleCfg IdleConfig) error

Start activates the Voice Agent session.

func (*Session) Stop

func (s *Session) Stop()

Stop deactivates the Voice Agent session.

type StartSensitivity

type StartSensitivity string

StartSensitivity controls how aggressively automatic activity detection commits speech start.

const (
	StartSensitivityLow    StartSensitivity = "low"
	StartSensitivityMedium StartSensitivity = "medium"
	StartSensitivityHigh   StartSensitivity = "high"
)

type State

type State string

State represents the current state of a Voice Agent session.

const (
	StateInactive     State = "inactive"
	StateConnecting   State = "connecting"
	StateListening    State = "listening"
	StateProcessing   State = "processing"
	StateSpeaking     State = "speaking"
	StateRecovering   State = "recovering"
	StateDeactivating State = "deactivating"
)

type ThinkingLevel

type ThinkingLevel string

ThinkingLevel controls how much deliberate reasoning Gemini Live should spend.

const (
	ThinkingLevelOff    ThinkingLevel = "off"
	ThinkingLevelLow    ThinkingLevel = "low"
	ThinkingLevelMedium ThinkingLevel = "medium"
	ThinkingLevelHigh   ThinkingLevel = "high"
)

type ThinkingPolicy

type ThinkingPolicy struct {
	Enabled         bool
	IncludeThoughts bool
	ThinkingBudget  int32
	ThinkingLevel   ThinkingLevel
}

ThinkingPolicy defines optional Gemini Live thinking behavior.

type ToolBehavior

type ToolBehavior string

ToolBehavior controls whether the model waits for a tool result.

const (
	ToolBehaviorUnspecified ToolBehavior = ""
	ToolBehaviorBlocking    ToolBehavior = "blocking"
	ToolBehaviorNonBlocking ToolBehavior = "non_blocking"
)

type ToolCall

type ToolCall struct {
	ID   string
	Name string
	Args map[string]any
}

ToolCall is a host-side action request emitted by the Voice Agent runtime.

type ToolDefinition

type ToolDefinition struct {
	Name                 string
	Description          string
	ParametersJSONSchema map[string]any
	ResponseJSONSchema   map[string]any
	Behavior             ToolBehavior
}

ToolDefinition exposes a host-side action the Voice Agent may call.

type ToolResponse

type ToolResponse struct {
	ID       string
	Name     string
	Response map[string]any

	Scheduling   ToolResponseScheduling
	WillContinue *bool
}

ToolResponse resolves a previously emitted tool call.

type ToolResponseScheduling

type ToolResponseScheduling string

ToolResponseScheduling controls how a non-blocking tool result is reintroduced into the conversation.

const (
	ToolResponseSchedulingUnspecified ToolResponseScheduling = ""
	ToolResponseSchedulingSilent      ToolResponseScheduling = "silent"
	ToolResponseSchedulingWhenIdle    ToolResponseScheduling = "when_idle"
	ToolResponseSchedulingInterrupt   ToolResponseScheduling = "interrupt"
)

type TurnCoverage

type TurnCoverage string

TurnCoverage controls how the live API builds a user turn from incoming activity.

const (
	TurnCoverageUnspecified               TurnCoverage = ""
	TurnCoverageTurnIncludesOnlyActivity  TurnCoverage = "turn_includes_only_activity"
	TurnCoverageTurnIncludesAllInput      TurnCoverage = "turn_includes_all_input"
	TurnCoverageTurnIncludesAudioActivity TurnCoverage = "turn_includes_audio_activity"
)

type WorkflowConfig

type WorkflowConfig struct {
	SequenceID string
	Completion string
	MaxTurns   int
	BasePrompt string
	Steps      []WorkflowStep
	// InitialStep selects the first active step. Out-of-range values fall
	// back to zero.
	InitialStep int
}

WorkflowConfig describes a deterministic, step-based Voice Agent behavior sequence. Durations are intentionally expressed as turns instead of wall clock time so tests and local installs can exercise long moderation flows quickly.

type WorkflowStep

type WorkflowStep struct {
	ID           string
	Instruction  string
	ExitCriteria string
	RequireTools []string
	MaxTurns     int
}

WorkflowStep describes one stage in a Voice Agent workflow.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL