live

package
v0.48.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 28, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package live exposes the low-level Voice Agent realtime-protocol types.

This is the public-API surface for building custom realtime providers (Gemini Live, OpenAI Realtime, or a third-party WebSocket model) and for libraries that drive a SpeechKit session directly without going through the higher-level [Service] in the parent package.

Most embedders use the parent [voiceagent] package or github.com/kombifyio/SpeechKit/pkg/speechkit/agentkit instead; reach for this package only when you need to plug in your own provider or read the raw LiveMessage stream.

Index

Constants

View Source
const DefaultOpenAIRealtimeModel = defaultOpenAIRealtimeModel

DefaultOpenAIRealtimeModel is the public runtime default for OpenAI-backed Voice Agent sessions.

Variables

View Source
var ErrNoMatchingProvider = errors.New("speechkit live: no provider satisfies intent")
View Source
var ErrUnknownLiveProvider = errors.New("speechkit live: unknown provider")

Functions

func NewProviderForConfig added in v0.47.0

func NewProviderForConfig(cfg LiveConfig) (LiveProvider, LiveConfig, error)

NewProviderForConfig normalizes a LiveConfig and returns the matching provider. It is the most convenient entrypoint for embedders that expose a provider/profile/model picker to users.

func NormalizeProviderID added in v0.47.0

func NormalizeProviderID(providerOrProfile string) string

NormalizeProviderID maps common provider aliases and public profile ids to the canonical provider id used by ProviderDescriptor.Provider.

func RenderHostInstructionUpdate

func RenderHostInstructionUpdate(cfg LiveConfig) string

RenderHostInstructionUpdate turns a workflow step change into a provider text update. Providers that implement LiveInstructionUpdater receive the structured LiveConfig instead.

Types

type ActivityDetectionPolicy

type ActivityDetectionPolicy struct {
	Automatic         bool
	StartSensitivity  StartSensitivity
	EndSensitivity    EndSensitivity
	PrefixPaddingMs   int32
	SilenceDurationMs int32
	ActivityHandling  ActivityHandling
	TurnCoverage      TurnCoverage
}

ActivityDetectionPolicy defines server-side VAD/session turn behavior.

type ActivityHandling

type ActivityHandling string

ActivityHandling controls what Gemini Live should do when new activity starts.

const (
	ActivityHandlingUnspecified               ActivityHandling = ""
	ActivityHandlingNoInterrupt               ActivityHandling = "no_interrupt"
	ActivityHandlingStartOfActivityInterrupts ActivityHandling = "start_of_activity_interrupts"
)

type AssemblyAILive added in v0.47.0

type AssemblyAILive struct {
	// contains filtered or unexported fields
}

AssemblyAILive implements LiveProvider against AssemblyAI's Voice Agent API. It sends session.update on connect, waits for session.ready before returning, and maps the provider's event vocabulary onto SpeechKit's LiveMessage shape.

func NewAssemblyAILive added in v0.47.0

func NewAssemblyAILive() *AssemblyAILive

func (*AssemblyAILive) Close added in v0.47.0

func (p *AssemblyAILive) Close() error

func (*AssemblyAILive) Connect added in v0.47.0

func (p *AssemblyAILive) Connect(ctx context.Context, cfg LiveConfig) error

func (*AssemblyAILive) Name added in v0.47.0

func (p *AssemblyAILive) Name() string

func (*AssemblyAILive) Receive added in v0.47.0

func (p *AssemblyAILive) Receive(ctx context.Context) (*LiveMessage, error)

func (*AssemblyAILive) Reconnect added in v0.47.0

func (p *AssemblyAILive) Reconnect(ctx context.Context) error

func (*AssemblyAILive) SendAudio added in v0.47.0

func (p *AssemblyAILive) SendAudio(chunk []byte) error

func (*AssemblyAILive) SendAudioStreamEnd added in v0.47.0

func (p *AssemblyAILive) SendAudioStreamEnd() error

func (*AssemblyAILive) SendText added in v0.47.0

func (p *AssemblyAILive) SendText(text string) error

func (*AssemblyAILive) SendToolResponse added in v0.47.0

func (p *AssemblyAILive) SendToolResponse(response ToolResponse) error

func (*AssemblyAILive) SessionCapabilities added in v0.47.0

func (p *AssemblyAILive) SessionCapabilities() SessionCapabilities

func (*AssemblyAILive) UpdateInstructions added in v0.47.0

func (p *AssemblyAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error

type Callbacks

type Callbacks struct {
	OnStateChange          func(state State)
	OnAudio                func(audio []byte) // Audio chunk to play
	OnText                 func(text string)  // Text for display (speech bubble)
	OnError                func(err error)
	OnInputTranscript      func(text string, done bool) // User speech transcribed
	OnOutputTranscript     func(text string, done bool) // Model speech transcribed
	OnToolCall             func(call ToolCall)
	OnToolCallCancellation func(ids []string)
	OnInterrupted          func() // User interrupted model (barge-in)
	OnSessionEnd           func() // Session ended (error, GoAway failure, or deactivation)
}

Callbacks are event handlers for UI integration with a low-level Session.

This is the rich Callbacks struct used by the low-level Session runtime. The higher-level [voiceagent.Callbacks] in the parent package carries only the three most-common handlers (OnAudio, OnText, OnError) and is the right shape for the embedded [voiceagent.Service]; reach for this struct only when wiring a custom realtime host.

type CapabilityRequirement added in v0.47.0

type CapabilityRequirement struct {
	Capability LiveCapabilityFlag `json:"capability"`
	Required   bool               `json:"required,omitempty"`
}

type ContextCompressionPolicy

type ContextCompressionPolicy struct {
	Enabled       bool
	TriggerTokens int64
	TargetTokens  int64
}

ContextCompressionPolicy defines how the live API should compress long sessions.

type DeepgramLive added in v0.46.0

type DeepgramLive struct {
	// Optional overrides; zero values fall back to the package defaults.
	ListenModel   string
	SpeakModel    string
	ThinkProvider string
	ThinkModel    string

	// ThinkEndpointURL + ThinkAPIKey switch the think leg to a bring-your-own
	// LLM deployment. When ThinkEndpointURL is set, the Settings message carries
	// an agent.think.endpoint block so Deepgram calls the operator's own LLM
	// instead of a Deepgram-managed model; ThinkAPIKey (when set) is sent as an
	// "Authorization: Bearer <key>" header on that endpoint. Leave both empty to
	// use Deepgram's managed LLM for ThinkProvider/ThinkModel (no key needed).
	ThinkEndpointURL string
	ThinkAPIKey      string
	// contains filtered or unexported fields
}

DeepgramLive implements LiveProvider against the Deepgram Voice Agent API (WebSocket). It mirrors GeminiLive/OpenAILive's surface so callers don't need to know which backend is active.

The think (LLM) leg is configurable: Deepgram drives the LLM server-side, so ThinkProvider/ThinkModel select which model reasons over the transcript. Defaults target a widely-available option; the wiring layer overrides them from deployment config. Listen defaults to Deepgram Flux for turn-aware conversational STT and speak defaults to a Deepgram Aura-2 voice.

func NewDeepgramLive added in v0.46.0

func NewDeepgramLive() *DeepgramLive

NewDeepgramLive returns a fresh Deepgram Voice Agent provider.

func (*DeepgramLive) Close added in v0.46.0

func (p *DeepgramLive) Close() error

Close terminates the WebSocket and stops the keepalive loop. Idempotent.

func (*DeepgramLive) ConfigureThink added in v0.46.0

func (p *DeepgramLive) ConfigureThink(provider, model, endpointURL, apiKey string)

ConfigureThink applies the deployment's think-LLM selection to the provider. Non-empty provider/model override the package defaults; empty values keep the Deepgram-managed default. endpointURL/apiKey select a bring-your-own think LLM (see ThinkEndpointURL/ThinkAPIKey) and are cleared when empty. Both Targets call this from their Voice Agent wiring with values resolved from config.

func (*DeepgramLive) Connect added in v0.46.0

func (p *DeepgramLive) Connect(ctx context.Context, cfg LiveConfig) error

Connect dials the Deepgram Voice Agent WebSocket and sends the initial Settings message describing listen/think/speak and the audio formats. The SettingsApplied acknowledgement is consumed asynchronously by Receive().

func (*DeepgramLive) Name added in v0.46.0

func (p *DeepgramLive) Name() string

Name identifies the provider in Voice Agent logs.

func (*DeepgramLive) Receive added in v0.46.0

func (p *DeepgramLive) Receive(ctx context.Context) (*LiveMessage, error)

Receive translates the next server frame into a LiveMessage. Binary frames are agent audio; text frames are JSON control events. Events that don't map to LiveMessage fields are swallowed and the loop fetches the next frame.

func (*DeepgramLive) SendAudio added in v0.46.0

func (p *DeepgramLive) SendAudio(chunk []byte) error

SendAudio forwards a 16 kHz PCM16 mic chunk as a binary frame. Deepgram accepts the mic rate directly (declared in the Settings input config), so no resample is needed. Empty chunks are no-ops.

func (*DeepgramLive) SendAudioStreamEnd added in v0.46.0

func (p *DeepgramLive) SendAudioStreamEnd() error

SendAudioStreamEnd is a no-op for Deepgram: the Voice Agent performs its own endpointing/turn-detection server-side and responds when the user stops speaking. There is no client-side commit in the protocol.

func (*DeepgramLive) SendText added in v0.46.0

func (p *DeepgramLive) SendText(text string) error

SendText injects a text user turn (e.g. an idle reminder) and lets the agent respond. Empty text is a no-op.

func (*DeepgramLive) SendToolResponse added in v0.46.0

func (p *DeepgramLive) SendToolResponse(response ToolResponse) error

SendToolResponse returns a host-side function result to the agent.

func (*DeepgramLive) SessionCapabilities added in v0.47.0

func (p *DeepgramLive) SessionCapabilities() SessionCapabilities

func (*DeepgramLive) UpdateInstructions added in v0.46.0

func (p *DeepgramLive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error

UpdateInstructions refreshes the agent's system prompt without a reconnect. Implements LiveInstructionUpdater.

type EndSensitivity

type EndSensitivity string

EndSensitivity controls how aggressively automatic activity detection commits speech end.

const (
	EndSensitivityLow    EndSensitivity = "low"
	EndSensitivityMedium EndSensitivity = "medium"
	EndSensitivityHigh   EndSensitivity = "high"
)

type GeminiLive added in v0.46.0

type GeminiLive struct {
	// contains filtered or unexported fields
}

GeminiLive implements LiveProvider using the Google GenAI Live API.

func NewGeminiLive added in v0.46.0

func NewGeminiLive() *GeminiLive

NewGeminiLive creates a Gemini Live provider.

func (*GeminiLive) Close added in v0.46.0

func (g *GeminiLive) Close() error

func (*GeminiLive) Connect added in v0.46.0

func (g *GeminiLive) Connect(ctx context.Context, cfg LiveConfig) error

func (*GeminiLive) Name added in v0.46.0

func (g *GeminiLive) Name() string

func (*GeminiLive) Receive added in v0.46.0

func (g *GeminiLive) Receive(ctx context.Context) (*LiveMessage, error)

func (*GeminiLive) Reconnect added in v0.46.0

func (g *GeminiLive) Reconnect(ctx context.Context) error

Reconnect re-establishes the session using the stored resumption handle. If the handle has expired (TTL) or been cleared, a fresh session is opened.

func (*GeminiLive) SendAudio added in v0.46.0

func (g *GeminiLive) SendAudio(chunk []byte) error

func (*GeminiLive) SendAudioStreamEnd added in v0.46.0

func (g *GeminiLive) SendAudioStreamEnd() error

func (*GeminiLive) SendText added in v0.46.0

func (g *GeminiLive) SendText(text string) error

func (*GeminiLive) SendToolResponse added in v0.46.0

func (g *GeminiLive) SendToolResponse(response ToolResponse) error

func (*GeminiLive) SessionCapabilities added in v0.47.0

func (g *GeminiLive) SessionCapabilities() SessionCapabilities

type IdleConfig

type IdleConfig struct {
	ReminderAfter   time.Duration // Default: 5 minutes
	DeactivateAfter time.Duration // Default: 15 minutes
}

IdleConfig configures the idle timer behavior.

func DefaultIdleConfig

func DefaultIdleConfig() IdleConfig

DefaultIdleConfig returns sensible defaults.

type IdleTimer

type IdleTimer struct {
	// contains filtered or unexported fields
}

IdleTimer manages reminder and auto-deactivation for Voice Agent.

func NewIdleTimer

func NewIdleTimer(cfg IdleConfig, session *Session) *IdleTimer

NewIdleTimer creates an idle timer bound to a session.

func (*IdleTimer) Reset

func (t *IdleTimer) Reset()

Reset restarts the idle countdown. Call after each user interaction.

func (*IdleTimer) Stop

func (t *IdleTimer) Stop()

Stop cancels all timers.

type LatencyProfile added in v0.47.0

type LatencyProfile string
const (
	LatencyProfileInteractive LatencyProfile = "interactive"
	LatencyProfileBalanced    LatencyProfile = "balanced"
	LatencyProfileAccuracy    LatencyProfile = "accuracy"
)

type LiveCapabilityFlag added in v0.47.0

type LiveCapabilityFlag string
const (
	LiveCapabilityRealtimeAudio       LiveCapabilityFlag = "realtime_audio"
	LiveCapabilityToolCalling         LiveCapabilityFlag = "tool_calling"
	LiveCapabilityNativeWords         LiveCapabilityFlag = "native_words"
	LiveCapabilityTranscript          LiveCapabilityFlag = "transcript"
	LiveCapabilityInterruptions       LiveCapabilityFlag = "interruptions"
	LiveCapabilitySessionResume       LiveCapabilityFlag = "session_resume"
	LiveCapabilityNativeContextPrompt LiveCapabilityFlag = "native_context_prompt"
	LiveCapabilityNativeKeyterms      LiveCapabilityFlag = "native_keyterms"
	LiveCapabilityLanguageHints       LiveCapabilityFlag = "language_hints"
	LiveCapabilitySpeakerStreaming    LiveCapabilityFlag = "speaker_streaming"
	LiveCapabilityPrivacyRedaction    LiveCapabilityFlag = "privacy_redaction"
	LiveCapabilityVoiceFocus          LiveCapabilityFlag = "voice_focus"
	LiveCapabilityMedicalDomain       LiveCapabilityFlag = "medical_domain"
	LiveCapabilityReasoningEffort     LiveCapabilityFlag = "reasoning_effort"
	LiveCapabilityTranslation         LiveCapabilityFlag = "translation"
	LiveCapabilityTranscriptionOnly   LiveCapabilityFlag = "transcription_only"
)

type LiveConfig

type LiveConfig struct {
	Provider  string // e.g. "google", "deepgram", "assemblyai", "openai"
	ProfileID string // e.g. "realtime.google.gemini-native-audio"
	Model     string // e.g. "gemini-3.1-flash-live-preview"
	// FallbackModel is tried when the primary Model's Connect fails. Empty
	// disables the fallback. Typical pairing in 2026: a preview model as
	// Model + the last GA model as FallbackModel, so transient preview
	// outages don't take down a Voice Agent deployment.
	FallbackModel    string
	APIKey           string
	Voice            string // Voice name
	FrameworkPrompt  string
	RefinementPrompt string
	VocabularyHint   string
	Locale           string
	// Region is the Google Cloud region the caller's API key / project is
	// pinned to (e.g. "europe-west3", "us-central1"). Used by providers that
	// support regional endpoints. For Gemini Live (as of May 2026) the API
	// exposes a single global WebSocket endpoint, so this field does NOT
	// redirect traffic — it is logged at connect time for compliance evidence
	// (byok.key_updated audit event) and reserved for future regional routing
	// once Google publishes per-region hostnames. Data residency is controlled
	// at the Google Cloud project level; both the project region AND this field
	// must agree for audit records to be accurate.
	// See docs/compliance/byok-gemini-region-pinning.md.
	Region          string
	Policies        LivePolicies
	Tools           []ToolDefinition
	Workflow        *WorkflowConfig
	Speaker         speaker.Options
	Options         provideropts.Values
	ProviderOptions provideropts.Values
}

LiveConfig configures a real-time session.

func DefaultLiveConfigForProvider added in v0.47.0

func DefaultLiveConfigForProvider(providerOrProfile string) (LiveConfig, bool)

DefaultLiveConfigForProvider returns a provider/profile/model tuple suitable for constructing an embedded Voice Agent session. Callers still supply API keys, prompts, tools, and policies.

func NormalizeLiveConfig added in v0.47.0

func NormalizeLiveConfig(cfg LiveConfig) (LiveConfig, error)

NormalizeLiveConfig fills Provider/ProfileID/Model from the provider descriptor catalog without touching credentials, prompts, tools, or policies. It accepts Provider, ProfileID, or a model id that is unique in the descriptor catalog. Existing non-empty fields remain stronger than descriptor defaults.

type LiveEventType added in v0.47.0

type LiveEventType string
const (
	LiveEventSessionReady     LiveEventType = "session_ready"
	LiveEventInputPartial     LiveEventType = "input_partial"
	LiveEventInputFinal       LiveEventType = "input_final"
	LiveEventOutputAudio      LiveEventType = "output_audio"
	LiveEventOutputText       LiveEventType = "output_text"
	LiveEventToolCall         LiveEventType = "tool_call"
	LiveEventToolResultAck    LiveEventType = "tool_result_ack"
	LiveEventInterrupted      LiveEventType = "interrupted"
	LiveEventTurnEnd          LiveEventType = "turn_end"
	LiveEventSessionResumable LiveEventType = "session_resumable"
	LiveEventSessionEnd       LiveEventType = "session_end"
)

func InferLiveEventTypes added in v0.47.0

func InferLiveEventTypes(msg *LiveMessage) []LiveEventType

InferLiveEventTypes returns the provider-neutral event meanings represented by a LiveMessage. Providers should set EventType/EventTypes themselves when translating native frames; this helper gives custom providers and tests the same fallback semantics.

type LiveInstructionUpdater

type LiveInstructionUpdater interface {
	UpdateInstructions(ctx context.Context, cfg LiveConfig) error
}

LiveInstructionUpdater is optionally implemented by providers that can refresh active host instructions without treating the update as a user turn.

type LiveMessage

type LiveMessage struct {
	EventType        LiveEventType   // Provider-neutral event type for hosts that need precise routing.
	EventTypes       []LiveEventType // All provider-neutral event meanings when a provider combines multiple events in one frame.
	ProviderMetadata map[string]any  // Optional provider-native metadata for debugging and advanced hosts.

	Audio []byte // PCM audio chunk (24kHz 16-bit mono)
	Text  string // Text transcript (may be partial or empty)
	Done  bool   // True when the model's turn is complete

	// Transcription fields (populated when transcription is enabled).
	InputTranscript        string  // User speech transcribed by server
	InputTranscriptDone    bool    // True when input transcription segment is final
	InputSpeakerLabel      string  // Optional diarized speaker label for input transcript segments
	InputPersonID          string  // Optional known person id for input transcript segments
	InputDisplayName       string  // Optional known display name for input transcript segments
	InputSpeakerConfidence float64 // Optional speaker-label confidence
	OutputTranscript       string  // Model speech transcribed by server
	OutputTranscriptDone   bool    // True when output transcription segment is final

	ToolCalls               []ToolCall
	ToolCallCancellationIDs []string
	Interrupted             bool // True when user interrupted model (barge-in)
	GoAway                  bool // True when server signals imminent session end
	SessionResumable        bool // True when the provider indicates reconnect/resume is possible.
}

LiveMessage is a message received from the real-time model.

type LiveModelDescriptor added in v0.47.0

type LiveModelDescriptor struct {
	Provider    string                   `json:"provider"`
	ModelID     string                   `json:"modelId"`
	Name        string                   `json:"name"`
	Lifecycle   framework.ModelLifecycle `json:"lifecycle"`
	Default     bool                     `json:"default,omitempty"`
	Recommended bool                     `json:"recommended,omitempty"`
	SourceURL   string                   `json:"sourceUrl"`
}

type LivePolicies

type LivePolicies struct {
	EnableInputAudioTranscription  bool
	EnableOutputAudioTranscription bool
	EnableAffectiveDialog          bool
	Thinking                       ThinkingPolicy
	ContextCompression             ContextCompressionPolicy
	ActivityDetection              ActivityDetectionPolicy
}

LivePolicies configures Google Live API features that shape Voice Agent behavior.

type LiveProvider

type LiveProvider interface {
	// Connect establishes a WebSocket session to the real-time model.
	Connect(ctx context.Context, cfg LiveConfig) error

	// SendAudio streams PCM audio chunks to the model.
	// Format: 16-bit signed int, little-endian, mono, 16kHz.
	SendAudio(chunk []byte) error

	// SendAudioStreamEnd signals that microphone input for the current turn ended.
	SendAudioStreamEnd() error

	// Receive blocks until the next server message arrives.
	// Returns audio chunks and/or text from the model.
	Receive(ctx context.Context) (*LiveMessage, error)

	// SendText injects a text prompt into the session (for idle reminders).
	SendText(text string) error

	// SendToolResponse sends the result of a host-side tool invocation back to the model.
	SendToolResponse(response ToolResponse) error

	// Close terminates the WebSocket session.
	Close() error

	// Name returns the provider identifier.
	Name() string
}

LiveProvider abstracts a real-time audio-to-audio model connection.

func NewProvider added in v0.47.0

func NewProvider(providerOrProfile string) (LiveProvider, error)

NewProvider returns a fresh built-in LiveProvider for a provider id, alias, or public realtime profile id.

func NewProviderWithFactories added in v0.47.0

func NewProviderWithFactories(providerOrProfile string, factories ProviderFactoryRegistry) (LiveProvider, error)

NewProviderWithFactories resolves provider aliases/profile ids against a custom factory registry. Missing registry entries fall back to the built-in providers, so callers can override only the providers they own.

type LiveReconnector

type LiveReconnector interface {
	Reconnect(ctx context.Context) error
}

LiveReconnector is an optional interface for providers that support session reconnection.

type LiveSessionCapabilities added in v0.47.0

type LiveSessionCapabilities interface {
	SessionCapabilities() SessionCapabilities
}

type ModelLifecyclePolicy added in v0.47.0

type ModelLifecyclePolicy string
const (
	ModelLifecycleAny       ModelLifecyclePolicy = "any"
	ModelLifecyclePreferGA  ModelLifecyclePolicy = "prefer_ga"
	ModelLifecycleRequireGA ModelLifecyclePolicy = "require_ga"
)

type OpenAILive added in v0.46.0

type OpenAILive struct {
	// contains filtered or unexported fields
}

OpenAILive implements LiveProvider against the OpenAI Realtime API (WebSocket). It mirrors GeminiLive's surface so callers don't need to know which backend is active.

func NewOpenAILive added in v0.46.0

func NewOpenAILive() *OpenAILive

NewOpenAILive returns a fresh OpenAI Realtime provider.

func (*OpenAILive) Close added in v0.46.0

func (p *OpenAILive) Close() error

Close terminates the WebSocket. Idempotent.

func (*OpenAILive) Connect added in v0.46.0

func (p *OpenAILive) Connect(ctx context.Context, cfg LiveConfig) error

Connect dials the OpenAI Realtime WebSocket, sends the configured instructions/voice/tools as a session.update, and waits for the session.updated acknowledgement before returning.

func (*OpenAILive) Name added in v0.46.0

func (p *OpenAILive) Name() string

Name identifies the provider in Voice Agent logs.

func (*OpenAILive) Receive added in v0.46.0

func (p *OpenAILive) Receive(ctx context.Context) (*LiveMessage, error)

Receive translates the next server event into a LiveMessage. Server events that don't map to LiveMessage fields (session.created/updated, rate-limit telemetry, etc.) are swallowed and the loop fetches the next frame so callers see an aligned event stream.

func (*OpenAILive) SendAudio added in v0.46.0

func (p *OpenAILive) SendAudio(chunk []byte) error

SendAudio resamples a 16 kHz mic chunk to 24 kHz and forwards it as a base64-encoded input_audio_buffer.append event. Empty chunks are no-ops.

func (*OpenAILive) SendAudioStreamEnd added in v0.46.0

func (p *OpenAILive) SendAudioStreamEnd() error

SendAudioStreamEnd flushes the input audio buffer and triggers a model response. With server VAD enabled OpenAI commits the buffer automatically at end-of-speech; this explicit commit covers push-to-talk style turns where the kernel decides when the user stopped speaking.

func (*OpenAILive) SendText added in v0.46.0

func (p *OpenAILive) SendText(text string) error

SendText injects a text-only user turn and triggers a response.

func (*OpenAILive) SendToolResponse added in v0.46.0

func (p *OpenAILive) SendToolResponse(response ToolResponse) error

SendToolResponse delivers the host-side tool result back to the model and triggers a follow-up response.

func (*OpenAILive) SessionCapabilities added in v0.47.0

func (p *OpenAILive) SessionCapabilities() SessionCapabilities

func (*OpenAILive) UpdateInstructions added in v0.46.0

func (p *OpenAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error

UpdateInstructions sends a fresh session.update with new instructions/tools. Implements LiveInstructionUpdater so the kernel can refresh persona prompts without forcing a reconnect.

type ProviderDescriptor added in v0.47.0

type ProviderDescriptor struct {
	Provider         string                  `json:"provider"`
	DisplayName      string                  `json:"displayName"`
	ProfileID        string                  `json:"profileId"`
	Capabilities     []LiveCapabilityFlag    `json:"capabilities"`
	Models           []LiveModelDescriptor   `json:"models,omitempty"`
	SupportedLocales []string                `json:"supportedLocales,omitempty"`
	NativeOptions    []provideropts.OptionID `json:"nativeOptions,omitempty"`
	AuthRequirement  string                  `json:"authRequirement,omitempty"`
	Transport        string                  `json:"transport,omitempty"`
	EvidenceURL      string                  `json:"evidenceUrl,omitempty"`
}

func DefaultProviderDescriptors added in v0.47.0

func DefaultProviderDescriptors() []ProviderDescriptor

DefaultProviderDescriptors returns the public live-provider catalog for embedders that want to switch providers by name/profile/model at runtime.

func FindProviderDescriptor added in v0.47.0

func FindProviderDescriptor(providerOrProfile string) (ProviderDescriptor, bool)

FindProviderDescriptor resolves provider ids, provider aliases, and profile ids to the canonical public descriptor.

func (ProviderDescriptor) DefaultModel added in v0.47.0

func (d ProviderDescriptor) DefaultModel() (LiveModelDescriptor, bool)

DefaultModel returns the provider's default live model. If no model is explicitly marked as default, it falls back to the first advertised model.

func (ProviderDescriptor) HasCapability added in v0.47.0

func (d ProviderDescriptor) HasCapability(flag LiveCapabilityFlag) bool

HasCapability reports whether this provider advertises a capability.

type ProviderFactory added in v0.47.0

type ProviderFactory func() LiveProvider

ProviderFactory constructs a fresh LiveProvider instance.

type ProviderFactoryRegistry added in v0.47.0

type ProviderFactoryRegistry map[string]ProviderFactory

ProviderFactoryRegistry lets embedders override or extend provider construction while still using SpeechKit's provider/profile normalization.

func DefaultProviderFactories added in v0.47.0

func DefaultProviderFactories() ProviderFactoryRegistry

DefaultProviderFactories returns factories for the built-in native realtime providers. The returned map is a copy and can be safely modified by callers.

type ProviderFallback added in v0.47.0

type ProviderFallback struct {
	Kind                        ProviderFallbackKind     `json:"kind"`
	Provider                    string                   `json:"provider"`
	ProfileID                   string                   `json:"profileId,omitempty"`
	Model                       string                   `json:"model,omitempty"`
	ModelLifecycle              framework.ModelLifecycle `json:"modelLifecycle,omitempty"`
	Reason                      string                   `json:"reason,omitempty"`
	MissingRequiredCapabilities []LiveCapabilityFlag     `json:"missingRequiredCapabilities,omitempty"`
	MissingRequiredOptions      []provideropts.OptionID  `json:"missingRequiredOptions,omitempty"`
	AuthRequirement             string                   `json:"authRequirement,omitempty"`
	Transport                   string                   `json:"transport,omitempty"`
	EvidenceURL                 string                   `json:"evidenceUrl,omitempty"`
}

type ProviderFallbackKind added in v0.47.0

type ProviderFallbackKind string
const (
	FallbackKindSameProviderModel ProviderFallbackKind = "same_provider_model"
	FallbackKindCrossProvider     ProviderFallbackKind = "cross_provider"
	FallbackKindCascaded          ProviderFallbackKind = "cascaded"
	FallbackKindCapabilityMissing ProviderFallbackKind = "capability_missing"
)

type ProviderIntent added in v0.47.0

type ProviderIntent struct {
	Mode                  string                  `json:"mode,omitempty"`
	Provider              string                  `json:"provider,omitempty"`
	ProfileID             string                  `json:"profileId,omitempty"`
	Model                 string                  `json:"model,omitempty"`
	RequiredCapabilities  []LiveCapabilityFlag    `json:"requiredCapabilities,omitempty"`
	PreferredCapabilities []LiveCapabilityFlag    `json:"preferredCapabilities,omitempty"`
	Requirements          []CapabilityRequirement `json:"requirements,omitempty"`
	RequiredOptions       []provideropts.OptionID `json:"requiredOptions,omitempty"`
	PreferredOptions      []provideropts.OptionID `json:"preferredOptions,omitempty"`
	Locale                string                  `json:"locale,omitempty"`
	LanguageHints         []string                `json:"languageHints,omitempty"`
	PrivacyRedaction      bool                    `json:"privacyRedaction,omitempty"`
	ResumePreferred       bool                    `json:"resumePreferred,omitempty"`
	LatencyProfile        LatencyProfile          `json:"latencyProfile,omitempty"`
	SelectionPolicy       ProviderSelectionPolicy `json:"selectionPolicy,omitempty"`
}

type ProviderIntentError added in v0.47.0

type ProviderIntentError struct {
	Intent                      ProviderIntent          `json:"intent"`
	MissingRequiredCapabilities []LiveCapabilityFlag    `json:"missingRequiredCapabilities,omitempty"`
	MissingRequiredOptions      []provideropts.OptionID `json:"missingRequiredOptions,omitempty"`
	Fallbacks                   []ProviderFallback      `json:"fallbacks,omitempty"`
	RejectedProviders           []ProviderRejection     `json:"rejectedProviders,omitempty"`
}

func (*ProviderIntentError) Error added in v0.47.0

func (e *ProviderIntentError) Error() string

func (*ProviderIntentError) Unwrap added in v0.47.0

func (e *ProviderIntentError) Unwrap() error

type ProviderRejection added in v0.47.0

type ProviderRejection struct {
	Provider                    string                   `json:"provider"`
	ProfileID                   string                   `json:"profileId,omitempty"`
	Model                       string                   `json:"model,omitempty"`
	ModelLifecycle              framework.ModelLifecycle `json:"modelLifecycle,omitempty"`
	FallbackKind                ProviderFallbackKind     `json:"fallbackKind,omitempty"`
	Reason                      string                   `json:"reason"`
	MissingRequiredCapabilities []LiveCapabilityFlag     `json:"missingRequiredCapabilities,omitempty"`
	MissingRequiredOptions      []provideropts.OptionID  `json:"missingRequiredOptions,omitempty"`
	AuthRequirement             string                   `json:"authRequirement,omitempty"`
	Transport                   string                   `json:"transport,omitempty"`
	EvidenceURL                 string                   `json:"evidenceUrl,omitempty"`
	UnsupportedLocale           string                   `json:"unsupportedLocale,omitempty"`
}

type ProviderSelectionPolicy added in v0.47.0

type ProviderSelectionPolicy struct {
	PreferredProviders []string             `json:"preferredProviders,omitempty"`
	AllowPreview       bool                 `json:"allowPreview,omitempty"`
	AllowLegacy        bool                 `json:"allowLegacy,omitempty"`
	ModelLifecycle     ModelLifecyclePolicy `json:"modelLifecycle,omitempty"`
}

type ResolvedLiveOptions added in v0.45.0

type ResolvedLiveOptions struct {
	Locale              string
	Voice               string
	ContextPrompt       string
	LanguageHints       []string
	Keyterms            []string
	ReasoningEffort     string
	Resume              bool
	TurnDetection       bool
	TurnDetectionSource provideropts.ValueSource
	EndpointingMs       int
	EndpointingSource   provideropts.ValueSource
	Effective           provideropts.EffectiveOptions
}

func ResolveLiveOptions added in v0.45.0

func ResolveLiveOptions(provider, profileID string, cfg LiveConfig, providerDefaults, providerOverrides provideropts.Values) ResolvedLiveOptions

func (ResolvedLiveOptions) HasEndpointingOverride added in v0.45.0

func (r ResolvedLiveOptions) HasEndpointingOverride() bool

func (ResolvedLiveOptions) HasTurnDetectionOverride added in v0.45.0

func (r ResolvedLiveOptions) HasTurnDetectionOverride() bool

type ResolvedProviderPlan added in v0.47.0

type ResolvedProviderPlan struct {
	Provider                         string                  `json:"provider"`
	ProfileID                        string                  `json:"profileId"`
	Model                            string                  `json:"model"`
	Descriptor                       ProviderDescriptor      `json:"descriptor"`
	ModelDescriptor                  LiveModelDescriptor     `json:"modelDescriptor"`
	SelectionReason                  string                  `json:"selectionReason,omitempty"`
	SelectedFallbackKind             ProviderFallbackKind    `json:"selectedFallbackKind,omitempty"`
	Fallbacks                        []ProviderFallback      `json:"fallbacks,omitempty"`
	RejectedProviders                []ProviderRejection     `json:"rejectedProviders,omitempty"`
	MatchedRequiredCapabilities      []LiveCapabilityFlag    `json:"matchedRequiredCapabilities,omitempty"`
	MatchedPreferredCapabilities     []LiveCapabilityFlag    `json:"matchedPreferredCapabilities,omitempty"`
	UnsupportedPreferredCapabilities []LiveCapabilityFlag    `json:"unsupportedPreferredCapabilities,omitempty"`
	MatchedRequiredOptions           []provideropts.OptionID `json:"matchedRequiredOptions,omitempty"`
	MatchedPreferredOptions          []provideropts.OptionID `json:"matchedPreferredOptions,omitempty"`
	UnsupportedPreferredOptions      []provideropts.OptionID `json:"unsupportedPreferredOptions,omitempty"`
	AuthRequirement                  string                  `json:"authRequirement,omitempty"`
	Transport                        string                  `json:"transport,omitempty"`
	LatencyProfile                   LatencyProfile          `json:"latencyProfile,omitempty"`
}

func ResolveProviderIntent added in v0.47.0

func ResolveProviderIntent(intent ProviderIntent, descriptors []ProviderDescriptor) (ResolvedProviderPlan, error)

func (ResolvedProviderPlan) LiveConfig added in v0.47.0

func (p ResolvedProviderPlan) LiveConfig() LiveConfig

func (ResolvedProviderPlan) String added in v0.47.0

func (p ResolvedProviderPlan) String() string

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session manages a Voice Agent conversation.

func NewSession

func NewSession(provider LiveProvider, callbacks Callbacks) *Session

NewSession creates a Voice Agent session with the given provider.

func (*Session) AdvanceWorkflowStep

func (s *Session) AdvanceWorkflowStep(ctx context.Context, reason string) error

AdvanceWorkflowStep moves a configured local workflow to its next step and updates the active provider instructions. It returns nil when no workflow is active or the workflow has already completed.

func (*Session) CurrentState

func (s *Session) CurrentState() State

State returns the current session state.

func (*Session) EndAudioStream

func (s *Session) EndAudioStream() error

EndAudioStream tells the live provider that the current microphone stream ended.

func (*Session) ProviderName

func (s *Session) ProviderName() string

func (*Session) SendAudio

func (s *Session) SendAudio(chunk []byte) error

SendAudio forwards a PCM audio chunk to the real-time model.

func (*Session) SendText

func (s *Session) SendText(text string) error

SendText injects a user text turn into the live session.

func (*Session) SendToolResponse

func (s *Session) SendToolResponse(response ToolResponse) error

SendToolResponse forwards the result of a host-side tool invocation to the model.

func (*Session) Start

func (s *Session) Start(ctx context.Context, cfg LiveConfig, idleCfg IdleConfig) error

Start activates the Voice Agent session.

func (*Session) Stop

func (s *Session) Stop()

Stop deactivates the Voice Agent session.

type SessionCapabilities added in v0.47.0

type SessionCapabilities struct {
	Provider         string               `json:"provider,omitempty"`
	ProfileID        string               `json:"profileId,omitempty"`
	Model            string               `json:"model,omitempty"`
	Capabilities     []LiveCapabilityFlag `json:"capabilities,omitempty"`
	ProviderMetadata map[string]any       `json:"providerMetadata,omitempty"`
}

type StartSensitivity

type StartSensitivity string

StartSensitivity controls how aggressively automatic activity detection commits speech start.

const (
	StartSensitivityLow    StartSensitivity = "low"
	StartSensitivityMedium StartSensitivity = "medium"
	StartSensitivityHigh   StartSensitivity = "high"
)

type State

type State string

State represents the current state of a Voice Agent session.

const (
	StateInactive     State = "inactive"
	StateConnecting   State = "connecting"
	StateListening    State = "listening"
	StateProcessing   State = "processing"
	StateSpeaking     State = "speaking"
	StateRecovering   State = "recovering"
	StateDeactivating State = "deactivating"
)

type ThinkingLevel

type ThinkingLevel string

ThinkingLevel controls how much deliberate reasoning Gemini Live should spend.

const (
	ThinkingLevelOff    ThinkingLevel = "off"
	ThinkingLevelLow    ThinkingLevel = "low"
	ThinkingLevelMedium ThinkingLevel = "medium"
	ThinkingLevelHigh   ThinkingLevel = "high"
)

type ThinkingPolicy

type ThinkingPolicy struct {
	Enabled         bool
	IncludeThoughts bool
	ThinkingBudget  int32
	ThinkingLevel   ThinkingLevel
}

ThinkingPolicy defines optional Gemini Live thinking behavior.

type ToolBehavior

type ToolBehavior string

ToolBehavior controls whether the model waits for a tool result.

const (
	ToolBehaviorUnspecified ToolBehavior = ""
	ToolBehaviorBlocking    ToolBehavior = "blocking"
	ToolBehaviorNonBlocking ToolBehavior = "non_blocking"
)

type ToolCall

type ToolCall struct {
	ID   string
	Name string
	Args map[string]any
}

ToolCall is a host-side action request emitted by the Voice Agent runtime.

type ToolDefinition

type ToolDefinition struct {
	Name                 string
	Description          string
	ParametersJSONSchema map[string]any
	ResponseJSONSchema   map[string]any
	Behavior             ToolBehavior
}

ToolDefinition exposes a host-side action the Voice Agent may call.

type ToolResponse

type ToolResponse struct {
	ID       string
	Name     string
	Response map[string]any

	Scheduling   ToolResponseScheduling
	WillContinue *bool
}

ToolResponse resolves a previously emitted tool call.

type ToolResponseScheduling

type ToolResponseScheduling string

ToolResponseScheduling controls how a non-blocking tool result is reintroduced into the conversation.

const (
	ToolResponseSchedulingUnspecified ToolResponseScheduling = ""
	ToolResponseSchedulingSilent      ToolResponseScheduling = "silent"
	ToolResponseSchedulingWhenIdle    ToolResponseScheduling = "when_idle"
	ToolResponseSchedulingInterrupt   ToolResponseScheduling = "interrupt"
)

type TurnCoverage

type TurnCoverage string

TurnCoverage controls how the live API builds a user turn from incoming activity.

const (
	TurnCoverageUnspecified               TurnCoverage = ""
	TurnCoverageTurnIncludesOnlyActivity  TurnCoverage = "turn_includes_only_activity"
	TurnCoverageTurnIncludesAllInput      TurnCoverage = "turn_includes_all_input"
	TurnCoverageTurnIncludesAudioActivity TurnCoverage = "turn_includes_audio_activity"
)

type WorkflowConfig

type WorkflowConfig struct {
	SequenceID string
	Completion string
	MaxTurns   int
	BasePrompt string
	Steps      []WorkflowStep
	// InitialStep selects the first active step. Out-of-range values fall
	// back to zero.
	InitialStep int
}

WorkflowConfig describes a deterministic, step-based Voice Agent behavior sequence. Durations are intentionally expressed as turns instead of wall clock time so tests and local installs can exercise long moderation flows quickly.

type WorkflowStep

type WorkflowStep struct {
	ID           string
	Instruction  string
	ExitCriteria string
	RequireTools []string
	MaxTurns     int
}

WorkflowStep describes one stage in a Voice Agent workflow.

Directories

Path Synopsis
Package livecontract provides reusable conformance checks for LiveProvider implementations.
Package livecontract provides reusable conformance checks for LiveProvider implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL