Documentation
¶
Overview ¶
Package live exposes the low-level Voice Agent realtime-protocol types.
This is the public-API surface for building custom realtime providers (Gemini Live, OpenAI Realtime, or a third-party WebSocket model) and for libraries that drive a SpeechKit session directly without going through the higher-level [Service] in the parent package.
Most embedders use the parent [voiceagent] package or github.com/kombifyio/SpeechKit/pkg/speechkit/agentkit instead; reach for this package only when you need to plug in your own provider or read the raw LiveMessage stream.
Index ¶
- Constants
- Variables
- func NewProviderForConfig(cfg LiveConfig) (LiveProvider, LiveConfig, error)
- func NormalizeProviderID(providerOrProfile string) string
- func RenderHostInstructionUpdate(cfg LiveConfig) string
- type ActivityDetectionPolicy
- type ActivityHandling
- type AssemblyAILive
- func (p *AssemblyAILive) Close() error
- func (p *AssemblyAILive) Connect(ctx context.Context, cfg LiveConfig) error
- func (p *AssemblyAILive) Name() string
- func (p *AssemblyAILive) Receive(ctx context.Context) (*LiveMessage, error)
- func (p *AssemblyAILive) Reconnect(ctx context.Context) error
- func (p *AssemblyAILive) SendAudio(chunk []byte) error
- func (p *AssemblyAILive) SendAudioStreamEnd() error
- func (p *AssemblyAILive) SendText(text string) error
- func (p *AssemblyAILive) SendToolResponse(response ToolResponse) error
- func (p *AssemblyAILive) SessionCapabilities() SessionCapabilities
- func (p *AssemblyAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
- type Callbacks
- type CapabilityRequirement
- type ContextCompressionPolicy
- type DeepgramLive
- func (p *DeepgramLive) Close() error
- func (p *DeepgramLive) ConfigureThink(provider, model, endpointURL, apiKey string)
- func (p *DeepgramLive) Connect(ctx context.Context, cfg LiveConfig) error
- func (p *DeepgramLive) Name() string
- func (p *DeepgramLive) Receive(ctx context.Context) (*LiveMessage, error)
- func (p *DeepgramLive) SendAudio(chunk []byte) error
- func (p *DeepgramLive) SendAudioStreamEnd() error
- func (p *DeepgramLive) SendText(text string) error
- func (p *DeepgramLive) SendToolResponse(response ToolResponse) error
- func (p *DeepgramLive) SessionCapabilities() SessionCapabilities
- func (p *DeepgramLive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
- type EndSensitivity
- type GeminiLive
- func (g *GeminiLive) Close() error
- func (g *GeminiLive) Connect(ctx context.Context, cfg LiveConfig) error
- func (g *GeminiLive) Name() string
- func (g *GeminiLive) Receive(ctx context.Context) (*LiveMessage, error)
- func (g *GeminiLive) Reconnect(ctx context.Context) error
- func (g *GeminiLive) SendAudio(chunk []byte) error
- func (g *GeminiLive) SendAudioStreamEnd() error
- func (g *GeminiLive) SendText(text string) error
- func (g *GeminiLive) SendToolResponse(response ToolResponse) error
- func (g *GeminiLive) SessionCapabilities() SessionCapabilities
- type IdleConfig
- type IdleTimer
- type LatencyProfile
- type LiveCapabilityFlag
- type LiveConfig
- type LiveEventType
- type LiveInstructionUpdater
- type LiveMessage
- type LiveModelDescriptor
- type LivePolicies
- type LiveProvider
- type LiveReconnector
- type LiveSessionCapabilities
- type ModelLifecyclePolicy
- type OpenAILive
- func (p *OpenAILive) Close() error
- func (p *OpenAILive) Connect(ctx context.Context, cfg LiveConfig) error
- func (p *OpenAILive) Name() string
- func (p *OpenAILive) Receive(ctx context.Context) (*LiveMessage, error)
- func (p *OpenAILive) SendAudio(chunk []byte) error
- func (p *OpenAILive) SendAudioStreamEnd() error
- func (p *OpenAILive) SendText(text string) error
- func (p *OpenAILive) SendToolResponse(response ToolResponse) error
- func (p *OpenAILive) SessionCapabilities() SessionCapabilities
- func (p *OpenAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
- type ProviderDescriptor
- type ProviderFactory
- type ProviderFactoryRegistry
- type ProviderFallback
- type ProviderFallbackKind
- type ProviderIntent
- type ProviderIntentError
- type ProviderRejection
- type ProviderSelectionPolicy
- type ResolvedLiveOptions
- type ResolvedProviderPlan
- type Session
- func (s *Session) AdvanceWorkflowStep(ctx context.Context, reason string) error
- func (s *Session) CurrentState() State
- func (s *Session) EndAudioStream() error
- func (s *Session) ProviderName() string
- func (s *Session) SendAudio(chunk []byte) error
- func (s *Session) SendText(text string) error
- func (s *Session) SendToolResponse(response ToolResponse) error
- func (s *Session) Start(ctx context.Context, cfg LiveConfig, idleCfg IdleConfig) error
- func (s *Session) Stop()
- type SessionCapabilities
- type StartSensitivity
- type State
- type ThinkingLevel
- type ThinkingPolicy
- type ToolBehavior
- type ToolCall
- type ToolDefinition
- type ToolResponse
- type ToolResponseScheduling
- type TurnCoverage
- type WorkflowConfig
- type WorkflowStep
Constants ¶
const DefaultOpenAIRealtimeModel = defaultOpenAIRealtimeModel
DefaultOpenAIRealtimeModel is the public runtime default for OpenAI-backed Voice Agent sessions.
Variables ¶
var ErrNoMatchingProvider = errors.New("speechkit live: no provider satisfies intent")
var ErrUnknownLiveProvider = errors.New("speechkit live: unknown provider")
Functions ¶
func NewProviderForConfig ¶ added in v0.47.0
func NewProviderForConfig(cfg LiveConfig) (LiveProvider, LiveConfig, error)
NewProviderForConfig normalizes a LiveConfig and returns the matching provider. It is the most convenient entrypoint for embedders that expose a provider/profile/model picker to users.
func NormalizeProviderID ¶ added in v0.47.0
NormalizeProviderID maps common provider aliases and public profile ids to the canonical provider id used by ProviderDescriptor.Provider.
func RenderHostInstructionUpdate ¶
func RenderHostInstructionUpdate(cfg LiveConfig) string
RenderHostInstructionUpdate turns a workflow step change into a provider text update. Providers that implement LiveInstructionUpdater receive the structured LiveConfig instead.
Types ¶
type ActivityDetectionPolicy ¶
type ActivityDetectionPolicy struct {
Automatic bool
StartSensitivity StartSensitivity
EndSensitivity EndSensitivity
PrefixPaddingMs int32
SilenceDurationMs int32
ActivityHandling ActivityHandling
TurnCoverage TurnCoverage
}
ActivityDetectionPolicy defines server-side VAD/session turn behavior.
type ActivityHandling ¶
type ActivityHandling string
ActivityHandling controls what Gemini Live should do when new activity starts.
const ( ActivityHandlingUnspecified ActivityHandling = "" ActivityHandlingNoInterrupt ActivityHandling = "no_interrupt" ActivityHandlingStartOfActivityInterrupts ActivityHandling = "start_of_activity_interrupts" )
type AssemblyAILive ¶ added in v0.47.0
type AssemblyAILive struct {
// contains filtered or unexported fields
}
AssemblyAILive implements LiveProvider against AssemblyAI's Voice Agent API. It sends session.update on connect, waits for session.ready before returning, and maps the provider's event vocabulary onto SpeechKit's LiveMessage shape.
func NewAssemblyAILive ¶ added in v0.47.0
func NewAssemblyAILive() *AssemblyAILive
func (*AssemblyAILive) Close ¶ added in v0.47.0
func (p *AssemblyAILive) Close() error
func (*AssemblyAILive) Connect ¶ added in v0.47.0
func (p *AssemblyAILive) Connect(ctx context.Context, cfg LiveConfig) error
func (*AssemblyAILive) Name ¶ added in v0.47.0
func (p *AssemblyAILive) Name() string
func (*AssemblyAILive) Receive ¶ added in v0.47.0
func (p *AssemblyAILive) Receive(ctx context.Context) (*LiveMessage, error)
func (*AssemblyAILive) Reconnect ¶ added in v0.47.0
func (p *AssemblyAILive) Reconnect(ctx context.Context) error
func (*AssemblyAILive) SendAudio ¶ added in v0.47.0
func (p *AssemblyAILive) SendAudio(chunk []byte) error
func (*AssemblyAILive) SendAudioStreamEnd ¶ added in v0.47.0
func (p *AssemblyAILive) SendAudioStreamEnd() error
func (*AssemblyAILive) SendText ¶ added in v0.47.0
func (p *AssemblyAILive) SendText(text string) error
func (*AssemblyAILive) SendToolResponse ¶ added in v0.47.0
func (p *AssemblyAILive) SendToolResponse(response ToolResponse) error
func (*AssemblyAILive) SessionCapabilities ¶ added in v0.47.0
func (p *AssemblyAILive) SessionCapabilities() SessionCapabilities
func (*AssemblyAILive) UpdateInstructions ¶ added in v0.47.0
func (p *AssemblyAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
type Callbacks ¶
type Callbacks struct {
OnStateChange func(state State)
OnAudio func(audio []byte) // Audio chunk to play
OnText func(text string) // Text for display (speech bubble)
OnError func(err error)
OnInputTranscript func(text string, done bool) // User speech transcribed
OnOutputTranscript func(text string, done bool) // Model speech transcribed
OnToolCall func(call ToolCall)
OnToolCallCancellation func(ids []string)
OnInterrupted func() // User interrupted model (barge-in)
OnSessionEnd func() // Session ended (error, GoAway failure, or deactivation)
}
Callbacks are event handlers for UI integration with a low-level Session.
This is the rich Callbacks struct used by the low-level Session runtime. The higher-level [voiceagent.Callbacks] in the parent package carries only the three most-common handlers (OnAudio, OnText, OnError) and is the right shape for the embedded [voiceagent.Service]; reach for this struct only when wiring a custom realtime host.
type CapabilityRequirement ¶ added in v0.47.0
type CapabilityRequirement struct {
Capability LiveCapabilityFlag `json:"capability"`
Required bool `json:"required,omitempty"`
}
type ContextCompressionPolicy ¶
ContextCompressionPolicy defines how the live API should compress long sessions.
type DeepgramLive ¶ added in v0.46.0
type DeepgramLive struct {
// Optional overrides; zero values fall back to the package defaults.
ListenModel string
SpeakModel string
ThinkProvider string
ThinkModel string
// ThinkEndpointURL + ThinkAPIKey switch the think leg to a bring-your-own
// LLM deployment. When ThinkEndpointURL is set, the Settings message carries
// an agent.think.endpoint block so Deepgram calls the operator's own LLM
// instead of a Deepgram-managed model; ThinkAPIKey (when set) is sent as an
// "Authorization: Bearer <key>" header on that endpoint. Leave both empty to
// use Deepgram's managed LLM for ThinkProvider/ThinkModel (no key needed).
ThinkEndpointURL string
ThinkAPIKey string
// contains filtered or unexported fields
}
DeepgramLive implements LiveProvider against the Deepgram Voice Agent API (WebSocket). It mirrors GeminiLive/OpenAILive's surface so callers don't need to know which backend is active.
The think (LLM) leg is configurable: Deepgram drives the LLM server-side, so ThinkProvider/ThinkModel select which model reasons over the transcript. Defaults target a widely-available option; the wiring layer overrides them from deployment config. Listen defaults to Deepgram Flux for turn-aware conversational STT and speak defaults to a Deepgram Aura-2 voice.
func NewDeepgramLive ¶ added in v0.46.0
func NewDeepgramLive() *DeepgramLive
NewDeepgramLive returns a fresh Deepgram Voice Agent provider.
func (*DeepgramLive) Close ¶ added in v0.46.0
func (p *DeepgramLive) Close() error
Close terminates the WebSocket and stops the keepalive loop. Idempotent.
func (*DeepgramLive) ConfigureThink ¶ added in v0.46.0
func (p *DeepgramLive) ConfigureThink(provider, model, endpointURL, apiKey string)
ConfigureThink applies the deployment's think-LLM selection to the provider. Non-empty provider/model override the package defaults; empty values keep the Deepgram-managed default. endpointURL/apiKey select a bring-your-own think LLM (see ThinkEndpointURL/ThinkAPIKey) and are cleared when empty. Both Targets call this from their Voice Agent wiring with values resolved from config.
func (*DeepgramLive) Connect ¶ added in v0.46.0
func (p *DeepgramLive) Connect(ctx context.Context, cfg LiveConfig) error
Connect dials the Deepgram Voice Agent WebSocket and sends the initial Settings message describing listen/think/speak and the audio formats. The SettingsApplied acknowledgement is consumed asynchronously by Receive().
func (*DeepgramLive) Name ¶ added in v0.46.0
func (p *DeepgramLive) Name() string
Name identifies the provider in Voice Agent logs.
func (*DeepgramLive) Receive ¶ added in v0.46.0
func (p *DeepgramLive) Receive(ctx context.Context) (*LiveMessage, error)
Receive translates the next server frame into a LiveMessage. Binary frames are agent audio; text frames are JSON control events. Events that don't map to LiveMessage fields are swallowed and the loop fetches the next frame.
func (*DeepgramLive) SendAudio ¶ added in v0.46.0
func (p *DeepgramLive) SendAudio(chunk []byte) error
SendAudio forwards a 16 kHz PCM16 mic chunk as a binary frame. Deepgram accepts the mic rate directly (declared in the Settings input config), so no resample is needed. Empty chunks are no-ops.
func (*DeepgramLive) SendAudioStreamEnd ¶ added in v0.46.0
func (p *DeepgramLive) SendAudioStreamEnd() error
SendAudioStreamEnd is a no-op for Deepgram: the Voice Agent performs its own endpointing/turn-detection server-side and responds when the user stops speaking. There is no client-side commit in the protocol.
func (*DeepgramLive) SendText ¶ added in v0.46.0
func (p *DeepgramLive) SendText(text string) error
SendText injects a text user turn (e.g. an idle reminder) and lets the agent respond. Empty text is a no-op.
func (*DeepgramLive) SendToolResponse ¶ added in v0.46.0
func (p *DeepgramLive) SendToolResponse(response ToolResponse) error
SendToolResponse returns a host-side function result to the agent.
func (*DeepgramLive) SessionCapabilities ¶ added in v0.47.0
func (p *DeepgramLive) SessionCapabilities() SessionCapabilities
func (*DeepgramLive) UpdateInstructions ¶ added in v0.46.0
func (p *DeepgramLive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
UpdateInstructions refreshes the agent's system prompt without a reconnect. Implements LiveInstructionUpdater.
type EndSensitivity ¶
type EndSensitivity string
EndSensitivity controls how aggressively automatic activity detection commits speech end.
const ( EndSensitivityLow EndSensitivity = "low" EndSensitivityMedium EndSensitivity = "medium" EndSensitivityHigh EndSensitivity = "high" )
type GeminiLive ¶ added in v0.46.0
type GeminiLive struct {
// contains filtered or unexported fields
}
GeminiLive implements LiveProvider using the Google GenAI Live API.
func NewGeminiLive ¶ added in v0.46.0
func NewGeminiLive() *GeminiLive
NewGeminiLive creates a Gemini Live provider.
func (*GeminiLive) Close ¶ added in v0.46.0
func (g *GeminiLive) Close() error
func (*GeminiLive) Connect ¶ added in v0.46.0
func (g *GeminiLive) Connect(ctx context.Context, cfg LiveConfig) error
func (*GeminiLive) Name ¶ added in v0.46.0
func (g *GeminiLive) Name() string
func (*GeminiLive) Receive ¶ added in v0.46.0
func (g *GeminiLive) Receive(ctx context.Context) (*LiveMessage, error)
func (*GeminiLive) Reconnect ¶ added in v0.46.0
func (g *GeminiLive) Reconnect(ctx context.Context) error
Reconnect re-establishes the session using the stored resumption handle. If the handle has expired (TTL) or been cleared, a fresh session is opened.
func (*GeminiLive) SendAudio ¶ added in v0.46.0
func (g *GeminiLive) SendAudio(chunk []byte) error
func (*GeminiLive) SendAudioStreamEnd ¶ added in v0.46.0
func (g *GeminiLive) SendAudioStreamEnd() error
func (*GeminiLive) SendText ¶ added in v0.46.0
func (g *GeminiLive) SendText(text string) error
func (*GeminiLive) SendToolResponse ¶ added in v0.46.0
func (g *GeminiLive) SendToolResponse(response ToolResponse) error
func (*GeminiLive) SessionCapabilities ¶ added in v0.47.0
func (g *GeminiLive) SessionCapabilities() SessionCapabilities
type IdleConfig ¶
type IdleConfig struct {
ReminderAfter time.Duration // Default: 5 minutes
DeactivateAfter time.Duration // Default: 15 minutes
}
IdleConfig configures the idle timer behavior.
func DefaultIdleConfig ¶
func DefaultIdleConfig() IdleConfig
DefaultIdleConfig returns sensible defaults.
type IdleTimer ¶
type IdleTimer struct {
// contains filtered or unexported fields
}
IdleTimer manages reminder and auto-deactivation for Voice Agent.
func NewIdleTimer ¶
func NewIdleTimer(cfg IdleConfig, session *Session) *IdleTimer
NewIdleTimer creates an idle timer bound to a session.
type LatencyProfile ¶ added in v0.47.0
type LatencyProfile string
const ( LatencyProfileInteractive LatencyProfile = "interactive" LatencyProfileBalanced LatencyProfile = "balanced" LatencyProfileAccuracy LatencyProfile = "accuracy" )
type LiveCapabilityFlag ¶ added in v0.47.0
type LiveCapabilityFlag string
const ( LiveCapabilityRealtimeAudio LiveCapabilityFlag = "realtime_audio" LiveCapabilityToolCalling LiveCapabilityFlag = "tool_calling" LiveCapabilityNativeWords LiveCapabilityFlag = "native_words" LiveCapabilityTranscript LiveCapabilityFlag = "transcript" LiveCapabilityInterruptions LiveCapabilityFlag = "interruptions" LiveCapabilitySessionResume LiveCapabilityFlag = "session_resume" LiveCapabilityNativeContextPrompt LiveCapabilityFlag = "native_context_prompt" LiveCapabilityNativeKeyterms LiveCapabilityFlag = "native_keyterms" LiveCapabilityLanguageHints LiveCapabilityFlag = "language_hints" LiveCapabilitySpeakerStreaming LiveCapabilityFlag = "speaker_streaming" LiveCapabilityPrivacyRedaction LiveCapabilityFlag = "privacy_redaction" LiveCapabilityVoiceFocus LiveCapabilityFlag = "voice_focus" LiveCapabilityMedicalDomain LiveCapabilityFlag = "medical_domain" LiveCapabilityReasoningEffort LiveCapabilityFlag = "reasoning_effort" LiveCapabilityTranslation LiveCapabilityFlag = "translation" LiveCapabilityTranscriptionOnly LiveCapabilityFlag = "transcription_only" )
type LiveConfig ¶
type LiveConfig struct {
Provider string // e.g. "google", "deepgram", "assemblyai", "openai"
ProfileID string // e.g. "realtime.google.gemini-native-audio"
Model string // e.g. "gemini-3.1-flash-live-preview"
// FallbackModel is tried when the primary Model's Connect fails. Empty
// disables the fallback. Typical pairing in 2026: a preview model as
// Model + the last GA model as FallbackModel, so transient preview
// outages don't take down a Voice Agent deployment.
FallbackModel string
APIKey string
Voice string // Voice name
FrameworkPrompt string
RefinementPrompt string
VocabularyHint string
Locale string
// Region is the Google Cloud region the caller's API key / project is
// pinned to (e.g. "europe-west3", "us-central1"). Used by providers that
// support regional endpoints. For Gemini Live (as of May 2026) the API
// exposes a single global WebSocket endpoint, so this field does NOT
// redirect traffic — it is logged at connect time for compliance evidence
// (byok.key_updated audit event) and reserved for future regional routing
// once Google publishes per-region hostnames. Data residency is controlled
// at the Google Cloud project level; both the project region AND this field
// must agree for audit records to be accurate.
// See docs/compliance/byok-gemini-region-pinning.md.
Region string
Policies LivePolicies
Tools []ToolDefinition
Workflow *WorkflowConfig
Speaker speaker.Options
Options provideropts.Values
ProviderOptions provideropts.Values
}
LiveConfig configures a real-time session.
func DefaultLiveConfigForProvider ¶ added in v0.47.0
func DefaultLiveConfigForProvider(providerOrProfile string) (LiveConfig, bool)
DefaultLiveConfigForProvider returns a provider/profile/model tuple suitable for constructing an embedded Voice Agent session. Callers still supply API keys, prompts, tools, and policies.
func NormalizeLiveConfig ¶ added in v0.47.0
func NormalizeLiveConfig(cfg LiveConfig) (LiveConfig, error)
NormalizeLiveConfig fills Provider/ProfileID/Model from the provider descriptor catalog without touching credentials, prompts, tools, or policies. It accepts Provider, ProfileID, or a model id that is unique in the descriptor catalog. Existing non-empty fields remain stronger than descriptor defaults.
type LiveEventType ¶ added in v0.47.0
type LiveEventType string
const ( LiveEventSessionReady LiveEventType = "session_ready" LiveEventInputPartial LiveEventType = "input_partial" LiveEventInputFinal LiveEventType = "input_final" LiveEventOutputAudio LiveEventType = "output_audio" LiveEventOutputText LiveEventType = "output_text" LiveEventToolCall LiveEventType = "tool_call" LiveEventToolResultAck LiveEventType = "tool_result_ack" LiveEventInterrupted LiveEventType = "interrupted" LiveEventTurnEnd LiveEventType = "turn_end" LiveEventSessionResumable LiveEventType = "session_resumable" LiveEventSessionEnd LiveEventType = "session_end" )
func InferLiveEventTypes ¶ added in v0.47.0
func InferLiveEventTypes(msg *LiveMessage) []LiveEventType
InferLiveEventTypes returns the provider-neutral event meanings represented by a LiveMessage. Providers should set EventType/EventTypes themselves when translating native frames; this helper gives custom providers and tests the same fallback semantics.
type LiveInstructionUpdater ¶
type LiveInstructionUpdater interface {
UpdateInstructions(ctx context.Context, cfg LiveConfig) error
}
LiveInstructionUpdater is optionally implemented by providers that can refresh active host instructions without treating the update as a user turn.
type LiveMessage ¶
type LiveMessage struct {
EventType LiveEventType // Provider-neutral event type for hosts that need precise routing.
EventTypes []LiveEventType // All provider-neutral event meanings when a provider combines multiple events in one frame.
ProviderMetadata map[string]any // Optional provider-native metadata for debugging and advanced hosts.
Audio []byte // PCM audio chunk (24kHz 16-bit mono)
Text string // Text transcript (may be partial or empty)
Done bool // True when the model's turn is complete
// Transcription fields (populated when transcription is enabled).
InputTranscript string // User speech transcribed by server
InputTranscriptDone bool // True when input transcription segment is final
InputSpeakerLabel string // Optional diarized speaker label for input transcript segments
InputPersonID string // Optional known person id for input transcript segments
InputDisplayName string // Optional known display name for input transcript segments
InputSpeakerConfidence float64 // Optional speaker-label confidence
OutputTranscript string // Model speech transcribed by server
OutputTranscriptDone bool // True when output transcription segment is final
ToolCalls []ToolCall
ToolCallCancellationIDs []string
Interrupted bool // True when user interrupted model (barge-in)
GoAway bool // True when server signals imminent session end
SessionResumable bool // True when the provider indicates reconnect/resume is possible.
}
LiveMessage is a message received from the real-time model.
type LiveModelDescriptor ¶ added in v0.47.0
type LiveModelDescriptor struct {
Provider string `json:"provider"`
ModelID string `json:"modelId"`
Name string `json:"name"`
Lifecycle framework.ModelLifecycle `json:"lifecycle"`
Default bool `json:"default,omitempty"`
Recommended bool `json:"recommended,omitempty"`
SourceURL string `json:"sourceUrl"`
}
type LivePolicies ¶
type LivePolicies struct {
EnableInputAudioTranscription bool
EnableOutputAudioTranscription bool
EnableAffectiveDialog bool
Thinking ThinkingPolicy
ContextCompression ContextCompressionPolicy
ActivityDetection ActivityDetectionPolicy
}
LivePolicies configures Google Live API features that shape Voice Agent behavior.
type LiveProvider ¶
type LiveProvider interface {
// Connect establishes a WebSocket session to the real-time model.
Connect(ctx context.Context, cfg LiveConfig) error
// SendAudio streams PCM audio chunks to the model.
// Format: 16-bit signed int, little-endian, mono, 16kHz.
SendAudio(chunk []byte) error
// SendAudioStreamEnd signals that microphone input for the current turn ended.
SendAudioStreamEnd() error
// Receive blocks until the next server message arrives.
// Returns audio chunks and/or text from the model.
Receive(ctx context.Context) (*LiveMessage, error)
// SendText injects a text prompt into the session (for idle reminders).
SendText(text string) error
// SendToolResponse sends the result of a host-side tool invocation back to the model.
SendToolResponse(response ToolResponse) error
// Close terminates the WebSocket session.
Close() error
// Name returns the provider identifier.
Name() string
}
LiveProvider abstracts a real-time audio-to-audio model connection.
func NewProvider ¶ added in v0.47.0
func NewProvider(providerOrProfile string) (LiveProvider, error)
NewProvider returns a fresh built-in LiveProvider for a provider id, alias, or public realtime profile id.
func NewProviderWithFactories ¶ added in v0.47.0
func NewProviderWithFactories(providerOrProfile string, factories ProviderFactoryRegistry) (LiveProvider, error)
NewProviderWithFactories resolves provider aliases/profile ids against a custom factory registry. Missing registry entries fall back to the built-in providers, so callers can override only the providers they own.
type LiveReconnector ¶
LiveReconnector is an optional interface for providers that support session reconnection.
type LiveSessionCapabilities ¶ added in v0.47.0
type LiveSessionCapabilities interface {
SessionCapabilities() SessionCapabilities
}
type ModelLifecyclePolicy ¶ added in v0.47.0
type ModelLifecyclePolicy string
const ( ModelLifecycleAny ModelLifecyclePolicy = "any" ModelLifecyclePreferGA ModelLifecyclePolicy = "prefer_ga" ModelLifecycleRequireGA ModelLifecyclePolicy = "require_ga" )
type OpenAILive ¶ added in v0.46.0
type OpenAILive struct {
// contains filtered or unexported fields
}
OpenAILive implements LiveProvider against the OpenAI Realtime API (WebSocket). It mirrors GeminiLive's surface so callers don't need to know which backend is active.
func NewOpenAILive ¶ added in v0.46.0
func NewOpenAILive() *OpenAILive
NewOpenAILive returns a fresh OpenAI Realtime provider.
func (*OpenAILive) Close ¶ added in v0.46.0
func (p *OpenAILive) Close() error
Close terminates the WebSocket. Idempotent.
func (*OpenAILive) Connect ¶ added in v0.46.0
func (p *OpenAILive) Connect(ctx context.Context, cfg LiveConfig) error
Connect dials the OpenAI Realtime WebSocket, sends the configured instructions/voice/tools as a session.update, and waits for the session.updated acknowledgement before returning.
func (*OpenAILive) Name ¶ added in v0.46.0
func (p *OpenAILive) Name() string
Name identifies the provider in Voice Agent logs.
func (*OpenAILive) Receive ¶ added in v0.46.0
func (p *OpenAILive) Receive(ctx context.Context) (*LiveMessage, error)
Receive translates the next server event into a LiveMessage. Server events that don't map to LiveMessage fields (session.created/updated, rate-limit telemetry, etc.) are swallowed and the loop fetches the next frame so callers see an aligned event stream.
func (*OpenAILive) SendAudio ¶ added in v0.46.0
func (p *OpenAILive) SendAudio(chunk []byte) error
SendAudio resamples a 16 kHz mic chunk to 24 kHz and forwards it as a base64-encoded input_audio_buffer.append event. Empty chunks are no-ops.
func (*OpenAILive) SendAudioStreamEnd ¶ added in v0.46.0
func (p *OpenAILive) SendAudioStreamEnd() error
SendAudioStreamEnd flushes the input audio buffer and triggers a model response. With server VAD enabled OpenAI commits the buffer automatically at end-of-speech; this explicit commit covers push-to-talk style turns where the kernel decides when the user stopped speaking.
func (*OpenAILive) SendText ¶ added in v0.46.0
func (p *OpenAILive) SendText(text string) error
SendText injects a text-only user turn and triggers a response.
func (*OpenAILive) SendToolResponse ¶ added in v0.46.0
func (p *OpenAILive) SendToolResponse(response ToolResponse) error
SendToolResponse delivers the host-side tool result back to the model and triggers a follow-up response.
func (*OpenAILive) SessionCapabilities ¶ added in v0.47.0
func (p *OpenAILive) SessionCapabilities() SessionCapabilities
func (*OpenAILive) UpdateInstructions ¶ added in v0.46.0
func (p *OpenAILive) UpdateInstructions(ctx context.Context, cfg LiveConfig) error
UpdateInstructions sends a fresh session.update with new instructions/tools. Implements LiveInstructionUpdater so the kernel can refresh persona prompts without forcing a reconnect.
type ProviderDescriptor ¶ added in v0.47.0
type ProviderDescriptor struct {
Provider string `json:"provider"`
DisplayName string `json:"displayName"`
ProfileID string `json:"profileId"`
Capabilities []LiveCapabilityFlag `json:"capabilities"`
Models []LiveModelDescriptor `json:"models,omitempty"`
SupportedLocales []string `json:"supportedLocales,omitempty"`
NativeOptions []provideropts.OptionID `json:"nativeOptions,omitempty"`
AuthRequirement string `json:"authRequirement,omitempty"`
Transport string `json:"transport,omitempty"`
EvidenceURL string `json:"evidenceUrl,omitempty"`
}
func DefaultProviderDescriptors ¶ added in v0.47.0
func DefaultProviderDescriptors() []ProviderDescriptor
DefaultProviderDescriptors returns the public live-provider catalog for embedders that want to switch providers by name/profile/model at runtime.
func FindProviderDescriptor ¶ added in v0.47.0
func FindProviderDescriptor(providerOrProfile string) (ProviderDescriptor, bool)
FindProviderDescriptor resolves provider ids, provider aliases, and profile ids to the canonical public descriptor.
func (ProviderDescriptor) DefaultModel ¶ added in v0.47.0
func (d ProviderDescriptor) DefaultModel() (LiveModelDescriptor, bool)
DefaultModel returns the provider's default live model. If no model is explicitly marked as default, it falls back to the first advertised model.
func (ProviderDescriptor) HasCapability ¶ added in v0.47.0
func (d ProviderDescriptor) HasCapability(flag LiveCapabilityFlag) bool
HasCapability reports whether this provider advertises a capability.
type ProviderFactory ¶ added in v0.47.0
type ProviderFactory func() LiveProvider
ProviderFactory constructs a fresh LiveProvider instance.
type ProviderFactoryRegistry ¶ added in v0.47.0
type ProviderFactoryRegistry map[string]ProviderFactory
ProviderFactoryRegistry lets embedders override or extend provider construction while still using SpeechKit's provider/profile normalization.
func DefaultProviderFactories ¶ added in v0.47.0
func DefaultProviderFactories() ProviderFactoryRegistry
DefaultProviderFactories returns factories for the built-in native realtime providers. The returned map is a copy and can be safely modified by callers.
type ProviderFallback ¶ added in v0.47.0
type ProviderFallback struct {
Kind ProviderFallbackKind `json:"kind"`
Provider string `json:"provider"`
ProfileID string `json:"profileId,omitempty"`
Model string `json:"model,omitempty"`
ModelLifecycle framework.ModelLifecycle `json:"modelLifecycle,omitempty"`
Reason string `json:"reason,omitempty"`
MissingRequiredCapabilities []LiveCapabilityFlag `json:"missingRequiredCapabilities,omitempty"`
MissingRequiredOptions []provideropts.OptionID `json:"missingRequiredOptions,omitempty"`
AuthRequirement string `json:"authRequirement,omitempty"`
Transport string `json:"transport,omitempty"`
EvidenceURL string `json:"evidenceUrl,omitempty"`
}
type ProviderFallbackKind ¶ added in v0.47.0
type ProviderFallbackKind string
const ( FallbackKindSameProviderModel ProviderFallbackKind = "same_provider_model" FallbackKindCrossProvider ProviderFallbackKind = "cross_provider" FallbackKindCascaded ProviderFallbackKind = "cascaded" FallbackKindCapabilityMissing ProviderFallbackKind = "capability_missing" )
type ProviderIntent ¶ added in v0.47.0
type ProviderIntent struct {
Mode string `json:"mode,omitempty"`
Provider string `json:"provider,omitempty"`
ProfileID string `json:"profileId,omitempty"`
Model string `json:"model,omitempty"`
RequiredCapabilities []LiveCapabilityFlag `json:"requiredCapabilities,omitempty"`
PreferredCapabilities []LiveCapabilityFlag `json:"preferredCapabilities,omitempty"`
Requirements []CapabilityRequirement `json:"requirements,omitempty"`
RequiredOptions []provideropts.OptionID `json:"requiredOptions,omitempty"`
PreferredOptions []provideropts.OptionID `json:"preferredOptions,omitempty"`
Locale string `json:"locale,omitempty"`
LanguageHints []string `json:"languageHints,omitempty"`
PrivacyRedaction bool `json:"privacyRedaction,omitempty"`
ResumePreferred bool `json:"resumePreferred,omitempty"`
LatencyProfile LatencyProfile `json:"latencyProfile,omitempty"`
SelectionPolicy ProviderSelectionPolicy `json:"selectionPolicy,omitempty"`
}
type ProviderIntentError ¶ added in v0.47.0
type ProviderIntentError struct {
Intent ProviderIntent `json:"intent"`
MissingRequiredCapabilities []LiveCapabilityFlag `json:"missingRequiredCapabilities,omitempty"`
MissingRequiredOptions []provideropts.OptionID `json:"missingRequiredOptions,omitempty"`
Fallbacks []ProviderFallback `json:"fallbacks,omitempty"`
RejectedProviders []ProviderRejection `json:"rejectedProviders,omitempty"`
}
func (*ProviderIntentError) Error ¶ added in v0.47.0
func (e *ProviderIntentError) Error() string
func (*ProviderIntentError) Unwrap ¶ added in v0.47.0
func (e *ProviderIntentError) Unwrap() error
type ProviderRejection ¶ added in v0.47.0
type ProviderRejection struct {
Provider string `json:"provider"`
ProfileID string `json:"profileId,omitempty"`
Model string `json:"model,omitempty"`
ModelLifecycle framework.ModelLifecycle `json:"modelLifecycle,omitempty"`
FallbackKind ProviderFallbackKind `json:"fallbackKind,omitempty"`
Reason string `json:"reason"`
MissingRequiredCapabilities []LiveCapabilityFlag `json:"missingRequiredCapabilities,omitempty"`
MissingRequiredOptions []provideropts.OptionID `json:"missingRequiredOptions,omitempty"`
AuthRequirement string `json:"authRequirement,omitempty"`
Transport string `json:"transport,omitempty"`
EvidenceURL string `json:"evidenceUrl,omitempty"`
UnsupportedLocale string `json:"unsupportedLocale,omitempty"`
}
type ProviderSelectionPolicy ¶ added in v0.47.0
type ProviderSelectionPolicy struct {
PreferredProviders []string `json:"preferredProviders,omitempty"`
AllowPreview bool `json:"allowPreview,omitempty"`
AllowLegacy bool `json:"allowLegacy,omitempty"`
ModelLifecycle ModelLifecyclePolicy `json:"modelLifecycle,omitempty"`
}
type ResolvedLiveOptions ¶ added in v0.45.0
type ResolvedLiveOptions struct {
Locale string
Voice string
ContextPrompt string
LanguageHints []string
Keyterms []string
ReasoningEffort string
Resume bool
TurnDetection bool
TurnDetectionSource provideropts.ValueSource
EndpointingMs int
EndpointingSource provideropts.ValueSource
Effective provideropts.EffectiveOptions
}
func ResolveLiveOptions ¶ added in v0.45.0
func ResolveLiveOptions(provider, profileID string, cfg LiveConfig, providerDefaults, providerOverrides provideropts.Values) ResolvedLiveOptions
func (ResolvedLiveOptions) HasEndpointingOverride ¶ added in v0.45.0
func (r ResolvedLiveOptions) HasEndpointingOverride() bool
func (ResolvedLiveOptions) HasTurnDetectionOverride ¶ added in v0.45.0
func (r ResolvedLiveOptions) HasTurnDetectionOverride() bool
type ResolvedProviderPlan ¶ added in v0.47.0
type ResolvedProviderPlan struct {
Provider string `json:"provider"`
ProfileID string `json:"profileId"`
Model string `json:"model"`
Descriptor ProviderDescriptor `json:"descriptor"`
ModelDescriptor LiveModelDescriptor `json:"modelDescriptor"`
SelectionReason string `json:"selectionReason,omitempty"`
SelectedFallbackKind ProviderFallbackKind `json:"selectedFallbackKind,omitempty"`
Fallbacks []ProviderFallback `json:"fallbacks,omitempty"`
RejectedProviders []ProviderRejection `json:"rejectedProviders,omitempty"`
MatchedRequiredCapabilities []LiveCapabilityFlag `json:"matchedRequiredCapabilities,omitempty"`
MatchedPreferredCapabilities []LiveCapabilityFlag `json:"matchedPreferredCapabilities,omitempty"`
UnsupportedPreferredCapabilities []LiveCapabilityFlag `json:"unsupportedPreferredCapabilities,omitempty"`
MatchedRequiredOptions []provideropts.OptionID `json:"matchedRequiredOptions,omitempty"`
MatchedPreferredOptions []provideropts.OptionID `json:"matchedPreferredOptions,omitempty"`
UnsupportedPreferredOptions []provideropts.OptionID `json:"unsupportedPreferredOptions,omitempty"`
AuthRequirement string `json:"authRequirement,omitempty"`
Transport string `json:"transport,omitempty"`
LatencyProfile LatencyProfile `json:"latencyProfile,omitempty"`
}
func ResolveProviderIntent ¶ added in v0.47.0
func ResolveProviderIntent(intent ProviderIntent, descriptors []ProviderDescriptor) (ResolvedProviderPlan, error)
func (ResolvedProviderPlan) LiveConfig ¶ added in v0.47.0
func (p ResolvedProviderPlan) LiveConfig() LiveConfig
func (ResolvedProviderPlan) String ¶ added in v0.47.0
func (p ResolvedProviderPlan) String() string
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session manages a Voice Agent conversation.
func NewSession ¶
func NewSession(provider LiveProvider, callbacks Callbacks) *Session
NewSession creates a Voice Agent session with the given provider.
func (*Session) AdvanceWorkflowStep ¶
AdvanceWorkflowStep moves a configured local workflow to its next step and updates the active provider instructions. It returns nil when no workflow is active or the workflow has already completed.
func (*Session) CurrentState ¶
State returns the current session state.
func (*Session) EndAudioStream ¶
EndAudioStream tells the live provider that the current microphone stream ended.
func (*Session) ProviderName ¶
func (*Session) SendToolResponse ¶
func (s *Session) SendToolResponse(response ToolResponse) error
SendToolResponse forwards the result of a host-side tool invocation to the model.
func (*Session) Start ¶
func (s *Session) Start(ctx context.Context, cfg LiveConfig, idleCfg IdleConfig) error
Start activates the Voice Agent session.
type SessionCapabilities ¶ added in v0.47.0
type StartSensitivity ¶
type StartSensitivity string
StartSensitivity controls how aggressively automatic activity detection commits speech start.
const ( StartSensitivityLow StartSensitivity = "low" StartSensitivityMedium StartSensitivity = "medium" StartSensitivityHigh StartSensitivity = "high" )
type ThinkingLevel ¶
type ThinkingLevel string
ThinkingLevel controls how much deliberate reasoning Gemini Live should spend.
const ( ThinkingLevelOff ThinkingLevel = "off" ThinkingLevelLow ThinkingLevel = "low" ThinkingLevelMedium ThinkingLevel = "medium" ThinkingLevelHigh ThinkingLevel = "high" )
type ThinkingPolicy ¶
type ThinkingPolicy struct {
Enabled bool
IncludeThoughts bool
ThinkingBudget int32
ThinkingLevel ThinkingLevel
}
ThinkingPolicy defines optional Gemini Live thinking behavior.
type ToolBehavior ¶
type ToolBehavior string
ToolBehavior controls whether the model waits for a tool result.
const ( ToolBehaviorUnspecified ToolBehavior = "" ToolBehaviorBlocking ToolBehavior = "blocking" ToolBehaviorNonBlocking ToolBehavior = "non_blocking" )
type ToolDefinition ¶
type ToolDefinition struct {
Name string
Description string
ParametersJSONSchema map[string]any
ResponseJSONSchema map[string]any
Behavior ToolBehavior
}
ToolDefinition exposes a host-side action the Voice Agent may call.
type ToolResponse ¶
type ToolResponse struct {
ID string
Name string
Response map[string]any
Scheduling ToolResponseScheduling
WillContinue *bool
}
ToolResponse resolves a previously emitted tool call.
type ToolResponseScheduling ¶
type ToolResponseScheduling string
ToolResponseScheduling controls how a non-blocking tool result is reintroduced into the conversation.
const ( ToolResponseSchedulingUnspecified ToolResponseScheduling = "" ToolResponseSchedulingSilent ToolResponseScheduling = "silent" ToolResponseSchedulingWhenIdle ToolResponseScheduling = "when_idle" ToolResponseSchedulingInterrupt ToolResponseScheduling = "interrupt" )
type TurnCoverage ¶
type TurnCoverage string
TurnCoverage controls how the live API builds a user turn from incoming activity.
const ( TurnCoverageUnspecified TurnCoverage = "" TurnCoverageTurnIncludesOnlyActivity TurnCoverage = "turn_includes_only_activity" TurnCoverageTurnIncludesAllInput TurnCoverage = "turn_includes_all_input" TurnCoverageTurnIncludesAudioActivity TurnCoverage = "turn_includes_audio_activity" )
type WorkflowConfig ¶
type WorkflowConfig struct {
SequenceID string
Completion string
MaxTurns int
BasePrompt string
Steps []WorkflowStep
// InitialStep selects the first active step. Out-of-range values fall
// back to zero.
InitialStep int
}
WorkflowConfig describes a deterministic, step-based Voice Agent behavior sequence. Durations are intentionally expressed as turns instead of wall clock time so tests and local installs can exercise long moderation flows quickly.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package livecontract provides reusable conformance checks for LiveProvider implementations.
|
Package livecontract provides reusable conformance checks for LiveProvider implementations. |