Documentation
¶
Overview ¶
Package cascaded implements a turn-based STT -> LLM -> TTS voice agent provider. It is the self-hosted-friendly alternative to real-time providers like Gemini Live and Moshi.
The package is platform-neutral pure Go. Server-Target (Linux container) and Device-Target (Windows Wails reference UI) both wrap it: see internal/server/voiceagent.CascadedProvider for the Linux server's adapter, and internal/voiceagent.LocalVoiceAgentProvider for the Device-Target's adapter.
Turn detection uses energy-based silence detection (RMS of the 16 kHz S16 PCM stream). Callers that want tighter control can send an audio_end frame which triggers immediate processing regardless of the silence heuristic.
Index ¶
- Variables
- func ChunkAudio(data []byte, size int) [][]byte
- func ChunkRMS(pcm []byte) float64
- func PCMDurationMs(pcm []byte) int64
- type Agent
- type AgentInput
- type AgentOutput
- type Config
- type Deps
- type Message
- type Provider
- func (p *Provider) Close() error
- func (p *Provider) Connect(ctx context.Context, cfg SessionConfig) error
- func (p *Provider) Name() string
- func (p *Provider) Receive(ctx context.Context) (*Message, error)
- func (p *Provider) SendAudio(chunk []byte) error
- func (p *Provider) SendAudioStreamEnd() error
- func (p *Provider) SendText(text string) error
- func (p *Provider) UpdateInstructions(ctx context.Context, cfg SessionConfig) error
- type STT
- type SessionConfig
- type SpeakerStreamer
- type TTS
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotConfigured is returned by Connect when a required dependency // (STT or Agent) was not supplied to New. ErrNotConfigured = errors.New("cascaded: dependency not configured") // ErrClosed is returned by Receive once the provider has been closed. ErrClosed = errors.New("cascaded: provider closed") )
Functions ¶
func ChunkAudio ¶
ChunkAudio splits data into chunks of at most size bytes. Returns a single-element slice when data fits in one chunk.
func ChunkRMS ¶
ChunkRMS computes the RMS level (0.0-1.0) of an S16LE PCM buffer. Matches the formula in internal/audio/pcm.PCMLevel; duplicated here to avoid pulling internal/audio into this package's import set.
func PCMDurationMs ¶
PCMDurationMs returns the duration of a 16 kHz S16 mono PCM buffer in milliseconds.
Types ¶
type Agent ¶
type Agent interface {
Run(ctx context.Context, input AgentInput) (AgentOutput, error)
}
Agent is the LLM surface the provider uses. In production this is the Genkit agent flow defined by flows.DefineAgentFlow.
type AgentInput ¶
type AgentInput struct {
Utterance string `json:"utterance"`
Locale string `json:"locale,omitempty"`
Selection string `json:"selection,omitempty"`
LastTranscription string `json:"lastTranscription,omitempty"`
SystemPrompt string `json:"systemPrompt,omitempty"`
}
AgentInput is the request passed to the Agent (LLM) for one turn. It mirrors the kernel's agent-flow input; embedders building a custom Agent fill these fields from the conversation.
type AgentOutput ¶
type AgentOutput struct {
Text string `json:"text"`
Action string `json:"action"` // "paste", "display", "silent"
}
AgentOutput is the Agent's response for one turn.
type Config ¶
type Config struct {
// SilenceRMSThreshold is the RMS level (0.0-1.0) below which a frame
// is considered silence. Default 0.02.
SilenceRMSThreshold float64
// SilenceTurnMs is the minimum silence duration before the current
// turn is committed. Default 800 ms.
SilenceTurnMs int
// MinTurnMs is the minimum accumulated non-silence needed to treat a
// buffer as a real turn. Default 300 ms. Filters coughs and clicks.
MinTurnMs int
// MaxTurnMs caps a runaway turn before forcing processing. Default
// 30_000 ms.
MaxTurnMs int
// HistoryTurns is how many (user, assistant) turns to keep as rolling
// context for the agent flow. Default 5.
HistoryTurns int
// TTSFormat is the audio format the TTS provider should emit.
// Default "mp3".
TTSFormat string
// TTSSpeed is the TTS speech rate. Default 1.0.
TTSSpeed float64
}
Config tunes turn detection and processing. Zero values are filled by NewProvider with sensible defaults via WithDefaults().
func (Config) WithDefaults ¶
WithDefaults returns a copy of cfg with every zero field replaced by a production default.
type Deps ¶
type Deps struct {
STT STT
Agent Agent
TTS TTS
SpeakerStreamer speaker.StreamingProvider
Config Config
}
Deps bundles everything the bootstrap hands to the provider.
type Message ¶
type Message struct {
Audio []byte
InputTranscript string
InputTranscriptDone bool
InputSpeakerLabel string
InputPersonID string
InputDisplayName string
InputSpeakerConfidence float64
OutputTranscript string
OutputTranscriptDone bool
}
Message is the subset of voice-agent message fields the cascaded path emits. Adapters wrap this into their richer LiveMessage type when forwarding to clients.
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider is a turn-based STT -> LLM -> TTS voice agent. It implements the small contract documented on this type's methods; adapters in internal/server/voiceagent and internal/voiceagent wrap it into the richer LiveProvider interface their callers expect.
func NewProvider ¶
NewProvider constructs a provider without starting background work. Connect() initializes the goroutine that performs turn processing.
func (*Provider) Connect ¶
func (p *Provider) Connect(ctx context.Context, cfg SessionConfig) error
Connect validates that the required dependencies are satisfied and starts the processor loop. Unlike Gemini Live, no external handshake happens.
func (*Provider) SendAudio ¶
SendAudio appends PCM to the current turn buffer and triggers processing when a silence boundary is reached.
func (*Provider) SendAudioStreamEnd ¶
SendAudioStreamEnd forces the current buffer to be treated as a complete turn, even if silence has not yet been detected.
func (*Provider) SendText ¶
SendText injects a text turn (skipping STT). Useful for testing and for clients that already have a transcript from their own STT.
func (*Provider) UpdateInstructions ¶
func (p *Provider) UpdateInstructions(ctx context.Context, cfg SessionConfig) error
UpdateInstructions changes future-turn host instructions without creating a synthetic user turn.
type STT ¶
type STT interface {
Route(ctx context.Context, audio []byte, audioDurationSecs float64, opts stt.TranscribeOpts) (*stt.Result, error)
}
STT is the STT surface the provider uses. In production this is the same *internal/router.Router that serves /v1/dictation/transcribe.
type SessionConfig ¶
type SessionConfig struct {
Locale string
Voice string
SystemPrompt string
RefinementPrompt string
Speaker speaker.Options
}
SessionConfig is the minimal per-session configuration the cascaded provider needs. Adapters in internal/server/voiceagent and internal/voiceagent translate their richer config types into this when calling Connect / UpdateInstructions.
type SpeakerStreamer ¶
type SpeakerStreamer = speaker.StreamingProvider
SpeakerStreamer is the optional realtime speaker attribution add-on.
Source Files