speechkit

package
v0.67.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 3, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package speechkit provides the public SDK for embedding SpeechKit voice capture, transcription, and assist/voice-agent pipelines into host applications.

Surface

The kernel exposes three strict interaction modes:

  • Dictation — speech to text only, no AI rewriting.
  • Assist — speech (or text) to a one-shot result, with optional TTS.
  • Voice Agent — realtime audio-to-audio dialogue.

The Mode enum carries two further constants that are capability surfaces, not interaction modes: ModeTTS exposes Text-to-Speech as a model-selection axis (its IntelligenceVoiceOutput contract is strictly text in, audio out), and ModeNone means no mode is selected.

Each mode is constructed via a small subpackage so host apps depend only on what they use:

Central types in this package

The root package is deliberately limited to contracts and value types: the Mode, Capability, ProviderKind and ExecutionMode enums, the ProviderProfile and ModeSettings descriptors, RuntimePolicy, the pipeline contracts (AudioRecorder, Transcriber, SegmentCollector, TranscriptOutput, TranscriptionObserver, JobSubmitter) and the value types that flow between them (Submission, Transcript, TranscriptionJob, Completion). Runtime owns shared state and the event channel that host apps read from. Everything in this package can be imported by any other SpeechKit package without creating a cycle, so custom providers, collectors and outputs only need this one import.

Implementations live in two sibling packages:

Stability

pkg/speechkit is the OSS public surface. Symbols here follow semver from v1.0 onward. Before v1.0 the surface may still evolve — see CHANGELOG.md and the release notes for breaking-change calls.

Package-level documentation lives in doc.go.

Index

Constants

View Source
const (
	AudioSampleRate     = audio.SampleRate
	AudioChannels       = audio.Channels
	AudioBitsPerSample  = audio.BitsPerSample
	AudioBytesPerSample = audio.BytesPerSample
)
View Source
const (
	NetworkScopeReasonCloudProvider   = "sk.privacy.disabled.cloud_provider"
	NetworkScopeReasonLocalService    = "sk.privacy.disabled.local_service_in_device_only"
	NetworkScopeReasonServerScope     = "sk.privacy.disabled.server_in_device_only"
	NetworkScopeReasonServerNotLocal  = "sk.privacy.disabled.server_url_not_local"
	NetworkScopeReasonAgentBridge     = "sk.privacy.disabled.agent_bridge"
	NetworkScopeReasonHomeAssistant   = "sk.privacy.disabled.home_assistant"
	NetworkScopeReasonEdgeBeta        = "sk.privacy.disabled.edge_beta"
	NetworkScopeReasonSetupTraffic    = "sk.privacy.disabled.setup_traffic"
	NetworkScopeReasonTelemetry       = "sk.privacy.disabled.telemetry"
	NetworkScopeReasonCloudAccount    = "sk.privacy.disabled.cloud_account"
	NetworkScopeReasonVoiceAgentCloud = "sk.privacy.disabled.voice_agent_cloud"
)

Stable, localizable disabled-reason IDs. The backend attaches these to blocked settings/providers; UIs must render the matching catalog message instead of inventing copy.

View Source
const (
	OutcomeEmptyFinalTranscript = "empty_final_transcript"
	OutcomePCMQueueDrop         = "pcm_queue_drop"
	OutcomeAssistEmptySpeak     = "assist_empty_speak"
)

Outcome names for RecordOutcome. Keep these stable — backends and alerts key on the string, not on log message text.

View Source
const (
	// TargetKindWindow is a native OS window the text is injected into.
	TargetKindWindow = "window"
	// TargetKindEditor is an in-app editor or text buffer.
	TargetKindEditor = "editor"
	// TargetKindClipboard means "copy only, do not inject".
	TargetKindClipboard = "clipboard"
	// TargetKindNone is the Output's default destination (a nil target).
	TargetKindNone = ""
)

Well-known target kinds. Hosts are free to add their own.

View Source
const (
	CaptureChannelMicrophone = "mic"
	CaptureChannelSystem     = "system"
)

Capture channels name the audio source behind a recording. A host that records one session from several sources at once — meeting capture takes the microphone and the system loopback in parallel — labels each controller so the resulting transcripts stay attributable.

View Source
const DefaultMinPCMBytes = 3200
View Source
const DefaultProcessingMessage = "Recording stopped · Transcribing"
View Source
const EmptyFinalTranscriptMessage = "No speech recognized · check the configured language"

EmptyFinalTranscriptMessage is shown when a provider returns a successful final transcript containing no text.

This is a named outcome rather than a silent drop because it is the visible half of a real data-loss bug: a provider answers HTTP 200 with a zero-length transcript when the pinned language does not match the speech, so the user's words disappear with nothing to alert on. The message names the most likely cause, since that is the one the user can act on.

View Source
const ReadinessSchemaVersion = "provider-readiness.v1"

Variables

View Source
var ErrCommandHandlerUnavailable = errors.New("speechkit: no command handler configured")

ErrCommandHandlerUnavailable is returned by CommandBus.Dispatch when no command handler has been configured on the Runtime.

View Source
var ErrUnknownNetworkScope = errors.New("speechkit: unknown network scope")

ErrUnknownNetworkScope is returned for scope values outside the known set. Config loading fails closed on it instead of guessing.

View Source
var ErrUnsupportedAudioFormat = errors.New("speechkit: unsupported audio format for this dictation stream provider")

ErrUnsupportedAudioFormat is returned by a DictationStreamProvider when the requested speaker.AudioFormat is one its realtime API cannot accept — e.g. AssemblyAI's v3 streaming API exposes no channel parameter and decodes the socket as a single channel, so stereo would transcribe as braided garbage rather than fail. Providers wrap it with %w.

Format support is per-provider, not a property of the format (Deepgram serves stereo natively), so this is deliberately not a protocol-level validation: the router's fallback loop treats it like any other start failure and tries the next candidate. Only when no provider can serve the format does it reach a caller, who should errors.Is it to report a format problem rather than invite a blind retry of the same doomed format.

Functions

func LowConfidenceWords added in v0.46.0

func LowConfidenceWords(words []WordConfidence, threshold float64) (terms []string, minConfidence float64)

LowConfidenceWords returns the distinct word texts whose per-word confidence is below threshold, together with the minimum confidence observed across all words. A threshold <= 0 disables detection. Words without per-word data (providers that do not expose it) yield (nil, 0). The returned terms are the raw STT tokens, so callers can match them against the (possibly rewritten) display text without depending on character offsets.

func NormalizeProviderProfileID added in v0.42.0

func NormalizeProviderProfileID(profileID string) string

NormalizeProviderProfileID maps legacy profile IDs to their current canonical IDs while preserving unknown custom IDs.

func PCMDurationSecs added in v0.24.0

func PCMDurationSecs(pcm []byte) float64

PCMDurationSecs returns the duration of 16kHz S16 mono PCM audio in seconds.

func PCMToWAV added in v0.24.0

func PCMToWAV(pcm []byte) []byte

PCMToWAV wraps raw 16kHz S16 mono PCM data in a WAV header.

func RecordOutcome added in v0.60.41

func RecordOutcome(ctx context.Context, name string, err error, attrs ...Attr)

RecordOutcome attaches a named framework result to the active span.

This is the vendor-neutral error/outcome seam: callers keep using slog for operators, but user-visible failures also land on the trace so a configured OTLP backend can surface them. With no TracerProvider installed (the local-only default) this is a zero-cost no-op.

func TargetKind added in v0.67.18

func TargetKind(target any) string

TargetKind reports the kind of an arbitrary target value handed through the `Target any` fields: the OutputTarget kind when the value implements it, TargetKindNone for nil, and "opaque" for legacy untyped values so that hosts can spot targets still awaiting migration.

func ValidateModeSettingsForPolicy added in v0.24.0

func ValidateModeSettingsForPolicy(profiles []ProviderProfile, settings ModeSettings, policy RuntimePolicy) error

ValidateModeSettingsForPolicy checks mode selections against a RuntimePolicy.

func ValidateProfileForMode added in v0.24.0

func ValidateProfileForMode(profile ProviderProfile, mode Mode) error

ValidateProfileForMode checks the stable v23 mode capability contract.

func ValidateRuntimePolicy added in v0.24.0

func ValidateRuntimePolicy(profiles []ProviderProfile, policy RuntimePolicy) error

ValidateRuntimePolicy checks that a policy references existing profiles and does not require a profile that violates its mode contract.

Types

type AssistRequest added in v0.24.0

type AssistRequest struct {
	Text              string                     `json:"text"`
	Locale            string                     `json:"locale,omitempty"`
	Selection         string                     `json:"selection,omitempty"`
	Context           string                     `json:"context,omitempty"`
	EditableTarget    bool                       `json:"editableTarget,omitempty"`
	ProviderProfileID string                     `json:"providerProfileId,omitempty"`
	SessionKey        string                     `json:"sessionKey,omitempty"`
	SpeakerOptions    speaker.Options            `json:"speakerOptions,omitempty"`
	Speakers          *speaker.DiarizationResult `json:"speakers,omitempty"`
}

AssistRequest is the mode-scoped input for Assist integrations.

type AssistResult added in v0.24.0

type AssistResult struct {
	Text       string                     `json:"text"`
	SpeakText  string                     `json:"speakText,omitempty"`
	Action     string                     `json:"action,omitempty"`
	Kind       string                     `json:"kind,omitempty"`
	Surface    AssistSurfaceDecision      `json:"surface"`
	ShortcutID string                     `json:"shortcutId,omitempty"`
	Locale     string                     `json:"locale,omitempty"`
	MessageID  localization.MessageID     `json:"messageId,omitempty"`
	ReasonCode string                     `json:"reasonCode,omitempty"`
	Audio      *AudioData                 `json:"audio,omitempty"`
	Format     string                     `json:"format,omitempty"`
	Speakers   *speaker.DiarizationResult `json:"speakers,omitempty"`
}

AssistResult is the public one-shot output contract for Assist Mode.

type AssistService added in v0.24.0

type AssistService interface {
	Process(context.Context, AssistRequest) (AssistResult, error)
}

AssistService is the mode-scoped SDK contract for one-shot utilities and work-product generation.

type AssistSetting added in v0.24.0

type AssistSetting struct {
	ModeSetting
	TTSEnabled      bool   `json:"ttsEnabled"`
	UtilityRegistry string `json:"utilityRegistry,omitempty"`
}

type AssistSurfaceDecision added in v0.24.0

type AssistSurfaceDecision string

AssistSurfaceDecision describes where an Assist result should be presented.

const (
	AssistSurfacePanel     AssistSurfaceDecision = "panel"
	AssistSurfaceInsert    AssistSurfaceDecision = "insert"
	AssistSurfaceReplace   AssistSurfaceDecision = "replace"
	AssistSurfaceActionAck AssistSurfaceDecision = "action_ack"
	AssistSurfaceSilent    AssistSurfaceDecision = "silent"
)

type Attr added in v0.64.3

type Attr struct {
	Key   string
	Value any
}

Attr is one key/value pair attached to a recorded outcome. It exists so RecordOutcome does not put OpenTelemetry in an embedder's own signatures: a host records outcomes with SpeechKit's own vocabulary, and whether a tracing backend is installed stays SpeechKit's business.

func BoolAttr added in v0.64.3

func BoolAttr(key string, v bool) Attr

func Float64Attr added in v0.64.3

func Float64Attr(key string, v float64) Attr

func Int64Attr added in v0.64.3

func Int64Attr(key string, v int64) Attr

func StringAttr added in v0.64.3

func StringAttr(key, value string) Attr

StringAttr, Int64Attr, Float64Attr and BoolAttr build an Attr of the matching type. They read better at a call site than a struct literal and keep the value's type explicit.

type AudioData added in v0.40.1

type AudioData []byte

AudioData carries optional synthesized audio without making AssistResult non-comparable for existing SDK consumers.

func NewAudioData added in v0.40.1

func NewAudioData(data []byte) *AudioData

func (*AudioData) Bytes added in v0.40.1

func (a *AudioData) Bytes() []byte

func (*AudioData) Len added in v0.40.1

func (a *AudioData) Len() int

type AudioIdleObserver added in v0.51.2

type AudioIdleObserver interface {
	IdleAudio() (silence time.Duration, lastFrame time.Time)
}

AudioIdleObserver is implemented by SegmentCollectors that can report silence in audio time: the cumulative duration of processed silent frames since the last detected speech, plus the wall-clock time of the most recently processed frame. When a collector satisfies this interface it is preferred over IdleObserver, because audio-anchored silence is immune to CPU-starvation stalls — when frame delivery stalls, the silence counter freezes instead of counting wall-clock seconds and auto-stopping mid-dictation.

type AudioRecorder

type AudioRecorder interface {
	Start() error
	Stop() ([]byte, error)
	SetPCMHandler(func([]byte))
}

AudioRecorder is the hardware abstraction for microphone capture.

type AudioSegment added in v0.24.0

type AudioSegment struct {
	PCM       []byte
	Duration  time.Duration
	Paragraph bool
	Final     bool
}

AudioSegment is a transcribable utterance extracted from a dictation recording. PCM is raw 16kHz S16 mono audio.

type Capability added in v0.24.0

type Capability string

Capability is a mode capability declared by a provider profile.

const (
	CapabilityTranscription         Capability = "transcription"
	CapabilitySTT                   Capability = "stt"
	CapabilityAudioInput            Capability = "audio_input"
	CapabilityLLM                   Capability = "llm"
	CapabilityTTS                   Capability = "tts"
	CapabilityRealtimeAudio         Capability = "realtime_audio"
	CapabilityPipelineFallback      Capability = "pipeline_fallback"
	CapabilityToolCalling           Capability = "tool_calling"
	CapabilityDictionaryPrompt      Capability = "dictionary_prompt"
	CapabilityDictionaryNativeHints Capability = "dictionary_native_hints"
	CapabilityWordsPrompt           Capability = "words_prompt"
	CapabilityWordsNativeHints      Capability = "words_native_hints"
	CapabilityPostSTTReplacements   Capability = "post_stt_replacements"
	CapabilitySessionSummary        Capability = "session_summary"
	CapabilityTranscript            Capability = "transcript"
	CapabilityInterruptions         Capability = "interruptions"
	CapabilitySessionResume         Capability = "session_resume"
	CapabilityNativeContextPrompt   Capability = "native_context_prompt"
	CapabilityNativeKeyterms        Capability = "native_keyterms"
	CapabilityNativeDictationStream Capability = "native_dictation_stream"
	CapabilityLanguageHints         Capability = "language_hints"
	CapabilitySpeakerStreaming      Capability = "speaker_streaming"
	CapabilityPrivacyRedaction      Capability = "privacy_redaction"
	CapabilityVoiceFocus            Capability = "voice_focus"
	CapabilityMedicalDomain         Capability = "medical_domain"
	CapabilityReasoningEffort       Capability = "reasoning_effort"
	CapabilityTranslation           Capability = "translation"
	CapabilityTranscriptionOnly     Capability = "transcription_only"
	CapabilitySpeakerDiarization    Capability = "speaker_diarization"
	CapabilitySpeakerIdentification Capability = "speaker_identification"
	CapabilitySpeakerAttribution    Capability = "speaker_attribution"
	CapabilitySpeakerEnrollment     Capability = "speaker_enrollment"
)

func RequiredCapabilities added in v0.24.0

func RequiredCapabilities(mode Mode, nativeRealtime bool) []Capability

RequiredCapabilities returns the minimum capability set for a profile to satisfy a mode contract.

type Command

type Command struct {
	Type     CommandType
	Text     string
	NoteID   int64
	Target   string
	Metadata map[string]string
}

Command is a request dispatched through the CommandBus.

func (Command) Clone

func (c Command) Clone() Command

type CommandBus

type CommandBus interface {
	Dispatch(context.Context, Command) error
}

CommandBus delivers Command values to the registered handler.

type CommandType

type CommandType string

CommandType identifies the action a Command requests.

const (
	CommandShowDashboard           CommandType = "dashboard.show"
	CommandStartDictation          CommandType = "dictation.start"
	CommandStopDictation           CommandType = "dictation.stop"
	CommandStartMode               CommandType = "mode.start"
	CommandStopMode                CommandType = "mode.stop"
	CommandSetActiveMode           CommandType = "mode.set_active"
	CommandOpenQuickNote           CommandType = "quicknote.open"
	CommandOpenQuickCapture        CommandType = "quicknote.capture.open"
	CommandCloseQuickCapture       CommandType = "quicknote.capture.close"
	CommandArmQuickNoteRecording   CommandType = "quicknote.record.arm"
	CommandCopyLastTranscription   CommandType = "transcription.copy_last"
	CommandInsertLastTranscription CommandType = "transcription.insert_last"
	CommandSummarizeSelection      CommandType = "selection.summarize"
)

type CommitObserver

type CommitObserver interface {
	OnCommit(completion Completion)
}

CommitObserver is notified after each successful [TranscriptionRunner.Commit].

type Completion

type Completion struct {
	Transcript             Transcript
	QuickNoteCommitted     bool
	QuickNoteCreated       bool
	QuickNoteID            int64
	TranscriptionPersisted bool
	AudioDurationMs        int64
}

Completion describes the outcome of a [TranscriptionRunner.Commit] call.

type CustomizationAction added in v0.47.0

type CustomizationAction struct {
	ReplacementID string         `json:"replacement_id,omitempty"`
	Kind          string         `json:"kind,omitempty"`
	Intent        string         `json:"intent,omitempty"`
	Text          string         `json:"text,omitempty"`
	Template      string         `json:"template,omitempty"`
	Payload       map[string]any `json:"payload,omitempty"`
	MatchedText   string         `json:"matched_text,omitempty"`
	Count         int            `json:"count,omitempty"`
}

CustomizationAction describes a command/snippet/template action produced by Words and Replacements v2. Known command intents can be executed by hosts; unknown intents remain structured metadata for event/API consumers.

type DictationRun added in v0.24.0

type DictationRun struct {
	ID               string                     `json:"id,omitempty"`
	Transcript       Transcript                 `json:"transcript"`
	StartedAt        time.Time                  `json:"startedAt,omitempty"`
	CompletedAt      time.Time                  `json:"completedAt,omitempty"`
	ProviderProfile  string                     `json:"providerProfile,omitempty"`
	DictionaryTerms  []string                   `json:"dictionaryTerms,omitempty"`
	AudioDurationMs  int64                      `json:"audioDurationMs,omitempty"`
	ProcessingTimeMs int64                      `json:"processingTimeMs,omitempty"`
	Speakers         *speaker.DiarizationResult `json:"speakers,omitempty"`
}

DictationRun is the public record produced by a completed Dictation request. Hosts may persist it directly or map it into their own history model.

type DictationService added in v0.24.0

type DictationService interface {
	Start(context.Context) error
	Stop(context.Context) (DictationRun, error)
}

DictationService is the mode-scoped SDK contract for text-only dictation.

type DictationSetting added in v0.24.0

type DictationSetting struct {
	ModeSetting
	DictionaryEnabled bool `json:"dictionaryEnabled"`
}

type DictationStream added in v0.48.0

type DictationStream interface {
	SendPCM(ctx context.Context, pcm []byte) error
	Finalize(ctx context.Context) error
	Receive(ctx context.Context) (DictationStreamEvent, error)
	Close() error
}

DictationStream is a provider-neutral realtime dictation session.

type DictationStreamEvent added in v0.48.0

type DictationStreamEvent struct {
	Sequence       int64
	SessionID      uint64
	SegmentID      uint64
	ProviderItemID string
	Text           string
	IsFinal        bool
	Language       string
	Provider       string
	Model          string
	Confidence     float64
	Words          []WordConfidence
	Speakers       *speaker.DiarizationResult
}

DictationStreamEvent is the provider-neutral event emitted by DictationStream.Receive.

func (DictationStreamEvent) Transcript added in v0.48.0

func (e DictationStreamEvent) Transcript() Transcript

type DictationStreamOptions added in v0.48.0

type DictationStreamOptions struct {
	SessionID         uint64
	ProviderProfileID string
	Language          string
	Model             string
	InterimResults    bool
	EndpointingMs     int
	TurnDetection     bool
	Keyterms          []string
	PromptHint        string
	Diarization       bool
}

DictationStreamOptions configures provider-native live transcription for Dictation and meeting transcription. It is intentionally separate from speaker.Options because plain dictation must not require diarization.

type DictationStreamProvider added in v0.48.0

type DictationStreamProvider interface {
	StartDictationStream(ctx context.Context, opts DictationStreamOptions, format speaker.AudioFormat) (DictationStream, error)
}

DictationStreamProvider is implemented by STT providers that can consume raw PCM frames and emit draft/final transcript events without waiting for a completed WAV upload.

type DictationStreamSink added in v0.48.0

type DictationStreamSink interface {
	HandleDictationStreamEvent(ctx context.Context, event DictationStreamEvent, opts DictationStreamSinkOptions) error
}

DictationStreamSink consumes provider-native live dictation events. Drafts are allowed to update UI state, but only final events may reach output or persistence.

type DictationStreamSinkOptions added in v0.48.0

type DictationStreamSinkOptions struct {
	// Target is the host delivery target of the final transcript; see
	// [OutputTarget]. Untyped values are accepted until the field becomes
	// OutputTarget in v0.69.0.
	Target             any
	QuickNote          bool
	QuickNoteID        int64
	Language           string
	RecordingSessionID int64
	// CaptureChannel names the capture source feeding this stream, and
	// CaptureEpoch is the wall clock the session's timeline is measured from.
	// Sinks stamp final events with the elapsed offset so parallel channels of
	// one meeting can be interleaved. A zero epoch disables the timeline.
	CaptureChannel string
	CaptureEpoch   time.Time
}

DictationStreamSinkOptions carries host metadata needed to commit final provider-stream events through the same path as batch transcription.

type EndpointClass added in v0.66.25

type EndpointClass string

EndpointClass groups outbound destinations by why SpeechKit dials them. The scope decides per class whether the dial is allowed and how strictly the URL and every resolved IP must be validated.

const (
	// EndpointClassManagedLoopback is a child process SpeechKit spawned
	// itself and reaches over loopback (whisper-server, managed llama-server,
	// local TTS engines).
	EndpointClassManagedLoopback EndpointClass = "managed_loopback"

	// EndpointClassLocalService is an external service the user points
	// SpeechKit at (Ollama, an OpenAI-compatible endpoint, Home Assistant,
	// a SpeechKit server target).
	EndpointClassLocalService EndpointClass = "local_service"

	// EndpointClassTelemetry is the optional OTLP audit/trace exporter.
	EndpointClassTelemetry EndpointClass = "telemetry"

	// EndpointClassCloud is any public SaaS endpoint (provider APIs, the
	// kombify cloud account, update/download hosts outside setup consent).
	EndpointClassCloud EndpointClass = "cloud"
)

type EndpointPolicy added in v0.66.25

type EndpointPolicy struct {
	// Allowed reports whether this class may be dialed at all.
	Allowed bool
	// ReasonID is the stable disabled reason ID when Allowed is false.
	ReasonID string
	// Enforce reports whether Validation must be applied to the URL and to
	// every resolved IP at dial time. False in the open scope, where call
	// sites keep their pre-existing endpoint-specific validation.
	Enforce bool
	// Validation is the netsec option set to enforce when Enforce is true.
	Validation netsec.ValidationOptions
}

EndpointPolicy is the per-scope decision for one endpoint class.

type Engine

type Engine interface {
	Start(context.Context) error
	Stop(context.Context) error
	Events() <-chan Event
	Commands() CommandBus
	State() Snapshot
}

Engine is the interface implemented by a full SpeechKit voice pipeline.

type Event

type Event struct {
	Type                 EventType
	Time                 time.Time
	Message              string
	Text                 string
	Provider             string
	Mode                 string
	SessionID            string
	QuickNote            bool
	Err                  error
	Shortcut             string
	Metadata             *Metadata
	CustomizationActions []CustomizationAction
}

Event is a notification published to the event channel returned by Runtime.Events. Consumers should switch on Type and inspect the relevant fields.

func (Event) Clone added in v0.40.1

func (e Event) Clone() Event

type EventType

type EventType string

EventType identifies the kind of event published to the event channel.

const (
	EventStateChanged            EventType = "state.changed"
	EventRecordingStarted        EventType = "recording.started"
	EventProcessingStarted       EventType = "processing.started"
	EventTranscriptionDraft      EventType = "transcription.draft"
	EventTranscriptionReady      EventType = "transcription.ready"
	EventTranscriptCommitted     EventType = "transcription.committed"
	EventQuickNoteModeArmed      EventType = "quicknote.mode_armed"
	EventQuickNoteUpdated        EventType = "quicknote.updated"
	EventWarningRaised           EventType = "warning.raised"
	EventErrorRaised             EventType = "error.raised"
	EventShortcutMatched         EventType = "shortcut.matched"
	EventWakeFired               EventType = "wake.fired"
	EventSkillExecuted           EventType = "skill.executed"
	EventCompanionSessionStarted EventType = "companion.session.started"
	EventCompanionSessionEnded   EventType = "companion.session.ended"
	EventVoiceAgentTurnFinalized EventType = "voiceagent.turn.finalized"
	EventTTSStarted              EventType = "tts.started"
	EventTTSFinished             EventType = "tts.finished"
	EventCustomizationAction     EventType = "customization.action"
)

type ExecutionMode added in v0.24.0

type ExecutionMode string

ExecutionMode describes the technical runtime behind a provider profile.

const (
	ExecutionModeLocal          ExecutionMode = "local"
	ExecutionModeSelfHostedHTTP ExecutionMode = "self_hosted_http"
	ExecutionModeHFRouted       ExecutionMode = "hf_routed"
	ExecutionModeOpenAI         ExecutionMode = "openai_api"
	ExecutionModeGroq           ExecutionMode = "groq_api"
	ExecutionModeGoogle         ExecutionMode = "google_api"
	ExecutionModeDeepgram       ExecutionMode = "deepgram_api"
	ExecutionModeAssemblyAI     ExecutionMode = "assemblyai_api"
	ExecutionModeOllama         ExecutionMode = "ollama_local"
	ExecutionModeOpenRouter     ExecutionMode = "openrouter_api"
	ExecutionModeFoundry        ExecutionMode = "foundry_api"
)

type Hooks

type Hooks struct {
	Start         func(context.Context) error
	Stop          func(context.Context) error
	HandleCommand func(context.Context, Command) error
}

Hooks are the lifecycle callbacks wired into a Runtime. Nil hooks are silently skipped.

type IdleObserver added in v0.35.21

type IdleObserver interface {
	IdleSince() time.Time
}

IdleObserver is implemented by SegmentCollectors that want to drive silence-based auto-stop. Returning the zero value tells the watcher "user is actively speaking; reset the timer." Returning a non-zero time tells the watcher "user has been silent since T."

type IntelligenceKind added in v0.24.0

type IntelligenceKind string

IntelligenceKind names the mode-specific intelligence contract.

const (
	IntelligenceUser          IntelligenceKind = "user"
	IntelligenceUtility       IntelligenceKind = "utility"
	IntelligenceBrainstorming IntelligenceKind = "brainstorming"
	// IntelligenceVoiceOutput is the contract for the TTS mode: render
	// generated text to audio. No user intelligence, no utility tools,
	// no brainstorming — strictly text in, audio out.
	IntelligenceVoiceOutput IntelligenceKind = "voice_output"
)

type JobSubmitter

type JobSubmitter interface {
	Submit(TranscriptionJob) error
}

JobSubmitter accepts a TranscriptionJob for async processing.

type Metadata added in v0.40.1

type Metadata map[string]string

Metadata carries optional event key/value data without making Event non-comparable for existing SDK consumers.

func NewMetadata added in v0.40.1

func NewMetadata(values map[string]string) *Metadata

func (*Metadata) Clone added in v0.40.1

func (m *Metadata) Clone() *Metadata

func (*Metadata) Get added in v0.40.1

func (m *Metadata) Get(key string) string

func (*Metadata) Map added in v0.40.1

func (m *Metadata) Map() map[string]string

type Modality added in v0.62.1

type Modality string

Modality classifies what a catalog entry does, independent of the three user-facing modes. Every profile a user can select maps onto a Mode as well; support entries a host needs but a user never picks — embeddings, rerankers, utility models — only have a Modality.

const (
	ModalitySTT           Modality = "stt"
	ModalityTTS           Modality = "tts"
	ModalityRealtimeVoice Modality = "realtime_voice"
	ModalityAssist        Modality = "assist"
	ModalityUtility       Modality = "utility"
	ModalityEmbedding     Modality = "embedding"
	ModalityReranker      Modality = "reranker"
)

func ModalityForMode added in v0.62.1

func ModalityForMode(mode Mode) Modality

ModalityForMode returns the modality a user-facing mode runs as, or "" for a mode that has none.

type Mode added in v0.24.0

type Mode string

Mode identifies one of SpeechKit's strict product modes.

const (
	ModeNone       Mode = "none"
	ModeDictation  Mode = "dictation"
	ModeAssist     Mode = "assist"
	ModeVoiceAgent Mode = "voice_agent"
	// ModeTTS exposes Text-to-Speech as a first-class model-selection axis
	// alongside the three product modes. The host activates an Assist or
	// Voice-Agent session and the TTS profile selected here drives which
	// provider speaks the response. v0.37 introduced this alongside the
	// Voice-Companion hands-free flow — Thalia + Companion Live need a
	// stable place to pin a TTS voice across deployments.
	ModeTTS Mode = "tts"
)

func ModeForModality added in v0.62.1

func ModeForModality(modality Modality) Mode

ModeForModality returns the user-facing mode a modality is selectable in, or ModeNone for support modalities a user never picks directly.

func NormalizeMode added in v0.24.0

func NormalizeMode(mode Mode) Mode

type ModeBehavior added in v0.24.0

type ModeBehavior string

ModeBehavior describes how much mode-specific intelligence a host enables.

const (
	// ModeBehaviorClean keeps a mode on its core contract, such as strict STT
	// for Dictation or deterministic utility handling for Assist.
	ModeBehaviorClean ModeBehavior = "clean"
	// ModeBehaviorIntelligence allows optional intelligence layers such as
	// LLM utility handling, TTS, summaries, or realtime tool use.
	ModeBehaviorIntelligence ModeBehavior = "intelligence"
)

type ModeContract added in v0.24.0

type ModeContract struct {
	Mode         Mode             `json:"mode"`
	Intelligence IntelligenceKind `json:"intelligence"`
	Input        string           `json:"input"`
	Output       string           `json:"output"`
	Allowed      []Capability     `json:"allowed"`
	Forbidden    []Capability     `json:"forbidden"`
}

ModeContract documents what a mode may and may not do. Hosts can use this to validate custom adapters before exposing them to users.

func DefaultModeContracts added in v0.24.0

func DefaultModeContracts() []ModeContract

type ModeSetting added in v0.24.0

type ModeSetting struct {
	Enabled           bool   `json:"enabled"`
	Hotkey            string `json:"hotkey,omitempty"`
	HotkeyBehavior    string `json:"hotkeyBehavior,omitempty"`
	PrimaryProfileID  string `json:"primaryProfileId,omitempty"`
	FallbackProfileID string `json:"fallbackProfileId,omitempty"`
	// ModeSource is "local" (default) or "server". When "server", this mode
	// runs against the speechkit-server pointed to by ServerConnection
	// instead of the in-process Framework kernel. Empty/missing is treated
	// as "local" for backwards compatibility with pre-0.26 hosts.
	ModeSource string `json:"modeSource,omitempty"`
}

ModeSetting is the public per-mode configuration shape used by the SDK and the versioned HTTP control plane.

type ModeSettings added in v0.24.0

type ModeSettings struct {
	Dictation        DictationSetting        `json:"dictation"`
	Assist           AssistSetting           `json:"assist"`
	VoiceAgent       VoiceAgentSetting       `json:"voiceAgent"`
	ServerConnection ServerConnectionSetting `json:"serverConnection"`
}

type ModelLifecycle added in v0.47.0

type ModelLifecycle string

ModelLifecycle classifies how a provider model ID should be treated by hosts: generally available, preview, legacy (still served but superseded) or deprecated (scheduled for removal). The catalog package assigns lifecycles to its model registry rows.

const (
	ModelLifecycleGA         ModelLifecycle = "ga"
	ModelLifecyclePreview    ModelLifecycle = "preview"
	ModelLifecycleLegacy     ModelLifecycle = "legacy"
	ModelLifecycleDeprecated ModelLifecycle = "deprecated"
)

type ModelVariant added in v0.24.0

type ModelVariant struct {
	ID          string `json:"id"`
	Name        string `json:"name"`
	ModelID     string `json:"modelId"`
	Description string `json:"description,omitempty"`
	Recommended bool   `json:"recommended,omitempty"`
}

ModelVariant is a concrete model choice inside a provider profile group.

type NetworkScope added in v0.66.25

type NetworkScope string

NetworkScope selects one of the three privacy/operating modes.

const (
	// NetworkScopeOpen is the default: every configured provider and
	// integration may be used, including public cloud endpoints.
	NetworkScopeOpen NetworkScope = "open"

	// NetworkScopeLocalNetwork permits on-device runtimes plus services
	// reachable via loopback or private/link-local addresses (e.g. Ollama on
	// another LAN machine, a self-hosted SpeechKit server). Public cloud
	// endpoints are blocked, including at dial time after DNS resolution.
	NetworkScopeLocalNetwork NetworkScope = "local_network"

	// NetworkScopeDeviceOnly permits only runtimes the SpeechKit app itself
	// manages as local child processes (whisper.cpp, managed llama-server,
	// local TTS engines, ONNX wake-word/VAD). Every external service — even
	// one on the same machine, such as a user-run Ollama — is blocked because
	// its own egress cannot be attested.
	NetworkScopeDeviceOnly NetworkScope = "device_only"
)

func NormalizeNetworkScope added in v0.66.25

func NormalizeNetworkScope(raw string) NetworkScope

NormalizeNetworkScope is the runtime-safe variant of ParseNetworkScope: unparseable values collapse to the strictest scope so an invalid value can never widen network access.

func ParseNetworkScope added in v0.66.25

func ParseNetworkScope(raw string) (NetworkScope, error)

ParseNetworkScope maps a raw config/API value onto a NetworkScope. The empty string is the backwards-compatible default (open); unknown values return ErrUnknownNetworkScope so callers fail closed instead of silently widening or narrowing the scope.

func (NetworkScope) AllowsProfile added in v0.66.25

func (s NetworkScope) AllowsProfile(p ProviderProfile) (bool, string)

AllowsProfile reports whether the catalog profile may be activated under this scope, with the disabled reason ID when it may not.

func (NetworkScope) AllowsProviderKind added in v0.66.25

func (s NetworkScope) AllowsProviderKind(kind ProviderKind) (bool, string)

AllowsProviderKind reports whether providers of the given kind may run under this scope. When blocked, the second return value is the stable disabled reason ID for the UI.

func (NetworkScope) EndpointPolicyFor added in v0.66.25

func (s NetworkScope) EndpointPolicyFor(class EndpointClass) EndpointPolicy

EndpointPolicyFor returns the dial policy for an endpoint class under this scope. Unknown scopes fail closed to device_only behaviour.

func (NetworkScope) Restricted added in v0.66.25

func (s NetworkScope) Restricted() bool

Restricted reports whether s blocks public cloud endpoints.

func (NetworkScope) Valid added in v0.66.25

func (s NetworkScope) Valid() bool

Valid reports whether s is one of the three known scopes.

type OutputTarget added in v0.67.18

type OutputTarget interface {
	// TargetKind names the target family. It must be stable and lower-case
	// (for example "window", "editor", "clipboard").
	TargetKind() string
}

OutputTarget identifies where a transcript or Assist result is delivered.

The framework never inspects a target: it travels unchanged from the RecordingStartOptions (or the Assist tool call) that produced the audio to the TranscriptOutput and TranscriptInterceptor that consume the result. Hosts therefore define their own concrete targets — a native window handle, an editor buffer, a chat channel — and implement this interface so that logs, audit events and interceptors can name the target family without type-asserting host internals.

Recognised kinds are documented in docs/speechkit-framework-api.md; hosts may introduce further kinds, prefixed with their product name (for example "companion.chat").

type Persistence

type Persistence interface {
	QuickNoteStore
	TranscriptionStore
}

Persistence combines QuickNoteStore and TranscriptionStore.

type ProviderKind added in v0.24.0

type ProviderKind string

ProviderKind is the product-facing provider group shown for every mode.

const (
	ProviderKindLocalBuiltIn   ProviderKind = "local_built_in"
	ProviderKindLocalProvider  ProviderKind = "local_provider"
	ProviderKindCloudProvider  ProviderKind = "cloud_provider"
	ProviderKindDirectProvider ProviderKind = "direct_provider"
)

type ProviderProfile added in v0.24.0

type ProviderProfile struct {
	ID   string `json:"id"`
	Mode Mode   `json:"mode"`
	// Modality is what the entry does. ProviderProfileWithDefaults derives it
	// from Mode when unset, and derives Mode from it for support entries that
	// carry no mode.
	Modality         Modality       `json:"modality,omitempty"`
	Name             string         `json:"name"`
	ProviderKind     ProviderKind   `json:"providerKind"`
	ExecutionMode    ExecutionMode  `json:"executionMode,omitempty"`
	Provider         string         `json:"provider,omitempty"`
	ModelID          string         `json:"modelId,omitempty"`
	Lifecycle        ModelLifecycle `json:"lifecycle,omitempty"`
	Source           string         `json:"source,omitempty"`
	Description      string         `json:"description,omitempty"`
	License          string         `json:"license,omitempty"`
	Capabilities     []Capability   `json:"capabilities,omitempty"`
	SupportedLocales []string       `json:"supportedLocales,omitempty"`
	NativeOptions    []string       `json:"nativeOptions,omitempty"`
	AuthRequirement  string         `json:"authRequirement,omitempty"`
	Transport        string         `json:"transport,omitempty"`
	EvidenceURL      string         `json:"evidenceUrl,omitempty"`
	AdapterKind      string         `json:"adapterKind,omitempty"`
	Variants         []ModelVariant `json:"variants,omitempty"`
	AllowInference   bool           `json:"inferenceAllowed,omitempty"`
	Default          bool           `json:"default,omitempty"`
	Recommended      bool           `json:"recommended,omitempty"`
	Experimental     bool           `json:"experimental,omitempty"`
}

ProviderProfile is the public catalog entry host applications can present or activate. ProviderKind is the stable user-facing grouping; ExecutionMode is the technical adapter underneath it.

func FilterProviderProfiles added in v0.24.0

func FilterProviderProfiles(profiles []ProviderProfile, policy RuntimePolicy) []ProviderProfile

FilterProviderProfiles returns the profiles visible under policy.

func (ProviderProfile) HasCapability added in v0.24.0

func (p ProviderProfile) HasCapability(capability Capability) bool

type QuickNoteStore

type QuickNoteStore interface {
	SaveQuickNote(ctx context.Context, text, language, provider string, durationMs, latencyMs int64, audioData []byte) (int64, error)
	GetQuickNoteText(ctx context.Context, id int64) (string, error)
	UpdateQuickNote(ctx context.Context, id int64, text string) error
	UpdateQuickNoteCapture(ctx context.Context, id int64, text, provider string, durationMs, latencyMs int64, audioData []byte) error
}

QuickNoteStore persists and retrieves Quick Note records.

type Readiness added in v0.24.0

type Readiness struct {
	SchemaVersion    string        `json:"schemaVersion,omitempty"`
	ProfileID        string        `json:"profileId"`
	Mode             Mode          `json:"mode"`
	ProviderKind     ProviderKind  `json:"providerKind"`
	ExecutionMode    ExecutionMode `json:"executionMode,omitempty"`
	ModelID          string        `json:"modelId,omitempty"`
	Source           string        `json:"source,omitempty"`
	Active           bool          `json:"active"`
	Default          bool          `json:"default"`
	Configured       bool          `json:"configured"`
	CredentialsReady bool          `json:"credentialsReady"`
	RuntimeReady     bool          `json:"runtimeReady"`
	CapabilityReady  bool          `json:"capabilityReady"`
	Ready            bool          `json:"ready"`
	// BlockedByScope reports that the active NetworkScope forbids this
	// profile. When true, DisabledReasonID carries the stable localizable
	// reason ID and Ready is forced false. Absent/false on targets without
	// a network scope (e.g. the Server-Target catalog).
	BlockedByScope   bool                   `json:"blockedByScope,omitempty"`
	DisabledReasonID string                 `json:"disabledReasonId,omitempty"`
	Missing          []string               `json:"missing,omitempty"`
	Requirements     []ReadinessRequirement `json:"requirements,omitempty"`
	Actions          []ReadinessAction      `json:"actions,omitempty"`
	Artifacts        []ReadinessArtifact    `json:"artifacts,omitempty"`
}

Readiness describes whether a provider profile can be used right now.

type ReadinessAction added in v0.24.0

type ReadinessAction struct {
	ID     string `json:"id"`
	Label  string `json:"label"`
	Kind   string `json:"kind"`
	Target string `json:"target,omitempty"`
}

ReadinessAction describes the next setup command a host can expose when a requirement is not ready.

type ReadinessArtifact added in v0.24.0

type ReadinessArtifact struct {
	ID             string `json:"id"`
	Name           string `json:"name"`
	Kind           string `json:"kind"`
	SizeLabel      string `json:"sizeLabel,omitempty"`
	SizeBytes      int64  `json:"sizeBytes,omitempty"`
	Available      bool   `json:"available"`
	Selected       bool   `json:"selected"`
	RuntimeReady   bool   `json:"runtimeReady,omitempty"`
	RuntimeProblem string `json:"runtimeProblem,omitempty"`
	Recommended    bool   `json:"recommended,omitempty"`
}

ReadinessArtifact describes downloadable or pullable model artifacts tied to a provider profile. Local Built-in profiles use this to expose concrete model choices through the same readiness API as credentials and runtime checks.

type ReadinessRequirement added in v0.24.0

type ReadinessRequirement struct {
	ID       string `json:"id"`
	Label    string `json:"label"`
	Category string `json:"category"`
	Required bool   `json:"required"`
	Ready    bool   `json:"ready"`
	Missing  string `json:"missing,omitempty"`
}

ReadinessRequirement is a machine-readable setup check for a provider profile. Hosts can render these checks directly instead of hard-coding provider-specific setup rules.

type RecordingCancelOptions added in v0.46.0

type RecordingCancelOptions struct {
	Label string
}

RecordingCancelOptions controls cancellation of an active recording without submitting captured audio. It is for host-level interruptions such as a mode switch, where the old buffer must be discarded rather than transcribed.

type RecordingObserver

type RecordingObserver interface {
	OnState(status, text string)
	OnLog(message, kind string)
}

type RecordingStartOptions

type RecordingStartOptions struct {
	// Context scopes provider-native streaming sessions. When nil,
	// context.Background is used.
	Context context.Context
	Label   string
	// Target is the host delivery target carried to every transcript this
	// recording produces. Pass a value implementing [OutputTarget]; untyped
	// values are accepted until the field becomes OutputTarget in v0.69.0.
	Target      any
	Language    string
	QuickNote   bool
	QuickNoteID int64
	// RecordingSessionID links final transcript commits to a persisted
	// long-running dictation or meeting session owned by the host.
	RecordingSessionID int64
	// CaptureChannel names the audio source this controller records, so a host
	// running several controllers over one recording session — meeting capture
	// records the microphone and the system loopback at the same time — can tell
	// the resulting transcripts apart. See CaptureChannel*.
	CaptureChannel string
	// CaptureEpoch is the wall clock that transcript timestamps are measured
	// from. Hosts recording one session across several controllers pass the same
	// epoch to all of them so the transcripts interleave on a single timeline.
	// Zero means "start of this recording".
	CaptureEpoch time.Time
	// StreamSegments enables live-ish dictation for this recording session.
	// Completed pause-bounded segments are queued before Stop(); Stop() then
	// flushes only pending/remaining tail segments. Leave false for Assist and
	// Voice Agent fallback capture, where a single full turn is the safer unit.
	StreamSegments bool
	// ProviderStream enables provider-native realtime dictation when the host
	// configured a DictationStreamProvider and DictationStreamSink. If stream
	// startup fails, the controller keeps using StreamSegments/full-capture
	// fallback behavior.
	ProviderStream bool
	// DictationStreamOptions are passed to the native provider stream. SessionID
	// and Language are filled from the active recording when left empty.
	DictationStreamOptions DictationStreamOptions
	// LiveCommitMode groups provider-finals before field injection.
	// Empty keeps immediate commit (tests and hosts that do not opt in).
	// Desktop dictation defaults to LiveCommitPassage.
	LiveCommitMode string
	// IdleTimeout, when greater than zero AND the underlying collector
	// implements [IdleObserver], arms a watcher that calls
	// OnIdleTimeoutCallback once the user has been silent for this long.
	// Zero (default) disables the watcher — typical for hold-to-talk
	// hotkey sessions that already terminate on KeyUp.
	IdleTimeout time.Duration
	// OnIdleTimeoutCallback fires once if IdleTimeout elapses without
	// observed speech. Wired by the host to dispatch a Stop command so
	// the dictate session ends after a silence window. The watcher
	// guarantees at-most-one invocation per Start() call.
	OnIdleTimeoutCallback func()
}

type RecordingStopOptions

type RecordingStopOptions struct {
	Label string
	// TailDelay keeps the recorder physically open for a short grace period
	// after a stop request. Hold-to-talk callers use this to avoid clipping
	// final syllables when the shortcut is released a few milliseconds early.
	TailDelay time.Duration
}

type Runtime

type Runtime struct {
	// contains filtered or unexported fields
}

Runtime manages shared observable state and event delivery for a SpeechKit session. Create one with NewRuntime and wire it into the host application via Runtime.Events and Runtime.Commands.

func NewRuntime

func NewRuntime(initial Snapshot, hooks Hooks) *Runtime

func (*Runtime) Close

func (r *Runtime) Close()

func (*Runtime) Commands

func (r *Runtime) Commands() CommandBus

func (*Runtime) Events

func (r *Runtime) Events() <-chan Event

Events returns the runtime's event channel. The channel is buffered with 64 slots; Runtime.Publish never blocks and drops events once the buffer is full, so consumers must drain the channel promptly to avoid losing events.

func (*Runtime) Publish

func (r *Runtime) Publish(event Event) bool

Publish delivers event to the channel returned by Runtime.Events. It never blocks: the event channel is buffered with 64 slots, and when the buffer is full (or the runtime is closed, or event.Type is empty) the event is silently dropped. The bool return reports whether the event was actually delivered to the buffer. Slow consumers must drain the events channel promptly, or events will be lost.

func (*Runtime) SetState

func (r *Runtime) SetState(snapshot Snapshot)

func (*Runtime) Start

func (r *Runtime) Start(ctx context.Context) error

func (*Runtime) State

func (r *Runtime) State() Snapshot

func (*Runtime) Stop

func (r *Runtime) Stop(ctx context.Context) error

func (*Runtime) UpdateState

func (r *Runtime) UpdateState(update func(*Snapshot)) Snapshot

type RuntimePolicy added in v0.24.0

type RuntimePolicy struct {
	EnabledModes    []Mode                `json:"enabledModes,omitempty"`
	AllowedProfiles []string              `json:"allowedProfiles,omitempty"`
	FixedProfiles   map[Mode]string       `json:"fixedProfiles,omitempty"`
	AllowFallbacks  bool                  `json:"allowFallbacks,omitempty"`
	ModeBehaviors   map[Mode]ModeBehavior `json:"modeBehaviors,omitempty"`
}

RuntimePolicy constrains which parts of the SpeechKit framework a host application exposes. Empty EnabledModes or AllowedProfiles mean "all".

type SegmentCollector

type SegmentCollector interface {
	FeedPCM([]byte) error
	CollectStopSegments(fullPCM []byte) ([]AudioSegment, error)
}

SegmentCollector accumulates real-time PCM frames and splits them into dictation segments when recording stops.

type SegmentCollectorFactory

type SegmentCollectorFactory func() SegmentCollector

type ServerConnectionSetting added in v0.26.0

type ServerConnectionSetting struct {
	Enabled              bool                     `json:"enabled"`
	ActiveTargetID       string                   `json:"activeTargetId,omitempty"`
	URL                  string                   `json:"url"`
	BearerTokenEnv       string                   `json:"bearerTokenEnv,omitempty"`
	AuthMode             string                   `json:"authMode,omitempty"`
	BetaInstallIDEnv     string                   `json:"betaInstallIdEnv,omitempty"`
	BetaInstallSecretEnv string                   `json:"betaInstallSecretEnv,omitempty"`
	BearerTokenSet       bool                     `json:"bearerTokenSet"`
	FallbackToLocal      bool                     `json:"fallbackToLocal"`
	RequestTimeoutSec    int                      `json:"requestTimeoutSec"`
	Targets              []ServerConnectionTarget `json:"targets,omitempty"`
}

ServerConnectionSetting exposes the [server_connection] config section to the control-plane API + frontend. The bearer token is never sent across this boundary — only the env var name + connection metadata.

type ServerConnectionTarget added in v0.31.0

type ServerConnectionTarget struct {
	ID                   string `json:"id"`
	Label                string `json:"label"`
	URL                  string `json:"url"`
	AuthMode             string `json:"authMode"`
	BearerTokenEnv       string `json:"bearerTokenEnv,omitempty"`
	BetaInstallIDEnv     string `json:"betaInstallIdEnv,omitempty"`
	BetaInstallSecretEnv string `json:"betaInstallSecretEnv,omitempty"`
	BearerTokenSet       bool   `json:"bearerTokenSet"`
	FallbackToLocal      bool   `json:"fallbackToLocal"`
	RequestTimeoutSec    int    `json:"requestTimeoutSec"`
}

type Snapshot

type Snapshot struct {
	Status                string
	Text                  string
	Level                 float64
	Hotkey                string
	ActiveMode            string
	Providers             []string
	ActiveProfiles        map[string]string
	Transcriptions        int
	QuickNoteMode         bool
	QuickCaptureMode      bool
	LastTranscriptionText string
}

Snapshot is a point-in-time copy of the Runtime's observable state. All slice and map fields are safe to read without holding any lock.

func (Snapshot) Clone

func (s Snapshot) Clone() Snapshot

type Submission

type Submission struct {
	PCM          []byte
	WAV          []byte
	DurationSecs float64
	Language     string
	Prefix       string
	QuickNote    bool
	QuickNoteID  int64
	SessionID    uint64
	SegmentID    uint64
	// RecordingSessionID is copied into the final Transcript and Completion so
	// host observers can attach committed text to a long-running session after
	// persistence/output succeeds.
	RecordingSessionID int64
	// CaptureChannel, CapturedStartMs and CapturedEndMs carry the capture
	// source and wall-clock placement through to the Transcript. See the
	// matching Transcript fields.
	CaptureChannel  string
	CapturedStartMs int64
	CapturedEndMs   int64
	// ProviderItemID carries provider-native turn/item IDs for realtime
	// streams. Segment-batch jobs leave it empty and rely on SessionID+SegmentID.
	ProviderItemID string
	SegmentFinal   bool
	// QueuedAt is set by TranscriptionWorker.Submit when the segment enters
	// the worker queue. Hosts can prefill it when replaying externally queued
	// work, but ordinary callers should leave it zero.
	QueuedAt time.Time
}

Submission carries a single audio segment and its metadata into the transcription pipeline.

type TargetRef added in v0.67.18

type TargetRef struct {
	Kind string
	ID   string
}

TargetRef is a ready-made OutputTarget for hosts that address targets by a stable string ID instead of a native handle.

func (TargetRef) TargetKind added in v0.67.18

func (t TargetRef) TargetKind() string

type Transcriber

type Transcriber interface {
	Transcribe(ctx context.Context, audio []byte, durationSecs float64, language string) (Transcript, error)
}

Transcriber converts raw WAV audio into a Transcript.

It is the host-facing contract: the dictation runtime, TranscriptionWorker and pipelines consume a Transcriber and nothing more specific. Provider implementations satisfy stt.STTProvider instead and are bridged with stt.AsTranscriber; a host never implements both.

type Transcript

type Transcript struct {
	Text       string
	Language   string
	Duration   time.Duration
	Provider   string
	Model      string
	Confidence float64
	// Session metadata is set for progressive dictation/meeting pipelines.
	// Draft transcripts may be replaced by later revisions; only final
	// segment IDs are committed by the worker ledger.
	SessionID      uint64
	SegmentID      uint64
	ProviderItemID string
	SegmentFinal   bool
	// RecordingSessionID links this transcript to a persisted long-running
	// dictation or meeting session in the host store.
	RecordingSessionID int64
	// CaptureChannel names the capture source this transcript came from (see
	// CaptureChannel*). Sessions that record more than one source at once —
	// meeting capture runs the microphone and the system loopback in parallel —
	// use it to keep the two apart. Empty for single-source captures.
	CaptureChannel string
	// CapturedStartMs and CapturedEndMs place this transcript on the capture
	// session's wall-clock timeline, relative to RecordingStartOptions.
	// CaptureEpoch. Both are zero when the host did not request a timeline.
	CapturedStartMs int64
	CapturedEndMs   int64
	// Words carries per-word acoustic confidence when available (Deepgram,
	// AssemblyAI). Used to surface likely-misrecognized terms; nil otherwise.
	Words                []WordConfidence
	Speakers             *speaker.DiarizationResult
	CustomizationActions []CustomizationAction `json:"customization_actions,omitempty"`
}

Transcript holds the result of a single transcription call.

type TranscriptInterceptor

type TranscriptInterceptor interface {
	Intercept(ctx context.Context, transcript Transcript, target any) (bool, error)
}

TranscriptInterceptor can handle a transcript before it reaches the normal output path. Return (true, nil) to signal that the transcript was consumed.

type TranscriptOutput

type TranscriptOutput interface {
	Deliver(ctx context.Context, transcript Transcript, target any) error
}

TranscriptOutput delivers a completed Transcript to the host application (e.g. clipboard injection or text-field paste).

type TranscriptSegmentKey added in v0.48.0

type TranscriptSegmentKey struct {
	RecordingSessionID int64
	CaptureChannel     string
	SessionID          uint64
	SegmentID          uint64
	ProviderItemID     string
}

TranscriptSegmentKey uniquely identifies a final transcript unit inside a progressive dictation or meeting session.

CaptureChannel is part of the identity because a meeting records several sources at once and each source numbers its own segments from one. Without it the two channels collide on their first segment and one of them is discarded as a duplicate of the other. RecordingSessionID separates fresh controllers, whose process-local counters restart from one, across durable meetings.

func (TranscriptSegmentKey) IsZero added in v0.48.0

func (k TranscriptSegmentKey) IsZero() bool

type TranscriptTransformer added in v0.45.0

type TranscriptTransformer interface {
	Transform(ctx context.Context, transcript Transcript) (Transcript, error)
}

TranscriptTransformer can apply final post-STT changes after all audio segments have been transcribed and merged, but before command routing or user-visible output.

type TranscriptionDraftObserver added in v0.48.0

type TranscriptionDraftObserver interface {
	OnTranscriptDraft(transcript Transcript)
}

TranscriptionDraftObserver is optionally implemented by observers that can surface live provider draft text. Drafts are never passed to output handlers.

type TranscriptionJob

type TranscriptionJob struct {
	Submission
	Segments []Submission
	// Target is the host delivery target forwarded unchanged to
	// [TranscriptOutput.Deliver] and [TranscriptInterceptor.Intercept]. Pass a
	// value implementing [OutputTarget]; untyped values are accepted until the
	// field becomes OutputTarget in v0.69.0.
	Target any
}

TranscriptionJob pairs a Submission with its delivery target.

func (TranscriptionJob) Clone

func (TranscriptionJob) EffectiveSegments added in v0.67.14

func (j TranscriptionJob) EffectiveSegments() []Submission

EffectiveSegments returns the submissions the worker should transcribe: Segments when the recorder produced several, otherwise the single Submission.

type TranscriptionObserver

type TranscriptionObserver interface {
	OnState(status, text string)
	OnLog(message, kind string)
	OnTranscriptCommitted(transcript Transcript, quickNote bool)
}

TranscriptionObserver receives real-time status and log updates from a [TranscriptionWorker] during processing.

type TranscriptionStore

type TranscriptionStore interface {
	SaveTranscription(ctx context.Context, text, language, provider, model string, durationMs, latencyMs int64, audioData []byte) error
}

TranscriptionStore persists completed dictation transcriptions.

type VoiceActivityDetector added in v0.24.0

type VoiceActivityDetector interface {
	ProcessFrame([]int16) (float32, error)
	Reset()
}

VoiceActivityDetector is the public VAD contract consumed by DictationSegmenter. It intentionally matches SpeechKit's internal Silero detector shape without exposing internal packages.

type VoiceAgentService added in v0.24.0

type VoiceAgentService interface {
	Start(context.Context) error
	Stop(context.Context) (VoiceAgentSession, error)
	SendText(context.Context, string) error
	CurrentSession(context.Context) (VoiceAgentSession, error)
}

VoiceAgentService is the mode-scoped SDK contract for realtime dialogue.

type VoiceAgentSession added in v0.24.0

type VoiceAgentSession struct {
	ID                string                     `json:"id,omitempty"`
	StartedAt         time.Time                  `json:"startedAt,omitempty"`
	EndedAt           time.Time                  `json:"endedAt,omitempty"`
	Locale            string                     `json:"locale,omitempty"`
	ProviderProfileID string                     `json:"providerProfileId,omitempty"`
	RuntimeKind       string                     `json:"runtimeKind,omitempty"`
	Turns             []VoiceAgentTurn           `json:"turns,omitempty"`
	Summary           VoiceAgentSessionSummary   `json:"summary"`
	Speakers          *speaker.DiarizationResult `json:"speakers,omitempty"`
}

VoiceAgentSession is the public record for a live dialogue.

type VoiceAgentSessionSummary added in v0.24.0

type VoiceAgentSessionSummary struct {
	Title         string   `json:"title,omitempty"`
	Summary       string   `json:"summary"`
	Ideas         []string `json:"ideas,omitempty"`
	Decisions     []string `json:"decisions,omitempty"`
	OpenQuestions []string `json:"openQuestions,omitempty"`
	NextSteps     []string `json:"nextSteps,omitempty"`
	RawText       string   `json:"rawText,omitempty"`
}

VoiceAgentSessionSummary is the structured handoff produced when a Voice Agent session ends.

type VoiceAgentSetting added in v0.24.0

type VoiceAgentSetting struct {
	ModeSetting
	SessionSummary   bool   `json:"sessionSummary"`
	PipelineFallback bool   `json:"pipelineFallback"`
	CloseBehavior    string `json:"closeBehavior,omitempty"`
	AgentProfileID   string `json:"agentProfileId,omitempty"`
	AgentSequenceID  string `json:"agentSequenceId,omitempty"`
}

type VoiceAgentTurn added in v0.24.0

type VoiceAgentTurn struct {
	Role              string    `json:"role"`
	Text              string    `json:"text"`
	CreatedAt         time.Time `json:"createdAt,omitempty"`
	SpeakerLabel      string    `json:"speakerLabel,omitempty"`
	PersonID          string    `json:"personId,omitempty"`
	DisplayName       string    `json:"displayName,omitempty"`
	SpeakerConfidence float64   `json:"speakerConfidence,omitempty"`
}

VoiceAgentTurn is one finalized turn in a realtime or fallback dialogue.

type WordConfidence added in v0.46.0

type WordConfidence struct {
	Text       string
	Confidence float64
	StartMs    int64
	EndMs      int64
}

WordConfidence is a recognized word with the provider's per-word acoustic confidence in [0,1]. It mirrors stt.WordConfidence (the kernel keeps Transcript decoupled from the stt package; stt.AsTranscriber is the public bridge that maps between them). nil when the provider does not expose word-level confidence.

Directories

Path Synopsis
Package agentbridge defines the framework-neutral seam through which a SpeechKit host fronts an external coding agent (adopted 2026-08-10, AI-VOICE-SPEECHKIT-TARGET.md "External Coding Agent Bridge").
Package agentbridge defines the framework-neutral seam through which a SpeechKit host fronts an external coding agent (adopted 2026-08-10, AI-VOICE-SPEECHKIT-TARGET.md "External Coding Agent Bridge").
codex
Package codex drives the official OpenAI Codex binary as an agentbridge.Agent.
Package codex drives the official OpenAI Codex binary as an agentbridge.Agent.
voicetools
Package voicetools binds an agentbridge.Agent to the voice agent's tool surface with the "Call GPT" semantics (owner decision 2026-08-10, AI-VOICE-SPEECHKIT-TARGET.md External Coding Agent Bridge):
Package voicetools binds an agentbridge.Agent to the voice agent's tool surface with the "Call GPT" semantics (owner decision 2026-08-10, AI-VOICE-SPEECHKIT-TARGET.md External Coding Agent Bridge):
Package agentkit provides a small Go harness for building SpeechKit Voice Agent hosts.
Package agentkit provides a small Go harness for building SpeechKit Voice Agent hosts.
Package assist provides an embeddable Assist Mode service.
Package assist provides an embeddable Assist Mode service.
genkitadapter
Package genkitadapter keeps Genkit-specific Assist wiring out of the core public assist package.
Package genkitadapter keeps Genkit-specific Assist wiring out of the core public assist package.
skills
Package skills exposes SpeechKit's Voice-Companion skill catalog — Time, Date, Math, Weather, Timer, Reminder, Wikipedia, plus a fail-closed Home Assistant boundary — as a public assist.ToolMatcher + assist.ToolExecutor pair, ready to plug into an assist.Service.
Package skills exposes SpeechKit's Voice-Companion skill catalog — Time, Date, Math, Weather, Timer, Reminder, Wikipedia, plus a fail-closed Home Assistant boundary — as a public assist.ToolMatcher + assist.ToolExecutor pair, ready to plug into an assist.Service.
toolbridge
Package toolbridge adapts Assist-mode tools (assist.ToolMatcher / assist.ToolExecutor — the deterministic skill layer, e.g.
Package toolbridge adapts Assist-mode tools (assist.ToolMatcher / assist.ToolExecutor — the deterministic skill layer, e.g.
Package audio provides the shared PCM audio primitives for the SpeechKit capture format: 16kHz S16 mono constants, WAV framing, duration math, and RMS level estimation.
Package audio provides the shared PCM audio primitives for the SpeechKit capture format: 16kHz S16 mono constants, WAV framing, duration math, and RMS level estimation.
capture
Package capture is the public microphone / system-audio capture layer of the SpeechKit framework: backend registry (RegisterBackend, Open), capture Session contract, device enumeration (ListCaptureDevices, ListOutputDevices), and the pooled PCM frame buffers (FramePool).
Package capture is the public microphone / system-audio capture layer of the SpeechKit framework: backend registry (RegisterBackend, Open), capture Session contract, device enumeration (ListCaptureDevices, ListOutputDevices), and the pooled PCM frame buffers (FramePool).
Package catalog ships the built-in SpeechKit provider and model catalog.
Package catalog ships the built-in SpeechKit provider and model catalog.
Package client provides a typed HTTP client for talking to a remote SpeechKit Server (the `cmd/speechkit-server` Linux container or any compatible deployment).
Package client provides a typed HTTP client for talking to a remote SpeechKit Server (the `cmd/speechkit-server` Linux container or any compatible deployment).
Package companion provides small composers for hands-free SpeechKit hosts.
Package companion provides small composers for hands-free SpeechKit hosts.
Package customize defines SpeechKit's public Words/Replacements contract.
Package customize defines SpeechKit's public Words/Replacements contract.
Package deviceagent implements the credential-minimal LAN-side SpeechKit device-agent client and its versioned wire contract.
Package deviceagent implements the credential-minimal LAN-side SpeechKit device-agent client and its versioned wire contract.
Package dictation provides an embeddable strict Dictation runtime.
Package dictation provides an embeddable strict Dictation runtime.
Package hostconfig turns a SpeechKit TOML configuration file into the public SDK types an embedding host drives the framework with: a speechkit.ModeSettings (which modes are on, their hotkeys and selected provider profiles) and a permissive speechkit.RuntimePolicy (which modes the host exposes and whether fallbacks are allowed).
Package hostconfig turns a SpeechKit TOML configuration file into the public SDK types an embedding host drives the framework with: a speechkit.ModeSettings (which modes are on, their hotkeys and selected provider profiles) and a permissive speechkit.RuntimePolicy (which modes the host exposes and whether fallbacks are allowed).
internal
Package lifecycle owns mode start/stop orchestration and refcounted shared dependencies for SpeechKit hosts.
Package lifecycle owns mode start/stop orchestration and refcounted shared dependencies for SpeechKit hosts.
Package localization resolves stable SpeechKit message IDs against the repository-owned locale catalogs.
Package localization resolves stable SpeechKit message IDs against the repository-owned locale catalogs.
Package netsec provides centralized network security primitives used by every HTTP-based provider in SpeechKit (STT, TTS, LLM, downloads).
Package netsec provides centralized network security primitives used by every HTTP-based provider in SpeechKit (STT, TTS, LLM, downloads).
Package pipeline is the composable capture-to-transcript engine of SpeechKit.
Package pipeline is the composable capture-to-transcript engine of SpeechKit.
Package procguard ties long-lived child processes to the lifetime of the process that spawned them.
Package procguard ties long-lived child processes to the lifetime of the process that spawned them.
Package provideropts defines SpeechKit's provider-neutral voice option vocabulary and the manifest/resolve types used by concrete provider adapters.
Package provideropts defines SpeechKit's provider-neutral voice option vocabulary and the manifest/resolve types used by concrete provider adapters.
Package speaker defines SpeechKit's public speaker diarization and attribution contracts.
Package speaker defines SpeechKit's public speaker diarization and attribution contracts.
Package storage defines the public storage-backend contract: backend capabilities and metadata, install/device/user/tenant scopes with their enforcement policies, and the configuration shape hosts use to construct a backend.
Package storage defines the public storage-backend contract: backend capabilities and metadata, install/device/user/tenant scopes with their enforcement policies, and the configuration shape hosts use to construct a backend.
stt
Package stt defines the SpeechKit speech-to-text provider interface and houses the concrete provider implementations: whisper.cpp (local built-in), HuggingFace, OpenAI, Groq, Google, an OpenAI-compatible adapter (covers Ollama and other compatible servers), and the self-hosted VPS adapter.
Package stt defines the SpeechKit speech-to-text provider interface and houses the concrete provider implementations: whisper.cpp (local built-in), HuggingFace, OpenAI, Groq, Google, an OpenAI-compatible adapter (covers Ollama and other compatible servers), and the self-hosted VPS adapter.
allproviders
Package allproviders is the batteries-included STT assembly layer: it knows every provider SpeechKit ships and turns a host's resolved credentials into a ready stt.Router.
Package allproviders is the batteries-included STT assembly layer: it knows every provider SpeechKit ships and turns a host's resolved credentials into a ready stt.Router.
sttcontract
Package sttcontract provides a reusable conformance suite that every stt.STTProvider implementation is expected to satisfy.
Package sttcontract provides a reusable conformance suite that every stt.STTProvider implementation is expected to satisfy.
vps
Package vps is the self-hosted whisper-server provider for SpeechKit: an OpenAI-compatible endpoint a user runs themselves, so audio never reaches a commercial provider.
Package vps is the self-hosted whisper-server provider for SpeechKit: an OpenAI-compatible endpoint a user runs themselves, so audio never reaches a commercial provider.
tts
Package tts exposes the embeddable SpeechKit text-to-speech surface.
Package tts exposes the embeddable SpeechKit text-to-speech surface.
ttscontract
Package ttscontract provides a reusable conformance suite that every tts.Provider implementation is expected to satisfy.
Package ttscontract provides a reusable conformance suite that every tts.Provider implementation is expected to satisfy.
Package ttsroute holds the single source of truth that maps a Voice-Output profile ID (e.g.
Package ttsroute holds the single source of truth that maps a Voice-Output profile ID (e.g.
Package voiceagent provides an embeddable Voice Agent service.
Package voiceagent provides an embeddable Voice Agent service.
cascaded
Package cascaded implements a turn-based STT -> LLM -> TTS voice agent provider.
Package cascaded implements a turn-based STT -> LLM -> TTS voice agent provider.
live
Package live exposes the low-level Voice Agent realtime-protocol types.
Package live exposes the low-level Voice Agent realtime-protocol types.
live/foundry
Package foundry adapts the OpenAI Realtime provider to Microsoft Foundry.
Package foundry adapts the OpenAI Realtime provider to Microsoft Foundry.
live/livecontract
Package livecontract provides reusable conformance checks for LiveProvider implementations.
Package livecontract provides reusable conformance checks for LiveProvider implementations.
local
Package local implements voiceagent.Provider on top of an in-process live session — realtime voice agents (Deepgram Voice Agent, Gemini Live, OpenAI Realtime, AssemblyAI, cascaded) without a speechkit-server.
Package local implements voiceagent.Provider on top of an in-process live session — realtime voice agents (Deepgram Voice Agent, Gemini Live, OpenAI Realtime, AssemblyAI, cascaded) without a speechkit-server.
Package wakeword exposes embeddable SpeechKit wake-word contracts.
Package wakeword exposes embeddable SpeechKit wake-word contracts.
sherpa
Package sherpa exposes the sherpa-onnx wake-word detector adapter.
Package sherpa exposes the sherpa-onnx wake-word detector adapter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL