dictation

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 9, 2026 License: MIT Imports: 28 Imported by: 0

Documentation

Overview

Package dictation implements speech-to-text capture and transcription for the composer: press a key, talk, and the transcript lands in the input for review — never auto-fired at the agent.

Two pipelines share a keybinding and a composer-insertion target:

  • Batch: record to a temp WAV file (Recorder.Start/Stop), then transcribe in one shot (Transcriber.Transcribe) — a local sherpa-onnx-offline exec or a single HTTPS POST to Groq/OpenAI. The only pipeline on Termux.
  • Streaming: capture raw PCM from the mic tool's stdout (Recorder.StartStreaming), feed 50ms chunks to a websocket (Transcriber.StreamTranscribe), and render partial transcripts live.

Every OS integration shells out via exec.Command with discrete argv — never a shell string, never CGO — matching internal/imageinput/clipboard.go. The external binaries (arecord/sox/ffmpeg/termux-microphone-record, sherpa-onnx) are looked up on PATH and degrade to a clear setup message when missing, matching internal/lsp's philosophy for language servers.

Index

Constants

View Source
const (
	ProviderLocal  = "local"
	ProviderGroq   = "groq"
	ProviderOpenAI = "openai"
)

Batch transcription providers (config value stt.provider).

View Source
const (
	StreamProviderLocal    = "local"
	StreamProviderDeepgram = "deepgram"
	StreamProviderOpenAI   = "openai"
)

Streaming transcription providers (config value stt.streamProvider).

View Source
const (
	// DefaultSampleRate is what the local engine and Deepgram consume.
	// OpenAI Realtime's pcm16 requires 24kHz; its transcriber constructs the
	// recorder accordingly.
	DefaultSampleRate = 16000
	// DefaultMaxDuration is the hard runaway-recording cap (§12), applied on
	// every platform, batch and streaming. Overridable via
	// stt.maxDurationSeconds.
	DefaultMaxDuration = 5 * time.Minute
)
View Source
const DefaultServerPort = 6006

DefaultServerPort is the localhost port the sherpa-onnx streaming server binds.

View Source
const DefaultSherpaVersion = "v1.13.3"

DefaultSherpaVersion is the pinned, smoke-tested release used unless stt.engineVersion overrides it. "latest" or any tag also works.

Variables

View Source
var ErrStreamingUnsupported = errors.New("this transcription provider does not support streaming")

ErrStreamingUnsupported reports that a batch-only Transcriber cannot stream; the caller falls back to the batch pipeline.

Functions

func AutoDownloadSupported

func AutoDownloadSupported() bool

AutoDownloadSupported reports whether the current platform has a known engine asset (so the F9 flow can offer the download). Termux/Android and unusual arches install manually or use a cloud provider.

func BatchProviders

func BatchProviders() []string

BatchProviders lists valid stt.provider values, for config validation.

func ChunkLevel

func ChunkLevel(chunk []byte) float64

ChunkLevel returns a perceptual 0..1 loudness for a little-endian PCM16 chunk, driving the live recording waveform from the actual microphone input. It uses an RMS→dBFS→linear mapping over [levelFloorDB, levelCeilDB].

func EngineDownloaded

func EngineDownloaded(destRoot, version string) bool

EngineDownloaded reports whether the shared sherpa-onnx engine is already on disk for this platform (one engine serves every model), so the picker can drop the engine's size from a model's download total once it's been fetched.

func ModelDownloaded

func ModelDownloaded(destRoot, dirName string) bool

ModelDownloaded reports whether a model variant is already extracted under destRoot (so the picker can mark it as installed). dirName is the variant's DirName.

func RequiredSampleRate

func RequiredSampleRate(t Transcriber) int

RequiredSampleRate returns the capture rate a Transcriber needs.

func StreamProviders

func StreamProviders() []string

StreamProviders lists valid stt.streamProvider values, for config validation.

Types

type AudioFormat

type AudioFormat string

AudioFormat identifies a recorded container format, sniffed from the bytes — desktop recorders produce WAV, Termux's recorder produces M4A/AAC.

const (
	FormatWAV     AudioFormat = "wav"
	FormatM4A     AudioFormat = "m4a"
	FormatUnknown AudioFormat = ""
)

func SniffFormat

func SniffFormat(data []byte) AudioFormat

SniffFormat detects the audio container from magic bytes. WAV is "RIFF....WAVE"; MP4-family (Termux's .m4a) has "ftyp" at offset 4.

func (AudioFormat) FileName

func (f AudioFormat) FileName() string

FileName returns a transcription-upload filename for the format.

type AuthError

type AuthError struct {
	Provider string
	Message  string
}

AuthError reports that a cloud provider rejected a transcription request for authentication reasons (HTTP 401/403) — a missing or invalid API key. It is a fixable credential problem, so callers (the TUI) can offer an inline key prompt for Provider instead of a dead-end error. Message is the already- classified, secret-redacted user-facing text.

func (*AuthError) Error

func (e *AuthError) Error() string

type CloudConfig

type CloudConfig struct {
	Provider string // ProviderGroq or ProviderOpenAI, for error messages
	BaseURL  string // e.g. https://api.groq.com/openai/v1
	APIKey   string
	Model    string // e.g. whisper-large-v3-turbo, whisper-1
	// Language optionally constrains recognition (ISO-639-1). Empty = auto.
	Language string

	// HTTPClient is injectable for tests; nil uses providerio's shared,
	// stall-hardened client.
	HTTPClient *http.Client
}

CloudConfig configures a batch cloud transcriber. Groq and OpenAI share the OpenAI-compatible /audio/transcriptions multipart endpoint — the design's "each owns its endpoint/model specifics while sharing the request shape" (§6). Key and base URL are resolved by the caller (TUI layer) from config/credstore, keeping this type decoupled and unit-testable.

type DeepgramConfig

type DeepgramConfig struct {
	APIKey     string
	Model      string // default "nova-3"
	Language   string
	SampleRate int // capture rate; default DefaultSampleRate (16kHz)
	// BaseURL overrides the wss endpoint (tests point it at a fake server).
	BaseURL string
}

DeepgramConfig configures the Deepgram streaming transcriber (§6b, default cloud streaming provider). The key is resolved by the caller.

type DownloadOptions

type DownloadOptions struct {
	// DestRoot is where extracted engines/models live (e.g. ~/.config/zero/stt).
	DestRoot string
	// EngineVersion selects the sherpa-onnx release tag ("" → DefaultSherpaVersion;
	// "latest" and any tag also work).
	EngineVersion string
	// HTTPClient is injectable for tests; nil uses a plain client.
	HTTPClient *http.Client
	// APIBase overrides the GitHub API base (tests). "" → api.github.com.
	APIBase string
	// Model selection (default: the Moonshine-tiny int8 batch model). Set these
	// from a ModelVariant to download a different model.
	ModelAssetName    string
	ModelPinnedDigest string
	ModelDirName      string
	// ModelLabel is the friendly model name shown in progress ("Zipformer 20M").
	ModelLabel string
	// Progress receives live status strings for the UI, including a percentage
	// while downloading (e.g. "Engine 45% · 57/126 MB").
	Progress func(status string)
	// contains filtered or unexported fields
}

DownloadOptions configures EnsureLocalEngine.

type EngineComponents

type EngineComponents struct {
	BinaryPath string // extracted sherpa-onnx-offline
	ServerPath string // extracted sherpa-onnx-online-websocket-server
	ModelPath  string // extracted model directory
}

EngineComponents identifies the resolved local-engine paths.

func EnsureLocalEngine

func EnsureLocalEngine(ctx context.Context, opts DownloadOptions) (EngineComponents, error)

EnsureLocalEngine downloads (if absent) and verifies the sherpa-onnx engine and the default model into DestRoot, returning the resolved paths. Idempotent: an already-extracted, still-present engine/model is reused without re-downloading. Every archive is SHA256-verified against the GitHub API digest before extraction.

type LocalConfig

type LocalConfig struct {
	// Binary is the sherpa-onnx-offline executable; looked up on PATH when a
	// bare name. Empty defaults to "sherpa-onnx-offline".
	Binary string
	// ModelPath is the directory holding the model's .onnx + tokens.txt files
	// (stt.localModelPath).
	ModelPath  string
	NumThreads int
	// contains filtered or unexported fields
}

LocalConfig configures the offline sherpa-onnx batch transcriber. Every tier (default/standard/high-end, §7) is just a different ModelPath — there are no tier-specific code paths; the model family is detected from the files present.

type ModelVariant

type ModelVariant struct {
	ID          string
	Label       string
	Description string
	Bytes       int64
	AssetName   string
	Digest      string
	DirName     string
	Streaming   bool
	// Recommended marks a curated, checksum-pinned, validated model (shown first).
	Recommended bool
}

ModelVariant is a downloadable model the /stt-model picker offers, with a pinned SHA256 (the model release predates GitHub's per-asset digests, so these must be pinned). Batch variants use the offline binary; the streaming variant (a transducer) drives the websocket server for a live transcript.

func ListModels

func ListModels(ctx context.Context, client *http.Client, apiBase string) ([]ModelVariant, error)

ListModels fetches every transcription model in the sherpa-onnx asr-models release and returns them as variants (the curated ones flagged Recommended and carrying a pinned digest; the rest with an empty digest, downloaded with TLS-only verification since that release predates GitHub's per-asset digests). Non-ASR assets (VAD, TTS, punctuation, speaker/keyword/…): filtered out.

func ModelVariants

func ModelVariants() []ModelVariant

ModelVariants returns the curated English models the download picker offers. This is a hand-picked shortlist (each pinned by SHA256 since the model release predates GitHub's per-asset digests), NOT the whole sherpa-onnx model zoo — many more models (other languages, sizes, and streaming/batch families) work via a manual stt.localModelPath. Streaming (live) options come first.

type OpenAIRealtimeConfig

type OpenAIRealtimeConfig struct {
	APIKey string
	Model  string // default "gpt-4o-transcribe"
	// BaseURL overrides the wss endpoint (tests point it at a fake server).
	BaseURL string
}

OpenAIRealtimeConfig configures the OpenAI Realtime transcription transcriber (§6b, the credential-reuse streaming alternative). Reuses the OpenAI key.

type Platform

type Platform string

Platform selects the audio-capture toolchain (§9 of the design doc).

const (
	PlatformLinux       Platform = "linux"
	PlatformDarwin      Platform = "darwin"
	PlatformWindows     Platform = "windows"
	PlatformTermux      Platform = "termux"
	PlatformUnsupported Platform = ""
)

func DetectPlatform

func DetectPlatform() Platform

DetectPlatform picks the capture toolchain for the current host. Termux is detected by its environment (it reports GOOS=linux/android): TERMUX_VERSION is exported to every Termux shell, and PREFIX carries the app package path.

func (Platform) StreamingSupported

func (p Platform) StreamingSupported() bool

StreamingSupported reports whether the platform's capture tool can write raw PCM to stdout. Termux's termux-microphone-record requires a regular output file — a hard tool limitation, so Termux is batch-only permanently.

type Recorder

type Recorder interface {
	Start() error
	// Stop ends a batch recording and returns the recorded audio bytes (WAV on
	// desktop, M4A on Termux — SniffFormat distinguishes them).
	Stop() ([]byte, error)
	// StartStreaming begins continuous capture of raw little-endian PCM16 mono
	// at the configured sample rate. Chunks arrive on the channel in ~50ms
	// units; the channel closes when capture ends (stop called, max duration
	// hit, or trailing-silence auto-stop). stop is idempotent.
	StartStreaming() (chunks <-chan []byte, stop func() error, err error)
}

Recorder captures microphone audio. Start/Stop is the batch path (used on Termux, and as the fallback everywhere); StartStreaming is the desktop-only continuous-PCM path.

func NewRecorder

func NewRecorder(opts RecorderOptions) Recorder

NewRecorder builds the platform recorder. Construction is cheap and side-effect free; tools are only exec'd on Start/StartStreaming.

type RecorderOptions

type RecorderOptions struct {
	Platform        Platform
	SampleRate      int
	MaxDuration     time.Duration
	SilenceAutoStop bool
	// WindowsAudioDevice is the dshow capture device name; auto-detected from
	// `ffmpeg -list_devices` when empty.
	WindowsAudioDevice string
	// contains filtered or unexported fields
}

RecorderOptions configures a Recorder. Zero values pick sensible defaults.

type SampleRate

type SampleRate interface {
	SampleRate() int
}

SampleRate lets a Transcriber declare the capture rate its wire format expects, so the recorder can be built to match (OpenAI Realtime wants 24kHz; everything else uses 16kHz). A Transcriber that does not implement this uses DefaultSampleRate.

type ServerConfig

type ServerConfig struct {
	// Binary is the streaming server executable (default
	// "sherpa-onnx-online-websocket-server"); looked up on PATH.
	Binary string
	// ModelPath is the sherpa-onnx streaming model directory (transducer:
	// encoder/decoder/joiner + tokens).
	ModelPath  string
	Port       int
	NumThreads int
	// contains filtered or unexported fields
}

ServerConfig configures the warm sherpa-onnx streaming server (§6a).

type ServerManager

type ServerManager struct {
	// contains filtered or unexported fields
}

ServerManager owns one long-lived sherpa-onnx streaming server, spawned lazily on first streaming use and kept warm for the session — a websocket server with ~1-2s startup latency can't be respawned per utterance (§6a). Its lifecycle mirrors internal/lsp's Manager: lazy start, health-checked reuse, one restart on a crashed process, torn down on exit. Safe for concurrent use.

func NewServerManager

func NewServerManager(cfg ServerConfig) *ServerManager

NewServerManager builds a manager. Construction is side-effect free; the server is only spawned on the first EnsureRunning.

func (*ServerManager) EnsureRunning

func (m *ServerManager) EnsureRunning(ctx context.Context) (string, error)

EnsureRunning returns the websocket URL of a live server, starting one if needed. A previously-started server whose process has since died is reaped and replaced (mirroring the LSP manager's crashed-session eviction).

func (*ServerManager) SetModelPath

func (m *ServerManager) SetModelPath(path string)

SetModelPath updates the model directory the server launches with, for a mid-session config change (e.g. an F9 auto-download). It takes effect the next time the server (re)starts; a running server is left as-is.

func (*ServerManager) Shutdown

func (m *ServerManager) Shutdown(ctx context.Context) error

Shutdown stops the running server. Called alongside the LSP manager's shutdown on exit — a leaked warm server holds a port and CPU.

func (*ServerManager) URL

func (m *ServerManager) URL() string

URL returns the current server URL without starting one ("" when not running).

type SetupError

type SetupError struct {
	Tool string
	Hint string
}

SetupError reports a missing external tool with install guidance. Missing binaries are an expected condition (the user hasn't set dictation up yet), not a bug — callers show the hint instead of a raw exec error.

func (*SetupError) Error

func (e *SetupError) Error() string

type Transcriber

type Transcriber interface {
	// Transcribe returns the final text for one recorded clip.
	Transcribe(ctx context.Context, audio []byte) (string, error)

	// StreamTranscribe consumes PCM16 chunks and reports incremental results
	// via onPartial(text, final) as decoding proceeds, returning the complete
	// final transcript when the chunk channel closes.
	//
	// onPartial's text is the CURRENT best transcript of the utterance so far
	// (cumulative, not a delta) so the composer can replace the live region
	// wholesale; final marks a segment the provider considers settled.
	//
	// On a mid-stream failure (network drop for cloud, subprocess crash for
	// local) StreamTranscribe returns the best-effort text accumulated from
	// onPartial calls made so far PLUS a non-nil error — already-streamed text
	// is never discarded, only the untranscribed tail after the failure is lost.
	StreamTranscribe(ctx context.Context, chunks <-chan []byte, onPartial func(text string, final bool)) (string, error)
}

Transcriber turns captured audio into text. Batch (Transcribe) and streaming (StreamTranscribe) are one interface but two independent code paths — a batch-only provider (Groq/OpenAI, sherpa-onnx-offline) leaves StreamTranscribe returning ErrStreamingUnsupported, and a streaming provider still implements Transcribe for the batch/Termux fallback.

func NewCloudTranscriber

func NewCloudTranscriber(cfg CloudConfig) (Transcriber, error)

NewCloudTranscriber builds a batch transcriber for Groq or OpenAI.

func NewDeepgramTranscriber

func NewDeepgramTranscriber(cfg DeepgramConfig) (Transcriber, error)

NewDeepgramTranscriber builds the Deepgram streaming transcriber.

func NewLocalStreamingTranscriber

func NewLocalStreamingTranscriber(manager *ServerManager) Transcriber

NewLocalStreamingTranscriber builds a streaming transcriber backed by the shared, session-long sherpa-onnx server manager.

func NewLocalTranscriber

func NewLocalTranscriber(cfg LocalConfig) (Transcriber, error)

NewLocalTranscriber builds the offline transcriber. It does not touch the filesystem or PATH until Transcribe runs, so a missing binary/model degrades to a clear setup message at use time (matching internal/lsp).

func NewOpenAIRealtimeTranscriber

func NewOpenAIRealtimeTranscriber(cfg OpenAIRealtimeConfig) (Transcriber, error)

NewOpenAIRealtimeTranscriber builds the OpenAI Realtime streaming transcriber.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL