audio

package
v1.5.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 7 Imported by: 2

Documentation

Overview

Package audio provides voice activity detection (VAD), turn detection, and audio session management for real-time voice AI applications.

The package follows industry-standard patterns for voice AI:

  • VAD (Voice Activity Detection): Detects when someone is speaking vs. silent
  • Turn Detection: Determines when a speaker has finished their turn
  • Interruption Handling: Manages user interrupting bot output

Architecture

Audio processing follows a two-stage approach:

  1. VADAnalyzer detects voice activity in real-time
  2. TurnDetector uses VAD output plus additional signals to detect turn boundaries

Usage Example 

vad := audio.NewSimpleVAD(audio.DefaultVADParams())
detector := audio.NewSilenceDetector(500 * time.Millisecond)

for chunk := range audioStream {
    vad.Analyze(ctx, chunk)
    if detector.DetectTurnEnd(ctx, vad) {
        // User finished speaking
    }
}

Package audio provides audio processing utilities.

Index

Constants

View Source 
const (
	SampleRate24kHz = 24000 // Common TTS output rate
	SampleRate16kHz = 16000 // Common STT/ASR input rate

	// DuplexRate is the native device rate for the single duplex stream.
	// Resample only at the STT (48→16 kHz) and TTS (24→48 kHz) seams.
	DuplexRate = 48000
)

Standard audio sample rates for common use cases.

View Source 
const (
	DefaultVADConfidence = 0.5
	DefaultVADStartSecs  = 0.2
	DefaultVADStopSecs   = 0.8
	DefaultVADMinVolume  = 0.01
	DefaultVADSampleRate = 16000
)

Default VAD parameter values.

View Source 
const DefaultMaxAudioBufferSize = 10 * 1024 * 1024

DefaultMaxAudioBufferSize is the maximum size of the audio buffer in bytes. At 16kHz/16-bit mono (32KB/s), 10MB holds approximately 5 minutes of audio.

Variables

This section is empty.

Functions

func Resample24kTo16k 

func Resample24kTo16k(input []byte) ([]byte, error)

Resample24kTo16k is a convenience function for the common case of resampling from 24kHz (TTS output) to 16kHz (Gemini input).

func ResamplePCM16 

func ResamplePCM16(input []byte, fromRate, toRate int) ([]byte, error)

ResamplePCM16 resamples PCM16 audio data from one sample rate to another. Uses linear interpolation for reasonable quality resampling. Input and output are little-endian 16-bit signed PCM samples.

Types

type AccumulatingTurnDetector 

type AccumulatingTurnDetector interface {
	TurnDetector

	// OnTurnComplete registers a callback for when a complete turn is detected.
	OnTurnComplete(callback TurnCallback)

	// GetAccumulatedAudio returns audio accumulated so far (may be incomplete turn).
	GetAccumulatedAudio() []byte

	// SetTranscript sets the transcript for the current turn (from external STT).
	SetTranscript(transcript string)
}

AccumulatingTurnDetector is a TurnDetector that accumulates audio during a turn.

type AdaptiveVAD added in v1.5.3 

type AdaptiveVAD struct {
	// contains filtered or unexported fields
}

AdaptiveVAD is a voice activity detector that adapts its speech threshold to the ambient noise level. It is well-suited for "quiet mic" environments where the speaker's voice is only slightly louder than the background — a condition where SimpleVAD's fixed threshold often fails to trigger.

Algorithm:

  1. Compute the RMS of each PCM16 chunk.
  2. Smooth with an exponential moving average (α = 0.3).
  3. Derive speechThreshold = max(noiseFloor × 3.0, 0.01).
  4. Adapt the noise floor upward only when smoothedRMS < speechThreshold (i.e. we are probably in silence, not speech).
  5. Map smoothedRMS → probability in [0, 1] using a linear scale between noiseFloor and 2 × (speechThreshold − noiseFloor).

AdaptiveVAD embeds *vadStateMachine, which promotes State(), OnStateChange(), and base Reset() to satisfy the VADAnalyzer interface.

func NewAdaptiveVAD added in v1.5.3 

func NewAdaptiveVAD(params VADParams) (*AdaptiveVAD, error)

NewAdaptiveVAD creates an AdaptiveVAD analyzer with the given parameters.

func (*AdaptiveVAD) Analyze added in v1.5.3 

func (v *AdaptiveVAD) Analyze(_ context.Context, audioData []byte) (float64, error)

Analyze processes audio and returns voice probability based on adaptive RMS analysis.

func (*AdaptiveVAD) Name added in v1.5.3 

func (v *AdaptiveVAD) Name() string

Name returns the analyzer identifier.

func (AdaptiveVAD) OnStateChange added in v1.5.3 

func (m AdaptiveVAD) OnStateChange() <-chan VADEvent

OnStateChange returns a channel that receives VADEvent values on each state transition. The channel is buffered; events are dropped when it is full.

func (*AdaptiveVAD) Reset added in v1.5.3 

func (v *AdaptiveVAD) Reset()

Reset clears accumulated state for a new conversation, resetting the smoothed RMS and noise floor back to their initial values.

func (AdaptiveVAD) State added in v1.5.3 

func (m AdaptiveVAD) State() VADState

State returns the current VAD state.

type Chunk added in v1.4.7 

type Chunk struct {
	// Data is the raw audio bytes.
	Data []byte
	// Index is the chunk sequence number (0-indexed).
	Index int
	// Final indicates this is the last chunk.
	Final bool
	// Error is set if an error occurred while producing the chunk.
	Error error
}

Chunk represents a chunk of audio data flowing through the runtime — produced by TTS providers, consumed by playback sinks, realtime LLM inputs, and pipeline stages. Carries the bytes plus stream-position metadata.

type Format added in v1.5.4 

type Format struct {
	// SampleRate is the number of audio samples per second (e.g. 16000, 24000, 48000).
	SampleRate int
	// Channels is 1 for mono, 2 for stereo.
	Channels int
}

Format describes the encoding of the media payload. Audio fields are present now; video fields (Width, Height, Codec) will be added later.

type InterruptionCallback 

type InterruptionCallback func()

InterruptionCallback is called when user interrupts the bot.

type InterruptionHandler 

type InterruptionHandler struct {
	// contains filtered or unexported fields
}

InterruptionHandler manages user interruption logic during bot output.

func NewInterruptionHandler 

func NewInterruptionHandler(strategy InterruptionStrategy, vad VADAnalyzer) *InterruptionHandler

NewInterruptionHandler creates an InterruptionHandler with the given strategy and VAD.

func (*InterruptionHandler) Interrupt added in v1.5.4 

func (h *InterruptionHandler) Interrupt()

Interrupt externally signals an interruption. Use this on the realtime/ASM path where barge-in is detected by the provider's server-side VAD rather than our local VAD — there is no ProcessVADState call to drive handleInterruption, so the provider stage signals the interruption directly. Idempotent within a turn; cleared by Reset.

func (*InterruptionHandler) Interrupted added in v1.5.4 

func (h *InterruptionHandler) Interrupted() <-chan struct{}

Interrupted returns a channel closed when an interruption fires for the current turn. Reset re-arms it, so callers should re-fetch after a Reset.

func (*InterruptionHandler) IsBotSpeaking 

func (h *InterruptionHandler) IsBotSpeaking() bool

IsBotSpeaking returns true if the bot is currently outputting audio.

func (*InterruptionHandler) NotifySentenceBoundary 

func (h *InterruptionHandler) NotifySentenceBoundary()

NotifySentenceBoundary notifies the handler of a sentence boundary. For deferred interruption strategy, this may trigger the pending interruption.

func (*InterruptionHandler) OnInterrupt 

func (h *InterruptionHandler) OnInterrupt(callback InterruptionCallback)

OnInterrupt registers a callback for when interruption occurs.

func (*InterruptionHandler) ProcessAudio 

func (h *InterruptionHandler) ProcessAudio(ctx context.Context, audio []byte) (bool, error)

ProcessAudio processes audio and detects user interruption. Returns true if an interruption was detected and should be acted upon.

func (*InterruptionHandler) ProcessVADState 

func (h *InterruptionHandler) ProcessVADState(ctx context.Context, state VADState) (bool, error)

ProcessVADState processes a VAD state update for interruption detection. Returns true if an interruption was detected and should be acted upon.

func (*InterruptionHandler) Reset 

func (h *InterruptionHandler) Reset()

Reset clears interruption state for a new turn.

func (*InterruptionHandler) SetBotSpeaking 

func (h *InterruptionHandler) SetBotSpeaking(speaking bool)

SetBotSpeaking sets whether the bot is currently outputting audio.

func (*InterruptionHandler) WasInterrupted 

func (h *InterruptionHandler) WasInterrupted() bool

WasInterrupted returns true if an interruption occurred.

type InterruptionStrategy 

type InterruptionStrategy int

InterruptionStrategy determines how to handle user interrupting bot. 

const (
	// InterruptionIgnore ignores user speech during bot output.
	InterruptionIgnore InterruptionStrategy = iota
	// InterruptionImmediate immediately stops bot and starts listening.
	InterruptionImmediate
	// InterruptionDeferred waits for bot's current sentence, then switches.
	InterruptionDeferred
)

func (InterruptionStrategy) String 

func (s InterruptionStrategy) String() string

String returns a human-readable representation of the interruption strategy.

type JitterBuffer added in v1.5.4 

type JitterBuffer struct {
	// contains filtered or unexported fields
}

JitterBuffer is a bounded FIFO ring buffer of int16 PCM samples (mono). It is safe for concurrent use: Push from a producer goroutine while Pull or Clear are called from the duplex playback loop.

Overflow policy: when a Push would exceed capacity the OLDEST samples are dropped to make room for the newest audio, and the dropped count is added to the cumulative Drops counter.

Underrun policy: Pull always returns exactly n samples; when fewer than n are buffered the tail is zero-filled (silence).

func NewJitterBuffer added in v1.5.4 

func NewJitterBuffer(capacitySamples int) *JitterBuffer

NewJitterBuffer returns a JitterBuffer with the given maximum capacity in samples. A capacity of zero is valid but all pushes will drop immediately.

func (*JitterBuffer) Clear added in v1.5.4 

func (j *JitterBuffer) Clear()

Clear drops all buffered samples. The next Pull will return silence.

func (*JitterBuffer) Drops added in v1.5.4 

func (j *JitterBuffer) Drops() int64

Drops returns the cumulative number of samples that have been dropped due to buffer overflow.

func (*JitterBuffer) Len added in v1.5.4 

func (j *JitterBuffer) Len() int

Len returns the number of samples currently in the buffer.

func (*JitterBuffer) Pull added in v1.5.4 

func (j *JitterBuffer) Pull(n int) []int16

Pull removes and returns exactly n samples from the front of the buffer. If fewer than n samples are available the returned slice is zero-filled from the point of underrun through index n-1.

func (*JitterBuffer) Push added in v1.5.4 

func (j *JitterBuffer) Push(samples []int16)

Push appends samples to the buffer. If appending would exceed capacity, the oldest samples are discarded first and the drop counter incremented.

type MediaFrame added in v1.5.4 

type MediaFrame struct {
	// Kind identifies the media type.
	Kind MediaKind
	// Data is the raw payload — PCM16 little-endian for audio.
	Data []byte
	// PTS is the presentation timestamp from the session clock.
	PTS time.Duration
	// Format describes the encoding of Data.
	Format Format
}

MediaFrame is a single unit of captured or synthesized media. PTS (presentation timestamp) is measured from the session clock and is the load-bearing field for AEC delay estimation and A/V sync.

type MediaKind added in v1.5.4 

type MediaKind int

MediaKind identifies the type of media carried in a MediaFrame. 

const (
	// KindAudio is a PCM16 little-endian audio frame.
	KindAudio MediaKind = iota
	// KindVideo is reserved for future use (YAGNI — not implemented).
	KindVideo
)

type MemSink added in v1.5.4 

type MemSink struct {
	// contains filtered or unexported fields
}

MemSink is an in-memory Sink for tests and headless use. It records every Write call; Flush drops the recorded frames. Safe for concurrent use.

func NewMemSink added in v1.5.4 

func NewMemSink(k MediaKind) *MemSink

NewMemSink creates a MemSink that accepts frames of the given kind.

func (*MemSink) Close added in v1.5.4 

func (m *MemSink) Close() error

Close is a no-op for MemSink; it satisfies the Sink interface.

func (*MemSink) Flush added in v1.5.4 

func (m *MemSink) Flush()

Flush drops all queued frames, simulating barge-in drain.

func (*MemSink) Kind added in v1.5.4 

func (m *MemSink) Kind() MediaKind

Kind returns the MediaKind this sink accepts.

func (*MemSink) Write added in v1.5.4 

func (m *MemSink) Write(f MediaFrame)

Write appends f to the internal frame log.

func (*MemSink) Written added in v1.5.4 

func (m *MemSink) Written() []MediaFrame

Written returns a copy of the frames written since the last Flush. The returned slice is independent of the internal buffer, making it safe to read concurrently with ongoing Write or Flush calls.

type MemSource added in v1.5.4 

type MemSource struct {
	// contains filtered or unexported fields
}

MemSource is an in-memory Source for tests and headless use. Push frames via Push; close the channel via Close so that range-loops terminate. Safe for concurrent use.

func NewMemSource added in v1.5.4 

func NewMemSource(kind MediaKind, buf int) *MemSource

NewMemSource creates a MemSource with a buffered channel of size buf.

func (*MemSource) Close added in v1.5.4 

func (m *MemSource) Close() error

Close closes the underlying channel, signaling end-of-stream to consumers. It is idempotent; calling Close more than once is safe.

func (*MemSource) Frames added in v1.5.4 

func (m *MemSource) Frames() <-chan MediaFrame

Frames returns the read-only channel of MediaFrames.

func (*MemSource) Kind added in v1.5.4 

func (m *MemSource) Kind() MediaKind

Kind returns the MediaKind produced by this source.

func (*MemSource) Push added in v1.5.4 

func (m *MemSource) Push(f MediaFrame)

Push sends f onto the internal channel. It blocks if the buffer is full.

type Session added in v1.5.4 

type Session interface {
	// Start begins media flow. It runs until ctx is canceled or Close is called.
	Start(ctx context.Context) error
	// Sources returns all Sources in this session.
	Sources() []Source
	// Sinks returns all Sinks in this session.
	Sinks() []Sink
	// Close stops all Sources and Sinks and releases session resources.
	Close() error
}

Session groups the Sources and Sinks that belong to one audio session (e.g. a single call leg: one microphone source + one speaker sink).

type SilenceDetector 

type SilenceDetector struct {
	// Threshold is the silence duration required to trigger turn end.
	Threshold time.Duration

	// MaxBufferSize is the maximum audio buffer size in bytes.
	// When exceeded, the oldest audio data is discarded to stay within the limit.
	// Default: DefaultMaxAudioBufferSize (10MB).
	MaxBufferSize int
	// contains filtered or unexported fields
}

SilenceDetector detects turn boundaries based on silence duration. It triggers end-of-turn when silence exceeds a configurable threshold.

func NewSilenceDetector 

func NewSilenceDetector(threshold time.Duration, opts ...SilenceDetectorOption) *SilenceDetector

NewSilenceDetector creates a SilenceDetector with the given threshold. threshold is the duration of silence required to trigger end-of-turn.

func (*SilenceDetector) GetAccumulatedAudio 

func (d *SilenceDetector) GetAccumulatedAudio() []byte

GetAccumulatedAudio returns audio accumulated so far.

func (*SilenceDetector) IsUserSpeaking 

func (d *SilenceDetector) IsUserSpeaking() bool

IsUserSpeaking returns true if user is currently speaking.

func (*SilenceDetector) Name 

func (d *SilenceDetector) Name() string

Name returns the detector identifier.

func (*SilenceDetector) OnTurnComplete 

func (d *SilenceDetector) OnTurnComplete(callback TurnCallback)

OnTurnComplete registers a callback for when a complete turn is detected.

func (*SilenceDetector) ProcessAudio 

func (d *SilenceDetector) ProcessAudio(ctx context.Context, audio []byte) (bool, error)

ProcessAudio processes an incoming audio chunk. This implementation delegates to ProcessVADState and expects VAD to be run separately. Returns true if end of turn is detected.

func (*SilenceDetector) ProcessVADState 

func (d *SilenceDetector) ProcessVADState(ctx context.Context, state VADState) (bool, error)

ProcessVADState processes a VAD state update and detects turn boundaries. Returns true if end of turn is detected.

func (*SilenceDetector) Reset 

func (d *SilenceDetector) Reset()

Reset clears state for a new conversation.

func (*SilenceDetector) SetTranscript 

func (d *SilenceDetector) SetTranscript(transcript string)

SetTranscript sets the transcript for the current turn.

type SilenceDetectorOption added in v1.3.10 

type SilenceDetectorOption func(*SilenceDetector)

SilenceDetectorOption configures a SilenceDetector.

func WithMaxAudioBufferSize added in v1.3.10 

func WithMaxAudioBufferSize(size int) SilenceDetectorOption

WithMaxAudioBufferSize sets the maximum audio buffer size in bytes. When the buffer exceeds this limit, the oldest data is trimmed.

type SimpleVAD 

type SimpleVAD struct {
	// contains filtered or unexported fields
}

SimpleVAD is a basic voice activity detector using RMS (Root Mean Square) analysis. It provides a lightweight VAD implementation without requiring external ML models. For more accurate detection, consider using SileroVAD.

SimpleVAD embeds *vadStateMachine, which promotes State(), OnStateChange(), and base Reset() — together they satisfy the VADAnalyzer interface.

func NewSimpleVAD 

func NewSimpleVAD(params VADParams) (*SimpleVAD, error)

NewSimpleVAD creates a SimpleVAD analyzer with the given parameters.

func (*SimpleVAD) Analyze 

func (v *SimpleVAD) Analyze(_ context.Context, audioData []byte) (float64, error)

Analyze processes audio and returns voice probability based on RMS volume.

func (*SimpleVAD) Name 

func (v *SimpleVAD) Name() string

Name returns the analyzer identifier.

func (SimpleVAD) OnStateChange 

func (m SimpleVAD) OnStateChange() <-chan VADEvent

OnStateChange returns a channel that receives VADEvent values on each state transition. The channel is buffered; events are dropped when it is full.

func (*SimpleVAD) Reset 

func (v *SimpleVAD) Reset()

Reset clears accumulated state for a new conversation, including the smoothed RMS.

func (SimpleVAD) State 

func (m SimpleVAD) State() VADState

State returns the current VAD state.

type Sink added in v1.5.4 

type Sink interface {
	// Write enqueues a MediaFrame for playback/storage.
	// Note: hardware sinks fix the playback sample rate at session construction
	// (WithPlaybackRate, default 24 kHz); MediaFrame.Format is informational and
	// is NOT resampled. Callers must write frames at the session's playback rate.
	// (Phase 3 adds resample-at-sink.)
	Write(MediaFrame)
	// Flush drops all queued output immediately (e.g. on barge-in).
	Flush()
	// Kind returns the MediaKind consumed by this sink.
	Kind() MediaKind
	// Close stops the sink and releases any underlying resources.
	Close() error
}

Sink is a write-only media stream. Implementations include hardware playback devices (speaker), file writers, and the in-memory MemSink test double.

type Source added in v1.5.4 

type Source interface {
	// Frames returns a channel that delivers captured MediaFrames.
	// The channel is closed when the source ends or Close is called.
	Frames() <-chan MediaFrame
	// Kind returns the MediaKind produced by this source.
	Kind() MediaKind
	// Close stops the source and releases any underlying resources.
	Close() error
}

Source is a read-only media stream. Implementations include hardware capture devices (microphone), file readers, and the in-memory MemSource test double.

type TurnCallback 

type TurnCallback func(audio []byte, transcript string)

TurnCallback is called when a complete user turn is detected. audio contains the accumulated audio for the turn. transcript contains any accumulated transcript (may be empty).

type TurnDetector 

type TurnDetector interface {
	// Name returns the detector identifier.
	Name() string

	// ProcessAudio processes an incoming audio chunk.
	// Returns true if end of turn is detected.
	ProcessAudio(ctx context.Context, audio []byte) (bool, error)

	// ProcessVADState processes a VAD state update.
	// Returns true if end of turn is detected based on VAD state.
	ProcessVADState(ctx context.Context, state VADState) (bool, error)

	// IsUserSpeaking returns true if user is currently speaking.
	IsUserSpeaking() bool

	// Reset clears state for a new conversation.
	Reset()
}

TurnDetector determines when a speaker has finished their turn. This is separate from VAD - VAD detects voice activity, turn detection determines conversation boundaries.

type VADAnalyzer 

type VADAnalyzer interface {
	// Name returns the analyzer identifier.
	Name() string

	// Analyze processes audio and returns voice probability (0.0-1.0).
	// audio should be raw PCM samples at the configured sample rate.
	Analyze(ctx context.Context, audio []byte) (float64, error)

	// State returns the current VAD state based on accumulated analysis.
	State() VADState

	// OnStateChange returns a channel that receives state transitions.
	// The channel is buffered and may drop events if not consumed.
	OnStateChange() <-chan VADEvent

	// Reset clears accumulated state for a new conversation.
	Reset()
}

VADAnalyzer analyzes audio for voice activity.

type VADEvent 

type VADEvent struct {
	State      VADState
	PrevState  VADState
	Timestamp  time.Time
	Duration   time.Duration // How long in the previous state
	Confidence float64       // Voice confidence at transition
}

VADEvent represents a state transition in VAD.

type VADParams 

type VADParams struct {
	// Confidence threshold for voice detection (0.0-1.0, default: 0.5).
	// Higher values require more confidence before triggering.
	Confidence float64

	// StartSecs is seconds of speech required to trigger VADStateSpeaking (default: 0.2).
	// Prevents false starts from brief noise.
	StartSecs float64

	// StopSecs is seconds of silence required to trigger VADStateQuiet (default: 0.8).
	// Allows natural pauses without ending turn.
	StopSecs float64

	// MinVolume is the minimum RMS volume threshold (default: 0.01).
	// Audio below this is treated as silence.
	MinVolume float64

	// SampleRate is the audio sample rate in Hz (default: 16000).
	SampleRate int
}

VADParams configures voice activity detection behavior.

func DefaultVADParams 

func DefaultVADParams() VADParams

DefaultVADParams returns sensible defaults for voice activity detection.

func (VADParams) Validate 

func (p VADParams) Validate() error

Validate checks that VAD parameters are within acceptable ranges.

type VADState 

type VADState int

VADState represents the current voice activity state. 

const (
	// VADStateQuiet indicates no voice activity detected.
	VADStateQuiet VADState = iota
	// VADStateStarting indicates voice is starting (within start threshold).
	VADStateStarting
	// VADStateSpeaking indicates active speech.
	VADStateSpeaking
	// VADStateStopping indicates voice is stopping (within stop threshold).
	VADStateStopping
)

func (VADState) String 

func (s VADState) String() string

String returns a human-readable representation of the VAD state.

type ValidationError 

type ValidationError struct {
	Field   string
	Message string
}

ValidationError represents a parameter validation error.

func (*ValidationError) Error 

func (e *ValidationError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.