Documentation
¶
Overview ¶
Package audio is the phone's microphone and speaker: one spoken turn at a time, for talking to an assistant. Listen opens the microphone with the phone's noise suppression and echo cancelling, and waits for someone close to it to say something and stop; Play plays a clip back. Both report how loud it is, 0…1, for something to move with the voice.
if ok, err := audio.RequestMicrophone(ctx); err != nil || !ok {
return // not allowed
}
l := &audio.Listener{OnLevel: setLevel}
wav, err := l.Listen(ctx) // a WAV, 16 kHz mono; nil if nothing was said
…
audio.Play(ctx, reply, setLevel)
audio.Release() // other apps' audio comes back
Add it to a project with `nimbus native plugin add audio`, which also asks for the microphone in the app's Info.plist and manifest. On the web the browser's microphone (with its own noise suppression and echo cancelling) and speaker stand in; Release isn't needed there.
Index ¶
Constants ¶
const Rate = 16000
Rate is the microphone's sample rate: 16-bit mono at 16 kHz.
Variables ¶
This section is empty.
Functions ¶
func Play ¶
Play plays a clip (WAV, MP3 or M4A) through the speaker and returns when it has finished. A cancelled ctx stops it, returning ctx's error. onLevel (may be nil) gets how loud it is, 0…1.
func Release ¶
func Release()
Release stops everything and lets go of the microphone and speaker, so other apps' audio comes back. Call it when the conversation ends.
func RequestMicrophone ¶
RequestMicrophone asks for the microphone, once: later calls answer at once with what the person chose.
Types ¶
type Listener ¶
type Listener struct {
OnLevel func(level float32) // how loud, 0…1, a few dozen times a second
OnSpeech func() // someone started talking
// contains filtered or unexported fields
}
Listener listens for one spoken turn at a time. Its callbacks run on Listen's goroutine.
type TurnDetector ¶
type TurnDetector struct {
// contains filtered or unexported fields
}
TurnDetector decides when a spoken turn starts and ends, against the room's own noise and the speaker's own voice.
The noise floor is a low percentile of the last few seconds' loudness, so it follows a room that gets louder and quieter again, and steady noise sinks below it. Speech is sound clearly above the floor and loud enough to be close to the phone, for long enough that a clatter does not count. Once someone is talking the detector learns how loud they are: a turn ends after a short quiet for them (the room may still be chattering well below), or at a hard limit. The clip keeps a little of the moment before speech began, and what is well below the speaker between their words is turned down. Feed it short frames (Listener uses 20 ms).
func NewTurnDetector ¶
func NewTurnDetector() *TurnDetector
NewTurnDetector starts listening for one turn.
func (*TurnDetector) Feed ¶
func (d *TurnDetector) Feed(samples []int16) (turn Turn, level float32, clip []byte)
Feed takes the next frame and reports what it brought, with how loud it was, 0…1. With TurnEnded comes the clip, a WAV.
func (*TurnDetector) Finish ¶
func (d *TurnDetector) Finish() []byte
Finish ends the turn now and returns it as a WAV, trailing silence trimmed; nil if nothing was said.
Source Files