audiosample

package
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2026 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package audiosample reads stretches of a book's audio from an Audiobookshelf server: seconds or minutes from anywhere in the book, as mono PCM at the rate asked for, across its files in play order. It is the groundwork for anything that listens to a book rather than reading its metadata - comparing two copies, transcribing a passage - and it holds the loudness envelope and the matching that compare two stretches.

ffmpeg does the decoding and reads the server's file route itself, seeking with byte ranges, so only the stretch asked for crosses the network. The API key never goes on ffmpeg's command line, where ps would show it: ffmpeg is pointed at a proxy on 127.0.0.1, inside this process, which adds the key.

Index

Constants

This section is empty.

Variables

View Source
var ErrNoFFmpeg = errors.New("ffmpeg is not installed or not on the PATH: reading a book's audio needs it")

ErrNoFFmpeg is returned by New when ffmpeg is not on the PATH.

Functions

func Envelope

func Envelope(pcm []int16, window int) []float64

Envelope is a stretch's loudness over time: the natural log of the RMS of each window of samples (200 samples at 4 kHz is 50 ms). Speech rises and falls with every syllable and pause, and that pattern survives a new encoding, a new bitrate and a small change of speed, where the samples themselves do not; two narrators reading the same words do not share it.

func SpectralMatch added in v0.6.0

func SpectralMatch(needle, haystack []int16, rate int, found Found, window int) float64

SpectralMatch is the spectral score of needle against the stretch of haystack that Match found for it, both PCM at rate: found.At is in envelope windows of window samples, and found.Speed is the speed the needle was played at to fit. The needle is taken a chunk at a time, each chunk lined up on its own within a window either side of where the envelope put it, as the envelope's window is coarser than a frame and a speed found to a quarter of a percent drifts half a frame over a chunk. The score is the mean over the chunks.

func SpectralScore added in v0.6.0

func SpectralScore(a, b [][]float64) float64

SpectralScore says how alike two spectrograms of the same length are: in each band the energy over time is scaled to mean 0 and deviation 1, which takes out loudness and the encoder's or the microphone's colouring, and the score is the mean of the bands' correlations. 1 is the same shape in every band, 0 no relation. A band flat in either is left out; when all are, the score is 0.

func Spectrogram added in v0.6.0

func Spectrogram(pcm []int16, rate, frame, hop, bands int) [][]float64

Spectrogram is the log energy of pcm in each of bands frequency bands, a row per frame: frames of frame samples every hop samples, Hann windowed, the bands spaced evenly in pitch from spectrumLowHz up to spectrumShare of the rate. Loudness is not taken out here; SpectralScore does that per band.

func Speeds

func Speeds(lo, hi float64, n int) []float64

Speeds is n speed factors spread evenly from lo to hi, for Match.

Types

type Book

type Book struct {
	ItemID string
	Tracks []Track
}

Book is an item's audio in play order.

func BookFromFiles added in v0.6.0

func BookFromFiles(ctx context.Context, id string, paths []string) (*Book, error)

BookFromFiles is a book made of audio files on this machine, in the order given, for a sampler from NewFiles: each file's length comes from ffprobe.

func BookOf

func BookOf(it *abs.LibraryItemExpanded) (*Book, error)

BookOf reads an item's play order from its expanded record: the server's own track list, which leaves excluded files out and carries each file's start, or, when the item came without one, its included audio files in index order, which is what the server builds the list from.

func (*Book) Duration

func (b *Book) Duration() float64

Duration is the whole book's length in seconds.

type Enveloper

type Enveloper struct {
	// contains filtered or unexported fields
}

Enveloper builds an Envelope a chunk of samples at a time, for a stretch read with Stream.

func NewEnveloper

func NewEnveloper(window int) *Enveloper

NewEnveloper starts an envelope of the given window.

func (*Enveloper) Envelope

func (e *Enveloper) Envelope() []float64

Envelope is the envelope of the whole windows written so far.

func (*Enveloper) Write

func (e *Enveloper) Write(pcm []int16)

Write adds samples to the envelope.

type Found

type Found struct {
	// Score is the correlation of the needle with the stretch of haystack it
	// fits: 1 is the same shape, 0 no relation.
	Score float64
	// At is where that stretch starts, in haystack samples.
	At int
	// Speed is the speed the needle was played at to fit: 1.02 means the
	// haystack runs 2% faster than the needle's recording.
	Speed float64
}

Found is where a needle fits a haystack best.

func Match

func Match(needle, haystack, speeds []float64) Found

Match finds where needle best fits in haystack, both envelopes, trying the needle played at each speed. The score at each offset is the normalised cross-correlation (Pearson's r of the needle against the stretch under it), so a copy recorded louder or quieter scores as high as the original. The correlation at every offset comes from one FFT of the haystack and two per speed, which is what makes searching an hour of envelope for 30 seconds of it, at 25 speeds, take a fraction of a second rather than minutes.

type Health added in v0.6.0

type Health struct {
	Seconds    float64  // audio decoded
	Complaints int      // lines ffmpeg wrote at its error level
	First      []string // the first few of them
	Failed     string   // why ffmpeg gave up on the file, when it did
}

Health is how a stretch of one file decoded: how much audio came back, and what ffmpeg complained of on the way.

type Sampler

type Sampler struct {
	// contains filtered or unexported fields
}

Sampler decodes stretches of books through ffmpeg. It serves ffmpeg the server's file route through a proxy on 127.0.0.1 while it is open, so it must be closed. The proxy answers only under a random path, and only for the items this sampler was asked to read.

func New

func New(ctx context.Context, gen *abs.Client) (*Sampler, error)

New starts a sampler reading through gen. It fails with ErrNoFFmpeg when ffmpeg is not installed.

func NewFiles added in v0.6.0

func NewFiles() (*Sampler, error)

NewFiles is a sampler over files on this machine rather than a server's: each Track's FileID is a path. It is what the calibration reads a corpus through, and what reads a book before it is on a server. It fails with ErrNoFFmpeg when ffmpeg is not installed.

func (*Sampler) BytesRead

func (s *Sampler) BytesRead() int64

BytesRead is how many bytes of audio files the server has sent so far.

func (*Sampler) Check added in v0.6.0

func (s *Sampler) Check(ctx context.Context, itemID, fileID string, off, length float64) (Health, error)

Check decodes length seconds of one file of an item from off seconds in and says how it went. What is wrong with the file - ffmpeg failing on it, complaining of it, or running out of it early - is in the Health; an error is the server or the proxy failing, which says nothing of the file.

func (*Sampler) Close

func (s *Sampler) Close() error

Close stops the proxy, and says why it had stopped if it stopped early.

func (*Sampler) Read

func (s *Sampler) Read(ctx context.Context, b *Book, start, length float64, rate int) ([]int16, error)

Read decodes length seconds of the book from start seconds in, as mono 16-bit PCM at rate samples a second, crossing from file to file in play order. ffmpeg seeks each file itself (-ss before -i), which over http is a ranged request, so only the stretch and the file's header are fetched; in an mp3 the seek lands where the file's table of contents or its bitrate says, which can be a second or so off. A stretch running past the end of the book comes back short.

func (*Sampler) Stream

func (s *Sampler) Stream(ctx context.Context, b *Book, start, length float64, rate int, fn func(pcm []int16) error) error

Stream is Read handing the samples to fn as ffmpeg decodes them, a chunk at a time, rather than holding the whole stretch: an hour at 4 kHz is 29 MB. fn must not keep the slice past the call. An error from fn stops the read and is returned.

type Track

type Track struct {
	FileID   string // the file's ino, which the server's file route takes
	Start    float64
	Duration float64
}

Track is one audio file of a book: where it starts in the whole book and how long it runs, in seconds.

Directories

Path Synopsis
Command calibration fetches the recordings item_compare_audio is calibrated on into a cache outside the repository, with the manifest the calibration test reads.
Command calibration fetches the recordings item_compare_audio is calibrated on into a cache outside the repository, with the manifest the calibration test reads.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL