Documentation
¶
Overview ¶
Package multimodal — audio transcription via whisper.cpp.
We shell out to `whisper-cli` (whisper.cpp's CLI binary) or `whisper` (OpenAI's Python reference). Either works; we probe in order and use whichever is on PATH. The CLI options differ but the contract we need from each is identical: in = audio file, out = transcript text.
Why not bundle a Go transcription library: there isn't a good one. Whisper.cpp has Go bindings (mutablelogic/go-whisper, etc.) but they require model files at known paths + careful CGO setup — way out of scope for a "drop an audio file in the chat" feature. Shell-out keeps the dependency story honest: user installs whisper, we use it. No models bundled.
Package multimodal — binary fallback extractor.
The "never say can't handle" floor. When no specific extractor claims a file (unknown binary format, corrupted bytes, uncategorized custom format), this one returns a useful Result: MIME + size + first-bytes hex. The agent gets enough to reason about the file's existence even when full content extraction isn't possible.
Register this LAST so specific extractors always win.
Package multimodal — default registry assembly.
Order matters. The router walks the list and takes the first extractor whose Supports() returns true. So specific extractors register before broad ones; the BinaryFallback always wins last.
Package multimodal — MIME + extension detection.
Two signals feed Lookup: the file extension (cheap, deterministic, reliable in agent workflows) AND the magic-byte sniff via http.DetectContentType. We return BOTH so extractors can match on whichever they prefer.
Extension wins when present + recognised. For files without an extension OR with a misleading one (".dat" containing PDF bytes), the sniff fills in. Some formats (DOCX, XLSX) need both signals because http.DetectContentType only sees "application/zip" — the underlying ZIP structure of OOXML files.
Package multimodal — DOCX / DOC / RTF / ODT extractor via pandoc.
Pandoc is the universal document converter. It handles every office format we'd plausibly see (docx, doc, rtf, odt, html, etc.) with one CLI. Same shell-out rationale as PDF/audio: the alternative is parsing OOXML + ODF ourselves, both of which are complex formats we'd never get right without months of work.
Pandoc isn't always installed; the extractor surfaces a clear "install pandoc" error rather than a generic exec failure.
Package multimodal — "understand anything" extraction pipeline (master plan §7.5 / Batch I).
The contract: hand the package a file path, get back text + a tiny metadata bag. PDFs come back as text + page count; audio comes back as transcript + duration; images come back as a vision-describer caption + dimensions; DOCX comes back via pandoc; plain text + code files come back verbatim. Never "I can't handle that" — at minimum the binary-fallback extractor reports MIME + size so the agent has a verbal handle on the file even when nothing extracts cleanly.
Design:
- Extractor interface so adapters plug in (PDF, audio, DOCX, image, text, binary-fallback). Each declares which MIME types and extensions it claims.
- Registry walks extractors in order, returns the first that Supports() the file. The binary-fallback is registered last so unknown content STILL gets a useful response.
- Shell-out extractors (pdftotext, whisper, pandoc) check the binary's presence up-front and return a clear "install X" error rather than failing mid-extraction. Tool sees the error, surfaces it to the user.
Package multimodal — image extractor via vision describer.
Hands the image bytes to the wired vision describer (Anthropic today, more later) and returns the caption as text. When no describer is wired (config didn't set it up), we still return a useful Result — bytes + dimensions + MIME — so the agent has a verbal handle on the image even without vision.
Package multimodal — PDF extractor via pdftotext (poppler-utils).
Why pdftotext: pure-Go PDF text extraction libraries exist (e.g. rsc.io/pdf, ledongthuc/pdf, unidoc) but they're either incomplete (rsc reads structured PDFs but skips embedded fonts), heavy (unidoc is GPL+commercial), or both. pdftotext is the industry- standard reference implementation, installed by default on most Linux distros and one brew/apt away otherwise. The shell-out cost is dwarfed by the LLM round-trip cost downstream.
Package multimodal — text / code / structured-text extractor.
The straight-through path: file is text-like by MIME or extension, we read it and return contents verbatim. Capped at 256KB so a giant log file doesn't blow context.
This is also the catch-all for ALL programming-language source — .go/.py/.rs/.ts/etc don't get a special MIME from http.DetectContentType (they sniff as text/plain) but the extension list keeps them on this happy path.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoExtractor = fmt.Errorf("multimodal: no extractor for this file")
ErrNoExtractor is returned when no extractor in the registry claims a file. With the default registry this never fires (the binary fallback claims everything).
Functions ¶
func Detect ¶
Detect returns (mime, extension) for path. The extension keeps its leading dot, lowercased. Mime is the http.DetectContentType result (always non-empty; falls back to "application/octet-stream").
File-not-found returns a clear error rather than masking it with a fallback — the tool layer surfaces this so the user knows their path was wrong, not their content was opaque.
func IsTextLike ¶
IsTextLike reports whether mime suggests human-readable text. Used by the text-fallback extractor to decide whether to dump bytes as content vs report "binary, sniffed mime=...". We accept text/* AND known structured-text MIME types (json, xml, yaml, etc.).
Types ¶
type AudioExtractor ¶
type AudioExtractor struct {
// Model is passed to whisper.cpp via --model (path to a .bin
// model) or to whisper via --model (model name like "base"). When
// empty we let whisper pick its default. The user is expected to
// have a model configured if they want better quality.
Model string
// Language hint passed through when non-empty.
Language string
// Timeout caps the per-extract budget. Default 5min — long
// enough for typical voice notes, short enough that a
// runaway transcribe doesn't hang the agent.
Timeout time.Duration
}
AudioExtractor transcribes audio via the first whisper binary on PATH (whisper-cli or whisper).
func NewAudioExtractor ¶
func NewAudioExtractor() *AudioExtractor
NewAudioExtractor returns an extractor with sensible defaults.
func (*AudioExtractor) Name ¶
func (a *AudioExtractor) Name() string
func (*AudioExtractor) Supports ¶
func (a *AudioExtractor) Supports(mime, ext string) bool
type BinaryFallback ¶
type BinaryFallback struct{}
BinaryFallback claims every file. Returns metadata-only Result.
func NewBinaryFallback ¶
func NewBinaryFallback() *BinaryFallback
NewBinaryFallback returns the fallback extractor.
func (*BinaryFallback) Name ¶
func (b *BinaryFallback) Name() string
func (*BinaryFallback) Supports ¶
func (b *BinaryFallback) Supports(mime, ext string) bool
Supports always returns true. This is the catch-all.
type ErrMissingDependency ¶
type ErrMissingDependency struct {
Tool string // "pdftotext", "whisper", "pandoc"
InstallEx string // example install command for the common platform
}
ErrMissingDependency is returned when a shell-out extractor can't find its binary (e.g. pdftotext, whisper). The caller propagates this to the user with the install hint embedded in the message.
func (*ErrMissingDependency) Error ¶
func (e *ErrMissingDependency) Error() string
type Extractor ¶
type Extractor interface {
Name() string
// Supports reports whether this extractor can extract a file
// with this MIME and extension. Lowercase comparisons only —
// callers normalize before calling.
Supports(mime, ext string) bool
// Extract reads the file and produces Result. Errors are
// transport (file unreadable, shell tool missing) rather than
// "couldn't extract well" — partial extractions return Result
// with a metadata note like "extraction_quality=partial".
Extract(ctx context.Context, path string) (Result, error)
}
Extractor is one content-type handler. Multiple extractors can claim the same MIME (e.g. text/plain via text and via fallback); the router picks the most-specific first.
type ImageExtractor ¶
type ImageExtractor struct {
Describer vision.Describer
// Prompt steers the description. Empty = describer's default
// ("describe this image in 1-2 sentences").
Prompt string
// Timeout caps the describer call. Default 30s.
Timeout time.Duration
// MaxBytes caps the image file size before reading. Default 10MB (B010).
MaxBytes int64
}
ImageExtractor wraps a vision.Describer. When Describer is nil the extractor still runs — it just returns dimensions + MIME without a caption.
func NewImageExtractor ¶
func NewImageExtractor(d vision.Describer) *ImageExtractor
NewImageExtractor returns an extractor that wraps the given describer. Pass nil to register the extractor without vision — useful for builds without an Anthropic key configured.
func (*ImageExtractor) Name ¶
func (e *ImageExtractor) Name() string
func (*ImageExtractor) Supports ¶
func (e *ImageExtractor) Supports(mime, ext string) bool
type PDFExtractor ¶
type PDFExtractor struct {
// Layout, when true, passes -layout to pdftotext to preserve
// approximate column alignment. Default false — most PDFs read
// better in raw mode for a model.
Layout bool
// MaxBytes caps the extracted text size so a 500-page document
// doesn't blow the model's context budget. Default 256KB. The
// metadata includes the original page count regardless of cap.
MaxBytes int
// Timeout caps the per-extract budget. Default 30s — enough for
// most documents, short enough that a malformed PDF doesn't
// hang the agent loop.
Timeout time.Duration
}
PDFExtractor extracts text + page count via pdftotext + pdfinfo.
func NewPDFExtractor ¶
func NewPDFExtractor() *PDFExtractor
NewPDFExtractor returns an extractor with sensible defaults.
func (*PDFExtractor) Name ¶
func (p *PDFExtractor) Name() string
func (*PDFExtractor) Supports ¶
func (p *PDFExtractor) Supports(mime, ext string) bool
type PandocExtractor ¶
PandocExtractor handles office documents via pandoc.
func NewPandocExtractor ¶
func NewPandocExtractor() *PandocExtractor
NewPandocExtractor returns an extractor with a 60s default timeout.
func (*PandocExtractor) Name ¶
func (e *PandocExtractor) Name() string
func (*PandocExtractor) Supports ¶
func (e *PandocExtractor) Supports(mime, ext string) bool
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the ordered list of extractors. Order matters: more specific extractors register first so they win the Lookup.
func DefaultRegistry ¶
DefaultRegistry returns a registry pre-loaded with the built-in extractors. Pass a non-nil Describer to enable image captioning; nil keeps images on the metadata-only path.
Registration order (first match wins):
- Pandoc (DOCX/DOC/RTF/ODT)
- Audio (whisper)
- Image (vision describer)
- Text (plain text + code + structured config)
- Binary fallback (claims everything)
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry returns an empty registry. Wire built-ins via DefaultRegistry() or Register() manually.
func (*Registry) Extract ¶
Extract is the top-level entry point. Resolves mime + ext from the path, picks an extractor, runs it. Returns ErrNoExtractor when the registry is empty (only possible with a custom registry — the default always has a binary fallback).
type Result ¶
type Result struct {
Text string
Metadata map[string]string
// Extractor names which adapter produced this result. Useful
// for audit logs + for the agent to know whether to trust the
// content depth ("pdftotext" vs "binary-fallback").
Extractor string
}
Result is what every extractor returns. Text is the primary payload — the model reads this directly. Metadata holds bounded key/value pairs (pages, duration, dimensions) so the agent can reason about scope without parsing prose.
type TextExtractor ¶
type TextExtractor struct {
// MaxBytes caps how much of the file we return. Default 256KB.
MaxBytes int
}
TextExtractor reads plain text + structured-text + code files.
func NewTextExtractor ¶
func NewTextExtractor() *TextExtractor
NewTextExtractor returns an extractor with sensible defaults.
func (*TextExtractor) Name ¶
func (t *TextExtractor) Name() string
func (*TextExtractor) Supports ¶
func (t *TextExtractor) Supports(mime, ext string) bool
Source Files