Documentation
¶
Overview ¶
Package extract turns file bytes into plain text for the search index's `content` field ("Bul" wave, v0.2).
Extractors register themselves from init() blocks (same pattern as the storage/db driver registries). The queue's content_index job asks For(mime, ext) whether a file is extractable and, if so, streams the source bytes through the matching Extractor.
Contract notes (shared with the OCR extractor added by S2):
- For() normalizes its inputs before calling Supports: `ext` arrives lower-cased without a leading dot ("pdf", not ".PDF"), `mime` arrives lower-cased with any parameters stripped ("text/plain", not "Text/Plain; charset=utf-8"). Supports implementations can rely on that.
- Extract must honor `limit` as the cap on the RETURNED text in bytes (<=0 means DefaultLimit). Source-size capping is the caller's job.
- "Could not extract" (scanned PDF, corrupt archive, no text layer) is NOT an error: return ("", nil). Reserve errors for transport-level failures (the reader itself failing) so the queue can retry those.
ocr.go — optional OCR extractor backed by the external `tesseract` binary (frozen v0.2 "Bul" contract #4).
Resolution: FILEX_TESSERACT_BIN when set (used exclusively — a bad value disables OCR instead of silently falling back), otherwise a $PATH lookup for "tesseract". No binary → Supports returns false, so the pipeline silently skips images; availability is surfaced to the UI via the capabilities `ocr` flag (internal/capability).
Only raster images are attempted: png / jpg / jpeg / webp / tiff.
Index ¶
Constants ¶
const DefaultLimit int64 = 200 << 10
DefaultLimit is the cap on extracted text, in bytes (200 KiB — frozen in the v0.2 "Bul" contract). Callers pass it (or their own smaller cap) to Extract; <=0 falls back to this value inside every bundled extractor.
Variables ¶
This section is empty.
Functions ¶
func NormalizeExt ¶
NormalizeExt lower-cases ext and strips a leading dot ("PDF"/".pdf" → "pdf").
func NormalizeMime ¶
NormalizeMime lower-cases mime and drops parameters ("Text/Plain; charset=utf-8" → "text/plain").
func Register ¶
func Register(e Extractor)
Register appends e to the extractor chain. Bundled extractors register from init(); external ones (OCR) register during server bootstrap.
func TesseractBin ¶
func TesseractBin() string
TesseractBin resolves the tesseract binary path. FILEX_TESSERACT_BIN, when set, is authoritative (invalid value → ""); otherwise $PATH is searched. Empty string means OCR is unavailable. Shared with internal/capability so the probe and the extractor can never disagree.
Types ¶
type Extractor ¶
type Extractor interface {
// Supports reports whether this extractor handles the (mime, ext)
// pair. Inputs are pre-normalized by For — see the package comment.
Supports(mime, ext string) bool
// Extract reads the source bytes from r and returns plain text capped
// at limit bytes. ("", nil) means "nothing extractable" — not an error.
Extract(ctx context.Context, r io.Reader, limit int64) (string, error)
}
Extractor converts one file format family into plain text.
type OCR ¶
type OCR struct{}
OCR extracts text from images via the external tesseract binary.
func (*OCR) Extract ¶
Extract runs tesseract over the image bytes and returns the recognized plain text, truncated to limit. Input is staged in a temp file (tesseract's stdin handling is unreliable across builds) and removed when done. Tries `-l tur+eng` first; any failure (typically a missing language pack) falls back to a plain invocation.
Source Files