extract

package
v1.21.31 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 12, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package extract is the framework-free document/content extraction library. It imports nothing from the monolith — no Gin, no GORM, no internal/config — so it can be linked into the sandboxed tmi-extractor worker and into the monolith's api package alike.

This package holds the foundation types relocated from package api during TMI Component Platform Plan 2 (issue #347): the extractor interfaces (ContentExtractor, ContextAwareExtractor, BoundedExtractor), the extraction result and entity-reference types, the sentinel errors and reason-code constants, the typed-error classifier (ClassifyError), and the extraction Limits struct with DefaultLimits. The extractor registry and the OOXML/PDF/HTML/plaintext extractors were relocated here by Plan 2 Task 6.

The move is a relocation, not a rewrite: the extraction logic is byte-for-byte the same; only the package boundary, the import paths, and the exported-identifier surface changed.

Index

Constants

View Source 
const (
	ReasonExtractionLimitCompressedSize   = "extraction_limit:compressed_size"
	ReasonExtractionLimitDecompressedSize = "extraction_limit:decompressed_size"
	ReasonExtractionLimitPartSize         = "extraction_limit:part_size"
	ReasonExtractionLimitPartCount        = "extraction_limit:part_count"
	ReasonExtractionLimitMarkdownSize     = "extraction_limit:markdown_size"
	ReasonExtractionLimitTimeout          = "extraction_limit:timeout"
	ReasonExtractionLimitXMLDepth         = "extraction_limit:xml_depth"
	ReasonExtractionLimitZipNested        = "extraction_limit:zip_nested"
	ReasonExtractionLimitZipPath          = "extraction_limit:zip_path"
	ReasonExtractionLimitCompressionRatio = "extraction_limit:compression_ratio"
	ReasonExtractionMalformed             = "extraction_malformed"
	ReasonExtractionUnsupported           = "extraction_unsupported"
	ReasonExtractionInternal              = "extraction_internal"
)

Extraction access-reason-code constants. Relocated verbatim from api/access_diagnostics.go so the monolith and the worker agree on the strings written to access_reason_code / the result envelope.

Variables

View Source 
var (
	ErrExtractionLimit = errors.New("extraction limit exceeded")
	ErrMalformed       = errors.New("malformed document")
	ErrUnsupported     = errors.New("unsupported document subformat")
)

Sentinel errors returned by extractors. Callers use errors.Is to classify.

Functions

func ExtractTextFromHTML 

func ExtractTextFromHTML(htmlContent string) string

ExtractTextFromHTML parses an HTML document and returns the concatenated visible text, skipping content inside <script> and <style> elements. This is the single source of truth shared by the HTMLExtractor and the api HTTP content provider.

func NewLimitError 

func NewLimitError(kind, detail string) error

NewLimitError builds a typed extraction-limit error for the given Kind (one of the documented Kind values: compressed_size, decompressed_size, part_size, part_count, markdown_size, xml_depth, zip_nested, zip_path, compression_ratio) with an optional human-readable detail. The result satisfies errors.Is(err, ErrExtractionLimit) and is classified by ClassifyError into the matching reason code.

The extractors themselves construct the package-private value directly; this constructor exists so callers outside the package (e.g. the monolith's pipeline/poller tests) can inject a classifiable limit error without depending on the private type.

Limit and Observed are left zero — this constructor is for classification and test injection, not for producing human-facing limit diagnostics.

Types

type BoundedExtractor 

type BoundedExtractor interface {
	Bounded() bool
}

BoundedExtractor is implemented by extractors that must run under a wall-clock deadline (CPU- or memory-heavy extractors that could otherwise run indefinitely on adversarial input). The pipeline calls Bounded() to detect the requirement; the value is informational and always true for types that implement it.

type Classification 

type Classification struct {
	ReasonCode   string
	ReasonDetail string
}

Classification describes how a typed extractor error maps to a reason code, plus an optional human-readable Detail. Relocated from api.ExtractionClassification; the Status field is dropped because access_status is a monolith concept — the worker reports only reason codes, and the monolith's result-consumer (Plan 3) derives access_status.

func ClassifyError 

func ClassifyError(err error) Classification

ClassifyError walks the error chain and returns the matching reason code. Default is ReasonExtractionInternal. A nil error returns the zero value.

type ContentExtractor 

type ContentExtractor interface {
	Name() string
	CanHandle(contentType string) bool
	Extract(data []byte, contentType string) (ExtractedContent, error)
}

ContentExtractor converts raw bytes into plain text.

type ContentExtractorRegistry 

type ContentExtractorRegistry struct {
	// contains filtered or unexported fields
}

ContentExtractorRegistry manages content extractors in priority order.

func NewContentExtractorRegistry 

func NewContentExtractorRegistry() *ContentExtractorRegistry

NewContentExtractorRegistry creates a new registry.

func (*ContentExtractorRegistry) FindExtractor 

func (r *ContentExtractorRegistry) FindExtractor(contentType string) (ContentExtractor, bool)

FindExtractor returns the first extractor that can handle the given content type.

func (*ContentExtractorRegistry) Register 

func (r *ContentExtractorRegistry) Register(extractor ContentExtractor)

Register adds an extractor to the registry.

type ContextAwareExtractor 

type ContextAwareExtractor interface {
	ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)
}

ContextAwareExtractor is implemented by extractors that can receive a deadline-bearing context for cooperative cancellation. When an extractor implements both BoundedExtractor and ContextAwareExtractor, the pipeline's ExtractWithDeadline wrapper calls ExtractCtx with the timeout-bounded context so any wall-clock cancellation aborts in-flight reads through the archive's boundedReader (which checks ctx.Err() per Read call).

Extractors that only implement BoundedExtractor (without ContextAwareExtractor) still get a wall-clock deadline at the goroutine boundary, but in-flight I/O continues until it finishes naturally — the pipeline returns DeadlineExceeded promptly while the extractor goroutine drains in the background.

Extract should remain implemented as the legacy entry point and typically delegates to ExtractCtx with a context.Background().

type DOCXExtractor 

type DOCXExtractor struct {
	// contains filtered or unexported fields
}

DOCXExtractor extracts Markdown-flavored text from a DOCX (OOXML) archive.

func NewDOCXExtractor 

func NewDOCXExtractor(limits Limits) *DOCXExtractor

NewDOCXExtractor returns an extractor configured with the given limits.

func (*DOCXExtractor) Bounded 

func (e *DOCXExtractor) Bounded() bool

Bounded marks DOCXExtractor as needing a wall-clock deadline.

func (*DOCXExtractor) CanHandle 

func (e *DOCXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the DOCX OOXML MIME type.

func (*DOCXExtractor) Extract 

func (e *DOCXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract parses a DOCX archive and produces Markdown-flavored text. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*DOCXExtractor) ExtractCtx 

func (e *DOCXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point used by the pipeline's ExtractWithDeadline wrapper. The supplied ctx is wired into the OOXML archive so that all member-level boundedReaders abort their reads on cancellation (wall-clock deadline or parent cancel).

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*DOCXExtractor) Name 

func (e *DOCXExtractor) Name() string

Name returns the extractor name as registered with the registry.

type EntityReference 

type EntityReference struct {
	EntityType string // "asset", "threat", "document", "note", "diagram", "repository"
	EntityID   string // UUID of the source entity
	URI        string // External URL (empty for DB-resident content)
	Name       string // Display name for progress reporting
}

EntityReference identifies a source entity for content extraction. For DB-resident content (notes, assets), URI is empty and the provider reads directly from the database using EntityType + EntityID. For external content (documents with URLs), URI is the fetch target.

type ExtractedContent 

type ExtractedContent struct {
	Text        string            // Extracted plain text
	Title       string            // Document title if available
	ContentType string            // Original content type (e.g., "application/pdf")
	Metadata    map[string]string // Provider-specific metadata
}

ExtractedContent holds the text extracted from a source entity.

func ExtractWithDeadline 

func ExtractWithDeadline(ctx context.Context, budget time.Duration, fn func(context.Context) (ExtractedContent, error)) (ExtractedContent, error)

ExtractWithDeadline runs an extraction function under a wall-clock budget, returning early if the budget elapses. It is the entry point the extractor worker uses to bound CPU/memory-heavy extractors on adversarial input. On timeout it returns context.DeadlineExceeded; on parent cancellation it returns ctx.Err(). The wrapped fn receives the deadline-bearing context so cooperative cancellation is possible.

type HTMLExtractor 

type HTMLExtractor struct{}

HTMLExtractor extracts visible text from HTML content, stripping script and style elements.

func NewHTMLExtractor 

func NewHTMLExtractor() *HTMLExtractor

NewHTMLExtractor creates a new HTMLExtractor.

func (*HTMLExtractor) CanHandle 

func (e *HTMLExtractor) CanHandle(contentType string) bool

CanHandle returns true for text/html content types.

func (*HTMLExtractor) Extract 

func (e *HTMLExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract strips HTML tags and returns the visible text. Script and style element content is excluded.

func (*HTMLExtractor) Name 

func (e *HTMLExtractor) Name() string

Name returns the extractor name.

type Limits 

type Limits struct {
	CompressedSizeBytes   int64
	DecompressedSizeBytes int64
	PartSizeBytes         int64
	MarkdownSizeBytes     int64
	MaxXMLElementDepth    int
	MaxCompressionRatio   int64
	PPTXSlides            int
	XLSXCells             int
	// WallClockBudget is the per-extraction deadline (0 disables).
	WallClockBudget time.Duration
}

Limits is the set of extraction caps the OOXML opener, the XML decoder, and the wall-clock budget enforce. Relocated and exported from the package-private ooxmlLimits.

func DefaultLimits 

func DefaultLimits() Limits

DefaultLimits returns the design-spec default values.

type PDFExtractor 

type PDFExtractor struct{}

PDFExtractor extracts plain text from PDF binary content. Because the PDF library requires a seekable reader, the bytes are written to a temporary file which is removed after extraction.

func NewPDFExtractor 

func NewPDFExtractor() *PDFExtractor

NewPDFExtractor creates a new PDFExtractor.

func (*PDFExtractor) CanHandle 

func (e *PDFExtractor) CanHandle(contentType string) bool

CanHandle returns true for application/pdf content types.

func (*PDFExtractor) Extract 

func (e *PDFExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract writes the PDF bytes to a temp file and extracts plain text page by page.

func (*PDFExtractor) Name 

func (e *PDFExtractor) Name() string

Name returns the extractor name.

type PPTXExtractor 

type PPTXExtractor struct {
	// contains filtered or unexported fields
}

PPTXExtractor extracts Markdown-flavored text from a PPTX (OOXML) archive.

func NewPPTXExtractor 

func NewPPTXExtractor(limits Limits) *PPTXExtractor

NewPPTXExtractor returns an extractor configured with the given limits.

func (*PPTXExtractor) Bounded 

func (e *PPTXExtractor) Bounded() bool

Bounded marks PPTXExtractor as needing a wall-clock deadline.

func (*PPTXExtractor) CanHandle 

func (e *PPTXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the PPTX OOXML MIME type.

func (*PPTXExtractor) Extract 

func (e *PPTXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract parses a PPTX archive and produces Markdown-flavored text. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*PPTXExtractor) ExtractCtx 

func (e *PPTXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point. Slides appear in document order under "## Slide N: <title>" headings; hidden slides are skipped but still consume slide numbers. Speaker notes (when present) follow each slide as a "### Notes" subsection. The supplied ctx is wired into the archive so all member-level reads abort on cancellation.

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*PPTXExtractor) Name 

func (e *PPTXExtractor) Name() string

Name returns the extractor name as registered with the registry.

type PlainTextExtractor 

type PlainTextExtractor struct{}

PlainTextExtractor extracts content from plain text and CSV content types.

func NewPlainTextExtractor 

func NewPlainTextExtractor() *PlainTextExtractor

NewPlainTextExtractor creates a new PlainTextExtractor.

func (*PlainTextExtractor) CanHandle 

func (e *PlainTextExtractor) CanHandle(contentType string) bool

CanHandle returns true for text/plain and text/csv content types.

func (*PlainTextExtractor) Extract 

func (e *PlainTextExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract returns the raw bytes as a string with no transformation.

func (*PlainTextExtractor) Name 

func (e *PlainTextExtractor) Name() string

Name returns the extractor name.

type TextChunker 

type TextChunker struct {
	// contains filtered or unexported fields
}

TextChunker splits text into chunks suitable for embedding

func NewTextChunker 

func NewTextChunker(maxChunkSize, overlap int) *TextChunker

NewTextChunker creates a chunker with the given size and overlap (in characters)

func (*TextChunker) Chunk 

func (tc *TextChunker) Chunk(text string) []string

Chunk splits text into chunks at sentence boundaries

type XLSXExtractor 

type XLSXExtractor struct {
	// contains filtered or unexported fields
}

XLSXExtractor extracts Markdown-flavored text from an XLSX (OOXML) workbook. Uses github.com/xuri/excelize/v2 for archive parsing, shared-strings handling, and number-format rendering. Limit enforcement (cell count, markdown cap) is applied during the streaming render.

Header detection follows the design spec: a per-row style fingerprint (dominant bgColor, fontSize, fontWeight, fontItalic) is computed across the first ≤5 non-empty rows, with a content-heuristic fallback when style fingerprints are uniform across the inspected rows. See xlsxDetectHeader.

func NewXLSXExtractor 

func NewXLSXExtractor(limits Limits) *XLSXExtractor

NewXLSXExtractor returns an extractor configured with the given limits.

func (*XLSXExtractor) Bounded 

func (e *XLSXExtractor) Bounded() bool

Bounded marks XLSXExtractor as needing a wall-clock deadline.

func (*XLSXExtractor) CanHandle 

func (e *XLSXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the XLSX OOXML MIME type.

func (*XLSXExtractor) Extract 

func (e *XLSXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract opens the workbook, walks visible sheets in tab order, and writes Markdown-flavored text into a bounded markdown builder. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*XLSXExtractor) ExtractCtx 

func (e *XLSXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point. Each visible sheet is rendered as `## Sheet: <name>` followed by a pipe table of trimmed cells. The supplied ctx is checked between sheets and inside the row loop so wall-clock cancellation aborts long workbooks promptly.

On non-nil error, the returned ExtractedContent is zero and must be discarded.

func (*XLSXExtractor) Name 

func (e *XLSXExtractor) Name() string

Name returns the extractor name as registered with the registry.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.