Help us shape the future of Go! Take the 2026 Go Developer Survey

extract

package
v1.21.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package extract is the framework-free document/content extraction library. It imports nothing from the monolith — no Gin, no GORM, no internal/config — so it can be linked into the sandboxed tmi-extractor worker and into the monolith's api package alike.

This package holds the foundation types relocated from package api during TMI Component Platform Plan 2 (issue #347): the extractor interfaces (ContentExtractor, ContextAwareExtractor, BoundedExtractor), the extraction result and entity-reference types, the sentinel errors and reason-code constants, the typed-error classifier (ClassifyError), and the extraction Limits struct with DefaultLimits. The extractor registry and the OOXML/PDF/HTML/plaintext extractors were relocated here by Plan 2 Task 6.

The move is a relocation, not a rewrite: the extraction logic is byte-for-byte the same; only the package boundary, the import paths, and the exported-identifier surface changed.

Index

Constants

View Source 
const (
	ReasonExtractionLimitCompressedSize   = "extraction_limit:compressed_size"
	ReasonExtractionLimitDecompressedSize = "extraction_limit:decompressed_size"
	ReasonExtractionLimitPartSize         = "extraction_limit:part_size"
	ReasonExtractionLimitPartCount        = "extraction_limit:part_count"
	ReasonExtractionLimitMarkdownSize     = "extraction_limit:markdown_size"
	ReasonExtractionLimitTimeout          = "extraction_limit:timeout"
	ReasonExtractionLimitXMLDepth         = "extraction_limit:xml_depth"
	ReasonExtractionLimitZipNested        = "extraction_limit:zip_nested"
	ReasonExtractionLimitZipPath          = "extraction_limit:zip_path"
	ReasonExtractionLimitCompressionRatio = "extraction_limit:compression_ratio"
	ReasonExtractionMalformed             = "extraction_malformed"
	ReasonExtractionUnsupported           = "extraction_unsupported"
	ReasonExtractionInternal              = "extraction_internal"
)

Extraction access-reason-code constants. Relocated verbatim from api/access_diagnostics.go so the monolith and the worker agree on the strings written to access_reason_code / the result envelope.

Variables

View Source 
var (
	ErrExtractionLimit = errors.New("extraction limit exceeded")
	ErrMalformed       = errors.New("malformed document")
	ErrUnsupported     = errors.New("unsupported document subformat")
)

Sentinel errors returned by extractors. Callers use errors.Is to classify.

Functions

func ExtractTextFromHTML 

func ExtractTextFromHTML(htmlContent string) string

ExtractTextFromHTML parses an HTML document and returns the concatenated visible text, skipping content inside <script> and <style> elements. This is the single source of truth shared by the HTMLExtractor and the api HTTP content provider. SEM@d056a3ea026249d40d05ab6af7f092a043f72c7a: parse an HTML document and concatenate visible text, skipping script and style subtrees (pure)

func NewLimitError 

func NewLimitError(kind, detail string) error

NewLimitError builds a typed extraction-limit error for the given Kind (one of the documented Kind values: compressed_size, decompressed_size, part_size, part_count, markdown_size, xml_depth, zip_nested, zip_path, compression_ratio) with an optional human-readable detail. The result satisfies errors.Is(err, ErrExtractionLimit) and is classified by ClassifyError into the matching reason code.

The extractors themselves construct the package-private value directly; this constructor exists so callers outside the package (e.g. the monolith's pipeline/poller tests) can inject a classifiable limit error without depending on the private type.

Limit and Observed are left zero — this constructor is for classification and test injection, not for producing human-facing limit diagnostics. SEM@d1fd850907490887fd11a6ccd4a691326ede6e4e: build a classifiable extraction-limit error for a given kind and detail (pure)

Types

type BoundedExtractor 

type BoundedExtractor interface {
	Bounded() bool
}

BoundedExtractor is implemented by extractors that must run under a wall-clock deadline (CPU- or memory-heavy extractors that could otherwise run indefinitely on adversarial input). The pipeline calls Bounded() to detect the requirement; the value is informational and always true for types that implement it. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: interface marking extractors that require a wall-clock deadline for safety (pure)

type Classification 

type Classification struct {
	ReasonCode   string
	ReasonDetail string
}

Classification describes how a typed extractor error maps to a reason code, plus an optional human-readable Detail. Relocated from api.ExtractionClassification; the Status field is dropped because access_status is a monolith concept — the worker reports only reason codes, and the monolith's result-consumer (Plan 3) derives access_status. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: pair a reason code with an optional human-readable detail for an extraction error (pure)

func ClassifyError 

func ClassifyError(err error) Classification

ClassifyError walks the error chain and returns the matching reason code. Default is ReasonExtractionInternal. A nil error returns the zero value. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: walk an error chain and return the matching extraction reason code (pure)

type ContentExtractor 

type ContentExtractor interface {
	Name() string
	CanHandle(contentType string) bool
	Extract(data []byte, contentType string) (ExtractedContent, error)
}

ContentExtractor converts raw bytes into plain text. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: interface for converting raw bytes of a known content type into extracted text (pure)

type ContentExtractorRegistry 

type ContentExtractorRegistry struct {
	// contains filtered or unexported fields
}

ContentExtractorRegistry manages content extractors in priority order. SEM@b4a403da2147ccb51a674e10d71891d4fccfe06a: ordered registry of content extractors matched by content type (pure)

func NewContentExtractorRegistry 

func NewContentExtractorRegistry() *ContentExtractorRegistry

NewContentExtractorRegistry creates a new registry. SEM@b4a403da2147ccb51a674e10d71891d4fccfe06a: build an empty content extractor registry (pure)

func (*ContentExtractorRegistry) FindExtractor 

func (r *ContentExtractorRegistry) FindExtractor(contentType string) (ContentExtractor, bool)

FindExtractor returns the first extractor that can handle the given content type. SEM@b4a403da2147ccb51a674e10d71891d4fccfe06a: return the first registered extractor that can handle the given content type (pure)

func (*ContentExtractorRegistry) Register 

func (r *ContentExtractorRegistry) Register(extractor ContentExtractor)

Register adds an extractor to the registry. SEM@b4a403da2147ccb51a674e10d71891d4fccfe06a: register a content extractor at the end of the priority list (mutates shared state)

type ContextAwareExtractor 

type ContextAwareExtractor interface {
	ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)
}

ContextAwareExtractor is implemented by extractors that can receive a deadline-bearing context for cooperative cancellation. When an extractor implements both BoundedExtractor and ContextAwareExtractor, the pipeline's ExtractWithDeadline wrapper calls ExtractCtx with the timeout-bounded context so any wall-clock cancellation aborts in-flight reads through the archive's boundedReader (which checks ctx.Err() per Read call).

Extractors that only implement BoundedExtractor (without ContextAwareExtractor) still get a wall-clock deadline at the goroutine boundary, but in-flight I/O continues until it finishes naturally — the pipeline returns DeadlineExceeded promptly while the extractor goroutine drains in the background.

Extract should remain implemented as the legacy entry point and typically delegates to ExtractCtx with a context.Background(). SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: interface for extractors that accept a context for cooperative cancellation (pure)

type DOCXExtractor 

type DOCXExtractor struct {
	// contains filtered or unexported fields
}

DOCXExtractor extracts Markdown-flavored text from a DOCX (OOXML) archive. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extractor that converts DOCX archives to Markdown-flavored text (pure)

func NewDOCXExtractor 

func NewDOCXExtractor(limits Limits) *DOCXExtractor

NewDOCXExtractor returns an extractor configured with the given limits. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a DOCXExtractor configured with the given extraction limits (pure)

func (*DOCXExtractor) Bounded 

func (e *DOCXExtractor) Bounded() bool

Bounded marks DOCXExtractor as needing a wall-clock deadline. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: signal that this extractor requires a wall-clock deadline context (pure)

func (*DOCXExtractor) CanHandle 

func (e *DOCXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the DOCX OOXML MIME type. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: validate that the content type matches the DOCX OOXML MIME type (pure)

func (*DOCXExtractor) Extract 

func (e *DOCXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract parses a DOCX archive and produces Markdown-flavored text. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: parse a DOCX archive and return Markdown text using a background context

func (*DOCXExtractor) ExtractCtx 

func (e *DOCXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point used by the pipeline's ExtractWithDeadline wrapper. The supplied ctx is wired into the OOXML archive so that all member-level boundedReaders abort their reads on cancellation (wall-clock deadline or parent cancel).

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: parse a DOCX archive with context cancellation and return Markdown text and title

func (*DOCXExtractor) Name 

func (e *DOCXExtractor) Name() string

Name returns the extractor name as registered with the registry. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the registered extractor name for DOCX files (pure)

type EntityReference 

type EntityReference struct {
	EntityType string // "asset", "threat", "document", "note", "diagram", "repository"
	EntityID   string // UUID of the source entity
	URI        string // External URL (empty for DB-resident content)
	Name       string // Display name for progress reporting
}

EntityReference identifies a source entity for content extraction. For DB-resident content (notes, assets), URI is empty and the provider reads directly from the database using EntityType + EntityID. For external content (documents with URLs), URI is the fetch target. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: identifies a source entity for content extraction by type, ID, and optional URI (pure)

type ExtractedContent 

type ExtractedContent struct {
	Text        string            // Extracted plain text
	Title       string            // Document title if available
	ContentType string            // Original content type (e.g., "application/pdf")
	Metadata    map[string]string // Provider-specific metadata
}

ExtractedContent holds the text extracted from a source entity. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: holds plain text and metadata produced by a content extractor (pure)

func ExtractWithDeadline 

func ExtractWithDeadline(ctx context.Context, budget time.Duration, fn func(context.Context) (ExtractedContent, error)) (ExtractedContent, error)

ExtractWithDeadline runs an extraction function under a wall-clock budget, returning early if the budget elapses. It is the entry point the extractor worker uses to bound CPU/memory-heavy extractors on adversarial input. On timeout it returns context.DeadlineExceeded; on parent cancellation it returns ctx.Err(). The wrapped fn receives the deadline-bearing context so cooperative cancellation is possible. SEM@4359a42e5ac4f24a1d76d8c8c8455c6cc91541c5: run an extractor under a wall-clock budget, cancelling on timeout or parent context

type HTMLExtractor 

type HTMLExtractor struct{}

HTMLExtractor extracts visible text from HTML content, stripping script and style elements. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extracts visible text from HTML content, excluding script and style elements (pure)

func NewHTMLExtractor 

func NewHTMLExtractor() *HTMLExtractor

NewHTMLExtractor creates a new HTMLExtractor. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a new HTMLExtractor (pure)

func (*HTMLExtractor) CanHandle 

func (e *HTMLExtractor) CanHandle(contentType string) bool

CanHandle returns true for text/html content types. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report whether the content type is text/html (pure)

func (*HTMLExtractor) Extract 

func (e *HTMLExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract strips HTML tags and returns the visible text. Script and style element content is excluded. SEM@f69d82c5864c65f768181d82416962e3bbd63667: parse HTML bytes and return visible text as extracted content (pure)

func (*HTMLExtractor) Name 

func (e *HTMLExtractor) Name() string

Name returns the extractor name. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the extractor's identifying name (pure)

type Limits 

type Limits struct {
	CompressedSizeBytes   int64
	DecompressedSizeBytes int64
	PartSizeBytes         int64
	MarkdownSizeBytes     int64
	MaxXMLElementDepth    int
	MaxCompressionRatio   int64
	PPTXSlides            int
	XLSXCells             int
	// WallClockBudget is the per-extraction deadline (0 disables).
	WallClockBudget time.Duration
}

Limits is the set of extraction caps the OOXML opener, the XML decoder, and the wall-clock budget enforce. Relocated and exported from the package-private ooxmlLimits. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: extraction size and depth caps applied to OOXML, XML decoding, and wall-clock budget (pure)

func DefaultLimits 

func DefaultLimits() Limits

DefaultLimits returns the design-spec default values. SEM@f7dfe970572e2574027691de97c695d5ae39d5b7: return design-spec default extraction limits (pure)

type PDFExtractor 

type PDFExtractor struct{}

PDFExtractor extracts plain text from PDF binary content. Because the PDF library requires a seekable reader, the bytes are written to a temporary file which is removed after extraction. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extractor that converts PDF binary content to plain text via a temporary file (pure)

func NewPDFExtractor 

func NewPDFExtractor() *PDFExtractor

NewPDFExtractor creates a new PDFExtractor. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a PDFExtractor instance (pure)

func (*PDFExtractor) CanHandle 

func (e *PDFExtractor) CanHandle(contentType string) bool

CanHandle returns true for application/pdf content types. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: validate that a content type is application/pdf (pure)

func (*PDFExtractor) Extract 

func (e *PDFExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract writes the PDF bytes to a temp file and extracts plain text page by page. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: convert PDF bytes to plain text via a temporary file, returning extracted content

func (*PDFExtractor) Name 

func (e *PDFExtractor) Name() string

Name returns the extractor name. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the canonical name of the PDF extractor (pure)

type PPTXExtractor 

type PPTXExtractor struct {
	// contains filtered or unexported fields
}

PPTXExtractor extracts Markdown-flavored text from a PPTX (OOXML) archive. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extractor for PPTX files that converts slides to Markdown text within configured limits (pure)

func NewPPTXExtractor 

func NewPPTXExtractor(limits Limits) *PPTXExtractor

NewPPTXExtractor returns an extractor configured with the given limits. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a PPTXExtractor configured with the given resource limits (pure)

func (*PPTXExtractor) Bounded 

func (e *PPTXExtractor) Bounded() bool

Bounded marks PPTXExtractor as needing a wall-clock deadline. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report that this extractor requires a wall-clock deadline (pure)

func (*PPTXExtractor) CanHandle 

func (e *PPTXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the PPTX OOXML MIME type. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report whether the content type is the PPTX OOXML MIME type (pure)

func (*PPTXExtractor) Extract 

func (e *PPTXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract parses a PPTX archive and produces Markdown-flavored text. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: parse a PPTX archive and return Markdown text using a background context (pure)

func (*PPTXExtractor) ExtractCtx 

func (e *PPTXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point. Slides appear in document order under "## Slide N: <title>" headings; hidden slides are skipped but still consume slide numbers. Speaker notes (when present) follow each slide as a "### Notes" subsection. The supplied ctx is wired into the archive so all member-level reads abort on cancellation.

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: parse a PPTX archive into Markdown with slide headings, speaker notes, and cancellation support (pure)

func (*PPTXExtractor) Name 

func (e *PPTXExtractor) Name() string

Name returns the extractor name as registered with the registry. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the extractor's registry name "pptx" (pure)

type PlainTextExtractor 

type PlainTextExtractor struct{}

PlainTextExtractor extracts content from plain text and CSV content types. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extractor for plain text and CSV content types (pure)

func NewPlainTextExtractor 

func NewPlainTextExtractor() *PlainTextExtractor

NewPlainTextExtractor creates a new PlainTextExtractor. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a PlainTextExtractor (pure)

func (*PlainTextExtractor) CanHandle 

func (e *PlainTextExtractor) CanHandle(contentType string) bool

CanHandle returns true for text/plain and text/csv content types. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report whether the extractor handles text/plain or text/csv content types (pure)

func (*PlainTextExtractor) Extract 

func (e *PlainTextExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract returns the raw bytes as a string with no transformation. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: convert raw bytes to extracted content with no transformation (pure)

func (*PlainTextExtractor) Name 

func (e *PlainTextExtractor) Name() string

Name returns the extractor name. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the extractor's canonical name (pure)

type TextChunker 

type TextChunker struct {
	// contains filtered or unexported fields
}

TextChunker splits text into chunks suitable for embedding SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: splits text into overlapping chunks sized for embedding (pure)

func NewTextChunker 

func NewTextChunker(maxChunkSize, overlap int) *TextChunker

NewTextChunker creates a chunker with the given size and overlap (in characters) SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build a TextChunker with a given max chunk size and overlap in characters (pure)

func (*TextChunker) Chunk 

func (tc *TextChunker) Chunk(text string) []string

Chunk splits text into chunks at sentence boundaries SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: split text into sentence-boundary-aligned chunks with optional overlap (pure)

type XLSXExtractor 

type XLSXExtractor struct {
	// contains filtered or unexported fields
}

XLSXExtractor extracts Markdown-flavored text from an XLSX (OOXML) workbook. Uses github.com/xuri/excelize/v2 for archive parsing, shared-strings handling, and number-format rendering. Limit enforcement (cell count, markdown cap) is applied during the streaming render.

Header detection follows the design spec: a per-row style fingerprint (dominant bgColor, fontSize, fontWeight, fontItalic) is computed across the first ≤5 non-empty rows, with a content-heuristic fallback when style fingerprints are uniform across the inspected rows. See xlsxDetectHeader. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extractor that converts XLSX workbook sheets to Markdown pipe tables within configured limits

func NewXLSXExtractor 

func NewXLSXExtractor(limits Limits) *XLSXExtractor

NewXLSXExtractor returns an extractor configured with the given limits. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: build an XLSXExtractor configured with the given extraction limits (pure)

func (*XLSXExtractor) Bounded 

func (e *XLSXExtractor) Bounded() bool

Bounded marks XLSXExtractor as needing a wall-clock deadline. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report that this extractor requires a wall-clock deadline (pure)

func (*XLSXExtractor) CanHandle 

func (e *XLSXExtractor) CanHandle(contentType string) bool

CanHandle returns true iff contentType is the XLSX OOXML MIME type. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: report whether the content type matches the XLSX OOXML MIME type (pure)

func (*XLSXExtractor) Extract 

func (e *XLSXExtractor) Extract(data []byte, contentType string) (ExtractedContent, error)

Extract opens the workbook, walks visible sheets in tab order, and writes Markdown-flavored text into a bounded markdown builder. This is the legacy entry point that delegates to ExtractCtx with a background context (no cooperative cancellation).

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extract Markdown text from XLSX bytes without cooperative cancellation

func (*XLSXExtractor) ExtractCtx 

func (e *XLSXExtractor) ExtractCtx(ctx context.Context, data []byte, contentType string) (ExtractedContent, error)

ExtractCtx is the context-aware extraction entry point. Each visible sheet is rendered as `## Sheet: <name>` followed by a pipe table of trimmed cells. The supplied ctx is checked between sheets and inside the row loop so wall-clock cancellation aborts long workbooks promptly.

On non-nil error, the returned ExtractedContent is zero and must be discarded. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: extract Markdown pipe tables from all visible XLSX sheets with context cancellation and limit enforcement

func (*XLSXExtractor) Name 

func (e *XLSXExtractor) Name() string

Name returns the extractor name as registered with the registry. SEM@d1c9c93fe4dd63680a390679e8df436b39c27a8b: return the registered extractor name for XLSX files (pure)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.