documents

package
v0.5.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 18, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package documents provides bounded, provenance-rich text extraction from local structured and markup files — CSV, TSV, JSON, JSONL, XML, and HTML — for the read_document agent tool. It is a pure content library: it knows nothing about cwd resolution or the credential-path denylist (that trust boundary lives at the agent-tool layer, same as read_file's ValidateReadPath), and takes an already-resolved, already-validated absolute path.

Every extractor enforces its byte/row/char caps before or during parsing rather than reading the whole file into memory first, so a pathological input (huge file, one giant line with no row boundaries) can't exhaust memory. Every returned section carries a provenance label ("rows 1-40", "JSONL line 412", "heading: Results") so the model can cite an exact location instead of an offset into an opaque blob.

Index

Constants

View Source
const (
	// DefaultMaxBytes bounds how much of the source file any extractor
	// reads, regardless of how many rows/records that covers. This is
	// the hard ceiling against pathological input (a single line with
	// no row boundaries, for instance) where row/char caps alone
	// wouldn't stop unbounded buffering.
	DefaultMaxBytes int64 = 5 * 1024 * 1024 // 5 MiB

	// DefaultMaxRows bounds how many logical rows/records (CSV/TSV rows,
	// JSONL records) an extractor samples into the preview.
	DefaultMaxRows = 200

	// DefaultMaxChars bounds the total extracted text (JSON/XML/HTML
	// content preview) so a single document can't dominate the model's
	// context window.
	DefaultMaxChars = 20000

	// DefaultMaxPages bounds how many PDF pages, or pptx slides, an
	// extractor reads text from. Every other extractor ignores it.
	DefaultMaxPages = 50

	// MaxAllowedBytes is the ceiling on a caller-supplied MaxBytes.
	// MaxBytes is the only cap that needs one: MaxRows and MaxChars are
	// transitively bounded by it (no extractor can sample more rows or
	// characters than the bytes it was allowed to read), so bounding
	// bytes bounds everything downstream.
	//
	// The ceiling exists mainly for the .json path, which is the one
	// extractor that buffers the whole allowance and decodes it into an
	// `any` tree — a representation several times larger than the source
	// text. The streaming extractors (CSV/TSV/JSONL/XML/HTML) grow only
	// linearly with the allowance. 32 MiB is ~6x the default, which
	// covers the realistic "my export is bigger than 5 MiB" case while
	// keeping even the amplified JSON worst case survivable on a dev
	// machine.
	MaxAllowedBytes int64 = 32 * 1024 * 1024 // 32 MiB
)
View Source
const (
	SlideWidthEMU  int64 = 12192000
	SlideHeightEMU int64 = 6858000
	// EMUPerInch converts inches to/from EMU (the OOXML native length
	// unit): 914400 EMU per inch, fixed by the OOXML spec.
	EMUPerInch int64 = 914400
)

SlideWidthEMU and SlideHeightEMU are the fixed widescreen (16:9) slide dimensions GeneratePPTX always renders at (see pptxPresentationXML). Exported so callers validating an ImageBounds against the slide can do it without duplicating these numbers.

View Source
const DocumentsImageHelperDir = "/opt/yottacode/doc-helpers"

DocumentsImageHelperDir is the fixed path infra/documents.Containerfile COPYs the driver scripts to inside the documents image. A podman-sandboxed call invokes a script directly at this path — see ResolvePyHelperScript.

Variables

View Source
var ErrUnsupported = fmt.Errorf("unsupported document type — read_document handles .csv, .tsv, .json, .jsonl, .xml, .html, .xlsx, .docx, .pptx, and (when a command sandbox can reach pdftotext) .pdf")

ErrUnsupported reports that no registered extractor recognizes the given path's extension. Kept here so every caller (the read_document tool today, any future caller later) shares one message.

Functions

func GeneratePPTX

func GeneratePPTX(slides []SlideModel) ([]byte, error)

GeneratePPTX renders slides into a minimal Office Open XML presentation. It is pure Go: no python3, python-pptx, LibreOffice, or sandbox process is required. The generated package includes the standard presentation parts, slide XML, optional note slides, and optional image relationships.

func GenerateXLSX

func GenerateXLSX(model SheetModel) ([]byte, error)

GenerateXLSX renders model into xlsx file bytes via excelize. Pure Go, no subprocess — this path never needs the command sandbox.

func RenderMarkdown

func RenderMarkdown(ast DocAST) (string, error)

RenderMarkdown renders ast as Pandoc-flavored Markdown text. Every block's text content is escaped (see escapeMarkdown) so user-supplied content can't inject unintended Markdown structure (a paragraph starting with "# " becoming a heading, a table cell containing "|" splitting a column, etc).

func ResolvePyHelperScript

func ResolvePyHelperScript(script PyHelperScript, isPodmanSandbox bool, cacheDir string) (string, error)

ResolvePyHelperScript returns the filesystem path script should be invoked at, given whether the active command sandbox is the documents-image-backed podman sandbox (scripts already baked in at DocumentsImageHelperDir by the Containerfile) or anything else — HostSandbox, or any other future backend — in which case the embedded script is materialized under cacheDir once.

isPodmanSandbox is a caller-supplied bool rather than this package inspecting a Sandbox itself: internal/documents cannot import internal/agent (see this package's own dependency-direction note in pdf.go's CommandRunner doc comment), so the caller — which owns the Sandbox interface — makes that call and passes the answer in, the same pattern CommandRunner itself already uses for the pandoc/pdftotext subprocess seam.

The materialized copy is refreshed whenever its on-disk content doesn't match the embedded content byte-for-byte (missing, corrupted, or left over from an older yottacode binary with different script source) — no separate version file to keep in sync; the comparison itself is the freshness check.

Types

type Block

type Block struct {
	// Type selects which other fields apply: "heading", "paragraph",
	// "list", "table", "code", or "image".
	Type string

	// Level is the heading level (1-6). heading only.
	Level int

	// Text is the block's plain text content. heading, paragraph, and
	// code. For heading/paragraph, Spans takes precedence when non-empty
	// — Text is the fallback plain-text path.
	Text string

	// Spans, when non-empty, renders inline-formatted text (bold/italic
	// runs) instead of Text. heading and paragraph only. Structured
	// spans rather than an inline-markdown string: each span's Text
	// still passes through escapeMarkdown, so a model can never smuggle
	// real Markdown/HTML syntax through a "formatting" field the way it
	// could through a raw markup string — see escapeMarkdown's own doc
	// comment for why that distinction matters.
	Spans []Span

	// Ordered selects a numbered (true) vs bulleted (false) list. list only.
	Ordered bool

	// Items is the list's plain-text entries, one per line. list only.
	// When ItemSpans is non-empty, it takes precedence per item.
	Items []string

	// ItemSpans, when non-empty, gives each list item inline formatting
	// the same way Spans does for heading/paragraph text. list only.
	// Must have the same length as Items when both are set; a shorter
	// ItemSpans falls back to the corresponding Items entry for the
	// missing tail.
	ItemSpans [][]Span

	// Header is the table's column headers. table only.
	Header []string

	// Rows is the table's body, one slice of cells per row. table only.
	Rows [][]string

	// Language is the code block's fenced-block language tag (e.g. "go",
	// "python"); empty renders a plain fence. code only.
	Language string

	// Path is the image's already-resolved, already-trust-validated
	// absolute file path. image only. Like ExtractRequest.Path, this
	// package never validates it — that trust boundary lives at the
	// agent-tool layer (see CreateDocumentTool.DenyReadPaths), the same
	// pattern read_document's Path field already documents.
	Path string

	// Alt is the image's alt text. image only.
	Alt string
}

Block is one node of a DocAST block tree. Which fields apply depends on Type; unused fields are left zero. Deliberately mirrors Pandoc's own block shape (heading, paragraph, list, table, code, image) rather than inventing a new schema, since Pandoc is the generation backend for docx/pdf — see docs/document-generation.md.

type CSVExtractor

type CSVExtractor struct {
	// Delimiter is the field separator assumed from the extension: ','
	// for CSV, '\t' for TSV. For .csv it is only a starting point —
	// see sniff.
	Delimiter rune
	// contains filtered or unexported fields
}

CSVExtractor parses comma- or tab-delimited files into a header + sampled-rows preview. It uses encoding/csv rather than line-splitting so a quoted field containing an embedded delimiter or newline is parsed as one logical row instead of being sheared into a bogus extra row.

func NewCSVExtractor

func NewCSVExtractor() *CSVExtractor

NewCSVExtractor returns a .csv extractor that defaults to comma and sniffs for the separator actually in use.

func NewTSVExtractor

func NewTSVExtractor() *CSVExtractor

NewTSVExtractor returns a tab-delimited .tsv extractor.

func (*CSVExtractor) Extract

func (*CSVExtractor) Match

func (e *CSVExtractor) Match(path string) bool

type Cell

type Cell struct {
	Value        any
	Formula      string
	Bold         bool
	Italic       bool
	NumberFormat string
}

Cell is one spreadsheet cell. Value holds a string, float64, int, or bool literal; Formula, when non-empty, is written as a formula instead and Value is ignored.

type CommandRunner

type CommandRunner func(ctx context.Context, command string) (stdout, stderr []byte, err error)

CommandRunner runs one command through whatever execution seam the caller has (host PATH or a podman sandbox) and returns its captured stdout/stderr. This is the seam that lets PDFExtractor stay a pure content library — see the package doc — while still shelling out to pdftotext/pdfinfo: internal/documents defines the seam, internal/agent (which owns the Sandbox interface) supplies the implementation, the same dependency direction generation already uses for pandoc. A nil CommandRunner means PDF extraction is unavailable.

type DocAST

type DocAST struct {
	Blocks []Block
}

DocAST is the block-tree intermediate representation for docx/pdf generation — see docs/document-generation.md's "Two canonical intermediate representations". Rendered to Pandoc-flavored Markdown by RenderMarkdown, then handed to pandoc for the actual docx/pdf conversion.

type DocumentMetadata

type DocumentMetadata struct {
	// Kind is the extractor's short format name: "csv", "tsv", "json",
	// "jsonl", "xml", "html", "pdf", "xlsx", "docx", or "pptx".
	Kind string

	// SizeBytes is the source file's size on disk (not the bounded
	// amount actually read).
	SizeBytes int64

	// Columns is the CSV/TSV header row, when the file has one. Empty
	// for other formats.
	Columns []string

	// RowCount is the number of logical rows/records the extractor
	// actually walked (which may be less than the file's true row
	// count when a cap stopped it early — see Warnings for that case).
	RowCount int

	// Shape is a free-form one-line structure summary: JSON top-level
	// keys/types, the XML root element name, or the HTML <title>.
	Shape string

	// Title, Author, and CreationDate are the document's own embedded
	// metadata, when the extractor found any — currently PDF only,
	// read from pdfinfo's output (already fetched for the page count).
	// CreationDate is poppler's own human-readable rendering of the
	// PDF's internal date, not reparsed here. Empty when absent or not
	// supported by the format.
	Title        string
	Author       string
	CreationDate string
}

DocumentMetadata is the structure summary shown above the content preview — what the model needs to decide whether to page further without having read the content yet.

type DocumentSection

type DocumentSection struct {
	Label string
	Text  string
}

DocumentSection is one labeled chunk of extracted text. Label carries provenance so the model can point back at an exact location instead of an offset into an opaque blob.

type DocxExtractor

type DocxExtractor struct {
	// Run is nil-safe: a nil Run skips straight to the native tier,
	// mirroring PDFExtractor's own optional-CommandRunner shape.
	Run CommandRunner
}

DocxExtractor extracts bounded, character-windowed text from docx files. Two tiers, tried in fidelity order:

  1. pandoc (when Run is set and succeeds): "pandoc -f docx -t gfm" preserves real tables (as pipe tables), bold/italic, and nested lists — the "full fidelity" tier of the docx parse fallback chain in docs/document-generation.md.
  2. Native zip+XML walk of word/document.xml (always available, no external binary): flattened paragraph/heading text only, no tables. Used when Run is nil or the pandoc tier fails for any reason — never a hard error, this tier is always a valid degrade target.

func (*DocxExtractor) Extract

func (*DocxExtractor) Match

func (e *DocxExtractor) Match(path string) bool

type ExtractRequest

type ExtractRequest struct {
	// Path is an already-resolved, already-trust-validated absolute
	// path. Extractors do not re-validate it.
	Path string

	MaxBytes int64
	MaxChars int
	MaxRows  int

	// Offset is where the preview window starts, counted in whatever
	// unit that format's preview is made of: data rows for CSV/TSV/xlsx,
	// records for JSONL, characters for the JSON/XML/HTML/docx text
	// preview, pages for PDF, slides for pptx. One name rather than
	// row_offset/char_offset because every result labels the window it
	// actually returned ("rows 201-400 of 5000"), so the unit is
	// unambiguous exactly where it's read.
	//
	// Skipping still requires streaming through what came before —
	// rows are variable-length and a quoted field may contain newlines,
	// so there is no seek. Skipped content is parsed and discarded
	// rather than retained, so paging costs time, not memory. It does
	// consume the MaxBytes budget, and a window that lies past the cap
	// says so rather than looking like the end of the file.
	Offset int

	// HasHeader overrides CSV/TSV header handling. Nil auto-detects
	// (see firstRowLooksLikeData); non-nil forces the answer, which is
	// the escape hatch for the cases the heuristic cannot get right —
	// a header of bare years, or a headerless file whose first row
	// happens to be all text. Ignored by non-tabular extractors.
	HasHeader *bool

	// MaxPages bounds how many PDF pages are read. PDF-only; every other
	// extractor ignores it.
	MaxPages int

	// OCRLang is the Tesseract language code (or "+"-joined codes, e.g.
	// "eng+fra") passed to the optional PDF OCR fallback tier. PDF-only;
	// every other extractor ignores it. Empty means "let pytesseract use
	// its own default" (English), not "OCR disabled" — the tier's
	// availability is controlled by whether the dependency is reachable
	// at all, not by this field.
	OCRLang string
}

ExtractRequest carries the local file to parse plus the caps that bound the work. Zero-value fields are filled with the Default* constants by withDefaults; callers only need to set what they want to override.

type ExtractResult

type ExtractResult struct {
	Metadata DocumentMetadata
	Sections []DocumentSection

	// Warnings surfaces truncation and degraded-parse notices — e.g.
	// "stopped at the 5 MiB read cap after 40 rows" or "malformed row
	// 12 skipped". Never silent: a capped or partial read always shows
	// up here.
	Warnings []string
}

ExtractResult is what read_document hands back to the model.

type Extractor

type Extractor interface {
	// Match reports whether this extractor handles path, based on its
	// extension. The Phase A format set is self-describing by suffix,
	// so content sniffing isn't needed.
	Match(path string) bool

	Extract(ctx context.Context, req ExtractRequest) (ExtractResult, error)
}

Extractor converts one local document into a bounded ExtractResult.

type HTMLExtractor

type HTMLExtractor struct{}

HTMLExtractor strips script/style content and returns visible text plus headings, using golang.org/x/net/html's tokenizer (already a go.mod dependency) rather than a naive tag strip — a regex-based strip can't tell a real close tag from one that merely looks like one inside a script string.

func (*HTMLExtractor) Extract

func (*HTMLExtractor) Match

func (e *HTMLExtractor) Match(path string) bool

type ImageBounds

type ImageBounds struct {
	X, Y, CX, CY int64
}

ImageBounds is an explicit picture bounding box in EMU (English Metric Units, the OOXML native length unit — 914400 EMU per inch).

type JSONExtractor

type JSONExtractor struct{}

JSONExtractor handles both .json (one document: object, array, or scalar) and .jsonl (one independent JSON value per line). JSON gets a top-level shape summary before the content preview; JSONL samples records and labels each with its source line number so the model can cite an exact line back to the user.

func (*JSONExtractor) Extract

func (*JSONExtractor) Match

func (e *JSONExtractor) Match(path string) bool

type PDFExtractor

type PDFExtractor struct {
	Run CommandRunner

	// ResolveScript enables the optional pdfplumber-backed table-
	// extraction tier (see extractTablesAndImages) and the optional
	// pytesseract-backed OCR tier (see extractOCR) — both are named by
	// PyHelperScript, so one resolver serves either. Nil-safe: a nil
	// resolver just skips both tiers, same as a nil Run skips
	// extraction entirely — the primary pdftotext-based text extraction
	// never depends on this field.
	ResolveScript ScriptResolver
}

PDFExtractor extracts text from PDF files via pdftotext/pdfinfo (poppler-utils). Unlike every other extractor in this package, it needs a subprocess — Run must be supplied by the caller (see CommandRunner).

func (*PDFExtractor) Extract

func (*PDFExtractor) Match

func (e *PDFExtractor) Match(path string) bool

type PptxExtractor

type PptxExtractor struct{}

PptxExtractor extracts bounded, slide-labeled text from pptx files: a zip archive with one ppt/slides/slideN.xml entry per slide. Native zip+XML walk (no external binary) — text extraction matches the "text only" tier of the pptx parse fallback chain in docs/document-generation.md, additionally augmented with real DrawingML table structure (see extractPptxSlideContent) so a table's cell text comes back as clean rows, not just run together with every other word on the slide, and with picture metadata (file name, size, alt text — never image bytes; see buildPptxImageSections) for every <p:pic> found on a slide.

func (*PptxExtractor) Extract

func (*PptxExtractor) Match

func (e *PptxExtractor) Match(path string) bool

type PyHelperScript

type PyHelperScript string

PyHelperScript names one of the embedded driver scripts. The string value is also its filename, both under pyhelpers/ in this embed and under DocumentsImageHelperDir in the documents image.

const (
	// ScriptExtractPDFTables wraps pdfplumber: prints
	// {"pages": [{"page": N, "tables": [{"rows": [...]}]}]} for a PDF.
	ScriptExtractPDFTables PyHelperScript = "extract_pdf_tables.py"

	// ScriptFillDocxTemplate wraps python-docx: replaces {{name}}
	// tokens in an existing docx and saves the result.
	ScriptFillDocxTemplate PyHelperScript = "fill_docx_template.py"

	// ScriptExtractPDFOCR wraps pytesseract+pdf2image: prints
	// {"pages": [{"page": N, "text": "..."}]} for a PDF with no
	// embedded text layer.
	ScriptExtractPDFOCR PyHelperScript = "extract_pdf_ocr.py"
)

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry dispatches to the first Extractor whose Match reports true, in registration order.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns a Registry pre-loaded with every extractor that needs no subprocess: the Phase A formats (CSV, TSV, JSON, JSONL, XML, HTML) plus the native zip/xlsx-based Phase C formats (xlsx via excelize, docx and pptx via zip+XML). PDF is deliberately NOT registered here — it needs a CommandRunner (pdftotext/pdfinfo), which only the agent-tool layer can supply (see PDFExtractor's doc comment); callers that want PDF support register it themselves after NewRegistry. docx is included with a nil Run (native tier only, always fully functional) — unlike PDF, docx doesn't need a CommandRunner to be useful, so a bare NewRegistry() call still gets working docx support. Callers that also want docx's optional pandoc tier use NewRegistryWithDocxTier instead.

func NewRegistryWithDocxTier

func NewRegistryWithDocxTier(run CommandRunner) *Registry

NewRegistryWithDocxTier is NewRegistry, but wires run into DocxExtractor.Run so its optional pandoc tier is available. A separate constructor rather than a second Register call after NewRegistry, because Registry.Lookup matches in registration order: a second DocxExtractor registered afterward would never be reached — the first (nil-Run, native-only) one registered by NewRegistry would always win.

func (*Registry) Lookup

func (r *Registry) Lookup(path string) Extractor

Lookup returns the first extractor whose Match reports true for path, or nil if no registered format recognizes it.

func (*Registry) Register

func (r *Registry) Register(e Extractor)

Register adds e to the dispatch list. Later registrations are tried after earlier ones, so more specific extractors should be registered first if extension ranges ever overlap.

type ScriptResolver

type ScriptResolver func(script PyHelperScript) (string, error)

ScriptResolver resolves a PyHelperScript to an invocable filesystem path, given whatever Sandbox-awareness the caller has — mirrors CommandRunner's own caller-supplies-the-seam shape (see pdf.go's CommandRunner doc comment for the dependency-direction reasoning: this package defines the seam, internal/agent supplies the implementation via documents.ResolvePyHelperScript). A nil ScriptResolver means the optional tier that needs it is unavailable, same as a nil CommandRunner today.

type Sheet

type Sheet struct {
	Name string
	Rows [][]Cell
}

Sheet is one spreadsheet tab: a name plus its rows, top to bottom, left to right.

type SheetModel

type SheetModel struct {
	Sheets []Sheet
}

SheetModel is the intermediate representation for xlsx generation — see docs/document-generation.md's "Two canonical intermediate representations". It is a thin wrapper over excelize's own object model rather than a new invention, since excelize already owns xlsx generation end to end (including formula recalculation), unlike the docx/pdf path.

type SlideModel

type SlideModel struct {
	Title    string
	Bullets  []string
	Notes    string
	Image    string
	ImageAlt string
	Layout   string

	// ImageBounds is the image's placement on the slide, in EMU. Nil
	// reproduces the original fixed right-half placement exactly, so a
	// slide built before this field existed generates byte-identical
	// output. Ignored when Image is empty. Presets and validation
	// against slide dimensions live in the create_document tool layer
	// (internal/agent), which owns the user-facing schema and error
	// messages; this package only draws the rectangle it's given.
	ImageBounds *ImageBounds
}

SlideModel is one slide in a generated PPTX deck. It intentionally supports a small, predictable subset of PowerPoint: title, bullets, speaker notes, and one optional image. That is enough for the agent's current structured create_document schema while keeping generation pure Go.

type Span

type Span struct {
	Text         string
	Bold, Italic bool
}

Span is one inline-formatted run of text within a heading, paragraph, or list item. See Block.Spans.

type XLSXExtractor

type XLSXExtractor struct{}

XLSXExtractor extracts bounded, sheet-labeled text from xlsx workbooks via excelize — the same library GenerateXLSX uses for writing, so reading and writing xlsx share one dependency.

func (*XLSXExtractor) Extract

func (*XLSXExtractor) Match

func (e *XLSXExtractor) Match(path string) bool

type XMLExtractor

type XMLExtractor struct{}

XMLExtractor streams an XML document token by token (encoding/xml's Decoder), so a decode error partway through still returns everything parsed before it instead of failing the whole call — malformed real-world XML is common enough that "here's what we got" beats an all-or-nothing error.

func (*XMLExtractor) Extract

func (*XMLExtractor) Match

func (e *XMLExtractor) Match(path string) bool

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL