Documentation
¶
Overview ¶
Package documentreading decides how to turn a file's bytes into markdown text, cheapest path first. It is the shared default reader for any host (the local backend, a server) that embeds the engine.
Convert tries the SDK's native, dependency-free extractors first (DOCX, PPTX, XLSX, HTML, and PDFs page by page through pdfsmart) and only reaches for an external converter, which may be nil, when a format genuinely needs it: scans, audio, images, or a PDF page with no usable text layer. Output from an external converter is checked for garbled text too. A host that has no converter configured still reads every native format.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var AllConvertibleExtensions = buildAllConvertibleExtensions()
AllConvertibleExtensions is every extension Convert can turn into markdown given a reachable external reader - native formats, PDF, and the external-only formats. Callers that need "is this worth attempting at all" (e.g. the upload-time conversion gate, the prompt annotation) should use this rather than re-deriving their own list.
var ExternalOnlyExtensions = map[string]bool{ ".wav": true, ".mp3": true, ".png": true, ".tiff": true, ".tif": true, ".jpeg": true, ".jpg": true, }
ExternalOnlyExtensions lists formats that always need an external reader: audio transcription and image OCR have no native equivalent here. PDF is intentionally absent - it's handled by Convert's own native-first/external fallback logic below, not a blanket "always try external" entry, since most PDFs have a real text layer and need no OCR at all.
HTMLExtensions are the HTML formats Convert turns into markdown: a saved page, an exported report, a documentation page, as a document. They are not in officetext.SupportedExtensions on purpose: the Read tool shows an HTML file as source, because the tags are what someone editing a web page needs to see.
var NativeExtensions = nativeExtensions()
NativeExtensions lists formats converted locally, no external service needed: the Office formats and HTML.
Functions ¶
func StripPageMarkers ¶
StripPageMarkers takes the page markers out of a text that has them.
Types ¶
type PageReadResult ¶
type PageReadResult struct {
Page int `json:"page"`
Source string `json:"source,omitempty"`
HasImage bool `json:"has_image,omitempty"`
}
PageReadResult records how one PDF page was read.
type Result ¶
type Result struct {
Markdown string
// PagedMarkdown is the same text with a marker line before each page (see internal/pagemark), for a PDF read
// page by page; empty for every other format. It is what goes to the chunkers, so that a chunk can say which
// pages it comes from. Markdown stays clean for anyone who shows the text.
PagedMarkdown string
Images []documentreader.ExtractedImage
PageCount int
Pages []PageReadResult
Source Source
}
Result is the outcome of a successful conversion.
func Convert ¶
func Convert(ctx context.Context, filePath string, externalConverter documentreader.Converter) (Result, bool, error)
Convert turns filePath's content into markdown. It tries native extraction first for DOCX/PPTX/XLSX/HTML (always) and PDF (page by page, see pdfsmart), then falls back to externalConverter (which may be nil) for everything else: scans, audio, images, a PDF page that needs OCR, and any native-extraction failure. A document whose native text is thin (a deck that is mostly pictures) is returned as it is when the external reader cannot improve on it. ok is false when no path could produce text - callers should not treat that as an error, just "nothing extracted".
func ConvertBytes ¶
func ConvertBytes(ctx context.Context, data []byte, filename string, externalConverter documentreader.Converter) (Result, bool, error)
ConvertBytes is Convert's counterpart for callers that only have the raw bytes and a filename, not a path on disk (e.g. RAG ingestion, which reads blobs out of storage rather than the workspace filesystem).
func (Result) TextForIndexing ¶
TextForIndexing is the text a chunker should read: the page-marked markdown when the document has pages, the markdown otherwise.
Source Files