pdfsmart

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package pdfsmart extracts a PDF's text page by page, sending only the pages that actually need it through docling instead of the whole document - most PDFs are mostly prose, and paying docling's per-page layout-detection pass (see its standard PDF pipeline) for every page of a large document is wasted work when only a handful of pages have an embedded image or an unreadable native text layer.

A page is routed to docling when:

  • it has at least one embedded raster image (a PDF XObject with /Subtype /Image, detected via pdfcpu's own resource inspection - a deterministic read of the file's structure, not a heuristic or an ML classification), or
  • its native text layer is too sparse (see pdftext.MinCharsPerPage), or looks like broken font-encoding extraction (see textquality.IsGarbledText).

This deliberately does NOT attempt to detect borderless tables or vector-drawn diagrams/charts - those aren't reliably detectable without a trained layout model (this is exactly why docling itself runs layout detection on every page unconditionally, rather than trying to guess which pages need it). A page containing one is only caught here if its native text also happens to be sparse or garbled; otherwise it's extracted natively, and the caller gets prose-quality text for a page that visually contained a table. Callers that can't accept ever missing a table this way (e.g. financial documents, invoices) should keep sending those documents through docling wholesale instead of this package - see Convert's own doc comment for the exact safety contract.

Index

Constants

View Source
const DefaultModelBudget = 20 * time.Second

DefaultModelBudget is the time a call may spend on the layout models when Options.ModelBudget is zero.

Variables

View Source
var ErrPasswordRequired = errors.New("the PDF is protected by a password")

ErrPasswordRequired is returned for a PDF that is encrypted with a password the caller did not give, or gave wrongly. An encrypted PDF whose user password is empty (an emailed statement with an owner password that only limits printing and copying) is not that: it is opened.

Functions

func ImageMarker

func ImageMarker(page int, picture Picture) string

ImageMarker writes the marker of a picture in the text of a page.

func PageCount

func PageCount(ctx context.Context, data []byte) (int, error)

PageCount is the number of pages of a PDF, without reading any of them.

func Unlock

func Unlock(ctx context.Context, data []byte, password string) ([]byte, error)

Unlock returns the bytes of a PDF in a form the page reader can open. A PDF that is not encrypted, or that the reader already opens (RC4 and AES-128 with an empty user password), comes back unchanged. One it cannot open (AES-256, and anything encrypted with a user password) is decrypted into a plain copy.

With no password the empty user password is tried, which is what every viewer does first. A password that is needed and not given, or wrong, gives ErrPasswordRequired.

Types

type Options

type Options struct {
	// ImagePagesNeedEngine sends a page with an embedded image to the engine even when its text layer
	// is fine, which is what Convert does. A reader that lets the caller look at the page itself
	// leaves it false: the text is read natively and the image is only reported.
	ImagePagesNeedEngine bool

	// MaxEnginePages bounds how many pages one call sends to the document reader (zero means no
	// bound). Pages past it are returned with Deferred set. Each engine call is slow, so a caller
	// that wants an answer in bounded time sets it and reads the rest in a later call.
	MaxEnginePages int

	// Password opens a PDF that is protected by a user password. A PDF whose user password is empty needs none.
	Password string

	// ModelBudget bounds the time one call spends asking the layout models about tables. A model looks at one page
	// in seconds on a CPU, and many pages of a long document look as if they might hold a table, so without a bound
	// a book takes minutes for the sake of the few pages that have one. Zero means DefaultModelBudget; a negative
	// value means no bound. Once it is spent the remaining pages are read without the models.
	ModelBudget time.Duration
}

Options tunes how ReadPages routes a page.

type PageRenderer

type PageRenderer interface {
	// RenderPage renders 1-indexed page pageNum of the PDF in data,
	// returning PNG-encoded image bytes.
	RenderPage(ctx context.Context, data []byte, pageNum int) ([]byte, error)
}

PageRenderer renders a single PDF page to a raster image, feeding the vision-LLM fallback (see Convert's own doc comment for where this slots into the page-processing pipeline). Implementations reuse Phase 1's internal/nativedoc/pdfium CGO rendering (internal/nativedoc/parser's Converter implements this directly), wired in as an interface exactly like documentreader.Converter is, so this package itself never depends on CGO or the nativedoc build tag.

type PageResult

type PageResult struct {
	Page             int
	Text             string
	Source           PageSource
	HasEmbeddedImage bool
	// Pictures are the pictures of the page the text does not say anything about, numbered from 1 on the page. A page read
	// natively has one marker in Text for each of them (see ImageMarker); a page read by an engine has none, since the
	// engine read the pictures.
	Pictures []Picture
	// Deferred is set on a page that needed the engine but was left unread because Options.MaxEnginePages
	// had been used up. It is not a failure: reading the page again later will read it.
	Deferred bool
}

PageResult is one page's contribution to a Result.

func (PageResult) HasText

func (p PageResult) HasText() bool

HasText says whether the page gave any text besides the markers of its pictures: a page that is only its markers has not been read, and a host that tells the pages it could not read from the others must not count it as read.

type PageSource

type PageSource string

PageSource identifies how a page's text was obtained.

const (
	PageSourceNative  PageSource = "native"
	PageSourceDocling PageSource = "docling"
	// PageSourceVision marks a page that needed the vision-LLM fallback -
	// native text extraction was too sparse/absent AND docling was either
	// unavailable or also couldn't produce usable text for it. See
	// VisionFallback's doc comment for how this stage is configured.
	PageSourceVision PageSource = "vision"
)

type Picture

type Picture struct {
	// Number counts the pictures of the page from 1, in the order the file lists them (which is not always the order on the
	// page). With the page number it names the picture for good, whatever pages were read.
	Number int
	// Title is the caption found in the text of the page ("Figure 2. ..."), or empty when there is none to attach.
	Title string
}

Picture is a picture of a page that its text does not say anything about.

type Result

type Result struct {
	Markdown string
	Pages    []PageResult
	// ModelPagesSkipped counts the pages whose tables the layout models were not asked about because the time
	// budget (Options.ModelBudget) was spent. Their text is read as usual, and so are the tables the rulings show.
	ModelPagesSkipped int
}

Result is the outcome of a page-aware PDF-to-markdown conversion.

func Convert

func Convert(ctx context.Context, data []byte, documentReader documentreader.Converter, vision VisionFallback) (Result, bool, error)

Convert extracts data's text page by page (see package doc for the native-vs-docling routing rule per page). A page that still has no usable text after docling gets one more attempt via vision, when configured (see VisionFallback) - render the page and transcribe it with a vision-capable LLM, gated on that model actually reporting vision support (see VisionTranscriber.IsAvailable) so this is never attempted against a text-only model.

ok is true only when EVERY page produced usable text, whether native, via docling, or via the vision fallback - if even one page that needed one of those couldn't get it, ok is false and Result should be discarded. This is a deliberate safety contract, not an oversight: a partial result missing one page's content is exactly the silent, permanent data loss this package exists to avoid on the pages it CAN cheaply skip the expensive paths for - it must never produce that same failure mode on a page it couldn't. Callers should fall back to sending the whole document through docling when ok is false, the same as if this package didn't exist.

func ReadPages

func ReadPages(ctx context.Context, data []byte, pages []int, opts Options, documentReader documentreader.Converter, vision VisionFallback) (Result, bool, error)

ReadPages is Convert for a chosen set of 1-indexed pages (nil means every page), so a caller that wants three pages of a long document does not pay for the rest. Result.Pages holds the requested pages in order; ok is true only when every one of them produced usable text, with the same contract as Convert.

type TableFinder

type TableFinder interface {
	FindTables(ctx context.Context, pdfData []byte, pageIndex int) ([]pdftext.TableStructure, error)
}

TableFinder is a document reader that can also say where the tables with no ruling lines are on a page, and how they are laid out, from a layout model. A reader that has this (the native one, with its models) is asked for a page whose text looks columnar; the page's own text fills the cells the model finds. pageIndex counts from zero. An error, or no tables, leaves the page as it would have been without the model.

type VisionFallback

type VisionFallback struct {
	Renderer    PageRenderer
	Transcriber VisionTranscriber
}

VisionFallback bundles the two collaborators the vision-LLM fallback stage needs. Both Renderer and Transcriber must be set (and Transcriber must report IsAvailable) for the fallback to actually run on a page - the zero value (or either field left nil) simply skips this stage, the same as a nil/unavailable docling client skips the docling stage. Passed as its own value rather than two more positional Convert parameters so a future fallback stage doesn't need another breaking signature change.

type VisionTranscriber

type VisionTranscriber interface {
	IsAvailable(ctx context.Context) bool
	// TranscribePage returns the page's transcribed text/markdown given its
	// PNG-encoded image bytes.
	TranscribePage(ctx context.Context, pngImage []byte) (string, error)
}

VisionTranscriber sends a rendered page image to a vision-capable LLM and returns its transcription. Implementations must report IsAvailable=false (rather than attempting the call and failing) when the configured model isn't multimodal - see internal/model.Registry.VisionCapable, the capability check used elsewhere in this codebase for the same purpose.

Directories

Path Synopsis
Package vision implements pdfsmart's optional vision-LLM fallback stage: transcribing a rendered PDF page image via a vision-capable LLM when neither native text extraction nor docling could produce usable text for it.
Package vision implements pdfsmart's optional vision-LLM fallback stage: transcribing a rendered PDF page image via a vision-capable LLM when neither native text extraction nor docling could produce usable text for it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL