text

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package text turns a stored PDF into searchable text. M3.

Three stages: extract text with layout hints, split into chunks that keep stable offsets back into the source page, and capture citation context — the sentence around each citation marker in the citing paper.

The citation context is why this package has the shape it does. Nobody hands it to you, and it is what makes retrieval better than plain search, so it is captured while the document is already parsed rather than by re-parsing later.

PDF extraction is the weakest part of the Go choice (ADR-009). Keep the extractor behind an interface so it can be replaced without disturbing chunking or context capture.

Index

Constants

View Source
const (
	MethodNumeric    = "numeric"     // [12], [3–5]: through the numbered reference list
	MethodAuthorYear = "author-year" // (Piwowar et al., 2018): straight to the reference
)

How a mention was found.

View Source
const ExtractorVersion = "pdfium/4" // 2: citation context is found as a PDF is read; 3: spaced accents joined; 4: a one-name marker never cites a paper of three authors, and "p. 23" does not end a sentence

ExtractorVersion names the extractor and the cleaning applied to its output. Text read by an older version is read again; raise it whenever a change here or in Clean would change what is stored.

View Source
const SectionReferences = "references"

SectionReferences marks the passages after a paper's last references heading. Search leaves them out unless asked: otherwise a search for an author's name fills up with bibliographies.

Variables

View Source
var ErrUnreadable = errors.New("unreadable PDF")

ErrUnreadable is a PDF the extractor refused: damaged, encrypted, or not a PDF at all. One such file fails one paper, never the run.

Functions

func Clean

func Clean(page string) string

Clean turns one page of PDFium's output into the text fil stores: one newline per line break, single spaces, words broken across lines joined again, Unicode NFC. Line breaks are kept, because chunking and the references heading are found by line.

What PDFium hands back, measured on real papers:

  • lines end in "\r\n";
  • a word hyphenated at the end of a line arrives already joined, with U+0002 where the hyphen was ("de\x02scribe"). Mostly syllable breaks, so the mark is dropped; a compound broken at its hyphen ("self-archiving") loses the hyphen too, as it does in pdftotext;
  • U+00AD, the soft hyphen, stands in for a printed hyphen inside a line ("Ninety­seven");
  • a hyphen that does end a line is a range ("1992-\n2003"), so the line break goes and the hyphen stays;
  • other control characters are glyphs with no Unicode meaning.

func Mentions

func Mentions(pages []string, title, surname string) bool

Mentions reports whether a work's title, and its first author's surname when known, appear in the text of a PDF's first pages: the check that lets a clue be trusted (ADR-014). Case, spacing, punctuation and line breaks are ignored. A title whose full form is not found passes on its main title — the part before a colon — if that is long enough to be telling, since subtitles are set apart or dropped.

func Resembles

func Resembles(pages []string, title, surname string) bool

Resembles is the looser check, for a work an identifier in the PDF itself named: a preprint's title is often not the published one OpenAlex keeps — "2D" for "two-dimensional", "the" dropped. It passes when the first author's surname is in the PDF as a word, and most of the title's words — three letters or more, at least 60% — are in the top lines of page 1, where a paper's own title sits; a cited paper's title sits in the references, further down. A title search's candidates get Mentions only.

Types

type Chunk

type Chunk struct {
	Page    int    // from 1
	Ordinal int    // position in the paper, from 0
	Section string // SectionReferences, or "" for the body
	// Start and End are byte offsets of Text in its page's cleaned text, so
	// a passage can be found again in its page and its neighbours stitched
	// back on.
	Start, End int
	Text       string
}

Chunk is one passage of a paper: what keyword search finds, and later what is embedded (M4). A chunk never crosses a page, so every hit has one page number.

func Chunks

func Chunks(pages []string) []Chunk

Chunks splits cleaned pages into passages.

type Clue

type Clue struct {
	ID    identity.ID
	Where string // in words: "the arXiv stamp", "a DOI label on page 1", …
}

Clue is an identifier a PDF carries, and where it was found. A clue is never trusted: import resolves it and checks the work's title is in the PDF before filing anything under it (ADR-014).

func Clues

func Clues(fileName string, f Front) []Clue

Clues lists the identifiers a PDF carries, the most telling first: the arXiv stamp; an arXiv download's file name; a DOI labelled as one in page 1's header or footer; the PDF's metadata; a labelled DOI anywhere on page 1; a labelled PMID; then any DOI on page 1, and on page 2. Page 1 carries other papers' DOIs — "cite this article", related articles, a short paper's references — which is why where a DOI sits matters.

type Extractor

type Extractor interface {
	Extract(ctx context.Context, pdf []byte) ([]string, error)
}

Extractor returns the text of each page of a PDF, in page order. It is an interface so PDFium can be replaced without touching cleaning or chunking (ADR-009).

type Front

type Front struct {
	// Pages is the cleaned text of the first pages, at most frontPages.
	Pages []string
	// Meta is the document information dictionary, where present: Title,
	// Subject, Keywords, and the doi key some publishers add. Templates
	// reuse old values, so it is a clue, never an answer.
	Meta map[string]string
	// Title is the largest text near the top of page 1 — the title, usually,
	// but a journal's name or a running head sometimes. A candidate only.
	Title string
}

Front is what the opening of a PDF says about which paper it is: the clues import works from (ADR-014). Nothing in it is trusted until an identifier found here resolves to a work whose title is in Pages.

type FrontReader

type FrontReader interface {
	Front(ctx context.Context, pdf []byte) (Front, error)
}

FrontReader reads a PDF's front matter. PDFium is one.

type Mention

type Mention struct {
	Ref        string // Ref.ID
	Page       int    // from 1
	Start, End int    // byte offsets of Sentence in its page's cleaned text
	Sentence   string
	Section    string // a heading's name above it, or ""
	Marker     string // as printed
	Method     string // MethodNumeric or MethodAuthorYear
	Confidence float64
}

Mention is one place the paper cites one of its references: the sentence, where it is, and how it was found (ADR-015).

func Contexts

func Contexts(pages []string, refs []Ref) []Mention

Contexts finds where a paper cites its references (ADR-015). pages are its cleaned pages; refs are the works OpenAlex says it cites. Nothing here adds a reference: a marker that cannot be tied to one of refs gives no mention. No match, no context — never a guess.

Two routes. Author–year markers name the work — surname and year — and are matched to refs directly, with no need for the reference list. Numeric markers name an entry of the numbered reference list, so the list is found and split, and each entry matched to a ref by its DOI or by most of its title's words, confirmed by year or surname (spike 11).

type PDFium

type PDFium struct {
	// contains filtered or unexported fields
}

PDFium is PDFium compiled to WebAssembly and run by wazero: no cgo (D20). It starts on first use — compiling the module takes a moment — so a command that reads no PDF pays nothing. Safe for concurrent use; at most two extractions run at once.

func Shared

func Shared() *PDFium

Shared returns the process's one PDFium. Its module is compiled once and kept until the process ends, however many libraries are opened.

func (*PDFium) Extract

func (p *PDFium) Extract(ctx context.Context, pdf []byte) ([]string, error)

Extract returns the text of each page. When ctx ends — a time limit, or the user cancelling — the extraction is stopped, not just abandoned, and ctx's error is returned. A PDF PDFium cannot open is ErrUnreadable.

func (*PDFium) Front

func (p *PDFium) Front(ctx context.Context, pdf []byte) (Front, error)

Front reads the opening of a PDF. Like Extract, it stops at ctx's end and returns ErrUnreadable for a file PDFium cannot open.

func (*PDFium) UseCache

func (p *PDFium) UseCache(dir string)

UseCache keeps the compiled module in dir, so the next process starts PDFium in a tenth of a second rather than three (measured: 3.1 s, then 80 ms; about 21 MB on disk). It takes effect only before the first extraction.

type Ref

type Ref struct {
	ID      string // the cited work's fil ID
	Title   string
	Year    int
	DOI     string
	Surname string // first author's
	Authors int    // how many authors; 0 when not known
}

Ref is a work the paper cites, as OpenAlex gave it: what citation context looks for in the paper's text.

type Result

type Result struct {
	Verdict
	Pages  int
	Chars  int     // visible characters, all pages
	Chunks []Chunk // only when Status is ok: gibberish would pollute search
	// Text is the cleaned pages, when Status is ok: what citation context is
	// found in (Contexts), while the PDF is read.
	Text []string
}

Result is what reading one PDF gave.

func Read

func Read(ctx context.Context, ex Extractor, pdf []byte) (Result, error)

Read extracts, cleans, judges and chunks one PDF.

A PDF that cannot be read is a Result, not an error: an unreadable file is StatusUnreadable, and one still being read when ctx's deadline passed is StatusTimedOut. The error is for everything else — the user cancelling, or PDFium failing to start — and means the run should stop.

type Status

type Status string

Status is what reading a PDF gave: searchable text, or why there is none.

const (
	StatusOK         Status = "ok"
	StatusNoText     Status = "no-text"    // no text layer: a scan, and fil does no OCR
	StatusGibberish  Status = "gibberish"  // a text layer that is not text
	StatusUnreadable Status = "unreadable" // damaged, encrypted, or not a PDF
	StatusTimedOut   Status = "timed-out"  // still being read at the time limit
)

type Verdict

type Verdict struct {
	Status Status
	Reason string // why, in words, when Status is not ok
}

Verdict is Judge's answer.

func Judge

func Judge(pages []string) Verdict

Judge decides whether cleaned pages are text worth searching.

Directories

Path Synopsis
Package pdftest writes small, valid PDFs for tests: real files PDFium opens, made in code, so no binary sits in testdata and every test says exactly what its PDF contains.
Package pdftest writes small, valid PDFs for tests: real files PDFium opens, made in code, so no binary sits in testdata and every test says exactly what its PDF contains.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL