Documentation
¶
Overview ¶
Package text turns a stored PDF into searchable text. M3.
Three stages: extract text with layout hints, split into chunks that keep stable offsets back into the source page, and capture citation context — the sentence around each citation marker in the citing paper.
The citation context is why this package has the shape it does. Nobody hands it to you, and it is what makes retrieval better than plain search, so it is captured while the document is already parsed rather than by re-parsing later.
PDF extraction is the weakest part of the Go choice (ADR-009). Keep the extractor behind an interface so it can be replaced without disturbing chunking or context capture.
Index ¶
- Constants
- Variables
- func Clean(page string) string
- func Mentions(pages []string, title, surname string) bool
- func Resembles(pages []string, title, surname string) bool
- type Chunk
- type Clue
- type Extractor
- type Front
- type FrontReader
- type Mention
- type PDFium
- type Ref
- type Result
- type Status
- type Verdict
Constants ¶
const ( MethodNumeric = "numeric" // [12], [3–5]: through the numbered reference list MethodAuthorYear = "author-year" // (Piwowar et al., 2018): straight to the reference )
How a mention was found.
const ExtractorVersion = "pdfium/4" // 2: citation context is found as a PDF is read; 3: spaced accents joined; 4: a one-name marker never cites a paper of three authors, and "p. 23" does not end a sentence
ExtractorVersion names the extractor and the cleaning applied to its output. Text read by an older version is read again; raise it whenever a change here or in Clean would change what is stored.
const SectionReferences = "references"
SectionReferences marks the passages after a paper's last references heading. Search leaves them out unless asked: otherwise a search for an author's name fills up with bibliographies.
Variables ¶
var ErrUnreadable = errors.New("unreadable PDF")
ErrUnreadable is a PDF the extractor refused: damaged, encrypted, or not a PDF at all. One such file fails one paper, never the run.
Functions ¶
func Clean ¶
Clean turns one page of PDFium's output into the text fil stores: one newline per line break, single spaces, words broken across lines joined again, Unicode NFC. Line breaks are kept, because chunking and the references heading are found by line.
What PDFium hands back, measured on real papers:
- lines end in "\r\n";
- a word hyphenated at the end of a line arrives already joined, with U+0002 where the hyphen was ("de\x02scribe"). Mostly syllable breaks, so the mark is dropped; a compound broken at its hyphen ("self-archiving") loses the hyphen too, as it does in pdftotext;
- U+00AD, the soft hyphen, stands in for a printed hyphen inside a line ("Ninetyseven");
- a hyphen that does end a line is a range ("1992-\n2003"), so the line break goes and the hyphen stays;
- other control characters are glyphs with no Unicode meaning.
func Mentions ¶
Mentions reports whether a work's title, and its first author's surname when known, appear in the text of a PDF's first pages: the check that lets a clue be trusted (ADR-014). Case, spacing, punctuation and line breaks are ignored. A title whose full form is not found passes on its main title — the part before a colon — if that is long enough to be telling, since subtitles are set apart or dropped.
func Resembles ¶
Resembles is the looser check, for a work an identifier in the PDF itself named: a preprint's title is often not the published one OpenAlex keeps — "2D" for "two-dimensional", "the" dropped. It passes when the first author's surname is in the PDF as a word, and most of the title's words — three letters or more, at least 60% — are in the top lines of page 1, where a paper's own title sits; a cited paper's title sits in the references, further down. A title search's candidates get Mentions only.
Types ¶
type Chunk ¶
type Chunk struct {
Page int // from 1
Ordinal int // position in the paper, from 0
Section string // SectionReferences, or "" for the body
// Start and End are byte offsets of Text in its page's cleaned text, so
// a passage can be found again in its page and its neighbours stitched
// back on.
Start, End int
Text string
}
Chunk is one passage of a paper: what keyword search finds, and later what is embedded (M4). A chunk never crosses a page, so every hit has one page number.
type Clue ¶
type Clue struct {
ID identity.ID
Where string // in words: "the arXiv stamp", "a DOI label on page 1", …
}
Clue is an identifier a PDF carries, and where it was found. A clue is never trusted: import resolves it and checks the work's title is in the PDF before filing anything under it (ADR-014).
func Clues ¶
Clues lists the identifiers a PDF carries, the most telling first: the arXiv stamp; an arXiv download's file name; a DOI labelled as one in page 1's header or footer; the PDF's metadata; a labelled DOI anywhere on page 1; a labelled PMID; then any DOI on page 1, and on page 2. Page 1 carries other papers' DOIs — "cite this article", related articles, a short paper's references — which is why where a DOI sits matters.
type Extractor ¶
Extractor returns the text of each page of a PDF, in page order. It is an interface so PDFium can be replaced without touching cleaning or chunking (ADR-009).
type Front ¶
type Front struct {
// Pages is the cleaned text of the first pages, at most frontPages.
Pages []string
// Meta is the document information dictionary, where present: Title,
// Subject, Keywords, and the doi key some publishers add. Templates
// reuse old values, so it is a clue, never an answer.
Meta map[string]string
// Title is the largest text near the top of page 1 — the title, usually,
// but a journal's name or a running head sometimes. A candidate only.
Title string
}
Front is what the opening of a PDF says about which paper it is: the clues import works from (ADR-014). Nothing in it is trusted until an identifier found here resolves to a work whose title is in Pages.
type FrontReader ¶
FrontReader reads a PDF's front matter. PDFium is one.
type Mention ¶
type Mention struct {
Ref string // Ref.ID
Page int // from 1
Start, End int // byte offsets of Sentence in its page's cleaned text
Sentence string
Section string // a heading's name above it, or ""
Marker string // as printed
Method string // MethodNumeric or MethodAuthorYear
Confidence float64
}
Mention is one place the paper cites one of its references: the sentence, where it is, and how it was found (ADR-015).
func Contexts ¶
Contexts finds where a paper cites its references (ADR-015). pages are its cleaned pages; refs are the works OpenAlex says it cites. Nothing here adds a reference: a marker that cannot be tied to one of refs gives no mention. No match, no context — never a guess.
Two routes. Author–year markers name the work — surname and year — and are matched to refs directly, with no need for the reference list. Numeric markers name an entry of the numbered reference list, so the list is found and split, and each entry matched to a ref by its DOI or by most of its title's words, confirmed by year or surname (spike 11).
type PDFium ¶
type PDFium struct {
// contains filtered or unexported fields
}
PDFium is PDFium compiled to WebAssembly and run by wazero: no cgo (D20). It starts on first use — compiling the module takes a moment — so a command that reads no PDF pays nothing. Safe for concurrent use; at most two extractions run at once.
func Shared ¶
func Shared() *PDFium
Shared returns the process's one PDFium. Its module is compiled once and kept until the process ends, however many libraries are opened.
func (*PDFium) Extract ¶
Extract returns the text of each page. When ctx ends — a time limit, or the user cancelling — the extraction is stopped, not just abandoned, and ctx's error is returned. A PDF PDFium cannot open is ErrUnreadable.
type Ref ¶
type Ref struct {
ID string // the cited work's fil ID
Title string
Year int
DOI string
Surname string // first author's
Authors int // how many authors; 0 when not known
}
Ref is a work the paper cites, as OpenAlex gave it: what citation context looks for in the paper's text.
type Result ¶
type Result struct {
Verdict
Pages int
Chars int // visible characters, all pages
Chunks []Chunk // only when Status is ok: gibberish would pollute search
// Text is the cleaned pages, when Status is ok: what citation context is
// found in (Contexts), while the PDF is read.
Text []string
}
Result is what reading one PDF gave.
func Read ¶
Read extracts, cleans, judges and chunks one PDF.
A PDF that cannot be read is a Result, not an error: an unreadable file is StatusUnreadable, and one still being read when ctx's deadline passed is StatusTimedOut. The error is for everything else — the user cancelling, or PDFium failing to start — and means the run should stop.
type Status ¶
type Status string
Status is what reading a PDF gave: searchable text, or why there is none.
const ( StatusOK Status = "ok" StatusNoText Status = "no-text" // no text layer: a scan, and fil does no OCR StatusGibberish Status = "gibberish" // a text layer that is not text StatusUnreadable Status = "unreadable" // damaged, encrypted, or not a PDF StatusTimedOut Status = "timed-out" // still being read at the time limit )
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package pdftest writes small, valid PDFs for tests: real files PDFium opens, made in code, so no binary sits in testdata and every test says exactly what its PDF contains.
|
Package pdftest writes small, valid PDFs for tests: real files PDFium opens, made in code, so no binary sits in testdata and every test says exactly what its PDF contains. |