Documentation
¶
Overview ¶
Package corpus builds and records the populations of real PDFs that the conformance tools are run against.
A corpus is a directory of files and a MANIFEST.tsv beside them. The manifest is what makes the corpus a measurement rather than a pile: it says where each file came from, when, how big it is and what its bytes hash to, so a figure quoted from it can be reproduced and a file that changed underneath can be noticed.
Index ¶
- Constants
- Variables
- func Digest(r io.Reader, w io.Writer) (string, int64, error)
- func Origins(entries []Entry) map[string]int
- func Write(dir string, entries []Entry) error
- type Archive
- func (a *Archive) Fetch(ctx context.Context, identifier, name string, w io.Writer) (string, int64, error)
- func (a *Archive) PDF(ctx context.Context, identifier string) (File, error)
- func (a *Archive) Search(ctx context.Context, query string, rows, page int) ([]string, error)
- func (a *Archive) URL(identifier, name string) string
- type Entry
- type File
- type Plan
- type Problem
Constants ¶
const ManifestName = "MANIFEST.tsv"
ManifestName is the file a corpus records itself in, beside its documents.
Variables ¶
var Now = func() time.Time { return time.Now() }
Now is the clock the harvester stamps entries with, so a test can hold it still.
Functions ¶
Types ¶
type Archive ¶
type Archive struct {
// Base is the site, so a test can point this at its own server.
Base string
// Client is the one used for every request; nil means http.DefaultClient.
Client *http.Client
}
An Archive fetches documents from the Internet Archive, which is where mass digitisation lives — and therefore where the codecs a form corpus never exercises are found. A scanned page is a fax: JBIG2 and CCITT are what it is stored in, and almost nothing else uses them.
func (*Archive) Fetch ¶
func (a *Archive) Fetch(ctx context.Context, identifier, name string, w io.Writer) (string, int64, error)
Fetch writes one of an item's files to w and returns its digest and size.
func (*Archive) PDF ¶
PDF finds an item's PDF and how big it is. An item may hold several — a scanned book often has both a full-colour one and a bitonal one — and the smallest is taken, because the bitonal one is the scanned page as a fax, which is the thing worth having here.
type Entry ¶
type Entry struct {
// Path is relative to the corpus directory, so a corpus can be moved.
Path string
// Origin is the population the document was drawn from — an issuing body,
// an archive collection — and is what a prevalence is computed over. Two
// documents from different origins are not interchangeable evidence.
Origin string
// Source is the URL it was fetched from.
Source string
Bytes int64
// SHA256 is of the bytes as fetched, so a figure can be tied to the exact
// document that produced it. Empty when the manifest did not say.
SHA256 string
// Fetched is when it was taken. The zero time means the manifest did not
// say — some documents predate the record that describes them.
Fetched time.Time
}
An Entry is one document and where it came from.
func Harvest ¶
Harvest adds documents to a corpus until the plan's origin holds Want of them, and records every one in the manifest.
It is resumable by construction: what is already in the manifest is skipped, so an interrupted run is continued by running it again, and a corpus is extended by asking for a larger Want.
func Read ¶
Read loads a corpus's manifest.
The header says what the columns are, so a manifest written by an earlier tool is still readable: the forms corpus gathered before this repository existed says "issuer|file|url|bytes|sha256-8|fetched", which is the same six facts under other names. Making the format self-describing was cheaper than re-fetching two thousand documents, and a corpus that has to be re-fetched to be read is not much of a record.
A directory with no manifest is an empty corpus rather than an error: that is what a corpus about to be built looks like.
type Plan ¶
type Plan struct {
// Dir is the corpus. It is created if it is not there.
Dir string
// Origin names the population these documents belong to. It is also the
// subdirectory they land in, so a corpus of several populations stays
// countable — a prevalence is per origin or it is not a prevalence.
Origin string
// Query is handed to the archive's search.
Query string
// Want is how many documents to end up with in this origin.
Want int
// MaxBytes refuses a document larger than this. A scanned book can be
// hundreds of megabytes and says no more about which codec it uses than a
// small one does.
MaxBytes int64
// Workers is how many are fetched at once. Be modest: this is somebody
// else's server.
Workers int
// Log, when set, is told what happened to each document.
Log func(string, ...any)
}
A Plan says what to add to a corpus and from where.
type Problem ¶
type Problem struct {
// Path is relative to the corpus directory, as the manifest writes it.
Path string
// Kind is why: "missing", "unreadable", "size", "digest" or
// "unrecorded".
Kind string
// Want and Got say what the manifest records and what is on disk. Both
// are empty for a kind where there is nothing to compare.
Want, Got string
}
A Problem is one way a corpus and the manifest beside it disagree.
func Check ¶
Check reports every way a corpus directory and its manifest disagree.
The package comment says the manifest is what makes a corpus a measurement rather than a pile, "so a figure quoted from it can be reproduced and a file that changed underneath can be noticed". Nothing noticed: the hash was recorded and never read back, and a file in no row was invisible to every tool here.
It is not a hypothetical. Measured 2026-10-04 over the two corpora this repository is run against, FOUR files of `pdfscans` were on disk and in no manifest row -- and three of those four are documents our reader refuses, so `images` and `compare`, which walk the manifest, counted 63 refusals where a sweep that walked the directory counted 66. Two true numbers for one corpus, and no way to tell which a figure came from. See baseline/README.md §33.
The digest is compared over AS MANY CHARACTERS AS THE MANIFEST RECORDS. The forms corpus, gathered before this repository existed, keeps sixteen of them under the header "sha256-8"; the scans corpus keeps all sixty-four. Sixteen hex characters is sixty-four bits, which answers "did this file change" perfectly well, and a check that demanded all sixty-four would call every one of 2 268 rows changed and be deleted by the first person to run it.
Source Files