README
¶
extract
Reads a PDF page back: the text on it, with where each piece of it sits, and the images it places.
text, err := extract.Text(doc, 1) // the page as words, lines put back together
runs, err := extract.Runs(doc, 1) // every piece, with its place and its size
imgs, err := extract.Images(doc, 1) // every picture, and where it lands
tabs, err := extract.Tables(doc, 1) // the grids its text lines up into
lines := extract.Lines(runs) // the runs gathered into their lines
s := extract.Join(lines[0]) // one line's pieces, spaced as words
Lines and Join are exported because the rule has to be the same rule:
Text uses them to put a page's text back together, and anything reading
structure out of a page — where a paragraph ends, which line is a heading, as
document does — has to agree with it
line for line, or the two disagree about what the page says. A test asserts
that agreement rather than trusting it.
Pure Go, no C. It builds for GOOS=js/wasm and every architecture the fleet
targets.
What a page actually holds
Not text. A page holds instructions for drawing glyphs, and the text is what
those glyphs were meant to say — which the document says only if it was asked
to. Where it does not, this says so rather than guessing: a run whose codes
could not be worked out comes back marked Unreadable, so a caller can tell
this page says nothing from this page could not be read.
Three things are tried, in order:
- the font's
/ToUnicodemap, which is the document's own word on what its text says; - the name its encoding gives the code — but only a name the document chose itself, since a symbolic font read through an assumed encoding gives the wrong letter with nothing to say so;
- what the embedded font program calls the code, read with
go-opentype— which is the honest answer for a symbolic font, and the reason a page of mathematics comes back as mathematics.
Text drawn in the mode that puts no ink on the page — how a scanner writes
what it read underneath the picture it read it from — is returned and marked
Invisible.
Images
A picture comes back as the file holds it, with the box it covers on the page.
A JPEG comes out as a JPEG: re-encoding it would lose something for nothing.
Turning the other kinds into pixels means reading their colour space, which is
render's work rather than this
package's.
Tables
A page holds no table. It holds text put at places, and a table is what a
reader sees when those places line up — so Tables reports an inference,
and there is nothing in the file to check it against.
A column boundary is a stretch of the page that every row of the table leaves empty. The first line of a block proposes its own gaps, each line after it takes away whatever it draws in them, and a line that leaves nothing closes the block and begins the next. That is the one decision everything else follows from, and it is subtraction rather than the intersection of the lines' gaps, which is where this started:
| a cell whose text wraps | leaves a line with one piece of text on it and so no gaps of its own. Intersecting with nothing ends the table, and a four-row table with one wrapped cell came back as two tables of two rows. A wrapped line draws nothing in the boundary beside it, so subtracting keeps it. |
Three things are not columns, and each one was found by running this over real documents rather than over a fixture:
| a bullet | a bulleted list is a two-column grid. What tells them apart is that one column says the same thing wherever it says anything — and a column like that distinguishes nothing. A mark of one or two characters, repeated, is dropped; CNRS repeated down a column is data and is kept. |
| a column nothing was drawn in | that is a stretch of page nothing was drawn in, which is what a boundary already is. A run of nothing but spaces splits a boundary in two and leaves an empty column between the halves. |
| a line far below the row above | a page number sits in the left-hand column and draws nothing in any boundary, which is exactly what a row of a table does. What says it is not one is the distance: more than three times the row's own text size. |
Measured on one 213-page document, as each of those went in:
588 tables, 1841 rows gaps the rows share
213 tables, 1631 rows + boundaries the rows leave empty (wrapped cells)
83 tables, 268 rows + a repeated mark is a bullet
40 tables, 120 rows + an empty column is not a column, and a page
number is not a row
The forty are the document's Code / Intitulé pairs, one per job
description, with their second cells wrapped. A 3-page prose report in the same
run reports none, which is the control: a detector that cannot say no to a
paragraph says yes to every page there is.
⛔ What it still gets wrong. A page laid out in two columns of prose is a grid too — the gutter is a stretch every line leaves empty. What saves the ordinary case is that prose does not stay out of its gutter for long, since one full-width line closes the block. And ruled lines are not read at all: this is the text strategy, the one a bordered table does not need, and a table whose columns are held apart by rules rather than by whitespace is not found by it.
How it is checked
100% of statements, including every branch that handles a page saying something it should not.
Then on the corpus — 118 833 real PDFs, 121 946 pages:
pages read 121946
with text 94865 (77.8%)
saying nothing 27081, of which 4812 did draw glyphs
characters 33976414 (17272226 of them letters)
runs 10797045
unreadable 203607 (1.89%)
invisible 29917
images 300685
panics 0
The 22% of pages that say nothing are figures — this corpus is arXiv's, and
most of it is plots. Of the pages that did draw glyphs and still said
nothing, nearly all are Type 3 fonts with no /ToUnicode: dvips bitmap fonts
whose glyphs are called a1 and s32, which no reader can turn back into
letters.
The rest of the family
reader reads and writes the format ·
ops is the verbs and the pdfops command ·
render turns a page into pixels ·
pdffont is what a document says about a font ·
extract reads a page back ·
coedit is a PDF several people edit at once ·
app is the browser workbench.
Documentation for all of it: https://go-pdfkit.github.io/docs/
Licence
BSD-3-Clause.
Documentation
¶
Overview ¶
Package extract reads a PDF page back: the text on it, with where each piece of it sits, and the images it places.
A page does not hold text. It holds instructions for drawing glyphs, and the text is what those glyphs were meant to say — which the document says only if it was asked to. Where it does not, this says so rather than guessing: a run whose characters could not be worked out comes back marked, so that a caller can tell "this page says nothing" from "this page could not be read".
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Assemble ¶
Assemble puts runs back together as lines of text: everything on one baseline in the order it sits across the page, with a space wherever the gap between two pieces is wide enough to be one.
func Join ¶ added in v0.7.0
Join puts one line's pieces back together, with a space wherever the gap between two of them is wide enough to be one and nothing where it is only kerning. It is Assemble for a single line, and exported for the same reason Lines is.
func Lines ¶ added in v0.7.0
Lines gathers runs into the lines they were drawn on, down the page and then left to right within each line. A line is allowed to wobble by a fraction of its own height, which is what a superscript does.
It is exported because the rule has to be the SAME rule: Assemble uses it to put a page's text back together, and anything that reads structure out of a page — where a paragraph ends, which line is a heading — has to agree with it line for line or the two disagree about what the page says.
Types ¶
type Image ¶
type Image struct {
// Name is what the page's resources call it, or empty for one written
// into the content stream.
Name string
// Width and Height are the picture's own, in samples.
Width, Height int
// X and Y are where its bottom left corner lands on the page, and
// DrawnWidth and DrawnHeight how large it is drawn, in points. A picture
// turned or skewed by the page still reports the box it covers.
X, Y float64
DrawnWidth, DrawnHeight float64
// Filter is the last filter still on the data: DCTDecode for a JPEG that
// comes out as one, JPXDecode for a JPEG 2000, and empty for data this
// has unfiltered into plain samples.
Filter reader.Name
// Data is the picture's bytes, with every filter this understands undone.
Data []byte
// Dict is the image's own dictionary, which says how to read the data:
// its colour space, how many bits a sample takes, and whether it is a
// mask.
Dict reader.Dict
// Inline reports a picture written into the content stream rather than
// held as a resource of its own.
Inline bool
}
An Image is a picture the page places, and where it puts it.
What comes back is the picture as the file holds it — the bytes and the dictionary describing them — rather than pixels. A JPEG comes out as a JPEG, which is what anyone pulling pictures out of a document wants: re-encoding it would lose something for nothing. Turning the other kinds into pixels means reading their colour space, which is the renderer's work rather than this package's.
type Run ¶
type Run struct {
// Text is what the run says.
Text string
// X and Y are where it starts, in the page's own coordinates — up from
// the bottom left corner, in points.
X, Y float64
// Width is how far the pen moved drawing it, and Size how tall the text
// is, both in points.
Width, Size float64
// Font is the name the page's resources give the font.
Font string
// Space is how wide a space would be here, which is what says whether a
// gap between two runs is a word break or only kerning.
Space float64
// Invisible reports text drawn in the mode that puts no ink on the page.
// It is how a scanner puts what it read underneath the picture it read it
// from, so it is text worth having — and worth knowing about.
Invisible bool
// Unreadable reports a run some of whose codes said nothing about which
// characters they were: the font carried no map and named its glyphs
// after numbers. What Text holds is then only the part that could be read.
Unreadable bool
}
A Run is a piece of text drawn in one go: one font, one size, one place.
type Table ¶ added in v0.6.0
type Table struct {
// Page is the page the table was found on, counting from one.
Page int
// Rows holds the cells, top row first and left to right within a row.
// Every row has the same number of cells, and a cell the page left empty
// is an empty string.
Rows [][]string
// X and Y are the bottom left corner of the TEXT the table is made of,
// and Width and Height its extent, in the page's own coordinates. A table
// drawn inside a frame has a frame a little larger than this; a table
// drawn without one has no frame at all, which is the usual case and the
// reason the box is measured from the words.
//
// It is every word of the block, including a bullet left out of
// [Table.Rows]: the box says where to look on the page, and a reader
// looking there wants to see the bullets too.
X, Y, Width, Height float64
}
A Table is a grid of text read off a page.
A PDF page does not contain a table. It contains text put at places, and a table is what a reader sees when those places line up. So a Table here is an inference, and Table.Rows is what the alignment said rather than what the author declared — there is nothing in the file to declare it with.
func Tables ¶ added in v0.6.0
Tables is every table on the page, top to bottom, with the default options.
func TablesWithOptions ¶ added in v0.6.0
TablesWithOptions is Tables with the thresholds said out loud.
⛔ A page laid out in two columns of prose is a grid too: the gutter is a stretch every line leaves empty, so a short enough stretch of such a page comes back as a two-column table whose cells are lines of text. There is nothing in the page that distinguishes the two. What saves the ordinary case is that prose does not stay out of the gutter for long — one full-width line closes the block — but a caller who knows the document is prose should not be asking. This reports alignment; it does not read minds.
type TableOptions ¶ added in v0.6.0
type TableOptions struct {
// ColumnGapSpaces is how wide a gap must be, counted in spaces, before it
// separates two columns rather than two words. The default is 1.5: one
// space is a word break, and a column boundary is wider than a word break
// or there would be no way to tell a table from a sentence.
ColumnGapSpaces float64
// MinRows and MinColumns are the smallest grid worth reporting, both 2 by
// default. One row is a line of text and one column is a paragraph.
MinRows, MinColumns int
}
TableOptions tunes what counts as a table. A zero field means the default, so TableOptions{} is the default everywhere and a caller changes one thing by naming one field.
Source Files