ops

ops

The verbs of go-pdfkit: what people actually do to a PDF they already have. Merge two files, pull out pages three to seven, turn a page the right way up, drop the metadata, split a report into chapters.

Built on reader, which takes a file apart and puts one back together; nothing outside the Go standard library is used, so this builds for GOOS=js/wasm and every architecture the fleet targets.

The model

A document here is an ordered list of pages, each borrowed from a source file, plus the document-level pieces. Every operation rearranges that list or annotates its entries; nothing is applied until the document is written out. An operation therefore costs nothing until it has to, several files can be mixed freely, and — the reason the model is shaped this way — the list is a plain value that two people can edit at the same time.

What it does

go install github.com/go-pdfkit/ops/cmd/pdfops@latest

pdfops merge whole.pdf part1.pdf part2.pdf
pdfops select -pages 3-7,last report.pdf extract.pdf
pdfops delete -pages even scan.pdf fronts.pdf
pdfops rotate -pages all -by 90 sideways.pdf upright.pdf
pdfops crop -box 20,20,575,820 wide.pdf trimmed.pdf
pdfops split -every 10 book.pdf chapters/
pdfops reverse back-to-front.pdf right-way-round.pdf
pdfops nup -n 4 slides.pdf handout.pdf
pdfops booklet chapter.pdf to-fold.pdf
pdfops overlay -with letterhead.pdf plain.pdf headed.pdf
pdfops blank -before 3 report.pdf report-with-a-gap.pdf
pdfops watermark -text CONFIDENTIAL contract.pdf marked.pdf
pdfops number -format "{page} of {pages}" report.pdf numbered.pdf
pdfops bates -prefix ACME- -start 1 exhibits.pdf stamped.pdf
pdfops stamp -text "seen 25 August" -at top-right -bold notes.pdf seen.pdf
pdfops sanitize downloaded.pdf safe.pdf
pdfops flatten filled-form.pdf final.pdf
pdfops strip -annotations -bookmarks private.pdf clean.pdf
pdfops compress fat.pdf lean.pdf
pdfops encrypt -user letmein -allow print,copy plain.pdf locked.pdf
pdfops -password letmein decrypt locked.pdf plain.pdf
pdfops -password letmein permissions locked.pdf
pdfops text paper.pdf
pdfops text -layout -pages 1 paper.pdf
pdfops images paper.pdf pictures/
pdfops info file.pdf

A page range is written 1-3,7,10- and may say all, even, odd or last. It keeps its own order and its own repeats, so select -pages 3-1 reverses three pages and select -pages 1,1 gives you two copies.

-password opens an encrypted file.

text reads the page back as words, and -layout says where every piece of it sits — a page, a place, a size, and what it says. A piece the document gave no way to read comes back marked rather than guessed at, so you can tell a page that says nothing from one that could not be read. images writes out the pictures a page places, a JPEG as a JPEG.

Links and bookmarks are carried over and pointed at the pages they became here, so extracting three pages of a book leaves the links between those three working and drops the ones that led out of them. A link or a bookmark that already led nowhere in the source keeps its place without a destination, because a file with broken bookmarks should keep its shape rather than lose it.

The catalogue is always written from nothing, so a document-level script, an action that runs on opening, or a tree of embedded files never survives any operation at all. sanitize deals with what travels attached to a page: its own actions, an annotation's actions, JavaScript, launching, form submission, the annotation types that exist to play or embed something, and files associated with a page. A link to the web is not executable and stays.

What a screen reader follows

A tagged document says which run of marks on which page is a heading, a paragraph, a table cell, the label of a form field. That is the reading order a screen reader follows, and for a government form it is often what the law requires — RGAA in France, Section 508 in the United States, EN 301 549 in the EU. It cannot be copied across a rebuild: every part of it points into the document, and the number tree that indexes it is keyed by a number each page carries. So it is rebuilt, element by element, around the pages that survived, and those numbers are handed out afresh. A page that kept the number it had in a file it is no longer part of is a page a reader would look up and be told, with every confidence, about somebody else's.

An element whose every page has gone is removed and its parent pruned; one with some pages gone keeps the children that are left. An empty table cell is kept, because the shape of a table is part of what it says. A page written twice carries the structure once, on the first copy, since an element names one page. Merging two files carries no tree at all: two files have two role maps in which the same name may mean two different things, and there is no honest way to choose between them.

Of 1 633 real government forms from eighteen issuers, 1 021 carry a structure tree. After a rotate 1 014 still do — the seven that do not are the seven whose tree was empty in the file we were given — and 991 of the 1 021 are identical down to the last element, mark and annotation reference. The thirty that differ are accounted for one by one in the commit that added this. What cannot be carried is named in catalogue.go with the reason, rather than disappearing quietly.

Text is drawn in the four faces every viewer already has — Helvetica, Helvetica-Bold, Courier and Courier-Bold — so nothing is embedded and a watermark costs about a kilobyte. Stamp text may say {page}, {pages} and {n}, the last being a counter for serial numbering.

Verified against real files

Every operation is checked on the same corpus of 118 863 real PDFs the reader is measured against — Matplotlib, cairo, pdfTeX, Ghostscript, Adobe, R, Apache FOP, PDF 1.3 through 1.7. For each of the 118 833 that open, four properties have to hold, compared on the bytes of every page's content stream rather than on an exit status:

• writing the document unchanged reproduces every page;
• reversing twice is the identity;
• merging a document with itself doubles it exactly;
• selecting the last page yields exactly that page.

All four hold on all 118 833 files.

Laying pages out is checked the same way: for each of the 1 959 multi-page files in that corpus, two-up produces the right number of sheets and every form drawn on them carries exactly the bytes of the page it stands for, blanks included. All 1 959 hold.

Testing

go test -covermode=set ./...

CI gates on exact 100% statement coverage, go vet, and a cross-compile across linux/{amd64,arm64,riscv64,loong64,ppc64le,s390x}, js/wasm, darwin/arm64 and windows/amd64.

The rest of the family

reader reads and writes the format · ops is the verbs and the pdfops command · render turns a page into pixels · pdffont is what a document says about a font · extract reads a page back · coedit is a PDF several people edit at once · app is the browser workbench.

Documentation for all of it: https://go-pdfkit.github.io/docs/

License

BSD-3-Clause — see LICENSE. Copyright the go-pdfkit/ops authors.

Forms

pdfops fields lists what a form asks for and what it holds; pdfops fill fills it in and saves it.

$ pdfops fields fw9.pdf
note: the file also carries an XFA form, which is not read; the standard one is.
topmostSubform[0].Page1[0].f1_01[0]                       text      ""
topmostSubform[0].Page1[0].Boxes3a-b_ReadOrder[0].c1_1[0] checkbox  "Off"
                                                            buttons [1]

$ pdfops fill -set 'topmostSubform[0].Page1[0].f1_01[0]=Wolfgang Amadeus Mozart' \
              -set 'topmostSubform[0].Page1[0].Boxes3a-b_ReadOrder[0].c1_1[0]=1' \
              fw9.pdf filled.pdf

A filled form is written as an incremental update: the original file, byte for byte, with the objects that changed appended after it and a new cross-reference section pointing back at the old one. That is how every program that saves a form saves one, and it is the safest thing a program can do to somebody's document — nothing already there is rewritten, so whatever this does not understand survives untouched, and if the update is wrong the original is still the first part of the file.

The update says where its objects went the same way the file already does. A file whose cross-reference is a stream cannot be pointed back at by a plain table: a reader following /Prev would find an object where it expected the word xref. That is not a nicety — macOS's own renderer draws nothing at all for such a file, which is how the mistake was found.

Two things it will not do. A document that had to be repaired to be read has no cross-reference section worth pointing back at, so it is refused rather than added to. A document that is encrypted has every string and stream in it written through a key, and this does not yet write into one.

Measured on a real form: fw9.pdf filled in every field, written out, read back with every value in place, and rendered by macOS — which drew what we drew, in the same places, comb cells and ticks included.

⚠ Filling a form is the only verb that keeps one. Every other verb here takes the pages apart and builds a new document round them, and a form is tied into a document by object number in a dozen places at once — so merging, splitting or rotating a form's pages loses the form. Use fill on the file itself.

The form inside a blank document

Some PDFs are blank to look at and are not blank. Their pages hold one panel — "Please wait... your PDF viewer may not be able to display this type of document" — and the real form exists only as XML beside them, laid out when Adobe's own reader opens it. Adobe removed the format from PDF 2.0; no browser and no other reader lays one out.

pdfops xfa lays it out and draws it, which turns such a file into one anything can show.

$ pdfops xfa t1135-fill-23e.pdf readable.pdf
6 sheets, 497 elements drawn

Measured over every dynamic form in a corpus of 2 240 real government forms — fourteen of them, from the Canada Revenue Agency, the French cerfa series, the US Department of Labor and the Office of Personnel Management: all fourteen laid out, 111 sheets, 9 808 elements, nothing left unplaced, and every file read back by poppler.

WHERE each element goes is go-pdfkit/xfa's, measured against pdf.js and pdfium. What is drawn INSIDE each box is this package's own and much simpler: the four standard faces, a greedy line break, a rule round each field. So this is legibility rather than fidelity — the difference between a blank sheet and a form somebody can read, not between this and Adobe.

Pictures are drawn — 27 of them across twelve of the fourteen. For most that is a logo; for French cerfa 12064 it is the whole printed form. That one carries 212 fields, 7 draws, 4 images and — counted, not guessed — no <text> and no <caption> whatever, so without its pictures it is a grid of empty boxes and with them it is the customs declaration it is. How big each one is drawn follows pdfium's own arithmetic (xfa/fxfa/cxfa_ffwidget.cpp:55-86), because pdf.js declines to settle fit and actual and leaves them to a browser's sizing of an <img>.

A picture's own size is its pixels at its own resolution, read from a PNG's pHYs chunk or a JPEG's JFIF density, and that is not a detail. Every Canada Revenue Agency form in the corpus writes aspect="actual", and their logos are 1200-dpi PNGs: 2617 pixels across is 157.02 points, which is to a hundredth of a point the width the template writes for the box. Taken at 72 dpi — one point per pixel, the assumption for a picture that declares nothing — the same logo is 2617 points wide and covers a third of the sheet in black. It did, until the render was looked at.

Page numbers come out right — "Page 3 of 11", and "Page 3 sur 4" on a French form — without running the form's scripts. A footer like that is not text: it is rich text holding a floating field naming a hidden field whose value a script sets. All sixteen of those in the corpus ask one of exactly two things:

this.rawValue = xfa.layout.page(this);
this.rawValue = xfa.layout.pageCount();

Those are questions asked of the layout, which has just answered them. Recognising the two and handing back what the pager computed is not evaluating anything. Neither reference does this — pdf.js's layout ignores xfa:embed entirely, pdfium runs a JavaScript engine — so it is named as ours. Anything else a floating field names is left as a gap, because a wrong number that looks right is worse than a blank: four cerfa fields carry a static default beside their script, and printing that would put "Page 1 of 1" on every sheet.

Two limits, both named rather than left to be discovered:

• A picture in a file outside the document is not fetched. That is pdf.js's position and its words: "we don't get remote data and use what we have in the pdf itself, so no picture for non null href". Two of cerfa 12818's are like that, and both name an absolute path on the machine of whoever drew the form — C:\Users\…\Downloads\logo ministere.PNG. They are reported, not passed over.
• Text is set smaller rather than cut off. The box was measured with the font the template names and this draws in Helvetica, which is wider; at the template's own size the words come to more lines than the box holds. The size comes down until they fit, with a floor of four points, because losing the end of a sentence is worse than setting it a point smaller.