README
¶
mobi
A Kindle MOBI ebook as a richdoc
document, in pure Go. Nothing outside the standard library and
golang.org/x/net/html is used, so this builds for js/wasm and every
architecture the fleet targets.
doc, err := mobi.Parse(src, mobi.Limits{})
// doc.Blocks is the book; doc.Meta has "title", "author", "language", …
From there it is a PDF through go-pdfkit, Markdown, HTML or DOCX through the rest of the family — a MOBI and an EPUB come out as the same kind of document and go through the same writers.
Three formats stacked on each other
Reading a MOBI is reading all three:
| a Palm Database (PDB) | a 78-byte header and a list of byte offsets, one per record. Everything else lives inside those records. |
| record 0 | a PalmDOC header, a MOBI header, and usually an EXTH block of metadata. |
| the book | ONE HTML document split across records 1..n and compressed, with its pictures in records of their own, referred to by number rather than by name. |
The HTML then goes through go-richdoc/html, exactly as an EPUB's chapters do.
⛔ The end of the last record is not written down. A record entry gives a start and nothing else, and the format's own documentation does not say how the last one ends; every reader takes it to be the end of the file, and so does this one. It is the one bound here that is a convention rather than a field.
What it reads
MOBI 6 — mobi type 2, uncompressed or PalmDOC-compressed, CP1252 or UTF-8 — which is what the format means in practice.
⛔ CP1252 is not Latin-1, and the difference is the punctuation of every
English book written on Windows. Taken as Latin-1, 0x92 — the apostrophe in
"don't" as Word writes it — becomes U+0092, an invisible control character: the
word comes out with a hole in it, in every contraction in the book. The table
is twenty-seven characters wide and it is in text.go.
Pictures, named two ways
<img recindex="00001"/> <!-- Mobipocket's own way -->
<img src="kindle:embed:0001?mime=image/jpeg"/> <!-- Calibre and KindleGen -->
Both resolve to the picture's record and come out as a data: URL. ⛔ The
number in kindle:embed is base 32 — digits then capitals, 0-9A-V — so
003C is 108 and 0010 is 32. Read as hexadecimal it would be wrong for every
picture past the thirty-first and would still look right for the first nine,
which is the kind of bug that passes a test written on a book with a cover and
nothing else.
⛔ And it is one-based from the first image record, which was measured rather than assumed: of the 164 references in the book below, all 164 land on a JPEG read one-based and 163 do read zero-based. The one that differs is what makes the question answerable.
A picture whose record is missing, or whose record turns out not to be a picture at all, is dropped — the records after the text hold indexes, a FLIS and a FCIS block and the end-of-file marker as well as the pictures — and the text around it is kept. Losing a picture must never lose a word.
Links inside the book
MOBI's internal links are byte offsets into the decompressed text —
<a filepos="0000012345"> — and not names. They come out as plain text: a
link that looks clickable and goes nowhere is worse than no link. The words are
kept in every case.
What it says no to, by name
mobi: "Bought From A Shop" is DRM'd (encryption type 2), and the text is
encrypted to a device key: there is nothing here to read
mobi: "A Commercial Title" is HUFF/CDIC-compressed (2 records from 300),
which this reader does not decode
mobi: "An AZW3" is a KF8 (AZW3) book: its text is a skeleton with fragments
woven into it, not one HTML document, and that is a different reader
⛔ A refusal says which of these it is. "Cannot read this file" would send somebody looking for a defect in their book, and none of these three files is defective.
⛔ And none of them is ErrFormat. That error means "not a Mobipocket file
at all", and a caller sniffing formats by trying readers in turn must not be
told "not mine" about a file that is.
A combined file — MOBI 6 and KF8 in one container, which is what Calibre writes by default — reads fine: record 0 describes the MOBI 6 half, and that is the half this reads.
Limits
Every field this format uses to size an allocation is a number in the file. It costs the author of that file nothing to claim four gigabytes of text.
mobi.Limits{
MaxRecords: 65535, // the Palm database's own maximum
MaxTextBytes: 64 << 20, // decompressed, across every record
MaxImageBytes: 32 << 20, // one picture
MaxTotalImageBytes: 256 << 20, // all of them
MaxDepth: 0, // handed to the HTML reader
MaxNodes: 0,
}
The zero value of each means the default above.
⛔ And the ceiling stops the work, not only the answer
A ceiling checked while a document is built fires after a delegated parser has already allocated everything — a refusal that arrives too late to be a defence. It cost go-richdoc/data 1.275 GB to refuse a 16 MiB array, go-richdoc/html 96 MB to refuse a 3.2 MB page, and go-richdoc/epub a node ceiling that bounded each chapter and not the book. All three read as correct.
So the control is ported here and committed: two books refused by the same ceiling, the second holding eight times the text of the first.
| 200 408-byte book, refused | 68 480 bytes allocated |
| 1 600 408-byte book, refused | 68 480 bytes allocated |
| ratio | 1.0 |
Identical to the byte: inflate is handed what is left of the budget and stops
there, so this reader was already right — and the control stays in the suite to
keep it that way. An accepted book costs 51.4 bytes of heap per byte, which
is where HTML content lands across this whole family (the HTML reader's own
figure is 56.3, and the cost here is mostly its).
⛔ Both figures are byte counts from runtime.MemStats, taken at a load average
above 50. They are exact under any load; a ratio of durations would not be,
so there is no timing claim here.
The trap of this format
The bytes at the end of a text record are not text, and they are not compressed either: they are indexing data and, for a multibyte character split across two records, the overlap. Which entries are present is a bitfield at offset 240 of record 0, each entry ends with its own size written backwards as a variable-width integer that counts itself, and the lowest bit is different again — its count is two bits of the last byte.
⛔ Feeding those bytes to the decompressor does not fail. It produces plausible
rubbish at the end of every record — wrong prose, which is the worst kind of
failure a reader has. The book measured below sets flags 0x3 on all 259 of
its text records.
Testing
100% of statements, measured as the profile and not the percentage, plus a
FuzzParse target run for a minute on every pull request: 16.3 million
executions in the session that wrote it, no panic.
⛔ The fixtures are built by hand, field by field, and that is not enough. A
test that reads a book this package wrote only says that the two halves of one
misreading agree. So the bytes of each PalmDOC case are typed from the format's
own description — and the first version of one of them had its back-reference
distance one short, which read "Hello ello": exactly what a reader with an
off-by-one produces on a real book.
The real measurement
Beside every book.mobi, TestARealCorpus uses a book.txt if there is one and
reports how much of it came out. Project Gutenberg publishes both formats of
every title it has:
curl -o pg1342.mobi https://www.gutenberg.org/ebooks/1342.kf8.images
curl -o pg1342.txt https://www.gutenberg.org/ebooks/1342.txt.utf-8
MOBI_CORPUS=. go test -run TestARealCorpus -v ./...
Pride and Prejudice, 25 MB, 440 records — 259 of text and 165 JPEGs, Calibre 9.5.0 — read on 2026-10-18:
| reference | with html v0.1.0 |
with html v0.1.2 |
|
|---|---|---|---|
| words | 126 630 | 125 816 | 126 512 |
| sentences of 60 characters or more | 4 215 | 3 638 (86.3%) | 4 125 (97.9%) |
164 pictures referenced, 163 carried as bytes; the remaining one is a
kindle:flow: SVG, which lives in the KF8 half.
⛔ The first column's 13.7% gap was not in this package, and reading this
book is what found it:
go-richdoc/html#4 dropped the
space following an inline element, so <i>Walt Whitman</i> has came out as one
word — under epub and email too, all along. The third column is the same
book through the one-line fix
(go-richdoc/html v0.1.2).
⛔ The 90 that still differ are not prose. They are the book's list of illustrations and its page-number column, which the two renderings lay out differently — a vocabulary mismatch, not lost text. A run reporting much worse than 97% means something here broke.
⛔ And getting the judge symmetric took three readings. The first said
76.8%, the second 79.4%, and every "missing" sentence inspected was present: the
plain-text rendering marks italics with underscores, writes
[Illustration: ...] where the ebook holds a picture, and writes an em dash as
-- where the ebook holds a real one. A judge carrying a markup layer the
subject does not have measures the markup.
⛔ There is no book in testdata, deliberately: a Kindle library is
somebody's own. Without MOBI_CORPUS the corpus test skips and says so,
because a test that passes by doing nothing is worse than one that is not there.
Bibliography
The format is not published by its owner. What this was written from:
- MOBI — the record-0 layout, the compression types, the EXTH record numbers, the extra-data flags.
- PDB — the Palm database header and record list.
- PalmDOC — the LZ77 scheme, byte range by byte range.
The rest of the family
richdoc is the document model; html, epub, fb2, markdown and email are its other readers.
License
BSD-3-Clause. See LICENSE.
Documentation
¶
Overview ¶
Package mobi reads a Kindle MOBI ebook into a richdoc.Document.
A MOBI file is three formats stacked on each other, and reading one is reading all three:
- a Palm Database (PDB): a 78-byte header and a list of byte offsets, one per record. Everything else lives inside those records.
- record 0, which holds a PalmDOC header, a MOBI header and usually an EXTH header of metadata.
- the book itself, which is ONE HTML document split across records 1..n and compressed, with its pictures in records of their own, referred to by number rather than by name.
The HTML then goes through github.com/go-richdoc/html, exactly as an EPUB's chapters do, so a Kindle book and an EPUB come out as the same kind of document and convert through the same writers.
What it reads, and what it says no to ¶
It reads MOBI 6 — mobi type 2, uncompressed or PalmDOC-compressed, in CP1252 or UTF-8 — which is what the format means in practice: the file this was measured against is Project Gutenberg's own Pride and Prejudice, 25 MB, 440 records, 259 of text and 165 JPEGs, produced by Calibre 9.5.0.
Three things are REFUSED BY NAME rather than half-read, because each one would otherwise come out as a book with its text missing:
- DRM (encryption type 1 or 2). The bytes are encrypted to a device key this package will never have. There is nothing to attempt.
- HUFF/CDIC compression (type 17480), a proprietary Huffman scheme whose only published description is a reference implementation. Rare outside commercial Mobipocket titles.
- KF8 (mobi type 248), the AZW3 format. Its text is not one HTML document but a skeleton with fragments woven into it, which is a different reader and not a flag on this one. A COMBINED file — MOBI 6 and KF8 in one container, which is what Calibre writes by default — reads fine: record 0 describes the MOBI 6 half, and that is the half this reads.
⛔ A refusal says which of these it is. "mobi: cannot read this file" would send somebody looking for a defect in their book, and three of these four files are not defective at all.
Links inside the book ¶
MOBI's internal links are byte offsets into the decompressed text — `<a filepos="0000012345">` — and not names. They come out as PLAIN TEXT, because the HTML reader treats an anchor with neither href nor id as content, and a link to nowhere is worse than no link: it looks clickable and goes nowhere. The text of the link is kept in every case.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrFormat = errors.New("mobi: not a Mobipocket file")
ErrFormat means the bytes are not a Mobipocket file at all — the wrong creator, or too short to hold a Palm database header.
⛔ It is deliberately NOT returned for a MOBI this package will not read: a DRM'd book, a HUFF/CDIC one and a KF8 one are all perfectly good Mobipocket files, and a caller sniffing formats by trying readers in turn must not be told "not mine" about a file that IS mine and is refused for a reason worth repeating to a person.
Functions ¶
func Parse ¶
Parse reads a MOBI book into a document.
The whole of the format is four steps: the Palm database gives the records, record 0 says how the text is compressed and where the pictures start, the text records concatenate into one HTML document, and that document goes through the HTML reader.
Types ¶
type Limits ¶
type Limits struct {
// MaxRecords is how many records the Palm database may hold. The format
// writes the count in two bytes, so this is a bound on WORK rather than on
// one allocation: sixty-five thousand records of four kilobytes is a
// quarter of a gigabyte of decompression.
MaxRecords int
// MaxTextBytes is how much text this will decompress in all, across every
// record. One record under the ceiling says nothing about two hundred of
// them.
MaxTextBytes int64
// MaxImageBytes is how large one picture may be and MaxTotalImageBytes how
// much picture this will carry in all. They are separate from the text
// because a book is mostly pictures and the two fail differently: one
// enormous cover, or ten thousand small decorations.
MaxImageBytes int64
MaxTotalImageBytes int64
// MaxDepth and MaxNodes are handed to the HTML reader. They are named here
// so that a caller sets one Limits rather than two.
MaxDepth int
MaxNodes int
}
Limits bound what a file can make this reader do.
⛔ Every one of these bounds an allocation decided by a number IN THE FILE. A MOBI's record 0 says how long its text is in a four-byte field: it costs the author of that file nothing to claim four gigabytes, and a reader that believes it allocates them.
Source Files