textmap

Overview ¶

Command textmap reports the source location of every text chunk and reconstructs the document from what it was told. The reconstruction is the point: it is how you find out that one of the two obvious recipes is not reliable, and what to use instead.

The unreliable recipe is the direct one - slice the input at each text chunk's range and concatenate. The SourceLocation documentation says the offsets are "absolute and unaffected by how the document was written in - one byte at a time gives the same numbers as one call", and that "slicing the input at the range works and measuring the reported string does not". Both hold for an element, an end tag, a comment and a doctype, measured at every write size. For a text chunk they hold only when no multi-byte character straddles a write boundary.

`<p>a€b</p>`, twelve bytes with one three-byte character, reports this:

write size   text chunks                     slices are the text   bytes named by nothing
1            3..4 "a" 6..7 "€" 7..8 "b"      no                    4, 5
2            3..4 "a" 6..8 "€b"              no                    4, 5
3            3..6 "a" 6..8 "€b"              no                    -
4            3..4 "a" 4..8 "€b"              yes                   -
5            3..5 "a" 5..8 "€b"              no                    -
6            3..6 "a" 6..8 "€b"              no                    -
7            3..7 "a€" 7..8 "b"              yes                   -
8 and up     3..8 "a€b"                      yes                   -

At size 3 the chunk whose text is the single byte "a" reports a three-byte range, because the two bytes of the euro sign that arrived in the same write are held for the rest of the character and charged to the chunk already emitted. At size 1 they are charged to nothing, so two bytes of the document are named by no chunk at all. The chunk text is right throughout; it is the range that moves.

So a proxy that copies from an io.Reader into the rewriter with a fixed buffer gets ranges that depend on where its buffer edges fell. Nothing errors, and ASCII never shows it.

The reliable recipe anchors on the units whose ranges do not move and takes everything between them from the caller's own buffer. Exact is that: element, end tag, comment and doctype ranges as anchors, gaps copied from the input. It is exact at every write size from one byte up, for a document with multi-byte text, for one with character references, for a stray end tag (which no handler reports at all - B194 - and which the gap copy therefore recovers for free), and for a body that is not HTML.

That last one is the reason to bother. A text handler makes a rewrite lossy: every byte that is not valid in the declared encoding becomes U+FFFD, so a gzip body comes out 67 bytes in and 117 out. The anchored reconstruction of that same body is byte-exact at every write size, because it never reads a chunk's text - it reads the caller's bytes. An observer that has to be lossless on a body that might not be text can be.