encodingmatrix

Overview ¶

Command encodingmatrix runs a document through every encoding the rewriter accepts and compares what comes out.

$ encodingmatrix < page.html
structure: identical in all 36 encodings
readings:  14 distinct readings of the text
  shift_jis, euc-jp, gbk, gb18030, big5, euc-kr   京都 (and 3 more spans)
  windows-1252, iso-8859-1                        ‹žì (and 3 more spans)
  ...

Two questions, and they have different answers. Which bytes are markup - where the tags are, what they are called, which spans are text - is the same in every encoding. What those bytes say is different in almost every one. The first is the property worth checking on a corpus; the second is the reason a document has to declare its encoding.

Why the structure is the interesting half ¶

In a browser, a legacy multi-byte encoding can hide a markup character: a lead byte takes the byte after it, and if that byte is a quote or a ">" then a filter reading the bytes and a browser reading the characters disagree about where the tag ended. That is a whole class of cross-site scripting, and it is the reason to check rather than assume.

Measured, it does not happen here: over a corpus that puts every markup character after nine different lead bytes, all 36 accepted encodings agree with each other and with x-user-defined - which is single-byte and maps every high byte to a character of its own, so it cannot combine bytes even in principle. The structure this program compares is the byte spans of the elements, their names, their attribute names, and the byte spans of the text and comments, all from lolhtml.Element.SourceLocation, which reports byte offsets whatever the encoding.

What it will not tell you ¶

Which encoding a document is actually in. This program says what each label would make of it, and a caller who wants to guess has to decide what looks like language - which is not a question a rewriter can answer. What it does say is when a label makes the text undecodable: a byte the decoder cannot use is reported as U+FFFD and counted here, and the label with the fewest of those is a reasonable first guess.