reencode

Overview ¶

Command reencode converts a document from a single-byte legacy encoding to UTF-8 and proves the text survived.

$ reencode -from windows-1252 < old.html > new.html
reencode: 1284 bytes in, 1299 out; 640 characters, fingerprints match; 1 charset declaration(s) rewritten to utf-8

The document's own charset declaration is rewritten too, because converting the bytes alone leaves a file that says it is windows-1252 and is not: a browser believes the declaration and decodes UTF-8 through the legacy table, which is the mojibake this program exists to avoid. The proof below cannot catch that, since both of its passes are told the encoding rather than reading it out of the document.

The conversion is a byte-for-character table and is not the interesting part. The proof is: the same document is read twice through the rewriter - once declared as the legacy encoding, once as UTF-8 - and the characters the handlers are given are fingerprinted both times. If the two fingerprints agree, every character of text, every attribute value and every comment came through unchanged. If they do not, the program says where they first differed and exits non-zero, leaving the caller with a document it does not vouch for.

Why the conversion is not done by the rewriter ¶

It cannot be. The rewriter decodes with the document's encoding and encodes back with the same one: there is no output-encoding option, and replacing every text chunk and every attribute with itself leaves the bytes exactly as they were - measured over windows-1252, iso-8859-2, shift_jis, euc-jp and gbk. What the rewriter is good for here is the other half: it is the only thing in the pipeline that knows how a browser will read either version, so it is the oracle rather than the converter.

What a lost byte looks like ¶

A byte the legacy decoder cannot use is handed to a handler as U+FFFD, and on the way out it becomes "�" - a reference, in the text, seven bytes where the document had one - because U+FFFD is not in a legacy repertoire and a reference is the fallback. In a document declared UTF-8 the same byte comes back as the character itself. Either way it is the signal this program looks for: a fingerprint that holds U+FFFD is a document that lost something before this program ever saw it, and the report says so rather than blaming the conversion.

References are counted as what the document wrote ¶

A handler is given "京" rather than the character it stands for, so both passes see the same six or eight bytes and the fingerprints agree. That is what makes the proof possible without a reference table: the conversion does not touch references and neither does the check.