Documentation
¶
Overview ¶
Package subtitleenc is the single authority for what encoding a subtitle's bytes are in and how to read them as UTF-8. Every site that asks "is this text, and what does it say?" reads the same decoder, so the content gate, the SRT parser and the post-processing step cannot disagree about whether a file is subtitle text or binary.
Normalize COMPUTES a UTF-8 view. It never decides whether that view replaces the original bytes, and no caller may treat it as having done so. That split is load-bearing: subflux persists a converted subtitle only when post_processing.normalize_utf8 is on, and a subtitle otherwise reaches disk byte-for-byte as the provider sent it. So a caller that needs to JUDGE the bytes (subtitlefile.Validate) or PARSE them (the sync paths) decodes a view and throws it away, while post-processing decodes and keeps the result. A caller that persists this function's output without consulting that setting has silently converted a user's file.
Zero dependencies on the rest of subflux, deliberately: it sits below the packages that read subtitle content so any of them can reach it.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Normalize ¶
Normalize returns the UTF-8 view of subtitle data, detecting UTF-8 (with or without BOM), UTF-16 LE/BE (with or without BOM), and the common single-byte encodings (Windows-1252, ISO-8859-1).
Any byte-order mark and every embedded NUL byte are removed. When the input is already valid, BOM-free, NUL-free UTF-8 the original slice is returned (not a copy); callers must not mutate the returned slice if they need the original data unchanged. The result is a fixed point: Normalize(Normalize(x)) always equals Normalize(x), which is what lets a caller decode for a decision and a later step decode again for persistence without the two disagreeing.
The caller owns persistence. See the package comment.
func TextView ¶
TextView returns the bytes a text probe should judge, decoding first only when the bytes cannot be judged without it. It is the one answer to "what do I run a text test against?", so the content gate and the archive probe cannot disagree.
A UTF-16 subtitle is about half NUL bytes, so any byte-level text test calls it binary; measured, that refused four of the eight members of a real SubSource pack. Decoding is therefore necessary for that family and for no other: an encoding this package could not NAME gets its raw bytes back, so a text verdict is never manufactured by the Windows-1252 fallback or by NUL stripping.
The result is a view. It is never the bytes anyone persists; see the package comment.
Types ¶
type Encoding ¶
type Encoding int
Encoding is what detection could positively conclude about a subtitle's bytes.
The encodings detection can name. Unknown means nothing identified the bytes and only the Windows-1252 catch-all would apply, which is a fallback rather than a finding: that decoder maps every possible byte to some rune, so it turns arbitrary binary into text-shaped output.
func Detect ¶
Detect reports the encoding it can positively identify, from a byte-order mark or from the NUL-interleaving pattern of ASCII-dominant UTF-16.
The pattern check must run before the utf8.Valid test, because NUL-interleaved ASCII ('H' 0x00 'i' 0x00) is itself technically valid UTF-8 and would otherwise be named UTF8 and passed through uncorrected. The pattern is only reliable for ASCII-dominant text: non-ASCII-heavy content with no BOM can be read BE for LE or the reverse, which real subtitle files never are.
Normalize decodes strictly more than this names, on purpose: it must always produce something, so it ends at Windows-1252 and at NUL stripping. A caller deciding whether bytes are TEXT must not lean on either, because both accept anything, so it asks this instead.