Documentation
¶
Overview ¶
Package subtitleenc answers what encoding a subtitle's bytes are in, by ICU's charset-detection models (internal/subtitleenc/icu) narrowed by the subtitle's language, and reads and writes them as the Encoding Standard does, on golang.org/x/text's decode tables.
Decoding computes a view and never decides whether it replaces the original bytes: subflux persists a converted subtitle only under post_processing.normalize_utf8, so a caller writing Decode's output without that setting has converted a user's file.
Index ¶
Constants ¶
const ( // SiblingMargin is the score lead an ISO-8859 page needs over its Windows // sibling, and the lead the top survivor needs over one decoding the // sample otherwise. Czech ISO-8859-2 holding only "š" and "ž" leads its // 1250 reading by 3, and Polish 1250 "Mąż" ties its ISO reading. SiblingMargin = 3 // SampleBytes is how much of a file the hint's pages read. SampleBytes = 64 << 10 )
Variables ¶
var ErrUndecodable = errors.New("subtitleenc: bytes not decodable in this encoding")
ErrUndecodable reports Unknown, or bytes the encoding cannot read or write.
Functions ¶
This section is empty.
Types ¶
type Encoding ¶
type Encoding uint8
Encoding values are stored in timing's baseline codec: append only, and a renumbering bumps timing.CodecVersion.
const ( Unknown Encoding = iota // bytes Detect could not name; never decoded UTF8 UTF16LE UTF16BE ISO2022JP // stateful 7-bit, named only by its ESC sequences Windows1250 Windows1251 Windows1252 Windows1253 Windows1254 Windows1255 Windows1256 ISO8859_2 ISO8859_5 ISO8859_6 ISO8859_7 ISO8859_8 // Hebrew, named only when windows-1255 reads a sample byte as undefined KOI8R KOI8U ShiftJIS EUCJP GB18030 Big5 EUCKR )
The encodings Detect names. The ISO-8859 names are x/text's charmap ones.
func Detect ¶
Detect answers a BOM, UTF-16 by NUL interleave, 7-bit ISO-2022-JP, valid UTF-8, then the hint row's pages over the first SampleBytes, else Unknown. A form Decode refuses is Unknown; a page it refuses or holding a C1 is out. Of a Windows page and its ISO-8859 sibling, one reading a differing byte as a non-letter is out, as is a Windows page whose refused sibling reads one as a letter; of two left the ISO page needs SiblingMargin, where ICU names it unless a C1 byte shows. The top score at or above its model's floor wins, unless one within SiblingMargin decodes otherwise.
func (Encoding) ASCIITransparent ¶
ASCIITransparent is false for UTF-16 and ISO-2022-JP: in every other, a line made only of bytes below 0x80 reads as ASCII.
func (Encoding) Decode ¶
Decode returns b as UTF-8 in a new slice, without a BOM. Unknown, and bytes the encoding's encoder would not write back from the text (read), give ErrUndecodable.
func (Encoding) Encode ¶
Encode writes valid UTF-8 text in the encoding, UTF-16 behind its BOM so Detect names it again, as bytes Decode reads back as the same text. Unknown, invalid UTF-8, a character the encoding has no code for, and text Decode would read back otherwise (GB18030's U+E000) give ErrUndecodable.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package icu is a Go port of ICU4C's charset detector (icu4c/source/i18n csrsbcs.cpp, csrmbcs.cpp and csr2022.cpp at 560e6db, ICU 69.1), under the Unicode License V3 (THIRD_PARTY_NOTICES.md).
|
Package icu is a Go port of ICU4C's charset detector (icu4c/source/i18n csrsbcs.cpp, csrmbcs.cpp and csr2022.cpp at 560e6db, ICU 69.1), under the Unicode License V3 (THIRD_PARTY_NOTICES.md). |