subtitleenc

package
v0.10.0-dev.39 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: AGPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package subtitleenc answers what encoding a subtitle's bytes are in, by ICU's charset-detection models (internal/subtitleenc/icu) narrowed by the subtitle's language, and reads and writes them as the Encoding Standard does, on golang.org/x/text's decode tables.

Decoding computes a view and never decides whether it replaces the original bytes: subflux persists a converted subtitle only under post_processing.normalize_utf8, so a caller writing Decode's output without that setting has converted a user's file.

Index

Constants

View Source
const (
	// SiblingMargin is the score lead an ISO-8859 page needs over its Windows
	// sibling, and the lead the top survivor needs over one decoding the
	// sample otherwise. Czech ISO-8859-2 holding only "š" and "ž" leads its
	// 1250 reading by 3, and Polish 1250 "Mąż" ties its ISO reading.
	SiblingMargin = 3
	// SampleBytes is how much of a file the hint's pages read.
	SampleBytes = 64 << 10
)

Variables

View Source
var ErrUndecodable = errors.New("subtitleenc: bytes not decodable in this encoding")

ErrUndecodable reports Unknown, or bytes the encoding cannot read or write.

Functions

This section is empty.

Types

type Encoding

type Encoding uint8

Encoding values are stored in timing's baseline codec: append only, and a renumbering bumps timing.CodecVersion.

const (
	Unknown Encoding = iota // bytes Detect could not name; never decoded
	UTF8
	UTF16LE
	UTF16BE
	ISO2022JP // stateful 7-bit, named only by its ESC sequences
	Windows1250
	Windows1251
	Windows1252
	Windows1253
	Windows1254
	Windows1255
	Windows1256
	ISO8859_2
	ISO8859_5
	ISO8859_6
	ISO8859_7
	ISO8859_8 // Hebrew, named only when windows-1255 reads a sample byte as undefined
	KOI8R
	KOI8U
	ShiftJIS
	EUCJP
	GB18030
	Big5
	EUCKR
)

The encodings Detect names. The ISO-8859 names are x/text's charmap ones.

func Detect

func Detect(data []byte, hint language.Tag) Encoding

Detect answers a BOM, UTF-16 by NUL interleave, 7-bit ISO-2022-JP, valid UTF-8, then the hint row's pages over the first SampleBytes, else Unknown. A form Decode refuses is Unknown; a page it refuses or holding a C1 is out. Of a Windows page and its ISO-8859 sibling, one reading a differing byte as a non-letter is out, as is a Windows page whose refused sibling reads one as a letter; of two left the ISO page needs SiblingMargin, where ICU names it unless a C1 byte shows. The top score at or above its model's floor wins, unless one within SiblingMargin decodes otherwise.

func (Encoding) ASCIITransparent

func (e Encoding) ASCIITransparent() bool

ASCIITransparent is false for UTF-16 and ISO-2022-JP: in every other, a line made only of bytes below 0x80 reads as ASCII.

func (Encoding) Decode

func (e Encoding) Decode(b []byte) ([]byte, error)

Decode returns b as UTF-8 in a new slice, without a BOM. Unknown, and bytes the encoding's encoder would not write back from the text (read), give ErrUndecodable.

func (Encoding) Encode

func (e Encoding) Encode(text []byte) ([]byte, error)

Encode writes valid UTF-8 text in the encoding, UTF-16 behind its BOM so Detect names it again, as bytes Decode reads back as the same text. Unknown, invalid UTF-8, a character the encoding has no code for, and text Decode would read back otherwise (GB18030's U+E000) give ErrUndecodable.

func (Encoding) String

func (e Encoding) String() string

String is the value's Go name, such as "Windows1250".

func (Encoding) Valid

func (e Encoding) Valid() bool

Valid reports whether e is a declared Encoding, Unknown included: a byte read back from storage may not be.

Directories

Path Synopsis
Package icu is a Go port of ICU4C's charset detector (icu4c/source/i18n csrsbcs.cpp, csrmbcs.cpp and csr2022.cpp at 560e6db, ICU 69.1), under the Unicode License V3 (THIRD_PARTY_NOTICES.md).
Package icu is a Go port of ICU4C's charset detector (icu4c/source/i18n csrsbcs.cpp, csrmbcs.cpp and csr2022.cpp at 560e6db, ICU 69.1), under the Unicode License V3 (THIRD_PARTY_NOTICES.md).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL