Documentation
¶
Overview ¶
Package encoding detects and converts file encodings for the built-in file tools. The detection cascade (BOM → strict UTF-8 → GB18030 → lossy UTF-8) mirrors v1's file-encoding.ts and keeps CJK Windows files editable without silently mangling their bytes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Decoder ¶
func Decoder(enc Kind) transform.Transformer
Decoder returns a streaming transform.Transformer for the given encoding, suitable for wrapping an io.Reader via dec.Reader(r). Returns nil for UTF-8 and LossyUTF8 (no transformation needed — the caller should read directly).
Types ¶
type Kind ¶
type Kind int
Kind identifies a detected file encoding.
const ( // UTF8 is plain UTF-8 without a BOM — the common case. UTF8 Kind = iota // UTF8BOM is UTF-8 with a leading BOM (EF BB BF). UTF8BOM // UTF16LE is UTF-16 Little-Endian with a BOM (FF FE). UTF16LE // UTF16BE is UTF-16 Big-Endian with a BOM (FE FF). UTF16BE // GB18030 is the Chinese national standard charset (superset of GBK). GB18030 // LossyUTF8 is not valid UTF-8 and not valid GB18030 — decoded lossily // as UTF-8 with replacement characters so the model sees something. LossyUTF8 // UTF16LENoBOM is UTF-16 Little-Endian without a BOM — common for source // files saved by Windows tools. Detected heuristically from the NUL-byte // pattern; written back without a BOM to preserve the original bytes. UTF16LENoBOM // UTF16BENoBOM is UTF-16 Big-Endian without a BOM. UTF16BENoBOM )
func Detect ¶
Detect returns the encoding kind for the given raw file bytes. The same bytes should then be passed to Decode for conversion to UTF-8.
func DetectQuick ¶
DetectQuick checks only for BOM prefixes in the first few bytes. This is the fast path for peek-based binary rejection: BOM-prefixed files (UTF-16, UTF-8 BOM) skip the NUL-byte check since 0x00 is normal in UTF-16. Returns UTF8 for non-BOM content (the caller should fall through to full Detect after verifying no NUL bytes).
func DetectUTF16NoBOM ¶
DetectUTF16NoBOM heuristically recognises BOM-less UTF-16 from the NUL-byte distribution: ASCII-range text encodes one byte of payload and one 0x00 per code unit, so the NULs cluster on odd offsets (LE) or even offsets (BE). It requires a strong skew — one parity heavily NUL, the other almost none — so genuine binary (NULs on both parities) and plain UTF-8 (no NULs) fall through.
Source Files