utf8stream

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 2 Imported by: 0

Documentation

Overview

Package utf8stream decodes UTF-8 chunks the way Node does: TextDecoder with { stream: true } and StringDecoder.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Decoder

type Decoder struct {
	StripBOM bool
	// contains filtered or unexported fields
}

Decoder mirrors the WHATWG UTF-8 decoder behind TextDecoder with { stream: true }: an incomplete trailing sequence waits for the next chunk, and each maximal invalid subpart becomes one U+FFFD.

With StripBOM set it also mirrors TextDecoder's default ignoreBOM: false, dropping a U+FEFF that is the stream's first code point, even when its bytes arrive split across chunks. Buffer.toString (file reads) keeps the BOM, so those callers leave StripBOM unset.

func (*Decoder) AppendDecode

func (d *Decoder) AppendDecode(dst, data []byte, stream bool) []byte

AppendDecode appends the decoding of data to dst. Between characters, the well-formed prefix of a chunk that ends at a character boundary is appended as it is; the decoder walks only what is left (an incomplete last character, or a chunk with an invalid sequence).

func (*Decoder) Decode

func (d *Decoder) Decode(data []byte, stream bool) string

Decode returns the decoding of data. Well-formed UTF-8 decodes to itself, so it is copied once; anything else runs the decoder.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL