htmlro

package
v0.1.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: MIT, BSD-3-Clause Imports: 12 Imported by: 0

README

encoding/htmlro

A read-only pull tokenizer for HTML, and on top of it a parser that builds the tree golang.org/x/net/html builds, with a third of the allocations. The tokenizer reads names, attributes and text in place and allocates nothing in steady state; the parser makes the strings a node keeps, once each, and little else.

The name says what it is: HTML, read only. It never writes; html.Render of golang.org/x/net/html writes the trees it builds.

Why not golang.org/x/net/html as it is

Its tokenizer allocates for every attribute: a copy and a string of the name to check it for repeats, a copy each of the name and of the value to replace NULs that are nearly never there, the value as a string, and the slice that holds them, grown one doubling at a time. Seven to ten allocations per attribute, two per element on top, on pages that have more attributes than elements. Measured on 267 real pages (32 MB, 136 thousand elements, 187 thousand attributes, the test pages of go-readability), darwin/arm64:

allocations per element bytes time
html.Parse 1,920,394 14.2 127 MB 159 ms
htmlro.ParseBytes 809,755 6.0 103 MB 124 ms

And go test -bench . ./encoding/htmlro, on a 1.1 MB page of 10,000 elements with 25,000 attributes:

ns/op MB/s B/op allocs/op
html.Tokenizer, zero-copy API 4,124,480 140 964,964 110,022
htmlro.Reader 2,645,419 218 3 0
html.Parse 6,670,829 86 5,526,762 190,040
htmlro.ParseBytes 4,542,680 127 3,674,371 55,023

What remains is the tree: a node per element, text node and comment, an attribute slice per element that has attributes, a string per attribute value and per text node. Tag names and attribute names that are atoms of golang.org/x/net/html/atom cost nothing; the others are made once per document.

Reading

r := htmlro.NewReader(src, htmlro.Options{})   // or htmlro.NewBytesReader(data, ...)
for {
	k, err := r.Next()
	if err != nil {
		return err
	}
	if k == htmlro.EOF {
		break
	}
	if k == htmlro.StartTag && r.NameIs("a") {
		if href, ok := r.Attr("href"); ok {
			links = append(links, href.String())
		}
	}
}

On a StartTag the caller reads Name, and the attributes with Attr by name or all of them in order with NextAttr; they were indexed as the tag was scanned, so neither copies anything. A name a tag repeats is reported once, with its first value. On a Text, Comment or Doctype token, Text returns the content.

Every slice the reader returns aliases its buffer and is valid until the next Next. A caller that keeps a name or a value copies it.

Values

Attr, NextAttr and Text return a Value: the bytes as written, with a note of what decoding them means where they were read. AppendTo, String and Equal decode on demand: "\r\n" and "\r" become "\n"; a NUL becomes U+FFFD in an attribute value, a comment and the text of a script or style element (the parser deals with the NULs of other text, as the specification has it); and in an attribute value or text that is not raw, a character reference becomes what it names, numeric or one of the 2,231 named ones, the legacy references that need no semicolon included, with the rule that keeps &copy=1 as written in an attribute value. Content with none of those, which is nearly all of it, is never copied: NeedsDecoding says.

UnescapeString is the same decoding for a string outside the reader, what golang.org/x/net/html.UnescapeString does, so a program that has this package needs neither that nor the standard library's html for it.

Trees

doc, err := htmlro.ParseBytes(data, htmlro.Options{MaxBufferBytes: len(data)})

Parse, ParseBytes and ParseFragment are html.Parse and html.ParseFragment of golang.org/x/net/html, the same tree construction stage over this tokenizer, and they build that package's html.Node, so a tree reads, renders and is walked with that package and with the libraries built on it (go-readability, cascadia, goquery). Options.NoScripting is its ParseOptionEnableScripting(false).

The two give the same tree. The html5lib tree construction suite passes here as it does there (the same six cases skipped), the html5lib tokenizer inputs and that package's own tokenizer cases tokenize the same through both, and TestParseMatchesXNetOnCorpus compares the dumps and the renders of every document under the directory HTMLRO_CORPUS names: the 267 pages above are identical, node for node. FuzzReaderMatchesTokenizer and FuzzParseMatchesXNet keep looking.

Bounds

A token may be as long as the input unless Options.MaxBufferBytes bounds it, since the text of a script element often is; a reader of untrusted input sets it, and gets ErrTooLarge past it. Elements nest at most Options.MaxDepth deep, 512 by default as in golang.org/x/net/html, and a document that needs more is refused with ErrTooDeep where that package panics.

What it is made of

The tokenizer states (scan.go), the tree construction stage (parse.go, foreign.go, doctype.go, const.go) and the test suites are those of golang.org/x/net/html, carried over function for function so that the two read alike, under its license (LICENSE). The entity table is generated from that package's by gen.go. What is new is the reader around the states, the Value, and the way the parser takes its tokens.

The zero-allocation tests run under tinygo test as well as go test, measuring bytes allocated, since TinyGo's testing.AllocsPerRun reports nothing.

Documentation

Overview

Package htmlro is a read-only pull tokenizer for HTML and, on top of it, a parser that builds the tree golang.org/x/net/html builds, with a third of the allocations.

The tokenizer is that of golang.org/x/net/html, carried over state for state so that the two tokenize alike, and what it returns is read in place: every name, attribute and text it hands out is a slice into its own buffer, valid until the next call to Next, and a caller copies only what it keeps. In steady state it allocates nothing.

Reading

Next advances one token at a time. On a StartTag the caller reads the name and the attributes it wants, as the tag was indexed when it was scanned; on a Text, Comment or Doctype token the content, as a Value that decodes on demand:

r := htmlro.NewReader(src, htmlro.Options{})
for {
	k, err := r.Next()
	if err != nil {
		return err
	}
	if k == htmlro.EOF {
		break
	}
	if k == htmlro.StartTag && r.NameIs("a") {
		if href, ok := r.Attr("href"); ok {
			links = append(links, href.String())
		}
	}
}

Trees

Parse, ParseBytes and ParseFragment run the tree construction stage of HTML5 over the tokens, as golang.org/x/net/html does, into that package's html.Node, so that a tree reads, renders and is walked with that package and the libraries built on it. The parser makes the strings a node keeps once each, interns the tag and attribute names that are not atoms, and gathers the text of a text node in one buffer before it becomes a string. The html5lib test suite passes as it does there, and on real pages the trees are the same, node for node.

Bounds

A token may be as long as the input unless Options.MaxBufferBytes bounds it, and elements nest at most Options.MaxDepth deep, 512 by default. A document that needs more is refused, with ErrTooLarge or ErrTooDeep.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrTooLarge is returned when a token does not fit in
	// Options.MaxBufferBytes.
	ErrTooLarge = errors.New("html: token exceeds MaxBufferBytes")
	// ErrTooDeep is returned by a parse whose open elements nest deeper
	// than Options.MaxDepth.
	ErrTooDeep = errors.New("html: elements nest deeper than MaxDepth")
)

Functions

func Equal

func Equal(b []byte, s string) bool

Equal reports whether b holds exactly the bytes of s. It allocates on neither compiler: the Go compiler elides the conversion in string(b) == s, but TinyGo does not, and copies b for every comparison, every switch on string(b) and every map index by string(b). Compare names through this, NameIs or Value.Equal when the binary is a TinyGo one.

func EqualFold

func EqualFold(b []byte, s string) bool

EqualFold reports whether b equals s under ASCII case folding of b. s is lower case.

func Parse

func Parse(src io.Reader, opts Options) (*html.Node, error)

Parse returns the parse tree for the HTML read from src, as golang.org/x/net/html.Parse builds it: a tree of html.Node that the tree construction rules of HTML5 make, with the elements the rules imply and without the tags they drop. The input is assumed to be UTF-8.

Parse refuses a document whose elements nest deeper than Options.MaxDepth with ErrTooDeep, and one with a token longer than Options.MaxBufferBytes with ErrTooLarge.

func ParseBytes

func ParseBytes(data []byte, opts Options) (*html.Node, error)

ParseBytes is Parse over data, read in place and not modified.

func ParseFragment

func ParseFragment(src io.Reader, context *html.Node, opts Options) ([]*html.Node, error)

ParseFragment parses a fragment of HTML and returns the nodes that were found. If the fragment is the InnerHTML for an existing element, pass that element in context. It has the same intricacies as Parse.

func UnescapeString

func UnescapeString(s string) string

UnescapeString decodes the character references of s as the tokenizer decodes text outside an attribute value, which is what golang.org/x/net/html.UnescapeString does: "a&lt;b" becomes "a<b". Line ends and NULs are left as they are. It returns s itself when s has no ampersand.

Types

type Kind

type Kind uint8

Kind is the kind of token the reader is positioned on.

const (
	// None is the kind before the first Next and after an error.
	None Kind = iota
	// StartTag is an opening tag, <a href="x">. Its attributes are read
	// with NextAttr or Attr.
	StartTag
	// EndTag is a closing tag, </a>.
	EndTag
	// SelfClosingTag is a tag that closes itself, <br/>. The tree
	// construction rules decide whether that means anything.
	SelfClosingTag
	// Text is character data, as written: references are not decoded and
	// whitespace is not trimmed. The text of a script or style element is
	// one Text token.
	Text
	// Comment is the body of a comment, <!--x-->, and also what the
	// tokenizer makes of bogus markup: <?x>, </>, <!x>.
	Comment
	// Doctype is a document type declaration, <!DOCTYPE x>, as the text
	// after the keyword.
	Doctype
	// EOF is reported once at the end of the input.
	EOF
)

func (Kind) String

func (k Kind) String() string

type Options

type Options struct {
	// BufferSize is the initial buffer of a Reader over a stream, 64 KiB by
	// default. It grows only when one token does not fit, and only up to
	// MaxBufferBytes.
	BufferSize int
	// MaxBufferBytes bounds the buffer, and so the largest single token: a
	// tag with its attributes, a run of text, the whole text of a script or
	// style element. 0, the default, leaves it unbounded, as the HTML
	// parser of golang.org/x/net does; a reader of untrusted input sets it.
	MaxBufferBytes int
	// MaxDepth bounds the stack of open elements of a parse, 512 by
	// default, which is also the bound of golang.org/x/net/html.
	MaxDepth int
	// NoScripting parses as a browser with scripting disabled does: the
	// content of noscript elements is parsed as markup.
	NoScripting bool
	// Fragment is the tag of the element whose inner HTML a Reader reads,
	// "script" or "textarea" for instance, so that the first token is read
	// as that element's text. ParseFragment sets it from its context.
	Fragment string
}

Options bounds a Reader and a parse. The zero value selects the defaults noted on each field.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads HTML tokens from an io.Reader or from a byte slice.

Every slice a Reader returns aliases its buffer and is valid until the next call to Next. Callers that keep a name or a value copy it.

A Reader is not safe for concurrent use. Reuse one across documents with Reset rather than allocating one per document.

func NewBytesReader

func NewBytesReader(data []byte, opts Options) *Reader

NewBytesReader returns a Reader over data, which it reads in place and does not modify.

func NewReader

func NewReader(src io.Reader, opts Options) *Reader

NewReader returns a Reader over src.

func (*Reader) AllowCDATA

func (r *Reader) AllowCDATA(allow bool)

AllowCDATA sets whether the reader recognizes <![CDATA[x]]> as the text "x". Off, the default, reads it as a bogus comment. The HTML tokenizer recognizes CDATA only in foreign content, SVG and MathML, which the reader cannot know by itself; the parser sets this as it goes.

func (*Reader) Attr

func (r *Reader) Attr(name string) (Value, bool)

Attr returns the value of the attribute named name, lower case, of a start tag, and whether the tag has it.

func (*Reader) Kind

func (r *Reader) Kind() Kind

Kind returns the kind of the current token.

func (*Reader) Name

func (r *Reader) Name() []byte

Name returns the lower-cased name of a tag token: the "img" of <IMG SRC="x">. A NUL in it is U+FFFD.

func (*Reader) NameIs

func (r *Reader) NameIs(s string) bool

NameIs reports whether the current token is a tag named s, lower case.

func (*Reader) Next

func (r *Reader) Next() (Kind, error)

Next advances to the next token and returns its kind. At the end of the input it returns EOF with a nil error, once; a token the input ends inside is dropped, as the tokenizer specification has it. After an error it returns None and the same error.

func (*Reader) NextAttr

func (r *Reader) NextAttr() (name []byte, value Value, ok bool)

NextAttr returns the next attribute of a start tag not yet returned, its name lower-cased and its value as written, and whether there was one. A name that a tag repeats is returned once, with its first value.

func (*Reader) NextIsNotRawText

func (r *Reader) NextIsNotRawText()

NextIsNotRawText tells the reader that the next token is markup although the last start tag was one of an element whose content is text: a title inside svg, a textarea the parser does not open. The parser calls it as the tree construction rules require.

func (*Reader) Offset

func (r *Reader) Offset() int64

Offset returns the byte offset of the current token from the start of the input.

func (*Reader) Reset

func (r *Reader) Reset(src io.Reader)

Reset makes the Reader read src from the start, keeping its buffer.

func (*Reader) ResetBytes

func (r *Reader) ResetBytes(data []byte)

ResetBytes makes the Reader read data, in place, from the start.

func (*Reader) Text

func (r *Reader) Text() Value

Text returns the content of a Text, Comment or Doctype token, as written; its Value decodes as the tokenizer would.

type Value

type Value struct {
	// contains filtered or unexported fields
}

Value is text, comment or attribute content as it appears in the document: character references undecoded, carriage returns and NULs as written. Its methods decode on demand, so content with none of those, which is nearly all of it, is never copied. A Value aliases the reader's buffer and is valid until the reader advances.

Decoding is what the HTML tokenizer does to the bytes: "\r\n" and a lone "\r" become "\n"; a NUL becomes U+FFFD in an attribute value, a comment and the text of a raw text element (the parser handles the NULs of other text); and in an attribute value or text that is not raw, a character reference, named or numeric, becomes the characters it names, with the legacy references that need no semicolon and the Windows-1252 numeric remappings of the specification.

func (Value) AppendTo

func (v Value) AppendTo(dst []byte) []byte

AppendTo appends the decoded content to dst.

func (Value) Equal

func (v Value) Equal(s string) bool

Equal reports whether the decoded content equals s.

func (Value) NeedsDecoding

func (v Value) NeedsDecoding() bool

NeedsDecoding reports whether decoding would change the bytes: a carriage return, a NUL to replace, or an ampersand where references are decoded.

func (Value) Raw

func (v Value) Raw() []byte

Raw returns the bytes as written.

func (Value) String

func (v Value) String() string

String returns the decoded content as a new string.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL