Documentation
¶
Overview ¶
Package htmlro is a read-only pull tokenizer for HTML and, on top of it, a parser that builds the tree golang.org/x/net/html builds, with a third of the allocations.
The tokenizer is that of golang.org/x/net/html, carried over state for state so that the two tokenize alike, and what it returns is read in place: every name, attribute and text it hands out is a slice into its own buffer, valid until the next call to Next, and a caller copies only what it keeps. In steady state it allocates nothing.
Reading ¶
Next advances one token at a time. On a StartTag the caller reads the name and the attributes it wants, as the tag was indexed when it was scanned; on a Text, Comment or Doctype token the content, as a Value that decodes on demand:
r := htmlro.NewReader(src, htmlro.Options{})
for {
k, err := r.Next()
if err != nil {
return err
}
if k == htmlro.EOF {
break
}
if k == htmlro.StartTag && r.NameIs("a") {
if href, ok := r.Attr("href"); ok {
links = append(links, href.String())
}
}
}
Trees ¶
Parse, ParseBytes and ParseFragment run the tree construction stage of HTML5 over the tokens, as golang.org/x/net/html does, into that package's html.Node, so that a tree reads, renders and is walked with that package and the libraries built on it. The parser makes the strings a node keeps once each, interns the tag and attribute names that are not atoms, and gathers the text of a text node in one buffer before it becomes a string. The html5lib test suite passes as it does there, and on real pages the trees are the same, node for node.
Bounds ¶
A token may be as long as the input unless Options.MaxBufferBytes bounds it, and elements nest at most Options.MaxDepth deep, 512 by default. A document that needs more is refused, with ErrTooLarge or ErrTooDeep.
Index ¶
- Variables
- func Equal(b []byte, s string) bool
- func EqualFold(b []byte, s string) bool
- func Parse(src io.Reader, opts Options) (*html.Node, error)
- func ParseBytes(data []byte, opts Options) (*html.Node, error)
- func ParseFragment(src io.Reader, context *html.Node, opts Options) ([]*html.Node, error)
- func UnescapeString(s string) string
- type Kind
- type Options
- type Reader
- func (r *Reader) AllowCDATA(allow bool)
- func (r *Reader) Attr(name string) (Value, bool)
- func (r *Reader) Kind() Kind
- func (r *Reader) Name() []byte
- func (r *Reader) NameIs(s string) bool
- func (r *Reader) Next() (Kind, error)
- func (r *Reader) NextAttr() (name []byte, value Value, ok bool)
- func (r *Reader) NextIsNotRawText()
- func (r *Reader) Offset() int64
- func (r *Reader) Reset(src io.Reader)
- func (r *Reader) ResetBytes(data []byte)
- func (r *Reader) Text() Value
- type Value
Constants ¶
This section is empty.
Variables ¶
var ( // ErrTooLarge is returned when a token does not fit in // Options.MaxBufferBytes. ErrTooLarge = errors.New("html: token exceeds MaxBufferBytes") // ErrTooDeep is returned by a parse whose open elements nest deeper // than Options.MaxDepth. ErrTooDeep = errors.New("html: elements nest deeper than MaxDepth") )
Functions ¶
func Equal ¶
Equal reports whether b holds exactly the bytes of s. It allocates on neither compiler: the Go compiler elides the conversion in string(b) == s, but TinyGo does not, and copies b for every comparison, every switch on string(b) and every map index by string(b). Compare names through this, NameIs or Value.Equal when the binary is a TinyGo one.
func EqualFold ¶
EqualFold reports whether b equals s under ASCII case folding of b. s is lower case.
func Parse ¶
Parse returns the parse tree for the HTML read from src, as golang.org/x/net/html.Parse builds it: a tree of html.Node that the tree construction rules of HTML5 make, with the elements the rules imply and without the tags they drop. The input is assumed to be UTF-8.
Parse refuses a document whose elements nest deeper than Options.MaxDepth with ErrTooDeep, and one with a token longer than Options.MaxBufferBytes with ErrTooLarge.
func ParseBytes ¶
ParseBytes is Parse over data, read in place and not modified.
func ParseFragment ¶
ParseFragment parses a fragment of HTML and returns the nodes that were found. If the fragment is the InnerHTML for an existing element, pass that element in context. It has the same intricacies as Parse.
func UnescapeString ¶
UnescapeString decodes the character references of s as the tokenizer decodes text outside an attribute value, which is what golang.org/x/net/html.UnescapeString does: "a<b" becomes "a<b". Line ends and NULs are left as they are. It returns s itself when s has no ampersand.
Types ¶
type Kind ¶
type Kind uint8
Kind is the kind of token the reader is positioned on.
const ( // None is the kind before the first Next and after an error. None Kind = iota // StartTag is an opening tag, <a href="x">. Its attributes are read // with NextAttr or Attr. StartTag // EndTag is a closing tag, </a>. EndTag // SelfClosingTag is a tag that closes itself, <br/>. The tree // construction rules decide whether that means anything. SelfClosingTag // Text is character data, as written: references are not decoded and // whitespace is not trimmed. The text of a script or style element is // one Text token. Text // Comment is the body of a comment, <!--x-->, and also what the // tokenizer makes of bogus markup: <?x>, </>, <!x>. Comment // Doctype is a document type declaration, <!DOCTYPE x>, as the text // after the keyword. Doctype // EOF is reported once at the end of the input. EOF )
type Options ¶
type Options struct {
// BufferSize is the initial buffer of a Reader over a stream, 64 KiB by
// default. It grows only when one token does not fit, and only up to
// MaxBufferBytes.
BufferSize int
// MaxBufferBytes bounds the buffer, and so the largest single token: a
// tag with its attributes, a run of text, the whole text of a script or
// style element. 0, the default, leaves it unbounded, as the HTML
// parser of golang.org/x/net does; a reader of untrusted input sets it.
MaxBufferBytes int
// MaxDepth bounds the stack of open elements of a parse, 512 by
// default, which is also the bound of golang.org/x/net/html.
MaxDepth int
// NoScripting parses as a browser with scripting disabled does: the
// content of noscript elements is parsed as markup.
NoScripting bool
// Fragment is the tag of the element whose inner HTML a Reader reads,
// "script" or "textarea" for instance, so that the first token is read
// as that element's text. ParseFragment sets it from its context.
Fragment string
}
Options bounds a Reader and a parse. The zero value selects the defaults noted on each field.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads HTML tokens from an io.Reader or from a byte slice.
Every slice a Reader returns aliases its buffer and is valid until the next call to Next. Callers that keep a name or a value copy it.
A Reader is not safe for concurrent use. Reuse one across documents with Reset rather than allocating one per document.
func NewBytesReader ¶
NewBytesReader returns a Reader over data, which it reads in place and does not modify.
func (*Reader) AllowCDATA ¶
AllowCDATA sets whether the reader recognizes <![CDATA[x]]> as the text "x". Off, the default, reads it as a bogus comment. The HTML tokenizer recognizes CDATA only in foreign content, SVG and MathML, which the reader cannot know by itself; the parser sets this as it goes.
func (*Reader) Attr ¶
Attr returns the value of the attribute named name, lower case, of a start tag, and whether the tag has it.
func (*Reader) Name ¶
Name returns the lower-cased name of a tag token: the "img" of <IMG SRC="x">. A NUL in it is U+FFFD.
func (*Reader) Next ¶
Next advances to the next token and returns its kind. At the end of the input it returns EOF with a nil error, once; a token the input ends inside is dropped, as the tokenizer specification has it. After an error it returns None and the same error.
func (*Reader) NextAttr ¶
NextAttr returns the next attribute of a start tag not yet returned, its name lower-cased and its value as written, and whether there was one. A name that a tag repeats is returned once, with its first value.
func (*Reader) NextIsNotRawText ¶
func (r *Reader) NextIsNotRawText()
NextIsNotRawText tells the reader that the next token is markup although the last start tag was one of an element whose content is text: a title inside svg, a textarea the parser does not open. The parser calls it as the tree construction rules require.
func (*Reader) Offset ¶
Offset returns the byte offset of the current token from the start of the input.
func (*Reader) ResetBytes ¶
ResetBytes makes the Reader read data, in place, from the start.
type Value ¶
type Value struct {
// contains filtered or unexported fields
}
Value is text, comment or attribute content as it appears in the document: character references undecoded, carriage returns and NULs as written. Its methods decode on demand, so content with none of those, which is nearly all of it, is never copied. A Value aliases the reader's buffer and is valid until the reader advances.
Decoding is what the HTML tokenizer does to the bytes: "\r\n" and a lone "\r" become "\n"; a NUL becomes U+FFFD in an attribute value, a comment and the text of a raw text element (the parser handles the NULs of other text); and in an attribute value or text that is not raw, a character reference, named or numeric, becomes the characters it names, with the legacy references that need no semicolon and the Windows-1252 numeric remappings of the specification.
func (Value) NeedsDecoding ¶
NeedsDecoding reports whether decoding would change the bytes: a carriage return, a NUL to replace, or an ampersand where references are decoded.