Documentation
¶
Overview ¶
Package dataapi is the Go port of hyper-html-api's read engine: it turns a document plus a set of extraction rules into JSON. It is a pure library — no net/http, no session, no filesystem — so the server can wrap it without the engine knowing anything about requests.
The contract it implements is not "reasonable JSON extraction", it is "byte-for-byte what the JavaScript engine already returns", quirks included. Three JS hosts (hyperclay, hyperclay-local, makerclay) answer this API today and their answers are what pages depend on. Where this package deliberately differs, the difference is recorded in the conformance corpus as a tier-3 case, never left to be discovered.
See plans/htmlclay/data-api-plan.md for the full parity ledger.
Index ¶
- Constants
- func Find(ctx *html.Node, selector string, opts FindOpts) ([]*html.Node, error)
- func Marshal(v Value) ([]byte, error)
- type Document
- type FindOpts
- type InvalidRulesToken
- type MaxRuleDepthExceeded
- type Object
- type RulesParseError
- type RulesTag
- type SelectorError
- type SelectorFailure
- type UnknownRulesVersion
- type UnsupportedSelector
- type Value
Constants ¶
const SupportedRulesVersion = "1"
SupportedRulesVersion is the only data-rules-version this engine accepts.
Variables ¶
This section is empty.
Functions ¶
func Find ¶
Find returns the descendants of ctx matching selector, in document order. Self is never included, matching cheerio's .find().
func Marshal ¶
Marshal renders a Value the way the JS hosts do: HTML characters are NOT escaped, because JSON.stringify does not escape them and extracted values routinely carry < and &. Go's default encoder escapes all three of < > & into \u00XX, which would make every response differ from the reference.
Two byte-level exceptions we cannot close and therefore document: Go escapes U+2028 and U+2029 even with escaping off, and Go replaces invalid UTF-8 with U+FFFD where JS preserves the lone surrogate. Lone surrogates are rejected upstream in rules.go rather than silently mangled here.
Types ¶
type Document ¶
Document is a parsed page with its <template> content detached.
Detaching is the only way to get browser semantics. A skip inside find() is not enough, because relational and structural pseudos still OBSERVE the skipped nodes: measured on <body><template><b id=inside>x</b></template></body>, cascadia matches body:has(b) and body:has(#inside) and does not consider the body empty. Filtering <b> out of the results still returns the wrong <body>. Once the children are off the tree there is no mechanism left that can see them.
The content is kept here rather than thrown away because @innerHTML and @outerHTML must still serialize the template markup, exactly as a browser does. So the accessor matrix falls out:
find() and every pseudo never see template content text(), @textContent exclude it, transitively, because it is off the tree @innerHTML, @outerHTML keep it, because the renderer consults this map
func Parse ¶
Parse reads a document, detaches its template content, and undoes the parser's attribute sort.
The source bytes are buffered because restoreAttrOrder needs a second, independent pass over them.
func ParseBytes ¶
ParseBytes is Parse over a byte slice, which is how the server has the file.
func (*Document) Extract ¶
Extract turns a document and a rule tree into a Value. Ported from extract.js.
The context node for a top-level rule is the document root, and matching runs over the shadow tree — template content was detached at parse time, so no selector here can observe it.
func (*Document) FindRulesIn ¶
FindRulesIn returns the first script[data-rules-name~="token"] in root, with its body parsed. A missing tag is (nil, nil) — not an error — because the HTTP status for "this page publishes no such endpoint" is the server's call, not the engine's.
The JS also has a no-token form that takes the first data-rules-name script of any name. That is a tooling path with no caller on the server side, so it is not ported.
type FindOpts ¶
type FindOpts struct {
// IncludeRulesTag keeps script[data-rules-name] elements in the result. Only the rules-tag
// lookup sets it, and it MUST: isRulesTag matches any such script, so without the flag the
// lookup would filter out the very tag it selects.
IncludeRulesTag bool
}
FindOpts is the read-path subset of the JS adapter's find options. The adapter also takes `skip` and `templateAttr`, both of which exist only for the CMS write path; leaving them out keeps the one caller that matters honest rather than carrying dead configuration.
type InvalidRulesToken ¶
type InvalidRulesToken struct {
Token string
}
InvalidRulesToken is a rules-tag token that failed validation before reaching the selector.
func (*InvalidRulesToken) Error ¶
func (e *InvalidRulesToken) Error() string
type MaxRuleDepthExceeded ¶
type MaxRuleDepthExceeded struct {
Path []string
}
MaxRuleDepthExceeded mirrors the JS error of the same name, message included, because the path it names is genuinely useful for finding the offending branch of a large rule tree.
func (*MaxRuleDepthExceeded) Error ¶
func (e *MaxRuleDepthExceeded) Error() string
type Object ¶
type Object struct {
// contains filtered or unexported fields
}
Object is a JSON object that remembers its key order, because the JS engine's output order is observable and pages rely on it. encoding/json marshals a Go map in sorted order, which would silently reorder every response.
func (*Object) Define ¶
Define stores a key the way JSON.parse does, keeping "__proto__" as a real own property.
The difference from Set is not a nicety, it is the whole reason both exist. JSON.parse builds objects with CreateDataProperty, which bypasses the prototype setter; assignment does not. So a "__proto__" rule key survives parsing, gets iterated, and has its selector resolved — errors and all — and then vanishes when the extractor assigns the result. Parse with Define, build output with Set, and both halves of that behavior fall out. Measured against node, both directions.
func (*Object) Keys ¶
Keys returns the JS property order: canonical array indices first in ascending numeric order, then every other key in insertion order. This is the ordinary-object [[OwnPropertyKeys]] rule, and it is why {b:…, "2":…, "0":…, a:…} serializes as 0, 2, b, a.
func (*Object) MarshalJSON ¶
func (*Object) Set ¶
Set stores a key, preserving first-insertion position on overwrite — JS assignment semantics, where re-assigning an existing property updates the value and leaves the key where it was.
__proto__ is dropped rather than stored. In JS, extract.js builds results with `const result = {}` and assigns `result[key] = …`, so an own "__proto__" key hits the legacy prototype setter instead of creating a property, and the key never appears in the output. A Go map would happily emit it. Matching JS here is the whole point; the upstream fix is to build with Object.create(null), and when that lands this drops out with a corpus change.
type RulesParseError ¶
RulesParseError is returned when rules cannot be parsed, from either face. It mirrors the JS engine's error of the same name.
The message text does NOT match the JS engine's, and cannot: it wraps Go's encoding/json diagnostics rather than V8's. That is deliberate and safe, because htmlclay maps errors to HTTP status by TYPE. The JS hosts map by sniffing message text, which is exactly why two sibling selector failures answer 400 and 500 there. See the error-parity table in the plan.
func (*RulesParseError) Error ¶
func (e *RulesParseError) Error() string
func (*RulesParseError) Unwrap ¶
func (e *RulesParseError) Unwrap() error
type SelectorError ¶
SelectorError is a selector cascadia would not compile.
This is the clearest place htmlclay departs from the JS hosts on STATUS rather than on data. Of the sixteen message classes css-what can produce, twelve carry no word the JS hosts' sniffing recognises, so those answer 500 while their siblings answer 400. Two failures of the same kind getting different statuses is not a contract worth reproducing; every selector failure is 400 here. Recorded as a divergence, not parity.
func (*SelectorError) Error ¶
func (e *SelectorError) Error() string
func (*SelectorError) Unwrap ¶
func (e *SelectorError) Unwrap() error
type SelectorFailure ¶
type SelectorFailure interface {
error
// contains filtered or unexported methods
}
SelectorFailure is implemented by both selector errors, so a caller mapping HTTP status can answer 400 for either without knowing which. The two stay distinct types because they mean different things to a human reading the log: one selector is broken, the other is fine but ambiguous across engines.
type UnknownRulesVersion ¶
type UnknownRulesVersion struct {
Version string
}
UnknownRulesVersion is a rules tag whose data-rules-version is not "1". A missing attribute reports as empty, matching the JS, where undefined !== "1" fails the same way.
func (*UnknownRulesVersion) Error ¶
func (e *UnknownRulesVersion) Error() string
type UnsupportedSelector ¶
UnsupportedSelector is the gate's refusal: a selector cheerio would accept and answer, which htmlclay declines because the two engines do not agree on what it means.
It is deliberately NOT the same type as SelectorError, even though both answer 400. A SelectorError is a selector nobody can run. An UnsupportedSelector is a divergence made loud on purpose: the reference produces data and htmlclay produces an error, which is the safe direction to differ but is still a difference, and it should be countable rather than blended into the generic parse failures. Reason says what actually differs, since "unsupported" alone leaves the author with nothing to act on.
func (*UnsupportedSelector) Error ¶
func (e *UnsupportedSelector) Error() string
type Value ¶
type Value any
Value is anything extraction can produce: nil, string, []Value, or *Object. There is no number or boolean case — the engine reads text and attributes, so every scalar is a string, and a non-string rule extracts to null (see extract.go).
func ParseRelaxed ¶
ParseRelaxed parses the ?data= parameter and script-tag bodies. Strict JSON is tried first and returned as-is when it works; anything else goes through a tokenizer that rewrites the input into JSON and parses that. The tokenizer came from hyperclay's legacy data-extractor.js and its behavior on odd input is load-bearing, not incidental — see the rules-* conformance cases.
func ParseStrict ¶
ParseStrict parses rules as strict JSON: no unquoted keys, no single quotes, no trailing commas. The script-tag face uses ParseRelaxed instead; this exists for callers that want strict semantics, matching the JS engine's public export.
Source Files