quarry

quarry

Deterministic, declarative, drift-resilient extraction of structured data from HTML for Go: define the fields you want as configuration, get typed data out, and be told when the page's markup changes instead of silently getting empty strings.

type Article struct {
    Title  string   `quarry:"h1 || [property='og:title']@content"`
    Author string   `quarry:".byline .author,optional"`
    Body   string   `quarry:"article .content@html"`
    Tags   []string `quarry:".tags a"`
}

var a Article
err := quarry.Extract(htmlBytes, &a)   // an error NAMES the field that failed

go get github.com/ophymx/quarry

Why

A hand-rolled doc.Find(".price").Text() returns "" both when the price is genuinely absent and when the site renamed the class last night — and you find out weeks later, from bad data. Across every scraping ecosystem (we surveyed ~30 tools), that silent failure is the default. quarry makes "the selector stopped matching" a first-class, detectable event:

• Fields are required by default. A required field whose selectors all fail is a machine-readable *FieldError — field path, spec, and per-alternative match counts — not a zero value.
• Selectors carry ordered fallbacks. || chains degrade gracefully under markup churn, and quarry records which alternative fired.
• Drift is observable before it breaks. Schema.Check probes a page without extracting; Aggregate folds reports across pages into coverage. "The primary selector died, a fallback is carrying the field" is a report line, not an outage.
• Fixes are config, not releases. The same grammar drives struct tags and runtime-loaded YAML schemas; Schema.Merge applies a drift patch at runtime.

The selector grammar

A field spec is one or more ||-separated alternatives; the first alternative that produces a non-empty value wins. Each alternative is a CSS selector, optionally suffixed with @ and a source:

Notes that matter:

• An element that merely exists does not win: empty text, an absent or empty attribute, whitespace-only markup — all fall through to the next alternative. <meta property="og:title" content=""> cannot pin an empty string.
• || exists because CSS's own comma is document order, not preference order — "h1, .title" can't say "prefer the og tag".
• Hashed CSS-module classes (_item_167zw_4): use the standard substring selector, [class*='_item'].
• @text, @html, @outerHtml are reserved words; attribute names are matched case-insensitively.
• Alternatives are evaluated in order, at most once per extraction, and extraction short-circuits on the first win.
• An alternative can be a JSON-LD path instead of CSS: jsonld:headline, jsonld:author.name (array steps map across elements), jsonld:@id (@ is part of the key). Evaluated against the application/ld+json blocks in the current scope; numbers come out exactly as written in the JSON; null, objects and empty strings fall through like an empty CSS match.

Slices take every non-empty value; struct, *struct and []struct fields scope their inner tags to matching containers:

type Result struct {
    Title string `quarry:"a.title"`
    URL   string `quarry:"a.title@href"`
}
type Listing struct {
    Results []Result `quarry:"[class*='result-item']"`  // one per container
}

Required means required: a required slice that matches nothing is an error. ,optional is the only optionality spelling. quarry.Lenient() relaxes everything for best-effort runs — the only silent mode, and you have to ask for it.

Two more tag options sharpen the win condition, because shape and count are stronger drift signals than presence:

Price string   `quarry:".price,match='^\\$\\d+\\.\\d{2}$'"` // "Sign in to see price" ≠ a price
Rows  []Result `quarry:".result-row,min=30"`                // 2 rows ≠ a listing page

,match=<re> drops mis-shaped values exactly like empty ones (Go regexp, unanchored — quote a regex containing commas); ,min=N makes an alternative win only with ≥ N usable values (containers: N matching containers). Both compose with || — a drifted primary falls through to the next rung, and a required field with no rung left fails loud. Options come after the whole spec, in any order.

Dynamic schemas: ship a patch, not a release

The same grammar loads from YAML (or JSON) at runtime:

$expect: "#app .product-page"        # page-identity precondition
title: "h1 || [property='og:title']@content"
price: "meta[itemprop=price]@content || .price"
results:
  selector: "[class*='result-item']"
  list: true
  children:
    title: "a.title"
    url: "a.title@href"

schema, _ := quarry.LoadSchema(yamlBytes)
data, err := schema.Extract(htmlBytes)   // map[string]any

When the site moves, the fix is a config overlay:

patch, _ := quarry.LoadSchema([]byte(`price: ".pricing-v2 .amount"`))
data, err = schema.Merge(patch).Extract(htmlBytes)

Validate a patch the way self-healing systems do: run the merged schema over archived HTML and compare against known-good output (quarry's own test suite does exactly this against real committed pages).

$expect distinguishes "this is a login wall / captcha / error page" (*ExpectError, no field noise) from "a field drifted" (*FieldError).

Drift monitoring

report := schema.Check(htmlBytes)     // probe, don't extract
report.OK()                           // all required fields match
report.Missing()                      // which don't
report.Degraded()                     // matching, but on a fallback — fix it
                                      // before the fallback dies too

cov := quarry.Aggregate(reports...)   // across many pages:
// per field: MatchRate(), Degraded count, WonBy (which alternative wins
// where), min/max/total match counts (a list going 30 → 3 is drift even
// though it "matched")

Check is exhaustive where extraction short-circuits: every alternative is censused, so a dead fallback behind a healthy primary is visible too. Thresholds and alerting are yours — a field legitimately absent on some pages is a coverage number, not a hardcoded policy.

Embedded structured data

Often the most drift-resistant data is what the site publishes for SEO — sites break CSS classes weekly and their schema.org markup almost never. JSON-LD is wired straight into the selector grammar as a fallback rung (".headline || jsonld:headline"); the structured subpackage gives you the full documents when you want more than single values:

og, _    := structured.OpenGraph(htmlBytes) // map[string][]string, repeated og:image kept
items, _ := structured.JSONLD(htmlBytes)    // []map[string]any; bad blocks skipped & reported
md, _    := structured.Microdata(htmlBytes) // itemscope/itemprop tree

Pure functions, stdlib-only, no URL resolution. (No Go equivalent of Python's extruct existed; this is it, minus the syntaxes the modern web abandoned.)

Fetching (optional)

The core never touches the network. fetch is a polite single-page GET for the batteries-included case — cross-goroutine rate limiting, bounded retries with backoff, Retry-After respect:

c := fetch.New(fetch.WithMinInterval(time.Second), fetch.WithRetries(3))
htmlBytes, err := c.Get(ctx, url)

Not a crawler. If you need frontier management, use a crawler and feed quarry the HTML.

The pipeline

quarry is the extraction stage of a text-corpus pipeline; it hands raw strings downstream, deliberately un-normalized:

htmlBytes, _ := c.Get(ctx, url)     // fetch     (quarry/fetch)
var a Article
_ = quarry.Extract(htmlBytes, &a)   // extract   (quarry)
clean := normalize(a.Body)          // normalize (your text normalizer)
sig := sketcher.Sketch(clean)       // dedup     (github.com/ophymx/semblance)

Writing drift-resistant selectors

Distilled from the wrapper-maintenance literature (Robula+, SIGMOD'09, VLDB'11) and a survey of what actually breaks:

1. Anchor on meaning, not styling. id, itemprop, data-*, [property='og:*'] outlive class names by years; classes tied to a design system die with the next redesign.
2. Never position. :nth-child(3), deep descendant chains, and anything that encodes today's layout are the most fragile selectors you can write.
3. Shortest discriminating selector wins. Every extra step is another thing that can change.
4. Make alternatives independent. A fallback chain of three spellings of the same class shares one failure mode; anchor each rung on a different page mechanism — a semantic attribute || a styling class || a jsonld: path. Embedded structured data is usually the sturdiest rung of all, and it sits right in the grammar.
5. Watch Degraded(). A field running on its fallback is a field one selector away from an outage. Fallback depth is a fragility score.

Determinism & stability

Extraction is a pure function of (HTML bytes, schema): no network, no clock, no locale, no randomness. Malformed markup is repaired by golang.org/x/net/html's HTML5 recovery, deterministic per module version (pinned in go.mod); well-formed input is unconditionally stable.

The selector grammar and struct-tag semantics are frozen. Two regions are pre-reserved so the grammar never needs a breaking change: scheme-prefixed alternatives and ,key=value tag options — errors today, features some day. (jsonld: is the first activated scheme, and ,match=/,min= the first activated options — schemas using them are loudly rejected, never misread, by older quarry versions; other schemes and option keys remain reserved.) Golden tests against committed real pages (HN, Wikipedia, go.dev) pin behavior; deviations from the original design are recorded in docs/design-notes.md.

What quarry is not

• Not a DOM library. goquery gives you a jQuery; quarry takes a declaration. (And unlike parsel's ::text/::attr() pseudo-elements, quarry's @source suffix keeps the selector standard CSS you can test in a browser console.)
• Not a crawler (Colly/Geziyor), not a query language runtime (Ferret), not a headless browser — pages needing JavaScript need a rendering step upstream.
• Not normalization — raw strings out; normalize downstream.
• Not heuristic extraction — quarry is the precise complement to trafilatura-style automatic content extractors.
• Among Go struct-tag binders: pagser has transform pipes but fails silently; goq binds but has no fallbacks, no required fields, no drift report. Fail-loud + fallbacks + check is the point.

License

MIT