twig

twig

A Go HTML scraping library with CSS and XPath selectors, fluent traversal, and schema-based extraction.

Installation

go get github.com/your-org/twig

Quick Start

page, err := gosoup.NewPage(rawHTML, "https://example.com")
if err != nil {
    log.Fatal(err)
}

title, err := page.CSS("h1").Text()
href, err := page.CSS("a.main-link").Href()

Querying

Both Page and Builder support CSS and XPath queries. Queries return a *Builder (single match) or Nodes (multiple matches).

// Single match
b := page.CSS("div.article")
b := page.XPath("//div[@class='article']")

// All matches
ns := page.CSSAll("ul li")
ns := page.XPathAll("//ul/li")

// Scoped to a node
b := page.CSS("div.article").CSS("h2")

XPath note: Use relative expressions (.//...) to scope to the current node. Absolute expressions (//...) always search from the document root.

Builder — Single Node

b.Text()           // inner text (all descendants)
b.HTML()           // outer HTML
b.Attr("data-id")  // attribute value
b.Href()           // resolved href URL
b.Src()            // resolved src URL
b.Tag()            // tag name
b.HasClass("foo")  // bool
b.NodeType()       // html.NodeType

Traversal

b.Parent()
b.Child()       // first element child
b.Child(2)      // nth element child (0-indexed)
b.Next()        // next element sibling
b.Prev()        // previous element sibling
b.Children()    // → Nodes
b.Siblings()    // → Nodes
b.Parents()     // → Nodes
b.NextAll()     // → Nodes
b.PrevAll()     // → Nodes

Nodes — Multiple Nodes

ns.First()         // → *Builder
ns.Last()          // → *Builder
ns.Get(i)          // → *Builder
ns.Len()           // (int, error)
ns.Each(func(i int, b *Builder) { ... })

ns.TextAll()       // []string
ns.AttrAll("href") // []string

// Chain further queries
ns.CSS("span")
ns.XPath(".//span")

// Predicates
ns.Some()                           // any nodes?
ns.None()                           // no nodes?
ns.Any(func(b *Builder) bool { })
ns.Every(func(b *Builder) bool { })

Schema Extraction

Extract structured data from a node or page in one call. Partial results are always returned even when some fields fail.

schema := twig.Schema{
    "title": twig.Field().CSS("h1").Text(),
    "link":  twig.Field().CSS("a").Href(),
    "id":    twig.Field().CSS("article").Attr("data-id"),
}

result, err := page.Extract(schema)
// result["title"], result["link"], result["id"]

Per-item extraction from a list

results, err := page.CSSAll("ul li").ExtractAll(schema)

Handling partial errors

result, err := page.Extract(schema)
if err != nil {
    if ee, ok := err.(*twig.ExtractionError); ok {
        for field, ferr := range ee.Fields {
            log.Printf("field %s failed: %v", field, ferr)
        }
    }
}
// result still contains successfully extracted fields

Errors

Errors propagate lazily through builder chains — you only need to check at the terminal call.

Utilities

// Panics on error — useful in tests and scripts
page := twig.Must(twig.NewPage(rawHTML, ""))
text := twig.Must(page.CSS("h1").Text())

Dependencies

• golang.org/x/net/html
• github.com/andybalholm/cascadia
• github.com/antchfx/xpath