README
¶
GTMLP - Go HTML Parsing Library
A powerful and easy-to-use Go library for parsing HTML documents with XPath selectors and pattern-based extraction.
Why GTMLP?
- Clean Chainable API - Fluent syntax for all operations including pattern-based extraction
- Pattern-Based Extraction - Define what to extract once, apply everywhere
- Data Transformation Pipes - Built-in pipes for cleaning, normalizing, and transforming data
- Production Ready - Random user-agents, retries, timeouts, error handling built-in
- Zero External Dependencies - Pure Go implementation with minimal dependencies
Features
- XPath Selectors - Query HTML documents using XPath 1.0 expressions
- Pattern-Based Extraction - Container patterns for structured data extraction
- Data Transformation Pipes - Built-in pipes (trim, decode, replace, case conversion, etc.)
- Chainable Fluent API - Clean, composable syntax for all operations
- HTML to JSON - Convert HTML documents to structured JSON format
- HTTP Client - Built-in client for fetching and parsing remote URLs
- Random User-Agents - Realistic rotating browser user-agents for anti-detection
- XPath Validation - Validate XPath patterns before scraping
- Alternative Patterns - Fallback patterns for robust extraction
- URL Health Check - Check URL availability concurrently
- XML Parsing - Parse XML documents (sitemaps, RSS feeds, SOAP responses)
Installation
go get github.com/Hanivan/gtmlp
Quick Start
Basic HTML Parsing
package main
import (
"fmt"
"log"
"github.com/Hanivan/gtmlp"
)
func main() {
html := `<html><body><h1>Hello, World!</h1></body></html>`
parser, err := gtmlp.Parse(html)
if err != nil {
log.Fatal(err)
}
h1, _ := parser.XPath("//h1")
fmt.Println(h1.Text()) // Output: Hello, World!
}
Pattern-Based Scraping (Recommended)
package main
import (
"fmt"
"log"
"github.com/Hanivan/gtmlp"
)
func main() {
// Define what to extract
patterns := []gtmlp.PatternField{
gtmlp.NewContainerPattern("container", "//div[@class='product']"),
{
Key: "name",
Patterns: []string{".//h2/text()"},
Pipes: []gtmlp.Pipe{gtmlp.NewTrimPipe()},
},
{
Key: "price",
Patterns: []string{".//span[@class='price']/text()"},
Pipes: []gtmlp.Pipe{gtmlp.NewTrimPipe()},
},
}
// Extract with chainable API
results, err := gtmlp.FromURL("https://example.com/products").
WithPatterns(patterns).
Extract()
if err != nil {
log.Fatal(err)
}
// Use the extracted data
for _, product := range results {
fmt.Printf("%s: %s\n", product["name"], product["price"])
}
}
Fluent API
// Fetch and parse URL
parser, err := gtmlp.ParseURL("https://example.com",
gtmlp.WithTimeout(10*time.Second),
gtmlp.WithUserAgent("MyBot/1.0"),
)
// Chain operations
text, _ := gtmlp.FromHTML(html).
XPath("//p[@class='content']").
Text()
// Convert to JSON
json, _ := gtmlp.New().
FromURL("https://example.com").
ToJSON()
Documentation
- API Reference - Complete API documentation
- Pattern Extraction Guide - Pattern-based extraction and pipes
- Examples - All examples and usage guides
Running Examples
# Basic examples
go run examples/basic/*.go -type=all
# Advanced examples
go run examples/advanced/*.go -type=all
# Specific example
make example-chainable
See docs/EXAMPLES.md for all available examples and commands.
Anti-Detection Features
Random User-Agents are enabled by default. All HTTP requests use realistic, rotating user-agent strings from a comprehensive database of real browsers (Chrome, Firefox, Safari, Edge, etc.).
// Random UA is enabled by default (recommended)
parser, err := gtmlp.ParseURL("https://example.com")
// Use custom user-agent instead
parser, err := gtmlp.ParseURL("https://example.com",
gtmlp.WithUserAgent("CustomBot/1.0"),
)
Benefits:
- Avoid bot detection - Requests appear to come from real browsers
- Reduced blocking - Websites less likely to block your scraper
- Better success rate - More consistent scraping results
- No extra configuration - Works out of the box
Project Structure
gtmlp/
├── internal/ # Internal packages
│ ├── parser/ # Core parsing functionality
│ ├── httpclient/ # HTTP client
│ ├── builder/ # Fluent API builder
│ ├── pipe/ # Data transformation pipes
│ ├── pattern/ # Pattern-based extraction
│ ├── validator/ # XPath validation
│ └── health/ # URL health checking
├── examples/
│ ├── basic/ # Basic examples (7 files)
│ └── advanced/ # Advanced examples (9 files)
├── docs/ # Documentation
├── types.go # Public types and errors
├── options.go # Configuration options
└── gtmlp.go # Root package (public API)
Dependencies
github.com/antchfx/htmlquery- HTML parsing with XPathgithub.com/antchfx/xpath- XPath 1.0 implementationgithub.com/lib4u/fake-useragent- Realistic user-agent stringsgolang.org/x/net- Additional networking libraries
License
MIT License
Contributing
- Fork the repository
- Create your feature branch
- Write tests for new features
- Commit your changes
- Push to the branch
- Create a Pull Request
Acknowledgments
Built with:
- htmlquery - HTML parsing
- fake-useragent - User-agent database
Documentation
¶
Index ¶
- Constants
- func ExtractSingle(p *Parser, field PatternField) (any, error)
- func ExtractWithPatterns(p *Parser, patterns []PatternField) ([]map[string]any, error)
- func FromHTML(html string) *builder.Builder
- func FromURL(url string) *builder.Builder
- func ListPipes() []string
- func New() *builder.Builder
- func RegisterPipe(name string, factory func() Pipe)
- func ToJSON(html string) ([]byte, error)
- func ToJSONWithOptions(html string, opts JSONOptions) ([]byte, error)
- func URLToJSON(url string, parseOpts []Option, jsonOpts JSONOptions) ([]byte, error)
- type Extractor
- type HealthResult
- type JSONOptions
- type MultipleType
- type Option
- func WithDisableRandomUA() Option
- func WithHeaders(h map[string]string) Option
- func WithMaxRetries(maxRetries int) Option
- func WithProxy(proxyURL string) Option
- func WithRandomUserAgent() Option
- func WithSuppressErrors() Option
- func WithTimeout(d time.Duration) Option
- func WithUserAgent(ua string) Option
- type ParseError
- type Parser
- type PatternField
- type PatternMeta
- type Pipe
- func CreatePipe(name string) (Pipe, error)
- func NewDateFormatPipe(format string) Pipe
- func NewDecodePipe() Pipe
- func NewExtractEmailPipe() Pipe
- func NewLowerCasePipe() Pipe
- func NewNumberNormalizePipe() Pipe
- func NewReplacePipe(pattern, with string) Pipe
- func NewTrimPipe() Pipe
- func NewURLResolvePipe(baseURL string) Pipe
- func NewUpperCasePipe() Pipe
- type ReturnType
- type Selection
- type ValidationResult
Constants ¶
const ( // ReturnTypeText returns plain text content. ReturnTypeText = parser.ReturnTypeText // ReturnTypeHTML returns HTML content. ReturnTypeHTML = parser.ReturnTypeHTML )
const ( // MultipleNone returns only the first match. MultipleNone = pattern.MultipleNone // MultipleArray returns all matches as an array. MultipleArray = pattern.MultipleArray // MultipleSpace returns all matches joined with spaces. MultipleSpace = pattern.MultipleSpace // MultipleComma returns all matches joined with commas. MultipleComma = pattern.MultipleComma )
Multiple type constants.
Variables ¶
This section is empty.
Functions ¶
func ExtractSingle ¶
func ExtractSingle(p *Parser, field PatternField) (any, error)
ExtractSingle extracts a single field using a pattern.
func ExtractWithPatterns ¶
func ExtractWithPatterns(p *Parser, patterns []PatternField) ([]map[string]any, error)
ExtractWithPatterns extracts data using pattern fields.
func RegisterPipe ¶
RegisterPipe registers a custom pipe factory.
func ToJSONWithOptions ¶
func ToJSONWithOptions(html string, opts JSONOptions) ([]byte, error)
ToJSONWithOptions converts HTML string to JSON with custom options.
Types ¶
type Extractor ¶
Extractor is an alias for internal pattern.Extractor.
func NewExtractor ¶
NewExtractor creates a new Extractor from a Parser.
type HealthResult ¶
type HealthResult = health.HealthResult
HealthResult is an alias for internal health.HealthResult.
func CheckURLHealth ¶
func CheckURLHealth(urls []string, timeout time.Duration) []HealthResult
CheckURLHealth checks the health of URLs concurrently.
func CheckURLHealthSequential ¶
func CheckURLHealthSequential(urls []string, timeout time.Duration) []HealthResult
CheckURLHealthSequential checks URLs sequentially.
func CheckURLHealthWithGet ¶
func CheckURLHealthWithGet(urls []string, timeout time.Duration) []HealthResult
CheckURLHealthWithGet checks URL health using GET requests.
type JSONOptions ¶
type JSONOptions = parser.JSONOptions
JSONOptions is an alias for internal parser.JSONOptions.
func DefaultJSONOptions ¶
func DefaultJSONOptions() JSONOptions
DefaultJSONOptions returns the default JSON conversion options.
type MultipleType ¶
type MultipleType = pattern.MultipleType
MultipleType is an alias for internal pattern.MultipleType.
type Option ¶
type Option func(*config)
Option is a function that configures parsing behavior.
func WithDisableRandomUA ¶
func WithDisableRandomUA() Option
WithDisableRandomUA disables random user agents and uses static user agent.
func WithHeaders ¶
WithHeaders sets custom HTTP headers.
func WithMaxRetries ¶
WithMaxRetries sets the maximum number of retries for failed HTTP requests.
func WithRandomUserAgent ¶
func WithRandomUserAgent() Option
WithRandomUserAgent explicitly enables random user agents (enabled by default).
func WithSuppressErrors ¶
func WithSuppressErrors() Option
WithSuppressErrors enables error suppression for XPath queries. When enabled, XPath errors return nil instead of error values.
func WithTimeout ¶
WithTimeout sets the HTTP request timeout.
type ParseError ¶
ParseError represents an error that occurred during parsing.
func NewParseError ¶
func NewParseError(message string, err error) *ParseError
NewParseError creates a new ParseError.
func (*ParseError) Unwrap ¶
func (e *ParseError) Unwrap() error
Unwrap returns the underlying error.
type Parser ¶
Parser is an alias for internal parser.Parser.
type PatternField ¶
type PatternField = pattern.PatternField
PatternField is an alias for internal pattern.PatternField.
func NewContainerPattern ¶
func NewContainerPattern(key string, xpath string) PatternField
NewContainerPattern creates a new container PatternField.
func NewPatternField ¶
func NewPatternField(key string, xpath string) PatternField
NewPatternField creates a new PatternField.
func NewPatternFieldWithHTML ¶
func NewPatternFieldWithHTML(key string, xpath string) PatternField
NewPatternFieldWithHTML creates a new PatternField that returns HTML content.
func NewPatternFieldWithMultiple ¶
func NewPatternFieldWithMultiple(key string, xpath string, multiple MultipleType) PatternField
NewPatternFieldWithMultiple creates a new PatternField with multiple value handling.
type PatternMeta ¶
type PatternMeta = pattern.PatternMeta
PatternMeta is an alias for internal pattern.PatternMeta.
func DefaultPatternMeta ¶
func DefaultPatternMeta() *PatternMeta
DefaultPatternMeta returns default pattern metadata.
type Pipe ¶
Pipe is an alias for internal pipe.Pipe.
func NewDateFormatPipe ¶
NewDateFormatPipe creates a new DateFormatPipe.
func NewExtractEmailPipe ¶
func NewExtractEmailPipe() Pipe
NewExtractEmailPipe creates a new ExtractEmailPipe.
func NewNumberNormalizePipe ¶
func NewNumberNormalizePipe() Pipe
NewNumberNormalizePipe creates a new NumberNormalizePipe.
func NewReplacePipe ¶
NewReplacePipe creates a new ReplacePipe.
func NewURLResolvePipe ¶
NewURLResolvePipe creates a new URLResolvePipe.
type ReturnType ¶
type ReturnType = parser.ReturnType
ReturnType is an alias for internal parser.ReturnType.
type ValidationResult ¶
type ValidationResult = validator.ValidationResult
ValidationResult is an alias for internal validator.ValidationResult.
func ValidateXPath ¶
func ValidateXPath(html string, xpaths []string, suppressErrors bool) []ValidationResult
ValidateXPath validates XPath expressions against HTML.
func ValidateXPathWithParser ¶
func ValidateXPathWithParser(p *Parser, xpaths []string) []ValidationResult
ValidateXPathWithParser validates XPath using an existing Parser.
Source Files
Directories