wat

package

v0.2.0 Latest Latest Go to latest Published: Jun 17, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/tamnd/ccrawl-cli

Links

Open Source Insights

Documentation ¶

Overview ¶

Package wat reads WAT files, the Common Crawl archive of per-page metadata: the response status and content type, the HTML title and meta tags, and the outbound links. A WAT file is a WARC whose metadata records carry a JSON envelope, so this package decodes that envelope on top of pkg/warc.

Index ¶

func Iterate(r io.Reader, crawlID string, fn func(Record) error) error
type Link
type Meta
type Record

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func Iterate ¶

func Iterate(r io.Reader, crawlID string, fn func(Record) error) error

Iterate reads a WAT file and calls fn for each parsed record.

Types ¶

type Link ¶

type Link struct {
	Path  string `json:"path"`
	URL   string `json:"url"`
	Text  string `json:"text,omitempty"`
	Title string `json:"title,omitempty"`
	Alt   string `json:"alt,omitempty"`
}

Link is a hyperlink extracted from page HTML.

type Meta ¶

type Meta struct {
	Name     string `json:"name,omitempty"`
	Property string `json:"property,omitempty"`
	Content  string `json:"content,omitempty"`
	Charset  string `json:"charset,omitempty"`
}

Meta is a <meta> tag extracted from page HTML.

type Record ¶

type Record struct {
	RecordID    string
	CrawlID     string
	URL         string
	Date        time.Time
	HTTPStatus  int
	ContentType string
	Title       string
	Links       []Link
	LinksCount  int
	Metas       []Meta
	WARCFile    string
	WARCOffset  int64
	WARCLength  int64
}

Record is metadata extracted by Common Crawl from a single page.

Source Files ¶

View all Source files

wat.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL