wet

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 17, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package wet reads WET files, the Common Crawl archive of extracted plain text. A WET file is a WARC whose conversion records hold the readable text of each page, so this package decodes those records on top of pkg/warc.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Iterate 

func Iterate(r io.Reader, crawlID string, fn func(Record) error) error

Iterate reads a WET file (WARC conversion records holding plain text) and calls fn for each record.

Types

type Record 

type Record struct {
	RecordID        string
	CrawlID         string
	URL             string
	Date            time.Time
	ContentLanguage string
	Text            string
}

Record is extracted plain text for one page.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.