readability

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

readability

Go Reference

readability extracts the main article and its metadata from HTML. It is an idiomatic Go port of Mozilla Readability.

The package can return these values:

  • Processed article HTML
  • Plain article text
  • The article title, author, excerpt, site name, language, text direction, and publication time
  • A parsed html.Node for the article

[!WARNING] Article.Content and Article.Node can contain unsafe HTML. The package does not sanitize these values. Sanitize the HTML before you add it to a web page.

Requirements

  • Go 1.23 or a later version

Installation

go get github.com/ryanfowler/readability

Quick start

Pass an HTML reader and the page URL to Parse:

package main

import (
    "fmt"
    "log"
    "strings"

    "github.com/ryanfowler/readability"
)

func main() {
    source := `<html>
<head><title>Example article</title></head>
<body><article><p>This is the article text.</p></article></body>
</html>`

    article, err := readability.Parse(strings.NewReader(source), "https://example.com/news/1")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(article.Title)
    fmt.Println(article.TextContent)
}

Use the source page URL when it is available. The package uses this URL and the document <base> element to resolve relative links and media URLs. The page URL can be empty. A nonempty page URL must be an absolute HTTP or HTTPS URL with a host.

Calling Parse without functional options selects the default settings.

Check a page before extraction

IsProbablyReaderable applies a fast heuristic. It does not extract the article. Use it when you only need to know if a page is likely to contain an article:

if readability.IsProbablyReaderable(source) {
    article, err := readability.Parse(strings.NewReader(source), pageURL)
    if err != nil {
        log.Printf("extraction failed: %v", err)
        return
    }
    fmt.Println(article.Title)
}

A true result is not a guarantee that extraction will succeed. A false result does not mean that the HTML is invalid.

Use a parsed HTML tree

Use ParseNode if you already parsed the HTML with golang.org/x/net/html. This prevents a second parse of the source:

doc, err := html.Parse(strings.NewReader(source))
if err != nil {
    // Handle the error.
}

if readability.IsProbablyReaderableNode(doc) {
    article, err := readability.ParseNode(doc, pageURL)
    if err != nil {
        log.Printf("extraction failed: %v", err)
        return
    }
    fmt.Println(article.Title)
}

The node functions accept a complete document or a tree that has a body root. They do not change the supplied tree. You can use the same tree in later calls. Do not change the tree while either node function uses it.

Configure extraction

Pass functional options after the page URL. Unspecified settings retain their defaults:

article, err := readability.Parse(
    strings.NewReader(source),
    pageURL,
    readability.WithCharThreshold(100),
    readability.WithMaxElemsToParse(50_000),
)

Options are reusable and are applied from left to right. If a setting appears more than once, the last option wins.

Functional option Default Function
WithMaxElemsToParse(0) 0 Sets the maximum number of HTML elements. Zero removes the limit.
WithNbTopCandidates(5) 5 Sets the number of top article candidates to compare.
WithCharThreshold(500) 500 Retries extraction when the result is shorter than this value. Zero prevents retries.
WithClassesToPreserve("page") []string{"page"} Lists the CSS classes to retain when class cleanup is active.
WithKeepClasses(false) false Retains all CSS classes when true.
WithDisableJSONLD(false) false Prevents metadata extraction from JSON-LD when true.
WithAllowedVideoRegex(pattern) Built-in allowlist Identifies video URLs that cleanup can retain. A nil pattern selects the built-in allowlist.
WithLinkDensityModifier(0) 0 Changes the link-density limits that remove a candidate.
WithLogger(logger) nil Receives extraction log records. Nil turns logs off.
WithDebug(false) false Adds verbose debug data to log records when true.

WithCharThreshold controls extraction retries. If a result is too short, the package retries with less strict removal and cleanup rules. If all results are too short, the package returns the longest nonempty result.

Pass WithLogger a *slog.Logger to receive logs. The logger handler controls the log level and output. The package does not use the global slog logger.

The heuristic has separate options:

likely := readability.IsProbablyReaderable(
    source,
    readability.WithMinContentLength(200),
    readability.WithMinScore(20),
)

Character counts

These values use UTF-16 code units:

  • Article.Length
  • The value passed to WithCharThreshold
  • The value passed to WithMinContentLength

This rule matches JavaScript String.length and Mozilla Readability. Most characters count as one unit. A character outside the Basic Multilingual Plane, such as many emoji, counts as two units.

Handle errors

Errors encountered while reading the input are returned directly. Use errors.Is with the package error values:

  • ErrNoBody: the input does not contain a required body element.
  • ErrInvalidURL: the page URL is not valid.
  • ErrNoContent: extraction did not produce article content.

Use errors.As to inspect an element-limit error:

_, err := readability.Parse(
    strings.NewReader(source),
    pageURL,
    readability.WithMaxElemsToParse(50_000),
)
if err != nil {
    var limitErr *readability.TooManyElementsError
    switch {
    case errors.As(err, &limitErr):
        log.Printf("document has %d elements; limit is %d", limitErr.Count, limitErr.Max)
    case errors.Is(err, readability.ErrNoContent):
        log.Print("no article content found")
    default:
        log.Printf("extraction failed: %v", err)
    }
}

Output safety

Treat all extracted data as untrusted input.

  • Sanitize Article.Content before you add it to a web page.
  • Sanitize or validate Article.Node before you render it.
  • Escape plain-text metadata for its output context.
  • Pass WithMaxElemsToParse when you process HTML from an untrusted source and need an element limit.

Compatibility

This package tracks Mozilla Readability.js at commit ab4027a8b37669745016869a37a504727992b2ba. The repository pins Mozilla's official 130-case test corpus in tests/readability-js.

Go parses and serializes the HTML instead of a browser DOM. As a result, attribute order and other HTML serialization details can differ.

License

The project is licensed under the Apache License 2.0; see NOTICE for Mozilla Readability and Arc90 Readability attribution. The pinned Mozilla test fixtures retain their original license.

Documentation

Overview

Package readability extracts the main article and its metadata from HTML.

Use Parse for HTML from an io.Reader. Use ParseNode for a tree parsed with golang.org/x/net/html. Use IsProbablyReaderable or IsProbablyReaderableNode when you only need a fast readerability check.

Parse and the readerability functions use defaults when called without options. Pass functional options to change selected settings.

Article.Content and Article.Node can contain unsafe HTML. The package does not sanitize them. Sanitize extracted HTML before you add it to a web page.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNoContent means that extraction did not produce article content.
	ErrNoContent = engine.ErrNoContent
	// ErrNoBody means that the supplied HTML tree does not have a body element.
	ErrNoBody = engine.ErrNoBody
	// ErrInvalidURL means that the nonempty page URL is not an absolute HTTP or
	// HTTPS URL with a host.
	ErrInvalidURL = engine.ErrInvalidURL
)

Functions

func IsProbablyReaderable 

func IsProbablyReaderable(input string, opts ...ReaderableOption) bool

IsProbablyReaderable reports whether input is likely to contain an article. It applies a fast heuristic and does not extract the article.

Example 
package main

import (
	"fmt"

	"github.com/ryanfowler/readability"
)

func main() {
	source := `<article><p>short</p></article>`
	fmt.Println(readability.IsProbablyReaderable(source))
}

Output:
false

func IsProbablyReaderableNode 

func IsProbablyReaderableNode(root *html.Node, opts ...ReaderableOption) bool

IsProbablyReaderableNode reports whether a parsed HTML tree is likely to contain an article. It does not change root.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/readability"
	"golang.org/x/net/html"
)

func main() {
	source := `<article><p>An article body with useful prose.</p></article>`
	document, err := html.Parse(strings.NewReader(source))
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(readability.IsProbablyReaderableNode(document))
}

Output:
false

Types

type Article 

type Article struct {
	// Title is the article title.
	Title string `json:"title"`
	// Byline identifies the article author.
	Byline string `json:"byline"`
	// Dir is the text direction, such as "ltr" or "rtl".
	Dir string `json:"dir"`
	// Lang is the article language from the document metadata.
	Lang string `json:"lang"`
	// Content is the processed inner HTML of Node. It is not sanitized.
	Content string `json:"content"`
	// Node is the processed article element. Its inner HTML is Content when the
	// package returns the Article. Node is not included in JSON because an
	// html.Node contains cyclic links.
	Node *html.Node `json:"-"`
	// TextContent is the article text. It uses one space for each normal
	// whitespace sequence. It retains whitespace in preformatted elements.
	TextContent string `json:"textContent"`
	// Length is the length of TextContent in UTF-16 code units.
	Length int `json:"length"`
	// Excerpt is the article description or a short extract from the content.
	Excerpt string `json:"excerpt"`
	// SiteName is the name of the source site.
	SiteName string `json:"siteName"`
	// PublishedTime is the publication time from the document metadata. The
	// package does not change its source format.
	PublishedTime string `json:"publishedTime"`
}

Article contains the extracted article and its metadata.

Content and Node can contain unsafe HTML. Sanitize them before you add them to a web page. Metadata fields are empty when the source does not supply a value.

func Parse 

func Parse(input io.Reader, pageURL string, opts ...Option) (*Article, error)

Parse reads HTML from input and extracts an article.

pageURL can be empty. If it is not empty, it must be an absolute HTTP or HTTPS URL with a host. Parse uses pageURL and the document base URL to resolve relative links and media URLs.

With no options, Parse uses the Mozilla defaults. Parse returns input read errors directly. Other errors support errors.Is with ErrNoBody, ErrInvalidURL, or ErrNoContent. Parse can also return a *TooManyElementsError.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/readability"
)

func main() {
	const source = `<html>
<head><title>Hello</title></head>
<body><article><h1>Hello</h1><p>This is a useful article paragraph.</p></article></body>
</html>`

	article, err := readability.Parse(
		strings.NewReader(source),
		"https://example.com/",
		readability.WithCharThreshold(0),
	)
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(article.Title)
}

Output:
Hello

func ParseNode 

func ParseNode(root *html.Node, pageURL string, opts ...Option) (*Article, error)

ParseNode extracts an article from a parsed HTML tree.

root can be a complete document or a tree with a body root. ParseNode does not change root. The caller must not change root while ParseNode uses it. pageURL can be empty; a nonempty value must be an absolute HTTP or HTTPS URL with a host. With no options, ParseNode uses the Mozilla defaults.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/readability"
	"golang.org/x/net/html"
)

func main() {
	const source = `<html>
<head><title>News</title></head>
<body><article><p>An article body with useful prose.</p></article></body>
</html>`

	document, err := html.Parse(strings.NewReader(source))
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	article, err := readability.ParseNode(
		document,
		"https://example.com/news",
		readability.WithCharThreshold(0),
	)
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(article.Title)
}

Output:
News

type Option added in v0.1.1 

type Option func(*options)

Option configures article extraction. Options are applied in order, so a later option overrides an earlier option for the same setting.

func WithAllowedVideoRegex added in v0.1.1 

func WithAllowedVideoRegex(pattern *regexp.Regexp) Option

WithAllowedVideoRegex sets the pattern used to identify video URLs that cleanup can retain. A nil pattern selects the built-in allowlist.

func WithCharThreshold added in v0.1.1 

func WithCharThreshold(threshold int) Option

WithCharThreshold sets the minimum result length in UTF-16 code units. The package retries extraction with less strict cleanup when a result is shorter. Zero prevents retries.

func WithClassesToPreserve added in v0.1.1 

func WithClassesToPreserve(classes ...string) Option

WithClassesToPreserve sets the CSS classes retained during class cleanup. It has no effect when WithKeepClasses(true) is also used.

func WithDebug added in v0.1.1 

func WithDebug(debug bool) Option

WithDebug controls additional verbose log records. A non-nil logger is required to receive these records.

func WithDisableJSONLD added in v0.1.1 

func WithDisableJSONLD(disable bool) Option

WithDisableJSONLD controls whether metadata extraction from JSON-LD is disabled.

func WithKeepClasses added in v0.1.1 

func WithKeepClasses(keep bool) Option

WithKeepClasses controls whether all CSS classes are retained.

func WithLinkDensityModifier added in v0.1.1 

func WithLinkDensityModifier(modifier float64) Option

WithLinkDensityModifier changes the link-density limits used by cleanup rules to remove a candidate.

func WithLogger added in v0.1.1 

func WithLogger(logger *slog.Logger) Option

WithLogger sets the logger that receives extraction records. A nil logger turns logging off. The package does not use the global slog logger.

func WithMaxElemsToParse added in v0.1.1 

func WithMaxElemsToParse(max int) Option

WithMaxElemsToParse sets the maximum number of HTML elements accepted during extraction. Zero removes the limit.

func WithNbTopCandidates added in v0.1.1 

func WithNbTopCandidates(count int) Option

WithNbTopCandidates sets the number of top article candidates to compare.

type ReaderableOption added in v0.1.1 

type ReaderableOption func(*readerableOptions)

ReaderableOption configures the fast readerability heuristic. Options are applied in order.

func WithMinContentLength added in v0.1.1 

func WithMinContentLength(length int) ReaderableOption

WithMinContentLength sets the minimum candidate length in UTF-16 code units.

func WithMinScore added in v0.1.1 

func WithMinScore(score float64) ReaderableOption

WithMinScore sets the score that a document must exceed to be considered readerable.

type TooManyElementsError 

type TooManyElementsError struct {
	// Count is the number of elements in the document.
	Count int
	// Max is the configured maximum number of elements.
	Max int
}

TooManyElementsError reports that a document exceeds the limit set by WithMaxElemsToParse.

func (*TooManyElementsError) Error 

func (e *TooManyElementsError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
internal
engine
Package engine contains readability's internal extraction implementation.
 Package engine contains readability's internal extraction implementation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.