README
¶
h2m
Turn raw HTML into clean, readable Markdown.
h2m strips the boilerplate from a web page (navigation, sidebars, ads,
footers, cookie banners) and converts what is left into GitHub-flavored
Markdown: headings, paragraphs, lists, tables, code blocks, links, and images.
Relative links and image sources are resolved to absolute URLs.
It is built for bulk web-to-text work where extraction quality matters: feeding crawl data into a search index, building a pretraining corpus, or archiving pages as readable text.
How it works
Two stages:
- Extract. go-trafilatura
tuned for recall (
FavorRecallplus the Readability and DomDistiller fallbacks) isolates the main content node and drops boilerplate. This keeps far more real pages than a Readability-only pass, which is the whole point: product pages, docs, forum threads, and long-form articles all survive. - Render. A direct node-tree walk emits Markdown without a second parse. Tables become GFM tables, code blocks keep their language hint where the source page exposed one, and links resolve against the page URL.
Install
go get github.com/tamnd/h2m
Usage
package main
import (
"fmt"
"github.com/tamnd/h2m"
)
func main() {
res := h2m.Convert(htmlBytes, "https://example.com/post")
if res.HasContent {
fmt.Println(res.Markdown)
}
}
Convert does no network I/O. Pass the page URL only so relative links and
images can be resolved to absolute URLs. It is safe to call from many
goroutines at once.
Result
type Result struct {
Markdown string // the converted Markdown
Title string // page title from metadata
Language string // detected language, when available
HasContent bool // true when an article was extracted
HTMLSize int // input bytes
MarkdownSize int // output bytes
HTMLTokens int // rough token estimate of the input
MarkdownTokens int // rough token estimate of the output
ConvertMs int // wall time for this conversion
Error string // set when HasContent is false
}
Fast path
ConvertFast skips trafilatura and uses go-readability alone. It is several
times faster but extracts fewer pages and less of each page. Use it when raw
throughput matters more than catching every page.
res := h2m.ConvertFast(htmlBytes, pageURL)
License
MIT. See LICENSE. Extraction is powered by go-trafilatura (Apache-2.0) and go-readability; see NOTICE.
Documentation
¶
Overview ¶
Package h2m converts raw HTML pages into clean, readable Markdown.
The pipeline is two stages: extraction then rendering. Extraction uses go-trafilatura tuned for recall (FavorRecall plus the Readability and DomDistiller fallbacks), so boilerplate like navigation, sidebars, ads, and footers is stripped while the main article body, tables, code blocks, links, and images are kept. Rendering walks the extracted node tree directly and emits GitHub-flavored Markdown with relative links resolved to absolute URLs.
The main entry point is Convert:
res := h2m.Convert(htmlBytes, "https://example.com/post")
if res.HasContent {
fmt.Println(res.Markdown)
}
Convert is safe for concurrent use. It does no network I/O: pass the page URL only so relative links and images can be resolved.
For a faster, lower-recall path that skips trafilatura and uses go-readability alone, use ConvertFast. It trades extraction quality for throughput and suits bulk jobs where occasional missed pages are acceptable.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CleanMarkdown ¶
CleanMarkdown is the exported version of cleanMarkdown for post-processing stored files.
func EstimateTokens ¶
EstimateTokens approximates token count: ~4 bytes per token for English text.
Types ¶
type ConvertOptions ¶
type ConvertOptions struct {
IncludeImages bool
// FullFallback forces trafilatura's external fallback (go-readability plus
// go-domdistiller) to run on every page. The default (false) runs it only
// when the primary extraction is shorter than defaultFallbackMinTextSize,
// which is much faster on bulk corpora at a small recall cost on borderline
// pages. Set it when conversion quality on short pages matters more than
// throughput.
FullFallback bool
}
type Result ¶
type Result struct {
Markdown string
Title string
Language string
HasContent bool // trafilatura found main content
HTMLSize int
MarkdownSize int
HTMLTokens int
MarkdownTokens int
ConvertMs int
Error string
}
Result holds the output of a single HTML → Markdown conversion.
func Convert ¶
Convert extracts readable content from raw HTML and converts it to Markdown. The pageURL is used for resolving relative links; it may be empty.
func ConvertFast ¶
ConvertFast extracts content using go-readability (a Mozilla Readability.js port) and converts to Markdown. It is 3-8x faster than Convert at the cost of lower extraction quality on noisy pages. Reach for it in bulk jobs where throughput matters more than edge-case accuracy.
func ConvertWithOptions ¶
func ConvertWithOptions(rawHTML []byte, pageURL string, convertOpts ConvertOptions) Result
Source Files