engine

package module
v0.3.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 10, 2026 License: BSD-3-Clause Imports: 30 Imported by: 0

README

go-webengine/engine

go-webengine / engine

CI coverage Go Reference Docs

A pure-Go, CGO_ENABLED=0 headless web engine: it fetches a URL, parses the HTML into a DOM, applies a real CSS subset (cascade + inheritance + var() + modern colour + dark-mode + gradients), runs the page's JavaScript against a real DOM binding, lays the content out with a full box model (block, inline, float, flexbox, CSS grid, tables and position), and paints anti-aliased text, backgrounds, gradients, box-shadows, images and SVG to an image.RGBAno Chromium, no cgo, no host web view. Give it a URL, get back an image of the page.

It is the rendering core of the browserproxy remote-browser service and the wasmdesk in-desktop browser: a server renders pages and streams frames (plus a click hit-map) to a thin client.

Quickstart

import (
    "context"
    "image"
    "os"

    "github.com/go-webengine/engine"
)

func main() {
    ctx := context.Background()

    // Render a live page to PNG bytes.
    png, err := engine.Screenshot(ctx, "https://example.com/", image.Rect(0, 0, 1024, 768))
    if err != nil {
        panic(err)
    }
    _ = os.WriteFile("out.png", png, 0o644)

    // Or get the raw image plus metadata (title, final URL, full content height).
    img, info, err := engine.Render(ctx, "https://example.com/", image.Rect(0, 0, 1024, 768))
    _ = img
    _ = info // info.Title, info.URL, info.ContentHeight
    _ = err
}

For more control use an *engine.Engine (from engine.New()): it adds RenderHTML (render a local HTML string offline), RenderWithLinks / RenderDocumentWithLinks (image plus a []Link hit-map for click-to-navigate), and a DisableJS field to render the static, no-JavaScript document.

The viewport width is fixed; the height grows to fit the whole page (at least the viewport height). There is also a CLI:

go run ./cmd/render -url https://example.com/ -out out.png -w 1024 -h 768
# or render a local file offline:
go run ./cmd/render -file page.html -base https://example.com/ -out out.png

Pipeline

Fetch (go-browserhttp)  →  Parse (x/net/html → dom)  →  Cascade (css: +var()/@media/dark-mode)
     →  JavaScript (js: goja + real DOM + fetch/XHR)
     →  Layout: block · inline · float · flex · grid · table · position (layout)
     ⟲   settle loop: JS reads laid-out metrics → re-cascade + re-layout to a bounded fixpoint
     →  Paint: AA text/SVG/gradients/shadows (paint) + backgrounds (go-widgets/painter) + images (go-images)
     →  image.RGBA  →  PNG (+ optional link hit-map)
Package Role
dom Owned DOM node tree built from golang.org/x/net/html.
css Real CSS subset: value model, stylesheet/declaration parser, tag/class/id + descendant/child/sibling combinators + :checked/:not() selectors with specificity, var() custom properties, @media width queries, modern colour (rgb()/hsl()), dark-mode, UA stylesheet, cascade + inheritance.
layout Full box model — block-and-inline flow, floats + clear, flexbox, CSS grid, tables, position (relative/absolute/fixed/sticky), margin collapsing, greedy word-wrap — driven by a Measurer interface (font-free, exactly testable).
js JavaScript execution via goja bound to a minimal real DOM, with fetch()/XHR and laid-out-geometry read-back (getBoundingClientRect, offset*, getComputedStyle).
paint Rasterises the box tree to *image.RGBA — AA text (real bold + italic), gradients, border-radius, box-shadow, opacity, images and SVG; also the real Measurer (go-opentype faces).
engine (root) Fetch, Render, Screenshot, RenderHTML, RenderWithLinks, the settle-then-render loop, image + SVG loading, and the anchor hit-map.
cmd/render CLI: render -url URL -out shot.png -w 1024 -h 768 (or -file page.html).

Everything reused is pure-Go and BSD/MIT — see SURVEY.md for the prior-art verdict (opossum/mycel studied, not built on) and the full reuse-vs-build decision.

What works / What doesn't

The full per-feature and per-page assessment (five live pages, committed golden PNGs, measured vs headless Chrome) is in FIDELITY.md and bench/REPORT.md. Short version:

Works today

  • HTML → DOM → full box-model layout at a real viewport width: block/inline flow, floats + clear, flexbox, CSS grid, tables, position (relative/absolute/fixed/sticky), margin collapsing, greedy word-wrap.
  • CSS cascade with specificity (inline > id > class > tag) and inheritance; var() custom properties; @media width queries; dark-mode (prefers-color-scheme); external <link> stylesheet fetch; UA defaults.
  • Selectors: tag/class/id/compound, descendant + child + sibling (~/+) combinators, :checked and :not() (the checkbox-hack that collapses MediaWiki dropdowns), attribute selectors handled by a "reduce, don't drop" rule.
  • Colour & decoration: named/#rgb/#rrggbb, modern rgb()/hsl(), background-color, linear & radial gradients, background-image: url(), border + border-radius, box-shadow, group opacity.
  • Text: anti-aliased proportional text (go-opentype) with real bold and italic faces (no faux-bold), serif / sans / mono, complex scripts (Cyrillic, Vietnamese, …); white-space: pre.
  • Images: <img> over http(s) + data: (PNG/JPEG) and SVG (oksvg/rasterx) via <img *.svg>, data:image/svg+xml and inline <svg>.
  • JavaScript: page scripts run via goja against a real DOM, with fetch()/XHR and read-back of real laid-out geometry (getBoundingClientRect, offsetWidth/Height, getComputedStyle). A settle-then-render loop re-cascades and re-lays-out after scripts mutate the DOM (incl. dynamically injected <script>/<style>/<link>), to a bounded fixpoint — so mw.loader-style runtime chrome is reflected in the output. The same JS-settled DOM drives the click hit-map.

Honest limits (not overclaimed)

  • No conic-gradient, CSS filter or mask; SVG has no <filter>/<mask>/ <pattern>/embedded <image>/<text>, and a per-page image budget caps very icon-heavy pages.
  • No <li> list-style marker discs yet; some icon-font / visually-hidden chrome renders as text where a browser shows an icon.
  • Large computed pages (pkg.go.dev, go.dev) render slower than Chrome — an open perf gap, not a fidelity one.
  • This is not a standards-complete browser and not "as good as Chromium". Measured mean windowed-SSIM across the five bench pages is ≈ 0.69, with clear diminishing returns; the Wikipedia number (≈ 0.44) is JS-confounded and noisy. See the numbers below.

Measured fidelity vs headless Chrome

From bench/REPORT.md (windowed SSIM over the common top-left region, 1024px width; speed× = chrome_ms / webengine_ms, >1 = faster; timings include the live network fetch and vary with it):

URL SSIM pixdiff % speed× note
example.com/ 0.954 1.5 34.8 near-parity, ~35× faster
react.dev/ 0.727 26.4 1.15 SPA; gradients + React SVG atom render
go.dev/blog/ 0.670 33.2 0.34 dark-mode + SVG logos render; slower
pkg.go.dev/net/http 0.629 36.5 0.14 large computed page; perf gap
en.wikipedia.org/wiki/Go 0.441 22.4 1.20 JS-confounded, noisy metric

example.com is at near-parity and much faster; the JS-heavy and large computed pages are the honest frontier. Re-run the harness with cd bench && go run ./cmd/compare -urls urls.txt (needs a Chrome/Chromium binary).

Test

CGO_ENABLED=0 go build ./...          # cgo-free build
CGO_ENABLED=0 go vet ./...
CGO_ENABLED=0 go test -short ./...     # -short skips the live-network render
bash scripts/coverage-gate.sh          # ratchet coverage gate (see below)

The pure logic (cascade/inheritance, line-breaker, box metrics, DOM, selector engine) is asserted at exact geometry; committed golden PNGs — including offline JS/dynamic/gradient/position/SVG fixtures with a DisableJS control — cover the paint path. scripts/coverage-gate.sh enforces a ratchet coverage floor per pure-logic package (css/layout/paint/dom), which CI fails below and which is raised (never lowered) toward 100% as the engine matures. The live-network paths (root engine package, cmd/render) are excluded from the gate because their coverage is not reproducible in CI. The bench/ fidelity harness is a separate nested module (it pulls chromedp) and is not in the CGO=0 six-arch CI. go.mod floor is go 1.26.4; cross-built for all six 64-bit Go targets.

License

BSD-3-Clause — see LICENSE. Copyright (c) the go-webengine/engine authors.

Documentation

Overview

Package engine is a pure-Go (CGO=0) web rendering engine: it fetches a URL, parses the HTML into a DOM, applies a real CSS subset (cascade, inheritance, var(), @media, dark-mode, gradients), runs the page's JavaScript against a real DOM binding, lays the content out with a full box model (block, inline, float, flex, grid, table, position), and paints anti-aliased text, backgrounds, gradients, box-shadows, images and SVG to an image.RGBA. A settle-then-render loop re-lays-out after scripts mutate the DOM, so script-driven changes are reflected in the output; set Engine.DisableJS to render the static, no-JavaScript document instead.

This file adds a hyperlink hit-map on top of the laid-out box tree: given a rendered page it reports the painted rectangle and resolved href of every <a href> anchor, so a consumer (e.g. the wasmdesk browserproxy) can turn a click at pixel (x, y) into a navigation target. It is kept deliberately in its own file — separate from engine.go and the js/ runtime work — so the hit-test can evolve without colliding with the rendering pipeline.

This file adds a last-resort OpenGraph/meta fallback: when a page renders to nothing — an un-hydratable JavaScript SPA whose content lives only in a React initial-state blob (Mastodon, X, many app-shell sites) — the engine synthesises a clean, readable card from the page's og:/meta tags and renders that instead of a blank frame. It is gated by Engine.MetaFallback and only ever replaces a genuinely empty render, so pages that render real content are completely unaffected.

This file adds STAGED progressive rendering on top of the batch pipeline. A browser paints before all resources/scripts are done and re-flows incrementally; the engine emulates that with a small number of fully-styled frames (external CSS already applied, so no unstyled flash):

  • "initial": the TEXT-FIRST frame — external sheets fetched and cascaded and the page laid out, but BEFORE any image is fetched (image boxes reserve from their attr/CSS sizes). The big perceived-latency win: the styled text paints without waiting on the slow, network-bound image fetch.
  • "images": after the images/background images are loaded and the page is re-laid-out with their real intrinsic sizes (deduped against "initial" by geometry — a page with no images that moved nothing yields just initial+final).
  • "settle": after each settle pass that visibly changed the geometry (deduped; bounded by the settle pass cap) — the interruptible-reflow analogue.
  • "final" (Final=true): the fully-settled render, byte-identical to RenderWithLinks for the same input.

Every frame carries an INDEPENDENT image snapshot (a fresh canvas per frame), so a consumer may retain it after onFrame returns.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EncodePNG

func EncodePNG(img image.Image) ([]byte, error)

EncodePNG encodes an image as PNG bytes.

func LinkAt

func LinkAt(links []Link, pt image.Point) (string, bool)

LinkAt returns the href of the first link whose rectangle contains pt (a full-page pixel point) and whether one was found. Links are tested in document order, so an earlier anchor wins when rectangles overlap.

func Screenshot

func Screenshot(ctx context.Context, rawurl string, viewport image.Rectangle) ([]byte, error)

Screenshot renders url with a default engine and returns PNG bytes.

Types

type Document

type Document struct {
	URL   string    // final URL after redirects
	Title string    // <title> text
	Root  *dom.Node // owned DOM tree
	HTML  string    // decoded (UTF-8) source
}

Document is a fetched and parsed page.

type Engine

type Engine struct {
	Client    *http.Client
	UserAgent string
	// MaxImages bounds how many RASTER (non-SVG) <img>/background images are
	// fetched and decoded per render — the expensive network + large-decode work.
	MaxImages int
	// MaxVectorImages bounds how many VECTOR images (inline <svg> and
	// <img src="*.svg"> / data:image/svg+xml) are rasterised per render. Vector
	// chrome (nav/footer/social/UI icons) is cheap but plentiful — real design
	// systems place many dozens ahead of the content — so it gets its OWN, more
	// generous budget. Without this split a wall of decorative icons early in the
	// DOM exhausts a single shared budget and starves the actual content photos
	// (raster) further down (the DeepMind hero black-void bug).
	MaxVectorImages int

	// DisableJS turns off the JavaScript pass. Offline fixture tests that must
	// stay byte-deterministic set this; the default (false) runs page scripts.
	DisableJS bool
	// JSTimeout bounds the total script + timer budget per render. Zero selects
	// js.DefaultTimeout.
	JSTimeout time.Duration
	// MaxJSHeapBytes bounds process heap growth during the JavaScript settle
	// stage (script execution AND the esbuild module-bundle fetch/parse). Some
	// real pages ship huge module graphs (GitHub, large SPAs) whose bundling
	// balloons memory into the gigabytes before any time budget trips; a watchdog
	// samples the heap and, once growth from the stage's start exceeds this bound,
	// cancels the stage so it falls back to the already-computed pre-script layout
	// instead of driving the process toward OOM. Zero selects a sane default; a
	// negative value disables the guard.
	MaxJSHeapBytes int64
	// JSLog, if non-nil, receives console.* and diagnostic lines from the script
	// pass (used for debugging; nil discards them).
	JSLog func(string)
	// Backdrop is the base colour painted under a page before its own background.
	// The zero value (transparent) selects the default white paper. A themed host
	// (e.g. a reader in dark mode) sets it to its surface colour so a page that
	// declares no background of its own — most plainly a bare image — is framed to
	// match instead of flashing white. A page that DOES declare a background still
	// paints its own over this, so normal sites are unchanged.
	Backdrop css.Color
	// MetaFallback, when true, renders a clean readable card synthesised from a
	// page's OpenGraph/meta tags (og:title/og:description/og:image, <title>) as a
	// LAST RESORT — only when the real render is empty (a JavaScript SPA the engine
	// cannot hydrate: Mastodon, X, app-shell sites). Default false: it fabricates
	// content that is not in the page's rendered DOM, so it is opt-in; a
	// link-preview / reader consumer enables it, while callers wanting a faithful
	// render (or to detect the blank SPA themselves) are unaffected. It never
	// triggers on a page that renders any real content, so normal pages are
	// byte-identical whether it is on or off.
	MetaFallback bool
}

Engine holds the HTTP client and render configuration.

func New

func New() *Engine

New returns an Engine with a browser-like HTTP client (Chrome TLS fingerprint, cookie jar, redirect following).

func (*Engine) Fetch

func (e *Engine) Fetch(ctx context.Context, rawurl string) (*Document, error)

Fetch retrieves and parses url into a Document.

func (*Engine) Render

func (e *Engine) Render(ctx context.Context, rawurl string, viewport image.Rectangle) (*image.RGBA, *RenderInfo, error)

Render fetches url and renders it into an image at the given viewport width (the height is grown to fit the full page, at least the viewport height).

func (*Engine) RenderDocument

func (e *Engine) RenderDocument(ctx context.Context, doc *Document, viewport image.Rectangle) (*image.RGBA, *RenderInfo, error)

RenderDocument renders an already-fetched Document. It is the offline entry point used by fixtures and tests (image sub-resources are still fetched from the network if their src is absolute/resolvable).

func (*Engine) RenderDocumentProgressive added in v0.3.0

func (e *Engine) RenderDocumentProgressive(ctx context.Context, doc *Document, viewport image.Rectangle, onFrame func(ProgressiveFrame)) (*RenderInfo, error)

RenderDocumentProgressive renders an already-fetched Document progressively. See RenderProgressive.

func (e *Engine) RenderDocumentWithLinks(ctx context.Context, doc *Document, viewport image.Rectangle) (*image.RGBA, *RenderInfo, []Link, error)

RenderDocumentWithLinks renders an already-fetched Document and returns its image, render info and anchor hit-map from a single layout pass. It runs the same cascade → layout → paint pipeline as RenderDocument.

func (*Engine) RenderHTML

func (e *Engine) RenderHTML(ctx context.Context, htmlSrc, baseURL string, viewport image.Rectangle) (*image.RGBA, *RenderInfo, error)

RenderHTML renders an HTML string (with baseURL used to resolve relative image sources) into an image at the given viewport. It is the offline entry point for local fixtures and demos, running the same cascade/layout/paint pipeline as Render but without fetching the page itself.

func (*Engine) RenderProgressive added in v0.3.0

func (e *Engine) RenderProgressive(ctx context.Context, rawurl string, viewport image.Rectangle, onFrame func(ProgressiveFrame)) (*RenderInfo, error)

RenderProgressive fetches rawurl and renders it progressively, invoking onFrame for each staged frame. It runs the same cascade → JS settle → layout → paint pipeline as RenderWithLinks; the Final frame's image, links and RenderInfo are identical to what RenderWithLinks returns for the same input. onFrame is always called at least once and the last call has Final=true.

func (e *Engine) RenderWithLinks(ctx context.Context, rawurl string, viewport image.Rectangle) (*image.RGBA, *RenderInfo, []Link, error)

RenderWithLinks fetches url, renders it to a full-page image and, in the same layout pass, returns the anchor hit-map. It mirrors (*Engine).Render but also threads the laid-out box tree into LinksFromBox so the image and the link rectangles are guaranteed to describe the exact same layout. It is kept here (rather than folded into engine.go's RenderDocument) so the hit-test feature stays isolated from the core pipeline and the js/ runtime work.

func (*Engine) Screenshot

func (e *Engine) Screenshot(ctx context.Context, rawurl string, viewport image.Rectangle) ([]byte, error)

Screenshot renders url and encodes the result as PNG.

type Link struct {
	Rect image.Rectangle
	Href string
}

Link is one hyperlink's painted rectangle (in full-page image pixels, the same coordinate space as the image returned by Render) and its href resolved against the page URL to an absolute URL.

func LinksFromBox

func LinksFromBox(root *layout.Box, baseURL string) []Link

LinksFromBox walks a laid-out box tree and returns, in document order, one Link per <a href> anchor that painted at least one inline atom (a word or an image). Each anchor's Rect is the union of the rectangles of all inline atoms it produced, so a multi-line link yields a single bounding box covering every line. baseURL is the page's own URL: each anchor's raw href is resolved against it, and links that resolve to a non-navigable scheme (javascript:, mailto:, an empty or pure-fragment href) are dropped.

The rectangles are in the box tree's own coordinate space, which is the full-page image space (layout origin is 0,0 at the page top-left), so a caller can hit-test a full-page click directly against them.

type ProgressiveFrame added in v0.3.0

type ProgressiveFrame struct {
	Img   *image.RGBA
	Links []Link
	Info  *RenderInfo
	Stage string
	Final bool
}

ProgressiveFrame is one staged snapshot delivered to a RenderProgressive callback. Img is an independent allocation; Links matches RenderWithLinks's shape; Stage is "initial" | "images" | "settle" | "final"; Final is true exactly once, on the last frame.

type RenderInfo

type RenderInfo struct {
	Title         string
	URL           string
	ContentHeight int
}

RenderInfo carries metadata about a completed render.

func Render

func Render(ctx context.Context, rawurl string, viewport image.Rectangle) (*image.RGBA, *RenderInfo, error)

Render renders url with a default engine.

Directories

Path Synopsis
cmd
render command
Command render fetches a URL and writes a PNG screenshot of the statically rendered page.
Command render fetches a URL and writes a PNG screenshot of the statically rendered page.
Package css implements a deliberately small but real subset of CSS: a value model, a tokenizer/parser for stylesheets and declaration blocks, tag/class/ id selectors with specificity, and a cascade with inheritance over a dom tree.
Package css implements a deliberately small but real subset of CSS: a value model, a tokenizer/parser for stylesheets and declaration blocks, tag/class/ id selectors with specificity, and a cascade with inheritance over a dom tree.
Package dom is a small, owned DOM node tree built from golang.org/x/net/html.
Package dom is a small, owned DOM node tree built from golang.org/x/net/html.
Package js runs a page's JavaScript against a minimal but real DOM binding on a pure-Go goja ECMAScript runtime.
Package js runs a page's JavaScript against a minimal but real DOM binding on a pure-Go goja ECMAScript runtime.
Package layout is a block-and-inline flow engine.
Package layout is a block-and-inline flow engine.
Package paint rasterises a laid-out box tree onto an *image.RGBA using go-opentype for anti-aliased text, go-widgets/painter for backgrounds and go-images-decoded bitmaps for <img>.
Package paint rasterises a laid-out box tree onto an *image.RGBA using go-opentype for anti-aliased text, go-widgets/painter for backgrounds and go-images-decoded bitmaps for <img>.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL