distill

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package distill is the orchestrator: it takes a URL and returns an artifact, choosing how much work to do along the way.

The shape of a run is always the same, whatever tier answers:

fetch  ->  score  ->  [render]  ->  [sweep]  ->  [recover]  ->  graph  ->  emit

Every rung is optional except the first and the last two, and the decision to climb is recorded in the artifact rather than left implicit. A caller can always see which tier answered and why.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Distiller

type Distiller struct {
	// contains filtered or unexported fields
}

Distiller runs distillations, reusing a browser across them.

func New

func New(opts Options) *Distiller

New builds a distiller. The browser is not started here: a caller that only ever hits tier 0 should never pay for one.

func (*Distiller) Close

func (d *Distiller) Close()

Close releases the browser, if one was started.

It waits for any speculative launch still in flight first. A browser that is half-started when the caller gives up is still a process, and the goroutine that would have adopted it is the only thing holding its handle.

func (*Distiller) Distill

func (d *Distiller) Distill(ctx context.Context, rawURL string) (*Result, error)

Distill produces an artifact for one URL.

func (*Distiller) Memory

func (d *Distiller) Memory() *escalate.Memory

Memory exposes the escalation memory so a caller can persist it.

type Options

type Options struct {
	// MaxTier caps the ladder. Setting it to TierFetch forbids the browser
	// entirely, which is what a caller with no Chromium wants.
	MaxTier escalate.Tier
	// MinTier forces at least this much work, for a caller who already knows
	// the page is heavy.
	MinTier escalate.Tier
	// Thresholds are pinned escalation constants.
	Thresholds escalate.Thresholds
	// Memory provides per-domain hysteresis. Optional but strongly recommended:
	// without it the tier decision is recomputed from scratch every run and a
	// page near the threshold will waver.
	Memory *escalate.Memory

	// Render configures the browser, when one is used.
	Render render.Options
	// Fetch configures tier 0.
	Fetch fetch.Options

	// Robots enforces robots.txt. A nil cache means no enforcement, which is
	// only appropriate for a site the caller owns.
	Robots *safety.RobotsCache
	// Limiter enforces politeness towards a single site.
	Limiter *safety.Limiter
	// Guard vets every URL.
	Guard *safety.Guard

	// Canvas configures recovery. Vision is off unless explicitly enabled.
	Canvas canvas.Options

	// Private marks the run as using an authenticated or user-profile session,
	// which makes the artifact ineligible for any shared cache.
	Private bool

	// Now is injectable so golden tests are not time-dependent.
	Now time.Time
	// Generator identifies the build.
	Generator string
	// Logf receives progress lines.
	Logf func(format string, args ...any)

	// OnProgress is called as stages complete, so a caller can emit a manifest
	// as soon as one exists rather than after the whole sweep. Time to first
	// useful token is the metric that matters; thirty seconds of silence and
	// thirty seconds with usable output at two seconds are different products.
	OnProgress func(Progress)
}

Options configures a distillation.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns a usable configuration.

type Progress

type Progress struct {
	Stage   string
	Tier    escalate.Tier
	Message string
	Elapsed time.Duration
	// Partial carries a graph good enough to answer some questions, when one
	// exists. It is always marked incomplete.
	Partial *graph.Graph
}

Progress reports a stage boundary.

type Result

type Result struct {
	Graph *graph.Graph
	// Freshness is what to store so the next run can skip work.
	Freshness fetch.Freshness
	// Decision records the tier and its reasoning.
	Decision escalate.Decision
	// Timing breaks down where the wall clock went.
	Timing map[string]time.Duration

	// Capture is the deduplicated observation the graph was built from. It is
	// retained so a snapshot can be recorded: storing the artifact would only
	// prove what the graph produced last time, whereas storing the capture lets
	// the whole graph stage be re-run against new code.
	Capture *capture.Merged
	// StaticHTML is the served document.
	StaticHTML string
	// Scene and Libraries are the page-level probes, kept for the same reason.
	Scene     *capture.SceneIntrospection
	Libraries []string
	Status    int64
}

Result is a completed distillation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL