Documentation
¶
Overview ¶
Package distill is the orchestrator: it takes a URL and returns an artifact, choosing how much work to do along the way.
The shape of a run is always the same, whatever tier answers:
fetch -> score -> [render] -> [sweep] -> [recover] -> graph -> emit
Every rung is optional except the first and the last two, and the decision to climb is recorded in the artifact rather than left implicit. A caller can always see which tier answered and why.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Distiller ¶
type Distiller struct {
// contains filtered or unexported fields
}
Distiller runs distillations, reusing a browser across them.
func New ¶
New builds a distiller. The browser is not started here: a caller that only ever hits tier 0 should never pay for one.
func (*Distiller) Close ¶
func (d *Distiller) Close()
Close releases the browser, if one was started.
It waits for any speculative launch still in flight first. A browser that is half-started when the caller gives up is still a process, and the goroutine that would have adopted it is the only thing holding its handle.
type Options ¶
type Options struct {
// MaxTier caps the ladder. Setting it to TierFetch forbids the browser
// entirely, which is what a caller with no Chromium wants.
MaxTier escalate.Tier
// MinTier forces at least this much work, for a caller who already knows
// the page is heavy.
MinTier escalate.Tier
// Thresholds are pinned escalation constants.
Thresholds escalate.Thresholds
// Memory provides per-domain hysteresis. Optional but strongly recommended:
// without it the tier decision is recomputed from scratch every run and a
// page near the threshold will waver.
Memory *escalate.Memory
// Render configures the browser, when one is used.
Render render.Options
// Fetch configures tier 0.
Fetch fetch.Options
// Robots enforces robots.txt. A nil cache means no enforcement, which is
// only appropriate for a site the caller owns.
Robots *safety.RobotsCache
// Limiter enforces politeness towards a single site.
Limiter *safety.Limiter
// Guard vets every URL.
Guard *safety.Guard
// Canvas configures recovery. Vision is off unless explicitly enabled.
Canvas canvas.Options
// Private marks the run as using an authenticated or user-profile session,
// which makes the artifact ineligible for any shared cache.
Private bool
// Now is injectable so golden tests are not time-dependent.
Now time.Time
// Generator identifies the build.
Generator string
// Logf receives progress lines.
Logf func(format string, args ...any)
// OnProgress is called as stages complete, so a caller can emit a manifest
// as soon as one exists rather than after the whole sweep. Time to first
// useful token is the metric that matters; thirty seconds of silence and
// thirty seconds with usable output at two seconds are different products.
OnProgress func(Progress)
}
Options configures a distillation.
type Progress ¶
type Progress struct {
Stage string
Tier escalate.Tier
Message string
Elapsed time.Duration
// Partial carries a graph good enough to answer some questions, when one
// exists. It is always marked incomplete.
Partial *graph.Graph
}
Progress reports a stage boundary.
type Result ¶
type Result struct {
Graph *graph.Graph
// Freshness is what to store so the next run can skip work.
Freshness fetch.Freshness
// Decision records the tier and its reasoning.
Decision escalate.Decision
// Timing breaks down where the wall clock went.
Timing map[string]time.Duration
// Capture is the deduplicated observation the graph was built from. It is
// retained so a snapshot can be recorded: storing the artifact would only
// prove what the graph produced last time, whereas storing the capture lets
// the whole graph stage be re-run against new code.
Capture *capture.Merged
// StaticHTML is the served document.
StaticHTML string
// Scene and Libraries are the page-level probes, kept for the same reason.
Scene *capture.SceneIntrospection
Libraries []string
Status int64
}
Result is a completed distillation.