Documentation
¶
Overview ¶
Package webscrape integrates web scraping as a service-task worker: a BPMN web-scraping task fetches a model-authored URL and extracts the elements matching a CSS selector through the job path (ADR-0118), mirroring how the rest package calls a model-authored HTTP endpoint (ADR-0067). ADR-0190 extends the same worker with explicit RSS and Atom extraction modes, and ADR-0231 adds per-item fields, richer feed entries, and a fetch that copes with the encodings and feed flavors real publishers ship. The integration inherits the job protocol's durability and non-blocking properties (ADR-0007):
- A task creates a job carrying the reserved compiler.WebScrapeJobType. The processor never performs the outbound fetch itself, so it stays allocation-free (invariant I1) and free of any HTTP/HTML/XML dependency.
- The in-process Handler — a job worker — pulls those jobs, fetches the document off the processor goroutine and after fsync (invariant I2, never inside applyToState / I4), extracts the authored representation, writes it into the task's result variable, and completes the job, which drives the token onward.
Like the REST worker, the URL and extraction settings are authored in the model; there is no server-registered worker and no credential. A scrape is a plain GET, so an at-least-once retry simply refetches — the operation is idempotent and side-effect-free.
Index ¶
Constants ¶
const UserAgent = "Atlas-Webscrape/1.0 (+https://github.com/pblumer/atlas)"
UserAgent is what a scrape says it is. Go's default ("Go-http-client/2.0") reads as an anonymous script and is refused outright by a large share of sites, which reaches the author as a bare 403 with nothing to act on. An honest identity with a project link is something a site operator can allow — or block deliberately, which is the point (ADR-0231).
Variables ¶
This section is empty.
Functions ¶
func Handler ¶
func Handler(store state.Reader, lookup ProcessLookup, client Client) job.OutputHandler
Handler builds a job handler that performs a web-scraping worker task. Register it under compiler.WebScrapeJobTypeIndex via HandleWithOutput. The handler resolves model data against the task's visible variables, runs the network/document work off the processor, and returns one durable result variable. HTML remains a JSON array of strings; RSS/Atom are arrays of stable feed-entry objects (ADR-0190).
func Items ¶ added in v0.5.0
Items is what a run's result becomes as the value of the process variable: the scraped strings for an HTML scrape with no fields, one object per match for a scrape that authored fields (ADR-0231), or the feed's entries as {title, link, description, published, guid, author, categories, image} objects for RSS/Atom (ADR-0190, extended).
Both halves call it. The in-process worker renders it through expr and one leased by another process sends it as JSON on the wire, but *what a scrape means* is decided here once — which is the property that was missing: the offloaded path built its own list from Values alone, so a feed reached a model as an empty array even on the days the format survived the hand-over at all.
Types ¶
type Client ¶
Client fetches a page and extracts an HTML scrape's matches. It remains the original interface so existing HTML clients and tests stay source-compatible. Feed-capable clients additionally implement FeedClient, and clients that can assemble one object per match implement RecordClient.
type FeedClient ¶ added in v0.5.0
FeedClient is the optional structured-feed half of a web-scrape client (ADR-0190). The built-in HTTPClient implements it; keeping it separate preserves source compatibility for HTML-only custom clients.
type FeedEntry ¶ added in v0.5.0
type FeedEntry struct {
Title string `json:"title"`
Link string `json:"link"`
Description string `json:"description"`
Published string `json:"published"`
Guid string `json:"guid"`
Author string `json:"author"`
Categories []string `json:"categories"`
Image string `json:"image"`
}
FeedEntry is the stable structured result one RSS item or Atom entry produces (ADR-0190, extended by ADR-0231). Every field is always serialized; a source that omits one leaves it empty rather than making Atlas invent data. Categories is always a list, empty when the source names none.
Guid is the entry's own identity as the publisher states it (RSS <guid>, Atom <id>) — it is what a recurring scrape deduplicates on across runs, which a title cannot do because publishers edit titles.
type Field ¶ added in v0.5.0
type Field struct {
Name string `json:"name"`
Selector string `json:"selector,omitempty"`
Attribute string `json:"attribute,omitempty"`
}
Field is one named field of a structured HTML scrape. Name is the key the value lands under in the item's object; Selector is optional and evaluated within the matched item (empty = the item element itself); Attribute is optional (empty = the element's text).
type HTTPClient ¶
type HTTPClient struct {
// contains filtered or unexported fields
}
HTTPClient scrapes a real document over HTTP.
func NewHTTPClient ¶
func NewHTTPClient() *HTTPClient
NewHTTPClient builds a web-scraping HTTP client bounded by the shared worker call budget (nettimeout.Default). The worker runs on the run-loop goroutine, so an unbounded call would let a hung site stall the whole engine; see nettimeout.
func (*HTTPClient) Scrape ¶
Scrape GETs r.URL as HTML and extracts the matches of r.Selector. A non-2xx status or fetch/parse failure is an error so the job is retried. An empty result is valid and becomes an empty JSON array. MaxItems is applied in document order.
func (*HTTPClient) ScrapeFeed ¶ added in v0.5.0
ScrapeFeed GETs r.URL and decodes the explicitly authored RSS or Atom format. Content-Type is used only for negotiation: r.Format, not the response, selects the parser (ADR-0190).
func (*HTTPClient) ScrapeRecords ¶ added in v0.5.0
ScrapeRecords GETs r.URL as HTML and returns one object per match of r.Selector, carrying the task's authored fields. The item selector and the field selectors are compiled before the fetch is parsed, so a typo names itself rather than producing an empty result.
type Job ¶ added in v0.3.0
type Job struct {
URL string `json:"url"`
Selector string `json:"selector,omitempty"`
Attribute string `json:"attribute,omitempty"`
Fields []Field `json:"fields,omitempty"`
Format string `json:"format,omitempty"`
MaxItems int32 `json:"maxItems,omitempty"`
// AbsoluteLinks resolves href/src reads against the fetched document's final URL
// (HTML); PlainText strips markup from a feed entry's description. Both are
// compile-time structure like Format, and both default to the behavior every
// model authored before ADR-0231 has.
AbsoluteLinks bool `json:"absoluteLinks,omitempty"`
PlainText bool `json:"plainText,omitempty"`
// Result names the process variable the scraped values are written to; empty
// means the task writes nothing back.
Result string `json:"resultVariable,omitempty"`
}
Job is a web-scrape task with everything already evaluated. It is what travels with a leased job. Format is always explicit for newly resolved work; an empty value remains HTML for backwards compatibility with pre-ADR-0190 payloads.
func Resolve ¶ added in v0.3.0
func Resolve(store state.Reader, cp *compiler.CompiledProcess, detail *compiler.ConnectorTaskDetail, ei *model.ElementInstanceValue, elementInstanceKey uint64) (Job, error)
Resolve turns a compiled web-scrape task into a Job by evaluating its authored values against the variables the task sees. Format and MaxItems are already compile-time structural data (ADR-0190), so resolution copies rather than interprets them.
type ProcessLookup ¶
type ProcessLookup func(defKey uint64) *compiler.CompiledProcess
ProcessLookup resolves a process-definition key to its compiled process. The worker uses it to find the resolved web-scrape configuration a job belongs to, so one handler serves every deployed process.
type RecordClient ¶ added in v0.5.0
type RecordClient interface {
ScrapeRecords(ctx context.Context, r Request) ([]map[string]string, error)
}
RecordClient is the optional structured-HTML half (ADR-0231): one object per selector match, keyed by the task's authored field names. Separate from Client for the same reason FeedClient is — a custom client written for ADR-0118 keeps compiling.
type Request ¶
type Request struct {
URL string
Selector string
Attribute string
Fields []Field
Format string
MaxItems int32
AbsoluteLinks bool
PlainText bool
}
Request is one scrape a web-scraping task performs. URL is the full, model-authored document to fetch. Format is the already-compiled representation (html/rss/atom); empty retains the pre-ADR-0190 HTML default. Selector/Attribute/ Fields/AbsoluteLinks apply only to HTML, PlainText only to feeds. MaxItems is the deterministic first-N bound; 0 is unlimited.
type Result ¶ added in v0.3.0
type Result struct {
ResultVariable string
Format string
Values []string
Records []map[string]string
Entries []FeedEntry
}
Result is what running a Job produces. HTML with no fields keeps Values as the historical string array; a field scrape populates Records and RSS/Atom populate Entries instead. Format tells the handler which result shape to persist without inspecting the returned content; Records is non-nil for every field scrape, so an empty field result is still a list of objects and not of strings.
Source Files