Documentation
¶
Overview ¶
Package webscrape integrates web scraping as a service-task connector: a BPMN web-scraping connector task fetches a model-authored URL and extracts the elements matching a CSS selector through the job path (ADR-0118), mirroring how the rest package calls a model-authored HTTP endpoint (ADR-0067). The integration inherits the job protocol's durability and non-blocking properties (ADR-0007):
- A connector task creates a job carrying the reserved compiler.WebScrapeJobType. The processor never performs the outbound fetch itself, so it stays allocation-free (invariant I1) and free of any HTTP/HTML dependency.
- The in-process Handler — a job worker — pulls those jobs, fetches the page off the processor goroutine and after fsync (invariant I2, never inside applyToState / I4), extracts the matches, writes them into the task's result variable as a JSON array, and completes the job, which drives the token onward.
Like the REST connector, the URL and selector are authored in the model; there is no server-registered connector and no credential. A scrape is a plain GET, so an at-least-once retry (a crash between "the page was fetched" and "job completed") simply refetches — the operation is idempotent and side-effect-free.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Handler ¶
func Handler(store state.Reader, lookup ProcessLookup, client Client) job.OutputHandler
Handler builds a job handler that performs a web-scraping connector task. Register it with a job.Runner under the reserved compiler.WebScrapeJobTypeIndex via HandleWithOutput; the runner then pulls activatable scrape jobs, and for each the handler resolves the connector task's url/selector/attribute/result-variable from the compiled process — evaluating any FEEL url/selector values over the instance's variables (the fx toggle, ADR-0067) — fetches the page through client, and returns the extracted values as the result variable (a JSON array of strings) to be written back into the instance on completion. Returning an error fails the job (retry, then an incident, ADR-0061); the runner completes it only on success.
Types ¶
type Client ¶
Client fetches a page and extracts a scrape's matches. It is an interface so the worker is testable without a live server.
type HTTPClient ¶
type HTTPClient struct {
// contains filtered or unexported fields
}
HTTPClient scrapes a real page over HTTP. It GETs Request.URL, parses the HTML, and returns the text (or the named attribute) of every element matching Request.Selector. A non-2xx status is returned as an error so the job stays pending and is retried (at-least-once).
func NewHTTPClient ¶
func NewHTTPClient() *HTTPClient
NewHTTPClient builds a web-scraping HTTP client bounded by the shared connector call budget (nettimeout.Default). The worker runs on the run-loop goroutine, so an unbounded call would let a hung site stall the whole engine; see the nettimeout package doc. A per-connector configurable timeout is a follow-up (ADR-0118).
type Job ¶ added in v0.3.0
type Job struct {
URL string `json:"url"`
Selector string `json:"selector"`
Attribute string `json:"attribute,omitempty"`
// Result names the process variable the scraped values are written to; empty
// means the task writes nothing back.
Result string `json:"resultVariable,omitempty"`
}
Job is a web-scrape task with everything already evaluated. It is what travels with a leased job.
func Resolve ¶ added in v0.3.0
func Resolve(store state.Reader, cp *compiler.CompiledProcess, detail *compiler.ConnectorTaskDetail, ei *model.ElementInstanceValue, elementInstanceKey uint64) (Job, error)
Resolve turns a compiled web-scrape task into a Job by evaluating its authored fields against the variables the task sees. Engine work by necessity — FEEL is compiled at deploy (ADR-0008/0015) and the scope lives in the store.
type ProcessLookup ¶
type ProcessLookup func(defKey uint64) *compiler.CompiledProcess
ProcessLookup resolves a process-definition key to its compiled process. The worker uses it to find the URL, selector, attribute, and result variable a scrape job belongs to, so one handler serves every deployed process.
type Request ¶
Request is one scrape a web-scraping connector task performs. URL is the full, model-authored page to fetch. Selector is the CSS selector whose matches are extracted. Attribute, when non-empty, names the HTML attribute read from each match; empty means read each match's text content.
Source Files