Documentation
¶
Overview ¶
Package webscrape integrates web scraping as a service-task connector: a BPMN web-scraping connector task fetches a model-authored URL and extracts the elements matching a CSS selector through the job path (ADR-0118), mirroring how the rest package calls a model-authored HTTP endpoint (ADR-0067). The integration inherits the job protocol's durability and non-blocking properties (ADR-0007):
- A connector task creates a job carrying the reserved compiler.WebScrapeJobType. The processor never performs the outbound fetch itself, so it stays allocation-free (invariant I1) and free of any HTTP/HTML dependency.
- The in-process Handler — a job worker — pulls those jobs, fetches the page off the processor goroutine and after fsync (invariant I2, never inside applyToState / I4), extracts the matches, writes them into the task's result variable as a JSON array, and completes the job, which drives the token onward.
Like the REST connector, the URL and selector are authored in the model; there is no server-registered connector and no credential. A scrape is a plain GET, so an at-least-once retry (a crash between "the page was fetched" and "job completed") simply refetches — the operation is idempotent and side-effect-free.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Handler ¶
func Handler(store *state.Store, lookup ProcessLookup, client Client) job.OutputHandler
Handler builds a job handler that performs a web-scraping connector task. Register it with a job.Runner under the reserved compiler.WebScrapeJobTypeIndex via HandleWithOutput; the runner then pulls activatable scrape jobs, and for each the handler resolves the connector task's url/selector/attribute/result-variable from the compiled process — evaluating any FEEL url/selector values over the instance's variables (the fx toggle, ADR-0067) — fetches the page through client, and returns the extracted values as the result variable (a JSON array of strings) to be written back into the instance on completion. Returning an error fails the job (retry, then an incident, ADR-0061); the runner completes it only on success.
Types ¶
type Client ¶
Client fetches a page and extracts a scrape's matches. It is an interface so the worker is testable without a live server.
type HTTPClient ¶
type HTTPClient struct {
// contains filtered or unexported fields
}
HTTPClient scrapes a real page over HTTP. It GETs Request.URL, parses the HTML, and returns the text (or the named attribute) of every element matching Request.Selector. A non-2xx status is returned as an error so the job stays pending and is retried (at-least-once).
func NewHTTPClient ¶
func NewHTTPClient() *HTTPClient
NewHTTPClient builds a web-scraping HTTP client backed by http.DefaultClient. A configurable timeout is a follow-up (ADR-0118).
type ProcessLookup ¶
type ProcessLookup func(defKey uint64) *compiler.CompiledProcess
ProcessLookup resolves a process-definition key to its compiled process. The worker uses it to find the URL, selector, attribute, and result variable a scrape job belongs to, so one handler serves every deployed process.
Source Files