Documentation
¶
Overview ¶
Package api provides HTTP handlers for bounded AI authoring endpoints.
Purpose: - Expose prompt-heavy AI preview and authoring routes without creating jobs.
Responsibilities: - Validate authoring requests and shared AI configuration. - Enforce strict JSON request parsing and bounded request sizes. - Adapt authoring results into stable API responses.
Scope: - AI authoring request handlers only.
Usage: - Mounted under `/v1/ai/*`.
Invariants/Assumptions: - Requests must use `application/json`. - AI handlers require the shared `aiauthoring.Service`.
Package api provides HTTP handlers for analytics endpoints.
This file handles: - Historical metrics queries - Host performance analytics - Job trend analysis - Dashboard data aggregation
This file does NOT handle: - Real-time metrics (metrics_handler.go handles that) - Data collection (analytics/collector.go handles that) - Report generation (analytics_reports.go handles that)
Package api provides HTTP handlers for authentication profile management endpoints. Auth handlers support listing, creating, updating, deleting auth profiles, importing/exporting profiles, and managing profile presets. It also provides OAuth 2.0 and OIDC authentication flow support.
Package api provides HTTP handlers for batch job operations.
Purpose: - Accept batch submissions and batch-status operations over the API.
Responsibilities: - List persisted batches with pagination metadata. - Validate and submit scrape, crawl, and research batches. - Return aggregate batch status and optionally paginated jobs. - Cancel batches and their constituent jobs.
Scope: - API request handling only; persistence and job execution live in internal/store and internal/jobs.
Usage: - Registered for /v1/jobs/batch and /v1/jobs/batch/* routes.
Invariants/Assumptions: - All batch requests are JSON and validated before any jobs are created. - Batch list responses expose aggregate summaries only; batch detail loads individual jobs separately. - Research batches create a single research job containing all submitted URLs.
Package api provides HTTP handlers for job chain management endpoints.
Purpose: - Expose chain CRUD and submission operations over the REST API.
Responsibilities: - Validate chain definitions before creation. - Normalize node request payloads onto the canonical submission contract. - Submit chain nodes through the shared operator-facing request-to-spec conversion layer.
Scope: - `/v1/chains` CRUD and submission routes only.
Usage: - Mounted by Server.Routes for chain management.
Invariants/Assumptions: - Chain node requests must stay on the same request contract as live job submissions. - Only chains with no active jobs can be deleted.
Package api provides HTTP handlers for crawl job endpoints.
Purpose: - Accept website crawl submissions over the API.
Responsibilities: - Validate crawl requests. - Build a crawl JobSpec with shared request defaults. - Create and enqueue the job, then return the sanitized record.
Scope: - Crawl request handling only; job execution lives in internal/jobs.
Usage: - Registered for POST /v1/crawl.
Invariants/Assumptions: - Requests must be JSON and include a URL. - Optional sitemap-only mode defaults to false.
Package api provides HTTP handlers for crawl state listing endpoints. Crawl state handlers support listing in-progress and completed crawl states with pagination support.
Package api provides deduplication API endpoints for the Spartan Scraper.
This file implements REST endpoints for querying content fingerprints and finding duplicate content across jobs.
Endpoints: - GET /v1/dedup/duplicates - Find duplicate content by simhash - GET /v1/dedup/history - Get content history for a URL - GET /v1/dedup/stats - Get deduplication statistics - DELETE /v1/dedup/job/{jobId} - Remove all dedup entries for a job
These endpoints enable clients to: - Detect duplicate content before crawling - Analyze content history across jobs - Monitor deduplication statistics - Clean up dedup data for deleted jobs
Package api centralizes shared health and recovery diagnostics.
Purpose: - Build canonical component health, diagnostic responses, and operator actions shared by REST, CLI, and MCP.
Responsibilities: - Classify browser, AI, and proxy-pool readiness with consistent operator-facing language. - Build read-only diagnostic responses and recovery actions for degraded optional subsystems. - Translate one-click recovery actions into surface-appropriate commands for CLI and MCP consumers.
Scope: - Shared diagnostics metadata and message generation only; HTTP handlers and CLI rendering live elsewhere.
Usage: - Used by `/healthz`, `/v1/diagnostics/*`, `spartan health`, and MCP diagnostics tools.
Invariants/Assumptions: - Optional subsystems should degrade without blocking core scraping workflows. - One-click diagnostic actions remain read-only and deterministic across surfaces.
Package api provides read-only recovery diagnostics for setup and degraded runtime states.
Purpose: - Expose safe one-click diagnostic endpoints and action builders that help operators recover inside the product.
Responsibilities: - Route browser, AI, and proxy-pool re-checks onto the shared diagnostic builders. - Keep setup-mode and normal-mode diagnostic responses consistent. - Preserve read-only semantics for repeated operator troubleshooting.
Scope: - HTTP diagnostic endpoint handling only; shared message construction lives in sibling helpers.
Usage: - Mounted by `Server.Routes()` for setup mode and normal mode surfaces.
Invariants/Assumptions: - Endpoints remain safe to call repeatedly. - Recovery actions should favor copy-ready commands plus browser-safe links.
Package api builds canonical export-outcome response envelopes.
Purpose: - Translate persisted export history records into operator-facing inspection payloads shared by REST, Web, CLI, and MCP.
Responsibilities: - Shape artifact metadata and optional inline content. - Derive export titles, messages, and recommended recovery actions. - Keep list and single-item export responses transport-consistent.
Scope: - Export response construction only; persistence and execution live elsewhere.
Usage: - Called by direct export handlers, export-history endpoints, CLI rendering, and MCP tools.
Invariants/Assumptions: - Inline content is included only when explicitly provided by the caller. - Collection responses always emit arrays, never null.
Package api provides HTTP handlers for automated export schedule management.
Purpose: - Manage export schedules and schedule history over HTTP.
Responsibilities: - Validate create/update requests for export schedules. - Return consistent JSON envelopes for list and history responses. - Normalize export defaults before persistence.
Scope: - CRUD handlers for export schedules and their history endpoints.
Usage: - Mounted under `/v1/export-schedules` and `/v1/export-schedules/{id}/history`.
Invariants/Assumptions: - JSON request bodies are decoded through shared strict helpers. - Cloud export defaults are applied consistently on create and update.
Package api provides HTTP handlers for health and diagnostic endpoints.
Purpose: - Expose structured setup, runtime, and optional-subsystem diagnostics for web, CLI, and automation clients.
Responsibilities: - Report overall service health and setup-mode state. - Classify component status as ok, degraded, disabled, setup_required, or error. - Surface non-fatal startup notices so operators can recover inside the product.
Scope: - Health response construction only; component initialization happens elsewhere.
Usage: - Mounted at `/healthz` by `Server.Routes()`.
Invariants/Assumptions: - Optional subsystems should degrade health instead of blocking startup by default. - Setup-mode servers must still answer `/healthz` with actionable recovery metadata.
Package api handles direct export inspection and delivery workflows.
Purpose: - Run direct exports, persist canonical export outcomes, and expose export inspection endpoints.
Responsibilities: - Validate direct-export requests and render export artifacts. - Persist export outcome history for API-triggered exports. - Return operator-facing export inspection envelopes instead of raw transport bodies. - Dispatch export-completed webhooks with the shared multipart contract when configured.
Scope: - Direct export API handlers only; raw result inspection stays in job_results.go.
Usage: - Mounted under POST /v1/jobs/{id}/export, GET /v1/jobs/{id}/exports, and GET /v1/exports/{id}.
Invariants/Assumptions: - Direct export failures that happen after request validation should still produce a persisted export outcome. - Inline artifact content is returned only on the immediate POST response.
Package api provides shared HTTP submission helpers for job-creation endpoints.
Purpose:
- Keep REST handlers thin while delegating operator-facing request conversion to the canonical internal/submission package.
Responsibilities: - Decode JSON requests and route them through shared submission builders. - Provide consistent request-default envelopes for live REST job creation. - Create and enqueue single jobs and batches with one response-shaping path.
Scope:
- API transport glue only; request validation and request-to-spec conversion live in internal/submission.
Usage: - Used by scrape, crawl, research, and batch API handlers.
Invariants/Assumptions: - REST creation endpoints should use the same operator-facing conversion rules as other surfaces. - Live REST submissions always resolve auth before persisting jobs.
Package api centralizes sanitized job and batch response envelope construction.
Purpose: - Provide one canonical response-shaping path for job and batch surfaces.
Responsibilities: - Wrap sanitized jobs in stable single-item and collection envelopes. - Derive run-history, queue-progression, and failure-context fields from persisted jobs and batches. - Shape batch metadata, aggregated stats, progress, and optional included job pages. - Keep REST, CLI direct-mode, and MCP outputs aligned so clients do not branch on transport-specific payloads.
Scope: - Response construction only; persistence and execution remain in other packages.
Usage: - Called by REST handlers, CLI direct-mode helpers, and MCP tool handlers before encoding responses.
Invariants/Assumptions: - Jobs are always sanitized before they leave trusted server boundaries. - Collection responses always emit arrays, never null. - Batch envelopes always carry aggregate stats and explicit progress, even when individual jobs are omitted.
Package api provides shared helpers for job result access across API handlers.
Purpose: - Centralize job-result readiness checks, path validation, and file opening.
Responsibilities: - Normalize status-based "results unavailable" errors. - Validate recorded result paths before handlers read from disk. - Provide reusable helpers for opening result files safely.
Scope: - API-layer result readers such as export, transform preview, and traffic replay.
Usage: - Call requireJobResultFile when a handler needs precise file/empty-file messaging. - Call openJobResultFile when a helper needs a validated readable file handle.
Invariants/Assumptions: - Result paths must stay within the store data directory. - Queued/running/failed/canceled jobs do not have readable results.
Package api provides HTTP handlers for job result retrieval endpoints. Job result handlers support retrieving persisted raw results for inspection and pagination. Shaped/transformed direct exports are handled separately by POST /v1/jobs/{id}/export.
Package api provides thin submission-contract adapters for schedules and non-HTTP callers.
Purpose:
- Keep schedule CRUD and MCP adapters aligned with the canonical operator-facing submission conversion that now lives in internal/submission.
Responsibilities: - Reconstruct public request payloads from persisted typed specs. - Convert raw schedule request JSON into typed specs using shared defaults. - Expose narrow forwarding helpers for existing non-HTTP API callers.
Scope: - Thin adapters only; validation and request-to-spec conversion live in internal/submission.
Usage: - Used by REST schedule handlers, MCP handlers, and tests.
Invariants/Assumptions: - Schedule payloads must stay on the same operator-facing request contract as live submissions. - Behavior changes should land in internal/submission first and flow through these adapters.
Package api provides HTTP handlers for job listing and management endpoints.
Purpose: - Expose canonical recent-run inspection and job control routes over HTTP.
Responsibilities: - List persisted jobs with pagination and optional status filtering. - List recent failed jobs with operator-meaningful failure context. - Retrieve single job details and cancel or force-delete jobs. - Route all job responses through the shared sanitized observability builders.
Scope: - Job transport handling only; persistence and execution stay in internal/store and internal/jobs.
Usage: - Registered for /v1/jobs, /v1/jobs/failures, and /v1/jobs/* routes.
Invariants/Assumptions: - Job list responses always return paginated envelopes. - Failure inspection is a filtered view over persisted failed jobs, not a separate history store. - Force delete permanently removes persisted job data and artifacts.
Package api provides the REST API server for Spartan Scraper. This file implements the metrics collection layer for performance monitoring and rate limiting visibility.
Responsibilities: - Collecting request metrics (duration, success/failure, fetcher type, host) - Tracking fetcher usage breakdown (HTTP, Chromedp, Playwright) - Recording job duration histograms - Exposing rate limiter state for all known hosts - Providing thread-safe access to metrics snapshots
This file does NOT: - Persist metrics to disk (all in-memory with circular buffers) - Expose raw URLs (only host-level aggregation) - Provide historical data beyond the configured retention window
Invariants: - All metrics operations are thread-safe via RWMutex - Circular buffers have fixed capacity to cap memory usage - Host names are sanitized before storage (no query params, paths)
Package api provides HTTP handlers for metrics endpoints. The metrics handler exposes performance and rate limiting data for monitoring.
Package api implements the REST API server for Spartan Scraper. It provides endpoints for enqueuing jobs, managing auth profiles, and retrieving job status and results.
Package api provides REST API handlers for pipeline JavaScript management.
Purpose: - Expose file-backed CRUD endpoints for pipeline JS scripts.
Responsibilities: - Route collection and item requests for /v1/pipeline-js. - Reuse the shared named-resource CRUD helpers with pipeline JS storage.
Scope: - HTTP handler wiring only; script persistence and validation live in internal/pipeline.
Usage: - Registered by the API server for GET/POST collection requests and GET/PUT/DELETE item requests.
Invariants/Assumptions: - Script names are stable identifiers carried in the URL path. - Validation is delegated to the pipeline package before files are written.
Package api implements the REST API server for Spartan Scraper.
Package api provides REST API handlers for render profile management.
Purpose: - Expose file-backed CRUD endpoints for render profiles.
Responsibilities: - Route collection and item requests for /v1/render-profiles. - Reuse the shared named-resource CRUD helpers with render-profile storage.
Scope: - HTTP handler wiring only; render-profile persistence and validation live in internal/fetch.
Usage: - Registered by the API server for GET/POST collection requests and GET/PUT/DELETE item requests.
Invariants/Assumptions: - Profile names are stable identifiers carried in the URL path. - Validation is delegated to the fetch package before files are written.
Package api provides shared HTTP request helpers for Spartan Scraper's API.
Purpose: - Centralize repeated request/response patterns used by API handlers.
Responsibilities: - Decode strict JSON request bodies consistently. - Write common JSON responses such as 201 Created payloads. - Provide reusable named-resource CRUD scaffolding for file-backed API endpoints. - Centralize path, pagination, and collection-mapping helpers for resource handlers.
Scope: - Helper utilities for handler implementations in this package only.
Usage:
- Call decodeJSONBody for JSON request parsing and handleNamedResourceCollection/ handleNamedResourceItem for collection-style CRUD endpoints.
Invariants/Assumptions: - JSON endpoints require application/json content types. - Unknown JSON fields are rejected. - Named resources are addressed by the path segment immediately after a stable route prefix.
Package api provides HTTP handlers for research job endpoints.
Purpose: - Accept multi-source research submissions over the API.
Responsibilities: - Validate research requests. - Build a research JobSpec with shared request defaults. - Create and enqueue the job, then return the sanitized record.
Scope: - Research request handling only; job execution lives in internal/jobs.
Usage: - Registered for POST /v1/research.
Invariants/Assumptions: - Requests must be JSON and include both a query and at least one URL. - Auth resolution for research uses the first target URL as the host context.
Package api provides HTTP handlers for retention management endpoints.
This file is responsible for: - Providing retention status information - Running manual cleanup operations with dry-run support
This file does NOT handle: - Scheduled cleanup execution (scheduler handles this) - Policy evaluation logic (retention package handles this)
Invariants: - Cleanup defaults to dry-run mode for safety - Force flag can override disabled retention - All errors use apperrors package for consistent handling
Package api builds capability-aware retention guidance shared by operator surfaces.
Purpose: - Derive operator-facing retention guidance from retention status metrics.
Responsibilities: - Classify disabled, warning, danger, and healthy retention states. - Attach actionable next steps that work across API, Web, and CLI surfaces. - Keep retention guidance language consistent with Settings-route recovery flows.
Scope: - Retention guidance derivation only; retention execution stays in the retention package.
Usage: - Called by retention API handlers and CLI status rendering.
Invariants/Assumptions: - Disabled retention is optional, not a failure. - Warning and danger states should recommend previewing cleanup before destructive action.
Package api provides HTTP handlers for scheduled job management endpoints.
Purpose: - Manage recurring scrape, crawl, and research schedules over HTTP.
Responsibilities: - Validate schedule creation inputs. - Translate operator-facing job requests into scheduler storage records. - Return consistent JSON responses for list/create/delete operations.
Scope: - Direct schedule CRUD handlers in the API package.
Usage: - Mounted under `/v1/schedules` and `/v1/schedules/{id}` by the API router.
Invariants/Assumptions: - Schedule creation requests use strict JSON decoding via shared helpers. - Supported kinds are scrape, crawl, and research.
Package api provides HTTP handlers for scrape job endpoints.
Purpose: - Accept single-page scrape submissions over the API.
Responsibilities: - Validate scrape requests. - Build a scrape JobSpec with shared request defaults. - Create and enqueue the job, then return the sanitized record.
Scope: - Scrape request handling only; job execution lives in internal/jobs.
Usage: - Registered for POST /v1/scrape.
Invariants/Assumptions: - Requests must be JSON and include a URL. - Method defaults to GET when omitted.
Package api implements the REST API server for Spartan Scraper.
Purpose: - Wire runtime services into the canonical HTTP surface for operator and browser clients.
Responsibilities: - Construct the API server with its backing store, manager, scheduler, and optional webhook dispatcher. - Register all HTTP routes exposed by the application. - Own server lifecycle concerns such as startup, shutdown, and HTTP listener configuration.
Scope: - Server assembly and route registration only; individual handler behavior lives in sibling files.
Usage: - Created by CLI/server entrypoints and exercised by API/system tests.
Invariants/Assumptions: - Route registration is the canonical REST surface for the product. - Long-running runtime dependencies are initialized before the server begins handling requests.
Package api implements the REST API server for Spartan Scraper. This file handles template preview endpoints for visual selector building.
Package api provides HTTP handlers for extraction template listing endpoints. Template handlers support listing available extraction templates for use in scrape, crawl, and research jobs.
Package api provides HTTP handlers for job result transformation endpoints. This file handles transformation preview using JMESPath and JSONata expressions.
Package api provides HTTP request and response types for the Spartan Scraper API.
Purpose: - Define stable request and response contracts shared across REST handlers, MCP adapters, and CLI direct-mode helpers.
Responsibilities: - Hold operator-facing request payload types for scrape, crawl, research, schedules, and batches. - Hold stable job and batch response envelope types used across transports. - Define derived observability payloads for recent runs, queue progression, and failure context.
Scope: - JSON contracts only; handler logic and response construction live elsewhere in this package.
Usage: - Imported by REST handlers, MCP handlers, CLI direct-mode helpers, tests, and OpenAPI maintenance work.
Invariants/Assumptions: - Job and batch automation surfaces should reuse the same response envelope shapes. - Response envelopes expose sanitized jobs rather than persisted raw records. - Derived run-history fields are transport-safe and never reveal host-local artifact paths.
Package api provides shared HTTP utility functions for the Spartan Scraper API.
Purpose: - Centralize low-level HTTP helpers that are reused across API handlers.
Responsibilities: - Decode and encode JSON payloads consistently. - Parse common request parameters and normalize response metadata. - Provide shared error writing, request ID, and content-type helpers.
Scope: - Transport-level API utilities only.
Usage: - Imported implicitly across the api package by request handlers and tests.
Invariants/Assumptions: - Error responses use the shared request ID envelope. - JSON decoding should reject malformed or oversized request bodies consistently.
Package api provides HTTP handlers for watch monitoring endpoints.
Purpose: - Expose CRUD and manual-check endpoints for configured watches.
Responsibilities: - Decode and normalize watch API requests. - Map storage not-found errors to stable HTTP responses. - Return consistent watch and check-result payloads.
Scope: - `/v1/watch` and nested watch detail routes only.
Usage: - Mounted by Server.Routes for watch management and manual checks.
Invariants/Assumptions: - Create requests apply schema defaults for omitted optionals. - Update requests preserve existing optional values when they are omitted.
Package api provides watch artifact response shaping and download handlers.
Purpose: - Convert internal watch artifact metadata into stable public response shapes.
Responsibilities: - Build watch-check artifact descriptors with download URLs. - Serve persisted watch artifacts through explicit download endpoints. - Hide host-local filesystem paths from public watch-check responses.
Scope: - `/v1/watch/{id}/artifacts/{artifactKind}` and watch-check response shaping.
Usage:
- Called by the watch handlers after manual checks and by the nested watch artifact route.
Invariants/Assumptions: - Public responses expose download URLs, never host-local paths. - Artifact downloads are resolved only from deterministic watch artifact files. - Artifact kinds are limited to the canonical watch artifact enum.
Package api provides HTTP handlers for persisted watch-check inspection routes.
Purpose: - Expose watch history list, detail, and artifact-download endpoints for operator inspection.
Responsibilities: - Paginate persisted watch check records. - Return canonical watch inspection envelopes with narratives and actions. - Serve check-scoped artifact snapshots without exposing host-local filesystem paths.
Scope: - `/v1/watch/{id}/history`, `/v1/watch/{id}/history/{checkId}`, and nested history artifact routes only.
Usage: - Mounted through the watch detail wrapper in Server.Routes.
Invariants/Assumptions: - Watch history is scoped to an existing watch. - History artifact downloads resolve from persisted check snapshots, not the mutable latest-artifact slots.
Package api builds canonical watch-check inspection response envelopes.
Purpose: - Translate persisted watch history records into operator-facing inspection payloads shared by REST, Web, CLI, and MCP.
Responsibilities: - Shape check narratives, artifact download metadata, and recommended next steps. - Keep list and single-item watch history responses transport-consistent. - Preserve watch-history details without exposing host-local artifact paths.
Scope: - Watch history response construction only; persistence and execution live in internal/watch.
Usage: - Called by watch history API handlers, CLI rendering, and MCP watch inspection tools.
Invariants/Assumptions: - Collection responses always emit arrays, never null. - Artifact downloads are check-scoped so historical records remain stable after later checks rotate the latest artifacts.
Package api centralizes watch response envelope construction.
Purpose: - Provide one canonical response-shaping path for watch collections and single-watch payloads.
Responsibilities: - Normalize watch response arrays for JSON collection envelopes. - Wrap paginated watch collections with stable metadata used across REST, Web, and MCP. - Reuse the existing watch response mapping so transports stay aligned.
Scope: - Watch configuration response construction only; persistence and execution remain elsewhere.
Usage: - Called by REST handlers and MCP tool handlers before encoding watch payloads.
Invariants/Assumptions: - Collection responses always emit arrays, never null. - Watch payloads should match the API-facing watch schema instead of exposing raw storage structs.
Package api provides HTTP handlers for webhook delivery history endpoints. These endpoints allow querying webhook delivery records for debugging and monitoring.
Package api provides WebSocket support for real-time job updates.
This module handles: - WebSocket connection lifecycle (upgrade, read, write, close) - Client connection management (register, unregister, broadcast) - Message framing and JSON serialization - Heartbeat/ping-pong for connection health
Invariants: - All Hub methods are goroutine-safe - Client send channel has fixed buffer; slow clients are disconnected - Hub must be started with Run() before use
Index ¶
- Constants
- func ApplyTransformation(data []any, expression, language string) ([]any, error)
- func DiagnosticActionPath(target string) string
- func DiagnosticTargetFromActionValue(value string) string
- func GetAPIKeyFromContext(ctx context.Context) (auth.APIKey, bool)
- func JobSpecFromCrawlRequest(cfg config.Config, defaults JobSubmissionDefaults, req CrawlRequest) (jobs.JobSpec, error)
- func JobSpecFromResearchRequest(cfg config.Config, defaults JobSubmissionDefaults, req ResearchRequest) (jobs.JobSpec, error)
- func JobSpecFromScrapeRequest(cfg config.Config, defaults JobSubmissionDefaults, req ScrapeRequest) (jobs.JobSpec, error)
- func NormalizeDiagnosticTarget(target string) string
- type AIExportShapeRequest
- type AIExportShapeResponse
- type AIExtractPreviewRequest
- type AIExtractPreviewResponse
- type AIExtractTemplateDebugRequest
- type AIExtractTemplateDebugResponse
- type AIExtractTemplateGenerateRequest
- type AIExtractTemplateGenerateResponse
- type AIHealthChecker
- type AIPipelineJSDebugRequest
- type AIPipelineJSDebugResponse
- type AIPipelineJSGenerateRequest
- type AIPipelineJSGenerateResponse
- type AIRenderProfileDebugRequest
- type AIRenderProfileDebugResponse
- type AIRenderProfileGenerateRequest
- type AIRenderProfileGenerateResponse
- type AIResearchRefineRequest
- type AIResearchRefineResponse
- type AITransformGenerateRequest
- type AITransformGenerateResponse
- type BatchCrawlRequest
- type BatchJobRequest
- type BatchListResponse
- type BatchProgress
- type BatchResearchRequest
- type BatchResponse
- func BuildBatchResponse(batch model.Batch, stats model.BatchJobStats, jobs []model.Job, total int, ...) BatchResponse
- func BuildCreatedBatchResponse(batchID string, kind model.Kind, createdJobs []model.Job) BatchResponse
- func BuildStoreBackedBatchResponse(ctx context.Context, st *store.Store, batch model.Batch, ...) (BatchResponse, error)
- type BatchScrapeRequest
- type BatchSummary
- type CapabilityGuidance
- type ChainCreateRequest
- type ChainSubmitRequest
- type ChainSubmitResponse
- type Client
- type ComponentStatus
- type CrawlRequest
- type CrawlStateResponse
- type CreateTemplateRequest
- type DOMElement
- type DOMNode
- type DiagnosticActionResponse
- func BuildAIDiagnosticResponse(ctx context.Context, cfg config.Config, aiExtractor AIHealthChecker) DiagnosticActionResponse
- func BuildBrowserDiagnosticResponse(cfg config.Config) DiagnosticActionResponse
- func BuildProxyPoolDiagnosticResponse(cfg config.Config, runtimeState ProxyPoolRuntimeState) DiagnosticActionResponse
- type ErrorResponse
- type ExportArtifact
- type ExportFailureContext
- type ExportInspection
- type ExportOutcomeListResponse
- type ExportOutcomeResponse
- type ExportScheduleRequest
- type ExportScheduleResponse
- type ExportScheduleRuntime
- type FetcherUsageBreakdown
- type HealthResponse
- type Hub
- func (h *Hub) Broadcast(msg WSMessage)
- func (h *Hub) BroadcastJobEvent(event JobEvent)
- func (h *Hub) BroadcastManagerStatus(queuedJobs, activeJobs int)
- func (h *Hub) BroadcastMetrics(snapshot MetricsSnapshot)
- func (h *Hub) ClientCount() int
- func (h *Hub) NewClient(conn net.Conn) *Client
- func (h *Hub) Run()
- func (h *Hub) Stop()
- func (h *Hub) Wait()
- type InspectableJob
- type JobEvent
- type JobEventPayload
- type JobEventType
- type JobFailureContext
- type JobListResponse
- type JobProgress
- type JobQueueProgress
- type JobResponse
- type JobRunSummary
- type JobSubmissionDefaults
- type ManagerStatusPayload
- type MetricsCollector
- func (m *MetricsCollector) Callback() func(duration time.Duration, success bool, fetcherType, rawURL string)
- func (m *MetricsCollector) GetRateLimitStatus() []RateLimitStatus
- func (m *MetricsCollector) GetSnapshot() MetricsSnapshot
- func (m *MetricsCollector) RecordJobDuration(duration time.Duration)
- func (m *MetricsCollector) RecordRequest(duration time.Duration, success bool, fetcherType, rawURL string)
- func (m *MetricsCollector) RegisterHostLimiter(host string, limiter *rate.Limiter, qps float64, burst int)
- func (m *MetricsCollector) Reset()
- func (m *MetricsCollector) SetDefaultRateLimit(qps float64, burst int)
- func (m *MetricsCollector) StartRequest()
- func (m *MetricsCollector) SyncHostLimiters(hostLimiter interface{ ... })
- type MetricsSnapshot
- type OAuthCallbackResponse
- type OAuthInitiateRequest
- type OAuthInitiateResponse
- type OAuthRefreshRequest
- type OIDCDiscoverRequest
- type ProxyPoolProvider
- type ProxyPoolRuntimeState
- type ProxyPoolStatusResponse
- type ProxyStatus
- type RateLimitStatus
- type RecommendedAction
- type RequestMetric
- type ResearchRequest
- type RetentionCleanupRequest
- type RetentionCleanupResponse
- type RetentionStatusResponse
- type RingBuffer
- type RuntimeNotice
- type ScheduleRequest
- type ScheduleResponse
- type ScrapeRequest
- type Server
- type SetupStatus
- type StatusResponse
- type TemplatePreviewResponse
- type TemplateResponse
- type TestSelectorRequest
- type TestSelectorResponse
- type TransformPreviewRequest
- type TransformPreviewResponse
- type TransformValidateRequest
- type TransformValidateResponse
- type WSMessage
- type WSMessageType
- type WatchArtifactResponse
- type WatchCheckHistoryResponse
- type WatchCheckInspection
- type WatchCheckInspectionResponse
- type WatchCheckStatus
- type WatchListResponse
- type WatchRequest
- type WatchResponse
- type WebhookConfig
Constants ¶
const ( DiagnosticTargetBrowser = "browser" DiagnosticTargetAI = "ai" DiagnosticTargetProxyPool = "proxy_pool" )
const ( // DefaultMetricsBufferSize is the default size for circular buffers DefaultMetricsBufferSize = 1000 // DefaultMetricsRetention is the time window for metrics calculations DefaultMetricsRetention = 5 * time.Minute )
const ( ActionKindRoute = "route" ActionKindCommand = "command" ActionKindEnv = "env" ActionKindCopy = "copy" ActionKindDoc = "doc" ActionKindExternalLink = "external-link" ActionKindOneClick = "one-click" )
Variables ¶
This section is empty.
Functions ¶
func ApplyTransformation ¶
ApplyTransformation applies a transformation expression to data. This wrapper preserves existing api package call sites and tests.
func DiagnosticActionPath ¶
func GetAPIKeyFromContext ¶
GetAPIKeyFromContext retrieves the API key from the request context Returns the APIKey and true if found, zero value and false otherwise
func JobSpecFromCrawlRequest ¶
func JobSpecFromCrawlRequest(cfg config.Config, defaults JobSubmissionDefaults, req CrawlRequest) (jobs.JobSpec, error)
func JobSpecFromResearchRequest ¶
func JobSpecFromResearchRequest(cfg config.Config, defaults JobSubmissionDefaults, req ResearchRequest) (jobs.JobSpec, error)
func JobSpecFromScrapeRequest ¶
func JobSpecFromScrapeRequest(cfg config.Config, defaults JobSubmissionDefaults, req ScrapeRequest) (jobs.JobSpec, error)
Types ¶
type AIExportShapeRequest ¶
type AIExportShapeRequest struct {
JobID string `json:"job_id"`
Format string `json:"format"`
CurrentShape exporter.ShapeConfig `json:"currentShape,omitempty"`
Instructions string `json:"instructions,omitempty"`
}
type AIExportShapeResponse ¶
type AIExportShapeResponse struct {
Issues []string `json:"issues,omitempty"`
InputStats aiauthoring.ExportShapeInputStats `json:"inputStats"`
Shape exporter.ShapeConfig `json:"shape"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
type AIExtractPreviewRequest ¶
type AIExtractPreviewRequest struct {
URL string `json:"url"`
HTML string `json:"html,omitempty"` // Optional: provide HTML directly
Mode extract.AIExtractionMode `json:"mode"`
Prompt string `json:"prompt,omitempty"`
Schema map[string]interface{} `json:"schema,omitempty"`
Fields []string `json:"fields,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
AIExtractPreviewRequest for POST /v1/ai/extract-preview
type AIExtractPreviewResponse ¶
type AIExtractPreviewResponse struct {
Fields map[string]extract.FieldValue `json:"fields"`
Confidence float64 `json:"confidence"`
Explanation string `json:"explanation,omitempty"`
TokensUsed int `json:"tokens_used"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
Cached bool `json:"cached"`
VisualContextUsed bool `json:"visual_context_used"`
}
AIExtractPreviewResponse for preview endpoint
type AIExtractTemplateDebugRequest ¶
type AIExtractTemplateDebugRequest struct {
URL string `json:"url,omitempty"`
HTML string `json:"html,omitempty"`
Template extract.Template `json:"template"`
Instructions string `json:"instructions,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
type AIExtractTemplateDebugResponse ¶
type AIExtractTemplateDebugResponse struct {
Issues []string `json:"issues,omitempty"`
ExtractedFields map[string]extract.FieldValue `json:"extracted_fields,omitempty"`
Explanation string `json:"explanation,omitempty"`
SuggestedTemplate *extract.Template `json:"suggested_template,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
}
type AIExtractTemplateGenerateRequest ¶
type AIExtractTemplateGenerateRequest struct {
URL string `json:"url,omitempty"`
HTML string `json:"html,omitempty"`
Description string `json:"description"`
SampleFields []string `json:"sample_fields,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
AIExtractTemplateGenerateRequest for POST /v1/ai/template-generate
type AIExtractTemplateGenerateResponse ¶
type AIExtractTemplateGenerateResponse struct {
Template extract.Template `json:"template"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
}
AIExtractTemplateGenerateResponse for template generation
type AIHealthChecker ¶
type AIHealthChecker interface {
HealthStatus(ctx context.Context) (extract.AIHealthSnapshot, error)
}
AIHealthChecker represents the minimal AI extractor health contract shared across surfaces.
type AIPipelineJSDebugRequest ¶
type AIPipelineJSDebugRequest struct {
URL string `json:"url"`
Script pipeline.JSTargetScript `json:"script"`
Instructions string `json:"instructions,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
type AIPipelineJSDebugResponse ¶
type AIPipelineJSDebugResponse struct {
Issues []string `json:"issues,omitempty"`
ResolvedGoal *aiauthoring.ResolvedGoal `json:"resolved_goal,omitempty"`
Explanation string `json:"explanation,omitempty"`
SuggestedScript *pipeline.JSTargetScript `json:"suggested_script,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
RecheckStatus int `json:"recheck_status,omitempty"`
RecheckEngine string `json:"recheck_engine,omitempty"`
RecheckError string `json:"recheck_error,omitempty"`
}
type AIPipelineJSGenerateRequest ¶
type AIPipelineJSGenerateRequest struct {
URL string `json:"url"`
Name string `json:"name,omitempty"`
HostPatterns []string `json:"host_patterns,omitempty"`
Instructions string `json:"instructions,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
type AIPipelineJSGenerateResponse ¶
type AIPipelineJSGenerateResponse struct {
Script pipeline.JSTargetScript `json:"script"`
ResolvedGoal *aiauthoring.ResolvedGoal `json:"resolved_goal,omitempty"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
}
type AIRenderProfileDebugRequest ¶
type AIRenderProfileDebugRequest struct {
URL string `json:"url"`
Profile fetch.RenderProfile `json:"profile"`
Instructions string `json:"instructions,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
type AIRenderProfileDebugResponse ¶
type AIRenderProfileDebugResponse struct {
Issues []string `json:"issues,omitempty"`
ResolvedGoal *aiauthoring.ResolvedGoal `json:"resolved_goal,omitempty"`
Explanation string `json:"explanation,omitempty"`
SuggestedProfile *fetch.RenderProfile `json:"suggested_profile,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
RecheckStatus int `json:"recheck_status,omitempty"`
RecheckEngine string `json:"recheck_engine,omitempty"`
RecheckError string `json:"recheck_error,omitempty"`
}
type AIRenderProfileGenerateRequest ¶
type AIRenderProfileGenerateRequest struct {
URL string `json:"url"`
Name string `json:"name,omitempty"`
HostPatterns []string `json:"host_patterns,omitempty"`
Instructions string `json:"instructions,omitempty"`
Images []extract.AIImageInput `json:"images,omitempty"`
Headless bool `json:"headless,omitempty"`
UsePlaywright bool `json:"playwright,omitempty"`
Visual bool `json:"visual,omitempty"`
}
type AIRenderProfileGenerateResponse ¶
type AIRenderProfileGenerateResponse struct {
Profile fetch.RenderProfile `json:"profile"`
ResolvedGoal *aiauthoring.ResolvedGoal `json:"resolved_goal,omitempty"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
VisualContextUsed bool `json:"visual_context_used"`
}
type AIResearchRefineRequest ¶
type AIResearchRefineResponse ¶
type AIResearchRefineResponse struct {
Issues []string `json:"issues,omitempty"`
InputStats aiauthoring.ResearchRefineInputStats `json:"inputStats"`
Refined piai.ResearchRefinedContent `json:"refined"`
Markdown string `json:"markdown"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
type AITransformGenerateRequest ¶
type AITransformGenerateRequest struct {
JobID string `json:"job_id"`
CurrentTransform exporter.TransformConfig `json:"currentTransform,omitempty"`
PreferredLanguage string `json:"preferredLanguage,omitempty"`
Instructions string `json:"instructions,omitempty"`
}
type AITransformGenerateResponse ¶
type AITransformGenerateResponse struct {
Issues []string `json:"issues,omitempty"`
InputStats aiauthoring.TransformInputStats `json:"inputStats"`
Transform exporter.TransformConfig `json:"transform"`
Preview []any `json:"preview,omitempty"`
Explanation string `json:"explanation,omitempty"`
RouteID string `json:"route_id,omitempty"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
type BatchCrawlRequest ¶
type BatchCrawlRequest = submission.BatchCrawlRequest
BatchCrawlRequest creates multiple crawl jobs.
type BatchJobRequest ¶
type BatchJobRequest = submission.BatchJobRequest
BatchJobRequest represents a single job within a batch.
type BatchListResponse ¶
type BatchListResponse struct {
Batches []BatchSummary `json:"batches"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
BatchListResponse represents a paginated collection of aggregate batch summaries.
func BuildBatchListResponse ¶
func BuildBatchListResponse(batches []model.Batch, stats []model.BatchJobStats, total int, limit int, offset int) BatchListResponse
BuildBatchListResponse returns the canonical paginated batch-summary collection envelope.
type BatchProgress ¶
type BatchProgress struct {
Completed int `json:"completed"`
Remaining int `json:"remaining"`
Percent int `json:"percent"`
}
BatchProgress exposes explicit queue progress for a batch.
type BatchResearchRequest ¶
type BatchResearchRequest = submission.BatchResearchRequest
BatchResearchRequest creates multiple research jobs.
type BatchResponse ¶
type BatchResponse struct {
Batch BatchSummary `json:"batch"`
Jobs []InspectableJob `json:"jobs"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
BatchResponse represents a stable batch envelope shared by create/get/cancel flows.
func BuildBatchResponse ¶
func BuildBatchResponse(batch model.Batch, stats model.BatchJobStats, jobs []model.Job, total int, limit int, offset int) BatchResponse
BuildBatchResponse returns the canonical batch response envelope.
func BuildCreatedBatchResponse ¶
func BuildCreatedBatchResponse(batchID string, kind model.Kind, createdJobs []model.Job) BatchResponse
BuildCreatedBatchResponse returns the canonical batch envelope immediately after submission.
func BuildStoreBackedBatchResponse ¶
func BuildStoreBackedBatchResponse(ctx context.Context, st *store.Store, batch model.Batch, stats model.BatchJobStats, jobs []model.Job, total int, limit int, offset int) (BatchResponse, error)
BuildStoreBackedBatchResponse returns the canonical batch response envelope with batch-aware job enrichment.
type BatchScrapeRequest ¶
type BatchScrapeRequest = submission.BatchScrapeRequest
BatchScrapeRequest creates multiple scrape jobs.
type BatchSummary ¶
type BatchSummary struct {
ID string `json:"id"`
Kind string `json:"kind"`
Status string `json:"status"`
JobCount int `json:"jobCount"`
Stats model.BatchJobStats `json:"stats"`
Progress BatchProgress `json:"progress"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
BatchSummary represents aggregate batch metadata and status.
func BuildBatchSummary ¶
func BuildBatchSummary(batch model.Batch, stats model.BatchJobStats) BatchSummary
BuildBatchSummary returns the canonical aggregate batch summary.
type CapabilityGuidance ¶
type CapabilityGuidance struct {
Status string `json:"status"`
Title string `json:"title,omitempty"`
Message string `json:"message,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
CapabilityGuidance describes a capability-aware explanation and follow-up actions.
func BuildRetentionCapabilityGuidance ¶
func BuildRetentionCapabilityGuidance(status RetentionStatusResponse) *CapabilityGuidance
type ChainCreateRequest ¶
type ChainCreateRequest struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Definition model.ChainDefinition `json:"definition"`
}
ChainCreateRequest represents a request to create a new chain.
type ChainSubmitRequest ¶
type ChainSubmitRequest struct {
Overrides map[string]json.RawMessage `json:"overrides,omitempty"`
}
ChainSubmitRequest represents a request to submit/instantiate a chain.
type ChainSubmitResponse ¶
ChainSubmitResponse represents the response from submitting a chain.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client represents a single WebSocket connection.
func (*Client) IsSubscribed ¶
IsSubscribed returns whether the client is subscribed to job events.
type ComponentStatus ¶
type ComponentStatus struct {
Status string `json:"status"`
Message string `json:"message,omitempty"`
Details interface{} `json:"details,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
ComponentStatus represents the health of a single system component.
func BuildAIComponentStatus ¶
func BuildAIComponentStatus(ctx context.Context, cfg config.Config, aiExtractor AIHealthChecker) ComponentStatus
func BuildBrowserComponentStatus ¶
func BuildBrowserComponentStatus(cfg config.Config) ComponentStatus
func BuildProxyPoolComponentStatus ¶
func BuildProxyPoolComponentStatus(cfg config.Config, runtimeState ProxyPoolRuntimeState) ComponentStatus
type CrawlRequest ¶
type CrawlRequest = submission.CrawlRequest
CrawlRequest represents a request to crawl a website.
type CrawlStateResponse ¶
type CrawlStateResponse struct {
URL string `json:"url"`
ETag string `json:"etag"`
LastModified string `json:"lastModified"`
ContentHash string `json:"contentHash"`
LastScraped time.Time `json:"lastScraped"`
Depth int `json:"depth,omitempty"`
JobID string `json:"jobId,omitempty"`
}
CrawlStateResponse exposes the public crawl-state contract.
type CreateTemplateRequest ¶
type CreateTemplateRequest struct {
Name string `json:"name"`
Selectors []extract.SelectorRule `json:"selectors"`
JSONLD []extract.JSONLDRule `json:"jsonld,omitempty"`
Regex []extract.RegexRule `json:"regex,omitempty"`
Normalize extract.NormalizeSpec `json:"normalize,omitempty"`
}
CreateTemplateRequest for POST /v1/templates
type DOMElement ¶
type DOMElement struct {
Tag string `json:"tag"`
Text string `json:"text"`
HTML string `json:"html"` // truncated preview
Path string `json:"path"`
}
DOMElement represents a matched element in selector testing
type DOMNode ¶
type DOMNode struct {
Tag string `json:"tag"`
ID string `json:"id,omitempty"`
Classes []string `json:"classes,omitempty"`
Attributes map[string]string `json:"attributes,omitempty"`
Text string `json:"text,omitempty"`
Children []DOMNode `json:"children,omitempty"`
Path string `json:"path"` // CSS path to this element
Depth int `json:"depth"`
}
DOMNode represents a simplified DOM element for the visual builder
type DiagnosticActionResponse ¶
type DiagnosticActionResponse struct {
Status string `json:"status"`
Title string `json:"title,omitempty"`
Message string `json:"message"`
Details interface{} `json:"details,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
DiagnosticActionResponse represents the result of a safe read-only diagnostic action.
func BuildAIDiagnosticResponse ¶
func BuildAIDiagnosticResponse(ctx context.Context, cfg config.Config, aiExtractor AIHealthChecker) DiagnosticActionResponse
func BuildBrowserDiagnosticResponse ¶
func BuildBrowserDiagnosticResponse(cfg config.Config) DiagnosticActionResponse
func BuildProxyPoolDiagnosticResponse ¶
func BuildProxyPoolDiagnosticResponse(cfg config.Config, runtimeState ProxyPoolRuntimeState) DiagnosticActionResponse
type ErrorResponse ¶
type ErrorResponse struct {
Error string `json:"error"`
RequestID string `json:"requestId,omitempty"`
}
ErrorResponse represents a standard error response.
type ExportArtifact ¶
type ExportArtifact struct {
Format string `json:"format"`
Filename string `json:"filename"`
ContentType string `json:"contentType"`
RecordCount int `json:"recordCount,omitempty"`
Size int64 `json:"size,omitempty"`
Encoding string `json:"encoding,omitempty"`
Content string `json:"content,omitempty"`
}
ExportArtifact describes a rendered export artifact and optional inline content.
type ExportFailureContext ¶
type ExportFailureContext = exporter.FailureContext
ExportFailureContext aliases the shared exporter failure context for transport-safe responses.
type ExportInspection ¶
type ExportInspection struct {
ID string `json:"id"`
ScheduleID string `json:"scheduleId,omitempty"`
JobID string `json:"jobId"`
Trigger string `json:"trigger"`
Status string `json:"status"`
Title string `json:"title"`
Message string `json:"message"`
Destination string `json:"destination,omitempty"`
ExportedAt time.Time `json:"exportedAt"`
CompletedAt *time.Time `json:"completedAt,omitempty"`
RetryCount int `json:"retryCount"`
Request exporter.ResultExportConfig `json:"request"`
Artifact *ExportArtifact `json:"artifact,omitempty"`
Failure *ExportFailureContext `json:"failure,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
ExportInspection represents a persisted, operator-facing export outcome.
func BuildExportInspection ¶
func BuildExportInspection(record scheduler.ExportRecord, content []byte) ExportInspection
type ExportOutcomeListResponse ¶
type ExportOutcomeListResponse struct {
Exports []ExportInspection `json:"exports"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
ExportOutcomeListResponse represents a paginated export outcome collection.
func BuildExportOutcomeListResponse ¶
func BuildExportOutcomeListResponse(records []scheduler.ExportRecord, total, limit, offset int) ExportOutcomeListResponse
type ExportOutcomeResponse ¶
type ExportOutcomeResponse struct {
Export ExportInspection `json:"export"`
}
ExportOutcomeResponse represents a single export outcome envelope.
type ExportScheduleRequest ¶
type ExportScheduleRequest struct {
Name string `json:"name"`
Enabled *bool `json:"enabled,omitempty"`
Filters scheduler.ExportFilters `json:"filters"`
Export scheduler.ExportConfig `json:"export"`
Retry *scheduler.ExportRetryConfig `json:"retry,omitempty"`
}
ExportScheduleRequest represents a request to create/update an export schedule.
type ExportScheduleResponse ¶
type ExportScheduleResponse struct {
ID string `json:"id"`
Name string `json:"name"`
Enabled bool `json:"enabled"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
Filters scheduler.ExportFilters `json:"filters"`
Export scheduler.ExportConfig `json:"export"`
Retry scheduler.ExportRetryConfig `json:"retry"`
}
ExportScheduleResponse represents an export schedule in API responses.
type ExportScheduleRuntime ¶
type ExportScheduleRuntime interface {
AddSchedule(schedule *scheduler.ExportSchedule)
UpdateSchedule(schedule *scheduler.ExportSchedule)
RemoveSchedule(scheduleID string)
}
type FetcherUsageBreakdown ¶
type FetcherUsageBreakdown struct {
HTTP uint64 `json:"http"`
Chromedp uint64 `json:"chromedp"`
Playwright uint64 `json:"playwright"`
}
FetcherUsageBreakdown tracks usage counts per fetcher type
type HealthResponse ¶
type HealthResponse struct {
Status string `json:"status"`
Version string `json:"version"`
Components map[string]ComponentStatus `json:"components"`
Notices []RuntimeNotice `json:"notices,omitempty"`
Setup *SetupStatus `json:"setup,omitempty"`
}
HealthResponse represents the overall health of the system.
type Hub ¶
type Hub struct {
// contains filtered or unexported fields
}
Hub manages WebSocket client connections and broadcasts messages.
func (*Hub) BroadcastJobEvent ¶
BroadcastJobEvent converts a JobEvent to a WSMessage and broadcasts it.
func (*Hub) BroadcastManagerStatus ¶
BroadcastManagerStatus broadcasts the current manager status to all clients.
func (*Hub) BroadcastMetrics ¶
func (h *Hub) BroadcastMetrics(snapshot MetricsSnapshot)
BroadcastMetrics broadcasts the current metrics snapshot to all clients.
func (*Hub) ClientCount ¶
ClientCount returns the number of connected clients.
func (*Hub) Run ¶
func (h *Hub) Run()
Run starts the hub's event loop. Must be called in a goroutine.
type InspectableJob ¶
type InspectableJob struct {
model.Job
Run JobRunSummary `json:"run"`
}
InspectableJob represents a sanitized job with derived observability fields.
type JobEvent ¶
type JobEvent struct {
Type JobEventType
Job model.Job
PrevStatus model.Status
}
JobEvent represents a job lifecycle event for WebSocket broadcasting.
type JobEventPayload ¶
type JobEventPayload struct {
JobID string `json:"jobId"`
Kind string `json:"kind"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
Progress *JobProgress `json:"progress,omitempty"`
UpdatedAt int64 `json:"updatedAt"`
}
JobEventPayload for job-related messages.
type JobEventType ¶
type JobEventType string
JobEventType represents the type of job lifecycle event.
const ( JobEventCreated JobEventType = "created" JobEventStarted JobEventType = "started" JobEventStatus JobEventType = "status" JobEventCompleted JobEventType = "completed" )
type JobFailureContext ¶
type JobFailureContext struct {
Category string `json:"category"`
Summary string `json:"summary"`
Retryable bool `json:"retryable"`
Terminal bool `json:"terminal"`
}
JobFailureContext summarizes operator-meaningful terminal failure details.
type JobListResponse ¶
type JobListResponse struct {
Jobs []InspectableJob `json:"jobs"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
JobListResponse represents a paginated collection of sanitized inspectable jobs.
func BuildJobListResponse ¶
BuildJobListResponse returns the canonical paginated job collection envelope.
func BuildStoreBackedJobListResponse ¶
func BuildStoreBackedJobListResponse(ctx context.Context, st *store.Store, jobs []model.Job, total int, limit int, offset int) (JobListResponse, error)
BuildStoreBackedJobListResponse returns the canonical paginated job collection envelope with batch metadata enrichment.
type JobProgress ¶
type JobProgress struct {
PagesCrawled int `json:"pagesCrawled"`
PagesTotal int `json:"pagesTotal,omitempty"`
DepthCurrent int `json:"depthCurrent,omitempty"`
DepthMax int `json:"depthMax,omitempty"`
}
JobProgress for crawl jobs.
type JobQueueProgress ¶
type JobQueueProgress struct {
BatchID string `json:"batchId,omitempty"`
Index int `json:"index,omitempty"`
Total int `json:"total,omitempty"`
Completed int `json:"completed,omitempty"`
Remaining int `json:"remaining,omitempty"`
Queued int `json:"queued,omitempty"`
Running int `json:"running,omitempty"`
Percent int `json:"percent,omitempty"`
}
JobQueueProgress summarizes a job's position within a persisted batch queue.
type JobResponse ¶
type JobResponse struct {
Job InspectableJob `json:"job"`
}
JobResponse represents a single sanitized inspectable job envelope.
func BuildJobResponse ¶
func BuildJobResponse(job model.Job) JobResponse
BuildJobResponse returns the canonical single-job response envelope.
func BuildStoreBackedJobResponse ¶
func BuildStoreBackedJobResponse(ctx context.Context, st *store.Store, job model.Job) (JobResponse, error)
BuildStoreBackedJobResponse returns the canonical single-job response envelope with batch metadata enrichment.
type JobRunSummary ¶
type JobRunSummary struct {
WaitMs int64 `json:"waitMs"`
RunMs int64 `json:"runMs"`
TotalMs int64 `json:"totalMs"`
Queue *JobQueueProgress `json:"queue,omitempty"`
Failure *JobFailureContext `json:"failure,omitempty"`
}
JobRunSummary exposes derived lifecycle timing, queue, and failure details for a job.
type JobSubmissionDefaults ¶
type JobSubmissionDefaults = submission.Defaults
type ManagerStatusPayload ¶
type ManagerStatusPayload struct {
QueuedJobs int `json:"queuedJobs"`
ActiveJobs int `json:"activeJobs"`
}
ManagerStatusPayload for manager status updates.
type MetricsCollector ¶
type MetricsCollector struct {
// contains filtered or unexported fields
}
MetricsCollector holds all runtime metrics
func NewMetricsCollector ¶
func NewMetricsCollector() *MetricsCollector
NewMetricsCollector creates a new metrics collector
func (*MetricsCollector) Callback ¶
func (m *MetricsCollector) Callback() func(duration time.Duration, success bool, fetcherType, rawURL string)
Callback adapts the collector to the fetch-layer metrics callback contract. A zero duration is treated as a request start marker; non-zero durations record completed request outcomes.
func (*MetricsCollector) GetRateLimitStatus ¶
func (m *MetricsCollector) GetRateLimitStatus() []RateLimitStatus
GetRateLimitStatus returns the current rate limit status for all known hosts
func (*MetricsCollector) GetSnapshot ¶
func (m *MetricsCollector) GetSnapshot() MetricsSnapshot
GetSnapshot returns a point-in-time snapshot of all metrics
func (*MetricsCollector) RecordJobDuration ¶
func (m *MetricsCollector) RecordJobDuration(duration time.Duration)
RecordJobDuration records a job duration measurement
func (*MetricsCollector) RecordRequest ¶
func (m *MetricsCollector) RecordRequest(duration time.Duration, success bool, fetcherType, rawURL string)
RecordRequest records a request metric
func (*MetricsCollector) RegisterHostLimiter ¶
func (m *MetricsCollector) RegisterHostLimiter(host string, limiter *rate.Limiter, qps float64, burst int)
RegisterHostLimiter registers a host with its rate limiter for tracking
func (*MetricsCollector) Reset ¶
func (m *MetricsCollector) Reset()
Reset clears all metrics (useful for testing)
func (*MetricsCollector) SetDefaultRateLimit ¶
func (m *MetricsCollector) SetDefaultRateLimit(qps float64, burst int)
SetDefaultRateLimit sets the default rate limit parameters for new hosts
func (*MetricsCollector) StartRequest ¶
func (m *MetricsCollector) StartRequest()
StartRequest marks the beginning of a request (for active request counting)
func (*MetricsCollector) SyncHostLimiters ¶
func (m *MetricsCollector) SyncHostLimiters(hostLimiter interface { GetHostStatus() []struct { Host string QPS float64 Burst int LastRequest time.Time } GetLimiter(host string) *rate.Limiter })
SyncHostLimiters synchronizes host limiters from the HostLimiter to the metrics collector. This should be called periodically to ensure new hosts are registered for metrics tracking.
type MetricsSnapshot ¶
type MetricsSnapshot struct {
// Request metrics
RequestsPerSec float64 `json:"requestsPerSec"`
SuccessRate float64 `json:"successRate"` // Percentage 0-100
AvgResponseTime float64 `json:"avgResponseTimeMs"` // Milliseconds
ActiveRequests int `json:"activeRequests"`
TotalRequests uint64 `json:"totalRequests"`
// Fetcher breakdown
FetcherUsage FetcherUsageBreakdown `json:"fetcherUsage"`
// Rate limit status
RateLimitStatus []RateLimitStatus `json:"rateLimitStatus"`
// Job metrics
JobThroughput float64 `json:"jobThroughputPerMin"` // Jobs per minute
AvgJobDuration float64 `json:"avgJobDurationMs"` // Milliseconds
// Timestamp
Timestamp int64 `json:"timestamp"`
}
MetricsSnapshot is a point-in-time view of all metrics
type OAuthCallbackResponse ¶
type OAuthCallbackResponse struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
ExpiresIn int `json:"expires_in,omitempty"`
Scope string `json:"scope,omitempty"`
}
OAuthCallbackResponse represents the response from an OAuth callback.
type OAuthInitiateRequest ¶
type OAuthInitiateRequest struct {
ProfileName string `json:"profile_name"`
RedirectURI string `json:"redirect_uri,omitempty"`
}
OAuthInitiateRequest represents a request to initiate an OAuth flow.
type OAuthInitiateResponse ¶
type OAuthInitiateResponse struct {
AuthorizationURL string `json:"authorization_url"`
State string `json:"state"`
}
OAuthInitiateResponse represents the response from initiating an OAuth flow.
type OAuthRefreshRequest ¶
type OAuthRefreshRequest struct {
ProfileName string `json:"profile_name"`
}
OAuthRefreshRequest represents a request to refresh an OAuth token.
type OIDCDiscoverRequest ¶
type OIDCDiscoverRequest struct {
DiscoveryURL string `json:"discovery_url,omitempty"`
Issuer string `json:"issuer,omitempty"`
}
OIDCDiscoverRequest represents a request to perform OIDC discovery.
type ProxyPoolProvider ¶
ProxyPoolProvider is an interface for providing proxy pool status.
type ProxyPoolRuntimeState ¶
type ProxyPoolRuntimeState string
const ( ProxyPoolRuntimeLoaded ProxyPoolRuntimeState = "loaded" ProxyPoolRuntimeUnloaded ProxyPoolRuntimeState = "unloaded" ProxyPoolRuntimeSetupMode ProxyPoolRuntimeState = "setup_mode" )
type ProxyPoolStatusResponse ¶
type ProxyPoolStatusResponse struct {
Strategy string `json:"strategy"`
TotalProxies int `json:"total_proxies"`
HealthyProxies int `json:"healthy_proxies"`
Regions []string `json:"regions"`
Tags []string `json:"tags"`
Proxies []ProxyStatus `json:"proxies"`
}
ProxyPoolStatusResponse represents the response for the proxy pool status endpoint.
func BuildProxyPoolStatusResponse ¶
func BuildProxyPoolStatusResponse(proxyPool *fetch.ProxyPool) ProxyPoolStatusResponse
BuildProxyPoolStatusResponse converts a loaded proxy pool into a stable status payload.
type ProxyStatus ¶
type ProxyStatus struct {
ID string `json:"id"`
Region string `json:"region,omitempty"`
Tags []string `json:"tags,omitempty"`
IsHealthy bool `json:"is_healthy"`
RequestCount uint64 `json:"request_count"`
SuccessCount uint64 `json:"success_count"`
FailureCount uint64 `json:"failure_count"`
SuccessRate float64 `json:"success_rate"`
AvgLatencyMs int64 `json:"avg_latency_ms"`
ConsecutiveFails int `json:"consecutive_fails"`
}
ProxyStatus represents the status of a single proxy.
type RateLimitStatus ¶
type RateLimitStatus struct {
Host string `json:"host"`
QPS float64 `json:"qps"`
Burst int `json:"burst"`
Tokens float64 `json:"tokens"` // Estimated current tokens
LastRequest int64 `json:"lastRequest"` // Unix timestamp
}
RateLimitStatus represents the current state for a single host
type RecommendedAction ¶
type RecommendedAction struct {
Label string `json:"label"`
Kind string `json:"kind"`
Value string `json:"value,omitempty"`
}
RecommendedAction describes an operator-facing next step for setup or recovery.
func CLIRecommendedActions ¶
func CLIRecommendedActions(actions []RecommendedAction, commandName string) []RecommendedAction
func MCPRecommendedActions ¶
func MCPRecommendedActions(actions []RecommendedAction) []RecommendedAction
type RequestMetric ¶
type RequestMetric struct {
Timestamp time.Time
Duration time.Duration
Success bool
FetcherType string // "http", "chromedp", "playwright"
Host string // sanitized host only
}
RequestMetric represents a single request measurement
type ResearchRequest ¶
type ResearchRequest = submission.ResearchRequest
ResearchRequest represents a request to perform deep research across multiple URLs.
type RetentionCleanupRequest ¶
type RetentionCleanupRequest struct {
DryRun bool `json:"dryRun"`
Force bool `json:"force,omitempty"`
OlderThan *int `json:"olderThan,omitempty"` // days
Kind string `json:"kind,omitempty"` // scrape|crawl|research
}
RetentionCleanupRequest represents a request to run retention cleanup.
type RetentionCleanupResponse ¶
type RetentionCleanupResponse struct {
JobsDeleted int `json:"jobsDeleted"`
JobsAttempted int `json:"jobsAttempted"`
CrawlStatesDeleted int64 `json:"crawlStatesDeleted"`
SpaceReclaimedMB int64 `json:"spaceReclaimedMB"`
DurationMs int64 `json:"durationMs"`
FailedJobIDs []string `json:"failedJobIDs,omitempty"`
Errors []string `json:"errors,omitempty"`
DryRun bool `json:"dryRun"`
}
RetentionCleanupResponse represents the result of a retention cleanup operation.
type RetentionStatusResponse ¶
type RetentionStatusResponse struct {
Enabled bool `json:"enabled"`
JobRetentionDays int `json:"jobRetentionDays"`
CrawlStateDays int `json:"crawlStateDays"`
MaxJobs int `json:"maxJobs"`
MaxStorageGB int `json:"maxStorageGB"`
TotalJobs int64 `json:"totalJobs"`
JobsEligible int64 `json:"jobsEligible"`
StorageUsedMB int64 `json:"storageUsedMB"`
Guidance *CapabilityGuidance `json:"guidance,omitempty"`
}
RetentionStatusResponse represents the retention system status.
type RingBuffer ¶
type RingBuffer[T any] struct { // contains filtered or unexported fields }
RingBuffer is a thread-safe circular buffer for fixed-size data
func NewRingBuffer ¶
func NewRingBuffer[T any](capacity int) *RingBuffer[T]
NewRingBuffer creates a new ring buffer with the specified capacity
func (*RingBuffer[T]) GetAll ¶
func (r *RingBuffer[T]) GetAll() []T
GetAll returns all items in the buffer (oldest first)
func (*RingBuffer[T]) Push ¶
func (r *RingBuffer[T]) Push(item T)
Push adds an item to the buffer, overwriting old data if full
func (*RingBuffer[T]) Size ¶
func (r *RingBuffer[T]) Size() int
Size returns the current number of items in the buffer
type RuntimeNotice ¶
type RuntimeNotice struct {
ID string `json:"id"`
Scope string `json:"scope"`
Severity string `json:"severity"`
Title string `json:"title"`
Message string `json:"message"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
RuntimeNotice summarizes a non-fatal setup, config, or runtime issue.
func BuildConfigRuntimeNotices ¶
func BuildConfigRuntimeNotices(in []config.StartupNotice) []RuntimeNotice
type ScheduleRequest ¶
type ScheduleRequest struct {
Kind string `json:"kind"`
IntervalSeconds int `json:"intervalSeconds"`
Request json.RawMessage `json:"request"`
}
ScheduleRequest represents a request to add a scheduled job.
type ScheduleResponse ¶
type ScheduleResponse struct {
ID string `json:"id"`
Kind string `json:"kind"`
IntervalSeconds int `json:"intervalSeconds"`
NextRun string `json:"nextRun"`
Request any `json:"request"`
}
ScheduleResponse represents a schedule in the response.
type ScrapeRequest ¶
type ScrapeRequest = submission.ScrapeRequest
ScrapeRequest represents a request to scrape a single page.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
func NewSetupServer ¶
func NewSetupServer(cfg config.Config, setup SetupStatus) *Server
NewSetupServer creates a minimal API server used when startup must remain in guided setup mode.
func (*Server) GetMetricsCollector ¶
func (s *Server) GetMetricsCollector() *MetricsCollector
GetMetricsCollector returns the server's metrics collector for external registration
func (*Server) Manager ¶
Manager returns the server's job manager when the server is running normally.
func (*Server) SetExportScheduleRuntime ¶
func (s *Server) SetExportScheduleRuntime(runtime ExportScheduleRuntime)
SetExportScheduleRuntime connects the HTTP handlers to the live export-schedule runtime.
func (*Server) Stop ¶
func (s *Server) Stop()
Stop gracefully shuts down the server's background services. This should be called during application shutdown to ensure analytics data is properly flushed to storage.
func (*Server) WebhookDispatcher ¶
func (s *Server) WebhookDispatcher() *webhook.Dispatcher
WebhookDispatcher returns the server's shared webhook dispatcher.
type SetupStatus ¶
type SetupStatus struct {
Required bool `json:"required"`
Code string `json:"code,omitempty"`
Title string `json:"title,omitempty"`
Message string `json:"message,omitempty"`
DataDir string `json:"dataDir,omitempty"`
SchemaVersion string `json:"schemaVersion,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
SetupStatus describes guided recovery information when the server starts in setup mode.
type StatusResponse ¶
type StatusResponse struct {
Status string `json:"status"`
RequestID string `json:"requestId,omitempty"`
}
StatusResponse represents a generic success response.
type TemplatePreviewResponse ¶
type TemplatePreviewResponse struct {
URL string `json:"url"`
Title string `json:"title"`
DOMTree DOMNode `json:"dom_tree"`
FetchTime int64 `json:"fetch_time_ms"`
Fetcher string `json:"fetcher"` // "http", "chromedp", or "playwright"
}
TemplatePreviewResponse returned by the preview endpoint
type TemplateResponse ¶
type TemplateResponse struct {
Name string `json:"name"`
IsBuiltIn bool `json:"is_built_in"`
Template extract.Template `json:"template"`
}
TemplateResponse with full template details
type TestSelectorRequest ¶
type TestSelectorRequest struct {
URL string `json:"url"`
Selector string `json:"selector"`
Headless bool `json:"headless,omitempty"`
Playwright bool `json:"playwright,omitempty"`
}
TestSelectorRequest for testing selectors
type TestSelectorResponse ¶
type TestSelectorResponse struct {
Selector string `json:"selector"`
Matches int `json:"matches"`
Elements []DOMElement `json:"elements"`
Error string `json:"error,omitempty"`
}
TestSelectorResponse with matching elements
type TransformPreviewRequest ¶
type TransformPreviewRequest struct {
JobID string `json:"jobId"` // Job to preview against
Expression string `json:"expression"` // JMESPath or JSONata expression
Language string `json:"language"` // "jmespath" or "jsonata"
Limit int `json:"limit,omitempty"` // Max results to return (default 10)
}
TransformPreviewRequest requests a transformation preview.
type TransformPreviewResponse ¶
type TransformPreviewResponse struct {
Results []any `json:"results"` // Transformed data
Error string `json:"error,omitempty"` // Error message if failed
ResultCount int `json:"resultCount"` // Number of results
}
TransformPreviewResponse returns transformed results.
type TransformValidateRequest ¶
type TransformValidateRequest struct {
Expression string `json:"expression"` // JMESPath or JSONata expression
Language string `json:"language"` // "jmespath" or "jsonata"
}
TransformValidateRequest requests validation of a transformation expression.
type TransformValidateResponse ¶
type TransformValidateResponse struct {
Valid bool `json:"valid"`
Error string `json:"error,omitempty"`
Message string `json:"message,omitempty"`
}
TransformValidateResponse returns validation result.
type WSMessage ¶
type WSMessage struct {
Type WSMessageType `json:"type"`
Timestamp int64 `json:"timestamp"`
Payload any `json:"payload"`
}
WSMessage is the envelope for all WebSocket messages.
type WSMessageType ¶
type WSMessageType string
WSMessageType represents the type of WebSocket message.
const ( // Server -> Client message types WSMessageJobCreated WSMessageType = "job_created" WSMessageJobStarted WSMessageType = "job_started" WSMessageStatusChanged WSMessageType = "job_status_changed" WSMessageJobCompleted WSMessageType = "job_completed" WSMessageManagerStatus WSMessageType = "manager_status" WSMessageMetrics WSMessageType = "metrics" WSMessagePing WSMessageType = "ping" // Client -> Server message types WSMessagePong WSMessageType = "pong" WSMessageSubscribeJobs WSMessageType = "subscribe_jobs" WSMessageUnsubscribeJobs WSMessageType = "unsubscribe_jobs" )
type WatchArtifactResponse ¶
type WatchArtifactResponse struct {
Kind string `json:"kind"`
Filename string `json:"filename"`
ContentType string `json:"contentType"`
ByteSize int64 `json:"byteSize,omitempty"`
DownloadURL string `json:"downloadUrl"`
}
WatchArtifactResponse describes a public watch artifact download.
type WatchCheckHistoryResponse ¶
type WatchCheckHistoryResponse struct {
Checks []WatchCheckInspection `json:"checks"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
WatchCheckHistoryResponse represents a paginated watch check history collection.
func BuildWatchCheckHistoryResponse ¶
func BuildWatchCheckHistoryResponse(records []watch.WatchCheckRecord, total, limit, offset int) WatchCheckHistoryResponse
type WatchCheckInspection ¶
type WatchCheckInspection struct {
ID string `json:"id"`
WatchID string `json:"watchId"`
URL string `json:"url"`
CheckedAt time.Time `json:"checkedAt"`
Status string `json:"status"`
Changed bool `json:"changed"`
Baseline bool `json:"baseline,omitempty"`
Title string `json:"title"`
Message string `json:"message"`
PreviousHash string `json:"previousHash,omitempty"`
CurrentHash string `json:"currentHash,omitempty"`
DiffText string `json:"diffText,omitempty"`
DiffHTML string `json:"diffHtml,omitempty"`
Error string `json:"error,omitempty"`
Selector string `json:"selector,omitempty"`
Artifacts []WatchArtifactResponse `json:"artifacts,omitempty"`
VisualHash string `json:"visualHash,omitempty"`
PreviousVisualHash string `json:"previousVisualHash,omitempty"`
VisualChanged bool `json:"visualChanged"`
VisualSimilarity float64 `json:"visualSimilarity,omitempty"`
TriggeredJobs []string `json:"triggeredJobs,omitempty"`
Actions []RecommendedAction `json:"actions,omitempty"`
}
WatchCheckInspection represents a persisted, operator-facing watch check outcome.
func BuildWatchCheckInspection ¶
func BuildWatchCheckInspection(record watch.WatchCheckRecord) WatchCheckInspection
type WatchCheckInspectionResponse ¶
type WatchCheckInspectionResponse struct {
Check WatchCheckInspection `json:"check"`
}
WatchCheckInspectionResponse represents a single watch check inspection envelope.
type WatchCheckStatus ¶
type WatchCheckStatus = watch.CheckStatus
WatchCheckStatus aliases the shared watch check status enum for transport-safe responses.
type WatchListResponse ¶
type WatchListResponse struct {
Watches []WatchResponse `json:"watches"`
Total int `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
}
WatchListResponse represents a paginated collection of watches.
func BuildWatchListResponse ¶
func BuildWatchListResponse(items []watch.Watch, total, limit, offset int) WatchListResponse
BuildWatchListResponse returns the canonical paginated watch collection envelope.
type WatchRequest ¶
type WatchRequest struct {
URL string `json:"url"`
Selector string `json:"selector,omitempty"`
IntervalSeconds int `json:"intervalSeconds"`
Enabled *bool `json:"enabled,omitempty"`
DiffFormat string `json:"diffFormat,omitempty"`
WebhookConfig *model.WebhookSpec `json:"webhookConfig,omitempty"`
NotifyOnChange bool `json:"notifyOnChange"`
MinChangeSize int `json:"minChangeSize,omitempty"`
IgnorePatterns []string `json:"ignorePatterns,omitempty"`
Headless bool `json:"headless"`
UsePlaywright bool `json:"usePlaywright"`
ExtractMode string `json:"extractMode,omitempty"`
ScreenshotEnabled bool `json:"screenshotEnabled,omitempty"`
ScreenshotConfig *fetch.ScreenshotConfig `json:"screenshotConfig,omitempty"`
VisualDiffThreshold *float64 `json:"visualDiffThreshold,omitempty"`
JobTrigger *watch.JobTrigger `json:"jobTrigger,omitempty"`
}
WatchRequest represents the request body for creating/updating a watch.
type WatchResponse ¶
type WatchResponse struct {
ID string `json:"id"`
URL string `json:"url"`
Selector string `json:"selector,omitempty"`
IntervalSeconds int `json:"intervalSeconds"`
Enabled bool `json:"enabled"`
CreatedAt time.Time `json:"createdAt"`
LastCheckedAt time.Time `json:"lastCheckedAt,omitempty"`
LastChangedAt time.Time `json:"lastChangedAt,omitempty"`
ChangeCount int `json:"changeCount"`
DiffFormat string `json:"diffFormat"`
WebhookConfig *model.WebhookSpec `json:"webhookConfig,omitempty"`
NotifyOnChange bool `json:"notifyOnChange"`
MinChangeSize int `json:"minChangeSize,omitempty"`
IgnorePatterns []string `json:"ignorePatterns,omitempty"`
Headless bool `json:"headless"`
UsePlaywright bool `json:"usePlaywright"`
ExtractMode string `json:"extractMode,omitempty"`
Status string `json:"status"`
ScreenshotEnabled bool `json:"screenshotEnabled,omitempty"`
ScreenshotConfig *fetch.ScreenshotConfig `json:"screenshotConfig,omitempty"`
VisualDiffThreshold float64 `json:"visualDiffThreshold,omitempty"`
JobTrigger *watch.JobTrigger `json:"jobTrigger,omitempty"`
}
WatchResponse represents a watch in API responses.
func BuildWatchResponse ¶
func BuildWatchResponse(item watch.Watch) WatchResponse
BuildWatchResponse returns the canonical single-watch payload.
type WebhookConfig ¶
type WebhookConfig = submission.WebhookConfig
WebhookConfig represents webhook configuration for job notifications.
Source Files
¶
- ai_extract.go
- analytics_handler.go
- auth.go
- batch.go
- chains.go
- crawl.go
- crawl_states.go
- dedup.go
- diagnostic_status.go
- diagnostics.go
- export_outcome_contracts.go
- export_schedules.go
- health.go
- job_export.go
- job_helpers.go
- job_response_contracts.go
- job_result_access.go
- job_results.go
- job_submission_contracts.go
- jobs.go
- metrics.go
- metrics_handler.go
- middleware.go
- pipeline_js_handlers.go
- proxy_pool.go
- render_profiles.go
- request_helpers.go
- research.go
- retention.go
- retention_guidance.go
- schedules.go
- scrape.go
- server.go
- template_preview_handler.go
- templates.go
- transform.go
- types.go
- util.go
- watch.go
- watch_artifacts.go
- watch_history.go
- watch_outcome_contracts.go
- watch_response_contracts.go
- webhook_handlers.go
- websocket.go