checkworker

package
v0.0.0-...-b74bf7c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: AGPL-3.0 Imports: 52 Imported by: 0

Documentation

Overview

Package checkworker implements distributed check job execution.

Index

Constants

View Source
const (

	// OutputKeyReportFileUID names the stored report of a finished run.
	OutputKeyReportFileUID = "reportFileUid"
)

Multi-step run limits (spec 2026-10-03-03).

Variables

View Source
var (
	ErrUnknownCheckType   = errors.New("unknown check type")
	ErrCheckerNotFound    = errors.New("checker not found for type")
	ErrFailedToParseConf  = errors.New("failed to parse config")
	ErrFailedToFetchCheck = errors.New("failed to fetch check from database")
	ErrNoCheckType        = errors.New("check job has no type set")
	// ErrDockerNotAvailableInSaaS is returned when a docker job reaches a SaaS
	// shared worker (spec 2026-09-25-22). The create/update gate
	// (checks.Service.validateDockerDeploymentConfig) is meant to keep such a
	// job from ever being scheduled, but a check created before that gate
	// shipped, or one whose placement moved off a private region since, must
	// still be refused here — without ever dialing the socket. Same message
	// as the create-time gate, so the two read as one rule from either side.
	ErrDockerNotAvailableInSaaS = errors.New("docker checks are not available on this deployment")

	// ErrCheckerPanic wraps a recovered panic from inside a checker's
	// Execute call (spec 2026-07-05-05 D2). The runner survives; the panic
	// is converted into a StatusError result instead of crashing the
	// process.
	ErrCheckerPanic = errors.New("checker panicked")

	// ErrCheckerAbandoned marks a checker execution the watchdog gave up
	// on because the checker never observed its context deadline within
	// execTimeout + abandonGrace (spec 2026-07-05-05 D1).
	ErrCheckerAbandoned = errors.New("check execution abandoned: checker did not honor its context")
)

Errors returned by the check runner.

View Source
var (
	// ErrStepNotSupported means this worker's backend cannot persist step
	// state (an agent): multi-step checks run on the server's workers only.
	ErrStepNotSupported = errors.New("multi-step checks are not supported on this worker")
	// ErrStepFailedTooOften ends a run after maxStepFailures failed slices.
	ErrStepFailedTooOften = errors.New("multi-step run failed 3 slices in a row")
)

Errors of the multi-step path.

Functions

This section is empty.

Types

type CheckWorker

type CheckWorker struct {
	// contains filtered or unexported fields
}

CheckWorker executes check jobs from the queue.

func NewAgentCheckWorker

func NewAgentCheckWorker(cfg *config.Config, workerBackend backend.WorkerBackend) *CheckWorker

NewAgentCheckWorker creates a check runner for agent mode: no database, no services registry — everything flows through the given (remote) backend. Self-stats and the entitlements gate are in-process concerns and are skipped; the server enforces per-org rate limits on the agent claim path.

func NewCheckWorker

func NewCheckWorker(
	dbService db.Service,
	cfg *config.Config,
	svc *services.Registry,
	checkJobSvc checkjobsvc.Service,
) *CheckWorker

NewCheckWorker creates a new check runner wired to the in-process DirectBackend (the production server path).

func (*CheckWorker) EgressGuard

func (r *CheckWorker) EgressGuard() *egress.Guard

EgressGuard returns the worker's egress policy, for the process's other outbound probes (the agent's path traces).

func (*CheckWorker) Run

func (r *CheckWorker) Run(ctx context.Context) error

Run starts the runner loop (blocking).

func (*CheckWorker) SetFaultLatch

func (r *CheckWorker) SetFaultLatch(latch *dbfault.Latch)

SetFaultLatch installs the process-wide structural-fault latch. Set by the server before Run; a worker without one never terminates the process.

type PassiveEvaluator

type PassiveEvaluator struct {
	// contains filtered or unexported fields
}

PassiveEvaluator evaluates passive checks (heartbeat, email) on the jobs node (spec 2026-09-25-04).

Before it, the "is the signal overdue?" evaluation ran as an ordinary regional check job, which failed three ways: a dead region silenced every dead-man's switch pinned to it (the 2026-09-24 `lauterbourg` outage), an agent that won the job could not evaluate it and opened an incident every period, and N regions wrote N identical evaluations. A passive check now owns exactly one NULL-region check_jobs row, every check worker and agent claim excludes passive types, and this loop is the only claimer.

Guarantees, all from the check_jobs lease:

  • it depends on no region and no check worker being alive — only on a node running the jobs role;
  • a restart loses nothing: claimed-but-unevaluated jobs are released on shutdown, and a crash only waits out the lease (scheduled_at + period + 30s) before the next claim picks the job up;
  • two jobs nodes never evaluate one tick twice: the claim locks rows (FOR UPDATE SKIP LOCKED on Postgres) and leases them in one transaction, and the release moves scheduled_at to the next tick.

Its lease owner is a `workers` row of its own (check_jobs.lease_worker_uid is a foreign key). Evaluation rows therefore carry a worker_uid, which is half of what keeps them from being read back as signals; the other half is the `output.evaluation` filter in GetLastSignalForChecks. Evaluation rows carry no region: they did not run anywhere.

func NewPassiveEvaluator

func NewPassiveEvaluator(
	dbService db.Service,
	cfg *config.Config,
	svc *services.Registry,
	checkJobSvc checkjobsvc.Service,
) *PassiveEvaluator

NewPassiveEvaluator builds the jobs node's passive evaluator, writing its results through the same in-process pipeline as the check worker (DirectBackend: save, ProcessCheckResult, release).

func (*PassiveEvaluator) Run

func (e *PassiveEvaluator) Run(ctx context.Context) error

Run registers the evaluator's workers row and evaluates due passive jobs until ctx is canceled.

func (*PassiveEvaluator) RunOnce

func (e *PassiveEvaluator) RunOnce(ctx context.Context) (int, time.Duration, error)

RunOnce claims every due passive job (up to one batch) and evaluates it. Returns how many jobs it claimed and the claim's next-eligible hint.

Directories

Path Synopsis
Package backend provides the WorkerBackend interface that abstracts how the check worker loop (internal/checkworker) talks to the master: DirectBackend calls the database and services in-process (the production server path), WSBackend (agent mode) speaks the WebSocket agent protocol to a remote server.
Package backend provides the WorkerBackend interface that abstracts how the check worker loop (internal/checkworker) talks to the master: DirectBackend calls the database and services in-process (the production server path), WSBackend (agent mode) speaks the WebSocket agent protocol to a remote server.
Package checkjobsvc provides check job queue operations for the distributed check runner system.
Package checkjobsvc provides check job queue operations for the distributed check runner system.
Package egressreport turns this process's live egress self-probe into the shape the worker row stores, for the in-process worker and the deported agent alike (spec 2026-08-15-11).
Package egressreport turns this process's live egress self-probe into the shape the worker row stores, for the in-process worker and the deported agent alike (spec 2026-08-15-11).
Package scheduling holds the pure, side-effect-free math for cost-aware, plan-weighted check scheduling (spec 2026-06-30-09, cost-only offset per spec 2026-07-01-02).
Package scheduling holds the pure, side-effect-free math for cost-aware, plan-weighted check scheduling (spec 2026-06-30-09, cost-only offset per spec 2026-07-01-02).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL