engine

package
v4.0.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: AGPL-3.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker requests tasks from a scheduler and executes them. Task results are sent to other Worker instances or back to the scheduler.

func NewWorker

func NewWorker(params WorkerParams, reg prometheus.Registerer) (*Worker, error)

NewWorker creates a new Worker instance. Use Worker.Service to manage the lifecycle of the Worker.

func (*Worker) RegisterMetrics

func (w *Worker) RegisterMetrics(reg prometheus.Registerer) error

RegisterMetrics registers metrics about w to report to reg.

func (*Worker) RegisterWorkerServer

func (w *Worker) RegisterWorkerServer(router *mux.Router)

RegisterWorkerServer registers the wire.Listener of the inner worker as http.Handler on the provided router.

func (*Worker) Service

func (w *Worker) Service() services.Service

Service returns the service used to manage the lifecycle of the Worker.

func (*Worker) UnregisterMetrics

func (w *Worker) UnregisterMetrics(reg prometheus.Registerer)

UnregisterMetrics unregisters metrics about w from reg.

type WorkerConfig

type WorkerConfig struct {
	WorkerThreads int

	SchedulerLookupAddress  string
	SchedulerLookupInterval time.Duration
}

WorkerConfig represents the configuration for the Worker.

type WorkerParams

type WorkerParams struct {
	Logger log.Logger      // Logger for optional log messages.
	Bucket objstore.Bucket // Bucket to read stored data from.

	// DataBucket reads source log objects during LogMerge compaction. When
	// Bucket is prefixed with the index-storage prefix (compaction wiring),
	// source log objects live at the unprefixed dataobj root and must be read
	// through this bucket. Optional; nil falls back to Bucket.
	DataBucket objstore.Bucket

	Config WorkerConfig // Configuration for the worker.

	// Address to advertise to other workers and schedulers. Required.
	AdvertiseAddr net.Addr

	// Absolute path of the endpoint where the frame handler is registered.
	// Used for connecting to scheduler and other workers.
	Endpoint string

	// ScratchStore holds the temporary data of compaction tasks. Optional; nil
	// keeps the data in memory.
	ScratchStore scratch.Store

	// IndexobjCfg is the builder config for compacted index objects.
	IndexobjCfg logsobj.BuilderBaseConfig

	// LogsobjCfg is the builder config for compacted log objects.
	LogsobjCfg logsobj.BuilderBaseConfig

	// IndexMergeObserver is used  by compaction to populate output-size
	// histograms. Optional; nil disables observation.
	IndexMergeObserver executor.IndexMergeObserver

	// NewLogMergeObserver returns the LogMerge observer for each worker
	// thread. Optional; nil disables observation.
	NewLogMergeObserver executor.NewLogMergeObserverFunc
}

WorkerParams holds parameters for constructing a new Worker.

Directories

Path Synopsis
Package compactor contains the dataobj compaction planner and worker service skeletons.
Package compactor contains the dataobj compaction planner and worker service skeletons.
internal/compactortest
Package compactortest provides object-backed compaction integration fixtures.
Package compactortest provides object-backed compaction integration fixtures.
internal
metrictimer
Package metrictimer provides a small helper for timing named operations.
Package metrictimer provides a small helper for timing named operations.
obslock
Package obslock provides an observed sync.RWMutex that records how long callers wait to acquire a lock and how long they hold it.
Package obslock provides an observed sync.RWMutex that records how long callers wait to acquire a lock and how long they hold it.
proto/ulid
Package ulid provides a ULID implementation.
Package ulid provides a ULID implementation.
scheduler
Package scheduler provides an implementation of workflow.Runner that works by scheduling tasks to be executed by a set of workers.
Package scheduler provides an implementation of workflow.Runner that works by scheduling tasks to be executed by a set of workers.
scheduler/schedulerstat
Package schedulerstat declares scheduler-recorded per-task observation statistics.
Package schedulerstat declares scheduler-recorded per-task observation statistics.
scheduler/wire
Package wire provides the wire protocol for how peers scheduler peers communicate.
Package wire provides the wire protocol for how peers scheduler peers communicate.
util/dag
Package dag provides utilities for working with directed acyclic graphs (DAGs).
Package dag provides utilities for working with directed acyclic graphs (DAGs).
util/ewma
Package ewma provides an implementation of an exponentially weighted moving average (EWMA) that can be reported as a Prometheus metric.
Package ewma provides an implementation of an exponentially weighted moving average (EWMA) that can be reported as a Prometheus metric.
util/queue/fair
Package fair implements a Hierarchical Fair Queue (HFQ), providing balanced service across a hierarchy of queues.
Package fair implements a Hierarchical Fair Queue (HFQ), providing balanced service across a hierarchy of queues.
worker
Package worker provides a mechanism to connect to the [scheduler] for executing tasks.
Package worker provides a mechanism to connect to the [scheduler] for executing tasks.
worker/workerstat
Package workerstat declares worker-recorded per-task observation statistics.
Package workerstat declares worker-recorded per-task observation statistics.
workflow
Package workflow defines how to represent physical plans as distributed workflows.
Package workflow defines how to represent physical plans as distributed workflows.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL