worker

package
v3.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 26, 2026 License: AGPL-3.0 Imports: 39 Imported by: 0

Documentation

Overview

Package worker provides a mechanism to connect to the [scheduler] for executing tasks.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Logger for optional log messages.
	Logger log.Logger

	// Metrics for the worker's wire connections.
	WireMetrics *wire.Metrics

	// Bucket to read stored data from.
	Bucket objstore.Bucket

	// Metastore client to access indexes.
	Metastore metastore.Metastore

	// Dialer to establish connections to scheduler and remote workers.
	Dialer wire.Dialer

	// Listener for accepting connections from workers.
	Listener wire.Listener

	// SchedulerAddress is the address of the remote scheduler to connect to.
	// SchedulerAddress is used when there is exactly one scheduler. Use
	// SchedulerLookupAddress to discover multiple schedulers by DNS SRV lookup.
	SchedulerAddress net.Addr

	// SchedulerLookupAddress is the address of the remote scheduler to look up.
	// Scheduler addresses are resolved using DNS SRV records at this address.
	SchedulerLookupAddress string

	// SchedulerLookupInterval is the frequency at which the worker will attempt
	// to find new remote schedulers.
	SchedulerLookupInterval time.Duration

	// BatchSize specifies the maximum number of rows to retrieve in a single
	// read call of a task pipeline, or to send in a single message to a peer (sink).
	BatchSize int64

	// PrefetchBytes controls the number of bytes prefetched when opening a
	// data object in scan tasks.
	PrefetchBytes int64

	// NumThreads is the number of worker threads to spawn. The number of
	// threads corresponds to the number of tasks that can be executed
	// concurrently.
	//
	// If NumThreads is set to 0, NumThreads defaults to [runtime.GOMAXPROCS].
	NumThreads int

	// Absolute path of the endpoint where the frame handler is registered.
	// Used for connecting to scheduler and other workers.
	Endpoint string

	// StreamFilterer is an optional filterer that can filter streams based on their labels.
	// When set, streams are filtered before scanning.
	StreamFilterer executor.RequestStreamFilterer `yaml:"-"`
}

Config holds configuration options for Worker.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker requests tasks from a set of [scheduler.Scheduler] instances and executes them. Task results are forwarded along streams, which are received by other Worker instances or the scheduler.

func New

func New(config Config) (*Worker, error)

New creates a new instance of a worker. Use Worker.Service to manage the lifecycle of the returned worker.

New returns an error if the provided config is invalid.

func (*Worker) RegisterMetrics

func (w *Worker) RegisterMetrics(reg prometheus.Registerer) error

RegisterMetrics registers metrics about s to report to reg.

func (*Worker) Service

func (w *Worker) Service() services.Service

Service returns the service used to manage the lifecycle of the worker.

func (*Worker) UnregisterMetrics

func (w *Worker) UnregisterMetrics(reg prometheus.Registerer)

UnregisterMetrics unregisters metrics about s from reg.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL