executor

package
v4.0.0-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: AGPL-3.0 Imports: 37 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// StatPipelineRowsOut counts rows produced by an executor pipeline node.
	StatPipelineRowsOut = xcap.NewStatisticInt64("rows.out", xcap.AggregationTypeSum, xcap.Local())

	// StatPipelineReadCalls counts calls to Read for an executor pipeline node.
	StatPipelineReadCalls = xcap.NewStatisticInt64("read.calls", xcap.AggregationTypeSum, xcap.Local())

	// StatPipelineReadDuration tracks wall time spent in Read for an executor
	// pipeline node, inclusive of child pipeline nodes.
	StatPipelineReadDuration = xcap.NewStatisticFloat64("read.duration", xcap.AggregationTypeSum, xcap.Local())
)

Common pipeline statistics tracked across executor nodes.

View Source
var (
	EOF = fmt.Errorf("pipeline exhausted: %w", io.EOF) //nolint:revive,staticcheck
)

Functions

This section is empty.

Types

type BufferedPipeline

type BufferedPipeline struct {
	// contains filtered or unexported fields
}

BufferedPipeline is a pipeline implementation that reads from a fixed set of Arrow records. It implements the Pipeline interface and returns the records in order.

func NewBufferedPipeline

func NewBufferedPipeline(records ...arrow.RecordBatch) *BufferedPipeline

NewBufferedPipeline creates a new BufferedPipeline from a set of Arrow records. The pipeline will return these records in sequence.

func (*BufferedPipeline) Close

func (p *BufferedPipeline) Close()

Close implements Pipeline. It drops the references to the unreturned records.

func (*BufferedPipeline) Open

Open implements Pipeline.

func (*BufferedPipeline) Read

Read implements Pipeline. It advances to the next record and returns EOF when all records have been read.

type Config

type Config struct {
	// ScratchStore holds the temporary data of compaction tasks. Optional; nil
	// keeps the data in memory.
	ScratchStore scratch.Store
	// IndexobjCfg is the builder config for index objects.
	IndexobjCfg logsobj.BuilderBaseConfig
	// LogsobjCfg is the builder config for log objects.
	LogsobjCfg logsobj.BuilderBaseConfig
	// BuilderMetrics is shared by logs object builders across worker tasks.
	BuilderMetrics *logsobj.BuilderMetrics

	// IndexMergeObserver is used  by compaction to populate output-size
	// histograms. Optional; nil disables observation.
	IndexMergeObserver IndexMergeObserver

	// LogMergeObserver is used by compaction to populate log-merge histograms.
	// Optional; nil disables observation.
	LogMergeObserver LogMergeObserver

	Bucket objstore.Bucket

	// DataBucket reads source log objects during LogMerge compaction. Unlike
	// Bucket (which the compaction wiring prefixes with the index-storage
	// prefix for index I/O and ToC), source log objects are stored at the
	// unprefixed dataobj root, so they must be read through this bucket.
	// Optional; when nil the executor falls back to Bucket.
	DataBucket objstore.Bucket

	// GetExternalInputs is an optional function called for each node in the
	// plan. Compaction nodes accept no inputs, so a node fails if
	// GetExternalInputs returns any Pipeline for it.
	GetExternalInputs func(ctx context.Context, node physical.Node) []Pipeline
}

type Context

type Context struct {
	// contains filtered or unexported fields
}

Context is the execution context

type IndexMergeObserver

type IndexMergeObserver interface {
	ObserveIndexMergeOutput(tenant string, compressedBytes, uncompressedBytes int64)
}

type LogMergeObservedStats

type LogMergeObservedStats struct {
	Outcome               string
	SourceObjects         int
	InputSections         int
	InputBytes            int64 // Log line bytes plus structured metadata value bytes.
	DuplicateRecords      int   // Records that repeat an earlier record's stream, timestamp, line, and metadata.
	OutputObjects         int
	OutputBytesCompressed int64
}

LogMergeObservedStats is the per-task compaction summary reported to LogMergeObserver and xcap statistics.

type LogMergeObserver

type LogMergeObserver interface {
	// ObserveLogMerge receives the summary of one finished task.
	ObserveLogMerge(tenant string, stats LogMergeObservedStats, duration time.Duration)

	// ObserveLogMergeInputBytes receives the size of a batch of merged records
	// while the task runs. It reports during the task, not once at the end,
	// so that rate() stays steady for tasks that run for many minutes.
	ObserveLogMergeInputBytes(bytes int64)
}

LogMergeObserver receives LogMerge statistics from one worker thread.

type NewLogMergeObserverFunc

type NewLogMergeObserverFunc func(thread int) LogMergeObserver

NewLogMergeObserverFunc returns the LogMergeObserver for the worker thread with the given index.

type Pipeline

type Pipeline interface {
	// Open initializes the pipeline resources and must be called before [Read].
	// The implementation must be safe to call multiple times.
	Open(context.Context) error
	// Read collects the next value ([arrow.RecordBatch]) from the pipeline and returns it to the caller.
	// It returns an error if reading fails or when the pipeline is exhausted. In this case, the function returns EOF.
	Read(context.Context) (arrow.RecordBatch, error)
	// Close closes the resources of the pipeline.
	// The implementation must close the pipelines it wraps and must be safe to call multiple times.
	Close()
}

Pipeline represents a data processing pipeline that can read Arrow records. It provides methods to read data, access the current record, and close resources.

func NewObservedPipeline

func NewObservedPipeline(name string, attrs []attribute.KeyValue, inner Pipeline) Pipeline

NewObservedPipeline wraps a pipeline to automatically collect common statistics. The xcap span and region are not created here; they are deferred to the first [Read] call so the span inherits its parent from the caller's context.

func Run

func Run(ctx context.Context, cfg Config, plan *physical.Plan, logger log.Logger) Pipeline

func TranslateEOF

func TranslateEOF(pipeline Pipeline) Pipeline

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL