Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // StatPipelineRowsOut counts rows produced by an executor pipeline node. StatPipelineRowsOut = xcap.NewStatisticInt64("rows.out", xcap.AggregationTypeSum, xcap.Local()) // StatPipelineReadCalls counts calls to Read for an executor pipeline node. StatPipelineReadCalls = xcap.NewStatisticInt64("read.calls", xcap.AggregationTypeSum, xcap.Local()) // StatPipelineReadDuration tracks wall time spent in Read for an executor // pipeline node, inclusive of child pipeline nodes. StatPipelineReadDuration = xcap.NewStatisticFloat64("read.duration", xcap.AggregationTypeSum, xcap.Local()) )
Common pipeline statistics tracked across executor nodes.
var (
EOF = fmt.Errorf("pipeline exhausted: %w", io.EOF) //nolint:revive,staticcheck
)
Functions ¶
This section is empty.
Types ¶
type BufferedPipeline ¶
type BufferedPipeline struct {
// contains filtered or unexported fields
}
BufferedPipeline is a pipeline implementation that reads from a fixed set of Arrow records. It implements the Pipeline interface and returns the records in order.
func NewBufferedPipeline ¶
func NewBufferedPipeline(records ...arrow.RecordBatch) *BufferedPipeline
NewBufferedPipeline creates a new BufferedPipeline from a set of Arrow records. The pipeline will return these records in sequence.
func (*BufferedPipeline) Close ¶
func (p *BufferedPipeline) Close()
Close implements Pipeline. It drops the references to the unreturned records.
func (*BufferedPipeline) Open ¶
func (p *BufferedPipeline) Open(_ context.Context) error
Open implements Pipeline.
func (*BufferedPipeline) Read ¶
func (p *BufferedPipeline) Read(_ context.Context) (arrow.RecordBatch, error)
Read implements Pipeline. It advances to the next record and returns EOF when all records have been read.
type Config ¶
type Config struct {
// ScratchStore holds the temporary data of compaction tasks. Optional; nil
// keeps the data in memory.
ScratchStore scratch.Store
// IndexobjCfg is the builder config for index objects.
IndexobjCfg logsobj.BuilderBaseConfig
// LogsobjCfg is the builder config for log objects.
LogsobjCfg logsobj.BuilderBaseConfig
// BuilderMetrics is shared by logs object builders across worker tasks.
BuilderMetrics *logsobj.BuilderMetrics
// IndexMergeObserver is used by compaction to populate output-size
// histograms. Optional; nil disables observation.
IndexMergeObserver IndexMergeObserver
// LogMergeObserver is used by compaction to populate log-merge histograms.
// Optional; nil disables observation.
LogMergeObserver LogMergeObserver
Bucket objstore.Bucket
// DataBucket reads source log objects during LogMerge compaction. Unlike
// Bucket (which the compaction wiring prefixes with the index-storage
// prefix for index I/O and ToC), source log objects are stored at the
// unprefixed dataobj root, so they must be read through this bucket.
// Optional; when nil the executor falls back to Bucket.
DataBucket objstore.Bucket
// GetExternalInputs is an optional function called for each node in the
// plan. Compaction nodes accept no inputs, so a node fails if
// GetExternalInputs returns any Pipeline for it.
GetExternalInputs func(ctx context.Context, node physical.Node) []Pipeline
}
type Context ¶
type Context struct {
// contains filtered or unexported fields
}
Context is the execution context
type IndexMergeObserver ¶
type LogMergeObservedStats ¶
type LogMergeObservedStats struct {
Outcome string
SourceObjects int
InputSections int
InputBytes int64 // Log line bytes plus structured metadata value bytes.
DuplicateRecords int // Records that repeat an earlier record's stream, timestamp, line, and metadata.
OutputObjects int
OutputBytesCompressed int64
}
LogMergeObservedStats is the per-task compaction summary reported to LogMergeObserver and xcap statistics.
type LogMergeObserver ¶
type LogMergeObserver interface {
// ObserveLogMerge receives the summary of one finished task.
ObserveLogMerge(tenant string, stats LogMergeObservedStats, duration time.Duration)
// ObserveLogMergeInputBytes receives the size of a batch of merged records
// while the task runs. It reports during the task, not once at the end,
// so that rate() stays steady for tasks that run for many minutes.
ObserveLogMergeInputBytes(bytes int64)
}
LogMergeObserver receives LogMerge statistics from one worker thread.
type NewLogMergeObserverFunc ¶
type NewLogMergeObserverFunc func(thread int) LogMergeObserver
NewLogMergeObserverFunc returns the LogMergeObserver for the worker thread with the given index.
type Pipeline ¶
type Pipeline interface {
// Open initializes the pipeline resources and must be called before [Read].
// The implementation must be safe to call multiple times.
Open(context.Context) error
// Read collects the next value ([arrow.RecordBatch]) from the pipeline and returns it to the caller.
// It returns an error if reading fails or when the pipeline is exhausted. In this case, the function returns EOF.
Read(context.Context) (arrow.RecordBatch, error)
// Close closes the resources of the pipeline.
// The implementation must close the pipelines it wraps and must be safe to call multiple times.
Close()
}
Pipeline represents a data processing pipeline that can read Arrow records. It provides methods to read data, access the current record, and close resources.
func NewObservedPipeline ¶
NewObservedPipeline wraps a pipeline to automatically collect common statistics. The xcap span and region are not created here; they are deferred to the first [Read] call so the span inherits its parent from the caller's context.