Documentation
¶
Index ¶
- func BuildTree(p *Plan, n Node) *tree.Node
- func CatalogRequestToMetastoreSectionsRequest(selector Expression, predicates []Expression, start, end time.Time) (metastore.SectionsRequest, error)
- func PrintAsTree(p *Plan) string
- func WriteMermaidFormat(w io.Writer, p *Plan)
- type Batching
- type BinaryExpr
- type BinaryExpression
- type Catalog
- type ColumnCompat
- type ColumnExpr
- type ColumnExpression
- type Context
- func (pc *Context) Clone() *Context
- func (pc *Context) GetResolveTimeRange() (from, through time.Time)
- func (pc *Context) WithDirection(direction SortOrder) *Context
- func (pc *Context) WithMaxQuerySeries(maxQuerySeries int) *Context
- func (pc *Context) WithRangeInterval(rangeInterval time.Duration) *Context
- func (pc *Context) WithTimeRange(from, through time.Time) *Context
- type DataObjLocation
- type DataObjScan
- type DataObjSections
- type Expression
- type ExpressionType
- type Filter
- type FunctionExpression
- type Grouping
- type Join
- type Limit
- type LiteralExpr
- type LiteralExpression
- type Merge
- type MetastoreCatalog
- type MetastorePlanner
- type MetastoreSectionsResolver
- type Node
- type NodeType
- type Parallelize
- type Plan
- func (p *Plan) CalculateMaxTimeRange() TimeRange
- func (p *Plan) Children(n Node) []Node
- func (p *Plan) DFSWalk(n Node, f dag.WalkFunc[Node], order dag.WalkOrder) error
- func (p *Plan) Graph() *dag.Graph[Node]
- func (p *Plan) Leaves() []Node
- func (p *Plan) Len() int
- func (p *Plan) Parent(n Node) []Node
- func (p *Plan) Root() (Node, error)
- func (p *Plan) Roots() []Node
- func (p *Plan) String() string
- type Planner
- type PointersScan
- type Projection
- type RangeAggregation
- type ScanSet
- type ScanTarget
- type ScanType
- type ShardInfo
- type ShardableNode
- type SortOrder
- type TimeRange
- type TopK
- type UnaryExpr
- type UnaryExpression
- type VariadicExpr
- type VectorAggregation
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildTree ¶
BuildTree converts a physical plan node and its children into a tree structure that can be used for visualization and debugging purposes.
func CatalogRequestToMetastoreSectionsRequest ¶ added in v3.7.0
func CatalogRequestToMetastoreSectionsRequest(selector Expression, predicates []Expression, start, end time.Time) (metastore.SectionsRequest, error)
func PrintAsTree ¶
PrintAsTree converts a physical Plan into a human-readable tree representation. It processes each root node in the plan graph, and returns the combined string output of all trees joined by newlines.
func WriteMermaidFormat ¶
Types ¶
type Batching ¶ added in v3.7.0
Batching is a plan node that controls how records are grouped into output batches. It wraps the root of the plan and is added after optimization.
func (*Batching) Clone ¶ added in v3.7.0
Clone returns a deep copy of the node with a new unique ID.
func (*Batching) ID ¶ added in v3.7.0
ID returns the ULID that uniquely identifies the node in the plan.
func (*Batching) Type ¶ added in v3.7.0
Type returns NodeTypeBatching.
type BinaryExpr ¶
type BinaryExpr struct {
Left, Right Expression
Op types.BinaryOp
}
BinaryExpr is an expression that implements the BinaryExpression interface.
func (*BinaryExpr) Clone ¶
func (e *BinaryExpr) Clone() Expression
Clone returns a copy of the BinaryExpr.
func (*BinaryExpr) String ¶
func (e *BinaryExpr) String() string
func (*BinaryExpr) Type ¶
func (*BinaryExpr) Type() ExpressionType
Type returns the type of the BinaryExpr.
type BinaryExpression ¶
type BinaryExpression interface {
Expression
// contains filtered or unexported methods
}
BinaryExpression is the common interface for all binary expressions in a physical plan.
type Catalog ¶
type Catalog interface {
// ResolveDataObjSections returns a list of
// DataObjSections objects, which each include:
// a data object path, a list of stream IDs for
// each data object path, a list of sections for
// each data object path, and a time range.
ResolveDataObjSections(Expression, []Expression, ShardInfo, time.Time, time.Time) ([]DataObjSections, error)
}
Catalog is an interface that provides methods for interacting with storage metadata. In traditional database systems there are system tables providing this information (e.g. pg_catalog, ...) whereas in Loki there is the Metastore.
type ColumnCompat ¶
type ColumnCompat struct {
NodeID ulid.ULID
// TODO(chaudum): These fields are poorly named. Come up with more descriptive names.
Source types.ColumnType // column type of the column that may colide with columns of the same name but with collision type
Destination types.ColumnType // column type of the generated _extracted column (should be same as source)
Collisions []types.ColumnType // column types of the columns that a source type column may collide with
}
ColumnCompat represents a compactibilty operation in the physical plan that moves a values from a conflicting metadata column with a label column into a new column suffixed with `_extracted`.
func (*ColumnCompat) Clone ¶
func (m *ColumnCompat) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*ColumnCompat) ID ¶
func (m *ColumnCompat) ID() ulid.ULID
ID implements the Node interface. Returns the ULID that uniquely identifies the node in the plan.
func (*ColumnCompat) Type ¶
func (m *ColumnCompat) Type() NodeType
Type implements the Node interface. Returns the type of the node.
type ColumnExpr ¶
ColumnExpr is an expression that implements the ColumnExpr interface.
func (*ColumnExpr) Clone ¶
func (e *ColumnExpr) Clone() Expression
Clone returns a copy of the ColumnExpr.
func (*ColumnExpr) String ¶
func (e *ColumnExpr) String() string
String returns the string representation of the column expression. It contains of the name of the column and its type, joined by a dot (`.`).
func (*ColumnExpr) Type ¶
func (e *ColumnExpr) Type() ExpressionType
Type returns the type of the ColumnExpr.
type ColumnExpression ¶
type ColumnExpression interface {
Expression
// contains filtered or unexported methods
}
ColumnExpression is the common interface for all column expressions in a physical plan.
type Context ¶
type Context struct {
// contains filtered or unexported fields
}
Context carries planning state that needs to be propagated down the plan tree. This enables each branch to have independent context, which is essential for complex queries: - Binary operations with different [$range] intervals, offsets or @timestamp.
Propagation: - Each process() method passes context to children. - Context can be cloned and modified by nodes (ex: RangeAggregation sets rangeInterval) that gets inherited by all descendants.
func NewContext ¶
func (*Context) GetResolveTimeRange ¶
func (*Context) WithDirection ¶
func (*Context) WithMaxQuerySeries ¶ added in v3.7.0
WithMaxQuerySeries sets the maximum number of unique series allowed for metric queries.
func (*Context) WithRangeInterval ¶
type DataObjLocation ¶
type DataObjLocation string
DataObjLocation is a string that uniquely identifies a data object location in object storage.
type DataObjScan ¶
type DataObjScan struct {
NodeID ulid.ULID
// Location is the unique name of the data object that is used as source for
// reading streams.
Location DataObjLocation
// Section is the section index inside the data object to scan.
Section int
// StreamIDs is a set of stream IDs inside the data object. These IDs are
// only unique in the context of a single data object.
StreamIDs []int64
// Projections are used to limit the columns that are read to the ones
// provided in the column expressions to reduce the amount of data that needs
// to be processed.
Projections []ColumnExpression
// Predicates are used to filter rows to reduce the amount of rows that are
// returned. Predicates would almost always contain a time range filter to
// only read the logs for the requested time range.
Predicates []Expression
// The maximum boundary of timestamps that scanning the
// data object can possibly emit. Does not account for
// predicates.
// MaxTimeRange is not read when executing a scan.
// It can be used as metadata to control physical plan execution.
MaxTimeRange TimeRange
}
DataObjScan represents a physical plan operation for reading data objects. It contains information about the object location, stream IDs, projections, predicates for reading data from a data object.
func (*DataObjScan) Clone ¶
func (s *DataObjScan) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*DataObjScan) ID ¶
func (s *DataObjScan) ID() ulid.ULID
ID implements the Node interface. Returns the ULID that uniquely identifies the node in the plan.
func (*DataObjScan) Type ¶
func (*DataObjScan) Type() NodeType
Type implements the Node interface. Returns the type of the node.
type DataObjSections ¶ added in v3.7.0
type DataObjSections struct {
Location DataObjLocation
Streams []int64
Sections []int
TimeRange TimeRange
PredicatesInStreams map[int64][]string
}
DataObjSections is a collection of sections from single data object.
type Expression ¶
type Expression interface {
fmt.Stringer
Clone() Expression
Type() ExpressionType
// contains filtered or unexported methods
}
Expression is the common interface for all expressions in a physical plan.
type ExpressionType ¶
type ExpressionType uint32
ExpressionType represents the type of expression in the physical plan.
const ( ExprTypeUnary ExpressionType ExprTypeBinary ExprTypeVariadic ExprTypeLiteral ExprTypeColumn )
func (ExpressionType) String ¶
func (t ExpressionType) String() string
String returns the string representation of the ExpressionType.
type Filter ¶
type Filter struct {
NodeID ulid.ULID
// Predicates is a list of filter expressions that are used to discard not
// matching rows during execution.
Predicates []Expression
}
Filter represents a filtering operation in the physical plan. It contains a list of predicates (conditional expressions) that are later evaluated against the input columns and produce a result that only contains rows that match the given conditions. The list of expressions are chained with a logical AND.
type FunctionExpression ¶ added in v3.7.0
type FunctionExpression interface {
Expression
// contains filtered or unexported methods
}
FunctionExpression is the common interface for all function expressions in a physical plan.
type Grouping ¶ added in v3.7.0
type Grouping struct {
Columns []ColumnExpression // The columns for grouping
Without bool // The grouping mode
}
Grouping represents the grouping by/without label(s) for vector aggregators and range vector aggregators.
type Join ¶ added in v3.7.0
Join represents a join operation in the physical plan. For now it is only an inner join on `timestamp`. Will be expanded later.
type Limit ¶
type Limit struct {
NodeID ulid.ULID
// Skip specifies how many initial rows should be skipped.
Skip uint32
// Fetch specifies how many rows should be returned in total.
Fetch uint32
}
Limit represents a limiting operation in the physical plan that applies offset and limit to the result set. The offset specifies how many rows to skip before starting to return results, while limit specifies the maximum number of rows to return.
type LiteralExpr ¶
type LiteralExpr struct {
// contains filtered or unexported fields
}
LiteralExpr is an expression that implements the LiteralExpression interface.
func NewLiteral ¶
func NewLiteral(value any) *LiteralExpr
func (*LiteralExpr) Clone ¶
func (e *LiteralExpr) Clone() Expression
Clone returns a copy of the LiteralExpr.
func (*LiteralExpr) Literal ¶ added in v3.7.0
func (e *LiteralExpr) Literal() types.Literal
Literal returns the underlying literal.
func (*LiteralExpr) String ¶
func (e *LiteralExpr) String() string
String returns the string representation of the literal value.
func (*LiteralExpr) Type ¶
func (*LiteralExpr) Type() ExpressionType
Type returns the type of the LiteralExpr.
func (*LiteralExpr) Value ¶ added in v3.7.0
func (e *LiteralExpr) Value() any
Value returns the value represented by the literal.
func (*LiteralExpr) ValueType ¶
func (e *LiteralExpr) ValueType() types.DataType
ValueType returns the kind of value represented by the literal.
type LiteralExpression ¶
type LiteralExpression interface {
Expression
ValueType() types.DataType
// contains filtered or unexported methods
}
LiteralExpression is the common interface for all literal expressions in a physical plan.
type Merge ¶ added in v3.7.0
Merge combines multiple input streams into a single stream with no guaranteed ordering.
Merge is primarily used as an aggregation point for distributed task execution where a parent task needs to read from many partitioned tasks.
type MetastoreCatalog ¶
type MetastoreCatalog struct {
// contains filtered or unexported fields
}
MetastoreCatalog is the default implementation of Catalog.
func NewMetastoreCatalog ¶
func NewMetastoreCatalog(sectionsResolver MetastoreSectionsResolver) *MetastoreCatalog
NewMetastoreCatalog creates a new instance of MetastoreCatalog for query planning.
func (*MetastoreCatalog) ResolveDataObjSections ¶ added in v3.7.0
func (c *MetastoreCatalog) ResolveDataObjSections(selector Expression, predicates []Expression, shard ShardInfo, start, end time.Time) ([]DataObjSections, error)
ResolveDataObjSections resolves an array of DataObjSections objects based on a given Expression. The expression is required to be a (tree of) BinaryExpression with a ColumnExpression on the left and a LiteralExpression on the right.
type MetastorePlanner ¶ added in v3.7.0
type MetastorePlanner struct {
// contains filtered or unexported fields
}
func NewMetastorePlanner ¶ added in v3.7.0
func NewMetastorePlanner(metastore metastore.Metastore, batchSize int) MetastorePlanner
func (MetastorePlanner) Plan ¶ added in v3.7.0
func (p MetastorePlanner) Plan(ctx context.Context, selector Expression, predicates []Expression, start time.Time, end time.Time) (*Plan, error)
type MetastoreSectionsResolver ¶ added in v3.7.0
type MetastoreSectionsResolver func(Expression, []Expression, time.Time, time.Time) ([]*metastore.DataobjSectionDescriptor, error)
type Node ¶
type Node interface {
// ID returns the ULID that uniquely identifies a node in the plan.
ID() ulid.ULID
// Type returns the node type
Type() NodeType
// Clone creates a deep copy of the Node. Cloned nodes do not retain the
// same ID.
Clone() Node
// contains filtered or unexported methods
}
Node represents a single operation in a physical execution plan. It defines the core interface that all physical plan nodes must implement. Each node represents a specific operation like scanning, filtering, or transforming data. Nodes can be connected to form a directed acyclic graph (DAG) representing the complete execution plan.
type NodeType ¶
type NodeType uint32
NodeType represents the type of a node in the physical execution plan.
const ( NodeTypeInvalid NodeType = iota // NodeTypeInvalid is an illegal NodeType. NodeTypeDataObjScan // NodeTypeDataObjScan represents a [DataObjScan]. NodeTypeProjection // NodeTypeProjection represents a [Projection]. NodeTypeFilter // NodeTypeFilter represents a [Filter]. NodeTypeLimit // NodeTypeLimit represents a [Limit]. NodeTypeRangeAggregation // NodeTypeRangeAggregation represents a [RangeAggregation]. NodeTypeVectorAggregation // NodeTypeVectorAggregation represents a [VectorAggregation]. NodeTypeMerge // NodeTypeMerge represents a [Merge]. NodeTypeCompat // NodeTypeCompat represents a [ColumnCompat]. NodeTypeTopK // NodeTypeTopK represents a [TopK]. NodeTypeParallelize // NodeTypeParallelize represents a [Parallelize]. NodeTypeScanSet // NodeTypeScanSet represents a [ScanSet]. NodeTypeJoin // NodeTypeJoin represents a [Join]. NodeTypePointersScan // NodeTypePointersScan represents a [PointersScan]. NodeTypeBatching // NodeTypeBatching represents a [Batching] node. )
type Parallelize ¶
Parallelize represents a hint to the engine to partition and parallelize the children branches of the Parallelize and emit results as a single sequence with no guaranteed order.
func (*Parallelize) Clone ¶
func (p *Parallelize) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*Parallelize) ID ¶
func (p *Parallelize) ID() ulid.ULID
ID returns the ULID that uniquely identifies the node in the plan.
type Plan ¶
type Plan struct {
// contains filtered or unexported fields
}
Plan represents a physical execution plan as a directed acyclic graph (DAG). It maintains the relationships between nodes, tracking parent-child connections and providing methods for graph traversal and manipulation.
The plan structure supports operations like adding nodes and edges, retrieving nodes by ID, retrieving parents and children of nodes, and walking the graph in different orders using the depth-first-search algorithm.
func WrapWithBatching ¶ added in v3.7.0
WrapWithBatching inserts a Batching node as the new root of plan, with the existing root as its only child. It modifies plan in-place and returns it.
func (*Plan) CalculateMaxTimeRange ¶ added in v3.7.0
CalculateMaxTimeRange calculates max time boundaries for the plan. Boundaries are defined by either the topmost RangeAggregation or by data scans.
func (*Plan) DFSWalk ¶
DFSWalk performs a depth-first traversal of the plan starting from node n. It applies the visitor v to each node according to the specified walk order. The order parameter determines if nodes are visited before their children (dag.PreOrderWalk) or after their children (dag.PostOrderWalk).
func (*Plan) Graph ¶ added in v3.7.0
Graph returns the underlying graph of the plan. Modifications to the returned graph will affect the Plan.
func (*Plan) Root ¶
Root returns the root node that have no parents. It returns an error if the plan has no or multiple root nodes.
func (*Plan) String ¶ added in v3.7.0
String returns a string representation of the plan. It is a convenience method for calling PrintAsTree.
type Planner ¶
type Planner struct {
// contains filtered or unexported fields
}
Planner creates an executable physical plan from a logical plan. Planning is done in two steps:
- Convert Instructions from the logical plan are converted into Nodes in the physical plan. Most instructions can be translated into a single Node. However, MakeTable translates into multiple DataObjScan nodes.
- Pushdown a) Push down the limit of the Limit node to the DataObjScan nodes. b) Push down the predicate from the Filter node to the DataObjScan nodes.
func NewPlanner ¶
NewPlanner creates a new planner instance with the given context.
type PointersScan ¶ added in v3.7.0
type PointersScan struct {
NodeID ulid.ULID
Location DataObjLocation
Selector Expression
Predicates []Expression
Start time.Time
End time.Time
}
func (*PointersScan) Clone ¶ added in v3.7.0
func (s *PointersScan) Clone() Node
func (*PointersScan) ID ¶ added in v3.7.0
func (s *PointersScan) ID() ulid.ULID
func (*PointersScan) MaxTimeRange ¶ added in v3.7.0
func (s *PointersScan) MaxTimeRange() TimeRange
func (*PointersScan) Type ¶ added in v3.7.0
func (s *PointersScan) Type() NodeType
type Projection ¶
type Projection struct {
NodeID ulid.ULID
// Expressions is a set of column expressions that are used to drop not needed
// columns that match the column expression, or to expand columns that result
// from the expressions.
Expressions []Expression
All bool // Marker for projecting all columns of input relation (similar to SQL `SELECT *`)
Expand bool // Indicates that projected columns should be added to input relation
Drop bool // Indicates that projected columns should be dropped from input Relation
}
Projection represents a column selection operation in the physical plan. It contains a list of columns (column expressions) that are later evaluated against the input columns to remove unnecessary colums from the intermediate result.
func (*Projection) Clone ¶
func (p *Projection) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*Projection) ID ¶
func (p *Projection) ID() ulid.ULID
ID implements the Node interface. Returns the ULID that uniquely identifies the node in the plan.
func (*Projection) Type ¶
func (*Projection) Type() NodeType
Type implements the Node interface. Returns the type of the node.
type RangeAggregation ¶
type RangeAggregation struct {
NodeID ulid.ULID
Grouping Grouping
Operation types.RangeAggregationType
Start time.Time
End time.Time
Step time.Duration // optional for instant queries
Range time.Duration
MaxQuerySeries int // maximum number of unique series allowed (0 means no limit)
}
TODO: Rename based on the actual implementation.
func (*RangeAggregation) Clone ¶
func (r *RangeAggregation) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*RangeAggregation) ID ¶
func (r *RangeAggregation) ID() ulid.ULID
ID returns the ULID that uniquely identifies the node in the plan.
func (*RangeAggregation) Type ¶
func (r *RangeAggregation) Type() NodeType
type ScanSet ¶
type ScanSet struct {
NodeID ulid.ULID
// Targets to scan.
Targets []*ScanTarget
// Projections are used to limit the columns that are read to the ones
// provided in the column expressions to reduce the amount of data that
// needs to be processed.
Projections []ColumnExpression
// Predicates are used to filter rows to reduce the amount of rows that are
// returned. Predicates would almost always contain a time range filter to
// only read the logs for the requested time range.
Predicates []Expression
}
ScanSet represents a physical plan operation for reading data from targets.
type ScanTarget ¶
type ScanTarget struct {
Type ScanType
// DataObj is non-nil if Type is [ScanTypeDataObject]. Despite DataObjScan
// implementing [Node], the value is not inserted into the graph as a node.
DataObject *DataObjScan
// Pointers is non-nil if Type is [ScanTypePointers]. Despite PointersScan
// implementing [Node], the value is not inserted into the graph as a node.
Pointers *PointersScan
}
ScanTarget represents a target of a ScanSet.
func (*ScanTarget) Clone ¶
func (t *ScanTarget) Clone() *ScanTarget
Clone returns a copy of the scan target.
type ScanType ¶
type ScanType int
ScanType represents the data being scanned in a target of a ScanSet.
type ShardableNode ¶ added in v3.7.0
type ShardableNode interface {
Node
// Shards produces a sequence of nodes that represent a fragment of the
// original node. Returned nodes do not need to be the same type as the
// original node.
//
// Implementations must produce unique values of Node in each call to
// Shards.
Shards() iter.Seq[Node]
}
ShardableNode is a Node that can be split into multiple smaller partitions.
type TopK ¶
type TopK struct {
NodeID ulid.ULID
// SortBy is the column to sort by.
SortBy ColumnExpression
Ascending bool // Sort lines in ascending order if true.
NullsFirst bool // When true, considers NULLs < non-NULLs when sorting.
K int // Number of top rows to return.
}
TopK represents a physical plan node that performs topK operation. It ranks rows based on sort expressions and limits the result to the top K rows. Implementations may not guarantee the topK rows to be in sorted order.
type UnaryExpr ¶
type UnaryExpr struct {
// Left is the expression being operated on
Left Expression
// Op is the unary operator to apply to the expression
Op types.UnaryOp
}
UnaryExpr is an expression that implements the UnaryExpression interface.
func (*UnaryExpr) Clone ¶
func (e *UnaryExpr) Clone() Expression
Clone returns a copy of the UnaryExpr.
func (*UnaryExpr) Type ¶
func (*UnaryExpr) Type() ExpressionType
Type returns the type of the UnaryExpr.
type UnaryExpression ¶
type UnaryExpression interface {
Expression
// contains filtered or unexported methods
}
UnaryExpression is the common interface for all unary expressions in a physical plan.
type VariadicExpr ¶ added in v3.7.0
type VariadicExpr struct {
// Op is the function operation to apply to the parameters
Op types.VariadicOp
// Expressions are the parameters paaws to the function
Expressions []Expression
}
VariadicExpr is an expression that implements the FunctionExpression interface.
func (*VariadicExpr) Clone ¶ added in v3.7.0
func (e *VariadicExpr) Clone() Expression
Clone returns a copy of the VariadicExpr.
func (*VariadicExpr) String ¶ added in v3.7.0
func (e *VariadicExpr) String() string
func (*VariadicExpr) Type ¶ added in v3.7.0
func (*VariadicExpr) Type() ExpressionType
Type returns the type of the VariadicExpr.
type VectorAggregation ¶
type VectorAggregation struct {
NodeID ulid.ULID
Grouping Grouping // Grouping of the data.
// Operation defines the type of aggregation operation to perform (e.g., sum, min, max)
Operation types.VectorAggregationType
// MaxQuerySeries is the maximum number of unique series allowed (0 means no limit)
MaxQuerySeries int
}
VectorAggregation represents a physical plan node that performs vector aggregations. It computes aggregations over time series data at each timestamp instant, grouping results by specified dimensions.
func (*VectorAggregation) Clone ¶
func (v *VectorAggregation) Clone() Node
Clone returns a deep copy of the node with a new unique ID.
func (*VectorAggregation) ID ¶
func (v *VectorAggregation) ID() ulid.ULID
ID implements the Node interface. Returns the ULID that uniquely identifies the node in the plan.
func (*VectorAggregation) Type ¶
func (*VectorAggregation) Type() NodeType
Type implements the Node interface. Returns the type of the node.