logical

package
v0.15.0-aggregation-en... Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Overview

Package logical provides logical query plan representation and optimization.

Index

Constants

View Source
const RowCountOnlyColumn = "__rowcount_only__"

RowCountOnlyColumn is the sentinel required-column emitted for scans that only need a row count (bare COUNT(*) / literal-only projections). The physical scan drops it when any real column is required and otherwise projects the narrowest schema column instead of all columns. The "__" prefix keeps it flowing through sanitizeScanNeeds and makes the distributed worker's all-or-nothing projection guard fall back to full width (safe) if it ever reaches that path.

Variables

View Source
var BushyJoinReorder atomic.Bool

BushyJoinReorder enables bushy subset-partition transitions in the DP join reorder (docs/design/bushy-join-cbo.md §3.2). Process-wide, set once at startup from --bushy-join-reorder / wadjet.Config; default off. Read at plan time, so tests may toggle it around a planning call.

View Source
var BushyJoinsPlanned atomic.Int64

BushyJoinsPlanned counts queries whose FINAL chosen join order contains at least one bushy join (a join of two composite intermediates). Mechanism marker for A/B runs: a nonzero count proves the enumeration actually changed a plan; dormancy tests assert it stays zero with the flag off.

View Source
var ScalarAggSemijoin atomic.Bool

ScalarAggSemijoin gates reduceDecorrelatedScalarAggs. Kill switch: WADJET_SCALAR_AGG_SEMIJOIN=0 (mirrors WADJET_SCALAR_DEFER / WADJET_EXCHANGE_ELIDE). Default on. Exported as an atomic (the BushyJoinReorder pattern) so tests that exercise the UNreduced decorrelated shape — aggregate-shuffle and dynamic-filter fixtures built on Q17/Q20 — can flip it with a defer-restore.

Functions

func HasRemainingSubqueries

func HasRemainingSubqueries(n *Node) bool

HasRemainingSubqueries walks an optimized logical plan and returns true if any filter predicate still contains un-decorrelated subquery references (EXISTS, NOT EXISTS, IN (SELECT), scalar subqueries). After successful decorrelation these become semi/anti join nodes and the predicates are removed. If this returns false, the scan-node structure is deterministic and safe for distributed execution.

func NodeColumnRefs

func NodeColumnRefs(n *Node) []string

NodeColumnRefs returns the column names referenced by a node's own expressions (filter predicates, sort keys, projections, ...) — the same collection column pruning uses, exported for the physical planner's top-N late-materialization rewrite.

func SetSemiPushdownEnabled

func SetSemiPushdownEnabled(on bool) bool

SetSemiPushdownEnabled toggles the rewrite (tests that pin plan shapes downstream of the legacy join order run with it off). Returns the previous value so callers can restore it.

func SubstitutePreComputedAggregates

func SubstitutePreComputedAggregates(root *Node, sigs []PreComputedAggregate) (map[string]bool, error)

SubstitutePreComputedAggregates walks the plan tree and replaces each Aggregate node whose shape matches one of sigs with a synthetic Scan. Returns the set of substitutions actually performed (alias → signature), so the caller can verify every pre-computed input was consumed. A signature that never matched is logged by the caller; this pass does not error on unmatched signatures (the worker may have re-planned a shape that no longer contains the aggregate, in which case falling back to the in-pipeline execution is correct).

Matching is conservative: a signature must match on GroupByCols (order and content) AND on the full set of AggExpr output names. Any other cases fall through untouched.

Types

type AggExpr

type AggExpr struct {
	Func      string // sum, count, min, max, avg
	InputCol  string
	OutputCol string
	Distinct  bool         // COUNT(DISTINCT col)
	InputExpr plansql.Node // AST for aggregate argument (nil for simple column refs)
}

AggExpr is an aggregation expression.

type ColumnPolicy

type ColumnPolicy struct {
	Column   string
	Denied   bool   // true = column excluded from results
	MaskExpr string // non-empty = column replaced with this value (e.g., "'***'", "0")
}

ColumnPolicy describes a column-level security policy for plan-level enforcement.

type HistogramStats

type HistogramStats interface {
	SelectivityLE(v any) float64
	SelectivityLT(v any) float64
	SelectivityRange(lo, hi any) float64
	SelectivityEQ(v any) float64
}

HistogramStats is the interface a per-column histogram must implement to participate in selectivity estimation. catalog.Histogram satisfies it. Kept as an interface so the logical package stays free of a catalog dependency.

type MergeInfo

type MergeInfo struct {
	GroupBy      []string
	AggExprs     []AggExpr
	OrderBy      []OrderExpr
	Limit        int
	HasAggregate bool
	HasDistinct  bool // DISTINCT requires deduplication across partials
}

MergeInfo describes how to merge probe-split partial results.

func ExtractMergeInfo

func ExtractMergeInfo(plan *Node) *MergeInfo

ExtractMergeInfo extracts the top-level aggregate and sort/limit information needed to merge probe-split partial results. Returns nil if the plan doesn't have a top-level aggregate (probe-split merge not needed).

type Node

type Node struct {
	Type     NodeType
	Children []*Node

	// Scan
	TableName         string
	TableAlias        string
	ScanColumns       []string                   // column names available from this scan (populated by physical planner)
	RequiredColumns   []string                   // columns actually needed from this scan (set by optimizer column pruning)
	PartitionFilter   map[string]string          // extracted partition key filters (year, month, day, hour)
	ScanPredicates    []Predicate                // pushed-down filter predicates for row group pruning
	ScanRowEstimate   int64                      // estimated row count from manifest (0 = unknown)
	ScanColStats      map[string]ScanColumnStats // aggregated column stats from catalog (nil = unavailable)
	FilterOnlyColumns []string                   // columns needed ONLY by the filter directly above this scan (candidates for scan-level filter evaluation without materialization)
	SampleMethod      string                     // TABLESAMPLE method: BERNOULLI, SYSTEM
	SamplePercent     float64                    // percentage for TABLESAMPLE (0-100)

	// Table Function (e.g., read_json, read_csv, unnest)
	IsTableFunc    bool              // true if this scan reads from a table function
	FuncName       string            // function name (e.g., "read_json")
	FuncArgs       []string          // positional arguments (e.g., URL/path)
	FuncNamedArgs  map[string]string // named arguments (e.g., delimiter="|")
	WithOrdinality bool              // UNNEST(...) WITH ORDINALITY
	FuncColAliases []string          // AS alias(col1, col2, ...)

	// Filter
	Predicates []Predicate

	// Project
	Projections []Projection
	// SecurityBarrier marks a projection injected by ABAC column-policy
	// enforcement (InjectColumnPolicies): masked columns replaced with
	// literal expressions, denied columns absent. The physical planner
	// must APPLY it at the scan (distributed walkStages treats ordinary
	// Projects as passthrough — a dropped barrier would leak raw values).
	SecurityBarrier bool

	// Aggregate
	// PreservesAggOutputs marks a synthetic Project inserted by a rewrite
	// directly above an Aggregate that passes every aggregate output
	// through under its original name (possibly adding finalizations,
	// e.g. the two-level AVG division). The physical builder's
	// aggregate-ancestor resolution walks through such projections so
	// SELECT-list aggregate references still resolve by output name.
	PreservesAggOutputs bool
	// ScanStrictIntCols marks scan columns whose vectors are plain
	// Int64/Int32 at runtime — exactly the set expr.BinOpNumeric resolves
	// to integer arithmetic. Planner-side int typing must stay a subset
	// of the runtime rule (a declared-int column over a float-mode expr
	// would read as NULL through the typed getter), so this is
	// deliberately narrower than ScanIntCols' int-class set.
	ScanStrictIntCols map[string]bool
	// ScanIntCols marks scan columns whose types land on the typed
	// integer aggregation paths (set by physical.AnnotateScanColumns;
	// consumed by the two-level distinct rewrite's cost gate).
	ScanIntCols map[string]bool

	GroupBy          []string
	GroupByExprs     []plansql.Node // AST for GROUP BY expressions (may be nil)
	AggExprs         []AggExpr
	GroupingSetNulls []string   // columns that should be NULL in this grouping set (legacy, per-node)
	GroupingSets     [][]string // single-pass grouping sets: each entry lists the columns in that set

	// Sort
	OrderBy []OrderExpr

	// Limit
	LimitVal  int
	OffsetVal int

	// Join
	JoinType      string // inner, left, right, full, cross, semi, anti
	JoinCond      string
	JoinFilter    string // non-equality join conditions for semi/anti join
	LeftKeys      []string
	RightKeys     []string
	NeededColumns []string // columns the parent needs from this join's output (set by optimizer)

	// Window
	WindowExprs []WindowExpr

	// Union
	UnionAll bool // true = UNION ALL, false = UNION (dedup)

	// CTEs — stored on the root node so the physical planner can resolve
	// CTE references inside scalar subqueries (e.g., Q15's HAVING/WHERE
	// subquery that references a CTE defined in the outer WITH clause).
	CTEs []plansql.CTEDef

	// CTEName — set on the root of a CTE sub-plan so the physical planner
	// can detect and materialize multi-referenced CTEs.
	CTEName string

	// ScalarDecorrelated marks a LEFT join produced by
	// decorrelateScalarSubqueries (children[1] is the grouped aggregate
	// materializing the subquery result). reduceDecorrelatedScalarAggs
	// uses it after predicate pushdown to semijoin-reduce the aggregate's
	// input by the outer plan's key-source branch.
	ScalarDecorrelated bool
}

Node is a node in the logical query plan tree.

func BuildFromSelect

func BuildFromSelect(info *plansql.SelectInfo) (*Node, error)

BuildFromSelect constructs a logical plan from a parsed SELECT query.

func BuildFromSelectWithCTEs

func BuildFromSelectWithCTEs(info *plansql.SelectInfo, ctes []plansql.CTEDef) (*Node, error)

BuildFromSelectWithCTEs constructs a logical plan, resolving CTE references to inline sub-plans instead of table scans.

func InjectColumnPolicies

func InjectColumnPolicies(plan *Node, tableName string, policies []ColumnPolicy) *Node

InjectColumnPolicies walks the logical plan and inserts a security projection above Scan nodes for the given table. Denied columns are removed and masked columns are replaced with literal expressions. This ensures restricted data never enters the execution pipeline.

func InjectRowFilter

func InjectRowFilter(plan *Node, tableName, filterSQL string) *Node

InjectRowFilter walks the logical plan tree and wraps Scan nodes for the given table with an additional Filter node containing the row filter predicate. This is used by row-level security policies to restrict which rows a role can see.

func NewAggregate

func NewAggregate(child *Node, groupBy []string, aggs []AggExpr) *Node

NewAggregate creates an aggregate node.

func NewDistinct

func NewDistinct(child *Node) *Node

NewDistinct creates a distinct node.

func NewExcept

func NewExcept(left, right *Node, all bool) *Node

NewExcept creates an except node. Returns rows from left that are not in right. If all is true (EXCEPT ALL), preserves duplicates; otherwise deduplicates.

func NewFilter

func NewFilter(child *Node, predicates []Predicate) *Node

NewFilter creates a filter node.

func NewIntersect

func NewIntersect(left, right *Node, all bool) *Node

NewIntersect creates an intersect node. Returns only rows present in both sides. If all is true (INTERSECT ALL), preserves duplicates; otherwise deduplicates.

func NewJoin

func NewJoin(left, right *Node, joinType, condition string) *Node

NewJoin creates a join node.

func NewLimit

func NewLimit(child *Node, limit, offset int) *Node

NewLimit creates a limit node.

func NewProject

func NewProject(child *Node, projections []Projection) *Node

NewProject creates a projection node.

func NewScan

func NewScan(table, alias string) *Node

NewScan creates a scan node.

func NewSort

func NewSort(child *Node, orderBy []OrderExpr) *Node

NewSort creates a sort node.

func NewUnion

func NewUnion(left, right *Node, all bool) *Node

NewUnion creates a union node. If all is true, it represents UNION ALL (no deduplication); otherwise it represents UNION (with deduplication).

func NewWindow

func NewWindow(child *Node, exprs []WindowExpr) *Node

NewWindow creates a window node.

func Optimize

func Optimize(plan *Node, annotators ...func(*Node)) *Node

Optimize applies logical optimizations to the plan tree.

An optional ScanAnnotator function may be provided to populate scan metadata (ScanColumns) on newly created scan nodes after IN-to-SemiJoin conversion. This enables subsequent scalar subquery decorrelation to resolve unqualified column references. Without an annotator, scalar decorrelation may fail for subqueries that use unqualified outer column references.

func StripTopSortLimit

func StripTopSortLimit(plan *Node) *Node

StripTopSortLimit removes the outermost Sort and Limit nodes from a logical plan. Used by probe-split pipeline: each worker produces partial aggregates without ordering or truncation; the coordinator merges and applies final sort + limit.

func (*Node) PrettyPrint

func (n *Node) PrettyPrint(indent int) string

PrettyPrint returns a formatted string representation of the plan tree.

type NodeType

type NodeType int

NodeType identifies the kind of logical plan node.

const (
	NodeScan NodeType = iota
	NodeFilter
	NodeProject
	NodeAggregate
	NodeSort
	NodeLimit
	NodeJoin
	NodeDistinct
	NodeWindow
	NodeUnion
	NodeIntersect
	NodeExcept
	NodeDual // single-row, zero-column source for table-less SELECT
)

func (NodeType) String

func (n NodeType) String() string

type OrderExpr

type OrderExpr struct {
	Column     string
	Desc       bool
	NullsFirst *bool // nil = default, true = NULLS FIRST, false = NULLS LAST
}

OrderExpr is a sort expression.

type PreComputedAggregate

type PreComputedAggregate struct {
	InputTable     string
	GroupByCols    []string
	AggOutputCols  []string // OutputCol names from each AggSpec, in order
	SyntheticAlias string   // unique scan alias to substitute
}

PreComputedAggregate is a signature describing a derived aggregate whose result is already materialized elsewhere. When a logical-plan Aggregate node matches a signature, SubstitutePreComputedAggregates replaces it with a Scan node bearing SyntheticAlias; the caller (worker) is expected to wire SyntheticAlias → CacheFiles into the physical planner's StreamingSources map so the scan reads the cached rows directly.

Phase 1 matches only aggregates that are GROUP BY GroupByCols over a single Scan of InputTable (no filters on the scan, no nested joins). More general shapes are rejected and left in-plan.

type Predicate

type Predicate struct {
	Column  string
	Op      string // =, !=, <, <=, >, >=, is_null, is_not_null, in, between
	Value   any
	Raw     string       // raw SQL expression
	ASTExpr plansql.Node // compiled AST expression node
	// PruneOnly marks predicates attached solely for storage-level pruning
	// (row-group stats / dictionary probes). The cardinality estimator
	// ignores them: attaching AST-decomposed conjuncts must not shift
	// distributed plan choices (Q08's orders join flipped shuffle →
	// broadcast off a 0.33^n selectivity guess the moment they appeared).
	// Estimate-visible attachment is a separate, SF100-validated change.
	PruneOnly bool
}

Predicate is a filter condition.

func SplitConjunctsForPushdown

func SplitConjunctsForPushdown(expr plansql.Node) ([]Predicate, []plansql.Node)

SplitConjunctsForPushdown decomposes an AST filter into top-level AND-conjuncts and partitions them: structured `column <op> literal` comparisons (as Predicates) and everything else (residual ASTs). The physical planner uses this to push eligible conjuncts into the scan and compile only the residue as the exec filter.

type Projection

type Projection struct {
	Column  string // column reference
	Alias   string // output name
	Expr    string // raw expression
	IsAgg   bool
	ASTExpr plansql.Node // compiled AST expression node (nil for aggregates)
}

Projection is a column expression in a SELECT.

type RelStats

type RelStats struct {
	Rows   float64
	ColNDV map[string]float64 // lowercase unqualified column name → estimated NDV
	// ColHist carries opaque histogram pointers (*catalog.Histogram) for
	// columns that have one in the catalog. Used by estimatePredSelectivity
	// to compute range/equality selectivity from real data distributions.
	// Stored as any to keep the logical package free of catalog imports.
	ColHist map[string]any
}

RelStats holds estimated statistics for a plan subtree.

func RelStatsOf

func RelStatsOf(n *Node) RelStats

RelStatsOf returns CBO-derived statistics for the given subtree. Exported so the physical planner can consult cardinality-driven estimates when making structural decisions (broadcast vs shuffle, dynamic-filter eligibility, etc.) without re-implementing the recursion.

type ScanColumnStats

type ScanColumnStats struct {
	MinValue  any
	MaxValue  any
	NullCount int64
	TotalRows int64
	// NDV is the catalog's merged HyperLogLog estimate of distinct values
	// across all files of this column. Zero means HLL wasn't collected
	// (legacy files / pre-ANALYZE state); planner falls back to the
	// min/max-range heuristic. When >0 it's preferred — orders of
	// magnitude more accurate for sparse-int / string columns where the
	// range overstates true cardinality.
	NDV int64
	// Histogram is the catalog's equi-depth histogram for this column,
	// merged across files from reservoir samples. Nil when not collected.
	// Used by estimatePredSelectivity to compute range/equality
	// selectivity from real value distributions instead of hardcoded
	// fractions (0.33 for <, 0.1 for =). The opaque any allows the
	// logical package to receive *catalog.Histogram without importing
	// the catalog package (avoids circular import).
	Histogram any
}

ScanColumnStats holds aggregated column statistics from the catalog.

type WindowBound

type WindowBound struct {
	Type   string // "unbounded_preceding", "preceding", "current_row", "following", "unbounded_following"
	Offset int
}

WindowBound describes one end of a window frame.

type WindowExpr

type WindowExpr struct {
	Func        string // row_number, rank, dense_rank, sum, count, avg, min, max
	InputCol    string // for aggregate window functions
	OutputCol   string
	PartitionBy []string
	OrderBy     []OrderExpr
	Frame       *WindowFrameSpec
}

WindowExpr is a window function expression.

type WindowFrameSpec

type WindowFrameSpec struct {
	Mode  string // "rows" or "range"
	Start WindowBound
	End   WindowBound
}

WindowFrameSpec describes a window frame specification.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL