pushdown

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrFallback = errors.New("pushdown: leaf falls back to Tier B")

ErrFallback is returned by Leaf.Exec when a leaf that is statically eligible cannot run in SQL for these parameters (e.g. too many buckets); the leaf then runs on Tier B.

View Source
var TierALeaves atomic.Int64

TierALeaves counts successful Tier A leaf executions (for coverage assertions).

Functions

func Map

func Map(expr syntax.SampleExpr) (mapped syntax.SampleExpr, leaves []syntax.SampleExpr, ok bool)

Map wraps every *syntax.RangeAggregationExpr in the sample expression in a logql.DownstreamSampleExpr so the downstream engine dispatches each leaf to the Downstreamer. It returns the mapped expression, the list of original leaves, and ok=false if the tree contains a node kind Map does not handle (the caller then runs the original query on Tier B).

A sum or sum by over a range aggregation is itself a leaf; the outer sum stays above it (as in Loki's shard mapper) and re-aggregates what the leaf returns, which lets Tier A group by only the structured-metadata keys the grouping can see. Other vector aggregations map their child as before.

Vector aggregations and arithmetic over mapped leaves may differ from Loki's inline evaluation in the last float bits: Loki builds each step's vector from a Go map, so its own summation order is random. Results agree within the oracle's 1e-9 relative tolerance.

The input must be a freshly parsed expression; Map never mutates it.

func NewConnBudget

func NewConnBudget(maxOpenConns int) *semaphore.Weighted

NewConnBudget returns the process-wide budget of ClickHouse connections that parallel pieces may reserve: half of the pool, and never less than one.

Types

type Engine

type Engine struct {
	Mode   Mode
	TierB  *logql.QueryEngine
	TierA  Leaf // nil until it is wired (Task 6)
	Limits logql.Limits
	Logger log.Logger
	// Split, when set, runs single-tenant Tier B evaluations through the time
	// splitter. It wraps TierB.
	Split *Splitter
}

Engine routes a query to the plain Tier B engine or to Loki's downstream engine with range-aggregation leaves wrapped for push-down.

func (*Engine) Downstreamer

func (e *Engine) Downstreamer(ctx context.Context) logql.Downstreamer

Downstreamer implements logql.Downstreamable: it executes each downstream query.

func (*Engine) Query

func (e *Engine) Query(ctx context.Context, p logql.Params) logql.Query

Query returns the query to execute for p: the mapped downstream query when the mode and the expression allow it, otherwise TierB.Query(p).

type Leaf

type Leaf interface {
	Eligible(e syntax.SampleExpr) bool
	Exec(ctx context.Context, e syntax.SampleExpr, p logql.Params) (logqlmodel.Result, error)
}

Leaf executes one leaf in SQL: a range aggregation or a sum over one (see Map).

type Mode

type Mode string

Mode selects the push-down behavior.

const (
	// ModeOn enables Tier A push-down for eligible leaves, falling back to Tier B.
	ModeOn Mode = "on"
	// ModeOff disables push-down; everything runs on the Phase 1 engine.
	ModeOff Mode = "off"
	// ModeDownstream maps every query but runs all leaves on Tier B.
	ModeDownstream Mode = "downstream"
)

type SplitLimits

type SplitLimits interface {
	logql.Limits

	QuerySplitDuration(userID string) time.Duration
	// contains filtered or unexported methods
}

SplitLimits are the per-tenant limits the Splitter reads; *validation.Overrides satisfies it.

type SplitMetrics

type SplitMetrics struct {
	// contains filtered or unexported fields
}

SplitMetrics are the splitter's Prometheus metrics. A nil *SplitMetrics observes nothing.

func NewSplitMetrics

func NewSplitMetrics(r prometheus.Registerer) *SplitMetrics

NewSplitMetrics registers loki_query_frontend_partitions on r.

Copied from grafana/loki@39c256e50a804c045d5eb4e3c817a31653b535fe pkg/querier/queryrange/split_by_interval.go; modifications: type and field names, registered by the caller's registerer.

type Splitter

type Splitter struct {
	Engine  *logql.QueryEngine
	Limits  SplitLimits
	Budget  *semaphore.Weighted // process-wide connection budget, see NewConnBudget
	Metrics *SplitMetrics
}

Splitter has the shape of *logql.QueryEngine. Its queries cut a metric range query into time legs on the query's own step grid, evaluate the legs on Engine in parallel (see runPieces) and merge them into the answer the unsplit query would give. Everything else runs unsplit on Engine.

func (*Splitter) Query

func (s *Splitter) Query(p logql.Params) logql.Query

Query returns the query to execute for p.

type TierA

type TierA struct {
	Conn       driver.Conn
	MaxBuckets int64 // default 100_000; above it Exec returns ErrFallback
	Limits     logql.Limits
	// Deletes, when set, is checked at the start of Exec: a pending delete
	// overlapping the leaf's range makes the leaf fall back to Tier B, which
	// applies Loki's delete filter.
	Deletes querierdeletion.DeleteGetter
}

TierA executes range-aggregation leaves in ClickHouse.

func (*TierA) Eligible

func (a *TierA) Eligible(e syntax.SampleExpr) bool

Eligible reports whether the leaf (a range aggregation, or a sum over one) can be executed in Tier A.

func (*TierA) Exec

Exec executes the leaf in Tier A. For a sum leaf it runs the inner range aggregation grouped by only the structured-metadata keys the sum's grouping can see and returns the matrix un-summed: the outer sum that Map keeps above the leaf completes the aggregation. A Tier B fallback returns the summed vector instead, which the outer sum leaves unchanged.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL