fetch

type Batch ¶

type Batch struct {
	ID         signal.SeriesID
	Series     signal.Series
	Timestamps []int64
	Values     []float64

	// ScaleFactors carries each sample's lossy-sampling weight (metrics only): a kept sample
	// with ScaleFactors[i] = N "represents" N original samples that budgeted sampling dropped
	// (DESIGN §8a). It is nil when no sampling occurred (every weight is 1), so the common path
	// is unaffected; when non-nil its length matches Values. The storage layer only *carries* the
	// weight — an embedder's aggregation multiplies it back into count/sum/rate to stay unbiased
	// (a gauge read ignores it). Use [Batch.ScaleFactor] to read it with the nil default.
	ScaleFactors []float64

	// Columns are the materialized per-record columns (logs); nil for metrics. Each column's
	// length matches Timestamps. The named layout is the engine's (e.g. severity, body, attrs).
	Columns []NamedColumn
	// contains filtered or unexported fields
}

Batch is one matching identity (a metric series, or a log stream) and its rows within the request window. For metrics the rows are (Timestamps, Values) samples. For logs the rows are the per-record Columns (the projected set); Timestamps still carries each record's time.

func Drain(ctx context.Context, it Iterator) ([]*Batch, error)

func MergeBatches(groups ...[]*Batch) []*Batch

func (b *Batch) Column(name string) (NamedColumn, bool)

func (b *Batch) Release()

func (b *Batch) ReleaseState() any

func (b *Batch) ScaleFactor(i int) float64

func (b *Batch) SetRelease(fn func(*Batch))

func (b *Batch) SetReleaseState(s any)

type Condition ¶ added in v0.2.0

type Condition struct {
	Column string
	Match  func(value signal.Value) bool
	Tokens [][]byte
	Equal  *EqualMatcher
}

Condition is one columnar predicate (logs): the rows whose value in column Column satisfy Match. Like Matcher it is operator-free — the language layer supplies the predicate.

A row that does not carry Column at all (no such fixed column, or no such per-record attribute) is offered to Match as signal.EmptyValue — which signal.Value.Kind reports as signal.KindEmpty, distinct from an empty string. Absence is a value the predicate judges, not a non-match the engine decides: a negation ("column != x") or an is-unset predicate must see those rows. A predicate that only accepts a concrete value must therefore reject signal.KindEmpty — which also keeps the hints below sound, since a pruning hint asserts a value is present.

Two optional, serializable hints let a fetcher prune whole parts before scanning (the engine always re-checks Match per row, so a hint only ever skips work, never changes results):

• Tokens: the full-text tokens the column value must contain (lowered) — consulted against a per-part token bloom for a `contains` condition (an empty Tokens ⇒ not full-text).
• Equal: an exact column=value equality — consulted against a per-part value bloom. For a per-record attribute condition, Column is the attribute key and Equal carries key=value.

type Counter ¶ added in v0.17.0

type Counter interface {
	Count(ctx context.Context, r Request) (int, error)
}

Counter is an optional Fetcher capability: it returns the number of series matching r.Matchers with at least one sample in [r.Start, r.End] without materializing samples or labels. It backs the PromQL `count(<selector>)` pushdown. A Fetcher that does not implement it simply opts out of the pushdown (the caller falls back to Fetch).

func CounterOf(f Fetcher) Counter

type EqualMatcher ¶

type EqualMatcher struct {
	Name  string
	Value string
}

EqualMatcher is the serializable form of an exact label-equality predicate (see Matcher.Spec). EqualMatcher.Predicate reconstructs the equivalent Matcher.Match.

func EqualitySpecs(matchers []Matcher) []EqualMatcher

func (m EqualMatcher) Predicate() func(signal.Value) bool

type Fetcher ¶

type Fetcher interface {
	Fetch(ctx context.Context, r Request) (Iterator, error)
}

Fetcher resolves a Request to an Iterator. It is implemented by the head, each part, and (later) the cluster fan-out.

func Filter(f Fetcher) Fetcher

func Merge(fetchers ...Fetcher) Fetcher

type GroupCounter ¶ added in v0.25.0

type GroupCounter interface {
	CountBy(ctx context.Context, r Request, label []byte) (map[string]int, error)
}

GroupCounter is an optional Fetcher capability, the grouped variant of Counter: CountBy returns, for each distinct canonical-text value of the label among the series matching r.Matchers, how many of them have at least one sample in [r.Start, r.End] — without materializing samples or projecting labels into results. It backs the PromQL `count by (label)(<selector>)` pushdown (and, via the map's length, `count(count by (label)(...))` = distinct label values). Matched series without the label group under the "" key. A Fetcher that does not implement it opts out of the pushdown (the caller falls back to Fetch).

func GroupCounterOf(f Fetcher) GroupCounter

type Iterator ¶

type Iterator interface {
	Next(ctx context.Context) (*Batch, error)
	Close() error
}

Iterator yields batches lazily; Next returns (nil, io.EOF) at the end.

type LabelLister ¶ added in v0.37.0

type LabelLister interface {
	LabelNames(ctx context.Context, r Request) ([]string, error)
	LabelValues(ctx context.Context, r Request, name []byte) ([]string, error)
}

LabelLister is an optional Fetcher capability, the label-metadata twin of SeriesLister: it returns the distinct label names, or one name's distinct canonical-text values, across the series matching r.Matchers that hold samples in [r.Start, r.End] — sorted and deduplicated.

It exists because the answer is a handful of strings while the identity set behind it can be millions: a producer backed by an inverted index answers an unmatched query in O(distinct values) without materializing a single identity. Callers must therefore pass **only pushable matchers** (the producer returns no identities, so the caller cannot re-check a matcher it could not push); with a non-pushable matcher, fall back to SeriesLister or Fetch. The window filter is coarse in the same way SeriesLister's is.

func LabelListerOf(f Fetcher) LabelLister

type Matcher ¶

type Matcher struct {
	Name  []byte
	Match func(value signal.Value) bool

	// Spec is an optional **serializable** form of an equality predicate, set by the language
	// layer when Match is an exact compare. It lets the cluster fan-out push a selective
	// matcher (e.g. `__name__="metric"`) to a peer node — the Match closure cannot cross the
	// wire. It is metadata only: a [Fetcher] always matches via Match; a peer reconstructs an
	// equivalent closure from Spec. Only equality is carried (it is exact, so a peer's pushdown
	// never drops a matching series); other predicates fall back to the requester's re-check.
	Spec *EqualMatcher
}

Matcher is one label condition: the predicate Match selects which values of the attribute Name satisfy it. The condition is a **callback**, not an operator enum, so the contract carries no query-language semantics — a language supplies the predicate (a compiled regexp, an exact compare, a typed numeric range, a custom rule) over the typed signal.Value. A Fetcher applies Match while scanning the label's distinct values.

Negation and absent-label semantics compose in the predicate, which is the only place a language can express them through this seam: a series that does not carry Name at all is offered signal.EmptyValue (signal.KindEmpty, distinct from an empty string), and a matcher that accepts it selects those series too. A matcher that wants only series carrying a concrete value must therefore reject signal.KindEmpty — which also keeps Matcher.Spec pushdown sound, since an equality asserts the label is present.

type NamedColumn ¶ added in v0.2.0

type NamedColumn struct {
	Name    string
	Int64   []int64
	Float64 []float64
	Bytes   [][]byte
}

NamedColumn is one materialized column of a log Batch: its name and exactly one populated typed slice (Int64/Float64/Bytes), matching the physical column kind. Row i of the batch is Int64[i] / Float64[i] / Bytes[i] for that column.

type Request ¶

type Request struct {
	Tenant     signal.TenantID
	Signal     signal.Signal // 0 ⇒ metric (the default vertical); Log for the logs read path
	Start, End int64         // unix nanos, inclusive
	Matchers   []Matcher

	// Conditions are columnar predicates applied per record (logs). Each names a column and
	// carries an operator-free Match callback over the row's typed value (mirroring Matcher).
	Conditions []Condition
	// AllConditions, when true, ANDs the conditions; a fetcher may still return a superset (an
	// approximate index like a bloom is re-checked by the requester). False ⇒ the fetcher need
	// not apply them at all (pure fetch-all; the caller filters).
	AllConditions bool
	// Projection names the columns to materialize for surviving rows (the second pass). Empty ⇒
	// the fetcher's default column set. Filter columns are decoded regardless of Projection.
	Projection []string
	// SecondPass, when set, is an engine-side row filter applied after the column Conditions —
	// for predicates not expressible as a single-column Match (e.g. a per-record attribute
	// decoded from the serialized attrs column). It sees the candidate row's materialized Batch.
	SecondPass func(*Batch) bool

	// Limit, when > 0, bounds the records returned to the most recent (Reverse) or oldest by
	// timestamp across all matched streams — the ordered top-N pushdown for limited log queries.
	// The result is a SUPERSET: the fetcher returns the Limit rows beyond the boundary timestamp
	// plus any rows that tie at that boundary, so a caller applying its own exact ordering+limit
	// never loses a boundary row (the fetch contract already permits a superset). It composes with
	// Matchers/Conditions/SecondPass — filtering happens first, the limit selects over survivors.
	// 0 ⇒ unlimited (every matching record). Honored by the record engine (logs/traces/profiles);
	// the metric engine ignores it (PromQL needs every sample).
	Limit int
	// Reverse selects the Limit direction: true keeps the newest records (largest timestamps, the
	// usual log-query default); false keeps the oldest. Ignored when Limit == 0.
	Reverse bool

	// Recycle opts the fetch into buffer pooling: the caller promises to call [Batch.Release] on
	// each returned batch once done with it, so the engine may hand out (and later reuse) pooled
	// result buffers. Default false — the engine allocates fresh buffers and the caller need not
	// release (so the non-recycling path takes no pool overhead at all). Misuse (reading a batch
	// after Release, or not releasing while Recycle is set) only forfeits the reuse, except that
	// reading after Release is undefined — never do it.
	Recycle bool

	// Scope names the logical query this read belongs to, so admission control can recognize a
	// caller that keeps several reads open at once instead of treating each as a stranger. Nil
	// (the default) is one read standing alone. See [Scope].
	Scope *Scope
}

Request selects series for a tenant within an inclusive time window, filtered by all matchers (their intersection).

The contract is **dual-shape** (DESIGN §7): Matchers resolve identity over the postings index (a metric series, a log stream), while Conditions filter the per-record columns *within* that identity (a log record's severity, body, attributes). Metrics use only Matchers; logs use both. The columnar fields are all zero-valued for a metrics request, so the metrics path is unaffected.

type Scope ¶ added in v0.37.0

type Scope struct {
	// contains filtered or unexported fields
}

Scope ties several reads to one logical query — a PromQL request and every selector, chunk and subquery under it — so admission control can tell "this query again" from "another query".

It is the read-side analog of Prometheus' `Storage.Querier`: the caller creates one per request, passes it on every Request it issues, and drops it when the request ends. An embedder that cannot thread it through every Request can install one on the request context instead (WithScope), which a Fetcher uses when Request.Scope is nil.

An embedder whose reads are one fetch each can leave it nil; the unscoped path is unchanged, and is no longer able to wedge: an unscoped read that hold-and-waits against its own reservation is released by its context, or force-admitted, rather than blocking forever.

It exists because the decode budget is reserved per fetch and released when that fetch ends, which is deadlock-free only while a caller keeps one fetch open at a time. A streaming consumer holds several iterators across an evaluation, and its second acquire is then hold-and-wait against its own reservation — the waiter is the only one who could release it. With a Scope the first read blocks for admission as usual and later reads under the same scope are charged without queueing.

A Scope is safe for concurrent use: an engine may evaluate two subtrees at once.

func NewScope() *Scope

func ScopeFrom(ctx context.Context) *Scope

func (s *Scope) Abort()

func (s *Scope) Charge(n int64)

func (s *Scope) Enter() (holding bool)

func (s *Scope) Release(n int64)

type SeriesLister ¶ added in v0.37.0

type SeriesLister interface {
	Series(ctx context.Context, r Request) ([]signal.Series, error)
}

SeriesLister is an optional Fetcher capability: it returns the identities of the series matching r.Matchers that hold samples in [r.Start, r.End], **ascending by series id** (the content hash of the identity) and deduplicated — the same order and dedup Iterator promises, so a lister's output composes as a child of another lister. SortSeries restores that order for a producer that gathers out of order.

The window filter may be **coarser than exact** (a producer may answer at part granularity, so a series whose samples sit just outside the window can be listed); the capability is for metadata, where Prometheus itself is block-granular. A caller needing the exact "has a sample in the window" test must fetch (or use Counter). It backs the label endpoints (label names/values, /series), whose answer is a list of strings but whose fetch-based implementation would decode every sample of every matching series — an unmatched selector then costs (cardinality x window). A Fetcher that does not implement it opts out (the caller falls back to Fetch).

The returned identities may alias source-owned memory; copy them to retain.

func SeriesListerOf(f Fetcher) SeriesLister

type SliceIterator ¶

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator is an Iterator over a fixed slice of batches — for simple fetchers and tests.

func NewSliceIterator(batches []*Batch) *SliceIterator

func (it *SliceIterator) Close() error

func (it *SliceIterator) Next(context.Context) (*Batch, error)

type Unwraper ¶ added in v0.17.0

type Unwraper interface {
	Unwrap() Fetcher
}

Unwraper is implemented by Fetcher decorators that wrap a single inner Fetcher (logging, caching, scoping, splitting). Multi-child fan-outs (merge, remote) are NOT Unwrapers — their count semantics are not a simple delegation, so CounterOf opts them out of the pushdown.