storagebackend

package
v0.49.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2026 License: Apache-2.0 Imports: 54 Imported by: 0

Documentation

Overview

Package storagebackend adapts the embeddable github.com/oteldb/storage engine to oteldb's query and ingestion interfaces, so any signal can be served from the native Go storage engine instead of ClickHouse.

All four signals are wired over a single shared *storage.Storage instance. Backend implements the metrics seam directly (Prometheus storage.Queryable + ExemplarQueryable, the PromQL engine's MetricsScanners, metricstorage.MetadataQuerier) and the ingestion sinks for every signal (ConsumeMetrics/ConsumeTraces/ConsumeLogs/ConsumeProfiles). Because the logs and profiles read interfaces declare colliding method names, each non-metric signal's query interface is implemented by a small wrapper obtained via Backend.Logs, Backend.Traces, and Backend.Profiles (see signals.go).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Backend 

type Backend struct {
	// contains filtered or unexported fields
}

Backend adapts a *storage.Storage to oteldb's metric query and ingestion interfaces. The zero value is not usable; construct with New.

func New 

func New(store *storage.Storage, opts ...Option) *Backend

New returns a Backend over store. The ingest side has no tenant callback, so every batch routes to the "default" tenant; the empty tenant id here normalizes to "default" on the read side, keeping reads and writes on the same tenant (which also makes cluster reads owner-aware).

func (*Backend) ConsumeLogs 

func (b *Backend) ConsumeLogs(ctx context.Context, ld plog.Logs) error

ConsumeLogs ingests an OTLP logs batch into the storage engine. It is the logs ingestion sink used when the storage backend serves logs.

func (*Backend) ConsumeMetrics 

func (b *Backend) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error

ConsumeMetrics ingests an OTLP metrics batch into the storage engine. It is the metrics ingestion sink used by the oteldb collector exporter when the storage backend is selected. Histogram, exponential-histogram, summary, and value-less points are not representable in the storage engine yet and are silently dropped by the conversion.

func (*Backend) ConsumeProfiles 

func (b *Backend) ConsumeProfiles(ctx context.Context, pd pprofile.Profiles) error

ConsumeProfiles ingests an OTLP profiles batch into the storage engine. It is the profiles ingestion sink used when the storage backend serves profiles.

func (*Backend) ConsumeTraces 

func (b *Backend) ConsumeTraces(ctx context.Context, td ptrace.Traces) error

ConsumeTraces ingests an OTLP traces batch into the storage engine. It is the traces ingestion sink used when the storage backend serves traces.

func (*Backend) EfficiencyStats added in v0.49.0 

func (b *Backend) EfficiencyStats(ctx context.Context) ([]storage.TenantEfficiency, error)

EfficiencyStats returns the per-tenant, per-signal capacity/efficiency breakdown (stored bytes, bytes per point, compression ratios). Unlike Inspect it performs backend I/O (per-part object sizes) — poll it at dashboard cadence, not per request.

func (*Backend) ExemplarQuerier 

func (b *Backend) ExemplarQuerier(context.Context) (promstorage.ExemplarQuerier, error)

ExemplarQuerier implements storage.ExemplarQueryable. The storage engine does not store exemplars yet, so this returns an empty querier.

func (*Backend) Inspect added in v0.49.0 

func (b *Backend) Inspect() storage.StoreStats

Inspect returns an in-memory snapshot of engine statistics (tenants, per-signal series/parts/head and WAL state, caches, and cluster membership when clustered). It performs no backend I/O and is safe to poll at a seconds cadence; it is the admin panel's primary storage view.

func (*Backend) Logs 

func (b *Backend) Logs() *LogQuerier

Logs returns the logs querier over the backend's storage engine.

func (*Backend) MaintainNow added in v0.49.0 

func (b *Backend) MaintainNow(ctx context.Context) error

MaintainNow runs one full maintenance cycle immediately (flush + merge + retention across every owned tenant and signal), i.e. the background maintenance loop's body on demand.

func (*Backend) MetricMetadata 

func (b *Backend) MetricMetadata(context.Context, metricstorage.MetadataParams) (metricstorage.Metadata, error)

MetricMetadata implements metricstorage.MetadataQuerier. The storage engine does not expose metric metadata yet, so this returns an empty set.

func (*Backend) MetricsScanners 

func (b *Backend) MetricsScanners() (enginestorage.Scanners, error)

MetricsScanners implements the oteldb PromQL engine's scanner seam.

func (*Backend) Profiles 

func (b *Backend) Profiles() *ProfileQuerier

Profiles returns the profiles querier over the backend's storage engine.

func (*Backend) Querier 

func (b *Backend) Querier(mint, maxt int64) (promstorage.Querier, error)

Querier implements storage.Queryable.

func (*Backend) Traces 

func (b *Backend) Traces() *TraceQuerier

Traces returns the traces querier over the backend's storage engine.

type LogQLOptimizer 

type LogQLOptimizer struct{}

LogQLOptimizer offloads LogQL line filters to the storage fetch layer for the embedded backend.

The leading positive substring filters (`|= "x"`) of a pipeline are pushed into the log fetch as body conditions, so the storage drops non-matching records before they are materialized into entries with label sets — the expensive part of the scan. The conditions carry an exact contains Match, so the result is identical to evaluating the filter in the engine (the engine still applies the full pipeline on the surviving entries, so the offload only ever skips work).

The condition also carries a bloom token hint (bloom.SafeTokens) so the storage can prune whole parts before scanning. The body bloom is word-tokenized, so the hint keeps only the interior whole tokens of the filter literal and drops its edge tokens, which a sub-word substring match may glue into a larger token (e.g. `|= "GET"` matching "xGETy"). A single-word literal therefore yields no hint (no pruning, but no false negatives), preserving the fetch contract's no-false-negatives guarantee while still pruning on multi-word literals.

Add it to a LogQL engine's optimizers when the querier is a storage LogQuerier; it is a no-op for any other node type.

func (*LogQLOptimizer) Name 

func (*LogQLOptimizer) Name() string

Name implements logqlengine.Optimizer.

func (*LogQLOptimizer) Optimize 

func (o *LogQLOptimizer) Optimize(_ context.Context, q logqlengine.Query) (logqlengine.Query, error)

Optimize implements logqlengine.Optimizer.

type LogQuerier 

type LogQuerier struct {
	// contains filtered or unexported fields
}

LogQuerier adapts the storage engine to oteldb's logs query interfaces (logstorage.Querier and logqlengine.Querier).

func (*LogQuerier) Capabilities 

func (q *LogQuerier) Capabilities() (caps logqlengine.QuerierCapabilities)

Capabilities implements logqlengine.Querier.

Every stream selector matcher shape is applied by this backend: [matchSelector] re-checks each materialized record against the whole selector with LogQL semantics (absent label reads as ""), and selector regexps are compiled anchored, matching the engine's own label matcher exactly. So the label ops are advertised and the engine builds no redundant prefilter — which also lets a bare selector query push its limit into the fetch (see [logStreamNode.EvalPipeline]).

No line op is advertised: line filters are not applied by this backend. LogQLOptimizer may offload some of them as fetch conditions, but those only ever skip work — the engine still evaluates the whole pipeline on the surviving entries.

func (*LogQuerier) DetectedFields 

func (q *LogQuerier) DetectedFields(ctx context.Context, opts logstorage.LabelsOptions) ([]logstorage.DetectedField, error)

DetectedFields implements logstorage.Querier. The storage backend does not parse record fields, so it reports the stream labels as string fields with their value cardinality.

func (*LogQuerier) DetectedLabels 

func (q *LogQuerier) DetectedLabels(ctx context.Context, opts logstorage.LabelsOptions) ([]logstorage.DetectedLabel, error)

DetectedLabels implements logstorage.Querier. It returns the cardinality of each stream label.

func (*LogQuerier) LabelNames 

func (q *LogQuerier) LabelNames(ctx context.Context, opts logstorage.LabelsOptions) ([]string, error)

LabelNames implements logstorage.Querier. It returns the distinct label names of the streams matching the options' selector.

func (*LogQuerier) LabelValues 

func (q *LogQuerier) LabelValues(ctx context.Context, labelName string, opts logstorage.LabelsOptions) (iterators.Iterator[logstorage.Label], error)

LabelValues implements logstorage.Querier. It returns the distinct values of labelName across the streams matching the options' selector.

func (*LogQuerier) Query 

func (q *LogQuerier) Query(_ context.Context, selector []logql.LabelMatcher) (logqlengine.PipelineNode, error)

Query implements logqlengine.Querier. It returns a node that streams the entries of the streams matching selector; the engine wraps it to evaluate the rest of the pipeline.

func (*LogQuerier) Series 

func (q *LogQuerier) Series(ctx context.Context, opts logstorage.SeriesOptions) (logstorage.Series, error)

Series implements logstorage.Querier. It returns the label sets of the streams matching any of the option selectors.

type Option 

type Option func(*Backend)

Option configures a Backend.

func WithLogParallelism 

func WithLogParallelism(n int) Option

WithLogParallelism enables concurrent materialization of LogQL query results across up to n workers. The fetched record set is split into contiguous chunks built in parallel and merged in order, so the result is identical to the sequential path regardless of scheduling. Opt-in: n <= 1 (the default) keeps the sequential path. Effective only above an internal record-count threshold.

func WithOverTimePushdown added in v0.42.0 

func WithOverTimePushdown(enabled bool) Option

WithOverTimePushdown toggles the instant *_over_time aggregate pushdown. It is on by default (the sidecar path is faster and correct); passing false restores the raw matrix-selector path (useful as a fallback or for differential testing).

func WithTraceQLPushdown added in v0.49.0 

func WithTraceQLPushdown(enabled bool) Option

WithTraceQLPushdown toggles the TraceQL span-matcher pushdown. It is on by default (the filtered candidate-trace scan is faster and returns the same traces); passing false restores the plain full window scan, for differential testing or as a fallback.

type ProfileQuerier 

type ProfileQuerier struct {
	// contains filtered or unexported fields
}

ProfileQuerier adapts the storage engine to oteldb's profiles query interface (profilestorage.Querier).

func (*ProfileQuerier) LabelNames 

func (q *ProfileQuerier) LabelNames(ctx context.Context, opts profilestorage.LabelNamesOptions) ([]string, error)

LabelNames implements profilestorage.Querier. It returns the distinct user-label names of the streams matching the selector, excluding the reserved profile-type labels.

func (*ProfileQuerier) LabelValues 

func (q *ProfileQuerier) LabelValues(ctx context.Context, label string, opts profilestorage.LabelValuesOptions) ([]string, error)

LabelValues implements profilestorage.Querier. It returns the distinct values the given label takes across the streams matching the selector.

func (*ProfileQuerier) ProfileTypes 

func (q *ProfileQuerier) ProfileTypes(ctx context.Context, opts profilestorage.ProfileTypesOptions) ([]profileql.ProfileType, error)

ProfileTypes implements profilestorage.Querier. It enumerates the distinct profile types of the tenant's streams in the window, reading the type out of each series' reserved labels.

func (*ProfileQuerier) SelectMergeProfile 

func (q *ProfileQuerier) SelectMergeProfile(ctx context.Context, params profilestorage.SelectProfileParams) (*profilestorage.FlameTree, error)

SelectMergeProfile implements profilestorage.Querier. It fetches every matching sample, resolves each sample's content-addressed stack to function frames, and merges them into a single flamegraph tree.

type TraceQuerier 

type TraceQuerier struct {
	// contains filtered or unexported fields
}

TraceQuerier adapts the storage engine to oteldb's traces query interfaces (tracestorage.Querier and traceqlengine.Querier).

func (*TraceQuerier) SearchTags 

func (q *TraceQuerier) SearchTags(ctx context.Context, tags map[string]string, opts tracestorage.SearchTagsOptions) (iterators.Iterator[tracestorage.Span], error)

SearchTags implements tracestorage.Querier. It returns the spans whose attributes match every requested tag and whose duration is within the optional bounds.

func (*TraceQuerier) SelectSpansets 

func (q *TraceQuerier) SelectSpansets(ctx context.Context, params traceqlengine.SelectSpansetsParams) (iterators.Iterator[traceqlengine.Trace], error)

SelectSpansets implements traceqlengine.Querier. It returns the traces whose spans fall in the window, grouped by trace id; the TraceQL engine evaluates the spanset matchers on the result.

The query's span matchers are first lowered to storage filters ([buildTracePushdown]) and run as a trace_id-only scan, so only the candidate traces are materialized with their attributes, events and links. The candidate set is a superset of the result and whole traces are returned, so the engine sees exactly what a full window scan would give it — structural operators and the spanset-level intrinsics still work. Nothing pushable (a bare `{}`, a `traceDuration` bound) falls back to the full window scan.

Resolving the candidates is itself a scan (of the filter columns and trace_id, not of the attribute/event/link blobs), so a predicate that matches nearly every trace pays for it without pruning anything: on the golden corpus a selective query is ~4x faster and a match-everything one ~18% slower than the plain scan.

params.Limit is deliberately not applied here: it counts *matching* traces, and the candidate set is only a superset of them. Truncating it in scan order would cap the result at "however many of the first N candidates happen to match", which for a selective query is usually fewer than N and often zero. The engine applies the limit once the matchers have run.

func (*TraceQuerier) TagNames 

func (q *TraceQuerier) TagNames(ctx context.Context, opts tracestorage.TagNamesOptions) ([]tracestorage.TagName, error)

TagNames implements tracestorage.Querier. It enumerates the distinct attribute names seen on the spans in the window, restricted to the requested scope.

func (*TraceQuerier) TagValues 

func (q *TraceQuerier) TagValues(ctx context.Context, attr traceql.Attribute, opts tracestorage.TagValuesOptions) (iterators.Iterator[tracestorage.Tag], error)

TagValues implements tracestorage.Querier. It enumerates the distinct values the attribute takes across the spans in the window.

func (*TraceQuerier) TraceByID 

func (q *TraceQuerier) TraceByID(ctx context.Context, id otelstorage.TraceID, _ tracestorage.TraceByIDOptions) (iterators.Iterator[tracestorage.Span], error)

TraceByID implements tracestorage.Querier. It fetches every span of one trace by id.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.