chstore

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: AGPL-3.0 Imports: 45 Imported by: 0

Documentation

Overview

Package chstore implements ClickHouse-backed storage for Loki's log data. It provides the Querier for log and sample queries, the Writer for ingest, and the schema migration and bucketing utilities.

Index

Constants

View Source
const (
	// MaxPushPartitions bounds the UTC day partitions one push may touch; wider pushes are rejected whole (split them by time range).
	// It stays well below ClickHouse's max_partitions_per_insert_block (100).
	MaxPushPartitions = 60
	// AsyncMaxPartitions is the widest push that goes through the async-insert
	// buffer: one buffer flush merges the pushes of all tenants, so a wide push
	// there could take the flush past ClickHouse's partition limit and fail
	// every push in it. Wider pushes are inserted synchronously, on their own.
	// Live pushes touch 1–2 partitions; late data under Loki's default
	// reject_old_samples_max_age (1w) at most 8.
	AsyncMaxPartitions = 8
)
View Source
const DeleteBatch = 2_000

DeleteBatch is the most keys one DELETE statement carries: a worst-case key renders to ~64 bytes of SQL, and the statement must stay under the server's max_query_size (256 KiB by default).

View Source
const StreamBatch = 5_000

StreamBatch is the most stream IDs one statement carries; 5 000 IDs render to well under the server's max_query_size (256 KiB).

Variables

View Source
var ErrSchemaV1 = errors.New("schema version 1: stop all runakefli instances and run runakefli -schema.migrate")

ErrSchemaV1 is returned (wrapped) by Migrate for a database that still has schema version 1 (a retention_days column); run MigrateV2 on it first.

View Source
var ErrSchemaV2 = errors.New("schema version 2: stop all runakefli instances and run runakefli -schema.migrate")

ErrSchemaV2 is returned (wrapped) by Migrate for a database that still has schema version 2 (a logs table without line_ascii_escape); run MigrateSchema on it first.

View Source
var ErrTooManyPartitions = errors.New("too many partitions")

ErrTooManyPartitions is returned (wrapped) by Write for a push touching more than MaxPushPartitions partitions; nothing is stored.

Functions

func ApplyDeletedMask

func ApplyDeletedMask(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) error

ApplyDeletedMask (of one of the retentionTables or logs) physically removes rows that lightweight DELETEs only masked. Without it they stay on disk until a merge rewrites their part, which for a day partition that no longer receives inserts may be never; Loki's compactor rewrites the chunks. Only parts carrying a mask are rewritten. mutations_sync=2 waits for every replica.

func DayPartitions

func DayPartitions(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) ([]time.Time, error)

DayPartitions returns the UTC days that have active parts in the storage table (on any replica of a cluster).

func DaysWithRows

func DaysWithRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, upTo time.Time) ([]time.Time, error)

DaysWithRows returns the UTC days <= upTo on which any of the streams still has rows.

func DeleteDays

func DeleteDays(ctx context.Context, conn driver.Conn, db string, topo Topology, table, tenant string, ids []uint64, days []time.Time) error

DeleteDays lightweight-deletes the given UTC days of the streams from the storage table (ON CLUSTER on a cluster), StreamBatch stream IDs per statement.

func DeleteRange

func DeleteRange(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, ids []uint64, fromNs, throughNs int64) error

DeleteRange removes every row of the given streams with fromNs <= ts <= throughNs.

func DeleteRows

func DeleteRows(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, keys []RowKey) error

DeleteRows removes the given rows with lightweight DELETEs on the storage table (ON CLUSTER on a cluster), DeleteBatch keys per statement. The tuple list is rendered from integers only.

func DropDatabase

func DropDatabase(ctx context.Context, conn driver.Conn, db string, topo Topology) error

DropDatabase removes db on every node of the topology. Used by tests.

func DropDay

func DropDay(ctx context.Context, conn driver.Conn, db string, topo Topology, table string, day time.Time) error

DropDay drops the partition of the given UTC day from the storage table. The partition ID is rendered from the time, never from a string.

func Migrate

func Migrate(ctx context.Context, conn driver.Conn, db string, topo Topology) error

Migrate creates all runakefli tables in db. Safe to run repeatedly, and from several runakefli instances at once.

func MigrateSchema

func MigrateSchema(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error

MigrateSchema brings a database of any older schema version to the current one and creates anything missing: version 1 tables are copied (MigrateV2, with the current DDL), version 2 tables are altered (MigrateV3). All other runakefli instances must be stopped.

func MigrateV2

func MigrateV2(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error

MigrateV2 migrates a schema version 1 database to version 2 (no retention_days); the copied tables are created with the current DDL. Call MigrateSchema to finish the job (it also creates anything missing). It copies each table into <table>_migrate, exchanges the two and drops the old data. It is resumable after a crash and a no-op on a version 2 database. All other runakefli instances must be stopped.

func MigrateV3

func MigrateV3(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error

MigrateV3 migrates a schema version 2 database to version 3: it adds the structured-metadata text indexes and the line_ascii_escape column to logs. Nothing is copied or re-indexed; only parts written or merged afterwards carry the new indexes. A no-op on a version 3 database. All other runakefli instances must be stopped.

func RegexExact

func RegexExact(re string) bool

RegexExact reports whether Go's regexp and ClickHouse's match() agree on every possible input for the pattern re, so it can be pushed down as an *exact* filter (match(line, re)) rather than merely a necessary one.

The gate is conservative: it walks the unsimplified parse tree and allows only ASCII-safe constructs. ASCII-only atoms never match a byte ≥ 0x80 in either engine, whether the line is valid UTF-8 or not, so UTF-8 decoding differences cannot change the outcome.

func RenderSchema

func RenderSchema(db string, topo Topology) ([]string, error)

RenderSchema returns the DDL statements for db under topo, in execution order.

func RenderTemplate

func RenderTemplate(src, db string, topo Topology) ([]string, error)

RenderTemplate renders a schema template (the syntax of schema.sql) for db under topo. The migration tests use it to create databases from older schema versions.

func RetentionTables

func RetentionTables() []string

RetentionTables returns the tables that hold per-day data subject to retention (a function because package-level mutable state is not allowed).

func ScanRows

func ScanRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, fromNs, throughNs int64, fn func(Row) error) error

ScanRows streams every raw row (no dedup: each copy must be deleted) of the given streams with fromNs <= ts <= throughNs (inclusive bounds).

func StreamIdentity

func StreamIdentity(lbls string) (labels.Labels, uint64, error)

StreamIdentity canonicalises a stream label string and returns its Loki-compatible hash.

func StreamsUpTo

func StreamsUpTo(ctx context.Context, conn driver.Conn, day time.Time, fn func(StreamRef) error) error

StreamsUpTo calls fn for every distinct (tenant, stream) registered on a UTC day <= day.

func WithQueryStats

func WithQueryStats(ctx context.Context) context.Context

WithQueryStats returns ctx with a ClickHouse progress callback that adds the rows and uncompressed bytes the server read to Loki's stats context in ctx (summary.totalLinesProcessed / totalBytesProcessed). ClickHouse reports progress as increments; existing query options in ctx are kept.

Types

type BucketOpts

type BucketOpts struct {
	Bytes  bool     // sum(length(line)); false reports Bytes = 0 and does not read line
	AllMD  bool     // group by the whole structured_metadata map
	MDKeys []string // when !AllMD: group by only these metadata keys (empty = none)
}

BucketOpts selects what BucketCounts reads and groups by.

type BucketRow

type BucketRow struct {
	StreamID uint64
	MD       map[string]string
	Bucket   int64
	Count    uint64
	Bytes    uint64
}

BucketRow holds the deduplicated row count and byte size for one stream in one bucket (time slice).

func BucketCounts

func BucketCounts(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, origin, through, g int64, filter Fragment, opt BucketOpts) ([]BucketRow, error)

BucketCounts groups the deduplicated (FINAL) rows of the given streams in (origin, through] into sub-buckets of width g ns. bucket = intDiv(toUnixTimestamp64Nano(ts) - origin + g - 1, g), so bucket j holds (origin + (j-1)*g, origin + j*g].

do_not_merge_across_partitions_select_final is safe here: duplicates share ts, hence the day partition.

type Fragment

type Fragment struct {
	SQL  string
	Args []any
}

Fragment is a SQL query fragment with its arguments.

func ExactFilterSQL

func ExactFilterSQL(expr syntax.LogSelectorExpr) (f Fragment, labelNames []string, ok bool)

ExactFilterSQL returns SQL accepting exactly the entries the pipeline's filters accept, or ok=false if the pipeline has any stage other than line filters that are literal (|=, !=, with or-chains) or regex that Loki simplifies to literals or that passes RegexExact (|~, !~), and label filters that compare a label to a non-empty string with = (alone or combined with and/or), which map to structured_metadata. labelNames lists the names of those label filters (each once, in source order; for name_extracted also name): the SQL is exact only if no stream carries such a label, because a stream label could satisfy the filter too, and a clash renames the metadata key to name_extracted.

func LeadingLineFilterSQL

func LeadingLineFilterSQL(expr syntax.LogSelectorExpr) Fragment

LeadingLineFilterSQL returns a necessary SQL condition for the literal line filters that precede every non-filter stage. Loki's pipeline re-applies all filters exactly.

func MetadataFilterSQL

func MetadataFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment

MetadataFilterSQL maps the label filters before the first parser stage of expr to conditions over structured_metadata, which the sm_keys and sm_vals text indexes answer. The filters come from Loki's bloom extractor (bloomv1.ExtractTestableLabelMatchers); they are ANDed.

The fragment is a necessary condition: the pipeline re-applies the filters in Go. A filter on a name that a resolved stream carries as a stream label is not pushed (the label may satisfy the filter), nor is a name starting with "__", nor an unsupported filter (a regex that is not an alternation of literals, !=, comparisons).

func ParsedFilterSQL

func ParsedFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment

ParsedFilterSQL maps label equality filters after a bare `| json` or `| logfmt` to conditions on the line, which the line_text index answers. The value of such a label comes from the line, so the verbatim runs of the value must occur in it. Three cases need no run: the label can also come from structured metadata of that name (mapContainsKey), and a line that hides an ASCII letter or digit behind a \u escape (line_ascii_escape) may not contain the run.

The fragment is a necessary condition: the pipeline re-applies the filters in Go. It walks the stages in order; line filters and label filters (which change no label) are passed over, any stage other than a parser ends the walk. Label filters before the first parser are MetadataFilterSQL's. A filter on a name that a resolved stream carries is not pushed, nor is a name starting with "__", a non-ASCII value, or a value with no run of at least four verbatim characters.

func StreamMatcherSQL

func StreamMatcherSQL(ms []*labels.Matcher) (Fragment, []*labels.Matcher)

StreamMatcherSQL turns equality matchers, and regex matchers that pass RegexExact, into SQL over streams.labels. Regex matchers are also returned: Go re-checks them.

func (Fragment) Empty

func (f Fragment) Empty() bool

Empty reports whether the fragment has no SQL.

type HourStat

type HourStat struct {
	StreamID, Entries, Bytes uint64
	MinTS, MaxTS             int64
}

HourStat is one (stream, hour) bucket of stream_stats_hourly; timestamps in ns.

func HourlyStats

func HourlyStats(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, from, through time.Time) ([]HourStat, error)

HourlyStats returns the hourly buckets of the given streams that can overlap [from, through]. Unmerged SimpleAggregateFunction rows are merged by the GROUP BY.

type Querier

type Querier struct {
	Conn driver.Conn
	// Deletes, when set, is queried for pending delete requests and the
	// matching lines are hidden from log and sample results, as Loki's querier
	// does. Nil = no deletes.
	Deletes querierdeletion.DeleteGetter
	// WindowRows is the target number of rows per read window; 0 means
	// defaultWindowRows.
	WindowRows uint64
	// Workers is the number of sample-extraction goroutines per metric query;
	// 0 means runtime.GOMAXPROCS(0) read at query time. The effective count is
	// capped at the number of matching streams.
	Workers int
	// StreamCache, when set, lets SelectLogs start reading lines while the
	// stream lookup of a repeated selector runs (speculateLogs). Nil disables
	// speculation.
	StreamCache *StreamSetCache
}

Querier executes LogQL queries against ClickHouse. It implements logql.Querier for both log and sample queries.

func (*Querier) DetectedLabels

DetectedLabels serves /loki/api/v1/detected_labels from the streams table: the store path of Loki's SingleTenantQuerier.DetectedLabels.

func (*Querier) IndexStats

IndexStats serves /loki/api/v1/index/stats from stream_stats_hourly.

func (*Querier) Label

Label returns the label names or values matching the request.

func (*Querier) SelectLogs

SelectLogs executes a log query and returns an iterator over the matching entries.

func (*Querier) SelectSamples

SelectSamples executes a sample (metrics) query and returns an iterator over the matching samples.

func (*Querier) Series

Series returns the time series matching the request.

func (*Querier) Volume

Volume serves one bucket of /loki/api/v1/index/volume(_range).

type Row

type Row struct {
	RowKey
	Line string
	MD   labels.Labels
}

Row is one raw stored log line plus its identifying key and metadata.

type RowKey

type RowKey struct {
	StreamID  uint64
	TS        int64
	EntryHash uint64
}

RowKey identifies one stored row exactly (the logs sorting key).

type Stream

type Stream struct {
	ID     uint64
	Labels labels.Labels
}

Stream is a log stream with its labels.

func ResolveStreams

func ResolveStreams(ctx context.Context, conn driver.Conn, tenant string, ms []*labels.Matcher, from, through time.Time) ([]Stream, error)

ResolveStreams returns the streams of tenant matching ms that have data on any UTC day in [from, through].

type StreamRef

type StreamRef struct {
	Tenant string
	ID     uint64
	Labels labels.Labels
}

StreamRef is a stream of a tenant with its labels.

type StreamSetCache

type StreamSetCache struct {
	// contains filtered or unexported fields
}

StreamSetCache remembers the streams a selector resolved to, so a repeated log query can start reading before its stream lookup returns. It is safe for concurrent use (the underlying LRU is thread-safe). A nil *StreamSetCache is a disabled cache.

func NewStreamSetCache

func NewStreamSetCache() *StreamSetCache

NewStreamSetCache returns an empty cache.

type Topology

type Topology struct {
	Cluster     string // remote_servers cluster name; empty = single server
	Replicated  bool   // use Replicated*MergeTree engines (needs Keeper)
	ReplicaPath string // optional Keeper path; empty = server's default_replica_path
	ReplicaName string // optional replica name; empty = server's default_replica_name
}

Topology describes how runakefli tables are laid out on ClickHouse. The zero value is a single server with plain MergeTree tables.

func (Topology) Engine

func (t Topology) Engine(base string, params ...string) string

Engine returns the storage engine clause for a MergeTree-family engine with the given engine parameters (e.g. ReplacingMergeTree's version column).

func (Topology) Local

func (t Topology) Local(name string) string

Local returns the name of the table that physically stores rows. On a cluster that is <name>_local; <name> itself is the Distributed table.

func (Topology) OnCluster

func (t Topology) OnCluster() string

OnCluster returns the ON CLUSTER clause for DDL and mutations, or "".

func (Topology) Settings

func (t Topology) Settings() clickhouse.Settings

Settings returns the session settings every runakefli connection needs for this topology: pushes are acknowledged only once all shards and an insert quorum of replicas hold the rows, so a query right after a push sees them.

type Writer

type Writer struct {
	Conn driver.Conn
	// contains filtered or unexported fields
}

Writer inserts pushes into ClickHouse. Batching follows ClickHouse's advice for many small concurrent inserts: every push of at most AsyncMaxPartitions partitions is one async insert per table, which the server buffers and flushes as a single part (async_insert_* settings tune the flush). Wider pushes are synchronous inserts.

func (*Writer) Write

func (w *Writer) Write(ctx context.Context, tenant string, streams []logproto.Stream) error

Write inserts entries into logs and registers each stream for every UTC day it touches.

Directories

Path Synopsis
Package chtest provides throwaway ClickHouse databases for tests.
Package chtest provides throwaway ClickHouse databases for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL