Documentation
¶
Overview ¶
Package chstore implements ClickHouse-backed storage for Loki's log data. It provides the Querier for log and sample queries, the Writer for ingest, and the schema migration and bucketing utilities.
Index ¶
- Constants
- Variables
- func ApplyDeletedMask(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) error
- func DayPartitions(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) ([]time.Time, error)
- func DaysWithRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, ...) ([]time.Time, error)
- func DeleteDays(ctx context.Context, conn driver.Conn, db string, topo Topology, ...) error
- func DeleteRange(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, ...) error
- func DeleteRows(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, ...) error
- func DropDatabase(ctx context.Context, conn driver.Conn, db string, topo Topology) error
- func DropDay(ctx context.Context, conn driver.Conn, db string, topo Topology, table string, ...) error
- func Migrate(ctx context.Context, conn driver.Conn, db string, topo Topology) error
- func MigrateSchema(ctx context.Context, conn driver.Conn, db string, topo Topology, ...) error
- func MigrateV2(ctx context.Context, conn driver.Conn, db string, topo Topology, ...) error
- func MigrateV3(ctx context.Context, conn driver.Conn, db string, topo Topology, ...) error
- func RegexExact(re string) bool
- func RenderSchema(db string, topo Topology) ([]string, error)
- func RenderTemplate(src, db string, topo Topology) ([]string, error)
- func RetentionTables() []string
- func ScanRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, ...) error
- func StreamIdentity(lbls string) (labels.Labels, uint64, error)
- func StreamsUpTo(ctx context.Context, conn driver.Conn, day time.Time, fn func(StreamRef) error) error
- func WithQueryStats(ctx context.Context) context.Context
- type BucketOpts
- type BucketRow
- type Fragment
- func ExactFilterSQL(expr syntax.LogSelectorExpr) (f Fragment, labelNames []string, ok bool)
- func LeadingLineFilterSQL(expr syntax.LogSelectorExpr) Fragment
- func MetadataFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment
- func ParsedFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment
- func StreamMatcherSQL(ms []*labels.Matcher) (Fragment, []*labels.Matcher)
- type HourStat
- type Querier
- func (q *Querier) DetectedLabels(ctx context.Context, req *logproto.DetectedLabelsRequest) (*logproto.DetectedLabelsResponse, error)
- func (q *Querier) IndexStats(ctx context.Context, req *logproto.IndexStatsRequest) (*logproto.IndexStatsResponse, error)
- func (q *Querier) Label(ctx context.Context, req *logproto.LabelRequest) (*logproto.LabelResponse, error)
- func (q *Querier) SelectLogs(ctx context.Context, p logql.SelectLogParams) (iter.EntryIterator, error)
- func (q *Querier) SelectSamples(ctx context.Context, p logql.SelectSampleParams) (iter.SampleIterator, error)
- func (q *Querier) Series(ctx context.Context, req *logproto.SeriesRequest) (*logproto.SeriesResponse, error)
- func (q *Querier) Volume(ctx context.Context, req *logproto.VolumeRequest) (*logproto.VolumeResponse, error)
- type Row
- type RowKey
- type Stream
- type StreamRef
- type StreamSetCache
- type Topology
- type Writer
Constants ¶
const ( // MaxPushPartitions bounds the UTC day partitions one push may touch; wider pushes are rejected whole (split them by time range). // It stays well below ClickHouse's max_partitions_per_insert_block (100). MaxPushPartitions = 60 // AsyncMaxPartitions is the widest push that goes through the async-insert // buffer: one buffer flush merges the pushes of all tenants, so a wide push // there could take the flush past ClickHouse's partition limit and fail // every push in it. Wider pushes are inserted synchronously, on their own. // Live pushes touch 1–2 partitions; late data under Loki's default // reject_old_samples_max_age (1w) at most 8. AsyncMaxPartitions = 8 )
const DeleteBatch = 2_000
DeleteBatch is the most keys one DELETE statement carries: a worst-case key renders to ~64 bytes of SQL, and the statement must stay under the server's max_query_size (256 KiB by default).
const StreamBatch = 5_000
StreamBatch is the most stream IDs one statement carries; 5 000 IDs render to well under the server's max_query_size (256 KiB).
Variables ¶
var ErrSchemaV1 = errors.New("schema version 1: stop all runakefli instances and run runakefli -schema.migrate")
ErrSchemaV1 is returned (wrapped) by Migrate for a database that still has schema version 1 (a retention_days column); run MigrateV2 on it first.
var ErrSchemaV2 = errors.New("schema version 2: stop all runakefli instances and run runakefli -schema.migrate")
ErrSchemaV2 is returned (wrapped) by Migrate for a database that still has schema version 2 (a logs table without line_ascii_escape); run MigrateSchema on it first.
var ErrTooManyPartitions = errors.New("too many partitions")
ErrTooManyPartitions is returned (wrapped) by Write for a push touching more than MaxPushPartitions partitions; nothing is stored.
Functions ¶
func ApplyDeletedMask ¶
func ApplyDeletedMask(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) error
ApplyDeletedMask (of one of the retentionTables or logs) physically removes rows that lightweight DELETEs only masked. Without it they stay on disk until a merge rewrites their part, which for a day partition that no longer receives inserts may be never; Loki's compactor rewrites the chunks. Only parts carrying a mask are rewritten. mutations_sync=2 waits for every replica.
func DayPartitions ¶
func DayPartitions(ctx context.Context, conn driver.Conn, db string, topo Topology, table string) ([]time.Time, error)
DayPartitions returns the UTC days that have active parts in the storage table (on any replica of a cluster).
func DaysWithRows ¶
func DaysWithRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, upTo time.Time) ([]time.Time, error)
DaysWithRows returns the UTC days <= upTo on which any of the streams still has rows.
func DeleteDays ¶
func DeleteDays(ctx context.Context, conn driver.Conn, db string, topo Topology, table, tenant string, ids []uint64, days []time.Time) error
DeleteDays lightweight-deletes the given UTC days of the streams from the storage table (ON CLUSTER on a cluster), StreamBatch stream IDs per statement.
func DeleteRange ¶
func DeleteRange(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, ids []uint64, fromNs, throughNs int64) error
DeleteRange removes every row of the given streams with fromNs <= ts <= throughNs.
func DeleteRows ¶
func DeleteRows(ctx context.Context, conn driver.Conn, db string, topo Topology, tenant string, keys []RowKey) error
DeleteRows removes the given rows with lightweight DELETEs on the storage table (ON CLUSTER on a cluster), DeleteBatch keys per statement. The tuple list is rendered from integers only.
func DropDatabase ¶
DropDatabase removes db on every node of the topology. Used by tests.
func DropDay ¶
func DropDay(ctx context.Context, conn driver.Conn, db string, topo Topology, table string, day time.Time) error
DropDay drops the partition of the given UTC day from the storage table. The partition ID is rendered from the time, never from a string.
func Migrate ¶
Migrate creates all runakefli tables in db. Safe to run repeatedly, and from several runakefli instances at once.
func MigrateSchema ¶
func MigrateSchema(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error
MigrateSchema brings a database of any older schema version to the current one and creates anything missing: version 1 tables are copied (MigrateV2, with the current DDL), version 2 tables are altered (MigrateV3). All other runakefli instances must be stopped.
func MigrateV2 ¶
func MigrateV2(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error
MigrateV2 migrates a schema version 1 database to version 2 (no retention_days); the copied tables are created with the current DDL. Call MigrateSchema to finish the job (it also creates anything missing). It copies each table into <table>_migrate, exchanges the two and drops the old data. It is resumable after a crash and a no-op on a version 2 database. All other runakefli instances must be stopped.
func MigrateV3 ¶
func MigrateV3(ctx context.Context, conn driver.Conn, db string, topo Topology, logger log.Logger) error
MigrateV3 migrates a schema version 2 database to version 3: it adds the structured-metadata text indexes and the line_ascii_escape column to logs. Nothing is copied or re-indexed; only parts written or merged afterwards carry the new indexes. A no-op on a version 3 database. All other runakefli instances must be stopped.
func RegexExact ¶
RegexExact reports whether Go's regexp and ClickHouse's match() agree on every possible input for the pattern re, so it can be pushed down as an *exact* filter (match(line, re)) rather than merely a necessary one.
The gate is conservative: it walks the unsimplified parse tree and allows only ASCII-safe constructs. ASCII-only atoms never match a byte ≥ 0x80 in either engine, whether the line is valid UTF-8 or not, so UTF-8 decoding differences cannot change the outcome.
func RenderSchema ¶
RenderSchema returns the DDL statements for db under topo, in execution order.
func RenderTemplate ¶
RenderTemplate renders a schema template (the syntax of schema.sql) for db under topo. The migration tests use it to create databases from older schema versions.
func RetentionTables ¶
func RetentionTables() []string
RetentionTables returns the tables that hold per-day data subject to retention (a function because package-level mutable state is not allowed).
func ScanRows ¶
func ScanRows(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, fromNs, throughNs int64, fn func(Row) error) error
ScanRows streams every raw row (no dedup: each copy must be deleted) of the given streams with fromNs <= ts <= throughNs (inclusive bounds).
func StreamIdentity ¶
StreamIdentity canonicalises a stream label string and returns its Loki-compatible hash.
func StreamsUpTo ¶
func StreamsUpTo(ctx context.Context, conn driver.Conn, day time.Time, fn func(StreamRef) error) error
StreamsUpTo calls fn for every distinct (tenant, stream) registered on a UTC day <= day.
func WithQueryStats ¶
WithQueryStats returns ctx with a ClickHouse progress callback that adds the rows and uncompressed bytes the server read to Loki's stats context in ctx (summary.totalLinesProcessed / totalBytesProcessed). ClickHouse reports progress as increments; existing query options in ctx are kept.
Types ¶
type BucketOpts ¶
type BucketOpts struct {
Bytes bool // sum(length(line)); false reports Bytes = 0 and does not read line
AllMD bool // group by the whole structured_metadata map
MDKeys []string // when !AllMD: group by only these metadata keys (empty = none)
}
BucketOpts selects what BucketCounts reads and groups by.
type BucketRow ¶
type BucketRow struct {
StreamID uint64
MD map[string]string
Bucket int64
Count uint64
Bytes uint64
}
BucketRow holds the deduplicated row count and byte size for one stream in one bucket (time slice).
func BucketCounts ¶
func BucketCounts(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, origin, through, g int64, filter Fragment, opt BucketOpts) ([]BucketRow, error)
BucketCounts groups the deduplicated (FINAL) rows of the given streams in (origin, through] into sub-buckets of width g ns. bucket = intDiv(toUnixTimestamp64Nano(ts) - origin + g - 1, g), so bucket j holds (origin + (j-1)*g, origin + j*g].
do_not_merge_across_partitions_select_final is safe here: duplicates share ts, hence the day partition.
type Fragment ¶
Fragment is a SQL query fragment with its arguments.
func ExactFilterSQL ¶
func ExactFilterSQL(expr syntax.LogSelectorExpr) (f Fragment, labelNames []string, ok bool)
ExactFilterSQL returns SQL accepting exactly the entries the pipeline's filters accept, or ok=false if the pipeline has any stage other than line filters that are literal (|=, !=, with or-chains) or regex that Loki simplifies to literals or that passes RegexExact (|~, !~), and label filters that compare a label to a non-empty string with = (alone or combined with and/or), which map to structured_metadata. labelNames lists the names of those label filters (each once, in source order; for name_extracted also name): the SQL is exact only if no stream carries such a label, because a stream label could satisfy the filter too, and a clash renames the metadata key to name_extracted.
func LeadingLineFilterSQL ¶
func LeadingLineFilterSQL(expr syntax.LogSelectorExpr) Fragment
LeadingLineFilterSQL returns a necessary SQL condition for the literal line filters that precede every non-filter stage. Loki's pipeline re-applies all filters exactly.
func MetadataFilterSQL ¶
func MetadataFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment
MetadataFilterSQL maps the label filters before the first parser stage of expr to conditions over structured_metadata, which the sm_keys and sm_vals text indexes answer. The filters come from Loki's bloom extractor (bloomv1.ExtractTestableLabelMatchers); they are ANDed.
The fragment is a necessary condition: the pipeline re-applies the filters in Go. A filter on a name that a resolved stream carries as a stream label is not pushed (the label may satisfy the filter), nor is a name starting with "__", nor an unsupported filter (a regex that is not an alternation of literals, !=, comparisons).
func ParsedFilterSQL ¶
func ParsedFilterSQL(expr syntax.LogSelectorExpr, streams []Stream) Fragment
ParsedFilterSQL maps label equality filters after a bare `| json` or `| logfmt` to conditions on the line, which the line_text index answers. The value of such a label comes from the line, so the verbatim runs of the value must occur in it. Three cases need no run: the label can also come from structured metadata of that name (mapContainsKey), and a line that hides an ASCII letter or digit behind a \u escape (line_ascii_escape) may not contain the run.
The fragment is a necessary condition: the pipeline re-applies the filters in Go. It walks the stages in order; line filters and label filters (which change no label) are passed over, any stage other than a parser ends the walk. Label filters before the first parser are MetadataFilterSQL's. A filter on a name that a resolved stream carries is not pushed, nor is a name starting with "__", a non-ASCII value, or a value with no run of at least four verbatim characters.
func StreamMatcherSQL ¶
StreamMatcherSQL turns equality matchers, and regex matchers that pass RegexExact, into SQL over streams.labels. Regex matchers are also returned: Go re-checks them.
type HourStat ¶
HourStat is one (stream, hour) bucket of stream_stats_hourly; timestamps in ns.
func HourlyStats ¶
func HourlyStats(ctx context.Context, conn driver.Conn, tenant string, ids []uint64, from, through time.Time) ([]HourStat, error)
HourlyStats returns the hourly buckets of the given streams that can overlap [from, through]. Unmerged SimpleAggregateFunction rows are merged by the GROUP BY.
type Querier ¶
type Querier struct {
Conn driver.Conn
// Deletes, when set, is queried for pending delete requests and the
// matching lines are hidden from log and sample results, as Loki's querier
// does. Nil = no deletes.
Deletes querierdeletion.DeleteGetter
// WindowRows is the target number of rows per read window; 0 means
// defaultWindowRows.
WindowRows uint64
// Workers is the number of sample-extraction goroutines per metric query;
// 0 means runtime.GOMAXPROCS(0) read at query time. The effective count is
// capped at the number of matching streams.
Workers int
// StreamCache, when set, lets SelectLogs start reading lines while the
// stream lookup of a repeated selector runs (speculateLogs). Nil disables
// speculation.
StreamCache *StreamSetCache
}
Querier executes LogQL queries against ClickHouse. It implements logql.Querier for both log and sample queries.
func (*Querier) DetectedLabels ¶
func (q *Querier) DetectedLabels(ctx context.Context, req *logproto.DetectedLabelsRequest) (*logproto.DetectedLabelsResponse, error)
DetectedLabels serves /loki/api/v1/detected_labels from the streams table: the store path of Loki's SingleTenantQuerier.DetectedLabels.
func (*Querier) IndexStats ¶
func (q *Querier) IndexStats(ctx context.Context, req *logproto.IndexStatsRequest) (*logproto.IndexStatsResponse, error)
IndexStats serves /loki/api/v1/index/stats from stream_stats_hourly.
func (*Querier) Label ¶
func (q *Querier) Label(ctx context.Context, req *logproto.LabelRequest) (*logproto.LabelResponse, error)
Label returns the label names or values matching the request.
func (*Querier) SelectLogs ¶
func (q *Querier) SelectLogs(ctx context.Context, p logql.SelectLogParams) (iter.EntryIterator, error)
SelectLogs executes a log query and returns an iterator over the matching entries.
func (*Querier) SelectSamples ¶
func (q *Querier) SelectSamples(ctx context.Context, p logql.SelectSampleParams) (iter.SampleIterator, error)
SelectSamples executes a sample (metrics) query and returns an iterator over the matching samples.
func (*Querier) Series ¶
func (q *Querier) Series(ctx context.Context, req *logproto.SeriesRequest) (*logproto.SeriesResponse, error)
Series returns the time series matching the request.
func (*Querier) Volume ¶
func (q *Querier) Volume(ctx context.Context, req *logproto.VolumeRequest) (*logproto.VolumeResponse, error)
Volume serves one bucket of /loki/api/v1/index/volume(_range).
type StreamSetCache ¶
type StreamSetCache struct {
// contains filtered or unexported fields
}
StreamSetCache remembers the streams a selector resolved to, so a repeated log query can start reading before its stream lookup returns. It is safe for concurrent use (the underlying LRU is thread-safe). A nil *StreamSetCache is a disabled cache.
func NewStreamSetCache ¶
func NewStreamSetCache() *StreamSetCache
NewStreamSetCache returns an empty cache.
type Topology ¶
type Topology struct {
Cluster string // remote_servers cluster name; empty = single server
Replicated bool // use Replicated*MergeTree engines (needs Keeper)
ReplicaPath string // optional Keeper path; empty = server's default_replica_path
ReplicaName string // optional replica name; empty = server's default_replica_name
}
Topology describes how runakefli tables are laid out on ClickHouse. The zero value is a single server with plain MergeTree tables.
func (Topology) Engine ¶
Engine returns the storage engine clause for a MergeTree-family engine with the given engine parameters (e.g. ReplacingMergeTree's version column).
func (Topology) Local ¶
Local returns the name of the table that physically stores rows. On a cluster that is <name>_local; <name> itself is the Distributed table.
func (Topology) Settings ¶
func (t Topology) Settings() clickhouse.Settings
Settings returns the session settings every runakefli connection needs for this topology: pushes are acknowledged only once all shards and an insert quorum of replicas hold the rows, so a query right after a push sees them.
type Writer ¶
Writer inserts pushes into ClickHouse. Batching follows ClickHouse's advice for many small concurrent inserts: every push of at most AsyncMaxPartitions partitions is one async insert per table, which the server buffers and flushes as a single part (async_insert_* settings tune the flush). Wider pushes are synchronous inserts.