recordengine

package
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 2, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package recordengine is the shared storage engine for **record-shaped** signals — logs and traces (later, profiles' sample table). A record-shaped signal is a stream (a Resource+Scope identity, indexed by the postings layer) of rows that each carry a primary timestamp plus a fixed set of typed columns. The engine is the structural twin of package engine (metrics) but generic over a Schema: it owns the in-memory head, columnar flush to immutable parts, the durable bucket-index + identity-index stateless read path, append-only merge with retention, per-part column blooms, lazy column decode, and the fetch.Fetcher contract — none of which is signal-specific. A signal package supplies the column Schema and projects its model into the engine's column vectors; the engine treats the columns opaquely.

The timestamp (the sort key) and the int128 stream id (the Resource+Scope hash) are implicit and not part of the Schema; the schema lists only the per-record columns.

Index

Constants

This section is empty.

Variables

View Source
var ErrCorruptKeys = errors.New("recordengine: corrupt record-keys footer")

ErrCorruptKeys is returned when a serialized record-keys footer fails to parse.

Functions

func EncodeWAL

func EncodeWAL(b *Batch) []byte

EncodeWAL frames a batch as a replication/WAL payload — a stream-identity record followed by the batch's records — exactly the form Engine.ApplyPrimary and Engine.ApplyReplicated replay. The cluster write path builds a tenant's payload by concatenating EncodeWAL over its streams and routing it to the ring primary.

Types

type AppendLimits added in v0.4.0

type AppendLimits struct {
	// MaxSeries caps the number of distinct streams buffered in the head. A record that would
	// register a new stream once the head already holds MaxSeries is rejected (the whole batch,
	// since a batch is one stream); known streams are unaffected. 0 ⇒ unlimited.
	MaxSeries int64
	// MaxInFlightBytes caps the head's buffered record bytes. A record arriving while the head is
	// at or over the cap is rejected (memory backpressure) until a flush drains it. 0 ⇒ unlimited.
	MaxInFlightBytes int64
}

AppendLimits are the per-call admission limits the record head enforces while buffering a batch. The zero value imposes no limit; they are passed per Engine.AppendBatch call so a consumer's hot-reloaded tenant policy takes effect on the next write. The engine stays policy-agnostic — it sees only these numbers. They mirror the metric engine's limits so the facade maps one tenant.Limits to both.

type AppendResult added in v0.4.0

type AppendResult struct {
	Accepted            int
	RejectedOOO         int // older than the out-of-order window
	RejectedCardinality int // would exceed AppendLimits.MaxSeries (a new stream)
	RejectedBytes       int // head at or over AppendLimits.MaxInFlightBytes
}

AppendResult reports the disposition of an Engine.AppendBatch run by reason, so the caller can attribute an OTLP partial-success precisely.

func (AppendResult) Rejected added in v0.4.0

func (r AppendResult) Rejected() int

Rejected returns the total number of rejected records across all reasons.

type Batch

type Batch struct {
	Stream   signal.SeriesID
	Identity func() signal.Series // materialized only when the stream is newly seen
	Ts       []int64
	Ints     [][]int64  // len == schema int count; Ints[k][row]
	Bytes    [][][]byte // len == schema byte count; Bytes[k][row]
	// Side is an optional encoded side-store delta (the content-addressed symbols this batch's
	// records reference) absorbed by [Config.SideStore]. nil when the engine has no side store.
	Side []byte
}

Batch is one stream's projected records in the engine's column layout: the primary timestamps plus the int and byte column vectors in the schema's per-kind order. The signal package builds it; the engine treats the columns opaquely. Byte slices may alias the source batch (the head clones them on append).

func (*Batch) ByteSize added in v0.4.0

func (b *Batch) ByteSize() int64

ByteSize returns the in-flight memory the batch's records occupy (timestamps, int columns, and the lengths of the byte columns) — the size a caller charges against an ingest-rate budget.

func (*Batch) Len

func (b *Batch) Len() int

Len returns the number of records in the batch.

type BloomMode

type BloomMode uint8

BloomMode says whether and how a column feeds its per-part bloom for predicate pruning.

const (
	// BloomNone builds no bloom for the column.
	BloomNone BloomMode = iota
	// BloomFullText tokenizes the column's value (lowercased words) so a `contains token`
	// condition can prune a part whose bloom lacks the token (e.g. a log body, a span name).
	BloomFullText
	// BloomAttrs treats the column as a serialized [signal.Attributes] blob and adds key-scoped
	// equality (`key‖value`) and full-text (`key‖word`) tokens, so per-record attribute equality
	// and contains conditions prune.
	BloomAttrs
	// BloomEquality adds the column's exact value as a token so a `column == value` condition can
	// prune (e.g. trace-by-id over the trace_id column).
	BloomEquality
)

type CardinalityStat added in v0.12.0

type CardinalityStat struct {
	TotalSeries        int64 // distinct streams
	DistinctLabelNames int
	SymbolCount        int
	Top                []LabelCard // sorted by Series desc, then Name; truncated to the requested top-N
}

CardinalityStat summarizes the engine's label cardinality (the head's index spans head ∪ flushed streams). TotalSeries and SymbolCount are exact; Top is the highest-cardinality label names.

type Column

type Column struct {
	Name  string
	Kind  Kind
	Codec chunk.Codec
	Bloom BloomMode
}

Column is one per-record column of a Schema: its name, physical kind, on-disk codec (zero ⇒ the kind's default), and bloom contribution.

type ColumnStat added in v0.12.0

type ColumnStat struct {
	Name     string
	Kind     string // physical type: int64 / float64 / bytes / int128
	Codec    string // value codec
	Compress string // block-compression algorithm
}

ColumnStat is one part column's physical description (from the manifest).

type Config

type Config struct {
	// Schema is the per-record column set this engine stores (required; the signal supplies it).
	Schema *Schema
	// OOOWindow rejects records older than newest-OOOWindow (nanoseconds). 0 disables.
	OOOWindow int64
	// WAL, when non-nil, durably logs streams and records for crash recovery. nil ⇒ ephemeral.
	WAL *wal.SegmentWriter
	// Backend stores flushed parts. Required for [Engine.Flush]; nil ⇒ head-only.
	Backend backend.Backend
	// Prefix is the backend key prefix under which this engine's parts are written.
	Prefix string
	// SideStore, when non-nil, is a signal-supplied content-addressed auxiliary store (e.g. the
	// profiles symbol store) that the engine persists as part sidecars on flush and unions on merge.
	// nil ⇒ no side data (logs, traces).
	SideStore SideStore
	// Obs is the observability handle (spans + metrics). nil ⇒ a no-op handle.
	Obs *obs.Obs
	// Signal is the signal label for this engine's metrics ("log"/"trace"/"profile"); the facade
	// sets it per signal. Empty ⇒ "record".
	Signal string
}

Config configures an Engine.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is one tenant's record store for a signal. Safe for concurrent use.

func New

func New(cfg Config) *Engine

New returns an engine with an empty head over cfg.Schema.

func (*Engine) AppendBatch

func (e *Engine) AppendBatch(b *Batch, limits AppendLimits) (AppendResult, error)

AppendBatch ingests one stream's records: it registers the stream on first sight, appends each record through the admission limits (OOO window, cardinality, in-flight bytes), and logs accepted records to the WAL. It returns an AppendResult breaking accepted/rejected down by reason, so the caller can report an exact OTLP partial-success. Safe for concurrent use.

func (*Engine) ApplyPrimary

func (e *Engine) ApplyPrimary(data []byte, limits AppendLimits) (accepted []byte, res AppendResult, err error)

ApplyPrimary applies a write as the stream's **primary**: it runs each record through the admission-checked append path (the single OOO decision for the shard, plus the cardinality and in-flight-memory valves from limits) and re-frames the *accepted* records into a WAL payload to replicate to the secondary owners. It returns that accepted payload and an AppendResult breaking the disposition down by reason, so the clustered ingest path attributes OTLP partial-success exactly like the single-node path. Every replica converges on the same data. Safe for concurrent use.

func (*Engine) ApplyReplicated

func (e *Engine) ApplyReplicated(data []byte) error

ApplyReplicated applies a replicated write from the primary verbatim (no OOO re-check — the primary already decided the accepted set), so all replicas hold identical data. Safe for concurrent use.

func (*Engine) Cardinality added in v0.12.0

func (e *Engine) Cardinality(topN int) CardinalityStat

Cardinality summarizes the engine's label cardinality from the head's inverted index (which spans every stream ever seen, flushed or not). topN bounds the returned Top slice (≤0 returns all label names). It takes a read lock and does no backend I/O.

func (*Engine) Close

func (e *Engine) Close(ctx context.Context) error

Close flushes any buffered records to a part and closes the WAL. It does not stop a background loop — the owner ([storage.Storage]) does that before calling Close.

func (*Engine) CloseWAL added in v0.4.0

func (e *Engine) CloseWAL() error

CloseWAL closes the engine's open WAL segment file handle without flushing the head or checkpointing — modeling a process crash, where the OS reclaims open descriptors but the on-disk WAL segments survive for replay. The head is left as-is (and lost, as a crash would lose it). A crash-recovery test uses this to release the file handle so the WAL directory can be removed even on platforms that refuse to delete a file held open by a live process (Windows). No-op without a WAL.

func (*Engine) Fetch

func (e *Engine) Fetch(ctx context.Context, r fetch.Request) (fetch.Iterator, error)

Fetch implements fetch.Fetcher over head ∪ flushed parts: it resolves matchers to streams, gathers each stream's in-window records (decoding only the referenced columns), applies the column conditions and projection, and returns one batch per stream sorted by timestamp.

func (*Engine) Flush

func (e *Engine) Flush(ctx context.Context) error

Flush writes the head's buffered records to a new immutable part and clears the buffers. No-op if the head is empty. Requires a Config.Backend.

func (*Engine) HeadBytes added in v0.4.0

func (e *Engine) HeadBytes() int64

HeadBytes returns the head's current buffered record bytes — the in-flight memory measure for AppendLimits.MaxInFlightBytes.

func (*Engine) HeadRecordCount

func (e *Engine) HeadRecordCount() int

HeadRecordCount returns the number of records buffered in the head across all streams.

func (*Engine) Keys added in v0.5.0

func (e *Engine) Keys(start, end int64) []KeyInfo

Keys enumerates the distinct attribute keys present across the engine's streams (head ∪ flushed parts) with at least one record in [start, end], each tagged with the scope(s) it appears in. A zero start AND end disables the time filter. Stream-identity keys (resource/scope) come from the authoritative series index; record-attribute keys come from the head buffers and each in-window part's persisted key footer. Window precision is part-granular and best-effort: a key whose records all fall outside the window may still be returned (harmless). Safe for concurrent use.

func (*Engine) LoadParts

func (e *Engine) LoadParts(ctx context.Context) error

LoadParts reconstructs the engine's durable state from the object store: the part set from the bucket index and the stream identity index from the persisted object. A head-only engine is a no-op. Replaces current parts and advances the sequence.

func (*Engine) Merge

func (e *Engine) Merge(ctx context.Context, retainFrom int64) error

Merge compacts every flushed part into a single new part, dropping records older than retainFrom (retention; retainFrom ≤ 0 disables it). No-op when there is nothing to gain — fewer than two parts and no retention cutoff. Records are append-only: a stream's records are concatenated across parts (no value dedup) and re-sorted by timestamp.

func (*Engine) MergeBacklog added in v0.12.0

func (e *Engine) MergeBacklog() int

MergeBacklog returns the number of flushed parts — the compaction-backlog proxy. Brief read lock.

func (*Engine) MergeRunning added in v0.12.0

func (e *Engine) MergeRunning() bool

MergeRunning reports whether a merge/compaction is currently executing on this engine (an in-memory liveness flag for introspection).

func (*Engine) PartCount

func (e *Engine) PartCount() int

PartCount returns the number of flushed parts (introspection).

func (*Engine) Parts added in v0.12.0

func (e *Engine) Parts() []PartStat

Parts returns an in-memory snapshot of the engine's flushed parts under a read lock — no backend I/O and no decode, safe to poll. For byte sizes, codecs, and chunk counts, use Engine.PartsDetailed.

func (*Engine) PartsDetailed added in v0.12.0

func (e *Engine) PartsDetailed(ctx context.Context) ([]PartDetailStat, error)

PartsDetailed augments Engine.Parts with each part's on-backend byte size, column/codec layout, and chunk (granule) count. It reads object sizes from the backend, so unlike Parts it is not hot-path-free — call it for a drill-down view, not a high-frequency poll. Each part is ref-held for the duration so a concurrent merge cannot reclaim its objects mid-read.

func (*Engine) RefreshReplica

func (e *Engine) RefreshReplica(ctx context.Context) error

RefreshReplica brings a replica node's view up to date with the shared object store: it reconstructs the flushed parts and trims its head to the still-unflushed window. With no shared store, a safe no-op.

func (*Engine) Replay

func (e *Engine) Replay(dir string) error

Replay rebuilds the head (and side store) from the WAL segments in dir (durable restart). It skips segments at or below the flush watermark recovered by Engine.LoadParts (call LoadParts first), so records already in a flushed part are not re-applied — exactly-once recovery.

func (*Engine) Reset

func (e *Engine) Reset(ctx context.Context) error

Reset discards all data (head + parts) and deletes this engine's part objects, returning it to the empty state without reallocating. Safe for concurrent use.

func (*Engine) Series

func (e *Engine) Series(matchers []fetch.Matcher, start, end int64) []signal.Series

Series returns the identities of the streams matching matchers that hold at least one record in [start, end] — the enumeration primitive behind profile-type / label listing. A zero start AND end disables the time filter (return every matching stream). The time filter is part-overlap granular (a returned stream is guaranteed to match the matchers; its in-window records are a superset check). Safe for concurrent use.

func (*Engine) SideSnapshot

func (e *Engine) SideSnapshot(ctx context.Context) (map[string][]byte, error)

SideSnapshot returns the engine's full side-store tables — the live head accumulator unioned with every flushed part's sidecars — as named payloads, for a signal to build a resolver over (e.g. the profiles symbol store). nil when the engine has no side store. Safe for concurrent use.

func (*Engine) Stats added in v0.10.0

func (e *Engine) Stats() Stats

Stats returns an in-memory snapshot of the engine's state under a single read lock (no backend I/O, no decode), safe to poll at dashboard cadence. Part byte sizes are not included.

func (*Engine) StreamCount

func (e *Engine) StreamCount() int

StreamCount returns the number of distinct streams in the head.

func (*Engine) SyncWAL added in v0.3.0

func (e *Engine) SyncWAL() error

SyncWAL fsyncs the engine's WAL, if any (the background WALSyncInterval path). No-op without a WAL.

func (*Engine) WALState added in v0.12.0

func (e *Engine) WALState() (segments int, bytes int64, epoch uint64, ok bool)

WALState returns the current WAL segment count, the open segment's byte size, and the active flush epoch. ok is false when the engine has no WAL (ephemeral). It takes a read lock, excluding concurrent appends (which hold the write lock).

type KeyInfo added in v0.5.0

type KeyInfo struct {
	Key   []byte
	Scope KeyScope
}

KeyInfo is a distinct attribute key and the union of the scopes it was observed in. Key aliases engine-owned bytes (a head identity or a decoded part-footer entry); copy it to retain.

type KeyScope added in v0.5.0

type KeyScope uint8

KeyScope is a bitset of the scopes an attribute key was observed in. A key can appear in more than one — e.g. as a resource attribute on one stream and a per-record attribute on another — and the bitset records every scope, so a caller can tell a stream label from a record attribute (or both).

const (
	// KeyScopeResource marks a resource attribute (part of the stream identity, postings-indexed).
	KeyScopeResource KeyScope = 1 << iota
	// KeyScopeScope marks an instrumentation-scope attribute (also stream identity).
	KeyScopeScope
	// KeyScopeRecord marks a per-record attribute (the serialized attrs column).
	KeyScopeRecord
)

type Kind

type Kind uint8

Kind is a column's physical type. Records use only int64 and byte-string columns (floats live in the metrics engine); a typed value is projected onto one of these at the language edge.

const (
	// KindInt64 is a signed 64-bit column (timestamps-relative, severities, durations, ids, …).
	KindInt64 Kind = iota
	// KindBytes is a byte-string column (bodies, names, trace ids, serialized attributes, …).
	KindBytes
)

type LabelCard added in v0.12.0

type LabelCard struct {
	Name           string
	Series         int64
	DistinctValues int
}

LabelCard is one label name's cardinality: how many streams carry it and how many distinct values it takes across them.

type PartDetailStat added in v0.12.0

type PartDetailStat struct {
	PartStat

	Bytes   int64        // sum of the part's backend object sizes
	Chunks  int          // sparse-index granules: ceil(RowCount / GranuleSize)
	Columns []ColumnStat // per-column physical layout
}

PartDetailStat augments PartStat with fields that need a backend read: the on-backend byte size (summed over the part's objects) and the column/codec layout and chunk count from the manifest (cached on the open part, so only Bytes incurs additional I/O).

type PartStat added in v0.12.0

type PartStat struct {
	ID      string // the part's backend key prefix
	MinTime int64  // inclusive unix-ns bounds of the part's records
	MaxTime int64
	Series  int   // distinct streams in the part (len of its row-range index)
	Rows    int64 // total records (sum of the per-stream row spans)
}

PartStat is one flushed part's in-memory shape (no backend I/O, no decode): identity, time bounds, and the stream/row counts from the part's in-memory row-range index.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema is the ordered set of per-record columns a signal stores. It is immutable after construction and shared by every engine, part, and record batch of that signal.

func NewSchema

func NewSchema(cols ...Column) *Schema

NewSchema builds a Schema from its columns (declaration order is preserved within each kind).

type SideStore

type SideStore interface {
	// Absorb merges one batch's encoded side delta ([Batch.Side]) into the live accumulator.
	Absorb(delta []byte) error
	// Encode serializes the accumulated side data into named sidecar payloads (name → bytes),
	// written as {prefix}/sym-{name}.bin at flush.
	Encode() map[string][]byte
	// Reset clears the live accumulator (after a flush drains the head).
	Reset()
	// Names returns the sidecar names to read back for a part on merge (the keys [SideStore.Encode]
	// may produce). A part missing a named sidecar is skipped.
	Names() []string
	// Union merges the loaded sidecars of the compacted parts (one map per part) and returns the
	// merged named payloads to write under the new part. Pure; ignores the live accumulator.
	Union(parts []map[string][]byte) (map[string][]byte, error)
}

SideStore is an optional per-engine auxiliary store that rides the part lifecycle. A signal whose records reference content-addressed side data (e.g. the profiles symbol store: strings, functions, locations, stacks) supplies one via Config.SideStore; the engine then absorbs each batch's delta (Batch.Side) into a live accumulator, persists the accumulator as part sidecars on flush, and unions the sidecars of compacted parts on merge. The engine treats the data opaquely — only the signal package knows the table formats.

Content-addressing is the load-bearing assumption: an entry's id is a hash of its content, so the same entry has the same id everywhere and SideStore.Union is a plain dedup with no id remap.

All methods are called under the engine's lock, so an implementation need not be safe for concurrent use. SideStore.Union must be a pure function of its arguments and must not read or mutate the live accumulator (it merges already-flushed part data, independent of the head).

type Stats added in v0.10.0

type Stats struct {
	Streams     int64 // distinct streams ever seen (index span: head ∪ flushed)
	HeadRecords int64 // records currently buffered in the head (unflushed)
	HeadBytes   int64 // head's buffered record bytes (the in-flight memory measure)
	Parts       int   // flushed immutable parts
	MinTime     int64 // oldest flushed record time (unix ns); 0 when no parts
	MaxTime     int64 // newest record time across parts and the head (unix ns); 0 when empty
}

Stats is an in-memory snapshot of a record engine's state for introspection (no backend I/O).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL