Documentation
¶
Overview ¶
Package recordengine is the shared storage engine for **record-shaped** signals — logs and traces (later, profiles' sample table). A record-shaped signal is a stream (a Resource+Scope identity, indexed by the postings layer) of rows that each carry a primary timestamp plus a fixed set of typed columns. The engine is the structural twin of package engine (metrics) but generic over a Schema: it owns the in-memory head, columnar flush to immutable parts, the durable bucket-index + identity-index stateless read path, append-only merge with retention, per-part column blooms, lazy column decode, and the fetch.Fetcher contract — none of which is signal-specific. A signal package supplies the column Schema and projects its model into the engine's column vectors; the engine treats the columns opaquely.
The timestamp (the sort key) and the int128 stream id (the Resource+Scope hash) are implicit and not part of the Schema; the schema lists only the per-record columns.
Index ¶
- Variables
- func EncodeWAL(b *Batch) []byte
- type AppendLimits
- type AppendResult
- type Batch
- type BloomMode
- type CardinalityStat
- type Column
- type ColumnStat
- type Config
- type Engine
- func (e *Engine) AppendBatch(b *Batch, limits AppendLimits) (AppendResult, error)
- func (e *Engine) ApplyPrimary(data []byte, limits AppendLimits) (accepted []byte, res AppendResult, err error)
- func (e *Engine) ApplyReplicated(data []byte) error
- func (e *Engine) Cardinality(topN int) CardinalityStat
- func (e *Engine) Close(ctx context.Context) error
- func (e *Engine) CloseWAL() error
- func (e *Engine) Fetch(ctx context.Context, r fetch.Request) (fetch.Iterator, error)
- func (e *Engine) Flush(ctx context.Context) error
- func (e *Engine) HeadBytes() int64
- func (e *Engine) HeadRecordCount() int
- func (e *Engine) Keys(start, end int64) []KeyInfo
- func (e *Engine) LoadParts(ctx context.Context) error
- func (e *Engine) Merge(ctx context.Context, retainFrom int64) error
- func (e *Engine) MergeBacklog() int
- func (e *Engine) MergeRunning() bool
- func (e *Engine) PartCount() int
- func (e *Engine) Parts() []PartStat
- func (e *Engine) PartsDetailed(ctx context.Context) ([]PartDetailStat, error)
- func (e *Engine) RefreshReplica(ctx context.Context) error
- func (e *Engine) Replay(dir string) error
- func (e *Engine) Reset(ctx context.Context) error
- func (e *Engine) Series(matchers []fetch.Matcher, start, end int64) []signal.Series
- func (e *Engine) SideSnapshot(ctx context.Context) (map[string][]byte, error)
- func (e *Engine) Stats() Stats
- func (e *Engine) StreamCount() int
- func (e *Engine) SyncWAL() error
- func (e *Engine) WALState() (segments int, bytes int64, epoch uint64, ok bool)
- type KeyInfo
- type KeyScope
- type Kind
- type LabelCard
- type PartDetailStat
- type PartStat
- type Schema
- type SideStore
- type Stats
Constants ¶
This section is empty.
Variables ¶
var ErrCorruptKeys = errors.New("recordengine: corrupt record-keys footer")
ErrCorruptKeys is returned when a serialized record-keys footer fails to parse.
Functions ¶
func EncodeWAL ¶
EncodeWAL frames a batch as a replication/WAL payload — a stream-identity record followed by the batch's records — exactly the form Engine.ApplyPrimary and Engine.ApplyReplicated replay. The cluster write path builds a tenant's payload by concatenating EncodeWAL over its streams and routing it to the ring primary.
Types ¶
type AppendLimits ¶ added in v0.4.0
type AppendLimits struct {
// MaxSeries caps the number of distinct streams buffered in the head. A record that would
// register a new stream once the head already holds MaxSeries is rejected (the whole batch,
// since a batch is one stream); known streams are unaffected. 0 ⇒ unlimited.
MaxSeries int64
// MaxInFlightBytes caps the head's buffered record bytes. A record arriving while the head is
// at or over the cap is rejected (memory backpressure) until a flush drains it. 0 ⇒ unlimited.
MaxInFlightBytes int64
}
AppendLimits are the per-call admission limits the record head enforces while buffering a batch. The zero value imposes no limit; they are passed per Engine.AppendBatch call so a consumer's hot-reloaded tenant policy takes effect on the next write. The engine stays policy-agnostic — it sees only these numbers. They mirror the metric engine's limits so the facade maps one tenant.Limits to both.
type AppendResult ¶ added in v0.4.0
type AppendResult struct {
Accepted int
RejectedOOO int // older than the out-of-order window
RejectedCardinality int // would exceed AppendLimits.MaxSeries (a new stream)
RejectedBytes int // head at or over AppendLimits.MaxInFlightBytes
}
AppendResult reports the disposition of an Engine.AppendBatch run by reason, so the caller can attribute an OTLP partial-success precisely.
func (AppendResult) Rejected ¶ added in v0.4.0
func (r AppendResult) Rejected() int
Rejected returns the total number of rejected records across all reasons.
type Batch ¶
type Batch struct {
Stream signal.SeriesID
Identity func() signal.Series // materialized only when the stream is newly seen
Ts []int64
Ints [][]int64 // len == schema int count; Ints[k][row]
Bytes [][][]byte // len == schema byte count; Bytes[k][row]
// Side is an optional encoded side-store delta (the content-addressed symbols this batch's
// records reference) absorbed by [Config.SideStore]. nil when the engine has no side store.
Side []byte
}
Batch is one stream's projected records in the engine's column layout: the primary timestamps plus the int and byte column vectors in the schema's per-kind order. The signal package builds it; the engine treats the columns opaquely. Byte slices may alias the source batch (the head clones them on append).
type BloomMode ¶
type BloomMode uint8
BloomMode says whether and how a column feeds its per-part bloom for predicate pruning.
const ( // BloomNone builds no bloom for the column. BloomNone BloomMode = iota // BloomFullText tokenizes the column's value (lowercased words) so a `contains token` // condition can prune a part whose bloom lacks the token (e.g. a log body, a span name). BloomFullText // BloomAttrs treats the column as a serialized [signal.Attributes] blob and adds key-scoped // equality (`key‖value`) and full-text (`key‖word`) tokens, so per-record attribute equality // and contains conditions prune. BloomAttrs // BloomEquality adds the column's exact value as a token so a `column == value` condition can // prune (e.g. trace-by-id over the trace_id column). BloomEquality )
type CardinalityStat ¶ added in v0.12.0
type CardinalityStat struct {
TotalSeries int64 // distinct streams
DistinctLabelNames int
SymbolCount int
Top []LabelCard // sorted by Series desc, then Name; truncated to the requested top-N
}
CardinalityStat summarizes the engine's label cardinality (the head's index spans head ∪ flushed streams). TotalSeries and SymbolCount are exact; Top is the highest-cardinality label names.
type Column ¶
Column is one per-record column of a Schema: its name, physical kind, on-disk codec (zero ⇒ the kind's default), and bloom contribution.
type ColumnStat ¶ added in v0.12.0
type ColumnStat struct {
Name string
Kind string // physical type: int64 / float64 / bytes / int128
Codec string // value codec
Compress string // block-compression algorithm
}
ColumnStat is one part column's physical description (from the manifest).
type Config ¶
type Config struct {
// Schema is the per-record column set this engine stores (required; the signal supplies it).
Schema *Schema
// OOOWindow rejects records older than newest-OOOWindow (nanoseconds). 0 disables.
OOOWindow int64
// WAL, when non-nil, durably logs streams and records for crash recovery. nil ⇒ ephemeral.
WAL *wal.SegmentWriter
// Backend stores flushed parts. Required for [Engine.Flush]; nil ⇒ head-only.
Backend backend.Backend
// Prefix is the backend key prefix under which this engine's parts are written.
Prefix string
// SideStore, when non-nil, is a signal-supplied content-addressed auxiliary store (e.g. the
// profiles symbol store) that the engine persists as part sidecars on flush and unions on merge.
// nil ⇒ no side data (logs, traces).
SideStore SideStore
// Obs is the observability handle (spans + metrics). nil ⇒ a no-op handle.
Obs *obs.Obs
// Signal is the signal label for this engine's metrics ("log"/"trace"/"profile"); the facade
// sets it per signal. Empty ⇒ "record".
Signal string
}
Config configures an Engine.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is one tenant's record store for a signal. Safe for concurrent use.
func (*Engine) AppendBatch ¶
func (e *Engine) AppendBatch(b *Batch, limits AppendLimits) (AppendResult, error)
AppendBatch ingests one stream's records: it registers the stream on first sight, appends each record through the admission limits (OOO window, cardinality, in-flight bytes), and logs accepted records to the WAL. It returns an AppendResult breaking accepted/rejected down by reason, so the caller can report an exact OTLP partial-success. Safe for concurrent use.
func (*Engine) ApplyPrimary ¶
func (e *Engine) ApplyPrimary(data []byte, limits AppendLimits) (accepted []byte, res AppendResult, err error)
ApplyPrimary applies a write as the stream's **primary**: it runs each record through the admission-checked append path (the single OOO decision for the shard, plus the cardinality and in-flight-memory valves from limits) and re-frames the *accepted* records into a WAL payload to replicate to the secondary owners. It returns that accepted payload and an AppendResult breaking the disposition down by reason, so the clustered ingest path attributes OTLP partial-success exactly like the single-node path. Every replica converges on the same data. Safe for concurrent use.
func (*Engine) ApplyReplicated ¶
ApplyReplicated applies a replicated write from the primary verbatim (no OOO re-check — the primary already decided the accepted set), so all replicas hold identical data. Safe for concurrent use.
func (*Engine) Cardinality ¶ added in v0.12.0
func (e *Engine) Cardinality(topN int) CardinalityStat
Cardinality summarizes the engine's label cardinality from the head's inverted index (which spans every stream ever seen, flushed or not). topN bounds the returned Top slice (≤0 returns all label names). It takes a read lock and does no backend I/O.
func (*Engine) Close ¶
Close flushes any buffered records to a part and closes the WAL. It does not stop a background loop — the owner ([storage.Storage]) does that before calling Close.
func (*Engine) CloseWAL ¶ added in v0.4.0
CloseWAL closes the engine's open WAL segment file handle without flushing the head or checkpointing — modeling a process crash, where the OS reclaims open descriptors but the on-disk WAL segments survive for replay. The head is left as-is (and lost, as a crash would lose it). A crash-recovery test uses this to release the file handle so the WAL directory can be removed even on platforms that refuse to delete a file held open by a live process (Windows). No-op without a WAL.
func (*Engine) Fetch ¶
Fetch implements fetch.Fetcher over head ∪ flushed parts: it resolves matchers to streams, gathers each stream's in-window records (decoding only the referenced columns), applies the column conditions and projection, and returns one batch per stream sorted by timestamp.
func (*Engine) Flush ¶
Flush writes the head's buffered records to a new immutable part and clears the buffers. No-op if the head is empty. Requires a Config.Backend.
func (*Engine) HeadBytes ¶ added in v0.4.0
HeadBytes returns the head's current buffered record bytes — the in-flight memory measure for AppendLimits.MaxInFlightBytes.
func (*Engine) HeadRecordCount ¶
HeadRecordCount returns the number of records buffered in the head across all streams.
func (*Engine) Keys ¶ added in v0.5.0
Keys enumerates the distinct attribute keys present across the engine's streams (head ∪ flushed parts) with at least one record in [start, end], each tagged with the scope(s) it appears in. A zero start AND end disables the time filter. Stream-identity keys (resource/scope) come from the authoritative series index; record-attribute keys come from the head buffers and each in-window part's persisted key footer. Window precision is part-granular and best-effort: a key whose records all fall outside the window may still be returned (harmless). Safe for concurrent use.
func (*Engine) LoadParts ¶
LoadParts reconstructs the engine's durable state from the object store: the part set from the bucket index and the stream identity index from the persisted object. A head-only engine is a no-op. Replaces current parts and advances the sequence.
func (*Engine) Merge ¶
Merge compacts every flushed part into a single new part, dropping records older than retainFrom (retention; retainFrom ≤ 0 disables it). No-op when there is nothing to gain — fewer than two parts and no retention cutoff. Records are append-only: a stream's records are concatenated across parts (no value dedup) and re-sorted by timestamp.
func (*Engine) MergeBacklog ¶ added in v0.12.0
MergeBacklog returns the number of flushed parts — the compaction-backlog proxy. Brief read lock.
func (*Engine) MergeRunning ¶ added in v0.12.0
MergeRunning reports whether a merge/compaction is currently executing on this engine (an in-memory liveness flag for introspection).
func (*Engine) Parts ¶ added in v0.12.0
Parts returns an in-memory snapshot of the engine's flushed parts under a read lock — no backend I/O and no decode, safe to poll. For byte sizes, codecs, and chunk counts, use Engine.PartsDetailed.
func (*Engine) PartsDetailed ¶ added in v0.12.0
func (e *Engine) PartsDetailed(ctx context.Context) ([]PartDetailStat, error)
PartsDetailed augments Engine.Parts with each part's on-backend byte size, column/codec layout, and chunk (granule) count. It reads object sizes from the backend, so unlike Parts it is not hot-path-free — call it for a drill-down view, not a high-frequency poll. Each part is ref-held for the duration so a concurrent merge cannot reclaim its objects mid-read.
func (*Engine) RefreshReplica ¶
RefreshReplica brings a replica node's view up to date with the shared object store: it reconstructs the flushed parts and trims its head to the still-unflushed window. With no shared store, a safe no-op.
func (*Engine) Replay ¶
Replay rebuilds the head (and side store) from the WAL segments in dir (durable restart). It skips segments at or below the flush watermark recovered by Engine.LoadParts (call LoadParts first), so records already in a flushed part are not re-applied — exactly-once recovery.
func (*Engine) Reset ¶
Reset discards all data (head + parts) and deletes this engine's part objects, returning it to the empty state without reallocating. Safe for concurrent use.
func (*Engine) Series ¶
Series returns the identities of the streams matching matchers that hold at least one record in [start, end] — the enumeration primitive behind profile-type / label listing. A zero start AND end disables the time filter (return every matching stream). The time filter is part-overlap granular (a returned stream is guaranteed to match the matchers; its in-window records are a superset check). Safe for concurrent use.
func (*Engine) SideSnapshot ¶
SideSnapshot returns the engine's full side-store tables — the live head accumulator unioned with every flushed part's sidecars — as named payloads, for a signal to build a resolver over (e.g. the profiles symbol store). nil when the engine has no side store. Safe for concurrent use.
func (*Engine) Stats ¶ added in v0.10.0
Stats returns an in-memory snapshot of the engine's state under a single read lock (no backend I/O, no decode), safe to poll at dashboard cadence. Part byte sizes are not included.
func (*Engine) StreamCount ¶
StreamCount returns the number of distinct streams in the head.
func (*Engine) SyncWAL ¶ added in v0.3.0
SyncWAL fsyncs the engine's WAL, if any (the background WALSyncInterval path). No-op without a WAL.
type KeyInfo ¶ added in v0.5.0
KeyInfo is a distinct attribute key and the union of the scopes it was observed in. Key aliases engine-owned bytes (a head identity or a decoded part-footer entry); copy it to retain.
type KeyScope ¶ added in v0.5.0
type KeyScope uint8
KeyScope is a bitset of the scopes an attribute key was observed in. A key can appear in more than one — e.g. as a resource attribute on one stream and a per-record attribute on another — and the bitset records every scope, so a caller can tell a stream label from a record attribute (or both).
const ( // KeyScopeResource marks a resource attribute (part of the stream identity, postings-indexed). KeyScopeResource KeyScope = 1 << iota // KeyScopeScope marks an instrumentation-scope attribute (also stream identity). KeyScopeScope // KeyScopeRecord marks a per-record attribute (the serialized attrs column). KeyScopeRecord )
type Kind ¶
type Kind uint8
Kind is a column's physical type. Records use only int64 and byte-string columns (floats live in the metrics engine); a typed value is projected onto one of these at the language edge.
type LabelCard ¶ added in v0.12.0
LabelCard is one label name's cardinality: how many streams carry it and how many distinct values it takes across them.
type PartDetailStat ¶ added in v0.12.0
type PartDetailStat struct {
PartStat
Bytes int64 // sum of the part's backend object sizes
Chunks int // sparse-index granules: ceil(RowCount / GranuleSize)
Columns []ColumnStat // per-column physical layout
}
PartDetailStat augments PartStat with fields that need a backend read: the on-backend byte size (summed over the part's objects) and the column/codec layout and chunk count from the manifest (cached on the open part, so only Bytes incurs additional I/O).
type PartStat ¶ added in v0.12.0
type PartStat struct {
ID string // the part's backend key prefix
MinTime int64 // inclusive unix-ns bounds of the part's records
MaxTime int64
Series int // distinct streams in the part (len of its row-range index)
Rows int64 // total records (sum of the per-stream row spans)
}
PartStat is one flushed part's in-memory shape (no backend I/O, no decode): identity, time bounds, and the stream/row counts from the part's in-memory row-range index.
type Schema ¶
type Schema struct {
// contains filtered or unexported fields
}
Schema is the ordered set of per-record columns a signal stores. It is immutable after construction and shared by every engine, part, and record batch of that signal.
type SideStore ¶
type SideStore interface {
// Absorb merges one batch's encoded side delta ([Batch.Side]) into the live accumulator.
Absorb(delta []byte) error
// Encode serializes the accumulated side data into named sidecar payloads (name → bytes),
// written as {prefix}/sym-{name}.bin at flush.
Encode() map[string][]byte
// Reset clears the live accumulator (after a flush drains the head).
Reset()
// Names returns the sidecar names to read back for a part on merge (the keys [SideStore.Encode]
// may produce). A part missing a named sidecar is skipped.
Names() []string
// Union merges the loaded sidecars of the compacted parts (one map per part) and returns the
// merged named payloads to write under the new part. Pure; ignores the live accumulator.
Union(parts []map[string][]byte) (map[string][]byte, error)
}
SideStore is an optional per-engine auxiliary store that rides the part lifecycle. A signal whose records reference content-addressed side data (e.g. the profiles symbol store: strings, functions, locations, stacks) supplies one via Config.SideStore; the engine then absorbs each batch's delta (Batch.Side) into a live accumulator, persists the accumulator as part sidecars on flush, and unions the sidecars of compacted parts on merge. The engine treats the data opaquely — only the signal package knows the table formats.
Content-addressing is the load-bearing assumption: an entry's id is a hash of its content, so the same entry has the same id everywhere and SideStore.Union is a plain dedup with no id remap.
All methods are called under the engine's lock, so an implementation need not be safe for concurrent use. SideStore.Union must be a pure function of its arguments and must not read or mutate the live accumulator (it merges already-flushed part data, independent of the head).
type Stats ¶ added in v0.10.0
type Stats struct {
Streams int64 // distinct streams ever seen (index span: head ∪ flushed)
HeadRecords int64 // records currently buffered in the head (unflushed)
HeadBytes int64 // head's buffered record bytes (the in-flight memory measure)
Parts int // flushed immutable parts
MinTime int64 // oldest flushed record time (unix ns); 0 when no parts
MaxTime int64 // newest record time across parts and the head (unix ns); 0 when empty
}
Stats is an in-memory snapshot of a record engine's state for introspection (no backend I/O).