Documentation
¶
Overview ¶
Package engine is the L3 single-node storage engine: an in-memory head that absorbs writes (indexing labels and buffering samples) with a bounded out-of-order window, optionally backed by a write-ahead log for crash recovery. It implements the fetch contract over the head. Flush to immutable parts and background merge (compaction, retention, downsampling) are added in later slices.
Index ¶
- type Config
- type Engine
- func (e *Engine) Append(s signal.Series, ts int64, value float64) (bool, error)
- func (e *Engine) AppendBatch(ids []signal.SeriesID, ts []int64, values []float64, ...) (accepted int, err error)
- func (e *Engine) ApplyPrimary(data []byte) (accepted []byte, rejected int, err error)
- func (e *Engine) ApplyReplicated(data []byte) error
- func (e *Engine) Close(ctx context.Context) error
- func (e *Engine) Fetch(ctx context.Context, r fetch.Request) (fetch.Iterator, error)
- func (e *Engine) Flush(ctx context.Context) error
- func (e *Engine) HeadSampleCount() int
- func (e *Engine) LoadParts(ctx context.Context) error
- func (e *Engine) Merge(ctx context.Context, retainFrom int64) error
- func (e *Engine) PartCount() int
- func (e *Engine) RefreshReplica(ctx context.Context) error
- func (e *Engine) Replay(dir string) error
- func (e *Engine) Reset(ctx context.Context) error
- func (e *Engine) SeriesCount() int
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// OOOWindow rejects samples older than newest-OOOWindow (nanoseconds). 0 disables.
OOOWindow int64
// WAL, when non-nil, durably logs series and samples for crash recovery. nil is the
// ephemeral in-memory engine.
WAL *wal.SegmentWriter
// Backend stores flushed parts. Required for [Engine.Flush]; nil is a head-only engine.
Backend backend.Backend
// Prefix is the backend key prefix under which this engine's parts are written
// (typically "{tenant}/metrics").
Prefix string
}
Config configures an Engine.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is a single tenant's storage engine. Safe for concurrent use.
func (*Engine) Append ¶
Append ingests one sample for series s, logging to the WAL when durable. It returns whether the sample was accepted (false ⇒ rejected as out-of-order beyond the window).
func (*Engine) AppendBatch ¶
func (e *Engine) AppendBatch( ids []signal.SeriesID, ts []int64, values []float64, materialize func(i int) signal.Series, ) (accepted int, err error)
AppendBatch ingests a run of samples whose content ids are already computed (by the projection layer, on a reused buffer). ids[i], ts[i], values[i] describe sample i; materialize(i) returns sample i's full identity and is called only when its series is new (first sight), so a repeat series costs just a map probe and a buffer append, with no per-point signal.Series construction or hashing. The whole run is appended under a single lock. It returns how many samples were accepted (the rest were out-of-order beyond the window). Safe for concurrent use.
func (*Engine) ApplyPrimary ¶ added in v0.2.0
ApplyPrimary applies a write as the shard's **primary**: it appends each sample through the out-of-order-checked path (the single OOO decision for the shard) and re-frames the *accepted* samples into a WAL payload to replicate to the secondary owners. It returns that accepted payload and the number of samples rejected as out-of-order. Because only the primary OOO-checks and it dictates the accepted set, every replica converges on the same data regardless of concurrent writers. Safe for concurrent use.
func (*Engine) ApplyReplicated ¶
ApplyReplicated applies a replicated write from the shard's primary to this secondary's head: it registers each series and appends its samples **verbatim** (no OOO re-check — the primary already decided the accepted set, the same way WAL Engine.Replay trusts the log), so all replicas hold identical data. A replica holds the unflushed head this way; after a flush the shared object store reconciles them. Safe for concurrent use.
func (*Engine) Close ¶
Close flushes any buffered samples to a part and closes the WAL. It does not stop a background loop — the owner ([storage.Storage]) does that before calling Close.
func (*Engine) Fetch ¶
Fetch implements fetch.Fetcher over the head ∪ flushed parts: it resolves the request's matchers to series (the index spans every series ever seen, flushed or not) and returns one batch per series with its samples in the window, merged across the head buffer and every part by timestamp.
func (*Engine) Flush ¶
Flush writes the head's buffered samples to a new immutable part and clears the buffers (the series index is retained). It is a no-op if the head holds no samples. Requires a Config.Backend.
func (*Engine) HeadSampleCount ¶
HeadSampleCount returns the number of samples currently buffered in the head (across all series) — for introspection and tests (e.g. to observe replica head trimming).
func (*Engine) LoadParts ¶
LoadParts reconstructs the engine's durable state from the object store: the part set from the bucket index, and the series identity index (postings + labels) from the persisted identity object. It is how a fresh engine over an existing prefix serves reads with no in-memory state carried over from the writer (the stateless read path); typically called once after New during recovery. WAL Engine.Replay is complementary — it restores the unflushed head samples — but is not required to query flushed data.
It replaces any current parts and advances the part sequence past the highest existing part. A head-only engine (no backend) is a no-op.
func (*Engine) Merge ¶
Merge compacts every flushed part into a single new part, dropping samples older than retainFrom (retention; retainFrom ≤ 0 disables it). It is a no-op when there is nothing to gain — fewer than two parts and no retention cutoff. Source parts are deleted from the backend after the new part is durably written.
Retention is expressed as an absolute timestamp (unix nanoseconds), so the engine stays free of wall-clock dependencies; the caller derives it from the tenant policy.
func (*Engine) RefreshReplica ¶
RefreshReplica brings a replica node's view up to date with the shared object store: it reconstructs the flushed parts from the bucket index and trims its head to the still-unflushed window — samples a primary has already flushed (covered by a part) are dropped, bounding replica memory. With no shared store (this node cannot see the parts), it is a safe no-op: nothing loads, so nothing is trimmed.
func (*Engine) Reset ¶
Reset discards all of the engine's data — the in-memory head (samples + series index) and every flushed part — returning it to the empty state of a freshly New'd engine, without reallocating the engine itself. Flushed part objects are deleted from the backend so none are orphaned. It is destructive (it wipes this engine's parts under Config.Prefix) and is meant for the ephemeral in-memory engine in tests and benchmarks, letting a long-lived engine be reused across runs. Safe for concurrent use.
func (*Engine) SeriesCount ¶
SeriesCount returns the number of distinct series in the head.
Source Files