recordstore

package
v0.1.37 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 15, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package recordstore keeps append-only record streams: the rows a long capture produces, written as they arrive and read back by position instead of being carried inside whatever reported the capture.

A stream is named by a stream id and holds rows of one kind. Every row gets a per-stream seq, contiguous from 1, which is the only position a reader resumes from — never a timestamp or a key, which neither order nor identify a row's position reliably. A kind may still declare a key, and then a stream holds each key once, which is what makes re-ingesting an overlapping source window idempotent. Rows leave a stream only from its low end: the whole stream expires, or Trim removes the oldest appends.

Backends live in subpackages: kv (a clicky cache.Store: in-process memory or valkey/redis), sqlite (a file that is also the query index a profile reads), and ndjson (a file per stream). An Indexer mirrors any of them into a sqlite index incrementally, which is what makes a stream written by one process pageable through another's query engine.

A stream has one writer at a time. Backends serialize appends made within one process; two processes appending to one stream is outside the contract.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotFound reports a stream that does not exist, or no longer does.
	ErrNotFound = errors.New("record stream not found")

	// ErrCapacity reports an append refused because the stream, or one row of
	// it, would exceed what the backend was configured to hold. The rows of a
	// refused append are not written — none of them.
	ErrCapacity = errors.New("record stream capacity exceeded")
)

Functions

func ValidateAppend

func ValidateAppend(stream, kind string) error

ValidateAppend checks an append's stream id and kind before a backend writes anything.

func ValidateKind

func ValidateKind(kind string) error

ValidateKind rejects a kind that could not name a table, a directory and a profile.

func ValidateStream

func ValidateStream(stream string) error

ValidateStream rejects a stream id a backend could not key by. The alphabet is narrow on purpose: an id becomes a Redis key segment, a file name and a query parameter, and every character outside it is a character one of those would have to escape.

func ValidateTTL

func ValidateTTL(ttl time.Duration) error

ValidateTTL rejects an expiry that is not in the future.

Types

type AppendResult

type AppendResult struct {
	Window  Window `json:"window"`
	Skipped int64  `json:"skipped"`
}

AppendResult is what one append stored: the window its rows were numbered into, and how many rows it skipped because their key was already stored.

func AppendTyped

func AppendTyped[T any](ctx context.Context, backend Backend, stream, kind string, items []T) (AppendResult, error)

AppendTyped appends items as rows through their JSON encoding, which is the shape every backend stores and every reader sees.

type Backend

type Backend interface {
	// Append adds rows to stream, creating it under kind when it does not
	// exist, and returns the window the rows were numbered into. Appending to
	// an existing stream under a different kind is an error.
	//
	// A keyed kind (KindOptions.Key) skips every row whose key the stream
	// already holds, atomically with the append, and numbers only the rows it
	// keeps; a batch naming one key twice is refused whole. A kind retaining
	// rows (RetainRows) slides the stream's expiry to the backend ttl from now
	// and trims the rows appended longer than that ago, in the same write.
	Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)

	// Meta describes stream, or returns ErrNotFound.
	Meta(ctx context.Context, stream string) (Meta, error)

	// Scan calls fn with every row after afterSeq, in seq order, stopping at
	// the first error fn returns. A scan from below the stream's low seq starts
	// at the low seq. An unknown stream is ErrNotFound.
	Scan(ctx context.Context, stream string, afterSeq int64, fn func(seq int64, row Row) error) error

	// Trim removes the rows appended before before — every append up to the
	// last one made before it — and returns the stream's metadata after. The
	// rows kept keep their seqs, the low seq moves to the first of them, and
	// the keys of the rows removed may be appended again. An unknown stream is
	// ErrNotFound.
	Trim(ctx context.Context, stream string, before time.Time) (Meta, error)

	// Expire removes stream ttl from now, rows appended later included. The
	// ttl must be positive; an unknown stream is ErrNotFound.
	Expire(ctx context.Context, stream string, ttl time.Duration) error

	Close() error
}

Backend stores record streams.

type BackendKind

type BackendKind string

BackendKind names a backend a store's streams can live in.

const (
	BackendKV     BackendKind = "kv"
	BackendSQLite BackendKind = "sqlite"
	BackendNDJSON BackendKind = "ndjson"
)

type ImportRequest

type ImportRequest struct {
	Source Meta
	First  int64
	Rows   []Row
}

ImportRequest is one source window copied into an index.

type Index

type Index interface {
	Backend

	// Prepare reconciles the storage for source's kind and returns the indexed
	// incarnation of its stream. A derived index removes an older generation
	// under the same stream id and reports it absent.
	Prepare(ctx context.Context, source Meta) (indexed Meta, found bool, err error)

	// Import stores request.Rows under their source seqs. First must be the seq
	// after the indexed stream's high seq, or for a stream the index does not
	// hold yet the source's low seq; a gap or different generation fails.
	Import(ctx context.Context, request ImportRequest) (Window, error)

	// TrimBelow mirrors a source trim: it drops the indexed rows below lowSeq,
	// and moves an indexed stream that had not reached lowSeq up to it empty.
	TrimBelow(ctx context.Context, stream string, lowSeq int64) (Meta, error)

	// Derived reports whether the index can discard rows and rebuild them from
	// a separate source.
	Derived() bool

	// SetExpiry mirrors the source's absolute expiry. Nil keeps the indexed
	// stream while its source exists.
	SetExpiry(ctx context.Context, stream string, expiresAt *time.Time) error
}

Index is a backend that takes rows under the seqs another backend gave them: what an Indexer mirrors a source into. The sqlite backend is one.

type Indexer

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer keeps an Index caught up with a source backend, one stream at a time and incrementally by seq: each Ensure copies only what the source gained since the last one.

func NewIndexer

func NewIndexer(source Backend, index Index) (*Indexer, error)

NewIndexer mirrors source into index.

func (*Indexer) Ensure

func (i *Indexer) Ensure(ctx context.Context, stream string) error

Ensure brings stream's index up to the source's high seq. A stream the source does not have is ErrNotFound. When the source is the index there is nothing to copy, and Ensure only confirms the stream exists.

type KindOptions

type KindOptions struct {
	// Key names a string column whose value identifies a row within a stream.
	// A keyed stream holds each key once: an append skips a row whose key is
	// already stored. Empty leaves the kind unkeyed, where every row appended
	// is stored.
	Key string

	// Retention says how long a stream keeps its rows.
	Retention Retention
}

KindOptions say how a kind's streams store its rows.

type KindSchema

type KindSchema struct {
	Kind    string
	Columns []query.ColumnDef
	Options KindOptions
}

KindSchema is everything a kind declares: its columns and how its streams store them.

func ResolveKind

func ResolveKind(resolver SchemaResolver, kind string) (KindSchema, error)

ResolveKind resolves kind through resolver and checks the schema it returns describes kind, so a backend never stores rows under a schema meant for another kind.

func (KindSchema) RetentionTTL

func (k KindSchema) RetentionTTL(ttl time.Duration) (time.Duration, error)

RetentionTTL is the ttl a stream of schema's kind keeps each row for, or zero for a kind that keeps its stream whole. A kind retaining rows needs a backend with a ttl: without one, nothing would ever leave the stream.

func (KindSchema) RowKeys

func (k KindSchema) RowKeys(rows []Row) ([]string, error)

RowKeys reads the key of every row of an append to a stream of schema's kind, in row order, or nil for an unkeyed kind. A row without a non-empty string key, or two rows with the same key, refuse the whole append.

func (KindSchema) Validate

func (k KindSchema) Validate() error

Validate refuses a schema no backend could store as declared.

type Meta

type Meta struct {
	Stream     string `json:"stream"`
	Kind       string `json:"kind"`
	Generation string `json:"generation"`

	// Total is how many rows the stream holds, LowSeq the seq of the first one
	// and HighSeq the seq of the last. Rows only leave a stream from its low end
	// (Trim), so Total is HighSeq-LowSeq+1 in a durable backend; all three are
	// reported because an index mirroring it may lag. An empty stream has
	// LowSeq HighSeq+1.
	Total   int64 `json:"total"`
	LowSeq  int64 `json:"lowSeq"`
	HighSeq int64 `json:"highSeq"`

	UpdatedAt time.Time `json:"updatedAt"`

	// ExpiresAt is when the stream is removed, or nil when it is kept until
	// something expires it.
	ExpiresAt *time.Time `json:"expiresAt,omitempty"`

	// Capped reports that an append was refused for capacity: the stream is
	// complete up to HighSeq and missing whatever that append carried.
	Capped bool `json:"capped,omitempty"`
}

Meta describes a stream.

func NewStreamMeta

func NewStreamMeta(stream, kind string, now time.Time) Meta

NewStreamMeta starts one incarnation of stream. Generation distinguishes a stream id reused after expiry from the rows an index previously held for it.

func (Meta) Expired

func (m Meta) Expired(now time.Time) bool

Expired reports whether the stream's expiry has passed at now.

func (Meta) Validate

func (m Meta) Validate() error

Validate refuses metadata that cannot identify one stream incarnation.

type Notifier

type Notifier struct {
	// contains filtered or unexported fields
}

Notifier is a Backend that wakes the readers waiting on a stream as soon as an append to it commits, which is what lets a reader follow a stream rather than poll it. It delegates every call to the backend it wraps.

Only appends made through the Notifier wake a waiter at once: a stream has one writer process (see the package documentation), so that writer appending through the Notifier is the whole of the contract. Anything else is seen at the next recheck.

func NewNotifier

func NewNotifier(backend Backend, options NotifierOptions) (*Notifier, error)

NewNotifier wraps backend.

func (*Notifier) Append

func (n *Notifier) Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)

Append appends through the wrapped backend and, once the append committed, wakes every waiter on stream.

func (*Notifier) Close

func (n *Notifier) Close() error

func (*Notifier) Expire

func (n *Notifier) Expire(ctx context.Context, stream string, ttl time.Duration) error

Expire sets the expiry through the wrapped backend and wakes the stream's waiters.

func (*Notifier) Meta

func (n *Notifier) Meta(ctx context.Context, stream string) (Meta, error)

func (*Notifier) Scan

func (n *Notifier) Scan(ctx context.Context, stream string, afterSeq int64, fn func(seq int64, row Row) error) error

func (*Notifier) Tail

func (n *Notifier) Tail(ctx context.Context, stream string, afterSeq int64, fn func(seq int64, row Row) error) error

Tail calls fn with every row of stream after afterSeq, in seq order, and then with every row appended after that as it is appended. It returns nil once ctx ends, fn's first error, and an ErrNotFound error when the stream does not exist or stops existing — expired, removed, or recreated as a new generation.

func (*Notifier) Trim

func (n *Notifier) Trim(ctx context.Context, stream string, before time.Time) (Meta, error)

Trim trims through the wrapped backend and wakes the stream's waiters, which re-read what is left.

func (*Notifier) Unwrap

func (n *Notifier) Unwrap() Backend

Unwrap is the backend the Notifier delegates to.

func (*Notifier) Wait

func (n *Notifier) Wait(ctx context.Context, stream string, afterSeq int64, generation string) (Meta, error)

Wait blocks until stream holds a row after afterSeq and returns its metadata. generation is the incarnation of the stream the caller has read: a stream that is gone, or that was recreated under another generation since, is ErrNotFound, because the seqs the caller holds no longer name its rows. A cancelled ctx returns ctx.Err().

The metadata is read after subscribing to the next append, so an append that commits between the read and the wait still wakes it.

type NotifierOptions

type NotifierOptions struct {
	// RecheckInterval is how often a waiter re-reads its stream's metadata
	// while no append wakes it. Appends made through the Notifier wake waiters
	// at once; the recheck is what notices what no append announces — a stream
	// expired, trimmed away or removed by the backend itself. Required.
	RecheckInterval time.Duration
}

NotifierOptions configure NewNotifier.

type Retention

type Retention int

Retention says how long a stream keeps its rows.

const (
	// RetainStream keeps a stream whole until it expires, the backend ttl after
	// its first append unless Expire moves it.
	RetainStream Retention = iota

	// RetainRows keeps each row the backend ttl after its own append: every
	// append slides the stream's expiry to the ttl from now and trims the rows
	// appended longer than the ttl ago. It is for a stream that accumulates
	// for as long as something writes to it.
	RetainRows
)

func (Retention) String

func (r Retention) String() string

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router is a Backend that resolves, per call, the backend of the route the call's context names, opening it on first use and keeping it — so a backend's per-stream append serialization holds across calls.

A stream id is not qualified by its route: routes are kept apart by each owning its backend, so a stream written on one route is ErrNotFound on every other.

func NewRouter

func NewRouter(options RouterOptions) (*Router, error)

NewRouter routes calls by options.Route to backends options.Open opens.

func (*Router) Append

func (r *Router) Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)

func (*Router) Close

func (r *Router) Close() error

Close closes every route's backend and refuses calls after it.

func (*Router) Expire

func (r *Router) Expire(ctx context.Context, stream string, ttl time.Duration) error

func (*Router) Forget

func (r *Router) Forget(route string, backend Backend) error

Forget drops and closes route's backend, so the next call on the route opens it afresh — for an owner whose store behind the route went away. It does nothing unless the route still holds backend (compared by identity), so an old owner releasing late cannot evict the backend its replacement opened. A call already holding the dropped backend finishes against it.

func (*Router) Meta

func (r *Router) Meta(ctx context.Context, stream string) (Meta, error)

func (*Router) Scan

func (r *Router) Scan(ctx context.Context, stream string, afterSeq int64, fn func(int64, Row) error) error

func (*Router) Trim

func (r *Router) Trim(ctx context.Context, stream string, before time.Time) (Meta, error)

type RouterOptions

type RouterOptions struct {
	// Route names the route a call's context belongs to — the tenant, the
	// environment — and fails for a context that carries none.
	Route func(ctx context.Context) (string, error)

	// Open opens one route's backend. It is called once per route, and again
	// only after Forget drops it. The backend it returns must belong to that
	// route alone: never shared with another route, and never the index a
	// registry reads, or one route's streams become readable through another.
	Open func(ctx context.Context, route string) (Backend, error)
}

RouterOptions configure NewRouter.

type Row

type Row = query.Row

Row is one record: a JSON-shaped map keyed by column name.

func DecodeRow

func DecodeRow(encoded []byte) (Row, error)

DecodeRow reads one JSON object as a row, keeping numbers as json.Number.

func EncodeRow

func EncodeRow(item any) (Row, error)

EncodeRow is item's JSON object as a row. Numbers are kept as json.Number so an int64 larger than a float64 can hold survives the trip.

func Unstored

func Unstored(rows []Row, keys []string, stored func(key string) bool) (kept []Row, keptKeys []string, skipped int64)

Unstored keeps the rows of an append whose key stored does not report as already in the stream, with their keys, and counts the rows it skipped. With no keys (an unkeyed kind) every row is kept.

type SchemaResolver

type SchemaResolver func(kind string) (KindSchema, error)

SchemaResolver resolves a kind to its schema, and refuses a kind nothing declared. Schemas.Kind is one.

type Schemas

type Schemas struct {
	// contains filtered or unexported fields
}

Schemas is the catalog of kinds and their columns. A backend resolves a kind through it — to store rows by column (sqlite), to find a kind's key and retention (every backend) — and a typed result registry fills it, so the two agree without either one owning the other.

func NewSchemas

func NewSchemas() *Schemas

NewSchemas returns an empty catalog.

func (*Schemas) Kind

func (s *Schemas) Kind(kind string) (KindSchema, error)

Kind resolves kind, and refuses a kind nothing declared.

func (*Schemas) Register

func (s *Schemas) Register(kind string, columns []query.ColumnDef, options KindOptions) error

Register declares kind's columns and options. A kind is declared once: a second declaration with different columns or options would leave rows already written under the first unreadable or wrongly deduplicated, so it is an error rather than a replacement.

type Settings

type Settings struct {
	// Prefix is the property prefix the settings were read under; errors about
	// a setting name the key under it.
	Prefix string

	// Backend is the backend asked for, or empty for the caller's default.
	Backend           BackendKind
	Dir               string
	TTL               time.Duration
	NDJSONMaxBytes    int64
	NDJSONKeepStreams int
}

Settings say where a store's streams live and for how long. A store reads them from commons/properties (-P, env, a properties file) under a prefix of its own:

<prefix>.backend             kv | sqlite | ndjson; unset lets Resolve pick
<prefix>.dir                 where sqlite files and ndjson streams live
<prefix>.ttl                 how long a stream is kept, e.g. 30d or 36h
<prefix>.ndjson.maxBytes     one ndjson stream's cap, e.g. 256MiB
<prefix>.ndjson.keepStreams  ndjson streams kept per kind

func ReadSettings

func ReadSettings(prefix string, defaults Settings) (Settings, error)

ReadSettings reads prefix's properties over defaults. A key that is unset keeps its default; a key that is set but unusable is an error naming it, never a silent fallback to the default.

func (Settings) Resolve

func (s Settings) Resolve(hasKV bool, fallback BackendKind) (BackendKind, error)

Resolve is the backend a store writes to. An explicit backend is used as asked; without one, kv wherever the caller has a kv store — the store every process can share — and fallback, a local file, where it has none.

hasKV is a fact about one caller, which may differ per tenant, so it is resolved per call. An explicit kv where there is no kv store is an error: writing somewhere else would lose the one property kv was asked for.

type StreamLocks

type StreamLocks struct {
	// contains filtered or unexported fields
}

StreamLocks serializes work on one stream while leaving other streams free, which is how a backend keeps the single-writer contract inside one process. The zero value is ready to use. A lock nobody holds or waits on is dropped, so the set does not grow with every stream ever written.

func (*StreamLocks) Lock

func (l *StreamLocks) Lock(stream string) (unlock func())

Lock blocks until stream is free and returns the function that frees it.

func (*StreamLocks) TryLock

func (l *StreamLocks) TryLock(stream string) (unlock func(), ok bool)

TryLock takes stream only when it is free, for work that should leave a stream in use alone rather than wait for it.

type Window

type Window struct {
	From int64 `json:"from"`
	To   int64 `json:"to"`
}

Window is an inclusive seq range. An empty window has From == To+1: an append of no rows reports the seq the next row will take.

func (Window) Len

func (w Window) Len() int64

Len is the number of seqs the window spans.

Directories

Path Synopsis
Package kv stores record streams in a clicky cache.Store, so one implementation runs in process (cache.NewMemory) and against valkey/redis (clicky/valkey.NewStore) — the backend a CLI writes to and a server reads.
Package kv stores record streams in a clicky cache.Store, so one implementation runs in process (cache.NewMemory) and against valkey/redis (clicky/valkey.NewStore) — the backend a CLI writes to and a server reads.
Package ndjson stores each record stream as a local file: one line per row in <dir>/<kind>/<stream>.ndjson, written {"seq":N,"row":{…}}, beside a <stream>.meta.json sidecar holding the stream's recordstore.Meta.
Package ndjson stores each record stream as a local file: one line per row in <dir>/<kind>/<stream>.ndjson, written {"seq":N,"row":{…}}, beside a <stream>.meta.json sidecar holding the stream's recordstore.Meta.
Package recordstoretest is the conformance suite every recordstore.Backend runs, so the kv, sqlite and ndjson backends can never disagree about what appending, scanning, expiring and describing a stream mean.
Package recordstoretest is the conformance suite every recordstore.Backend runs, so the kv, sqlite and ndjson backends can never disagree about what appending, scanning, expiring and describing a stream mean.
Package sqlite stores record streams in a SQLite file: a table per kind, keyed (stream_id, seq), with the kind's columns typed from its schema, and a record_streams table describing every stream.
Package sqlite stores record streams in a SQLite file: a table per kind, keyed (stream_id, seq), with the kind's columns typed from its schema, and a record_streams table describing every stream.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL