Documentation
¶
Overview ¶
Package recordstore keeps append-only record streams: the rows a long capture produces, written as they arrive and read back by position instead of being carried inside whatever reported the capture.
A stream is named by a stream id and holds rows of one kind. Every row gets a per-stream seq, contiguous from 1, which is the only position a reader resumes from — never a timestamp or a key, which neither order nor identify a row's position reliably. A kind may still declare a key, and then a stream holds each key once, which is what makes re-ingesting an overlapping source window idempotent. Rows leave a stream only from its low end: the whole stream expires, or Trim removes the oldest appends.
Backends live in subpackages: kv (a clicky cache.Store: in-process memory or valkey/redis), sqlite (a file that is also the query index a profile reads), and ndjson (a file per stream). An Indexer mirrors any of them into a sqlite index incrementally, which is what makes a stream written by one process pageable through another's query engine.
A stream has one writer at a time. Backends serialize appends made within one process; two processes appending to one stream is outside the contract.
Index ¶
- Variables
- func ValidateAppend(stream, kind string) error
- func ValidateKind(kind string) error
- func ValidateStream(stream string) error
- func ValidateTTL(ttl time.Duration) error
- type AppendResult
- type Backend
- type BackendKind
- type ImportRequest
- type Index
- type Indexer
- type KindOptions
- type KindSchema
- type Meta
- type Notifier
- func (n *Notifier) Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)
- func (n *Notifier) Close() error
- func (n *Notifier) Expire(ctx context.Context, stream string, ttl time.Duration) error
- func (n *Notifier) Meta(ctx context.Context, stream string) (Meta, error)
- func (n *Notifier) Scan(ctx context.Context, stream string, afterSeq int64, ...) error
- func (n *Notifier) Tail(ctx context.Context, stream string, afterSeq int64, ...) error
- func (n *Notifier) Trim(ctx context.Context, stream string, before time.Time) (Meta, error)
- func (n *Notifier) Unwrap() Backend
- func (n *Notifier) Wait(ctx context.Context, stream string, afterSeq int64, generation string) (Meta, error)
- type NotifierOptions
- type Retention
- type Router
- func (r *Router) Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)
- func (r *Router) Close() error
- func (r *Router) Expire(ctx context.Context, stream string, ttl time.Duration) error
- func (r *Router) Forget(route string, backend Backend) error
- func (r *Router) Meta(ctx context.Context, stream string) (Meta, error)
- func (r *Router) Scan(ctx context.Context, stream string, afterSeq int64, fn func(int64, Row) error) error
- func (r *Router) Trim(ctx context.Context, stream string, before time.Time) (Meta, error)
- type RouterOptions
- type Row
- type SchemaResolver
- type Schemas
- type Settings
- type StreamLocks
- type Window
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotFound reports a stream that does not exist, or no longer does. ErrNotFound = errors.New("record stream not found") // ErrCapacity reports an append refused because the stream, or one row of // it, would exceed what the backend was configured to hold. The rows of a // refused append are not written — none of them. ErrCapacity = errors.New("record stream capacity exceeded") )
Functions ¶
func ValidateAppend ¶
ValidateAppend checks an append's stream id and kind before a backend writes anything.
func ValidateKind ¶
ValidateKind rejects a kind that could not name a table, a directory and a profile.
func ValidateStream ¶
ValidateStream rejects a stream id a backend could not key by. The alphabet is narrow on purpose: an id becomes a Redis key segment, a file name and a query parameter, and every character outside it is a character one of those would have to escape.
func ValidateTTL ¶
ValidateTTL rejects an expiry that is not in the future.
Types ¶
type AppendResult ¶
AppendResult is what one append stored: the window its rows were numbered into, and how many rows it skipped because their key was already stored.
func AppendTyped ¶
func AppendTyped[T any](ctx context.Context, backend Backend, stream, kind string, items []T) (AppendResult, error)
AppendTyped appends items as rows through their JSON encoding, which is the shape every backend stores and every reader sees.
type Backend ¶
type Backend interface {
// Append adds rows to stream, creating it under kind when it does not
// exist, and returns the window the rows were numbered into. Appending to
// an existing stream under a different kind is an error.
//
// A keyed kind (KindOptions.Key) skips every row whose key the stream
// already holds, atomically with the append, and numbers only the rows it
// keeps; a batch naming one key twice is refused whole. A kind retaining
// rows (RetainRows) slides the stream's expiry to the backend ttl from now
// and trims the rows appended longer than that ago, in the same write.
Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)
// Meta describes stream, or returns ErrNotFound.
Meta(ctx context.Context, stream string) (Meta, error)
// Scan calls fn with every row after afterSeq, in seq order, stopping at
// the first error fn returns. A scan from below the stream's low seq starts
// at the low seq. An unknown stream is ErrNotFound.
Scan(ctx context.Context, stream string, afterSeq int64, fn func(seq int64, row Row) error) error
// Trim removes the rows appended before before — every append up to the
// last one made before it — and returns the stream's metadata after. The
// rows kept keep their seqs, the low seq moves to the first of them, and
// the keys of the rows removed may be appended again. An unknown stream is
// ErrNotFound.
Trim(ctx context.Context, stream string, before time.Time) (Meta, error)
// Expire removes stream ttl from now, rows appended later included. The
// ttl must be positive; an unknown stream is ErrNotFound.
Expire(ctx context.Context, stream string, ttl time.Duration) error
Close() error
}
Backend stores record streams.
type BackendKind ¶
type BackendKind string
BackendKind names a backend a store's streams can live in.
const ( BackendKV BackendKind = "kv" BackendSQLite BackendKind = "sqlite" BackendNDJSON BackendKind = "ndjson" )
type ImportRequest ¶
ImportRequest is one source window copied into an index.
type Index ¶
type Index interface {
Backend
// Prepare reconciles the storage for source's kind and returns the indexed
// incarnation of its stream. A derived index removes an older generation
// under the same stream id and reports it absent.
Prepare(ctx context.Context, source Meta) (indexed Meta, found bool, err error)
// Import stores request.Rows under their source seqs. First must be the seq
// after the indexed stream's high seq, or for a stream the index does not
// hold yet the source's low seq; a gap or different generation fails.
Import(ctx context.Context, request ImportRequest) (Window, error)
// TrimBelow mirrors a source trim: it drops the indexed rows below lowSeq,
// and moves an indexed stream that had not reached lowSeq up to it empty.
TrimBelow(ctx context.Context, stream string, lowSeq int64) (Meta, error)
// Derived reports whether the index can discard rows and rebuild them from
// a separate source.
Derived() bool
// SetExpiry mirrors the source's absolute expiry. Nil keeps the indexed
// stream while its source exists.
SetExpiry(ctx context.Context, stream string, expiresAt *time.Time) error
}
Index is a backend that takes rows under the seqs another backend gave them: what an Indexer mirrors a source into. The sqlite backend is one.
type Indexer ¶
type Indexer struct {
// contains filtered or unexported fields
}
Indexer keeps an Index caught up with a source backend, one stream at a time and incrementally by seq: each Ensure copies only what the source gained since the last one.
func NewIndexer ¶
NewIndexer mirrors source into index.
type KindOptions ¶
type KindOptions struct {
// Key names a string column whose value identifies a row within a stream.
// A keyed stream holds each key once: an append skips a row whose key is
// already stored. Empty leaves the kind unkeyed, where every row appended
// is stored.
Key string
// Retention says how long a stream keeps its rows.
Retention Retention
}
KindOptions say how a kind's streams store its rows.
type KindSchema ¶
type KindSchema struct {
Kind string
Columns []query.ColumnDef
Options KindOptions
}
KindSchema is everything a kind declares: its columns and how its streams store them.
func ResolveKind ¶
func ResolveKind(resolver SchemaResolver, kind string) (KindSchema, error)
ResolveKind resolves kind through resolver and checks the schema it returns describes kind, so a backend never stores rows under a schema meant for another kind.
func (KindSchema) RetentionTTL ¶
RetentionTTL is the ttl a stream of schema's kind keeps each row for, or zero for a kind that keeps its stream whole. A kind retaining rows needs a backend with a ttl: without one, nothing would ever leave the stream.
func (KindSchema) RowKeys ¶
func (k KindSchema) RowKeys(rows []Row) ([]string, error)
RowKeys reads the key of every row of an append to a stream of schema's kind, in row order, or nil for an unkeyed kind. A row without a non-empty string key, or two rows with the same key, refuse the whole append.
func (KindSchema) Validate ¶
func (k KindSchema) Validate() error
Validate refuses a schema no backend could store as declared.
type Meta ¶
type Meta struct {
Stream string `json:"stream"`
Kind string `json:"kind"`
Generation string `json:"generation"`
// Total is how many rows the stream holds, LowSeq the seq of the first one
// and HighSeq the seq of the last. Rows only leave a stream from its low end
// (Trim), so Total is HighSeq-LowSeq+1 in a durable backend; all three are
// reported because an index mirroring it may lag. An empty stream has
// LowSeq HighSeq+1.
Total int64 `json:"total"`
LowSeq int64 `json:"lowSeq"`
HighSeq int64 `json:"highSeq"`
UpdatedAt time.Time `json:"updatedAt"`
// ExpiresAt is when the stream is removed, or nil when it is kept until
// something expires it.
ExpiresAt *time.Time `json:"expiresAt,omitempty"`
// Capped reports that an append was refused for capacity: the stream is
// complete up to HighSeq and missing whatever that append carried.
Capped bool `json:"capped,omitempty"`
}
Meta describes a stream.
func NewStreamMeta ¶
NewStreamMeta starts one incarnation of stream. Generation distinguishes a stream id reused after expiry from the rows an index previously held for it.
type Notifier ¶
type Notifier struct {
// contains filtered or unexported fields
}
Notifier is a Backend that wakes the readers waiting on a stream as soon as an append to it commits, which is what lets a reader follow a stream rather than poll it. It delegates every call to the backend it wraps.
Only appends made through the Notifier wake a waiter at once: a stream has one writer process (see the package documentation), so that writer appending through the Notifier is the whole of the contract. Anything else is seen at the next recheck.
func NewNotifier ¶
func NewNotifier(backend Backend, options NotifierOptions) (*Notifier, error)
NewNotifier wraps backend.
func (*Notifier) Append ¶
func (n *Notifier) Append(ctx context.Context, stream, kind string, rows []Row) (AppendResult, error)
Append appends through the wrapped backend and, once the append committed, wakes every waiter on stream.
func (*Notifier) Expire ¶
Expire sets the expiry through the wrapped backend and wakes the stream's waiters.
func (*Notifier) Tail ¶
func (n *Notifier) Tail(ctx context.Context, stream string, afterSeq int64, fn func(seq int64, row Row) error) error
Tail calls fn with every row of stream after afterSeq, in seq order, and then with every row appended after that as it is appended. It returns nil once ctx ends, fn's first error, and an ErrNotFound error when the stream does not exist or stops existing — expired, removed, or recreated as a new generation.
func (*Notifier) Trim ¶
Trim trims through the wrapped backend and wakes the stream's waiters, which re-read what is left.
func (*Notifier) Wait ¶
func (n *Notifier) Wait(ctx context.Context, stream string, afterSeq int64, generation string) (Meta, error)
Wait blocks until stream holds a row after afterSeq and returns its metadata. generation is the incarnation of the stream the caller has read: a stream that is gone, or that was recreated under another generation since, is ErrNotFound, because the seqs the caller holds no longer name its rows. A cancelled ctx returns ctx.Err().
The metadata is read after subscribing to the next append, so an append that commits between the read and the wait still wakes it.
type NotifierOptions ¶
type NotifierOptions struct {
// RecheckInterval is how often a waiter re-reads its stream's metadata
// while no append wakes it. Appends made through the Notifier wake waiters
// at once; the recheck is what notices what no append announces — a stream
// expired, trimmed away or removed by the backend itself. Required.
RecheckInterval time.Duration
}
NotifierOptions configure NewNotifier.
type Retention ¶
type Retention int
Retention says how long a stream keeps its rows.
const ( // RetainStream keeps a stream whole until it expires, the backend ttl after // its first append unless Expire moves it. RetainStream Retention = iota // RetainRows keeps each row the backend ttl after its own append: every // append slides the stream's expiry to the ttl from now and trims the rows // appended longer than the ttl ago. It is for a stream that accumulates // for as long as something writes to it. RetainRows )
type Router ¶
type Router struct {
// contains filtered or unexported fields
}
Router is a Backend that resolves, per call, the backend of the route the call's context names, opening it on first use and keeping it — so a backend's per-stream append serialization holds across calls.
A stream id is not qualified by its route: routes are kept apart by each owning its backend, so a stream written on one route is ErrNotFound on every other.
func NewRouter ¶
func NewRouter(options RouterOptions) (*Router, error)
NewRouter routes calls by options.Route to backends options.Open opens.
func (*Router) Forget ¶
Forget drops and closes route's backend, so the next call on the route opens it afresh — for an owner whose store behind the route went away. It does nothing unless the route still holds backend (compared by identity), so an old owner releasing late cannot evict the backend its replacement opened. A call already holding the dropped backend finishes against it.
type RouterOptions ¶
type RouterOptions struct {
// Route names the route a call's context belongs to — the tenant, the
// environment — and fails for a context that carries none.
Route func(ctx context.Context) (string, error)
// Open opens one route's backend. It is called once per route, and again
// only after Forget drops it. The backend it returns must belong to that
// route alone: never shared with another route, and never the index a
// registry reads, or one route's streams become readable through another.
Open func(ctx context.Context, route string) (Backend, error)
}
RouterOptions configure NewRouter.
type Row ¶
Row is one record: a JSON-shaped map keyed by column name.
func EncodeRow ¶
EncodeRow is item's JSON object as a row. Numbers are kept as json.Number so an int64 larger than a float64 can hold survives the trip.
func Unstored ¶
func Unstored(rows []Row, keys []string, stored func(key string) bool) (kept []Row, keptKeys []string, skipped int64)
Unstored keeps the rows of an append whose key stored does not report as already in the stream, with their keys, and counts the rows it skipped. With no keys (an unkeyed kind) every row is kept.
type SchemaResolver ¶
type SchemaResolver func(kind string) (KindSchema, error)
SchemaResolver resolves a kind to its schema, and refuses a kind nothing declared. Schemas.Kind is one.
type Schemas ¶
type Schemas struct {
// contains filtered or unexported fields
}
Schemas is the catalog of kinds and their columns. A backend resolves a kind through it — to store rows by column (sqlite), to find a kind's key and retention (every backend) — and a typed result registry fills it, so the two agree without either one owning the other.
func (*Schemas) Kind ¶
func (s *Schemas) Kind(kind string) (KindSchema, error)
Kind resolves kind, and refuses a kind nothing declared.
type Settings ¶
type Settings struct {
// Prefix is the property prefix the settings were read under; errors about
// a setting name the key under it.
Prefix string
// Backend is the backend asked for, or empty for the caller's default.
Backend BackendKind
Dir string
TTL time.Duration
NDJSONMaxBytes int64
NDJSONKeepStreams int
}
Settings say where a store's streams live and for how long. A store reads them from commons/properties (-P, env, a properties file) under a prefix of its own:
<prefix>.backend kv | sqlite | ndjson; unset lets Resolve pick <prefix>.dir where sqlite files and ndjson streams live <prefix>.ttl how long a stream is kept, e.g. 30d or 36h <prefix>.ndjson.maxBytes one ndjson stream's cap, e.g. 256MiB <prefix>.ndjson.keepStreams ndjson streams kept per kind
func ReadSettings ¶
ReadSettings reads prefix's properties over defaults. A key that is unset keeps its default; a key that is set but unusable is an error naming it, never a silent fallback to the default.
func (Settings) Resolve ¶
func (s Settings) Resolve(hasKV bool, fallback BackendKind) (BackendKind, error)
Resolve is the backend a store writes to. An explicit backend is used as asked; without one, kv wherever the caller has a kv store — the store every process can share — and fallback, a local file, where it has none.
hasKV is a fact about one caller, which may differ per tenant, so it is resolved per call. An explicit kv where there is no kv store is an error: writing somewhere else would lose the one property kv was asked for.
type StreamLocks ¶
type StreamLocks struct {
// contains filtered or unexported fields
}
StreamLocks serializes work on one stream while leaving other streams free, which is how a backend keeps the single-writer contract inside one process. The zero value is ready to use. A lock nobody holds or waits on is dropped, so the set does not grow with every stream ever written.
func (*StreamLocks) Lock ¶
func (l *StreamLocks) Lock(stream string) (unlock func())
Lock blocks until stream is free and returns the function that frees it.
func (*StreamLocks) TryLock ¶
func (l *StreamLocks) TryLock(stream string) (unlock func(), ok bool)
TryLock takes stream only when it is free, for work that should leave a stream in use alone rather than wait for it.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package kv stores record streams in a clicky cache.Store, so one implementation runs in process (cache.NewMemory) and against valkey/redis (clicky/valkey.NewStore) — the backend a CLI writes to and a server reads.
|
Package kv stores record streams in a clicky cache.Store, so one implementation runs in process (cache.NewMemory) and against valkey/redis (clicky/valkey.NewStore) — the backend a CLI writes to and a server reads. |
|
Package ndjson stores each record stream as a local file: one line per row in <dir>/<kind>/<stream>.ndjson, written {"seq":N,"row":{…}}, beside a <stream>.meta.json sidecar holding the stream's recordstore.Meta.
|
Package ndjson stores each record stream as a local file: one line per row in <dir>/<kind>/<stream>.ndjson, written {"seq":N,"row":{…}}, beside a <stream>.meta.json sidecar holding the stream's recordstore.Meta. |
|
Package recordstoretest is the conformance suite every recordstore.Backend runs, so the kv, sqlite and ndjson backends can never disagree about what appending, scanning, expiring and describing a stream mean.
|
Package recordstoretest is the conformance suite every recordstore.Backend runs, so the kv, sqlite and ndjson backends can never disagree about what appending, scanning, expiring and describing a stream mean. |
|
Package sqlite stores record streams in a SQLite file: a table per kind, keyed (stream_id, seq), with the kind's columns typed from its schema, and a record_streams table describing every stream.
|
Package sqlite stores record streams in a SQLite file: a table per kind, keyed (stream_id, seq), with the kind's columns typed from its schema, and a record_streams table describing every stream. |