storage

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package storage persists what a resolver learns: the query log, per-device statistics, and whatever else a deployment wants to keep across a restart.

It defines the interfaces and ships one implementation that needs nothing — an in-memory store — because the engine must remain useful to an embedder who wants no persistence at all, and because a durable store is a dependency this module cannot take.

Why SQLite is not here

The roadmap says SQLite, and SQLite is the right default for the daemon. It is not the right thing for THIS module: every Go driver for it is either cgo, which would end the CGO_ENABLED=0 static binary and cross-compilation to four platforms, or a pure-Go reimplementation that is a large dependency to place in the graph of every embedder — including the ones that wanted a DNS library and no database at all.

So the boundary is here and the driver is downstream. Store is what the engine writes through, Memory is what it does when nobody supplies anything, and the daemon — a separate module, free to take dependencies for exactly this reason — provides the SQLite implementation. This is what ADR 0015 bought, and the first place the engine actually spends it.

Writing must never block a query

A query log is written on the query path, which is the one place a resolver cannot afford to wait. A disk that has filled, an fsync that stalls, a table lock held by a compaction — none of them may turn into DNS latency, because a resolver that answers slowly is indistinguishable from one that is down.

So every write is asynchronous and bounded. Recorder buffers entries and flushes them in batches; past the bound it DROPS, and counts what it dropped. Losing log entries under pressure is a small, visible loss. Adding latency to every query to avoid it is a large, invisible one.

Retention is a promise, not a suggestion

A query log is a record of every name every person on a network looked up. It is the most sensitive thing this software touches, and an unbounded one is a liability that grows. RetentionPolicy is therefore part of the store's configuration rather than an operational afterthought, the default is finite, and a store is required to enforce it without being asked.

Index

Constants

View Source
const (
	// DefaultBuffer is how many entries may be waiting to be written. At ten
	// thousand queries a second this is a fifth of a second of slack, which is
	// far longer than a healthy write and far shorter than an unhealthy one —
	// which is the shape you want: absorb a hiccup, notice an outage.
	DefaultBuffer = 2048

	// DefaultBatch is how many entries one write carries. Batching is most of
	// the value of writing asynchronously at all: a per-query round trip to any
	// durable store costs more than the query did.
	DefaultBatch = 256

	// DefaultFlushInterval bounds how long an entry waits when traffic is too
	// light to fill a batch. A query log that only appears once the network is
	// busy is one nobody can debug a quiet problem with.
	DefaultFlushInterval = time.Second
)

Defaults for RecorderOptions.

View Source
const CloseTimeout = 15 * time.Second

CloseTimeout bounds how long Recorder.Close waits for the writer goroutine.

It is longer than the write deadline in [Recorder.write], so a store that honours its context always finishes on its own and this never fires.

View Source
const DefaultLimit = 100

DefaultLimit bounds an unqualified read.

Variables

View Source
var DefaultRetention = RetentionPolicy{
	MaxAge:     7 * 24 * time.Hour,
	MaxEntries: 1_000_000,
}

DefaultRetention keeps a week or a million entries, whichever comes first.

A week is long enough to answer "what did this device do on Saturday", which is the question a query log is usually opened for, and short enough that the record of a household's browsing does not accumulate indefinitely because nobody chose a value.

Functions

This section is empty.

Types

type Filter

type Filter struct {
	Since, Until time.Time
	Device       string
	// Name matches the entry's name exactly, canonically.
	Name string
	// Blocked, when set, selects only blocked or only unblocked entries.
	Blocked *bool
	// Limit caps the rows returned. Zero selects [DefaultLimit]; a query log
	// has no natural size and an unbounded read of one is how a management API
	// runs a server out of memory.
	Limit int
	// Offset skips rows, for paging.
	Offset int
}

Filter selects entries to read back.

Every field is optional and they combine with AND. It is deliberately narrow: a query log is read to answer "what did this device ask for", "who asked for this name" and "what got blocked", and a general query language would be a larger surface for every implementation to get right.

type Memory

type Memory struct {
	// contains filtered or unexported fields
}

Memory is a Store that keeps entries in a ring buffer.

It exists so that the engine is useful to an embedder who wants a query log and no database, and so that everything above Store can be tested without one. It is not a lesser implementation with a real one to come: for a household resolver keeping a day of history, a bounded ring in memory is the correct amount of machinery, and a durable store is what you reach for when the history must survive a restart.

It is safe for concurrent use. Reads take a read lock and copy, so a caller walking a month of history cannot be handed entries that a concurrent append is overwriting.

func NewMemory

func NewMemory(opts MemoryOptions) (*Memory, error)

NewMemory returns an in-memory store.

func (*Memory) Append

func (m *Memory) Append(_ context.Context, entries ...QueryEntry) error

Append implements Store.

func (*Memory) Close

func (m *Memory) Close() error

Close implements Store. Nothing is released; the entries go when the store does.

func (*Memory) Count

func (m *Memory) Count(_ context.Context, f Filter) (uint64, error)

Count implements Store.

func (*Memory) Len

func (m *Memory) Len() int

Len reports how many entries are held.

func (*Memory) Prune

func (m *Memory) Prune(context.Context) (int, error)

Prune implements Store.

func (*Memory) Query

func (m *Memory) Query(_ context.Context, f Filter) ([]QueryEntry, error)

Query implements Store, newest first.

func (*Memory) TopDevices

func (m *Memory) TopDevices(_ context.Context, f Filter, n int) ([]Stat, error)

TopDevices implements Store.

func (*Memory) TopNames

func (m *Memory) TopNames(_ context.Context, f Filter, n int) ([]Stat, error)

TopNames implements Store.

type MemoryOptions

type MemoryOptions struct {
	// Retention bounds what is kept. The zero policy selects
	// [DefaultRetention]; keeping everything is [Unlimited], which has to be
	// said rather than fallen into — see [RetentionPolicy].
	Retention RetentionPolicy `json:"retention"`
	Clock     clock.Clock     `json:"-"`
}

MemoryOptions configure a Memory store.

type QueryEntry

type QueryEntry struct {
	// At is when the query was answered.
	At time.Time `json:"at"`
	// Device and Client identify who asked. Device is the stable identifier
	// something above the resolver assigned; Client is the address observed.
	Device string `json:"device,omitempty"`
	Client string `json:"client,omitempty"`
	// Name, Type and Class are the question, with Name canonical.
	Name  string `json:"name"`
	Type  string `json:"type"`
	Class string `json:"class,omitempty"`
	// RCode is the answer's response code, as a mnemonic rather than a number
	// so that a log stays readable without a lookup table.
	RCode string `json:"rcode"`
	// Source says where the answer came from: cache, upstream, stale, local.
	Source string `json:"source"`
	// Upstream names the provider that answered, empty when none was asked.
	Upstream string `json:"upstream,omitempty"`
	// Blocked reports a policy answer, and Rule the rule responsible.
	Blocked bool   `json:"blocked,omitempty"`
	Rule    string `json:"rule,omitempty"`
	// Duration is the whole resolution.
	Duration time.Duration `json:"duration"`
	// Answers is how many records came back, which is what distinguishes an
	// empty NOERROR from a real one at a glance.
	Answers int `json:"answers"`
}

QueryEntry is one resolved query, as recorded.

It is a flat value with no pointers into the message it came from, because the entry outlives the query by design: it sits in a buffer, is written in a batch, and may be read back hours later. Holding a *dnsmsg.Message here would pin every record of every answer for as long as the buffer lived.

func EntryFor

func EntryFor(at time.Time, q dnsmsg.Question, rcode dnsmsg.RCode, answers int) QueryEntry

EntryFor builds a QueryEntry from the pieces the resolver has.

It lives here rather than in the resolver so that every store sees the same shape, and so the canonicalisation happens once: the name is lower-cased, because a log searched for "example.com" must find the query that arrived as "ExAmPle.CoM" under DNS-0x20 or from a client that shouts.

func (QueryEntry) String

func (e QueryEntry) String() string

String renders an entry in a form a person can read in a terminal.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder writes query entries without ever blocking the query that produced them.

This is the whole reason it exists. A query log is written on the one path a resolver cannot afford to wait on: a disk that has filled, an fsync that stalls, a lock held by a compaction, a network store that has gone away — none of them may become DNS latency, because a resolver that answers slowly is indistinguishable to a user from one that is down.

So Recorder.Record is a non-blocking send onto a bounded channel and nothing else. Past the bound it drops and counts. Losing log entries under pressure is a small loss that shows up in Recorder.Dropped; adding a millisecond to every query to avoid it is a large loss that shows up nowhere.

func NewRecorder

func NewRecorder(opts RecorderOptions) (*Recorder, error)

NewRecorder starts a Recorder writing to opts.Store.

func (*Recorder) Close

func (r *Recorder) Close() error

Close flushes what is queued and stops the writer. It is idempotent, and every caller — not only the first — returns once the writer goroutine has actually exited or CloseTimeout has passed.

Both halves of that matter. Two goroutines closing at once is the ordinary shape of a careful shutdown (an explicit call plus a defer), and a second caller told the writer had exited when it had not may go on to tear down the store the writer is still inside.

The timeout is the other half. A Store that ignores the context it is given can park the writer inside Append forever, and an unbounded wait here turns that into a daemon that never exits and is eventually killed — losing the flush this method exists to perform. After the timeout Close returns an error naming the problem and abandons the writer, which is a leaked goroutine during shutdown and strictly better than not shutting down.

func (*Recorder) Dropped

func (r *Recorder) Dropped() uint64

Dropped is shorthand for the number of entries lost to a full buffer.

func (*Recorder) Record

func (r *Recorder) Record(e QueryEntry)

Record queues one entry. It never blocks and never fails.

It returns nothing, and that is deliberate: there is no useful thing a caller on the query path could do with an error from the query log, and offering one would invite somebody to check it and, eventually, to wait on it.

func (*Recorder) Stats

func (r *Recorder) Stats() RecorderStats

Stats returns a snapshot.

type RecorderOptions

type RecorderOptions struct {
	// Store receives the batches. Required.
	Store Store
	// Buffer, Batch and FlushInterval take the defaults above when zero.
	Buffer        int           `json:"buffer"`
	Batch         int           `json:"batch"`
	FlushInterval time.Duration `json:"flush_interval"`

	Clock  clock.Clock  `json:"-"`
	Logger *slog.Logger `json:"-"`
}

RecorderOptions configure a Recorder.

type RecorderStats

type RecorderStats struct {
	Recorded uint64 `json:"recorded"`
	Written  uint64 `json:"written"`
	// Dropped is entries refused because the buffer was full. Non-zero means
	// the store is slower than the query rate.
	Dropped uint64 `json:"dropped"`
	// Failed is entries the store rejected.
	Failed uint64 `json:"failed"`
	// Pending is entries queued but not yet written.
	Pending int `json:"pending"`
}

RecorderStats reports what the recorder has done.

type RetentionPolicy

type RetentionPolicy struct {
	// MaxAge discards entries older than this. Zero means no age limit.
	MaxAge time.Duration `json:"max_age"`
	// MaxEntries discards the oldest beyond this count. Zero means no count
	// limit.
	MaxEntries int `json:"max_entries"`
	// contains filtered or unexported fields
}

RetentionPolicy bounds what a store keeps.

Both limits apply; whichever bites first wins. A policy with neither set is rejected by RetentionPolicy.Validate rather than treated as "keep everything": a query log records every name every person on a network looked up, and an unbounded one is a liability that grows. Choosing to keep everything must be a thing somebody typed.

func Unlimited

func Unlimited() RetentionPolicy

Unlimited is the retention policy that keeps everything, for a caller who has decided to.

It exists so that "keep everything" is a thing somebody wrote down. The zero RetentionPolicy is refused precisely so it cannot be arrived at by omission.

func (RetentionPolicy) MarshalJSON

func (p RetentionPolicy) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*RetentionPolicy) UnmarshalJSON

func (p *RetentionPolicy) UnmarshalJSON(b []byte) error

UnmarshalJSON implements json.Unmarshaler.

func (RetentionPolicy) Validate

func (p RetentionPolicy) Validate() error

Validate reports every problem with p.

type Stat

type Stat struct {
	Key   string `json:"key"`
	Count uint64 `json:"count"`
}

Stat is an aggregate over the log.

type Store

type Store interface {
	// Append records entries. It may buffer, and it may return before anything
	// is durable — see [Recorder], which is what the engine actually writes
	// through and which never blocks a query.
	Append(ctx context.Context, entries ...QueryEntry) error

	// Query reads entries matching f, newest first.
	Query(ctx context.Context, f Filter) ([]QueryEntry, error)

	// Count reports how many entries match f, ignoring Limit and Offset.
	Count(ctx context.Context, f Filter) (uint64, error)

	// TopNames and TopDevices are the two aggregates every DNS dashboard shows.
	// They are named methods rather than a general GROUP BY because a store
	// backed by a real database can answer them with an index, and one backed
	// by a map can answer them without inventing a query planner.
	TopNames(ctx context.Context, f Filter, n int) ([]Stat, error)
	TopDevices(ctx context.Context, f Filter, n int) ([]Stat, error)

	// Prune deletes entries outside the retention policy and reports how many
	// went. A store enforces retention itself; this exists so a caller can
	// force it and learn the result.
	Prune(ctx context.Context) (int, error)

	// Close releases the store. It is idempotent.
	Close() error
}

Store persists query entries and answers questions about them.

Implementations must be safe for concurrent use. Every method takes a context: a store may be a network round trip away, and a management API asking for a month of history must be cancellable.

func Nop

func Nop() Store

Nop returns a Store that discards everything and answers nothing.

It is what a deployment that wants no query log uses, and it is a real choice rather than a degenerate one: a resolver that keeps no record of what anyone looked up is the most private configuration available, and the engine should not make it awkward to pick.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL