Documentation
¶
Overview ¶
Package storage persists what a resolver learns: the query log, per-device statistics, and whatever else a deployment wants to keep across a restart.
It defines the interfaces and ships one implementation that needs nothing — an in-memory store — because the engine must remain useful to an embedder who wants no persistence at all, and because a durable store is a dependency this module cannot take.
Why SQLite is not here ¶
The roadmap says SQLite, and SQLite is the right default for the daemon. It is not the right thing for THIS module: every Go driver for it is either cgo, which would end the CGO_ENABLED=0 static binary and cross-compilation to four platforms, or a pure-Go reimplementation that is a large dependency to place in the graph of every embedder — including the ones that wanted a DNS library and no database at all.
So the boundary is here and the driver is downstream. Store is what the engine writes through, Memory is what it does when nobody supplies anything, and the daemon — a separate module, free to take dependencies for exactly this reason — provides the SQLite implementation. This is what ADR 0015 bought, and the first place the engine actually spends it.
Writing must never block a query ¶
A query log is written on the query path, which is the one place a resolver cannot afford to wait. A disk that has filled, an fsync that stalls, a table lock held by a compaction — none of them may turn into DNS latency, because a resolver that answers slowly is indistinguishable from one that is down.
So every write is asynchronous and bounded. Recorder buffers entries and flushes them in batches; past the bound it DROPS, and counts what it dropped. Losing log entries under pressure is a small, visible loss. Adding latency to every query to avoid it is a large, invisible one.
Retention is a promise, not a suggestion ¶
A query log is a record of every name every person on a network looked up. It is the most sensitive thing this software touches, and an unbounded one is a liability that grows. RetentionPolicy is therefore part of the store's configuration rather than an operational afterthought, the default is finite, and a store is required to enforce it without being asked.
Index ¶
- Constants
- Variables
- type Filter
- type Memory
- func (m *Memory) Append(_ context.Context, entries ...QueryEntry) error
- func (m *Memory) Close() error
- func (m *Memory) Count(_ context.Context, f Filter) (uint64, error)
- func (m *Memory) Len() int
- func (m *Memory) Prune(context.Context) (int, error)
- func (m *Memory) Query(_ context.Context, f Filter) ([]QueryEntry, error)
- func (m *Memory) TopDevices(_ context.Context, f Filter, n int) ([]Stat, error)
- func (m *Memory) TopNames(_ context.Context, f Filter, n int) ([]Stat, error)
- type MemoryOptions
- type QueryEntry
- type Recorder
- type RecorderOptions
- type RecorderStats
- type RetentionPolicy
- type Stat
- type Store
Constants ¶
const ( // DefaultBuffer is how many entries may be waiting to be written. At ten // thousand queries a second this is a fifth of a second of slack, which is // far longer than a healthy write and far shorter than an unhealthy one — // which is the shape you want: absorb a hiccup, notice an outage. DefaultBuffer = 2048 // DefaultBatch is how many entries one write carries. Batching is most of // the value of writing asynchronously at all: a per-query round trip to any // durable store costs more than the query did. DefaultBatch = 256 // DefaultFlushInterval bounds how long an entry waits when traffic is too // light to fill a batch. A query log that only appears once the network is // busy is one nobody can debug a quiet problem with. DefaultFlushInterval = time.Second )
Defaults for RecorderOptions.
const CloseTimeout = 15 * time.Second
CloseTimeout bounds how long Recorder.Close waits for the writer goroutine.
It is longer than the write deadline in [Recorder.write], so a store that honours its context always finishes on its own and this never fires.
const DefaultLimit = 100
DefaultLimit bounds an unqualified read.
Variables ¶
var DefaultRetention = RetentionPolicy{ MaxAge: 7 * 24 * time.Hour, MaxEntries: 1_000_000, }
DefaultRetention keeps a week or a million entries, whichever comes first.
A week is long enough to answer "what did this device do on Saturday", which is the question a query log is usually opened for, and short enough that the record of a household's browsing does not accumulate indefinitely because nobody chose a value.
Functions ¶
This section is empty.
Types ¶
type Filter ¶
type Filter struct {
Since, Until time.Time
Device string
// Name matches the entry's name exactly, canonically.
Name string
// Blocked, when set, selects only blocked or only unblocked entries.
Blocked *bool
// Limit caps the rows returned. Zero selects [DefaultLimit]; a query log
// has no natural size and an unbounded read of one is how a management API
// runs a server out of memory.
Limit int
// Offset skips rows, for paging.
Offset int
}
Filter selects entries to read back.
Every field is optional and they combine with AND. It is deliberately narrow: a query log is read to answer "what did this device ask for", "who asked for this name" and "what got blocked", and a general query language would be a larger surface for every implementation to get right.
type Memory ¶
type Memory struct {
// contains filtered or unexported fields
}
Memory is a Store that keeps entries in a ring buffer.
It exists so that the engine is useful to an embedder who wants a query log and no database, and so that everything above Store can be tested without one. It is not a lesser implementation with a real one to come: for a household resolver keeping a day of history, a bounded ring in memory is the correct amount of machinery, and a durable store is what you reach for when the history must survive a restart.
It is safe for concurrent use. Reads take a read lock and copy, so a caller walking a month of history cannot be handed entries that a concurrent append is overwriting.
func NewMemory ¶
func NewMemory(opts MemoryOptions) (*Memory, error)
NewMemory returns an in-memory store.
func (*Memory) Append ¶
func (m *Memory) Append(_ context.Context, entries ...QueryEntry) error
Append implements Store.
func (*Memory) Close ¶
Close implements Store. Nothing is released; the entries go when the store does.
func (*Memory) TopDevices ¶
TopDevices implements Store.
type MemoryOptions ¶
type MemoryOptions struct {
// Retention bounds what is kept. The zero policy selects
// [DefaultRetention]; keeping everything is [Unlimited], which has to be
// said rather than fallen into — see [RetentionPolicy].
Retention RetentionPolicy `json:"retention"`
Clock clock.Clock `json:"-"`
}
MemoryOptions configure a Memory store.
type QueryEntry ¶
type QueryEntry struct {
// At is when the query was answered.
At time.Time `json:"at"`
// Device and Client identify who asked. Device is the stable identifier
// something above the resolver assigned; Client is the address observed.
Device string `json:"device,omitempty"`
Client string `json:"client,omitempty"`
// Name, Type and Class are the question, with Name canonical.
Name string `json:"name"`
Type string `json:"type"`
Class string `json:"class,omitempty"`
// RCode is the answer's response code, as a mnemonic rather than a number
// so that a log stays readable without a lookup table.
RCode string `json:"rcode"`
// Source says where the answer came from: cache, upstream, stale, local.
Source string `json:"source"`
// Upstream names the provider that answered, empty when none was asked.
Upstream string `json:"upstream,omitempty"`
// Blocked reports a policy answer, and Rule the rule responsible.
Blocked bool `json:"blocked,omitempty"`
Rule string `json:"rule,omitempty"`
// Duration is the whole resolution.
Duration time.Duration `json:"duration"`
// Answers is how many records came back, which is what distinguishes an
// empty NOERROR from a real one at a glance.
Answers int `json:"answers"`
}
QueryEntry is one resolved query, as recorded.
It is a flat value with no pointers into the message it came from, because the entry outlives the query by design: it sits in a buffer, is written in a batch, and may be read back hours later. Holding a *dnsmsg.Message here would pin every record of every answer for as long as the buffer lived.
func EntryFor ¶
EntryFor builds a QueryEntry from the pieces the resolver has.
It lives here rather than in the resolver so that every store sees the same shape, and so the canonicalisation happens once: the name is lower-cased, because a log searched for "example.com" must find the query that arrived as "ExAmPle.CoM" under DNS-0x20 or from a client that shouts.
func (QueryEntry) String ¶
func (e QueryEntry) String() string
String renders an entry in a form a person can read in a terminal.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder writes query entries without ever blocking the query that produced them.
This is the whole reason it exists. A query log is written on the one path a resolver cannot afford to wait on: a disk that has filled, an fsync that stalls, a lock held by a compaction, a network store that has gone away — none of them may become DNS latency, because a resolver that answers slowly is indistinguishable to a user from one that is down.
So Recorder.Record is a non-blocking send onto a bounded channel and nothing else. Past the bound it drops and counts. Losing log entries under pressure is a small loss that shows up in Recorder.Dropped; adding a millisecond to every query to avoid it is a large loss that shows up nowhere.
func NewRecorder ¶
func NewRecorder(opts RecorderOptions) (*Recorder, error)
NewRecorder starts a Recorder writing to opts.Store.
func (*Recorder) Close ¶
Close flushes what is queued and stops the writer. It is idempotent, and every caller — not only the first — returns once the writer goroutine has actually exited or CloseTimeout has passed.
Both halves of that matter. Two goroutines closing at once is the ordinary shape of a careful shutdown (an explicit call plus a defer), and a second caller told the writer had exited when it had not may go on to tear down the store the writer is still inside.
The timeout is the other half. A Store that ignores the context it is given can park the writer inside Append forever, and an unbounded wait here turns that into a daemon that never exits and is eventually killed — losing the flush this method exists to perform. After the timeout Close returns an error naming the problem and abandons the writer, which is a leaked goroutine during shutdown and strictly better than not shutting down.
func (*Recorder) Record ¶
func (r *Recorder) Record(e QueryEntry)
Record queues one entry. It never blocks and never fails.
It returns nothing, and that is deliberate: there is no useful thing a caller on the query path could do with an error from the query log, and offering one would invite somebody to check it and, eventually, to wait on it.
type RecorderOptions ¶
type RecorderOptions struct {
// Store receives the batches. Required.
Store Store
// Buffer, Batch and FlushInterval take the defaults above when zero.
Buffer int `json:"buffer"`
Batch int `json:"batch"`
FlushInterval time.Duration `json:"flush_interval"`
Clock clock.Clock `json:"-"`
Logger *slog.Logger `json:"-"`
}
RecorderOptions configure a Recorder.
type RecorderStats ¶
type RecorderStats struct {
Recorded uint64 `json:"recorded"`
Written uint64 `json:"written"`
// Dropped is entries refused because the buffer was full. Non-zero means
// the store is slower than the query rate.
Dropped uint64 `json:"dropped"`
// Failed is entries the store rejected.
Failed uint64 `json:"failed"`
// Pending is entries queued but not yet written.
Pending int `json:"pending"`
}
RecorderStats reports what the recorder has done.
type RetentionPolicy ¶
type RetentionPolicy struct {
// MaxAge discards entries older than this. Zero means no age limit.
MaxAge time.Duration `json:"max_age"`
// MaxEntries discards the oldest beyond this count. Zero means no count
// limit.
MaxEntries int `json:"max_entries"`
// contains filtered or unexported fields
}
RetentionPolicy bounds what a store keeps.
Both limits apply; whichever bites first wins. A policy with neither set is rejected by RetentionPolicy.Validate rather than treated as "keep everything": a query log records every name every person on a network looked up, and an unbounded one is a liability that grows. Choosing to keep everything must be a thing somebody typed.
func Unlimited ¶
func Unlimited() RetentionPolicy
Unlimited is the retention policy that keeps everything, for a caller who has decided to.
It exists so that "keep everything" is a thing somebody wrote down. The zero RetentionPolicy is refused precisely so it cannot be arrived at by omission.
func (RetentionPolicy) MarshalJSON ¶
func (p RetentionPolicy) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler.
func (*RetentionPolicy) UnmarshalJSON ¶
func (p *RetentionPolicy) UnmarshalJSON(b []byte) error
UnmarshalJSON implements json.Unmarshaler.
func (RetentionPolicy) Validate ¶
func (p RetentionPolicy) Validate() error
Validate reports every problem with p.
type Store ¶
type Store interface {
// Append records entries. It may buffer, and it may return before anything
// is durable — see [Recorder], which is what the engine actually writes
// through and which never blocks a query.
Append(ctx context.Context, entries ...QueryEntry) error
// Query reads entries matching f, newest first.
Query(ctx context.Context, f Filter) ([]QueryEntry, error)
// Count reports how many entries match f, ignoring Limit and Offset.
Count(ctx context.Context, f Filter) (uint64, error)
// TopNames and TopDevices are the two aggregates every DNS dashboard shows.
// They are named methods rather than a general GROUP BY because a store
// backed by a real database can answer them with an index, and one backed
// by a map can answer them without inventing a query planner.
TopNames(ctx context.Context, f Filter, n int) ([]Stat, error)
TopDevices(ctx context.Context, f Filter, n int) ([]Stat, error)
// Prune deletes entries outside the retention policy and reports how many
// went. A store enforces retention itself; this exists so a caller can
// force it and learn the result.
Prune(ctx context.Context) (int, error)
// Close releases the store. It is idempotent.
Close() error
}
Store persists query entries and answers questions about them.
Implementations must be safe for concurrent use. Every method takes a context: a store may be a network round trip away, and a management API asking for a month of history must be cancellable.
func Nop ¶
func Nop() Store
Nop returns a Store that discards everything and answers nothing.
It is what a deployment that wants no query log uses, and it is a real choice rather than a degenerate one: a resolver that keeps no record of what anyone looked up is the most private configuration available, and the engine should not make it awkward to pick.