tracestore

package
v0.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package tracestore is the durable store for sing-box trace data: the assembled connection records, the raw log lines a trace session asked to keep, and the five-minute rollups the usage and failure views read.

It owns its own SQLite database (trace.db), a sibling of logstore's logs.db in the same directory and with the same permissions. SQLite rather than bbolt because the query shape here is relational: an operator filters connections by any combination of user, line, node, destination, close reason and time, then pages newest-first across nodes. Hand-maintaining that many secondary indexes on bbolt is a permanent cost, and it is a relational database's day job. The driver is modernc.org/sqlite, which is pure Go: keeping lattice-server a single static binary with no cgo is a hard product property, so the cgo driver is not an option.

Unlabelled node log lines stay in logstore. Only what a session marked comes here.

Design: Lattice/SINGBOX-TRACE-DESIGN.md section 4.8.

Index

Constants

View Source
const (
	// DefaultRecordTTL, DefaultLineTTL and DefaultRollupTTL are the retention
	// floors from design section 4.8. Records outlive raw lines because a
	// record is a summary an operator still wants a week later, while a raw
	// line is bulk evidence for a capture that has already been read.
	DefaultRecordTTL = 14 * 24 * time.Hour
	DefaultLineTTL   = 7 * 24 * time.Hour
	DefaultRollupTTL = 90 * 24 * time.Hour

	// DefaultMaxBytes caps the whole database. Over it, oldest records are
	// deleted first, because the newest data is what an operator is looking at
	// during an incident.
	DefaultMaxBytes = int64(2) << 30 // 2 GiB

	// DefaultQueryLimit / MaxQueryLimit bound one page of records, matching
	// logstore's numbers so the two views page at the same rate.
	DefaultQueryLimit = 200
	MaxQueryLimit     = 1000

	// DefaultRollupLimit / MaxRollupLimit bound one rollup query. A rollup row
	// is small, and a chart over a week at five-minute resolution for one user
	// is about 2000 buckets, so the ceiling has to sit well above that.
	DefaultRollupLimit = 2000
	MaxRollupLimit     = 20000

	// RollupBucket is the rollup resolution. Five minutes is the design's
	// number: fine enough to see a failure burst, coarse enough that a busy
	// fleet does not write a row per connection.
	RollupBucket = 5 * time.Minute
)

Variables

View Source
var ErrBadCursor = errors.New("tracestore: malformed cursor")

ErrBadCursor is returned when a pagination cursor does not decode. Callers map it to a 400: a cursor is client-supplied input, and a bad one must never degrade into a silent full scan or a panic.

Functions

This section is empty.

Types

type Filter

type Filter struct {
	Since, Until time.Time
	NodeIDs      []string
	UserIDs      []string
	LineUUIDs    []string
	SessionIDs   []string
	// DstContains is a case-insensitive substring of the destination host, which
	// is what an operator actually types. It cannot use the dst_host index (no
	// index serves a leading wildcard), so it narrows whatever the other
	// predicates already selected.
	DstContains  string
	CloseReasons []string
	UserKinds    []string
	OnlyStalled  bool
	// IncludeOpen defaults false: a periodic snapshot of a still-running
	// connection is not a result an operator asked for, and mixing snapshots
	// into a list of finished connections double-reports live traffic.
	IncludeOpen bool
	Limit       int    // clamped to [1, MaxQueryLimit]; 0 means DefaultQueryLimit
	Cursor      string // opaque, from RecordPage.NextCursor
}

Filter is one page request against conn_records. Empty fields mean "no constraint on this dimension"; the fields combine with AND, and a slice combines with OR inside its own dimension.

type Options

type Options struct {
	RecordTTL time.Duration // default DefaultRecordTTL
	LineTTL   time.Duration // default DefaultLineTTL
	RollupTTL time.Duration // default DefaultRollupTTL
	MaxBytes  int64         // default DefaultMaxBytes
}

Options configures retention. A zero value takes every default.

type RecordPage

type RecordPage struct {
	Records           []model.ConnRecord `json:"records"`
	NextCursor        string             `json:"next_cursor,omitempty"`
	CollectedTotal    int64              `json:"collected_total"`
	CollectedNewestAt time.Time          `json:"collected_newest_at,omitzero"`
}

RecordPage is a newest-first page of records. An empty NextCursor means the result was exhausted.

CollectedTotal and CollectedNewestAt describe what the store holds for the nodes the caller may see, before any operator filter. An empty Records with CollectedTotal 0 means nothing has been collected (every policy off, or no agent has reported yet); an empty Records with CollectedTotal above zero means the filter matched nothing. Without the distinction both cases were one empty list, and the console told an operator with tracing switched off that "nothing matched these filters".

type RetainResult

type RetainResult struct {
	// Expired counts come from the three TTLs.
	RecordsExpired int64 `json:"records_expired"`
	LinesExpired   int64 `json:"lines_expired"`
	RollupsExpired int64 `json:"rollups_expired"`
	// Evicted counts come from the MaxBytes ceiling, oldest first.
	RecordsEvicted int64 `json:"records_evicted"`
	LinesEvicted   int64 `json:"lines_evicted"`

	BytesBefore int64 `json:"bytes_before"`
	BytesAfter  int64 `json:"bytes_after"`
	// Truncated reports that the batch ceiling was reached with work left. It is
	// surfaced rather than hidden: a store that never gets back under its cap
	// should be visible, not quietly growing.
	Truncated bool `json:"truncated"`
}

RetainResult is what one Retain call removed.

type Rollup

type Rollup struct {
	BucketStart time.Time `json:"bucket_start"`
	UserID      string    `json:"user_id,omitempty"`
	LineUUID    string    `json:"line_uuid,omitempty"`
	NodeID      string    `json:"node_id,omitempty"`

	// Connections counts every final record in the bucket.
	Connections int64 `json:"connections"`
	// BytesKnownCount is how many of those connections had their bytes actually
	// measured. Upload and Download sum only those. The two numbers travel
	// together on purpose: a caller that renders the sums without saying they
	// cover BytesKnownCount of Connections is presenting a partial total as a
	// whole one, which is the exact lie this feature exists to prevent.
	BytesKnownCount int64 `json:"bytes_known_count"`
	Upload          int64 `json:"upload"`
	Download        int64 `json:"download"`

	// CloseReasons counts final records per close reason. The counts sum to
	// Connections; a record with no reason is counted as unknown.
	CloseReasons map[string]int64 `json:"close_reasons,omitempty"`
}

Rollup is one five-minute bucket.

type RollupFilter

type RollupFilter struct {
	Since, Until time.Time
	UserIDs      []string
	LineUUIDs    []string
	NodeIDs      []string
	Limit        int // clamped to [1, MaxRollupLimit]; 0 means DefaultRollupLimit
}

RollupFilter mirrors the rollup grain: time, user, line, node.

type Stats

type Stats struct {
	Path           string    `json:"path"`
	SchemaVersion  int       `json:"schema_version"`
	Records        int64     `json:"records"`
	OpenRecords    int64     `json:"open_records"`
	Lines          int64     `json:"lines"`
	Rollups        int64     `json:"rollups"`
	OldestRecordAt time.Time `json:"oldest_record_at,omitzero"`
	NewestRecordAt time.Time `json:"newest_record_at,omitzero"`
	SizeBytes      int64     `json:"size_bytes"`
	MaxBytes       int64     `json:"max_bytes"`
	CipherEnabled  bool      `json:"cipher_enabled"`
}

Stats is the diagnostic projection of the store.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is the SQLite-backed trace store.

func Open

func Open(path string, cipher secret.Cipher, opts Options) (*Store, error)

Open opens (creating if needed) the trace store at path and runs any pending migration. A nil or disabled cipher stores the sealed columns in plaintext, exactly as logstore does when there is no master key.

func (*Store) AppendLines

func (s *Store) AppendLines(ls []model.TraceLine) (int, error)

AppendLines writes raw session lines. Re-delivering a line replaces it, so an agent that retries a batch cannot duplicate the evidence.

func (*Store) AppendRecords

func (s *Store) AppendRecords(rs []model.ConnRecord) (int, error)

AppendRecords writes a batch of connection records and folds the final ones into the five-minute rollups, all in one transaction. It returns the number of records applied.

The batch is all or nothing: an invalid record fails the call and writes nothing, so the caller can answer the agent with a 400 rather than silently keeping part of a batch it believes was rejected.

func (*Store) Close

func (s *Store) Close() error

Close releases the underlying database.

func (*Store) Collected

func (s *Store) Collected(nodeIDs []string) (int64, time.Time, error)

Collected counts every record the store holds for the given nodes, open or final, and the start time of the newest one. An empty nodeIDs counts the whole store.

func (*Store) Path

func (s *Store) Path() string

Path returns the on-disk path of the database file.

func (*Store) QueryLines

func (s *Store) QueryLines(sessionID string, afterSeq uint64, limit int) ([]model.TraceLine, error)

func (*Store) QueryRecords

func (s *Store) QueryRecords(f Filter) (RecordPage, error)

QueryRecords returns one newest-first page.

Paging is keyset, not OFFSET: the cursor carries the last row's full primary key so page N+1 is an index seek regardless of depth. OFFSET would re-walk every skipped row, and it would also silently duplicate or skip rows when ingest inserts underneath an operator who is paging.

func (*Store) Reattribute

func (s *Store) Reattribute(key model.ConnRecordKey, startedAt time.Time, to model.ConnRecord) error

Reattribute moves a record's identity and its aggregate contribution from the grain it was stored under to the one it now resolves to.

A record that arrived while the line read model was cold is stored unresolved. Repairing the row alone would leave the rollup counted under the empty user and line, so the aggregate would disagree with the record forever. Both moves happen in one transaction, or neither does.

func (*Store) RecordByKey

func (s *Store) RecordByKey(nodeID string, coreGeneration uint64, logID uint32, startedAt time.Time) (model.ConnRecord, bool, error)

QueryLines returns the raw lines of one session with seq greater than afterSeq, oldest first. That is the tail shape the dashboard polls with: pass back the last seq you saw and you get only what is new. RecordByKey returns one record by its identity. It exists because a scan of the newest page cannot answer this: a record older than that page is present in the database and absent from the scan, so a caller looking one up by key would report "not found" for something it is storing. startedAt completes the identity: one core generation can reuse a log id, and the store keeps both rows because its primary key includes the start time. A zero startedAt asks for the newest, which is the best a caller that does not know the exact connection can be given.

func (*Store) Retain

func (s *Store) Retain(now time.Time) (RetainResult, error)

Retain enforces the three TTLs and then the total size ceiling, deleting oldest first, and reports what it removed. now is passed in rather than read from the clock so the sweeper and the tests drive the same code path.

func (*Store) Rollups

func (s *Store) Rollups(f RollupFilter) ([]Rollup, error)

Rollups returns five-minute buckets in ascending time order, which is the order a chart draws them in.

func (*Store) Stats

func (s *Store) Stats() (Stats, error)

Stats returns the diagnostic projection of the store.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL