Documentation
¶
Overview ¶
Package logstore is the Badger-backed, time-ordered persistent request-log history engine. Extracted from package main's logstore.go + the deletion passes of logguard.go per ADR-0002 (BadgerDB containment, the catdb rationale).
It complements the two main-side surfaces rather than replacing them:
- the in-memory ring (main store.go `logs`) backs the live tail / SSE feed;
- the JSONL writer (main initRequestLog) remains for plain-text export;
- this store is the queryable, retention-managed HISTORY that survives restart and supports deep pagination ("page 20 = yesterday").
Key layout: 8-byte big-endian unix-millis ++ 4-byte big-endian seq (12 B)
The timestamp prefix keeps entries in chronological order so a reverse iterator yields newest-first; the seq disambiguates entries within the same millisecond and preserves insertion order.
Value layout: JSON-encoded Entry.
Retention is two-dimensional, matching the admin's choice (time + size):
- Age: a per-key Badger TTL (native, reliable). Expired entries are skipped on read and reclaimed during compaction/GC.
- Size: a best-effort janitor that, when the on-disk size exceeds the cap, deletes the oldest entries in bounded batches and runs value-log GC. Cleanup removes LOW-priority entries (access/traffic, Level INFO/DEBUG) before HIGH-priority security entries (WARN/ERROR) whenever possible.
What stays in package main: the process-wide singleton + enable/disable/ purge lifecycle, the disk-pressure ORCHESTRATOR (disk usage, minimal-mode state, audit trail, GUI status), and the admin retention API. The engine exposes two inversion points for it: the `minimal` hook injected at OpenTTL (Add's emergency skip reads main's minimal-mode state) and RunRetention returning its cleanup results so main records the audit/pressure event.
Index ¶
- Variables
- func Dropped() int64
- func EncKey(dir, passphrase string) ([]byte, error)
- func LowPriority(level string) bool
- func Pruned() int64
- func QuarantinedCopies(dir string) []string
- type DecryptionBlock
- type Entry
- type PageResult
- type Stats
- type Store
- func (s *Store) Add(e Entry)
- func (s *Store) BytesUsed() int64
- func (s *Store) CleanupBytes(need int64) (freed, count int64, levels map[string]int64)
- func (s *Store) Close() error
- func (s *Store) Encrypted() bool
- func (s *Store) PurgeAll() error
- func (s *Store) Query(fromMs, toMs int64, offset, limit int, filter func(*Entry) bool) ([]Entry, int, error)
- func (s *Store) QueryPage(fromMs, toMs int64, afterTS int64, afterSeq uint32, limit int, ...) (PageResult, error)
- func (s *Store) QueryPageWithBudget(fromMs, toMs int64, afterTS int64, afterSeq uint32, limit, scanBudget int, ...) (PageResult, error)
- func (s *Store) RetentionDays() int
- func (s *Store) RetentionMaxGB() float64
- func (s *Store) RunRetention() (freed, count int64, levels map[string]int64)
- func (s *Store) SetCancelJanitor(cancel context.CancelFunc)
- func (s *Store) SetRetention(days int, gb float64)
- func (s *Store) Stats() Stats
Constants ¶
This section is empty.
Variables ¶
var ErrEncMismatch = errors.New("saved logs use a different encryption key")
ErrEncMismatch is returned when an existing store can't be opened with the configured key (passphrase added/changed, or store was plaintext). The remediation is to purge the on-disk store and re-enable.
var ErrSaltUnusable = errors.New("saved logs exist but their encryption salt is missing or damaged")
ErrSaltUnusable is returned when a passphrase is configured, history already exists on disk, and the salt sidecar that derives its key is missing or damaged. See EncKey for why this is a refusal rather than a fresh salt.
Functions ¶
func Dropped ¶
func Dropped() int64
Dropped returns the cumulative count of entries dropped at the full queue.
func EncKey ¶
EncKey derives the AES key from the configured passphrase plus a persistent random salt (sidecar file dir+".salt"). Returns (nil, nil) when no passphrase is configured (encryption disabled).
A NEW SALT IS MINTED ONLY WHEN THERE IS NO STORE IT COULD LOCK US OUT OF (CHAOS-62). Minting unconditionally on an unreadable sidecar looks like making the common case work, and is in fact a one-way destruction of key material by the READ path: the derived key is a pure function of (passphrase, salt), so overwriting a torn 32-byte sidecar next to an existing encrypted store replaces the only value that could ever decrypt it. The store then fails to open with "different encryption key" — indistinguishable from an ordinary passphrase change — and the operator's remedy becomes "purge", destroying history that was intact right up until this function ran. That the sidecar can be truncated by exactly the unclean shutdown this store has to survive makes it reachable, not theoretical.
This is the same rule `internal/alerts` adopted for webhook signing secrets (SEC-WHSIGN-1): a failed decrypt never mints a key; creation belongs to first use, which for this store means a directory with nothing in it yet.
Refusing costs nothing that minting bought. Both paths fail to open; this one fails BEFORE overwriting the sidecar, so restoring it from a backup still recovers the history, and the operator who has no backup reaches the same purge they would have reached anyway.
func LowPriority ¶
LowPriority classifies an entry's storage priority by level. INFO, DEBUG, and empty are LOW priority (access/traffic); WARN and ERROR are HIGH priority (security: threats, malware, auth failures, policy violations).
func Pruned ¶
func Pruned() int64
Pruned returns the cumulative count of entries deleted by the size janitor.
func QuarantinedCopies ¶ added in v1.0.222
QuarantinedCopies returns the `.corrupt.*` siblings of dir, newest last, so a caller can re-surface an unreconciled quarantine from a PRIOR run: the in-memory record is process-local, so without this a self-healed store looks pristine on the next restart while the evidence — and the disk it occupies — is still sitting there.
Types ¶
type DecryptionBlock ¶ added in v1.0.115
type DecryptionBlock struct {
SchemaVersion int `json:"schema_version"` // independent version for the dec block
Outcome string `json:"outcome"` // decryptobs.Outcome
DecisionSource string `json:"decision_source"` // decryptobs.DecisionSource
RuleID string `json:"rule_id,omitempty"` // matched forward-proxy rule ULID
RuleName string `json:"rule_name,omitempty"` // matched rule display name
ProfileID string `json:"profile_id,omitempty"` // decryption-profile stable ID (autoexclude scopeID)
ProfileName string `json:"profile_name,omitempty"` // decryption-profile display name
Host string `json:"host"` // CONNECT authority / normalized host (redactable by hashing)
SNI string `json:"sni,omitempty"` // client-hello SNI when available (redactable)
TLSVersion string `json:"tls_version"` // decryptobs.TLSVersion
Cipher string `json:"cipher,omitempty"` // IANA suite name (record-only, never a metric label)
ALPN string `json:"alpn"` // decryptobs.ALPN ("" is a valid, explicit member)
CertVerify string `json:"cert_verify"` // decryptobs.CertVerify
FailStage string `json:"fail_stage"` // decryptobs.FailStage ("none" when no failure)
FailCategory string `json:"fail_category"` // decryptobs.FailCategory ("none" when no failure)
ExclReason string `json:"excl_reason"` // autoexclude.Reason ("" = no exclusion — explicit)
ExclScope string `json:"excl_scope,omitempty"` // owning profile ID (present only when excluded)
CacheConsulted bool `json:"cache_consulted"` // fail-open read path ran
CacheHit bool `json:"cache_hit"` // a learned entry bypassed this session
CacheLearned bool `json:"cache_learned"` // this session's evidence was recorded
Rescued bool `json:"rescued"` // live-rescue fired for this session
ScopeRuleCount int `json:"scope_rule_count"` // rules referencing the owning profile (explicit 0)
NodeID string `json:"node_id,omitempty"` // CP/DP NodeID; empty in single-binary mode
CertFingerprint string `json:"cert_fingerprint,omitempty"` // bounded SPKI/cert SHA-256 hash (privacy opt-in)
}
DecryptionBlock is the nested "dec" object on Entry (ADR-0011 §2.1). Fields are plain scalars (the enum .String() values) so logstore stays dependency-free and mirrors the flat auth_* precedent; the typed, validated source is main.DecryptionOutcome, which projects into this shape.
SERIALIZATION RULE (ADR-0011 §2.1 + PR #758 red-team): once the block is present, booleans, the required categorical enums, AND the int fields are NON-omitempty so an explicit false / "none" / 0 always serializes and stays queryable. Only genuinely optional strings (sni, cipher, cert_fingerprint, excl_scope, node_id, and the id/name pairs that are absent when no rule/profile matched) keep omitempty. `host` and `alpn` are non-omitempty: host is always known on a decisioned session (redacted by hashing to a fixed-length token, never by omission), and "" is a VALID alpn member that must serialize explicitly.
type Entry ¶
type Entry struct {
TS int64 `json:"ts"`
Time string `json:"time"`
IP string `json:"ip"`
Identity string `json:"identity,omitempty"` // authenticated username/email, empty if unauthenticated
Method string `json:"method"`
Host string `json:"host"`
URI string `json:"uri,omitempty"` // full request URL (host+path, no query); only set when the matched rule has LogFullURI
Status string `json:"status"` // OK | BLOCKED | AUTH_FAIL | RATE_LIMITED | IP_BLOCKED | POLICY_*
Level string `json:"level"` // INFO | WARN | ERROR
RuleMatched string `json:"ruleMatched"` // policy rule name that matched, if any
RuleID string `json:"ruleId,omitempty"` // stable ULID of the matched forward-proxy policy rule (rename-safe decision attribution, §1); omitted when no rule matched
ActionTaken string `json:"actionTaken"` // policy action taken, if any
BytesSent int64 `json:"bytesSent,omitempty"` // bytes sent to upstream (request body / tunnel client→dest)
BytesRecv int64 `json:"bytesRecv,omitempty"` // bytes received from upstream (response body / tunnel dest→client)
SSLAction string `json:"sslAction,omitempty"` // "inspect", "bypass", or empty (non-CONNECT)
DurationMs int64 `json:"durationMs,omitempty"` // connection lifetime; set on TUNNEL_CLOSED accounting entries
// Normalized authentication-policy SIEM fields (Phase 0 seam, §1.8; the
// auth_* observability block is finalized in Phase 1 Slice 5). Declared as the
// durable SIEM contract but populated only when an auth decision supplies them
// — all are omitempty so wire output stays byte-identical for requests with no
// auth decision. NO identity is carried in the auth_* block: an Exempt decision
// is logged by outcome + rule id/name (+ low-cardinality subject predicate
// types and the matched rule's subject schema version) only.
SchemaVersion int `json:"schema_version,omitempty"` // event schema version
AuthSource string `json:"auth_source,omitempty"` // categorical: idp|local|oidc:x|saml:x|exempt|unauth
AuthOutcome string `json:"auth_outcome,omitempty"` // Stage-1 outcome (e.g. "Exempt"); "" = none
AuthPolicyRuleID string `json:"auth_policy_rule_id,omitempty"` // ULID of matched Stage-1 rule
AuthPolicyRuleName string `json:"auth_policy_rule_name,omitempty"` // display name of matched Stage-1 rule
AccessRuleID string `json:"access_rule_id,omitempty"` // dormant Stage-2 auth-observability seam (unpopulated); forward-proxy decision attribution uses the top-level RuleID/ruleId field paired with RuleMatched
AuthSubjectMatchTypes []string `json:"auth_subject_match_types,omitempty"` // low-cardinality predicate type names (e.g. ["cidr"])
AuthSchemaVersion int `json:"auth_schema_version,omitempty"` // matched rule's SubjectMatch schema version
// Normalized decryption-observability block (ADR-0011). Nested pointer, block-level
// omitempty: when no decryption decision occurred (plain non-CONNECT, feature-off)
// Dec is nil and the "dec" key is ABSENT — wire stays byte-identical. When present,
// every categorical/boolean/int field serializes EXPLICITLY (see DecryptionBlock),
// so a negative outcome (cache_consulted:false, fail_stage:"none", scope_rule_count:0)
// is queryable rather than indistinguishable from an old/forgotten record. Populated
// only on the decryption decision path (a later ADR-0011 slice); nil until then.
Dec *DecryptionBlock `json:"dec,omitempty"`
}
Entry is one request-log record (moved from package main's LogEntry; main re-exposes it via a type alias). It is the wire type shared by the in-memory ring, the JSONL writer, the SSE live feed, and this store.
type PageResult ¶ added in v1.0.210
type PageResult struct {
Entries []Entry
// NextTS/NextSeq name the LAST entry RETURNED; a follow-up QueryPage
// passing them resumes strictly AFTER it (exclusive) in newest-first
// order. Meaningful only when len(Entries) > 0. When ScanLimited is
// false this is the correct continuation point — it deliberately does
// NOT skip the look-ahead match that proved HasMore.
NextTS int64
NextSeq uint32
// LastScanTS/LastScanSeq name the LAST raw entry VISITED and evaluated
// (matched or not). Meaningful when Scanned > 0. When ScanLimited is
// true this is the continuation point: resuming strictly after it never
// rescans a raw range this walk already proved non-matching, so a
// bounded continuation makes forward progress even when ZERO matching
// rows were returned (the sparse-filter forward-progress guarantee).
LastScanTS int64
LastScanSeq uint32
// HasMore reports that this walk did not exhaust the requested window:
// either a look-ahead MATCH proved another result exists (ScanLimited
// false), or the scan budget ran out first (ScanLimited true) — in the
// latter case more HISTORY remains to be SEARCHED, but no further match
// is proven to exist.
HasMore bool
// ScanLimited reports the walk stopped because the scan budget was
// exhausted before the window was exhausted and before a look-ahead
// match ended the page normally. Distinguishes "more matches are known"
// from "more history remains to be searched".
ScanLimited bool
// Scanned counts raw entries visited AND evaluated (matched or not) —
// the deterministic cost seam the Monitor scale gate asserts on: page
// cost is bounded by the entries visited to fill ONE page, never by how
// many pages precede it. Never exceeds the scan budget.
Scanned int
}
PageResult is one keyset page (ADR-FE-002 Monitor query contract).
type Stats ¶
type Stats struct {
Bytes int64 `json:"bytes"` // tracked logical size (same counter as the size cap)
Count int64 `json:"count"` // entries scanned (capped)
Capped bool `json:"capped"` // true when Count hit the scan cap
OldestMs int64 `json:"oldestMs,omitempty"` // timestamp of the oldest entry
}
Stats reports current usage for the admin retention panel.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the history store handle. Nil-safe on all methods.
func OpenResilientTTL ¶ added in v1.0.222
func OpenResilientTTL(dir string, ttl time.Duration, maxBytes int64, encKey []byte, minimal func() bool) (*Store, storeguard.Recovery, error)
OpenResilientTTL opens the history store the way OpenTTL does, but survives a directory the previous run could not: a store a prior process died inside of is quarantined before badger is allowed near it again, and a returned error positively identified as corruption is quarantined and the open retried once.
Returns (store, recovery, nil) on success. On failure the caller MUST degrade (run with history off) rather than treat it as fatal — see the file comment.
Direct OpenTTL remains the entry point for callers that already know the directory is sound.
func OpenTTL ¶
func OpenTTL(dir string, ttl time.Duration, maxBytes int64, encKey []byte, minimal func() bool) (*Store, error)
OpenTTL opens (or creates) the history store. It is the low-level constructor: main's enable path converts the admin's retention settings (days, GB) and passes the derived TTL/byte limits, the encryption key from EncKey (nil = unencrypted), and the minimal-mode hook (nil = never minimal). Tests use it directly for sub-day TTLs and tiny byte caps.
func (*Store) Add ¶
Add enqueues an entry for asynchronous batched persistence. It never blocks the caller (the proxy hot path): a full queue drops the entry and bumps a counter rather than stalling request handling.
func (*Store) BytesUsed ¶
BytesUsed returns the tracked logical size (the same counter the size cap uses). Nil-safe.
func (*Store) CleanupBytes ¶
CleanupBytes deletes stored entries to free at least `need` logical bytes, removing LOW-priority entries (oldest first) before HIGH-priority ones. Returns bytes freed, entries removed, and a per-level category breakdown. Held under the close read-lock; bounded by the scan cap per pass. Called by RunRetention (size cap) and by main's disk-critical handler (which owns the disk-usage check, minimal-mode state, and audit recording).
func (*Store) PurgeAll ¶
PurgeAll deletes all stored history (Badger DropAll) and resets the byte counter. The store stays open and usable. Held under the read lock so Close cannot close the DB mid-purge.
func (*Store) Query ¶
func (s *Store) Query(fromMs, toMs int64, offset, limit int, filter func(*Entry) bool) ([]Entry, int, error)
Query returns up to limit entries newest-first within [fromMs, toMs], applying filter, skipping offset, and reporting the total number of matches in the window (capped at the scan cap). offset/limit over a frozen time window give stable deep pagination. fromMs<=0 means "from the beginning", toMs<=0 means "up to now".
func (*Store) QueryPage ¶ added in v1.0.210
func (s *Store) QueryPage(fromMs, toMs int64, afterTS int64, afterSeq uint32, limit int, filter func(*Entry) bool) (PageResult, error)
QueryPage is keyset (cursor) pagination newest-first within [fromMs, toMs], applying filter, under the production scan budget (scanCap). See QueryPageWithBudget for the full contract.
func (*Store) QueryPageWithBudget ¶ added in v1.0.210
func (s *Store) QueryPageWithBudget(fromMs, toMs int64, afterTS int64, afterSeq uint32, limit, scanBudget int, filter func(*Entry) bool) (PageResult, error)
QueryPageWithBudget is QueryPage with an explicit raw-scan budget (scanBudget <= 0 ⇒ the production scanCap; the parameter exists so the bounded-continuation algorithm is testable with a small budget without a mutable global). Unlike Query it computes NO exact total and never scans past the page (plus one look-ahead match for HasMore), so the cost of page N does not grow with page depth. afterTS/afterSeq zero ⇒ first page; otherwise they name a previously returned OR previously scanned key and iteration resumes strictly after it. The (timestamp, seq) key pair is a total order, so paging is stable under concurrent appends: new entries get NEWER keys and can never duplicate or displace entries below an existing cursor.
Bounded scan-continuation contract (sparse-filter forward progress):
- Page filled and a look-ahead MATCH found ⇒ HasMore=true, ScanLimited=false; continue from NextTS/NextSeq (the look-ahead match is re-visited and returned by the next page — never skipped).
- Scan budget exhausted first ⇒ HasMore=true, ScanLimited=true; continue from LastScanTS/LastScanSeq (strictly after every raw entry already evaluated — an already-proven non-matching range is never rescanned, even when the page returned zero rows).
- Window exhausted ⇒ HasMore=false, ScanLimited=false: a true terminal empty/partial page.
func (*Store) RetentionDays ¶
RetentionDays returns the current age limit in whole days (0 = no limit).
func (*Store) RetentionMaxGB ¶
RetentionMaxGB returns the current size cap in GB (0 = no limit).
func (*Store) RunRetention ¶
RunRetention enforces the size cap: when the tracked size exceeds maxBytes it removes entries (low-priority first) until back under the cap and runs value-log GC. Age retention is handled natively by per-key TTL. One bounded pass per call; main's janitor calls it on a timer so the cap converges across passes despite lazy LSM size accounting. Returns what was cleaned so the caller can record the audit/pressure event (count == 0 → nothing ran).
func (*Store) SetCancelJanitor ¶
func (s *Store) SetCancelJanitor(cancel context.CancelFunc)
SetCancelJanitor records the cancel func for this store's janitor goroutine so Close can stop it. Called by main's enable path before the store is published; not synchronized (publish-before-share, like `minimal`).
func (*Store) SetRetention ¶
SetRetention updates the retention policy at runtime. The new TTL applies to newly written entries only (Badger TTL is fixed per key at write time); the size cap takes effect on the next janitor pass. days<=0 disables age expiry; gb<=0 disables the size cap.