Documentation
¶
Overview ¶
Package logstore is the Badger-backed, time-ordered persistent request-log history engine. Extracted from package main's logstore.go + the deletion passes of logguard.go per ADR-0002 (BadgerDB containment, the catdb rationale).
It complements the two main-side surfaces rather than replacing them:
- the in-memory ring (main store.go `logs`) backs the live tail / SSE feed;
- the JSONL writer (main initRequestLog) remains for plain-text export;
- this store is the queryable, retention-managed HISTORY that survives restart and supports deep pagination ("page 20 = yesterday").
Key layout: 8-byte big-endian unix-millis ++ 4-byte big-endian seq (12 B)
The timestamp prefix keeps entries in chronological order so a reverse iterator yields newest-first; the seq disambiguates entries within the same millisecond and preserves insertion order.
Value layout: JSON-encoded Entry.
Retention is two-dimensional, matching the admin's choice (time + size):
- Age: a per-key Badger TTL (native, reliable). Expired entries are skipped on read and reclaimed during compaction/GC.
- Size: a best-effort janitor that, when the on-disk size exceeds the cap, deletes the oldest entries in bounded batches and runs value-log GC. Cleanup removes LOW-priority entries (access/traffic, Level INFO/DEBUG) before HIGH-priority security entries (WARN/ERROR) whenever possible.
What stays in package main: the process-wide singleton + enable/disable/ purge lifecycle, the disk-pressure ORCHESTRATOR (disk usage, minimal-mode state, audit trail, GUI status), and the admin retention API. The engine exposes two inversion points for it: the `minimal` hook injected at OpenTTL (Add's emergency skip reads main's minimal-mode state) and RunRetention returning its cleanup results so main records the audit/pressure event.
Index ¶
- Variables
- func Dropped() int64
- func EncKey(dir, passphrase string) ([]byte, error)
- func LowPriority(level string) bool
- func Pruned() int64
- type DecryptionBlock
- type Entry
- type Stats
- type Store
- func (s *Store) Add(e Entry)
- func (s *Store) BytesUsed() int64
- func (s *Store) CleanupBytes(need int64) (freed, count int64, levels map[string]int64)
- func (s *Store) Close() error
- func (s *Store) Encrypted() bool
- func (s *Store) PurgeAll() error
- func (s *Store) Query(fromMs, toMs int64, offset, limit int, filter func(*Entry) bool) ([]Entry, int, error)
- func (s *Store) RetentionDays() int
- func (s *Store) RetentionMaxGB() float64
- func (s *Store) RunRetention() (freed, count int64, levels map[string]int64)
- func (s *Store) SetCancelJanitor(cancel context.CancelFunc)
- func (s *Store) SetRetention(days int, gb float64)
- func (s *Store) Stats() Stats
Constants ¶
This section is empty.
Variables ¶
var ErrEncMismatch = errors.New("saved logs use a different encryption key")
ErrEncMismatch is returned when an existing store can't be opened with the configured key (passphrase added/changed, or store was plaintext). The remediation is to purge the on-disk store and re-enable.
Functions ¶
func Dropped ¶
func Dropped() int64
Dropped returns the cumulative count of entries dropped at the full queue.
func EncKey ¶
EncKey derives the AES key from the configured passphrase plus a persistent random salt (sidecar file dir+".salt"). Returns (nil, nil) when no passphrase is configured (encryption disabled).
func LowPriority ¶
LowPriority classifies an entry's storage priority by level. INFO, DEBUG, and empty are LOW priority (access/traffic); WARN and ERROR are HIGH priority (security: threats, malware, auth failures, policy violations).
Types ¶
type DecryptionBlock ¶ added in v1.0.115
type DecryptionBlock struct {
SchemaVersion int `json:"schema_version"` // independent version for the dec block
Outcome string `json:"outcome"` // decryptobs.Outcome
DecisionSource string `json:"decision_source"` // decryptobs.DecisionSource
RuleID string `json:"rule_id,omitempty"` // matched forward-proxy rule ULID
RuleName string `json:"rule_name,omitempty"` // matched rule display name
ProfileID string `json:"profile_id,omitempty"` // decryption-profile stable ID (autoexclude scopeID)
ProfileName string `json:"profile_name,omitempty"` // decryption-profile display name
Host string `json:"host"` // CONNECT authority / normalized host (redactable by hashing)
SNI string `json:"sni,omitempty"` // client-hello SNI when available (redactable)
TLSVersion string `json:"tls_version"` // decryptobs.TLSVersion
Cipher string `json:"cipher,omitempty"` // IANA suite name (record-only, never a metric label)
ALPN string `json:"alpn"` // decryptobs.ALPN ("" is a valid, explicit member)
CertVerify string `json:"cert_verify"` // decryptobs.CertVerify
FailStage string `json:"fail_stage"` // decryptobs.FailStage ("none" when no failure)
FailCategory string `json:"fail_category"` // decryptobs.FailCategory ("none" when no failure)
ExclReason string `json:"excl_reason"` // autoexclude.Reason ("" = no exclusion — explicit)
ExclScope string `json:"excl_scope,omitempty"` // owning profile ID (present only when excluded)
CacheConsulted bool `json:"cache_consulted"` // fail-open read path ran
CacheHit bool `json:"cache_hit"` // a learned entry bypassed this session
CacheLearned bool `json:"cache_learned"` // this session's evidence was recorded
Rescued bool `json:"rescued"` // live-rescue fired for this session
ScopeRuleCount int `json:"scope_rule_count"` // rules referencing the owning profile (explicit 0)
NodeID string `json:"node_id,omitempty"` // CP/DP NodeID; empty in single-binary mode
CertFingerprint string `json:"cert_fingerprint,omitempty"` // bounded SPKI/cert SHA-256 hash (privacy opt-in)
}
DecryptionBlock is the nested "dec" object on Entry (ADR-0011 §2.1). Fields are plain scalars (the enum .String() values) so logstore stays dependency-free and mirrors the flat auth_* precedent; the typed, validated source is main.DecryptionOutcome, which projects into this shape.
SERIALIZATION RULE (ADR-0011 §2.1 + PR #758 red-team): once the block is present, booleans, the required categorical enums, AND the int fields are NON-omitempty so an explicit false / "none" / 0 always serializes and stays queryable. Only genuinely optional strings (sni, cipher, cert_fingerprint, excl_scope, node_id, and the id/name pairs that are absent when no rule/profile matched) keep omitempty. `host` and `alpn` are non-omitempty: host is always known on a decisioned session (redacted by hashing to a fixed-length token, never by omission), and "" is a VALID alpn member that must serialize explicitly.
type Entry ¶
type Entry struct {
TS int64 `json:"ts"`
Time string `json:"time"`
IP string `json:"ip"`
Identity string `json:"identity,omitempty"` // authenticated username/email, empty if unauthenticated
Method string `json:"method"`
Host string `json:"host"`
URI string `json:"uri,omitempty"` // full request URL (host+path, no query); only set when the matched rule has LogFullURI
Status string `json:"status"` // OK | BLOCKED | AUTH_FAIL | RATE_LIMITED | IP_BLOCKED | POLICY_*
Level string `json:"level"` // INFO | WARN | ERROR
RuleMatched string `json:"ruleMatched"` // policy rule name that matched, if any
RuleID string `json:"ruleId,omitempty"` // stable ULID of the matched forward-proxy policy rule (rename-safe decision attribution, §1); omitted when no rule matched
ActionTaken string `json:"actionTaken"` // policy action taken, if any
BytesSent int64 `json:"bytesSent,omitempty"` // bytes sent to upstream (request body / tunnel client→dest)
BytesRecv int64 `json:"bytesRecv,omitempty"` // bytes received from upstream (response body / tunnel dest→client)
SSLAction string `json:"sslAction,omitempty"` // "inspect", "bypass", or empty (non-CONNECT)
DurationMs int64 `json:"durationMs,omitempty"` // connection lifetime; set on TUNNEL_CLOSED accounting entries
// Normalized authentication-policy SIEM fields (Phase 0 seam, §1.8; the
// auth_* observability block is finalized in Phase 1 Slice 5). Declared as the
// durable SIEM contract but populated only when an auth decision supplies them
// — all are omitempty so wire output stays byte-identical for requests with no
// auth decision. NO identity is carried in the auth_* block: an Exempt decision
// is logged by outcome + rule id/name (+ low-cardinality subject predicate
// types and the matched rule's subject schema version) only.
SchemaVersion int `json:"schema_version,omitempty"` // event schema version
AuthSource string `json:"auth_source,omitempty"` // categorical: idp|local|oidc:x|saml:x|exempt|unauth
AuthOutcome string `json:"auth_outcome,omitempty"` // Stage-1 outcome (e.g. "Exempt"); "" = none
AuthPolicyRuleID string `json:"auth_policy_rule_id,omitempty"` // ULID of matched Stage-1 rule
AuthPolicyRuleName string `json:"auth_policy_rule_name,omitempty"` // display name of matched Stage-1 rule
AccessRuleID string `json:"access_rule_id,omitempty"` // dormant Stage-2 auth-observability seam (unpopulated); forward-proxy decision attribution uses the top-level RuleID/ruleId field paired with RuleMatched
AuthSubjectMatchTypes []string `json:"auth_subject_match_types,omitempty"` // low-cardinality predicate type names (e.g. ["cidr"])
AuthSchemaVersion int `json:"auth_schema_version,omitempty"` // matched rule's SubjectMatch schema version
// Normalized decryption-observability block (ADR-0011). Nested pointer, block-level
// omitempty: when no decryption decision occurred (plain non-CONNECT, feature-off)
// Dec is nil and the "dec" key is ABSENT — wire stays byte-identical. When present,
// every categorical/boolean/int field serializes EXPLICITLY (see DecryptionBlock),
// so a negative outcome (cache_consulted:false, fail_stage:"none", scope_rule_count:0)
// is queryable rather than indistinguishable from an old/forgotten record. Populated
// only on the decryption decision path (a later ADR-0011 slice); nil until then.
Dec *DecryptionBlock `json:"dec,omitempty"`
}
Entry is one request-log record (moved from package main's LogEntry; main re-exposes it via a type alias). It is the wire type shared by the in-memory ring, the JSONL writer, the SSE live feed, and this store.
type Stats ¶
type Stats struct {
Bytes int64 `json:"bytes"` // tracked logical size (same counter as the size cap)
Count int64 `json:"count"` // entries scanned (capped)
Capped bool `json:"capped"` // true when Count hit the scan cap
OldestMs int64 `json:"oldestMs,omitempty"` // timestamp of the oldest entry
}
Stats reports current usage for the admin retention panel.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the history store handle. Nil-safe on all methods.
func OpenTTL ¶
func OpenTTL(dir string, ttl time.Duration, maxBytes int64, encKey []byte, minimal func() bool) (*Store, error)
OpenTTL opens (or creates) the history store. It is the low-level constructor: main's enable path converts the admin's retention settings (days, GB) and passes the derived TTL/byte limits, the encryption key from EncKey (nil = unencrypted), and the minimal-mode hook (nil = never minimal). Tests use it directly for sub-day TTLs and tiny byte caps.
func (*Store) Add ¶
Add enqueues an entry for asynchronous batched persistence. It never blocks the caller (the proxy hot path): a full queue drops the entry and bumps a counter rather than stalling request handling.
func (*Store) BytesUsed ¶
BytesUsed returns the tracked logical size (the same counter the size cap uses). Nil-safe.
func (*Store) CleanupBytes ¶
CleanupBytes deletes stored entries to free at least `need` logical bytes, removing LOW-priority entries (oldest first) before HIGH-priority ones. Returns bytes freed, entries removed, and a per-level category breakdown. Held under the close read-lock; bounded by the scan cap per pass. Called by RunRetention (size cap) and by main's disk-critical handler (which owns the disk-usage check, minimal-mode state, and audit recording).
func (*Store) PurgeAll ¶
PurgeAll deletes all stored history (Badger DropAll) and resets the byte counter. The store stays open and usable. Held under the read lock so Close cannot close the DB mid-purge.
func (*Store) Query ¶
func (s *Store) Query(fromMs, toMs int64, offset, limit int, filter func(*Entry) bool) ([]Entry, int, error)
Query returns up to limit entries newest-first within [fromMs, toMs], applying filter, skipping offset, and reporting the total number of matches in the window (capped at the scan cap). offset/limit over a frozen time window give stable deep pagination. fromMs<=0 means "from the beginning", toMs<=0 means "up to now".
func (*Store) RetentionDays ¶
RetentionDays returns the current age limit in whole days (0 = no limit).
func (*Store) RetentionMaxGB ¶
RetentionMaxGB returns the current size cap in GB (0 = no limit).
func (*Store) RunRetention ¶
RunRetention enforces the size cap: when the tracked size exceeds maxBytes it removes entries (low-priority first) until back under the cap and runs value-log GC. Age retention is handled natively by per-key TTL. One bounded pass per call; main's janitor calls it on a timer so the cap converges across passes despite lazy LSM size accounting. Returns what was cleaned so the caller can record the audit/pressure event (count == 0 → nothing ran).
func (*Store) SetCancelJanitor ¶
func (s *Store) SetCancelJanitor(cancel context.CancelFunc)
SetCancelJanitor records the cancel func for this store's janitor goroutine so Close can stop it. Called by main's enable path before the store is published; not synchronized (publish-before-share, like `minimal`).
func (*Store) SetRetention ¶
SetRetention updates the retention policy at runtime. The new TTL applies to newly written entries only (Badger TTL is fixed per key at write time); the size cap takes effect on the next janitor pass. days<=0 disables age expiry; gb<=0 disables the size cap.
Source Files