Documentation
¶
Index ¶
Constants ¶
const DefaultMemoryCap int64 = 500 * 1024 * 1024
DefaultMemoryCap is how many bytes of pending labels the manager will retain across all stats before refusing new entries (and incrementing the per-stat dropped counter). Sized large enough that "dropped" stays at 0 in any realistic workload — even at 500k lines/sec sustained, the worker drains the queue an order of magnitude faster than this cap.
Variables ¶
This section is empty.
Functions ¶
func ExtractFieldString ¶
ExtractFieldString resolves `path` against l's parsed JSON value and returns the leaf as a string suitable for label matching. Returns false if l is not JSON, the path doesn't resolve, or the leaf is a structural value (object / array) — those don't make sense as frequency labels.
Types ¶
type BackfillState ¶
BackfillState reflects the progress of the one-shot backfill goroutine that scans the previous N lines when a stat is created.
type Definition ¶
type Definition struct {
ID int
Name string // display title; defaults to leaf field key
FieldPath []string // JSON path from the root object, e.g. ["api_name"]
Type StatType
Groups []GroupRule // empty -> every distinct value forms its own group; values not matching any rule also form their own group
BackfillSize int // how many recent lines to backfill (0 = none)
}
Definition fully describes a stat to be tracked. The Manager owns one running aggregator per Definition. Most fields are user-configurable in the setup modal; ID is assigned by the Manager.
type GroupCount ¶
GroupCount is one bucket in a frequency snapshot.
type GroupRule ¶
GroupRule maps every value matching Pattern to the group labelled Title. Patterns are anchored automatically (^pattern$) so partial matches don't silently bucket more values than the user expects. The first rule whose pattern matches a value wins, so order matters — put catch-alls last.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager owns every active stat and fans out Observe calls. It is safe for concurrent use from the ingestor (Observe), the UI (Add/All/Stop), and the backfill goroutines.
func NewManager ¶
func NewManager() *Manager
NewManager returns an empty Manager with the default 500MB memory cap.
func (*Manager) Add ¶
func (m *Manager) Add(def Definition) (*Stat, error)
Add registers a new stat from a Definition, compiles its group regexes, starts its worker goroutine, and returns the running Stat. Returns an error if any group pattern is invalid.
func (*Manager) All ¶
All returns a snapshot of the currently registered stats. The returned slice is a copy; callers may iterate without holding any lock.
func (*Manager) MemoryUsed ¶
MemoryUsed returns the current pending bytes across all stats.
func (*Manager) Observe ¶
Observe forwards a line to every active stat. Each stat extracts the field inline and queues just the resolved label string — never blocks, never pins the *LogLine.
Non-JSON lines short-circuit at the manager level so we don't pay the per-stat type-check cost on every plain-text line.
func (*Manager) SetMemoryCap ¶
SetMemoryCap overrides the shared pending-bytes budget. Tests use this to deliberately force the dropped path.
type Snapshot ¶
type Snapshot struct {
Title string
Type StatType
Total uint64
Counts []GroupCount
Dropped uint64
Pending int
Backfill BackfillState
}
Snapshot is an immutable view of a stat at one moment. Returned by Stat.Snapshot for the renderer to consume without holding the aggregator's lock.
type Stat ¶
type Stat struct {
// contains filtered or unexported fields
}
Stat is one running aggregator. The producer side (Manager.Observe) does the JSON-path extraction up-front and queues just the resolved label string — never the full *LogLine, so pending entries stay small (≈ tens of bytes each) and don't pin the underlying log lines past the store's own offload cycle. The worker goroutine swap-drains the queue and runs the regex match + counter increment.
Counter storage uses two parallel layouts that may both be active on the same stat:
- Explicit groups (def.Groups != nil): one entry per group rule in fixedCounters, lock-free atomic increments. The hot path doesn't touch any mutex when a value matches.
- Dynamic: one map entry per distinct value, RWMutex'd. Reads use RLock + atomic increment on the *uint64 stored in the map; only first-sight inserts take the write lock. Used both when no explicit groups are defined AND as a fallback bucket for values that don't match any explicit rule (so each unmatched value forms its own group).
Either way the increment itself is one atomic op, so the worker can sustain ~10M lines/sec/stat on commodity hardware.
func (*Stat) AdvanceBackfill ¶
AdvanceBackfill increments the processed counter. Called by the backfill goroutine after each batch.
func (*Stat) Definition ¶
func (s *Stat) Definition() Definition
Definition returns a copy of the stat's definition.
func (*Stat) Feed ¶
Feed enqueues a line for processing. Non-blocking; the producer never stalls on the worker. Memory budget is enforced at the manager level — only when the global pending byte count would exceed the cap does this increment the dropped counter and skip the line. With the default 500MB cap that case is essentially unreachable in normal use.
func (*Stat) FinishBackfill ¶
func (s *Stat) FinishBackfill()
FinishBackfill clears the active flag.
func (*Stat) MarkBackfill ¶
MarkBackfill announces that a backfill of `total` lines is about to start. The renderer can show a progress hint while it runs.
Source Files