Documentation
¶
Overview ¶
Package drift — behavioral drift detection (§8.3 Wave 4).
Track per-session metrics, maintain a running per-user baseline (mean + variance via Welford's online algorithm), and flag when the current session is N standard deviations from baseline.
Why: model swaps, prompt changes, even silent provider-side updates can shift the agent's behavior. The user may not notice until the agent does something it never used to do. Drift detection surfaces "this session looks unusual" before the user hits the surprise.
What we track per session:
- ToolCallsPerTurn: agent's average tool-use intensity.
- ErrorRate: error entries / total entries.
- RecoveryRate: successful recoveries / total errors.
- SessionLength: total entries in the session.
- AvgTurnDuration: time between user_input entries.
Storage: rolling baseline state at ~/.overkill/drift/baseline.json. Per-session metrics computed on demand from the flight recorder. Welford's algorithm means we don't need to keep individual session metrics around once they've been folded in — O(1) memory.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var AllMetrics = []Metric{ MetricToolCallsPerTurn, MetricErrorRate, MetricRecoveryRate, MetricSessionLength, MetricAvgTurnDuration, }
AllMetrics is the canonical list — useful for ranging over.
Functions ¶
func FormatFinding ¶
FormatFinding renders a one-line summary suitable for a session- open heads-up: "tool_calls_per_turn unusually above baseline (observed 8.2, mean 3.1 ± 1.4, z=+3.6σ)".
func FormatFindings ¶
FormatFindings renders all findings joined by "; ". Empty when no findings.
Types ¶
type Baseline ¶
type Baseline struct {
Metrics map[Metric]*Stats `json:"metrics"`
UpdatedAt time.Time `json:"updated_at"`
Sessions int64 `json:"sessions"` // total folded-in
WarmupHint string `json:"warmup_hint,omitempty"`
// contains filtered or unexported fields
}
Baseline holds per-metric running stats for one user.
type CompareOptions ¶
type CompareOptions struct {
// Threshold is the |z-score| above which we flag drift.
// Default 2.0 — caught at the ~5% tail.
Threshold float64
// MinSessions is the minimum baseline session count before we
// trust the variance estimate. Below this we skip flagging.
// Default 5.
MinSessions int64
}
CompareOptions tunes Compare.
type Finding ¶
type Finding struct {
Metric Metric
Observed float64
Mean float64
StdDev float64
ZScore float64
// Direction is "above" or "below" so the caller can say "the
// agent was unusually QUIET this session" vs. "unusually NOISY".
Direction string
}
Finding is one drift signal.
type Metric ¶
type Metric string
Metric is one named axis we track. Keeping these as constants keeps the JSON shape stable across versions.
type Stats ¶
Stats is the Welford running-stats record for one metric. Count + Mean + M2 (sum of squared diffs from mean). Variance = M2 / (Count - 1). Memory cost: O(1) per metric regardless of how many sessions you fold in.
func (*Stats) Update ¶
Update folds a new observation in via Welford's online algorithm. Stable + numerically robust even for huge counts.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store persists a Baseline at a fixed path. Atomic temp+rename writes so a crash mid-save leaves the prior baseline intact.
Source Files