filter

package
v1.1.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 26, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Overview

Package filter provides signal filtering and prediction algorithms for NornicDB.

This package implements a lightweight Kalman filter based on the imu-f project (https://github.com/heliorc/imu-f) designed for real-time state estimation and future value prediction with minimal computational overhead.

The filter is optimized for:

  • Memory decay score prediction
  • Co-access pattern confidence filtering
  • Query latency prediction
  • Similarity score smoothing

Key Features:

  • Adaptive measurement noise (R) based on signal variance
  • Setpoint-based error boosting for faster convergence
  • Velocity-based state projection for prediction
  • No matrix operations - pure scalar math for speed

Example Usage: 

// Create a filter for memory decay prediction
filter := filter.NewKalman(filter.DefaultConfig())

// Process observations
for _, observation := range decayScores {
	filtered := filter.Process(observation, targetScore)
	fmt.Printf("Filtered: %.3f\n", filtered)
}

// Predict future value
futureScore := filter.Predict(5) // 5 steps ahead

ELI12 (Explain Like I'm 12):

Imagine you're trying to guess where a ball will land. Each time you see the ball, you update your guess. But your eyes aren't perfect (measurement noise), and the ball might suddenly change direction (process noise).

The Kalman filter is like having a really smart friend who: 1. Remembers where the ball was before 2. Guesses where it's going based on how fast it was moving 3. Updates the guess when they see new info, but doesn't completely forget the old guess 4. Trusts new info MORE when their guess was way off (error boosting)

Original implementation: https://github.com/heliorc/imu-f/blob/master/src/filter/kalman.c

Package filter - Adaptive Kalman filter that auto-switches modes.

KalmanAdaptive monitors signal characteristics and dynamically switches between basic (smoothing) and velocity (tracking) modes based on:

  • Trend detection (is the signal drifting?)
  • Variance analysis (is noise high or low?)
  • Prediction error (is current mode working well?)

This provides the best of both worlds:

  • Use basic mode for stable signals → maximum noise rejection
  • Use velocity mode for trends → accurate tracking
  • Switch automatically when conditions change

Example usage: 

filter := NewKalmanAdaptive(DefaultAdaptiveConfig())
for _, observation := range data {
	filtered := filter.Process(observation)
	fmt.Printf("Mode: %s, Value: %.3f\n", filter.Mode(), filtered)
}

ELI12 (Explain Like I'm 12):

Imagine you have two friends helping you catch a ball:

  • Friend A is great at catching balls thrown straight at you
  • Friend B is great at catching balls that curve through the air

The adaptive filter is like having a coach who watches and says "Hey, this ball is curving - Friend B, you take this one!" It automatically picks the best helper for each situation.

Package filter - Velocity-state Kalman filter for trend tracking.

This implements a 2-state Kalman filter that explicitly estimates both position and velocity, providing much better trend tracking than the basic scalar filter.

Use KalmanVelocity when:

  • Signal has trends or drift
  • Prediction accuracy is critical
  • Temporal patterns need tracking

Use basic Kalman when:

  • Signal is stationary (stable value + noise)
  • Maximum noise rejection is priority
  • Simplicity and speed are paramount

The 2-state model: 

State vector: [position, velocity]ᵀ
Transition:   x(k+1) = F * x(k) + process_noise
              where F = [1, dt; 0, 1]
Measurement:  z(k) = H * x(k) + measurement_noise
              where H = [1, 0] (we only measure position)

ELI12 (Explain Like I'm 12):

The basic Kalman filter is like guessing where a ball is, but forgetting how fast it was going. This velocity filter remembers BOTH where the ball is AND how fast it's moving. So when you predict where it'll be next, you say "it was HERE, moving THIS FAST, so it'll probably be THERE."

This makes it WAY better at following moving targets!

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AdaptiveConfig 

type AdaptiveConfig struct {
	// BasicConfig for the underlying basic filter
	BasicConfig Config

	// VelocityConfig for the underlying velocity filter
	VelocityConfig VelocityConfig

	// TrendThreshold - velocity magnitude above which we switch to velocity mode
	// Default: 0.1 (10% of signal range per step)
	TrendThreshold float64

	// StabilityThreshold - velocity magnitude below which we switch to basic mode
	// Default: 0.02 (2% of signal range per step)
	StabilityThreshold float64

	// ErrorThreshold - prediction error above which we consider switching
	// Default: 2.0 (2x expected noise)
	ErrorThreshold float64

	// SwitchHysteresis - minimum observations before mode can switch again
	// Prevents rapid oscillation between modes
	// Default: 10
	SwitchHysteresis int

	// WindowSize - number of observations for trend/variance detection
	// Default: 20
	WindowSize int

	// InitialMode - starting mode (ModeBasic, ModeVelocity, or ModeAuto)
	InitialMode FilterMode
}

AdaptiveConfig holds configuration for the adaptive filter.

func DefaultAdaptiveConfig 

func DefaultAdaptiveConfig() AdaptiveConfig

DefaultAdaptiveConfig returns sensible defaults for auto-switching.

func SmoothingOptimizedConfig 

func SmoothingOptimizedConfig() AdaptiveConfig

SmoothingOptimizedConfig favors basic mode, only switches for strong trends.

func TrackingOptimizedConfig 

func TrackingOptimizedConfig() AdaptiveConfig

TrackingOptimizedConfig favors velocity mode, only switches for very stable signals.

type AdaptiveStats 

type AdaptiveStats struct {
	Mode            FilterMode
	ForcedMode      FilterMode
	Observations    int
	SwitchCount     int
	TrendScore      float64
	PredictionError float64
	CurrentState    float64
	CurrentVelocity float64
}

AdaptiveStats holds statistics for the adaptive filter.

type Config 

type Config struct {
	// ProcessNoise (Q) - how much we expect the true state to change between measurements.
	// Higher values = more responsive to changes, but noisier output.
	// Default: 0.1 (scaled by 0.001 internally like imu-f)
	ProcessNoise float64

	// MeasurementNoise (R) - how much we distrust individual measurements.
	// Higher values = smoother output, but slower to respond.
	// Default: 88.0 (seed value from imu-f)
	MeasurementNoise float64

	// InitialCovariance (P) - initial uncertainty in our estimate.
	// Default: 30.0 (seed value from imu-f)
	InitialCovariance float64

	// VarianceScale - multiplier for adaptive R calculation.
	// Default: 10.0
	VarianceScale float64
}

Config holds Kalman filter configuration.

func CoAccessConfig 

func CoAccessConfig() Config

CoAccessConfig returns config optimized for co-access pattern filtering.

func DecayPredictionConfig 

func DecayPredictionConfig() Config

DecayPredictionConfig returns config optimized for memory decay prediction.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns sensible defaults based on imu-f tuning.

func LatencyConfig 

func LatencyConfig() Config

LatencyConfig returns config optimized for query latency prediction.

type FilterMode 

type FilterMode string

FilterMode represents the current filtering strategy. 

const (
	// ModeBasic uses the scalar Kalman filter (optimal for stable signals)
	ModeBasic FilterMode = "basic"

	// ModeVelocity uses the 2-state Kalman filter (optimal for trends)
	ModeVelocity FilterMode = "velocity"

	// ModeAuto lets the filter decide automatically
	ModeAuto FilterMode = "auto"
)

type Kalman 

type Kalman struct {
	// contains filtered or unexported fields
}

Kalman implements a simple scalar Kalman filter with velocity-based prediction.

Based on the imu-f flight controller implementation, this filter provides:

  • State estimation with adaptive noise handling
  • Setpoint-based error boosting for faster convergence
  • Future state prediction using velocity

func NewKalman 

func NewKalman(cfg Config) *Kalman

NewKalman creates a new Kalman filter with the given configuration.

The Kalman filter provides optimal state estimation by combining predictions with noisy measurements. It's widely used in aerospace, robotics, and signal processing for tracking and smoothing time-series data.

Parameters:

  • cfg: Configuration parameters (use DefaultConfig() for general use)

Returns:

  • *Kalman ready to process measurements

Example 1 - Smoothing Noisy Sensor Data: 

filter := filter.NewKalman(filter.DefaultConfig())

// Simulate noisy temperature readings
trueTemp := 25.0
for i := 0; i < 10; i++ {
	// Measurement with noise
	noisy := trueTemp + (rand.Float64()-0.5)*2.0 // ±1°C noise

	filtered := filter.Process(noisy, trueTemp)
	fmt.Printf("Raw: %.2f°C, Filtered: %.2f°C\n", noisy, filtered)
}
// Filtered values are much smoother than raw readings

Example 2 - Memory Decay Score Prediction: 

filter := filter.NewKalman(filter.DecayPredictionConfig())

// Track memory decay over time
for day := 0; day < 30; day++ {
	// Calculate current decay score
	score := calculateDecayScore(memory, day)

	// Filter the score
	smoothed := filter.Process(score, 0.5) // Target: keep at 0.5

	// Predict score 7 days ahead
	predicted := filter.Predict(7)

	if predicted < 0.1 {
		fmt.Println("Memory will decay below threshold in a week!")
	}
}

Example 3 - Query Latency Tracking: 

filter := filter.NewKalman(filter.DefaultConfig())

// Track database query latency
for {
	start := time.Now()
	executeQuery()
	latencyMs := time.Since(start).Milliseconds()

	// Smooth latency measurements
	smoothed := filter.Process(float64(latencyMs), 0)

	// Alert if smoothed latency exceeds threshold
	if smoothed > 100 {
		log.Printf("High latency detected: %.1fms", smoothed)
	}
}

ELI12:

Imagine you're trying to guess your friend's bedtime by asking them every day:

  • Day 1: "11pm" → Your guess: 11pm
  • Day 2: "10pm" → Your guess: 10:30pm (average of old guess + new info)
  • Day 3: "11pm" → Your guess: 10:45pm (slowly adjusting)

But sometimes they lie or make mistakes! The Kalman filter is like being EXTRA smart:

  • If they usually say 11pm, and suddenly say "2am", you don't believe it completely (measurement noise handling)
  • If bedtime has been getting earlier (velocity), you predict it'll keep getting earlier
  • When your guess is WAY off, you trust new measurements more (error boosting)

This makes your guess better than simple averaging!

When to Use:

  • Smoothing noisy sensor data (temperature, GPS, accelerometer)
  • Tracking trends with predictions (decay scores, query latency)
  • Filtering user behavior patterns (access frequency, session length)
  • Real-time state estimation (object tracking, signal processing)

Performance:

  • O(1) per measurement - extremely fast
  • No matrix operations - pure scalar math
  • Memory: ~200 bytes per filter
  • Thread-safe with mutex protection

Thread Safety: 

All methods are thread-safe for concurrent access.

func NewKalmanWithInitial 

func NewKalmanWithInitial(cfg Config, initialState float64) *Kalman

NewKalmanWithInitial creates a filter with an initial state estimate.

func (*Kalman) Covariance 

func (k *Kalman) Covariance() float64

Covariance returns the current estimate uncertainty.

func (*Kalman) Gain 

func (k *Kalman) Gain() float64

Gain returns the current Kalman gain (0-1). Higher gain = trusting measurements more.

func (*Kalman) GetStats 

func (k *Kalman) GetStats() Stats

GetStats returns current filter statistics.

func (*Kalman) Observations 

func (k *Kalman) Observations() int

Observations returns the number of measurements processed.

func (*Kalman) Predict 

func (k *Kalman) Predict(steps int) float64

Predict estimates the state n steps into the future.

Uses the current velocity (rate of change) to project forward. Does not update the filter state.

func (*Kalman) PredictIfEnabled 

func (k *Kalman) PredictIfEnabled(feature string, steps int) config.FilteredValue

PredictIfEnabled returns a predicted value if the feature is enabled. If disabled, returns the current state unchanged.

func (*Kalman) PredictWithUncertainty 

func (k *Kalman) PredictWithUncertainty(steps int) (value, uncertainty float64)

PredictWithUncertainty returns the predicted value and its uncertainty.

func (*Kalman) Process 

func (k *Kalman) Process(measurement, target float64) float64

Process updates the filter with a new measurement and optional setpoint target.

This is the core Kalman filter update step. It combines the current prediction with the new measurement to produce an optimal estimate. The filter automatically adapts to measurement noise and uses velocity-based projection for prediction.

Parameters:

  • measurement: The observed value at this timestep
  • target: The desired setpoint (use 0 if no specific target)

Returns:

  • Filtered state estimate (smoothed value)

Example 1 - Simple Smoothing: 

filter := filter.NewKalman(filter.DefaultConfig())

measurements := []float64{10.2, 9.8, 10.5, 9.9, 10.1, 10.3}
for _, m := range measurements {
	smoothed := filter.Process(m, 0)
	fmt.Printf("Raw: %.2f → Filtered: %.2f\n", m, smoothed)
}
// Output shows smoothed values with reduced noise

Example 2 - With Target Setpoint: 

filter := filter.NewKalman(filter.DefaultConfig())

// Try to maintain temperature at 25°C
targetTemp := 25.0
for {
	currentTemp := readSensor()
	filtered := filter.Process(currentTemp, targetTemp)

	// When far from target, filter becomes more responsive
	error := targetTemp - filtered
	adjustHeater(error)
}

Example 3 - Real-time Anomaly Detection: 

filter := filter.NewKalman(filter.DefaultConfig())

for {
	value := getMetric()
	expected := filter.Process(value, 0)

	// Check if measurement deviates significantly from prediction
	deviation := math.Abs(value - expected)
	if deviation > 3*filter.Covariance() { // 3-sigma rule
		log.Printf("ANOMALY: Expected %.2f, got %.2f", expected, value)
	}
}

Example 4 - Memory Decay with Reinforcement: 

filter := filter.NewKalman(filter.DecayPredictionConfig())
targetScore := 0.6 // Want to maintain this decay score

for day := 0; day < 30; day++ {
	score := calculateDecayScore(memory)
	smoothed := filter.Process(score, targetScore)

	// If smoothed score drops below target, reinforce the memory
	if smoothed < targetScore {
		reinforceMemory(memory)
	}
}

ELI12:

Think of Process like updating your guess about the weather:

  1. You predicted: "It'll be 70°F"
  2. Thermometer says: "72°F"
  3. You think: "My prediction was close, but I'll adjust slightly"
  4. New guess: "71°F" (between prediction and measurement)

The cool part: If you said "70°F" and the thermometer says "90°F", you don't immediately believe it! You think: "That's weird, maybe the thermometer is broken. I'll adjust a little, but not all the way." That's measurement noise handling.

If the temperature has been rising (velocity), you'll predict it'll keep rising. That's velocity-based projection.

The target parameter is like a goal: "I want it to be 72°F". When you're far from the goal, you trust new measurements MORE to get back on track faster. That's error boosting.

How it Decides:

  • Far from measurement → Trust measurement more
  • Far from target → Trust measurement more (error boosting)
  • High measurement noise → Trust prediction more
  • Consistent trend (velocity) → Project forward

Performance:

  • O(1) constant time
  • Pure scalar math, no allocations
  • Adaptive noise handling for changing conditions

Thread Safety: 

Safe to call concurrently from multiple goroutines.

func (*Kalman) ProcessBatch 

func (k *Kalman) ProcessBatch(measurements []float64, target float64) []float64

ProcessBatch processes multiple measurements efficiently.

func (*Kalman) ProcessIfEnabled 

func (k *Kalman) ProcessIfEnabled(feature string, measurement, target float64) config.FilteredValue

ProcessIfEnabled applies filtering if the feature is enabled. If disabled, returns the raw measurement unchanged.

Parameters:

  • feature: The feature flag to check (e.g., FeatureKalmanDecay)
  • measurement: The observed value
  • target: The desired setpoint (use 0 if no target)

Returns a config.FilteredValue containing both raw and filtered values.

func (*Kalman) Reset 

func (k *Kalman) Reset()

Reset resets the filter to initial state.

func (*Kalman) SetState 

func (k *Kalman) SetState(state float64)

SetState manually sets the current state (use sparingly).

func (*Kalman) State 

func (k *Kalman) State() float64

State returns the current state estimate.

func (*Kalman) UpdateAdaptiveR 

func (k *Kalman) UpdateAdaptiveR()

UpdateAdaptiveR updates measurement noise based on innovation variance. Call periodically (e.g., every 10-20 observations) for adaptive filtering.

This implements the variance-based R adaptation from imu-f.

func (*Kalman) Velocity 

func (k *Kalman) Velocity() float64

Velocity returns the current rate of change.

type KalmanAdaptive 

type KalmanAdaptive struct {
	// contains filtered or unexported fields
}

KalmanAdaptive wraps both filter types and switches dynamically.

func NewKalmanAdaptive 

func NewKalmanAdaptive(cfg AdaptiveConfig) *KalmanAdaptive

NewKalmanAdaptive creates a new adaptive filter.

func (*KalmanAdaptive) GetStats 

func (k *KalmanAdaptive) GetStats() AdaptiveStats

GetStats returns current statistics.

func (*KalmanAdaptive) Mode 

func (k *KalmanAdaptive) Mode() FilterMode

Mode returns the current filtering mode.

func (*KalmanAdaptive) Observations 

func (k *KalmanAdaptive) Observations() int

Observations returns total observations processed.

func (*KalmanAdaptive) Predict 

func (k *KalmanAdaptive) Predict(steps int) float64

Predict estimates future state.

func (*KalmanAdaptive) PredictionError 

func (k *KalmanAdaptive) PredictionError() float64

PredictionError returns the smoothed prediction error.

func (*KalmanAdaptive) Process 

func (k *KalmanAdaptive) Process(measurement float64) float64

Process filters a new measurement, auto-switching modes if needed.

func (*KalmanAdaptive) ProcessBatch 

func (k *KalmanAdaptive) ProcessBatch(measurements []float64) []float64

ProcessBatch processes multiple measurements.

func (*KalmanAdaptive) ProcessIfEnabled 

func (k *KalmanAdaptive) ProcessIfEnabled(feature string, measurement float64) config.FilteredValue

ProcessIfEnabled applies filtering if enabled.

func (*KalmanAdaptive) Reset 

func (k *KalmanAdaptive) Reset()

Reset resets both filters and statistics.

func (*KalmanAdaptive) SetMode 

func (k *KalmanAdaptive) SetMode(mode FilterMode)

SetMode forces a specific mode (ModeBasic, ModeVelocity) or re-enables auto (ModeAuto).

func (*KalmanAdaptive) State 

func (k *KalmanAdaptive) State() float64

State returns the current filtered state.

func (*KalmanAdaptive) SwitchCount 

func (k *KalmanAdaptive) SwitchCount() int

SwitchCount returns how many times the mode has switched.

func (*KalmanAdaptive) TrendScore 

func (k *KalmanAdaptive) TrendScore() float64

TrendScore returns the current trend strength estimate (0-1+).

func (*KalmanAdaptive) Velocity 

func (k *KalmanAdaptive) Velocity() float64

Velocity returns the current velocity estimate.

type KalmanVelocity 

type KalmanVelocity struct {
	// contains filtered or unexported fields
}

KalmanVelocity implements a 2-state Kalman filter with position and velocity.

func NewKalmanVelocity 

func NewKalmanVelocity(cfg VelocityConfig) *KalmanVelocity

NewKalmanVelocity creates a new 2-state Kalman filter.

func NewKalmanVelocityWithInitial 

func NewKalmanVelocityWithInitial(cfg VelocityConfig, initialPos, initialVel float64) *KalmanVelocity

NewKalmanVelocityWithInitial creates a filter with initial state.

func (*KalmanVelocity) Covariance 

func (k *KalmanVelocity) Covariance() float64

Covariance returns the position variance (uncertainty squared).

func (*KalmanVelocity) GetStats 

func (k *KalmanVelocity) GetStats() VelocityStats

GetStats returns current filter statistics.

func (*KalmanVelocity) Observations 

func (k *KalmanVelocity) Observations() int

Observations returns the number of measurements processed.

func (*KalmanVelocity) Position 

func (k *KalmanVelocity) Position() float64

Position returns the current position estimate (alias for State).

func (*KalmanVelocity) Predict 

func (k *KalmanVelocity) Predict(steps int) float64

Predict estimates the state n steps into the future.

func (*KalmanVelocity) PredictIfEnabled 

func (k *KalmanVelocity) PredictIfEnabled(feature string, steps int) config.FilteredValue

PredictIfEnabled returns prediction if enabled.

func (*KalmanVelocity) PredictWithUncertainty 

func (k *KalmanVelocity) PredictWithUncertainty(steps int) (position, uncertainty float64)

PredictWithUncertainty returns predicted position and its uncertainty.

func (*KalmanVelocity) Process 

func (k *KalmanVelocity) Process(measurement float64) float64

Process updates the filter with a new measurement. Returns the filtered position estimate.

func (*KalmanVelocity) ProcessBatch 

func (k *KalmanVelocity) ProcessBatch(measurements []float64) []float64

ProcessBatch processes multiple measurements efficiently.

func (*KalmanVelocity) ProcessIfEnabled 

func (k *KalmanVelocity) ProcessIfEnabled(feature string, measurement float64) config.FilteredValue

ProcessIfEnabled applies filtering if the feature is enabled.

func (*KalmanVelocity) Reset 

func (k *KalmanVelocity) Reset()

Reset resets the filter to initial state.

func (*KalmanVelocity) SetState 

func (k *KalmanVelocity) SetState(pos, vel float64)

SetState manually sets position and velocity.

func (*KalmanVelocity) State 

func (k *KalmanVelocity) State() float64

State returns the current position estimate.

func (*KalmanVelocity) Velocity 

func (k *KalmanVelocity) Velocity() float64

Velocity returns the current velocity estimate.

func (*KalmanVelocity) VelocityCovariance 

func (k *KalmanVelocity) VelocityCovariance() float64

VelocityCovariance returns the velocity variance.

type Stats 

type Stats struct {
	State            float64
	Velocity         float64
	Covariance       float64
	Gain             float64
	MeasurementNoise float64
	Observations     int
}

Stats returns filter statistics.

type VarianceTracker 

type VarianceTracker struct {
	// contains filtered or unexported fields
}

VarianceTracker tracks signal variance for adaptive filtering. Based on imu-f's update_kalman_covariance.

func NewVarianceTracker 

func NewVarianceTracker(windowSize int) *VarianceTracker

NewVarianceTracker creates a variance tracker with the specified window size.

func (*VarianceTracker) AdaptiveNoise 

func (v *VarianceTracker) AdaptiveNoise(scale float64) float64

AdaptiveNoise returns a noise value based on current variance.

func (*VarianceTracker) Mean 

func (v *VarianceTracker) Mean() float64

Mean returns the current mean.

func (*VarianceTracker) StdDev 

func (v *VarianceTracker) StdDev() float64

StdDev returns the current standard deviation.

func (*VarianceTracker) Update 

func (v *VarianceTracker) Update(sample float64)

Update adds a new sample and updates variance statistics.

func (*VarianceTracker) Variance 

func (v *VarianceTracker) Variance() float64

Variance returns the current variance.

type VelocityConfig 

type VelocityConfig struct {
	// ProcessNoisePos - uncertainty in position prediction
	ProcessNoisePos float64

	// ProcessNoiseVel - uncertainty in velocity prediction
	ProcessNoiseVel float64

	// MeasurementNoise - uncertainty in measurements
	MeasurementNoise float64

	// InitialPosVariance - initial uncertainty in position
	InitialPosVariance float64

	// InitialVelVariance - initial uncertainty in velocity
	InitialVelVariance float64

	// Dt - time step between measurements (default: 1.0)
	Dt float64
}

VelocityConfig holds configuration for the 2-state Kalman filter.

func AggressiveTrackingConfig 

func AggressiveTrackingConfig() VelocityConfig

AggressiveTrackingConfig returns config for fast-changing signals.

func DefaultVelocityConfig 

func DefaultVelocityConfig() VelocityConfig

DefaultVelocityConfig returns sensible defaults for trend tracking.

func TemporalTrackingConfig 

func TemporalTrackingConfig() VelocityConfig

TemporalTrackingConfig returns config optimized for temporal pattern tracking.

type VelocityStats 

type VelocityStats struct {
	Position         float64
	Velocity         float64
	PositionVariance float64
	VelocityVariance float64
	CrossCovariance  float64
	MeasurementNoise float64
	Observations     int
}

VelocityStats returns detailed filter statistics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.