Documentation
¶
Overview ¶
Package sampling provides load-adaptive sampling utilities.
Overview ¶
This package prevents the "Observer Effect" where 100% logging/tracing causes performance degradation. The AdaptiveSampler dynamically adjusts sampling rate based on system load (measured by latency).
Components ¶
- AdaptiveSampler: Interface for load-adaptive sampling
- DefaultAdaptiveSampler: Production implementation
- HeadSampler: Samples only first N items
- RateLimitedSampler: Samples at most N items per second
Example Usage ¶
sampler := sampling.NewAdaptiveSampler(sampling.DefaultSamplingConfig())
defer sampler.Stop()
// In request handler:
if sampler.ShouldSample() {
trace.Start()
defer trace.End()
}
// Record latency for adaptive behavior:
sampler.RecordLatency(time.Since(start))
Thread Safety ¶
All types in this package are safe for concurrent use.
Index ¶
- func AlwaysSampleError(sampler AdaptiveSampler, isError bool) bool
- func HeadSampler(n int) func() bool
- func RateLimitedSampler(perSecond int) func() bool
- type AdaptiveSampler
- type Config
- type DefaultAdaptiveSampler
- func (s *DefaultAdaptiveSampler) ForceEnable(duration time.Duration)
- func (s *DefaultAdaptiveSampler) GetSamplingRate() float64
- func (s *DefaultAdaptiveSampler) RecordLatency(latency time.Duration)
- func (s *DefaultAdaptiveSampler) SetBaseSamplingRate(rate float64)
- func (s *DefaultAdaptiveSampler) ShouldSample() bool
- func (s *DefaultAdaptiveSampler) ShouldSampleContext(ctx context.Context) (context.Context, bool)
- func (s *DefaultAdaptiveSampler) Stats() SamplerStats
- func (s *DefaultAdaptiveSampler) Stop()
- type SamplerStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AlwaysSampleError ¶
func AlwaysSampleError(sampler AdaptiveSampler, isError bool) bool
AlwaysSampleError is a helper that always samples errors.
Description ¶
Wraps ShouldSample to always return true for errors. Useful when you want to ensure all errors are captured.
Inputs ¶
- sampler: The sampler to use
- isError: Whether this is an error case
Outputs ¶
- bool: True if should sample (always true for errors)
Example ¶
if AlwaysSampleError(sampler, err != nil) {
recordTrace()
}
Limitations ¶
- May increase sampling volume if errors are frequent
Assumptions ¶
- sampler is not nil
func HeadSampler ¶
HeadSampler creates a sampler that only samples the first N items.
Description ¶
Useful for sampling the start of a batch operation. Thread-safe via atomic counter.
Inputs ¶
- n: Number of items to sample (must be > 0)
Outputs ¶
- func(): Function that returns true for first N calls
Example ¶
sampler := HeadSampler(10)
for _, item := range items {
if sampler() {
log.Printf("Processing item: %v", item)
}
}
Limitations ¶
- Once N items are sampled, never samples again
Assumptions ¶
- N is positive; if N <= 0, returns a sampler that never samples
func RateLimitedSampler ¶
RateLimitedSampler creates a sampler with per-second rate limiting.
Description ¶
Samples at most N items per second, regardless of base rate. Returns a function for consistency with HeadSampler API.
Inputs ¶
- perSecond: Maximum samples per second (must be > 0)
Outputs ¶
- func() bool: Function that returns true if under rate limit
Example ¶
sampler := RateLimitedSampler(100) // Max 100 per second
for event := range events {
if sampler() {
processEvent(event)
}
}
Limitations ¶
- Window resets every second (not sliding window)
Assumptions ¶
- perSecond is positive; if <= 0, returns a sampler that never samples
Types ¶
type AdaptiveSampler ¶
type AdaptiveSampler interface {
// ShouldSample returns true if this item should be sampled.
ShouldSample() bool
// ShouldSampleContext returns true and records the decision in context.
ShouldSampleContext(ctx context.Context) (context.Context, bool)
// RecordLatency records a latency measurement for adaptive adjustment.
RecordLatency(latency time.Duration)
// GetSamplingRate returns the current sampling rate (0.0-1.0).
GetSamplingRate() float64
// SetBaseSamplingRate sets the base sampling rate.
SetBaseSamplingRate(rate float64)
// Stats returns current sampler statistics.
Stats() SamplerStats
// ForceEnable temporarily enables 100% sampling (for debugging).
ForceEnable(duration time.Duration)
}
AdaptiveSampler defines the interface for load-adaptive sampling.
Description ¶
AdaptiveSampler prevents the "Observer Effect" where 100% logging/tracing causes performance degradation. It dynamically adjusts sampling rate based on system load.
Thread Safety ¶
Implementations must be safe for concurrent use.
Example ¶
var sampler AdaptiveSampler = NewAdaptiveSampler(DefaultSamplingConfig())
if sampler.ShouldSample() {
trace.Start()
defer trace.End()
}
Limitations ¶
- Sampling is probabilistic; important events may be missed
- Rate adjustment has some latency
Assumptions ¶
- Caller records latencies for adaptive behavior
- Missing some samples is acceptable
type Config ¶
type Config struct {
// BaseSamplingRate is the default sampling rate (0.0-1.0).
// Default: 0.1 (10%)
BaseSamplingRate float64
// MinSamplingRate is the minimum rate even under high load.
// Default: 0.01 (1%)
MinSamplingRate float64
// MaxSamplingRate is the maximum rate under low load.
// Default: 1.0 (100%)
MaxSamplingRate float64
// LatencyThreshold triggers throttling when exceeded.
// Default: 100ms
LatencyThreshold time.Duration
// LatencyWindow is the window for averaging latency.
// Default: 1 minute
LatencyWindow time.Duration
// AdjustmentInterval is how often to recalculate rate.
// Default: 10 seconds
AdjustmentInterval time.Duration
// LatencyBufferSize is the number of latency samples to store in the ring buffer.
// A larger buffer provides more stable averages but uses more memory.
// Default: 1024
LatencyBufferSize int
}
Config configures the adaptive sampler.
Description ¶
Defines the parameters for adaptive sampling behavior.
Example ¶
config := Config{
BaseSamplingRate: 0.1, // 10% base rate
MinSamplingRate: 0.01, // Never go below 1%
MaxSamplingRate: 1.0, // Can go up to 100%
LatencyThreshold: 100 * time.Millisecond,
}
Limitations ¶
- All rate values must be in range [0.0, 1.0]
Assumptions ¶
- MinSamplingRate <= BaseSamplingRate <= MaxSamplingRate
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns sensible defaults.
Description ¶
Returns configuration with 10% base sampling rate that adapts based on system latency.
Inputs ¶
- None
Outputs ¶
- Config: Default configuration
Example ¶
config := DefaultConfig() config.BaseSamplingRate = 0.2 // Override to 20% sampler := NewAdaptiveSampler(config)
Limitations ¶
- Defaults may not be optimal for all workloads
Assumptions ¶
- Caller will tune for their specific use case
type DefaultAdaptiveSampler ¶
type DefaultAdaptiveSampler struct {
// contains filtered or unexported fields
}
DefaultAdaptiveSampler implements AdaptiveSampler.
Description ¶
Dynamically adjusts sampling rate based on system latency to prevent observability overhead from causing performance issues. When latency increases, sampling rate decreases automatically.
Use Cases ¶
- Trace sampling during high load
- Log sampling for high-volume paths
- Metric collection rate adjustment
Thread Safety ¶
DefaultAdaptiveSampler is safe for concurrent use.
Algorithm ¶
The sampler uses a simple proportional adjustment:
- If avg latency > threshold: rate = rate * (threshold / latency)
- If avg latency < threshold/2: rate = min(rate * 1.1, max)
Example ¶
sampler := NewAdaptiveSampler(DefaultConfig())
defer sampler.Stop()
// In request handler:
if sampler.ShouldSample() {
trace.Start()
defer trace.End()
}
// Record latency after request:
sampler.RecordLatency(time.Since(start))
Limitations ¶
- Uses simple random sampling (not reservoir)
- Latency window uses ring buffer (fixed size)
Assumptions ¶
- Caller calls Stop() on shutdown
- Latency recordings represent actual system load
func NewAdaptiveSampler ¶
func NewAdaptiveSampler(config Config) *DefaultAdaptiveSampler
NewAdaptiveSampler creates a new adaptive sampler.
Description ¶
Creates a sampler that automatically adjusts rate based on latency. Starts a background goroutine for periodic adjustment. Caller must call Stop() to clean up the background goroutine.
Inputs ¶
- config: Configuration for sampling behavior
Outputs ¶
- *DefaultAdaptiveSampler: New sampler
Example ¶
sampler := NewAdaptiveSampler(Config{
BaseSamplingRate: 0.2,
LatencyThreshold: 50 * time.Millisecond,
})
defer sampler.Stop()
Limitations ¶
- Starts a background goroutine that must be stopped
Assumptions ¶
- Caller will call Stop() on shutdown
func (*DefaultAdaptiveSampler) ForceEnable ¶
func (s *DefaultAdaptiveSampler) ForceEnable(duration time.Duration)
ForceEnable temporarily enables 100% sampling.
Description ¶
Forces 100% sampling for a specified duration. Useful for debugging specific issues. Thread-safe: uses mutex to ensure both forceEnabled and forceUntil are updated atomically.
Inputs ¶
- duration: How long to force 100% sampling
Outputs ¶
- None
Example ¶
sampler.ForceEnable(5 * time.Minute) // 100% sampling for 5 minutes
Limitations ¶
- May cause performance degradation if duration is long
Assumptions ¶
- Duration is reasonable for the system load
func (*DefaultAdaptiveSampler) GetSamplingRate ¶
func (s *DefaultAdaptiveSampler) GetSamplingRate() float64
GetSamplingRate returns the current sampling rate.
Description ¶
Returns the current effective sampling rate (0.0-1.0).
Inputs ¶
- None (receiver only)
Outputs ¶
- float64: Current sampling rate
Example ¶
rate := sampler.GetSamplingRate()
log.Printf("Current sampling rate: %.1f%%", rate*100)
Limitations ¶
- Value may change immediately after return
Assumptions ¶
- Receiver is not nil
func (*DefaultAdaptiveSampler) RecordLatency ¶
func (s *DefaultAdaptiveSampler) RecordLatency(latency time.Duration)
RecordLatency records a latency measurement.
Description ¶
Records latency for use in adaptive rate adjustment. Uses a ring buffer to maintain a rolling window of measurements.
Performance Note ¶
The latencyMu mutex serializes all latency recordings. On systems with very high request rates and many cores, this could become a contention point. If profiling reveals this as a bottleneck, consider using a buffered channel with a single consumer goroutine, or sharded mutexes. For most use cases (< 100k req/s), the current implementation is adequate.
Inputs ¶
- latency: The measured latency
Outputs ¶
- None
Example ¶
start := time.Now() doWork() sampler.RecordLatency(time.Since(start))
Limitations ¶
- High-frequency recording may cause mutex contention
Assumptions ¶
- Latency values are representative of system load
func (*DefaultAdaptiveSampler) SetBaseSamplingRate ¶
func (s *DefaultAdaptiveSampler) SetBaseSamplingRate(rate float64)
SetBaseSamplingRate sets the base sampling rate.
Description ¶
Updates the base rate. The actual rate may still be adjusted based on load. Thread-safe: uses mutex to protect config access.
Note: This does NOT immediately update currentRate. The background adjustLoop is the single source of truth for currentRate and will pick up the new BaseSamplingRate on its next tick. This prevents a race condition where adjustRate could overwrite the user's setting with a value calculated from stale data.
Inputs ¶
- rate: New base rate (0.0-1.0)
Outputs ¶
- None
Example ¶
sampler.SetBaseSamplingRate(0.5) // Set to 50%
Limitations ¶
- Rate change takes effect on next adjustment cycle
Assumptions ¶
- Rate is in valid range [0.0, 1.0]
func (*DefaultAdaptiveSampler) ShouldSample ¶
func (s *DefaultAdaptiveSampler) ShouldSample() bool
ShouldSample returns true if this item should be sampled.
Description ¶
Uses the current sampling rate to make a probabilistic decision. Thread-safe and fast (lock-free for the common path).
Inputs ¶
- None (receiver only)
Outputs ¶
- bool: True if item should be sampled
Example ¶
if sampler.ShouldSample() {
// Record trace, log, etc.
}
Limitations ¶
- Decision is probabilistic; results vary between calls
Assumptions ¶
- Receiver is not nil
func (*DefaultAdaptiveSampler) ShouldSampleContext ¶
ShouldSampleContext returns sampling decision and annotated context.
Description ¶
Same as ShouldSample but also stores the decision in context for consistent sampling of related operations. If a decision was already made (stored in context), it returns that decision without double-counting.
Inputs ¶
- ctx: Request context
Outputs ¶
- context.Context: Context with sampling decision
- bool: True if item should be sampled
Example ¶
ctx, sampled := sampler.ShouldSampleContext(ctx)
if sampled {
// All operations with this ctx will consistently sample
}
Limitations ¶
- Context must be propagated for consistent sampling
Assumptions ¶
- ctx is not nil
func (*DefaultAdaptiveSampler) Stats ¶
func (s *DefaultAdaptiveSampler) Stats() SamplerStats
Stats returns current sampler statistics.
Description ¶
Returns a snapshot of current sampling statistics.
Inputs ¶
- None (receiver only)
Outputs ¶
- SamplerStats: Current statistics
Example ¶
stats := sampler.Stats()
log.Printf("Sampled: %d, Dropped: %d, Rate: %.1f%%",
stats.TotalSampled, stats.TotalDropped, stats.CurrentRate*100)
Limitations ¶
- Values are point-in-time snapshot
Assumptions ¶
- Receiver is not nil
func (*DefaultAdaptiveSampler) Stop ¶
func (s *DefaultAdaptiveSampler) Stop()
Stop stops the sampler's background goroutine.
Description ¶
Stops the adjustment loop. Should be called on shutdown. Idempotent: safe to call multiple times without panic.
Inputs ¶
- None (receiver only)
Outputs ¶
- None
Example ¶
sampler := NewAdaptiveSampler(config) defer sampler.Stop()
Limitations ¶
- Sampler should not be used after Stop()
Assumptions ¶
- Receiver is not nil
type SamplerStats ¶
type SamplerStats struct {
// TotalSampled is the count of items that were sampled.
TotalSampled int64
// TotalDropped is the count of items that were not sampled.
TotalDropped int64
// CurrentRate is the current sampling rate.
CurrentRate float64
// AverageLatency is the rolling average latency.
AverageLatency time.Duration
// IsThrottled indicates if sampling is reduced due to load.
IsThrottled bool
// ThrottleReason explains why sampling is reduced.
ThrottleReason string
// ForceEnabled indicates if force-enable is active.
ForceEnabled bool
}
SamplerStats contains sampling statistics.
Description ¶
Provides visibility into sampling behavior for monitoring.
Source Files