observability

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2026 License: Apache-2.0 Imports: 34 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	LevelWarning  = "warning"
	LevelCritical = "critical"
)

Alert levels.

View Source 
const PersistErrors = true

PersistErrors, when true, persists every error trace (status >= 400) regardless of latency, so a fast 401/403/422/500 is still captured for debugging.

View Source 
const SlowTraceThresholdUS = 50_000

SlowTraceThresholdUS is the per-request duration above which a trace is persisted to slow_traces (Phase 1: a fixed 50ms, not a per-tenant p95).

View Source 
const TraceIDKey = contextKey("trace_id")

TraceIDKey is the context key under which the per-request trace id is stored.

Variables

This section is empty.

Functions

func AdminAuth 

func AdminAuth(adminKey string, next http.Handler) http.Handler

AdminAuth wraps next so it is reachable only with a matching X-Admin-Key header. This is the single gate shared by the /debug/* endpoints and /metrics.

func FilterHeaders 

func FilterHeaders(h http.Header) map[string]string

FilterHeaders flattens an http.Header into a name→value map, redacting the values of sensitive (credential) headers to "[Filtered]". Only the first value of each header is kept. Called only when a trace is persisted (off the hot path), so its allocation never taxes the 200-OK common case.

func NewTraceID 

func NewTraceID() string

NewTraceID returns a 16-hex-character trace id (8 random bytes, no hyphens), using crypto/rand and the stdlib only — no google/uuid. crypto/rand.Read in modern Go is backed by a fast userspace CSPRNG, so this stays cheap.

func ParseUserAgent 

func ParseUserAgent(ua string) (browser, os string)

ParseUserAgent extracts a coarse browser and OS from a User-Agent string using stdlib-only substring matching — no external dependency, ~1µs. Unknown values fall back to "Unknown".

Note: Android UAs also contain "Linux" (Android is Linux-based) and iOS UAs contain "like Mac OS X", so Android/iOS are matched BEFORE Linux/macOS.

func ShouldPersistTrace 

func ShouldPersistTrace(s Sample) bool

ShouldPersistTrace reports whether a request's trace should be written to slow_traces. A trace is persisted when it is either slow (DurUS above the threshold) or — when PersistErrors — an error response (Status >= 400).

(A third heuristic, "any span with dur_us == 0", was considered for detecting a prematurely-cut pipeline but rejected: sub-microsecond stages such as the RBAC map lookup legitimately round to 0µs, so it would persist almost every request. "How far the pipeline got" is instead conveyed by which spans are present — error paths now mark their own stage.)

func TraceIDFromCtx 

func TraceIDFromCtx(ctx context.Context) string

TraceIDFromCtx returns the trace id from ctx, or "" if none is set.

func WithSpanTracker 

func WithSpanTracker(ctx context.Context) context.Context

WithSpanTracker returns a context carrying a fresh SpanTracker.

func WithTraceID 

func WithTraceID(ctx context.Context, id string) context.Context

WithTraceID returns a context carrying the given trace id.

Types

type Alert 

type Alert struct {
	TenantID string
	Level    string // LevelWarning | LevelCritical
	Message  string
	BurnRate float64
	P95ms    float64
}

Alert is a single SLO-breach notification.

type Alerter 

type Alerter interface {
	Send(ctx context.Context, a Alert) error
}

Alerter delivers SLO alerts somewhere (Slack, a no-op sink, …).

func NewSlackAlerterFromEnv 

func NewSlackAlerterFromEnv() Alerter

NewSlackAlerterFromEnv returns a SlackAlerter when SLACK_WEBHOOK_URL is set, or a NoopAlerter otherwise — so the server always starts cleanly without alerting config.

type AnomalyDetector 

type AnomalyDetector struct {
	// contains filtered or unexported fields
}

AnomalyDetector flags requests whose latency is statistically anomalous using an exponentially-weighted moving average of mean and variance.

func NewAnomalyDetector 

func NewAnomalyDetector() *AnomalyDetector

func (*AnomalyDetector) GetCount 

func (d *AnomalyDetector) GetCount(tenantID string) int64

GetCount returns the total anomaly count for tenantID since startup.

func (*AnomalyDetector) IncrCounter 

func (d *AnomalyDetector) IncrCounter(tenantID string)

IncrCounter increments the anomaly count for tenantID by one.

func (*AnomalyDetector) Observe 

func (d *AnomalyDetector) Observe(tenantID string, ms float64) (bool, float64)

Observe updates the EWMA model for tenantID with a new latency sample (µs). Returns (true, z-score) when the sample is anomalous, (false, 0) otherwise.

func (*AnomalyDetector) RecentAnomalies 

func (d *AnomalyDetector) RecentAnomalies(tenantID string, n int) []AnomalyEvent

RecentAnomalies returns up to n of tenantID's most recent anomalies, newest first. Returns a non-nil empty slice when none are recorded, so it marshals to [] not null.

func (*AnomalyDetector) RecordAnomaly 

func (d *AnomalyDetector) RecordAnomaly(tenantID string, latencyUS, z float64)

RecordAnomaly increments the tenant's anomaly counter AND appends the event to its recent-anomaly ring. Call it (instead of IncrCounter) when Observe reports an anomaly: latencyUS is the offending request's latency and z the detected z-score. Off the common request path — runs only on a detection (z > zThreshold).

type AnomalyEvent 

type AnomalyEvent struct {
	TS        int64   `json:"ts"`         // unix microseconds (matches trace TS)
	LatencyUS float64 `json:"latency_us"` // request latency in microseconds
	ZScore    float64 `json:"z_score"`
}

AnomalyEvent is one detected latency anomaly, retained so the admin observability panel can show WHEN it happened, the request latency, and the z-score — data the detector already computes on detection but previously only logged + counted.

type Check 

type Check struct {
	Name     string
	URL      string
	Method   string
	Headers  map[string]string
	Expected int // expected HTTP status code; 0 = any 2xx
}

Check describes one synthetic health probe.

type CheckResult 

type CheckResult struct {
	Name      string    `json:"name"`
	Status    string    `json:"status"`
	LatencyMs int64     `json:"latency_ms"`
	LastCheck time.Time `json:"last_check"`
	LastError string    `json:"last_error,omitempty"`
	Uptime    float64   `json:"uptime_pct"`
	// contains filtered or unexported fields
}

CheckResult is the last known state of a synthetic check.

type CooldownAlerter 

type CooldownAlerter struct {
	// contains filtered or unexported fields
}

CooldownAlerter wraps another Alerter and drops repeat alerts for the same (tenant, level) pair that arrive within the cooldown window.

func NewCooldownAlerter 

func NewCooldownAlerter(inner Alerter, cooldown time.Duration) *CooldownAlerter

NewCooldownAlerter wraps inner so that each (tenant, level) fires at most once per cooldown.

func (*CooldownAlerter) Send 

func (c *CooldownAlerter) Send(ctx context.Context, a Alert) error

Send forwards to the inner Alerter unless an alert for the same (tenant, level) was sent less than cooldown ago, in which case it is suppressed (returns nil).

type DynamicCheck 

type DynamicCheck struct {
	Name    string
	Resolve func(ctx context.Context) (*Check, string)
}

DynamicCheck resolves its probe lazily on every tick. Resolve returns the Check to run, or nil plus a human-readable reason — reported as status "pending" instead of a failing probe. This exists for canaries whose target only makes sense once external state exists (e.g. an API canary on a fresh install, before any tenant is registered: probing would just spam 4xx for a route nobody created).

type ErrGroup 

type ErrGroup struct {
	Count     atomic.Int64
	FirstSeen int64
	LastSeen  atomic.Int64
	Message   string
	Stack     []Frame // symbolized only on first occurrence
	// contains filtered or unexported fields
}

ErrGroup aggregates repeated occurrences of the same logical error.

type ErrorCapture 

type ErrorCapture struct {
	TraceID   string  `json:"trace_id"`
	TenantID  string  `json:"tenant_id"`
	Route     string  `json:"route"`
	Method    string  `json:"method"`
	ErrMsg    string  `json:"error_msg"`
	Stack     []Frame `json:"stack"`
	UserID    string  `json:"user_id"`
	Role      string  `json:"role"`
	Timestamp int64   `json:"ts"`
}

ErrorCapture is a snapshot of a server error (HTTP 500) linked to the request's trace, so the stack trace can be shown next to the timeline. For client errors (4xx) the Stack is empty — those are not server bugs and never pay for runtime.Callers. UserID/Role/Route/Method are filled by the caller (the handler) because observability must not import auth (auth imports observability).

func CaptureError 

func CaptureError(ctx context.Context, err error) ErrorCapture

CaptureError snapshots the current goroutine's stack for a server error (500) and links it to the request's trace_id and tenant (from ctx). It symbolizes the stack at most once per call site; repeat occurrences reuse the cached frames with ZERO allocations.

The hit path captures only 3 PCs into a stack-local array used solely for the fingerprint (so nothing escapes to the heap); the expensive full-stack capture + symbolization lives in captureAndSymbolize, called only on a cache miss, so its unavoidable PC-slice escape never taxes the common (repeat) path.

Returned BY VALUE so a caller that discards it allocates nothing. NEVER call on the happy path — even a hit pays for runtime.Callers.

type ErrorStore 

type ErrorStore struct {
	// contains filtered or unexported fields
}

ErrorStore deduplicates errors per tenant by fingerprinting type + message prefix + call site.

func NewErrorStore 

func NewErrorStore() *ErrorStore

func (*ErrorStore) Record 

func (es *ErrorStore) Record(tenantID string, err error)

Record fingerprints err, captures a cheap PC snapshot on every call, and symbolizes the stack exactly once per unique fingerprint via stackOnce.

func (*ErrorStore) TopN 

func (es *ErrorStore) TopN(tenantID string, n int) []map[string]any

TopN returns the n most-frequent error groups for tenantID, sorted by count descending.

type Frame 

type Frame struct {
	Function string `json:"function"`
	File     string `json:"file"`
	Line     int    `json:"line"`
}

Frame is a single symbolized stack frame.

type FullSnapshot 

type FullSnapshot struct {
	Cached   *PercentileSnapshot `json:"cached,omitempty"`
	Uncached *PercentileSnapshot `json:"uncached,omitempty"`
}

FullSnapshot splits latency into cache-hit vs cache-miss buckets.

type GeoLookup 

type GeoLookup struct {
	// contains filtered or unexported fields
}

GeoLookup resolves an IP to an ISO country code using an in-memory GeoLite2 database. Lookups are ~1µs and never touch the network. A GeoLookup whose database failed to load returns "" for everything — graceful degradation, so a missing/corrupt mmdb never blocks startup or panics.

func DefaultGeoLookup 

func DefaultGeoLookup() *GeoLookup

DefaultGeoLookup builds a lookup from the embedded GeoLite2 database.

func NewGeoLookup 

func NewGeoLookup(data []byte) *GeoLookup

NewGeoLookup builds a lookup from a GeoLite2-Country mmdb buffer. An empty or invalid buffer yields a usable no-op GeoLookup (Country → "") rather than an error.

func (*GeoLookup) Country 

func (g *GeoLookup) Country(ip string) string

Country returns the ISO-3166 country code for ip ("CO", "US", "MX", …), or "" when the database is unavailable, the ip is invalid/private/loopback, or no country is recorded.

type HistoryPoint 

type HistoryPoint struct {
	TS         int64   `json:"ts"`
	P50US      int64   `json:"p50_us"`
	P95US      int64   `json:"p95_us"`
	BurnRate   float64 `json:"burn_rate"`
	ErrorRatio float64 `json:"error_ratio"`
	SLOStatus  string  `json:"slo_status"`
}

HistoryPoint is the trimmed per-snapshot projection returned under "history". BurnRate/ErrorRatio come straight from the persisted snapshot so the panel can plot the SLO burn-rate over time (with the multi-window thresholds overlaid) without a second query.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds the four Prometheus collectors described in DEPLOYMENT_INFRA.md, backed by a dedicated registry rather than the global default. The dedicated registry keeps construction side-effect-free (safe to build more than once, e.g. in tests) and scopes /metrics output to exactly these series plus the standard Go/process collectors.

func NewMetrics 

func NewMetrics() *Metrics

NewMetrics constructs and registers all collectors.

func (*Metrics) Gatherer 

func (m *Metrics) Gatherer() prometheus.Gatherer

Gatherer exposes the dedicated registry as a prometheus.Gatherer. It enables end-to-end metric assertions (prometheus/testutil.GatherAndCompare) against exactly this Metrics' series, without reaching for the global default registry. Like Handler(), it carries no auth of its own.

func (*Metrics) Handler 

func (m *Metrics) Handler() http.Handler

Handler returns the Prometheus exposition handler scoped to this registry. It carries no auth of its own — mount it behind the admin-key middleware.

func (*Metrics) IncGoroutinePanic 

func (m *Metrics) IncGoroutinePanic()

IncGoroutinePanic records one panic recovered inside a Ctx.SafeGo goroutine.

func (*Metrics) IncRequestPanic 

func (m *Metrics) IncRequestPanic()

IncRequestPanic records one panic recovered by the request-chain Recoverer.

func (*Metrics) ObserveMigration 

func (m *Metrics) ObserveMigration(tenantID, status string, durationSeconds float64)

ObserveMigration records the duration and outcome of a tenant migration.

func (*Metrics) ObserveRequest 

func (m *Metrics) ObserveRequest(tenantID, method, path, status string, durationSeconds float64)

ObserveRequest records one served request: increments the counter and observes the duration histogram. path should be the chi route pattern (e.g. "/api/{entity}") rather than the raw URL path, to keep label cardinality bounded.

func (*Metrics) SetActiveTenants 

func (m *Metrics) SetActiveTenants(n int)

SetActiveTenants updates the gauge of tenants currently loaded in cache.

type MinuteBucket 

type MinuteBucket struct {
	Count  uint32
	Errors uint32
	SumUS  uint32
}

MinuteBucket is one minute of aggregated traffic. Three uint32 fields, stored in a fixed array — no per-minute allocations.

type NoopAlerter 

type NoopAlerter struct{}

NoopAlerter discards alerts (used when no SLACK_WEBHOOK_URL is configured).

func (NoopAlerter) Send 

func (NoopAlerter) Send(_ context.Context, a Alert) error

Send logs that the alert was suppressed and returns nil.

type ObsServer 

type ObsServer struct {
	// contains filtered or unexported fields
}

ObsServer exposes internal observability data over HTTP.

func NewObsServer 

func NewObsServer(
	hist *TenantHistogram,
	errors *ErrorStore,
	anomaly *AnomalyDetector,
	synthmon *SyntheticMonitor,
	rings *Rings,
	slo *SLOEngine,
	store *ObsStore,
) *ObsServer

func (*ObsServer) DebugRouter 

func (s *ObsServer) DebugRouter(adminKey string) http.Handler

DebugRouter returns an admin-gated sub-router serving /tenant/{id} and /synthetic. Mount it at "/debug" on any router (the main :8080 server or the control plane).

func (*ObsServer) Router 

func (s *ObsServer) Router(adminKey string) *chi.Mux

Router returns a chi.Mux exposing the /debug/... paths, admin-protected. Mount at "/" on an existing router (used by the control plane on :9090).

func (*ObsServer) ServeTenantData 

func (s *ObsServer) ServeTenantData(w http.ResponseWriter, r *http.Request)

ServeTenantData serves the per-tenant observability JSON (the same payload as the admin-gated GET /debug/tenant/{id}), reading the tenant id from the chi "id" URL param. It performs NO authorization itself — the caller (e.g. the admin API in pkg/platformadmin) is responsible for authorizing the request first (platform super-admin → any tenant; tenant admin → its own). This exists so the observability logic is REUSED with a different authorization gate, never duplicated. The data is already tenant-scoped (filtered by tenant_id), so no cross-tenant data can leak through it (R5 of the persistence audit).

func (*ObsServer) SetGeo 

func (s *ObsServer) SetGeo(g *GeoLookup)

SetGeo installs the GeoLite2 lookup used to enrich recent_traces with country.

func (*ObsServer) SetTracesHandler 

func (s *ObsServer) SetTracesHandler(h http.Handler)

SetTracesHandler installs the handler served at /debug/traces. It is injected from package main (where the HTML is go:embed'd and the admin key lives) so the route applies ITS OWN auth — query param (?key=) OR X-Admin-Key header — which is intentionally looser than the header-only gate on the JSON debug APIs, so the page can be opened directly in a browser.

type ObsStore 

type ObsStore struct {
	// contains filtered or unexported fields
}

ObsStore persists observability snapshots to a local SQLite file (modernc, no CGO).

func OpenStore 

func OpenStore(path string) (*ObsStore, error)

OpenStore opens (creating if needed) the SQLite store at path and ensures the schema exists. An empty path selects defaultObsDBPath (a persistent location). The parent directory is created if missing; if it cannot be created or written, the store falls back to an ephemeral temp file so observability still works — a bad path is logged as a WARNING, never a boot failure. A resolved path on an ephemeral filesystem (/tmp or a tmpfs) is also logged, so the operator knows the history will not survive a restart.

func (*ObsStore) Close 

func (s *ObsStore) Close() error

Close closes the underlying database.

func (*ObsStore) Flush 

func (s *ObsStore) Flush(tenantID string, snap Snapshot) error

Flush writes (or replaces) one snapshot for the given tenant.

func (*ObsStore) History 

func (s *ObsStore) History(tenantID string, hours int) ([]Snapshot, error)

History returns the tenant's snapshots from the last `hours`, newest first.

func (*ObsStore) Path 

func (s *ObsStore) Path() string

Path returns the resolved on-disk location the store opened. It may differ from the requested path when an unwritable directory forced the ephemeral fallback.

func (*ObsStore) Prune 

func (s *ObsStore) Prune() error

Prune enforces retention: it deletes snapshots and slow traces older than the retention window, AND caps slow_traces at maxSlowTraceRows (oldest dropped). It must be called periodically, not only at startup — otherwise a long-running server never reclaims space.

func (*ObsStore) SaveSlowTrace 

func (s *ObsStore) SaveSlowTrace(tenantID string, tv TraceView) error

SaveSlowTrace persists a single slow request's trace (id, route, total, span breakdown). tv.TS is the request start in unix microseconds, consistent with the ring's Sample.Start. Designed to be called asynchronously from the request path so it never blocks the response.

func (*ObsStore) SlowTraces 

func (s *ObsStore) SlowTraces(tenantID string, hours int) ([]TraceView, error)

SlowTraces returns the tenant's persisted slow traces from the last `hours`, newest first (capped at 200), with spans decoded back into TraceViews.

type PercentileSnapshot 

type PercentileSnapshot struct {
	P50Us  float64 `json:"p50_us"`
	P95Us  float64 `json:"p95_us"`
	P99Us  float64 `json:"p99_us"`
	P999Us float64 `json:"p999_us"`
	Count  int64   `json:"count"`
	Mean   float64 `json:"mean_us"`
}

PercentileSnapshot holds a point-in-time latency summary for one bucket. TenantID is intentionally absent — the response envelope provides it.

type RecentRequest 

type RecentRequest struct {
	StartUS int64  `json:"start_us"`
	DurUS   int32  `json:"dur_us"`
	QueryUS int32  `json:"query_us"`
	Route   string `json:"route"`
	Status  uint16 `json:"status"`
}

RecentRequest is the JSON-friendly projection of a Sample with its route id resolved back to a human-readable pattern.

type Rings 

type Rings struct {
	// contains filtered or unexported fields
}

Rings holds one TenantRing per tenant plus a route-pattern interner that maps variable-length route strings to compact uint16 ids stored in each Sample.

func NewRings 

func NewRings() *Rings

NewRings returns an empty registry ready for use.

func (*Rings) Count 

func (rs *Rings) Count() int

Count returns the number of distinct tenants that have recorded at least one request.

func (*Rings) Recent 

func (rs *Rings) Recent(tenantID string) []RecentRequest

Recent returns the tenant's recent requests as RecentRequest values, ordered most-recent first, with route ids resolved to strings.

func (*Rings) RecentTraces 

func (rs *Rings) RecentTraces(tenantID string, n int) []TraceView

RecentTraces returns up to n of the tenant's most recent requests projected as TraceViews (trace id resolved to hex, route resolved to string, spans copied out of the fixed array).

func (*Rings) Record 

func (rs *Rings) Record(tenantID string, s Sample)

Record appends a sample to the given tenant's ring, creating the ring on first use.

func (*Rings) RouteID 

func (rs *Rings) RouteID(route string) uint16

RouteID returns a stable uint16 id for route, interning it on first sight. Reads are lock-free-ish (RLock); only the first occurrence of a route takes the write lock. Ids saturate at the uint16 max to avoid overflow wraparound.

func (*Rings) RouteName 

func (rs *Rings) RouteName(id uint16) string

RouteName resolves an interned id back to its route string.

func (*Rings) Snapshot 

func (rs *Rings) Snapshot(tenantID string) []Sample

Snapshot returns the tenant's retained samples (most-recent first), or an empty slice when the tenant has no recorded requests.

func (*Rings) TenantIDs 

func (rs *Rings) TenantIDs() []string

TenantIDs returns the IDs of every tenant that has recorded at least one request. Used by the SLO engine to sweep all active tenants each tick.

type SLOConfig 

type SLOConfig struct {
	TargetUptime  float64       // e.g. 0.999 (99.9%)
	LatencySLOms  float64       // e.g. 100.0 ms — requests slower than this count as errors
	AlertCooldown time.Duration // e.g. 15min between repeat alerts of the same level
}

SLOConfig holds the per-tenant objectives and alert cadence.

func DefaultSLOConfig 

func DefaultSLOConfig() SLOConfig

DefaultSLOConfig is the objective applied to every tenant unless overridden.

type SLOEngine 

type SLOEngine struct {
	// contains filtered or unexported fields
}

SLOEngine rolls the per-tenant request ring up into minute buckets every 60s, evaluates multi-window burn rates, and fires alerts through alerter. It is fully independent of the AnomalyDetector.

func NewSLOEngine 

func NewSLOEngine(rings *Rings, hist *TenantHistogram, alerter Alerter) *SLOEngine

NewSLOEngine builds an engine with the default SLO config. The provided alerter is wrapped in a CooldownAlerter so each (tenant, level) fires at most once per cooldown.

func NewSLOEngineWithConfig 

func NewSLOEngineWithConfig(rings *Rings, hist *TenantHistogram, alerter Alerter, cfg SLOConfig) *SLOEngine

NewSLOEngineWithConfig is NewSLOEngine with an explicit config (used in tests to shrink the cooldown window).

func (*SLOEngine) Run 

func (e *SLOEngine) Run(ctx context.Context)

Run evaluates all tenants immediately, then every 60s until ctx is cancelled.

func (*SLOEngine) Snapshot 

func (e *SLOEngine) Snapshot(tid string) SLOSnapshot

Snapshot returns the current SLO state for a tenant (used by /debug and tests).

type SLOSnapshot 

type SLOSnapshot struct {
	ErrorRatio5m float64 `json:"error_ratio_5m"`
	BurnRate5m   float64 `json:"burn_rate_5m"`
	ErrorRatio1h float64 `json:"error_ratio_1h"`
	BurnRate1h   float64 `json:"burn_rate_1h"`
	Status       string  `json:"status"`
}

SLOSnapshot is the JSON projection exposed under /debug/tenant/{id} → "slo".

type Sample 

type Sample struct {
	Start   int64
	DurUS   int32
	QueryUS int32
	Route   uint16
	Status  uint16
	TraceID [8]byte
	Spans   [maxSpans]Span
	NSpans  uint8
	// ErrMsg is the error message for an errored request ("" otherwise).
	// ErrorCapture carries the symbolized stack for a 500 (nil otherwise) — a
	// pointer, so non-error samples cost only 8 bytes and Record stays alloc-free.
	ErrMsg       string
	ErrorCapture *ErrorCapture
	// Client context (raw): browser/OS are parsed and country geo-resolved lazily
	// in the projection, so the hot path only stores two strings.
	IP        string
	UserAgent string
}

Sample is a fixed-size record of a single request. All fields are value types so a Sample never escapes to the heap and TenantRing.Record stays allocation-free. 

Start   request start time, unix microseconds
DurUS   total request duration, microseconds
QueryUS time spent in DB queries, microseconds (0 when unmeasured)
Route   interned route-pattern id (resolve via Rings.RouteName)
Status  HTTP status code
TraceID first 8 bytes of the request trace id (16 hex chars decoded)
Spans   per-stage durations captured by the request's SpanTracker
NSpans  number of valid entries in Spans

Spans/TraceID are value types too, so Record stays allocation-free.

type SlackAlerter 

type SlackAlerter struct {
	// contains filtered or unexported fields
}

SlackAlerter posts alerts to a Slack incoming-webhook URL. The HTTP client is the shared SSRF-safe egress client (blocks loopback/private/ link-local and refuses redirects), so a misconfigured webhook can't be used to reach internal services.

func NewSlackAlerter 

func NewSlackAlerter(webhookURL string) *SlackAlerter

NewSlackAlerter builds a SlackAlerter posting to webhookURL with a 5s timeout.

func (*SlackAlerter) Send 

func (s *SlackAlerter) Send(ctx context.Context, a Alert) error

Send posts the formatted alert to Slack. A blank webhook URL is treated as a soft no-op (log + nil) rather than an error or panic.

type Snapshot 

type Snapshot struct {
	TenantID   string  `json:"tenant_id"`
	TS         int64   `json:"ts"` // unix seconds
	P50US      int64   `json:"p50_us"`
	P95US      int64   `json:"p95_us"`
	ErrorRatio float64 `json:"error_ratio"`
	BurnRate   float64 `json:"burn_rate"`
	SLOStatus  string  `json:"slo_status"`
}

Snapshot is one persisted point-in-time observation for a tenant.

type Span 

type Span struct {
	Name  string `json:"name"`
	DurUS int32  `json:"dur_us"`
}

Span is one timed stage within a request. Name is a constant literal (e.g. "jwt"), so storing it never allocates.

type SpanTracker 

type SpanTracker struct {

	// HasError/ErrMsg are set on ANY error response (4xx/5xx) via RecordError.
	// Capture is set only for 500s (carries the symbolized stack) via SetCapture.
	HasError bool
	ErrMsg   string
	Capture  *ErrorCapture
	// contains filtered or unexported fields
}

SpanTracker records up to maxSpans stage durations for a single request. It is a fixed array (not a slice) so Mark never allocates and the tracker never escapes beyond the one heap allocation made by NewSpanTracker. It is NOT safe for concurrent Mark calls — but a request is handled by one goroutine through the middleware chain, so the spans are marked sequentially.

func NewSpanTracker 

func NewSpanTracker() *SpanTracker

NewSpanTracker starts a tracker whose clock begins now.

func SpanTrackerFromCtx 

func SpanTrackerFromCtx(ctx context.Context) *SpanTracker

SpanTrackerFromCtx returns the SpanTracker in ctx, or nil if none is set. Callers must nil-check before calling Mark.

func (*SpanTracker) Finish 

func (t *SpanTracker) Finish() []Span

Finish records a final "done" span (the tail since the last mark) and returns the recorded spans. The returned slice is backed by the tracker's array — no allocation — and must be copied if it needs to outlive the tracker.

func (*SpanTracker) Mark 

func (t *SpanTracker) Mark(name string)

Mark records the elapsed time since the previous Mark (or since creation) as a span named name. Once maxSpans are recorded, further marks are ignored (the clock still advances), so a runaway caller can never overflow or panic.

func (*SpanTracker) RecordError 

func (t *SpanTracker) RecordError(msg string)

RecordError marks the request as errored and stores the error message. Cheap (no stack): used for client errors (401/403/422) and as the message for 500s.

func (*SpanTracker) SetCapture 

func (t *SpanTracker) SetCapture(c *ErrorCapture)

SetCapture attaches a symbolized error capture (the stack) to the request — used only for server errors (500). Implies HasError.

func (*SpanTracker) TotalUS 

func (t *SpanTracker) TotalUS() int32

TotalUS returns the sum of all recorded span durations.

type SyntheticMonitor 

type SyntheticMonitor struct {
	// contains filtered or unexported fields
}

SyntheticMonitor periodically probes a list of HTTP endpoints.

func NewSyntheticMonitor 

func NewSyntheticMonitor(checks []Check) *SyntheticMonitor

func (*SyntheticMonitor) AddDynamic 

func (sm *SyntheticMonitor) AddDynamic(d DynamicCheck)

AddDynamic registers a lazily-resolved check. Must be called before Start.

func (*SyntheticMonitor) Results 

func (sm *SyntheticMonitor) Results() map[string]*CheckResult

Results returns all current check results as a map keyed by check name.

func (*SyntheticMonitor) Start 

func (sm *SyntheticMonitor) Start(ctx context.Context, interval time.Duration)

Start launches the probe loop. Runs until ctx is cancelled.

type TenantHistogram 

type TenantHistogram struct {
	// contains filtered or unexported fields
}

TenantHistogram records request latency per tenant using HDR histograms, split by whether the response was served from cache.

func NewTenantHistogram 

func NewTenantHistogram() *TenantHistogram

func (*TenantHistogram) FullSnapshot 

func (th *TenantHistogram) FullSnapshot(tenantID string) *FullSnapshot

FullSnapshot returns the cached/uncached latency split for a tenant, or nil buckets when no data has been recorded for that category.

func (*TenantHistogram) Record 

func (th *TenantHistogram) Record(tenantID string, durationUs int64, fromCache bool)

Record adds a latency sample (in microseconds) for a given tenant. fromCache distinguishes cache hits (sub-ms) from full DB-backed responses.

func (*TenantHistogram) Snapshot 

func (th *TenantHistogram) Snapshot(tenantID string) *PercentileSnapshot

Snapshot returns the uncached latency snapshot. Kept for test backward-compatibility; prefer FullSnapshot in production paths.

type TenantRing 

type TenantRing struct {
	// contains filtered or unexported fields
}

TenantRing is a fixed circular buffer of the last ringSize samples for one tenant.

A mutex — not lock-free atomics — guards the buffer on purpose: once head laps the array, two writers separated by ringSize records target the SAME physical slot, so an atomic head only serializes index allocation, not the 24-byte Sample copy. Those concurrent same-slot writes are a genuine data race (caught by -race). The lock keeps Record allocation-free (a single array assignment, no slices).

func (*TenantRing) Record 

func (r *TenantRing) Record(s Sample)

Record stores s in the next slot, overwriting the oldest sample once the buffer is full. Allocation-free.

func (*TenantRing) Snapshot 

func (r *TenantRing) Snapshot() []Sample

Snapshot returns a copy of the retained samples ordered most-recent first. Returns a non-nil empty slice (not nil) when nothing has been recorded, so it marshals to a JSON [] rather than null.

type TraceView 

type TraceView struct {
	TraceID string  `json:"trace_id"`
	TS      int64   `json:"ts"` // request start, unix microseconds
	Route   string  `json:"route"`
	TotalUS int32   `json:"total_us"`
	Status  uint16  `json:"status"`
	Spans   []Span  `json:"spans"`
	ErrMsg  string  `json:"error_msg,omitempty"` // set for error responses
	Stack   []Frame `json:"stack,omitempty"`     // set only for 500s
	// Request context for the error panel — populated for recent_traces from the
	// in-memory ErrorCapture (slow_traces persists only error_msg + stack).
	Method string `json:"method,omitempty"`
	UserID string `json:"user_id,omitempty"`
	Role   string `json:"role,omitempty"`
	// Client context — IP, parsed browser/OS, and geo country.
	IP        string `json:"ip,omitempty"`
	UserAgent string `json:"user_agent,omitempty"`
	Browser   string `json:"browser,omitempty"`
	OS        string `json:"os,omitempty"`
	Country   string `json:"country,omitempty"`
	// Reproducible-request context (persisted traces only): filtered headers +
	// full URL. Method (above) completes the curl reconstruction.
	Headers map[string]string `json:"headers,omitempty"`
	FullURL string            `json:"full_url,omitempty"`
}

TraceView is the JSON projection of a Sample's trace: id, route, total, and the per-stage span breakdown. Served under /debug/tenant/{id} → "recent_traces".

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.