analytics

package
v1.7.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2026 License: AGPL-3.0 Imports: 11 Imported by: 0

Documentation

Overview

Package analytics turns Goma Gateway's per-request event stream into minute-bucketed rollups (models.AnalyticsRollup) and answers the Traffic, Performance and Web Analytics queries over them. It holds no per-request rows and no PII: latency is a histogram, uniques are a HyperLogLog sketch, and the visitor id is a daily-salted hash produced at the edge (never an IP).

Index

Constants

View Source
const DefaultLiveWindow = 5 * time.Minute

DefaultLiveWindow is how far back a visitor still counts as "live" — the convention analytics products settled on (Plausible, Cloudflare, Fathom).

View Source
const SummaryTopCountries = 8

Variables

View Source
var LatencyBoundsMs = []int64{5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}

LatencyBoundsMs are the histogram bucket upper bounds (milliseconds). A value is counted in the first bucket whose bound it does not exceed; anything above the last bound lands in a trailing overflow bucket (index len(bounds)).

Functions

func IsBotUA added in v1.7.0

func IsBotUA(ua string) bool

classifyUA derives a coarse browser family, OS and device from a User-Agent string with cheap substring matching — enough for the analytics breakdowns, no dependency, no fingerprinting beyond family. bot is true for common crawlers. IsBotUA reports whether a user agent looks automated, using the same rule as the rollups' bot/human split — exported so callers outside the aggregator (live visitors) classify traffic identically.

func Merge

func Merge(dst, src *models.AnalyticsRollup)

Merge adds src into dst: counters, element-wise histograms, sums, top-K maps, and the HLL sketch. Used by the repository to combine a flushed bucket with an existing row (multi-consumer / re-flush safe).

func MergeUniques

func MergeUniques(sketches [][]byte) int64

MergeUniques merges several serialized sketches and returns the estimate — the unique-visitor count over a range is the merge of its buckets' sketches.

func Percentile

func Percentile(hist []int64, p float64) float64

Percentile estimates the p-th percentile (0..1) latency in ms from a histogram, returning each bucket's upper bound (the overflow bucket returns the last bound as a floor). Approximate by design — bucketed, cheap, and mergeable.

func UniquesOf

func UniquesOf(data []byte) int64

UniquesOf returns the estimated unique visitors from a serialized sketch.

func WorkspaceIDFromRoute

func WorkspaceIDFromRoute(name string) uint

WorkspaceIDFromRoute parses the workspace id out of a Goma route name of the form "mb-ws<id>-<slug>". Returns 0 when the name isn't in that form (a route Miabi doesn't own — e.g. the platform gateway's own route).

Types

type Aggregator

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator accumulates events into per-(workspace, app, route, minute) buckets in memory, and hands over "closed" buckets (older than a grace window) as models.AnalyticsRollup for persistence. Safe for concurrent Ingest/Flush.

func NewAggregator

func NewAggregator() *Aggregator

func (*Aggregator) Flush

func (a *Aggregator) Flush(before time.Time) []*models.AnalyticsRollup

Flush removes and returns every bucket strictly older than `before`, serializing its HLL sketch into VisitorsHLL. Call with now-grace so an in-progress minute keeps accumulating.

func (*Aggregator) Ingest

func (a *Aggregator) Ingest(e *Event, ws, app uint)

Ingest folds one event into its bucket. ws/app are resolved by the caller (workspace from the route name, app via a route lookup); an unresolved app is 0.

func (*Aggregator) Pending

func (a *Aggregator) Pending() int

Pending reports how many open buckets are held (for metrics/tests).

type Category

type Category struct {
	Label string `json:"label"`
	Count int64  `json:"count"`
}

Category is one labelled count in a breakdown.

type Event

type Event struct {
	Ts           int64  `json:"ts"` // unix millis
	Gateway      string `json:"gw"`
	Route        string `json:"name"` // "mb-ws<workspaceID>-<slug>"
	Host         string `json:"host"`
	Method       string `json:"method"`
	Status       int    `json:"status"`
	Path         string `json:"path"`
	PathTemplate string `json:"path_template"`
	ReqBytes     int64  `json:"req_bytes"`
	RespBytes    int64  `json:"resp_bytes"`
	DurationMs   int64  `json:"duration_ms"`
	UpstreamMs   int64  `json:"upstream_ms"`
	VID          string `json:"vid"`
	Country      string `json:"country"`
	UA           string `json:"ua"`
	RefererHost  string `json:"referer_host"`
}

Event is one request as emitted by Goma onto the analytics stream. Field tags mirror the gateway's AnalyticsEvent JSON.

type LiveTracker added in v1.7.0

type LiveTracker struct {
	// contains filtered or unexported fields
}

LiveTracker answers "how many distinct visitors are active right now", from the same Goma event stream the rollups are built from. It is separate from the gateway's own visitor tracker, which is a single gateway-wide Prometheus gauge: this one is per workspace and per app. Both key on the event's daily-salted visitor id, so no client address is stored.

One sorted set per scope — member = visitor id, score = last-seen unix seconds — so a count is a range query and expiry a range delete, both O(log N). State lives in Redis rather than in the worker, so several workers (or control-plane replicas) agree on one number and it survives a restart.

Correctness comes from the score range on read, not from sweeping: a visitor outside the window is never counted even if their entry is still there. Sweep only bounds memory.

func NewLiveTracker added in v1.7.0

func NewLiveTracker(rdb *redis.Client, window time.Duration) *LiveTracker

func (*LiveTracker) Count added in v1.7.0

func (t *LiveTracker) Count(ctx context.Context, ws, app uint) (int64, error)

Count returns the distinct visitors active within the window for a scope (app 0 = the whole workspace). Exact, and unaffected by how recently a sweep ran, because the window is applied as a score range.

func (*LiveTracker) Sweep added in v1.7.0

func (t *LiveTracker) Sweep(ctx context.Context)

Sweep drops visitors who have fallen outside the window from every scope this process writes to. Purely housekeeping — counts are already window-correct — so a failure is logged by the caller at most, never fatal.

func (*LiveTracker) Touch added in v1.7.0

func (t *LiveTracker) Touch(ctx context.Context, visits []LiveVisit)

Touch records a batch of sightings as active now. Callers batch by their natural boundary (one stream read) so this is a handful of round trips per batch rather than one per event. Best-effort: a Redis failure costs a live count, never an ingest.

func (*LiveTracker) Window added in v1.7.0

func (t *LiveTracker) Window() time.Duration

Window is the "active within" period the counts describe.

type LiveVisit added in v1.7.0

type LiveVisit struct {
	Workspace uint
	App       uint
	VID       string
}

LiveVisit is one visitor sighting: the scope it belongs to, and who.

type Performance

type Performance struct {
	RequestP50  float64 `json:"request_p50_ms"`
	RequestP95  float64 `json:"request_p95_ms"`
	RequestP99  float64 `json:"request_p99_ms"`
	UpstreamP50 float64 `json:"upstream_p50_ms"`
	UpstreamP95 float64 `json:"upstream_p95_ms"`
	UpstreamP99 float64 `json:"upstream_p99_ms"`
	// Mean total, upstream (backend) and gateway overhead (total − upstream) in ms.
	AvgRequestMs  float64     `json:"avg_request_ms"`
	AvgUpstreamMs float64     `json:"avg_upstream_ms"`
	AvgOverheadMs float64     `json:"avg_overhead_ms"`
	SlowRoutes    []RouteStat `json:"slow_routes"`
}

Performance covers the latency pillar: request vs upstream percentiles plus the slowest routes by p95.

type Report

type Report struct {
	Range       Window        `json:"range"`
	Granularity string        `json:"granularity"` // "minute" | "hour" | "day"
	Totals      Totals        `json:"totals"`
	Series      []SeriesPoint `json:"series"`
	Status      StatusBreak   `json:"status"`
	Performance Performance   `json:"performance"`
	Web         WebAnalytics  `json:"web"`
	// Compare holds the totals of the immediately preceding, equal-length window,
	// so the UI can render period-over-period deltas. Nil when not requested.
	Compare *Totals `json:"compare,omitempty"`
	// RetentionDays is the effective retention cap in days (-1 = unlimited), and
	// Exportable reports whether analytics export is entitled. Both are set by the
	// handler from the license so the UI can bound the range picker and show the
	// Enterprise upgrade hints. They are edition metadata, not computed from rows.
	RetentionDays int  `json:"retention_days"`
	Exportable    bool `json:"exportable"`
}

Report is the combined analytics answer for a workspace (optionally one app) over a time range: the Traffic, Performance and Web Analytics pillars share one pass over the same rollups so the three dashboard tabs come from a single query.

func BuildReport

func BuildReport(rows []models.AnalyticsRollup, since, until time.Time) Report

BuildReport reduces the raw minute rollups of a range into the combined dashboard report. granularity is auto-selected from the span. It performs one pass for the totals/status/web breakdowns and a grouped pass for the series, merging latency histograms for percentiles and HLL sketches for uniques.

type RouteStat

type RouteStat struct {
	Route     string  `json:"route"`
	Requests  int64   `json:"requests"`
	P95       float64 `json:"p95_latency_ms"`
	ErrorRate float64 `json:"error_rate"`
}

RouteStat is a per-route performance summary (used for the slowest-routes list).

type SeriesPoint

type SeriesPoint struct {
	T          time.Time `json:"t"`
	Requests   int64     `json:"requests"`
	BytesIn    int64     `json:"bytes_in"`
	BytesOut   int64     `json:"bytes_out"`
	Errors     int64     `json:"errors"`     // 4xx+5xx (kept for compatibility)
	Errors4xx  int64     `json:"errors_4xx"` // client errors
	Errors5xx  int64     `json:"errors_5xx"` // server errors
	Uniques    int64     `json:"unique_visitors"`
	AvgLatency float64   `json:"avg_latency_ms"`
	P95Latency float64   `json:"p95_latency_ms"`
}

SeriesPoint is one time bucket of the request/latency time series at the chosen granularity.

type StatusBreak

type StatusBreak struct {
	S2xx int64 `json:"s2xx"`
	S3xx int64 `json:"s3xx"`
	S4xx int64 `json:"s4xx"`
	S5xx int64 `json:"s5xx"`
}

type Summary added in v1.7.0

type Summary struct {
	Range       Window        `json:"range"`
	Granularity string        `json:"granularity"`
	Totals      Totals        `json:"totals"`
	Series      []SeriesPoint `json:"series"`
	Status      StatusBreak   `json:"status"`
	// TopCountries is the SummaryTopCountries busiest origins, for the dashboard's
	// country panel. Empty when the gateway has no GeoIP database — which is
	// indistinguishable here from "no traffic", so the panel simply doesn't render.
	TopCountries []Category `json:"top_countries"`
	// Compare holds the totals of the immediately preceding, equal-length window,
	// for period-over-period deltas. Nil when not requested.
	Compare *Totals `json:"compare,omitempty"`
}

Summary is the dashboard's slice of a report: the headline totals, the status split, the request series and the top countries. It deliberately omits the remaining categorical breakdowns, per-route stats and upstream percentiles the analytics pages need, so the dashboard's traffic card doesn't pay for them.

func BuildSummary added in v1.7.0

func BuildSummary(rows []models.AnalyticsRollup, since, until time.Time) Summary

BuildSummary reduces rollups to the dashboard's view of a range. Feed it rows from AnalyticsRepository.RangeSummary — it reads only the counters, the latency histogram, the visitor sketch and the country top-K.

type Totals

type Totals struct {
	Requests    int64   `json:"requests"`
	BytesIn     int64   `json:"bytes_in"`
	BytesOut    int64   `json:"bytes_out"`
	UniqueVisit int64   `json:"unique_visitors"`
	ErrorRate   float64 `json:"error_rate"` // (4xx+5xx)/requests, 0..1
	AvgLatency  float64 `json:"avg_latency_ms"`
	P50Latency  float64 `json:"p50_latency_ms"`
	P95Latency  float64 `json:"p95_latency_ms"`
	P99Latency  float64 `json:"p99_latency_ms"`
}

Totals are the headline traffic numbers over the whole range.

type WebAnalytics

type WebAnalytics struct {
	UniqueVisitors int64      `json:"unique_visitors"`
	BotRequests    int64      `json:"bot_requests"`
	HumanRequests  int64      `json:"human_requests"`
	TopPaths       []Category `json:"top_paths"`
	TopReferrers   []Category `json:"top_referrers"`
	TopCountries   []Category `json:"top_countries"`
	TopBrowsers    []Category `json:"top_browsers"`
	TopOS          []Category `json:"top_os"`
	TopDevices     []Category `json:"top_devices"`
	TopMethods     []Category `json:"top_methods"`
}

WebAnalytics is the cookieless web pillar: uniques plus the categorical breakdowns (paths, referrers, countries, browsers, methods).

type Window

type Window struct {
	Since time.Time `json:"since"`
	Until time.Time `json:"until"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL