Documentation
¶
Overview ¶
Package analytics turns Goma Gateway's per-request event stream into minute-bucketed rollups (models.AnalyticsRollup) and answers the Traffic, Performance and Web Analytics queries over them. It holds no per-request rows and no PII: latency is a histogram, uniques are a HyperLogLog sketch, and the visitor id is a daily-salted hash produced at the edge (never an IP).
Index ¶
- Constants
- Variables
- func IsBotUA(ua string) bool
- func Merge(dst, src *models.AnalyticsRollup)
- func MergeUniques(sketches [][]byte) int64
- func Percentile(hist []int64, p float64) float64
- func UniquesOf(data []byte) int64
- func WorkspaceIDFromRoute(name string) uint
- type Aggregator
- type Category
- type Event
- type LiveTracker
- type LiveVisit
- type Performance
- type Report
- type RouteStat
- type SeriesPoint
- type StatusBreak
- type Summary
- type Totals
- type WebAnalytics
- type Window
Constants ¶
const DefaultLiveWindow = 5 * time.Minute
DefaultLiveWindow is how far back a visitor still counts as "live" — the convention analytics products settled on (Plausible, Cloudflare, Fathom).
const SummaryTopCountries = 8
Variables ¶
var LatencyBoundsMs = []int64{5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}
LatencyBoundsMs are the histogram bucket upper bounds (milliseconds). A value is counted in the first bucket whose bound it does not exceed; anything above the last bound lands in a trailing overflow bucket (index len(bounds)).
Functions ¶
func IsBotUA ¶ added in v1.7.0
classifyUA derives a coarse browser family, OS and device from a User-Agent string with cheap substring matching — enough for the analytics breakdowns, no dependency, no fingerprinting beyond family. bot is true for common crawlers. IsBotUA reports whether a user agent looks automated, using the same rule as the rollups' bot/human split — exported so callers outside the aggregator (live visitors) classify traffic identically.
func Merge ¶
func Merge(dst, src *models.AnalyticsRollup)
Merge adds src into dst: counters, element-wise histograms, sums, top-K maps, and the HLL sketch. Used by the repository to combine a flushed bucket with an existing row (multi-consumer / re-flush safe).
func MergeUniques ¶
MergeUniques merges several serialized sketches and returns the estimate — the unique-visitor count over a range is the merge of its buckets' sketches.
func Percentile ¶
Percentile estimates the p-th percentile (0..1) latency in ms from a histogram, returning each bucket's upper bound (the overflow bucket returns the last bound as a floor). Approximate by design — bucketed, cheap, and mergeable.
func WorkspaceIDFromRoute ¶
WorkspaceIDFromRoute parses the workspace id out of a Goma route name of the form "mb-ws<id>-<slug>". Returns 0 when the name isn't in that form (a route Miabi doesn't own — e.g. the platform gateway's own route).
Types ¶
type Aggregator ¶
type Aggregator struct {
// contains filtered or unexported fields
}
Aggregator accumulates events into per-(workspace, app, route, minute) buckets in memory, and hands over "closed" buckets (older than a grace window) as models.AnalyticsRollup for persistence. Safe for concurrent Ingest/Flush.
func NewAggregator ¶
func NewAggregator() *Aggregator
func (*Aggregator) Flush ¶
func (a *Aggregator) Flush(before time.Time) []*models.AnalyticsRollup
Flush removes and returns every bucket strictly older than `before`, serializing its HLL sketch into VisitorsHLL. Call with now-grace so an in-progress minute keeps accumulating.
func (*Aggregator) Ingest ¶
func (a *Aggregator) Ingest(e *Event, ws, app uint)
Ingest folds one event into its bucket. ws/app are resolved by the caller (workspace from the route name, app via a route lookup); an unresolved app is 0.
func (*Aggregator) Pending ¶
func (a *Aggregator) Pending() int
Pending reports how many open buckets are held (for metrics/tests).
type Event ¶
type Event struct {
Ts int64 `json:"ts"` // unix millis
Gateway string `json:"gw"`
Route string `json:"name"` // "mb-ws<workspaceID>-<slug>"
Host string `json:"host"`
Method string `json:"method"`
Status int `json:"status"`
Path string `json:"path"`
PathTemplate string `json:"path_template"`
ReqBytes int64 `json:"req_bytes"`
RespBytes int64 `json:"resp_bytes"`
DurationMs int64 `json:"duration_ms"`
UpstreamMs int64 `json:"upstream_ms"`
VID string `json:"vid"`
Country string `json:"country"`
UA string `json:"ua"`
RefererHost string `json:"referer_host"`
}
Event is one request as emitted by Goma onto the analytics stream. Field tags mirror the gateway's AnalyticsEvent JSON.
type LiveTracker ¶ added in v1.7.0
type LiveTracker struct {
// contains filtered or unexported fields
}
LiveTracker answers "how many distinct visitors are active right now", from the same Goma event stream the rollups are built from. It is separate from the gateway's own visitor tracker, which is a single gateway-wide Prometheus gauge: this one is per workspace and per app. Both key on the event's daily-salted visitor id, so no client address is stored.
One sorted set per scope — member = visitor id, score = last-seen unix seconds — so a count is a range query and expiry a range delete, both O(log N). State lives in Redis rather than in the worker, so several workers (or control-plane replicas) agree on one number and it survives a restart.
Correctness comes from the score range on read, not from sweeping: a visitor outside the window is never counted even if their entry is still there. Sweep only bounds memory.
func NewLiveTracker ¶ added in v1.7.0
func NewLiveTracker(rdb *redis.Client, window time.Duration) *LiveTracker
func (*LiveTracker) Count ¶ added in v1.7.0
Count returns the distinct visitors active within the window for a scope (app 0 = the whole workspace). Exact, and unaffected by how recently a sweep ran, because the window is applied as a score range.
func (*LiveTracker) Sweep ¶ added in v1.7.0
func (t *LiveTracker) Sweep(ctx context.Context)
Sweep drops visitors who have fallen outside the window from every scope this process writes to. Purely housekeeping — counts are already window-correct — so a failure is logged by the caller at most, never fatal.
func (*LiveTracker) Touch ¶ added in v1.7.0
func (t *LiveTracker) Touch(ctx context.Context, visits []LiveVisit)
Touch records a batch of sightings as active now. Callers batch by their natural boundary (one stream read) so this is a handful of round trips per batch rather than one per event. Best-effort: a Redis failure costs a live count, never an ingest.
func (*LiveTracker) Window ¶ added in v1.7.0
func (t *LiveTracker) Window() time.Duration
Window is the "active within" period the counts describe.
type LiveVisit ¶ added in v1.7.0
LiveVisit is one visitor sighting: the scope it belongs to, and who.
type Performance ¶
type Performance struct {
RequestP50 float64 `json:"request_p50_ms"`
RequestP95 float64 `json:"request_p95_ms"`
RequestP99 float64 `json:"request_p99_ms"`
UpstreamP50 float64 `json:"upstream_p50_ms"`
UpstreamP95 float64 `json:"upstream_p95_ms"`
UpstreamP99 float64 `json:"upstream_p99_ms"`
// Mean total, upstream (backend) and gateway overhead (total − upstream) in ms.
AvgRequestMs float64 `json:"avg_request_ms"`
AvgUpstreamMs float64 `json:"avg_upstream_ms"`
AvgOverheadMs float64 `json:"avg_overhead_ms"`
SlowRoutes []RouteStat `json:"slow_routes"`
}
Performance covers the latency pillar: request vs upstream percentiles plus the slowest routes by p95.
type Report ¶
type Report struct {
Range Window `json:"range"`
Granularity string `json:"granularity"` // "minute" | "hour" | "day"
Totals Totals `json:"totals"`
Series []SeriesPoint `json:"series"`
Status StatusBreak `json:"status"`
Performance Performance `json:"performance"`
Web WebAnalytics `json:"web"`
// Compare holds the totals of the immediately preceding, equal-length window,
// so the UI can render period-over-period deltas. Nil when not requested.
Compare *Totals `json:"compare,omitempty"`
// RetentionDays is the effective retention cap in days (-1 = unlimited), and
// Exportable reports whether analytics export is entitled. Both are set by the
// handler from the license so the UI can bound the range picker and show the
// Enterprise upgrade hints. They are edition metadata, not computed from rows.
RetentionDays int `json:"retention_days"`
Exportable bool `json:"exportable"`
}
Report is the combined analytics answer for a workspace (optionally one app) over a time range: the Traffic, Performance and Web Analytics pillars share one pass over the same rollups so the three dashboard tabs come from a single query.
func BuildReport ¶
func BuildReport(rows []models.AnalyticsRollup, since, until time.Time) Report
BuildReport reduces the raw minute rollups of a range into the combined dashboard report. granularity is auto-selected from the span. It performs one pass for the totals/status/web breakdowns and a grouped pass for the series, merging latency histograms for percentiles and HLL sketches for uniques.
type RouteStat ¶
type RouteStat struct {
Route string `json:"route"`
Requests int64 `json:"requests"`
P95 float64 `json:"p95_latency_ms"`
ErrorRate float64 `json:"error_rate"`
}
RouteStat is a per-route performance summary (used for the slowest-routes list).
type SeriesPoint ¶
type SeriesPoint struct {
T time.Time `json:"t"`
Requests int64 `json:"requests"`
BytesIn int64 `json:"bytes_in"`
BytesOut int64 `json:"bytes_out"`
Errors int64 `json:"errors"` // 4xx+5xx (kept for compatibility)
Errors4xx int64 `json:"errors_4xx"` // client errors
Errors5xx int64 `json:"errors_5xx"` // server errors
Uniques int64 `json:"unique_visitors"`
AvgLatency float64 `json:"avg_latency_ms"`
P95Latency float64 `json:"p95_latency_ms"`
}
SeriesPoint is one time bucket of the request/latency time series at the chosen granularity.
type StatusBreak ¶
type Summary ¶ added in v1.7.0
type Summary struct {
Range Window `json:"range"`
Granularity string `json:"granularity"`
Totals Totals `json:"totals"`
Series []SeriesPoint `json:"series"`
Status StatusBreak `json:"status"`
// TopCountries is the SummaryTopCountries busiest origins, for the dashboard's
// country panel. Empty when the gateway has no GeoIP database — which is
// indistinguishable here from "no traffic", so the panel simply doesn't render.
TopCountries []Category `json:"top_countries"`
// Compare holds the totals of the immediately preceding, equal-length window,
// for period-over-period deltas. Nil when not requested.
Compare *Totals `json:"compare,omitempty"`
}
Summary is the dashboard's slice of a report: the headline totals, the status split, the request series and the top countries. It deliberately omits the remaining categorical breakdowns, per-route stats and upstream percentiles the analytics pages need, so the dashboard's traffic card doesn't pay for them.
func BuildSummary ¶ added in v1.7.0
func BuildSummary(rows []models.AnalyticsRollup, since, until time.Time) Summary
BuildSummary reduces rollups to the dashboard's view of a range. Feed it rows from AnalyticsRepository.RangeSummary — it reads only the counters, the latency histogram, the visitor sketch and the country top-K.
type Totals ¶
type Totals struct {
Requests int64 `json:"requests"`
BytesIn int64 `json:"bytes_in"`
BytesOut int64 `json:"bytes_out"`
UniqueVisit int64 `json:"unique_visitors"`
ErrorRate float64 `json:"error_rate"` // (4xx+5xx)/requests, 0..1
AvgLatency float64 `json:"avg_latency_ms"`
P50Latency float64 `json:"p50_latency_ms"`
P95Latency float64 `json:"p95_latency_ms"`
P99Latency float64 `json:"p99_latency_ms"`
}
Totals are the headline traffic numbers over the whole range.
type WebAnalytics ¶
type WebAnalytics struct {
UniqueVisitors int64 `json:"unique_visitors"`
BotRequests int64 `json:"bot_requests"`
HumanRequests int64 `json:"human_requests"`
TopPaths []Category `json:"top_paths"`
TopReferrers []Category `json:"top_referrers"`
TopCountries []Category `json:"top_countries"`
TopBrowsers []Category `json:"top_browsers"`
TopOS []Category `json:"top_os"`
TopDevices []Category `json:"top_devices"`
TopMethods []Category `json:"top_methods"`
}
WebAnalytics is the cookieless web pillar: uniques plus the categorical breakdowns (paths, referrers, countries, browsers, methods).