analytics

package
v1.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package analytics turns Goma Gateway's per-request event stream into minute-bucketed rollups and answers the Traffic, Performance and Web Analytics queries over them. It holds no per-request rows and no PII: latency is a histogram, uniques an HLL sketch, and the visitor id a daily-salted hash.

Index

Constants

View Source
const (
	MaxBatch = 500
	MaxSkew  = 15 * time.Minute
)

Ingest limits. MaxBatch bounds one POST; MaxSkew bounds how far a node's clock may be from the manager's before its timestamps are distrusted.

View Source
const DefaultLiveWindow = 5 * time.Minute

DefaultLiveWindow is how far back a visitor still counts as "live" — the convention analytics products settled on (Plausible, Cloudflare, Fathom).

View Source
const StreamEventField = "e"

StreamEventField is the stream entry field Goma writes each event under, and the only one the consumer reads. Anything appended under another name is dropped without a trace, so writers and the reader share this constant.

View Source
const SummaryTopCountries = 8

Variables

View Source
var (
	ErrBatchTooLarge = errors.New("batch exceeds the maximum size")
	ErrNoEvents      = errors.New("batch declares no events")
	ErrNoBatchID     = errors.New("batch id is required")
)
View Source
var LatencyBoundsMs = []int64{5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}

LatencyBoundsMs are the histogram bucket upper bounds (milliseconds). A value is counted in the first bucket whose bound it does not exceed; anything above the last bound lands in a trailing overflow bucket (index len(bounds)).

Functions

func IsBotUA added in v1.7.0

func IsBotUA(ua string) bool

IsBotUA reports whether a user agent looks automated, using the same rule as the rollups' bot/human split. Exported so callers outside the aggregator — live visitors — classify traffic identically.

func Merge

func Merge(dst, src *models.AnalyticsRollup)

Merge adds src into dst: counters, element-wise histograms, sums, top-K maps, and the HLL sketch. Used by the repository to combine a flushed bucket with an existing row (multi-consumer / re-flush safe).

func MergeUniques

func MergeUniques(sketches [][]byte) int64

MergeUniques merges several serialized sketches and returns the estimate — the unique-visitor count over a range is the merge of its buckets' sketches.

func Percentile

func Percentile(hist []int64, p float64) float64

Percentile estimates the p-th percentile (0..1) latency in ms from a histogram, returning each bucket's upper bound (the overflow bucket returns the last bound as a floor). Approximate by design — bucketed, cheap, and mergeable.

func UniquesOf

func UniquesOf(data []byte) int64

UniquesOf returns the estimated unique visitors from a serialized sketch.

func WorkspaceIDFromRoute

func WorkspaceIDFromRoute(name string) uint

WorkspaceIDFromRoute parses the workspace id out of a Goma route name of the form "mb-ws<id>-<slug>". Returns 0 when the name isn't in that form (a route Miabi doesn't own — e.g. the platform gateway's own route).

Types

type Aggregator

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator accumulates events into per-(workspace, app, route, minute) buckets in memory, and hands over "closed" buckets (older than a grace window) as models.AnalyticsRollup for persistence. Safe for concurrent Ingest/Flush.

func NewAggregator

func NewAggregator() *Aggregator

func (*Aggregator) Flush

func (a *Aggregator) Flush(before time.Time) []*models.AnalyticsRollup

Flush removes and returns every bucket strictly older than `before`, serializing its HLL sketch into VisitorsHLL. Call with now-grace so an in-progress minute keeps accumulating.

func (*Aggregator) Ingest

func (a *Aggregator) Ingest(e *Event, ws, app uint)

Ingest folds one event into its bucket. ws/app are resolved by the caller (workspace from the route name, app via a route lookup); an unresolved app is 0.

func (*Aggregator) Pending

func (a *Aggregator) Pending() int

Pending reports how many open buckets are held (for metrics/tests).

type Batch added in v1.8.2

type Batch struct {
	BatchID string  `json:"batch_id"`
	Events  []Event `json:"events"`
}

Batch is one node's forwarded run of gateway events.

type Category

type Category struct {
	Label string `json:"label"`
	Count int64  `json:"count"`
}

Category is one labelled count in a breakdown.

type Event

type Event struct {
	Ts           int64  `json:"ts"` // unix millis
	Gateway      string `json:"gw"`
	Route        string `json:"name"` // "mb-ws<workspaceID>-<slug>"
	Host         string `json:"host"`
	Method       string `json:"method"`
	Status       int    `json:"status"`
	Path         string `json:"path"`
	PathTemplate string `json:"path_template"`
	ReqBytes     int64  `json:"req_bytes"`
	RespBytes    int64  `json:"resp_bytes"`
	DurationMs   int64  `json:"duration_ms"`
	UpstreamMs   int64  `json:"upstream_ms"`
	VID          string `json:"vid"`
	Country      string `json:"country"`
	UA           string `json:"ua"`
	RefererHost  string `json:"referer_host"`
}

Event is one request as emitted by Goma onto the analytics stream. Field tags mirror the gateway's AnalyticsEvent JSON.

type IngestResult added in v1.8.2

type IngestResult struct {
	Accepted  int  `json:"accepted"`
	Duplicate bool `json:"duplicate"`
	// Rejected counts events dropped for a route the node does not serve.
	Rejected int `json:"rejected"`
	// Reclocked counts events whose timestamp was too far off to trust.
	Reclocked int `json:"reclocked"`
}

IngestResult reports what the manager did with a batch. Rejected events are counted rather than failing the batch: one bad event must not strand the rest.

type Ingester added in v1.8.2

type Ingester struct {
	// contains filtered or unexported fields
}

Ingester appends events forwarded by an edge node to the platform stream the AnalyticsConsumer reads, so edge traffic converges with the manager's.

func NewIngester added in v1.8.2

func NewIngester(rdb *redis.Client, owner RouteOwnership, stream string) *Ingester

func (*Ingester) Ingest added in v1.8.2

func (i *Ingester) Ingest(ctx context.Context, serverID uint, b Batch) (IngestResult, error)

Ingest validates a node's batch and appends the events it is allowed to report. Duplicate batches return the already-accepted result so the forwarder can ACK.

type LiveTracker added in v1.7.0

type LiveTracker struct {
	// contains filtered or unexported fields
}

LiveTracker answers "how many distinct visitors are active right now" from the same Goma event stream the rollups use, but per workspace and per app rather than one gateway-wide gauge. One sorted set per scope keys on the daily-salted visitor id, so counting and expiry are both O(log N).

func NewLiveTracker added in v1.7.0

func NewLiveTracker(rdb *redis.Client, window time.Duration) *LiveTracker

func (*LiveTracker) Count added in v1.7.0

func (t *LiveTracker) Count(ctx context.Context, ws, app uint) (int64, error)

Count returns the distinct visitors active within the window for a scope (app 0 = the whole workspace). Exact, and unaffected by how recently a sweep ran, because the window is applied as a score range.

func (*LiveTracker) Sweep added in v1.7.0

func (t *LiveTracker) Sweep(ctx context.Context)

Sweep drops visitors who have fallen outside the window from every scope this process writes to. Purely housekeeping — counts are already window-correct — so a failure is logged by the caller at most, never fatal.

func (*LiveTracker) Touch added in v1.7.0

func (t *LiveTracker) Touch(ctx context.Context, visits []LiveVisit)

Touch records a batch of sightings as active now. Callers batch by their natural boundary — one stream read — so this is a handful of round trips per batch rather than one per event. Best-effort: a Redis failure costs a live count, never an ingest.

func (*LiveTracker) Window added in v1.7.0

func (t *LiveTracker) Window() time.Duration

Window is the "active within" period the counts describe.

type LiveVisit added in v1.7.0

type LiveVisit struct {
	Workspace uint
	App       uint
	VID       string
}

LiveVisit is one visitor sighting: the scope it belongs to, and who.

type Performance

type Performance struct {
	RequestP50  float64 `json:"request_p50_ms"`
	RequestP95  float64 `json:"request_p95_ms"`
	RequestP99  float64 `json:"request_p99_ms"`
	UpstreamP50 float64 `json:"upstream_p50_ms"`
	UpstreamP95 float64 `json:"upstream_p95_ms"`
	UpstreamP99 float64 `json:"upstream_p99_ms"`
	// Mean total, upstream (backend) and gateway overhead (total − upstream) in ms.
	AvgRequestMs  float64     `json:"avg_request_ms"`
	AvgUpstreamMs float64     `json:"avg_upstream_ms"`
	AvgOverheadMs float64     `json:"avg_overhead_ms"`
	SlowRoutes    []RouteStat `json:"slow_routes"`
}

Performance covers the latency pillar: request vs upstream percentiles plus the slowest routes by p95.

type Report

type Report struct {
	Range       Window        `json:"range"`
	Granularity string        `json:"granularity"` // "minute" | "hour" | "day"
	Totals      Totals        `json:"totals"`
	Series      []SeriesPoint `json:"series"`
	Status      StatusBreak   `json:"status"`
	Performance Performance   `json:"performance"`
	Web         WebAnalytics  `json:"web"`
	// Compare holds the totals of the immediately preceding, equal-length window,
	// so the UI can render period-over-period deltas. Nil when not requested.
	Compare *Totals `json:"compare,omitempty"`
	// RetentionDays is the effective retention cap in days (-1 = unlimited), and Exportable reports whether
	// analytics export is entitled. Both are set by the handler from the license so the UI can bound the
	// range picker and show upgrade hints. They are edition metadata, not computed from rows.
	RetentionDays int  `json:"retention_days"`
	Exportable    bool `json:"exportable"`
}

Report is the combined analytics answer for a workspace (optionally one app) over a time range: the Traffic, Performance and Web Analytics pillars share one pass over the same rollups so the three dashboard tabs come from a single query.

func BuildReport

func BuildReport(rows []models.AnalyticsRollup, since, until time.Time) Report

BuildReport reduces the raw minute rollups of a range into the combined dashboard report, with granularity auto-selected from the span. One pass for the totals and breakdowns, a grouped pass for the series, merging latency histograms for percentiles and HLL sketches for uniques.

type RouteOwnership added in v1.8.2

type RouteOwnership interface {
	RouteNamesForNode(serverID uint) (map[string]bool, error)
}

RouteOwnership answers which Goma route names a node is allowed to report on.

type RouteStat

type RouteStat struct {
	Route     string  `json:"route"`
	Requests  int64   `json:"requests"`
	P95       float64 `json:"p95_latency_ms"`
	ErrorRate float64 `json:"error_rate"`
}

RouteStat is a per-route performance summary (used for the slowest-routes list).

type SeriesPoint

type SeriesPoint struct {
	T          time.Time `json:"t"`
	Requests   int64     `json:"requests"`
	BytesIn    int64     `json:"bytes_in"`
	BytesOut   int64     `json:"bytes_out"`
	Errors     int64     `json:"errors"`     // 4xx+5xx (kept for compatibility)
	Errors4xx  int64     `json:"errors_4xx"` // client errors
	Errors5xx  int64     `json:"errors_5xx"` // server errors
	Uniques    int64     `json:"unique_visitors"`
	AvgLatency float64   `json:"avg_latency_ms"`
	P95Latency float64   `json:"p95_latency_ms"`
}

SeriesPoint is one time bucket of the request/latency time series at the chosen granularity.

type StatusBreak

type StatusBreak struct {
	S2xx int64 `json:"s2xx"`
	S3xx int64 `json:"s3xx"`
	S4xx int64 `json:"s4xx"`
	S5xx int64 `json:"s5xx"`
}

type Summary added in v1.7.0

type Summary struct {
	Range       Window        `json:"range"`
	Granularity string        `json:"granularity"`
	Totals      Totals        `json:"totals"`
	Series      []SeriesPoint `json:"series"`
	Status      StatusBreak   `json:"status"`
	// TopCountries is the SummaryTopCountries busiest origins, for the dashboard's
	// country panel. Empty when the gateway has no GeoIP database — which is
	// indistinguishable here from "no traffic", so the panel simply doesn't render.
	TopCountries []Category `json:"top_countries"`
	// Compare holds the totals of the immediately preceding, equal-length window,
	// for period-over-period deltas. Nil when not requested.
	Compare *Totals `json:"compare,omitempty"`
}

Summary is the dashboard's slice of a report: headline totals, the status split, the request series and the top countries. It deliberately omits the remaining breakdowns, per-route stats and upstream percentiles the analytics pages need, so the dashboard's traffic card doesn't pay for them.

func BuildSummary added in v1.7.0

func BuildSummary(rows []models.AnalyticsRollup, since, until time.Time) Summary

BuildSummary reduces rollups to the dashboard's view of a range. Feed it rows from AnalyticsRepository.RangeSummary — it reads only the counters, the latency histogram, the visitor sketch and the country top-K.

type Totals

type Totals struct {
	Requests    int64   `json:"requests"`
	BytesIn     int64   `json:"bytes_in"`
	BytesOut    int64   `json:"bytes_out"`
	UniqueVisit int64   `json:"unique_visitors"`
	ErrorRate   float64 `json:"error_rate"` // (4xx+5xx)/requests, 0..1
	AvgLatency  float64 `json:"avg_latency_ms"`
	P50Latency  float64 `json:"p50_latency_ms"`
	P95Latency  float64 `json:"p95_latency_ms"`
	P99Latency  float64 `json:"p99_latency_ms"`
}

Totals are the headline traffic numbers over the whole range.

type WebAnalytics

type WebAnalytics struct {
	UniqueVisitors int64      `json:"unique_visitors"`
	BotRequests    int64      `json:"bot_requests"`
	HumanRequests  int64      `json:"human_requests"`
	TopPaths       []Category `json:"top_paths"`
	TopReferrers   []Category `json:"top_referrers"`
	TopCountries   []Category `json:"top_countries"`
	TopBrowsers    []Category `json:"top_browsers"`
	TopOS          []Category `json:"top_os"`
	TopDevices     []Category `json:"top_devices"`
	TopMethods     []Category `json:"top_methods"`
}

WebAnalytics is the cookieless web pillar: uniques plus the categorical breakdowns (paths, referrers, countries, browsers, methods).

type Window

type Window struct {
	Since time.Time `json:"since"`
	Until time.Time `json:"until"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL