analytics

package
v1.10.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 25, 2026 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package analytics turns Goma Gateway's per-request event stream into minute-bucketed rollups and answers the Traffic, Performance and Web Analytics queries over them. It holds no per-request rows and no PII: latency is a histogram, uniques an HLL sketch, and the visitor id a daily-salted hash.

Index

Constants

View Source
const (
	MaxBatch = 500
	MaxSkew  = 15 * time.Minute
)

Ingest limits. MaxBatch bounds one POST; MaxSkew bounds how far a node's clock may be from the manager's before its timestamps are distrusted.

View Source
const DefaultLiveWindow = 5 * time.Minute

DefaultLiveWindow is how far back a visitor still counts as "live" — the convention analytics products settled on (Plausible, Cloudflare, Fathom).

View Source
const StreamEventField = "e"

StreamEventField is the stream entry field Goma writes each event under, and the only one the consumer reads. Anything appended under another name is dropped without a trace, so writers and the reader share this constant.

View Source
const SummaryTopCountries = 8

Variables

View Source
var (
	ErrBatchTooLarge = errors.New("batch exceeds the maximum size")
	ErrNoEvents      = errors.New("batch declares no events")
	ErrNoBatchID     = errors.New("batch id is required")
)
View Source
var LatencyBoundsMs = []int64{5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}

LatencyBoundsMs are the histogram bucket upper bounds (milliseconds). A value is counted in the first bucket whose bound it does not exceed; anything above the last bound lands in a trailing overflow bucket (index len(bounds)).

Functions

func IsBotUA added in v1.7.0

func IsBotUA(ua string) bool

IsBotUA reports whether a user agent looks automated, using the same rule as the rollups' bot/human split. Exported so callers outside the aggregator — live visitors — classify traffic identically.

func Merge

func Merge(dst, src *models.AnalyticsRollup)

Merge adds src into dst: counters, element-wise histograms, sums, top-K maps, and the HLL sketch. Used by the repository to combine a flushed bucket with an existing row (multi-consumer / re-flush safe).

func MergeUniques

func MergeUniques(sketches [][]byte) int64

MergeUniques merges several serialized sketches and returns the estimate — the unique-visitor count over a range is the merge of its buckets' sketches.

func Percentile

func Percentile(hist []int64, p float64) float64

Percentile estimates the p-th percentile (0..1) latency in ms from a histogram, returning each bucket's upper bound (the overflow bucket returns the last bound as a floor). Approximate by design — bucketed, cheap, and mergeable.

func UniquesOf

func UniquesOf(data []byte) int64

UniquesOf returns the estimated unique visitors from a serialized sketch.

func WorkspaceIDFromRoute

func WorkspaceIDFromRoute(name string) uint

WorkspaceIDFromRoute parses the workspace id out of a Goma route name of the form "mb-ws<id>-<slug>". Returns 0 when the name isn't in that form (a route Miabi doesn't own — e.g. the platform gateway's own route).

Types

type Aggregator

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator accumulates events into per-(workspace, app, route, minute) buckets in memory, and hands over "closed" buckets (older than a grace window) as models.AnalyticsRollup for persistence. Safe for concurrent Ingest/Flush.

func NewAggregator

func NewAggregator() *Aggregator

func (*Aggregator) Flush

func (a *Aggregator) Flush(before time.Time) []*models.AnalyticsRollup

Flush removes and returns every bucket strictly older than `before`, serializing its HLL sketch into VisitorsHLL. Call with now-grace so an in-progress minute keeps accumulating.

func (*Aggregator) Ingest

func (a *Aggregator) Ingest(e *Event, ws, app uint)

Ingest folds one event into its bucket. ws/app are resolved by the caller (workspace from the route name, app via a route lookup); an unresolved app is 0.

func (*Aggregator) Pending

func (a *Aggregator) Pending() int

Pending reports how many open buckets are held (for metrics/tests).

type Batch added in v1.8.2

type Batch struct {
	BatchID string  `json:"batch_id"`
	Events  []Event `json:"events"`
}

Batch is one node's forwarded run of gateway events.

type Category

type Category struct {
	Label string `json:"label"`
	Count int64  `json:"count"`
}

Category is one labelled count in a breakdown.

type Event

type Event struct {
	Ts           int64  `json:"ts"` // unix millis
	Gateway      string `json:"gw"`
	Route        string `json:"name"` // "mb-ws<workspaceID>-<slug>"
	Host         string `json:"host"`
	Method       string `json:"method"`
	Status       int    `json:"status"`
	Path         string `json:"path"`
	PathTemplate string `json:"path_template"`
	ReqBytes     int64  `json:"req_bytes"`
	RespBytes    int64  `json:"resp_bytes"`
	DurationMs   int64  `json:"duration_ms"`
	UpstreamMs   int64  `json:"upstream_ms"`
	VID          string `json:"vid"`
	Country      string `json:"country"`
	UA           string `json:"ua"`
	RefererHost  string `json:"referer_host"`
}

Event is one request as emitted by Goma onto the analytics stream. Field tags mirror the gateway's AnalyticsEvent JSON.

type IngestResult added in v1.8.2

type IngestResult struct {
	Accepted  int  `json:"accepted"`
	Duplicate bool `json:"duplicate"`
	// Rejected counts events dropped for a route the node does not serve.
	Rejected int `json:"rejected"`
	// Reclocked counts events whose timestamp was too far off to trust.
	Reclocked int `json:"reclocked"`
}

IngestResult reports what the manager did with a batch. Rejected events are counted rather than failing the batch: one bad event must not strand the rest.

type Ingester added in v1.8.2

type Ingester struct {
	// contains filtered or unexported fields
}

Ingester appends events forwarded by an edge node to the platform stream the AnalyticsConsumer reads, so edge traffic converges with the manager's.

func NewIngester added in v1.8.2

func NewIngester(rdb *redis.Client, owner RouteOwnership, stream string) *Ingester

func (*Ingester) Ingest added in v1.8.2

func (i *Ingester) Ingest(ctx context.Context, serverID uint, b Batch) (IngestResult, error)

Ingest validates a node's batch and appends the events it is allowed to report. Duplicate batches return the already-accepted result so the forwarder can ACK.

type LiveTracker added in v1.7.0

type LiveTracker struct {
	// contains filtered or unexported fields
}

LiveTracker answers "how many distinct visitors are active right now" from the same Goma event stream the rollups use, but per workspace and per app rather than one gateway-wide gauge. One sorted set per scope keys on the daily-salted visitor id, so counting and expiry are both O(log N).

func NewLiveTracker added in v1.7.0

func NewLiveTracker(rdb *redis.Client, window time.Duration) *LiveTracker

func (*LiveTracker) Count added in v1.7.0

func (t *LiveTracker) Count(ctx context.Context, ws, app uint) (int64, error)

Count returns the distinct visitors active within the window for a scope (app 0 = the whole workspace). Exact, and unaffected by how recently a sweep ran, because the window is applied as a score range.

func (*LiveTracker) Sweep added in v1.7.0

func (t *LiveTracker) Sweep(ctx context.Context)

Sweep drops visitors who have fallen outside the window from every scope this process writes to. Purely housekeeping — counts are already window-correct — so a failure is logged by the caller at most, never fatal.

func (*LiveTracker) Touch added in v1.7.0

func (t *LiveTracker) Touch(ctx context.Context, visits []LiveVisit)

Touch records a batch of sightings as active now. Callers batch by their natural boundary — one stream read — so this is a handful of round trips per batch rather than one per event. Best-effort: a Redis failure costs a live count, never an ingest.

func (*LiveTracker) Window added in v1.7.0

func (t *LiveTracker) Window() time.Duration

Window is the "active within" period the counts describe.

type LiveVisit added in v1.7.0

type LiveVisit struct {
	Workspace uint
	App       uint
	VID       string
}

LiveVisit is one visitor sighting: the scope it belongs to, and who.

type Performance

type Performance struct {
	RequestP50    float64     `json:"request_p50_ms"`
	RequestP95    float64     `json:"request_p95_ms"`
	RequestP99    float64     `json:"request_p99_ms"`
	UpstreamP50   float64     `json:"upstream_p50_ms"`
	UpstreamP95   float64     `json:"upstream_p95_ms"`
	UpstreamP99   float64     `json:"upstream_p99_ms"`
	AvgRequestMs  float64     `json:"avg_request_ms"`
	AvgUpstreamMs float64     `json:"avg_upstream_ms"`
	AvgOverheadMs float64     `json:"avg_overhead_ms"`
	SlowRoutes    []RouteStat `json:"slow_routes"`
}

Performance covers the latency pillar: request vs upstream percentiles plus the slowest routes by p95.

type Report

type Report struct {
	Range         Window        `json:"range"`
	Granularity   string        `json:"granularity"` // "minute" | "hour" | "day"
	Totals        Totals        `json:"totals"`
	Series        []SeriesPoint `json:"series"`
	Status        StatusBreak   `json:"status"`
	Performance   Performance   `json:"performance"`
	Web           WebAnalytics  `json:"web"`
	Compare       *Totals       `json:"compare,omitempty"`
	RetentionDays int           `json:"retention_days"`
	Exportable    bool          `json:"exportable"`
}

Report is the combined analytics answer for a workspace (optionally one app) over a time range: the Traffic, Performance and Web Analytics pillars share one pass over the same rollups so the three dashboard tabs come from a single query.

func BuildReport

func BuildReport(rows []models.AnalyticsRollup, since, until time.Time) Report

BuildReport reduces the raw minute rollups of a range into the combined dashboard report, with granularity auto-selected from the span. One pass for the totals and breakdowns, a grouped pass for the series, merging latency histograms for percentiles and HLL sketches for uniques.

type RouteOwnership added in v1.8.2

type RouteOwnership interface {
	RouteNamesForNode(serverID uint) (map[string]bool, error)
}

RouteOwnership answers which Goma route names a node is allowed to report on.

type RouteStat

type RouteStat struct {
	Route     string  `json:"route"`
	Requests  int64   `json:"requests"`
	P95       float64 `json:"p95_latency_ms"`
	ErrorRate float64 `json:"error_rate"`
}

RouteStat is a per-route performance summary (used for the slowest-routes list).

type SeriesPoint

type SeriesPoint struct {
	T          time.Time `json:"t"`
	Requests   int64     `json:"requests"`
	BytesIn    int64     `json:"bytes_in"`
	BytesOut   int64     `json:"bytes_out"`
	Errors     int64     `json:"errors"`     // 4xx+5xx (kept for compatibility)
	Errors4xx  int64     `json:"errors_4xx"` // client errors
	Errors5xx  int64     `json:"errors_5xx"` // server errors
	Uniques    int64     `json:"unique_visitors"`
	AvgLatency float64   `json:"avg_latency_ms"`
	P95Latency float64   `json:"p95_latency_ms"`
}

SeriesPoint is one time bucket of the request/latency time series at the chosen granularity.

type StatusBreak

type StatusBreak struct {
	S2xx int64 `json:"s2xx"`
	S3xx int64 `json:"s3xx"`
	S4xx int64 `json:"s4xx"`
	S5xx int64 `json:"s5xx"`
}

type Summary added in v1.7.0

type Summary struct {
	Range        Window        `json:"range"`
	Granularity  string        `json:"granularity"`
	Totals       Totals        `json:"totals"`
	Series       []SeriesPoint `json:"series"`
	Status       StatusBreak   `json:"status"`
	TopCountries []Category    `json:"top_countries"`
	Compare      *Totals       `json:"compare,omitempty"`
}

Summary is the dashboard's slice of a report: headline totals, the status split, the request series and the top countries. It deliberately omits the remaining breakdowns, per-route stats and upstream percentiles the analytics pages need, so the dashboard's traffic card doesn't pay for them.

func BuildSummary added in v1.7.0

func BuildSummary(rows []models.AnalyticsRollup, since, until time.Time) Summary

type Totals

type Totals struct {
	Requests    int64   `json:"requests"`
	BytesIn     int64   `json:"bytes_in"`
	BytesOut    int64   `json:"bytes_out"`
	UniqueVisit int64   `json:"unique_visitors"`
	ErrorRate   float64 `json:"error_rate"` // (4xx+5xx)/requests, 0..1
	AvgLatency  float64 `json:"avg_latency_ms"`
	P50Latency  float64 `json:"p50_latency_ms"`
	P95Latency  float64 `json:"p95_latency_ms"`
	P99Latency  float64 `json:"p99_latency_ms"`
}

Totals are the headline traffic numbers over the whole range.

type WebAnalytics

type WebAnalytics struct {
	UniqueVisitors int64      `json:"unique_visitors"`
	BotRequests    int64      `json:"bot_requests"`
	HumanRequests  int64      `json:"human_requests"`
	TopPaths       []Category `json:"top_paths"`
	TopReferrers   []Category `json:"top_referrers"`
	TopCountries   []Category `json:"top_countries"`
	TopBrowsers    []Category `json:"top_browsers"`
	TopUserAgents  []Category `json:"top_user_agents"`
	TopOS          []Category `json:"top_os"`
	TopDevices     []Category `json:"top_devices"`
	TopMethods     []Category `json:"top_methods"`
}

WebAnalytics is the cookieless web pillar: uniques plus the categorical breakdowns (paths, referrers, countries, browsers, methods). Countries, paths and user agents return every entry tracked (at most topKCap), for the scrollable lists.

type Window

type Window struct {
	Since time.Time `json:"since"`
	Until time.Time `json:"until"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL