Documentation
¶
Overview ¶
Package analytics turns Goma Gateway's per-request event stream into minute-bucketed rollups and answers the Traffic, Performance and Web Analytics queries over them. It holds no per-request rows and no PII: latency is a histogram, uniques an HLL sketch, and the visitor id a daily-salted hash.
Index ¶
- Constants
- Variables
- func IsBotUA(ua string) bool
- func Merge(dst, src *models.AnalyticsRollup)
- func MergeUniques(sketches [][]byte) int64
- func Percentile(hist []int64, p float64) float64
- func UniquesOf(data []byte) int64
- func WorkspaceIDFromRoute(name string) uint
- type Aggregator
- type Batch
- type Category
- type Event
- type IngestResult
- type Ingester
- type LiveTracker
- type LiveVisit
- type Performance
- type Report
- type RouteOwnership
- type RouteStat
- type SeriesPoint
- type StatusBreak
- type Summary
- type Totals
- type WebAnalytics
- type Window
Constants ¶
const ( MaxBatch = 500 MaxSkew = 15 * time.Minute )
Ingest limits. MaxBatch bounds one POST; MaxSkew bounds how far a node's clock may be from the manager's before its timestamps are distrusted.
const DefaultLiveWindow = 5 * time.Minute
DefaultLiveWindow is how far back a visitor still counts as "live" — the convention analytics products settled on (Plausible, Cloudflare, Fathom).
const StreamEventField = "e"
StreamEventField is the stream entry field Goma writes each event under, and the only one the consumer reads. Anything appended under another name is dropped without a trace, so writers and the reader share this constant.
const SummaryTopCountries = 8
Variables ¶
var ( ErrBatchTooLarge = errors.New("batch exceeds the maximum size") ErrNoEvents = errors.New("batch declares no events") ErrNoBatchID = errors.New("batch id is required") )
var LatencyBoundsMs = []int64{5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}
LatencyBoundsMs are the histogram bucket upper bounds (milliseconds). A value is counted in the first bucket whose bound it does not exceed; anything above the last bound lands in a trailing overflow bucket (index len(bounds)).
Functions ¶
func IsBotUA ¶ added in v1.7.0
IsBotUA reports whether a user agent looks automated, using the same rule as the rollups' bot/human split. Exported so callers outside the aggregator — live visitors — classify traffic identically.
func Merge ¶
func Merge(dst, src *models.AnalyticsRollup)
Merge adds src into dst: counters, element-wise histograms, sums, top-K maps, and the HLL sketch. Used by the repository to combine a flushed bucket with an existing row (multi-consumer / re-flush safe).
func MergeUniques ¶
MergeUniques merges several serialized sketches and returns the estimate — the unique-visitor count over a range is the merge of its buckets' sketches.
func Percentile ¶
Percentile estimates the p-th percentile (0..1) latency in ms from a histogram, returning each bucket's upper bound (the overflow bucket returns the last bound as a floor). Approximate by design — bucketed, cheap, and mergeable.
func WorkspaceIDFromRoute ¶
WorkspaceIDFromRoute parses the workspace id out of a Goma route name of the form "mb-ws<id>-<slug>". Returns 0 when the name isn't in that form (a route Miabi doesn't own — e.g. the platform gateway's own route).
Types ¶
type Aggregator ¶
type Aggregator struct {
// contains filtered or unexported fields
}
Aggregator accumulates events into per-(workspace, app, route, minute) buckets in memory, and hands over "closed" buckets (older than a grace window) as models.AnalyticsRollup for persistence. Safe for concurrent Ingest/Flush.
func NewAggregator ¶
func NewAggregator() *Aggregator
func (*Aggregator) Flush ¶
func (a *Aggregator) Flush(before time.Time) []*models.AnalyticsRollup
Flush removes and returns every bucket strictly older than `before`, serializing its HLL sketch into VisitorsHLL. Call with now-grace so an in-progress minute keeps accumulating.
func (*Aggregator) Ingest ¶
func (a *Aggregator) Ingest(e *Event, ws, app uint)
Ingest folds one event into its bucket. ws/app are resolved by the caller (workspace from the route name, app via a route lookup); an unresolved app is 0.
func (*Aggregator) Pending ¶
func (a *Aggregator) Pending() int
Pending reports how many open buckets are held (for metrics/tests).
type Event ¶
type Event struct {
Ts int64 `json:"ts"` // unix millis
Gateway string `json:"gw"`
Route string `json:"name"` // "mb-ws<workspaceID>-<slug>"
Host string `json:"host"`
Method string `json:"method"`
Status int `json:"status"`
Path string `json:"path"`
PathTemplate string `json:"path_template"`
ReqBytes int64 `json:"req_bytes"`
RespBytes int64 `json:"resp_bytes"`
DurationMs int64 `json:"duration_ms"`
UpstreamMs int64 `json:"upstream_ms"`
VID string `json:"vid"`
Country string `json:"country"`
UA string `json:"ua"`
RefererHost string `json:"referer_host"`
}
Event is one request as emitted by Goma onto the analytics stream. Field tags mirror the gateway's AnalyticsEvent JSON.
type IngestResult ¶ added in v1.8.2
type IngestResult struct {
Accepted int `json:"accepted"`
Duplicate bool `json:"duplicate"`
// Rejected counts events dropped for a route the node does not serve.
Rejected int `json:"rejected"`
// Reclocked counts events whose timestamp was too far off to trust.
Reclocked int `json:"reclocked"`
}
IngestResult reports what the manager did with a batch. Rejected events are counted rather than failing the batch: one bad event must not strand the rest.
type Ingester ¶ added in v1.8.2
type Ingester struct {
// contains filtered or unexported fields
}
Ingester appends events forwarded by an edge node to the platform stream the AnalyticsConsumer reads, so edge traffic converges with the manager's.
func NewIngester ¶ added in v1.8.2
func NewIngester(rdb *redis.Client, owner RouteOwnership, stream string) *Ingester
type LiveTracker ¶ added in v1.7.0
type LiveTracker struct {
// contains filtered or unexported fields
}
LiveTracker answers "how many distinct visitors are active right now" from the same Goma event stream the rollups use, but per workspace and per app rather than one gateway-wide gauge. One sorted set per scope keys on the daily-salted visitor id, so counting and expiry are both O(log N).
func NewLiveTracker ¶ added in v1.7.0
func NewLiveTracker(rdb *redis.Client, window time.Duration) *LiveTracker
func (*LiveTracker) Count ¶ added in v1.7.0
Count returns the distinct visitors active within the window for a scope (app 0 = the whole workspace). Exact, and unaffected by how recently a sweep ran, because the window is applied as a score range.
func (*LiveTracker) Sweep ¶ added in v1.7.0
func (t *LiveTracker) Sweep(ctx context.Context)
Sweep drops visitors who have fallen outside the window from every scope this process writes to. Purely housekeeping — counts are already window-correct — so a failure is logged by the caller at most, never fatal.
func (*LiveTracker) Touch ¶ added in v1.7.0
func (t *LiveTracker) Touch(ctx context.Context, visits []LiveVisit)
Touch records a batch of sightings as active now. Callers batch by their natural boundary — one stream read — so this is a handful of round trips per batch rather than one per event. Best-effort: a Redis failure costs a live count, never an ingest.
func (*LiveTracker) Window ¶ added in v1.7.0
func (t *LiveTracker) Window() time.Duration
Window is the "active within" period the counts describe.
type LiveVisit ¶ added in v1.7.0
LiveVisit is one visitor sighting: the scope it belongs to, and who.
type Performance ¶
type Performance struct {
RequestP50 float64 `json:"request_p50_ms"`
RequestP95 float64 `json:"request_p95_ms"`
RequestP99 float64 `json:"request_p99_ms"`
UpstreamP50 float64 `json:"upstream_p50_ms"`
UpstreamP95 float64 `json:"upstream_p95_ms"`
UpstreamP99 float64 `json:"upstream_p99_ms"`
// Mean total, upstream (backend) and gateway overhead (total − upstream) in ms.
AvgRequestMs float64 `json:"avg_request_ms"`
AvgUpstreamMs float64 `json:"avg_upstream_ms"`
AvgOverheadMs float64 `json:"avg_overhead_ms"`
SlowRoutes []RouteStat `json:"slow_routes"`
}
Performance covers the latency pillar: request vs upstream percentiles plus the slowest routes by p95.
type Report ¶
type Report struct {
Range Window `json:"range"`
Granularity string `json:"granularity"` // "minute" | "hour" | "day"
Totals Totals `json:"totals"`
Series []SeriesPoint `json:"series"`
Status StatusBreak `json:"status"`
Performance Performance `json:"performance"`
Web WebAnalytics `json:"web"`
// Compare holds the totals of the immediately preceding, equal-length window,
// so the UI can render period-over-period deltas. Nil when not requested.
Compare *Totals `json:"compare,omitempty"`
// RetentionDays is the effective retention cap in days (-1 = unlimited), and Exportable reports whether
// analytics export is entitled. Both are set by the handler from the license so the UI can bound the
// range picker and show upgrade hints. They are edition metadata, not computed from rows.
RetentionDays int `json:"retention_days"`
Exportable bool `json:"exportable"`
}
Report is the combined analytics answer for a workspace (optionally one app) over a time range: the Traffic, Performance and Web Analytics pillars share one pass over the same rollups so the three dashboard tabs come from a single query.
func BuildReport ¶
func BuildReport(rows []models.AnalyticsRollup, since, until time.Time) Report
BuildReport reduces the raw minute rollups of a range into the combined dashboard report, with granularity auto-selected from the span. One pass for the totals and breakdowns, a grouped pass for the series, merging latency histograms for percentiles and HLL sketches for uniques.
type RouteOwnership ¶ added in v1.8.2
RouteOwnership answers which Goma route names a node is allowed to report on.
type RouteStat ¶
type RouteStat struct {
Route string `json:"route"`
Requests int64 `json:"requests"`
P95 float64 `json:"p95_latency_ms"`
ErrorRate float64 `json:"error_rate"`
}
RouteStat is a per-route performance summary (used for the slowest-routes list).
type SeriesPoint ¶
type SeriesPoint struct {
T time.Time `json:"t"`
Requests int64 `json:"requests"`
BytesIn int64 `json:"bytes_in"`
BytesOut int64 `json:"bytes_out"`
Errors int64 `json:"errors"` // 4xx+5xx (kept for compatibility)
Errors4xx int64 `json:"errors_4xx"` // client errors
Errors5xx int64 `json:"errors_5xx"` // server errors
Uniques int64 `json:"unique_visitors"`
AvgLatency float64 `json:"avg_latency_ms"`
P95Latency float64 `json:"p95_latency_ms"`
}
SeriesPoint is one time bucket of the request/latency time series at the chosen granularity.
type StatusBreak ¶
type Summary ¶ added in v1.7.0
type Summary struct {
Range Window `json:"range"`
Granularity string `json:"granularity"`
Totals Totals `json:"totals"`
Series []SeriesPoint `json:"series"`
Status StatusBreak `json:"status"`
// TopCountries is the SummaryTopCountries busiest origins, for the dashboard's
// country panel. Empty when the gateway has no GeoIP database — which is
// indistinguishable here from "no traffic", so the panel simply doesn't render.
TopCountries []Category `json:"top_countries"`
// Compare holds the totals of the immediately preceding, equal-length window,
// for period-over-period deltas. Nil when not requested.
Compare *Totals `json:"compare,omitempty"`
}
Summary is the dashboard's slice of a report: headline totals, the status split, the request series and the top countries. It deliberately omits the remaining breakdowns, per-route stats and upstream percentiles the analytics pages need, so the dashboard's traffic card doesn't pay for them.
func BuildSummary ¶ added in v1.7.0
func BuildSummary(rows []models.AnalyticsRollup, since, until time.Time) Summary
BuildSummary reduces rollups to the dashboard's view of a range. Feed it rows from AnalyticsRepository.RangeSummary — it reads only the counters, the latency histogram, the visitor sketch and the country top-K.
type Totals ¶
type Totals struct {
Requests int64 `json:"requests"`
BytesIn int64 `json:"bytes_in"`
BytesOut int64 `json:"bytes_out"`
UniqueVisit int64 `json:"unique_visitors"`
ErrorRate float64 `json:"error_rate"` // (4xx+5xx)/requests, 0..1
AvgLatency float64 `json:"avg_latency_ms"`
P50Latency float64 `json:"p50_latency_ms"`
P95Latency float64 `json:"p95_latency_ms"`
P99Latency float64 `json:"p99_latency_ms"`
}
Totals are the headline traffic numbers over the whole range.
type WebAnalytics ¶
type WebAnalytics struct {
UniqueVisitors int64 `json:"unique_visitors"`
BotRequests int64 `json:"bot_requests"`
HumanRequests int64 `json:"human_requests"`
TopPaths []Category `json:"top_paths"`
TopReferrers []Category `json:"top_referrers"`
TopCountries []Category `json:"top_countries"`
TopBrowsers []Category `json:"top_browsers"`
TopOS []Category `json:"top_os"`
TopDevices []Category `json:"top_devices"`
TopMethods []Category `json:"top_methods"`
}
WebAnalytics is the cookieless web pillar: uniques plus the categorical breakdowns (paths, referrers, countries, browsers, methods).