analytics

package
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 16, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package analytics is fabriq's cross-tenant analytics sink: a denormalized read model fed by the shared event stream so operators can run fleet-wide reporting without touching per-tenant databases. It is the ONE place tenant data is deliberately co-located; see the Sink docs and ADR.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Redact

func Redact(raw json.RawMessage, spec *registry.AnalyticsSpec, salt string) (json.RawMessage, error)

Redact projects a payload down to an AnalyticsSpec's allow-list, exactly as the live applier does at ingest, applying the salt to any Hash fields. Exported so the Reprojector can re-apply a (typically tightened) current spec to already-stored payloads. Deterministic.

func RunSinkConformance

func RunSinkConformance(t *testing.T, newSink func() Sink)

RunSinkConformance is the single behavioral contract every analytics.Sink must satisfy. fabriqtest runs it against the fake; adapters/pganalytics runs it against real Postgres. Drift between fake and adapter is a failing test.

Types

type Applier

type Applier struct {
	// contains filtered or unexported fields
}

Applier turns one event.Envelope into a redacted Fact + Event using the aggregate's registry AnalyticsSpec. It is PURE and deterministic: the same envelope always yields the same records (so live apply and backfill agree). Deny-by-default: aggregates without an AnalyticsSpec produce no records.

func NewApplier

func NewApplier(reg *registry.Registry, opts ...ApplierOption) *Applier

func (*Applier) Apply

func (a *Applier) Apply(env event.Envelope) (Fact, Event, bool, error)

Apply returns (fact, event, ok, err). ok=false means the aggregate is not analyticized — the caller skips it (cheap, no allocation of records).

type ApplierOption

type ApplierOption func(*Applier)

ApplierOption configures an Applier.

func WithHashSalt

func WithHashSalt(salt string) ApplierOption

WithHashSalt sets the deployment salt used to pseudonymize AnalyticsSpec.Hash fields. It must be stable across restarts and equal everywhere (live apply, backfill, reproject) so the same value always hashes the same.

type Backfiller

type Backfiller struct {
	Snapshot SnapshotFunc
	Applier  *Applier
	Sink     Sink
	Batch    int // default 128
}

Backfiller replays a tenant snapshot into the analytics sink.

func (*Backfiller) AllTenants

func (b *Backfiller) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]int, error)

AllTenants backfills each tenant with bounded concurrency. One tenant's failure is recorded in the returned error but does not abort the others (fleet-wide operation). Concurrency <= 0 defaults to 4.

func (*Backfiller) Tenant

func (b *Backfiller) Tenant(ctx context.Context, tenantID string) (int, error)

Tenant backfills one tenant, returning the count of analyticized rows.

type Consumer

type Consumer struct {
	Group     string // "proj:analytics"
	Source    projection.Source
	Applier   *Applier
	Sink      Sink
	Upcasters *event.UpcasterChain // optional

	// OnApplied and OnFailure are optional observability hooks: OnApplied
	// fires once per envelope that is successfully applied to the Sink
	// (all writes succeeded); OnFailure fires once per envelope whose Sink
	// write returned an error. Neither fires on the skip/poison paths
	// (un-migratable payload, malformed payload, un-derivable tenant, or an
	// unmarked entity) — those are neither applied nor failed. Both are
	// nil-safe: callers that don't need metrics leave them unset. Kept as
	// bare func() (not a metrics dependency) so core/analytics stays free
	// of internal/metrics — callers wire prometheus.Counter.Inc directly.
	OnApplied func()
	OnFailure func()
}

Consumer drives the proj:analytics group: it reads the shared event stream through the exported projection.Source seam and applies each envelope to the analytics Sink. It deliberately does NOT reuse projection.Engine, which is hardwired to the closed []Mutation set and graph/search-specific AppliedRecorder/TargetsFor. Idempotency lives in the Sink (version gate), so there is no per-tenant applied bookkeeping and the apply path never touches a tenant database.

func (*Consumer) Run

func (c *Consumer) Run(ctx context.Context, name string) error

Run consumes until ctx ends. Scale by running replicas with distinct names.

type Event

type Event struct {
	TenantID  string          `json:"tenantId"`
	Aggregate string          `json:"aggregate"`
	AggID     string          `json:"aggId"`
	Version   int64           `json:"version"`
	Type      string          `json:"type"`
	Payload   json.RawMessage `json:"payload"` // already redacted by the applier
	At        time.Time       `json:"at"`
}

Event is one append-only history record. The sink dedupes on (TenantID, Aggregate, AggID, Version).

type Fact

type Fact struct {
	TenantID  string          `json:"tenantId"`
	Aggregate string          `json:"aggregate"`
	AggID     string          `json:"aggId"`
	Version   int64           `json:"version"`
	Payload   json.RawMessage `json:"payload"` // already redacted by the applier
	At        time.Time       `json:"at"`
	Deleted   bool            `json:"deleted"`
}

Fact is the latest denormalized state of one aggregate instance. The sink keeps exactly one Fact per (TenantID, Aggregate, AggID), version-gated.

type Reconciler

type Reconciler struct {
	Snapshot SnapshotFunc
	Applier  *Applier
	Sink     Sink
}

Reconciler detects and heals divergence between the source databases (the truth) and the analytics store. The analytics consumer skips events it cannot upcast or apply (poison-avoidance), which can leave a fact permanently missing or stale — graph and search heal this from Postgres via their reconcilers; this is the analytics equivalent. It re-projects each marked aggregate's CURRENT source state and, comparing against the stored watermark, re-applies only the aggregates that are missing or behind.

func (*Reconciler) AllTenants

func (rc *Reconciler) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]Report, error)

AllTenants reconciles each tenant with bounded concurrency, returning a report per tenant. One tenant's failure is recorded (first error) but does not abort the others. Concurrency <= 0 defaults to 4.

func (*Reconciler) Tenant

func (rc *Reconciler) Tenant(ctx context.Context, tenantID string) (Report, error)

Tenant reconciles one tenant, returning what it found and healed.

type Report

type Report struct {
	Checked int `json:"checked"`
	Missing int `json:"missing"`
	Stale   int `json:"stale"`
	Healed  int `json:"healed"`
}

Report summarizes one tenant's reconciliation: how many source aggregates were checked, how many were absent or behind in the analytics store (drift a skipped/poison event left), and how many were healed by re-applying.

func (Report) Drifted

func (r Report) Drifted() int

Drifted is Missing + Stale — aggregates whose analytics state diverged from the source of truth.

type Reprojector

type Reprojector struct {
	Reg  *registry.Registry
	Sink Sink
	// Salt must match the deployment's Config.Analytics.HashSalt so re-projected
	// Hash fields hash identically to how the live applier wrote them.
	Salt string
}

Reprojector retroactively applies each analytics-marked entity's CURRENT redaction allow-list to already-stored facts and events. It is the privacy correction the version gate otherwise blocks: tightening an entity's `Include` (or dropping a field) does not re-redact existing rows on its own, because a same-version backfill is a no-op. Reprojection rewrites the stored payloads in place through the current spec.

It can only NARROW: it re-projects the already-stored (possibly wider) payload, so it removes fields that should no longer be co-located. Widening (bringing a previously-stripped field back) is not possible from stored data — that needs a forced backfill from the source database, out of scope here.

func (*Reprojector) AllTenants

func (r *Reprojector) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]int64, error)

AllTenants reprojects each tenant with bounded concurrency. One tenant's failure is recorded (first error returned) but does not abort the others. Concurrency <= 0 defaults to 4.

func (*Reprojector) Tenant

func (r *Reprojector) Tenant(ctx context.Context, tenantID string) (int64, error)

Tenant re-projects every analytics-marked aggregate's stored payloads for one tenant through its current spec, returning the number of rows rewritten.

type Sink

type Sink interface {
	UpsertFacts(ctx context.Context, facts []Fact) error
	AppendEvents(ctx context.Context, events []Event) error
	Watermark(ctx context.Context, tenantID, aggregate, aggID string) (int64, error)
	SetWatermark(ctx context.Context, ws []Watermark) error
	// AllWatermarks returns every applied watermark for a tenant in one read —
	// the reconciler compares these against the source's current versions to
	// find drift (facts a skipped event left missing or stale) without a
	// per-aggregate round-trip.
	AllWatermarks(ctx context.Context, tenantID string) ([]Watermark, error)
	// LagByTenant reports per-tenant read-model freshness: for each tenant with
	// at least one fact, now() - (that tenant's newest fact commit time), in
	// seconds. An empty map means the sink holds no facts. Per-tenant (rather
	// than one fleet-wide number) so a single stalled tenant cannot hide behind
	// others still flowing; the poller derives the worst-case gauge and the
	// tenants-behind count from it without emitting a per-tenant metric series.
	LagByTenant(ctx context.Context) (map[string]float64, error)
	// ReprojectTenant re-writes stored fact AND event payloads for a tenant in
	// place, applying transform to each. aggregate "" means every aggregate.
	// It returns the number of rows whose payload actually changed. Used to
	// retroactively apply a tightened redaction allow-list to already-stored
	// data (a privacy correction): the transform re-projects each stored payload
	// through the entity's current AnalyticsSpec. Idempotent — a second run with
	// the same transform rewrites nothing.
	ReprojectTenant(ctx context.Context, tenantID, aggregate string, transform func(payload json.RawMessage) (json.RawMessage, error)) (rowsRewritten int64, err error)
	// PruneEvents deletes append-only history events committed strictly before
	// olderThan, across all tenants, and returns the count removed. It is the
	// retention control for the unbounded event log; facts (latest state) are
	// never pruned — only the history. Idempotent.
	PruneEvents(ctx context.Context, olderThan time.Time) (rowsDeleted int64, err error)
	// MaintainPartitions is the partition-management hook for sinks that
	// range-partition the event log: it creates upcoming partitions and, when
	// retention > 0, drops partitions entirely older than retention (instant
	// reclaim). Sinks that do not partition return (0, 0, nil).
	MaintainPartitions(ctx context.Context, retention time.Duration) (created, dropped int, err error)
	// PurgeTenant hard-deletes ALL of one tenant's co-located data — facts,
	// events, and watermarks — and returns the number of rows removed. It is the
	// erasure primitive for tenant offboarding and right-to-be-forgotten
	// requests: the ONE deliberately destructive operation on the sink, and the
	// only supported way to remove a tenant from the shared store. Idempotent
	// (purging an absent tenant deletes nothing and returns 0). Never invoked
	// automatically — an explicit operator step, like dropping a tenant database.
	PurgeTenant(ctx context.Context, tenantID string) (rowsDeleted int64, err error)
	Close() error
}

Sink persists analytics records. All methods are idempotent under at-least-once delivery: UpsertFacts version-gates (older/equal versions are no-ops), AppendEvents dedupes on the natural key, SetWatermark advances monotonically. Implemented by adapters/pganalytics and fabriqtest.FakeAnalyticsSink (one conformance suite gates both).

type SnapshotFunc

type SnapshotFunc func(ctx context.Context, tenantID string, fn func(event.Envelope) error) error

SnapshotFunc streams a tenant's current-state envelopes (satisfied by postgres.Adapter.SnapshotEntities and the catalog DynamicSet-routed snapshot). Backfill drives the SAME applier as live consume, so the two agree; the sink's version gate makes a re-run a no-op and lets live traffic run concurrently.

type Watermark

type Watermark struct {
	TenantID  string `json:"tenantId"`
	Aggregate string `json:"aggregate"`
	AggID     string `json:"aggId"`
	Version   int64  `json:"version"`
}

Watermark records the highest applied version per aggregate instance, for lag metrics and resumable backfill.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL