Documentation
¶
Overview ¶
Package analytics is fabriq's cross-tenant analytics sink: a denormalized read model fed by the shared event stream so operators can run fleet-wide reporting without touching per-tenant databases. It is the ONE place tenant data is deliberately co-located; see the Sink docs and ADR.
Index ¶
- func Redact(raw json.RawMessage, spec *registry.AnalyticsSpec, salt string) (json.RawMessage, error)
- func RunSinkConformance(t *testing.T, newSink func() Sink)
- type Applier
- type ApplierOption
- type Backfiller
- type Consumer
- type Event
- type Fact
- type Reconciler
- type Report
- type Reprojector
- type Sink
- type SnapshotFunc
- type Watermark
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Redact ¶
func Redact(raw json.RawMessage, spec *registry.AnalyticsSpec, salt string) (json.RawMessage, error)
Redact projects a payload down to an AnalyticsSpec's allow-list, exactly as the live applier does at ingest, applying the salt to any Hash fields. Exported so the Reprojector can re-apply a (typically tightened) current spec to already-stored payloads. Deterministic.
func RunSinkConformance ¶
RunSinkConformance is the single behavioral contract every analytics.Sink must satisfy. fabriqtest runs it against the fake; adapters/pganalytics runs it against real Postgres. Drift between fake and adapter is a failing test.
Types ¶
type Applier ¶
type Applier struct {
// contains filtered or unexported fields
}
Applier turns one event.Envelope into a redacted Fact + Event using the aggregate's registry AnalyticsSpec. It is PURE and deterministic: the same envelope always yields the same records (so live apply and backfill agree). Deny-by-default: aggregates without an AnalyticsSpec produce no records.
func NewApplier ¶
func NewApplier(reg *registry.Registry, opts ...ApplierOption) *Applier
type ApplierOption ¶
type ApplierOption func(*Applier)
ApplierOption configures an Applier.
func WithHashSalt ¶
func WithHashSalt(salt string) ApplierOption
WithHashSalt sets the deployment salt used to pseudonymize AnalyticsSpec.Hash fields. It must be stable across restarts and equal everywhere (live apply, backfill, reproject) so the same value always hashes the same.
type Backfiller ¶
type Backfiller struct {
Snapshot SnapshotFunc
Applier *Applier
Sink Sink
Batch int // default 128
}
Backfiller replays a tenant snapshot into the analytics sink.
func (*Backfiller) AllTenants ¶
func (b *Backfiller) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]int, error)
AllTenants backfills each tenant with bounded concurrency. One tenant's failure is recorded in the returned error but does not abort the others (fleet-wide operation). Concurrency <= 0 defaults to 4.
type Consumer ¶
type Consumer struct {
Group string // "proj:analytics"
Source projection.Source
Applier *Applier
Sink Sink
Upcasters *event.UpcasterChain // optional
// OnApplied and OnFailure are optional observability hooks: OnApplied
// fires once per envelope that is successfully applied to the Sink
// (all writes succeeded); OnFailure fires once per envelope whose Sink
// write returned an error. Neither fires on the skip/poison paths
// (un-migratable payload, malformed payload, un-derivable tenant, or an
// unmarked entity) — those are neither applied nor failed. Both are
// nil-safe: callers that don't need metrics leave them unset. Kept as
// bare func() (not a metrics dependency) so core/analytics stays free
// of internal/metrics — callers wire prometheus.Counter.Inc directly.
OnApplied func()
OnFailure func()
}
Consumer drives the proj:analytics group: it reads the shared event stream through the exported projection.Source seam and applies each envelope to the analytics Sink. It deliberately does NOT reuse projection.Engine, which is hardwired to the closed []Mutation set and graph/search-specific AppliedRecorder/TargetsFor. Idempotency lives in the Sink (version gate), so there is no per-tenant applied bookkeeping and the apply path never touches a tenant database.
type Event ¶
type Event struct {
TenantID string `json:"tenantId"`
Aggregate string `json:"aggregate"`
AggID string `json:"aggId"`
Version int64 `json:"version"`
Type string `json:"type"`
Payload json.RawMessage `json:"payload"` // already redacted by the applier
At time.Time `json:"at"`
}
Event is one append-only history record. The sink dedupes on (TenantID, Aggregate, AggID, Version).
type Fact ¶
type Fact struct {
TenantID string `json:"tenantId"`
Aggregate string `json:"aggregate"`
AggID string `json:"aggId"`
Version int64 `json:"version"`
Payload json.RawMessage `json:"payload"` // already redacted by the applier
At time.Time `json:"at"`
Deleted bool `json:"deleted"`
}
Fact is the latest denormalized state of one aggregate instance. The sink keeps exactly one Fact per (TenantID, Aggregate, AggID), version-gated.
type Reconciler ¶
type Reconciler struct {
Snapshot SnapshotFunc
Applier *Applier
Sink Sink
}
Reconciler detects and heals divergence between the source databases (the truth) and the analytics store. The analytics consumer skips events it cannot upcast or apply (poison-avoidance), which can leave a fact permanently missing or stale — graph and search heal this from Postgres via their reconcilers; this is the analytics equivalent. It re-projects each marked aggregate's CURRENT source state and, comparing against the stored watermark, re-applies only the aggregates that are missing or behind.
func (*Reconciler) AllTenants ¶
func (rc *Reconciler) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]Report, error)
AllTenants reconciles each tenant with bounded concurrency, returning a report per tenant. One tenant's failure is recorded (first error) but does not abort the others. Concurrency <= 0 defaults to 4.
type Report ¶
type Report struct {
Checked int `json:"checked"`
Missing int `json:"missing"`
Stale int `json:"stale"`
Healed int `json:"healed"`
}
Report summarizes one tenant's reconciliation: how many source aggregates were checked, how many were absent or behind in the analytics store (drift a skipped/poison event left), and how many were healed by re-applying.
type Reprojector ¶
type Reprojector struct {
Reg *registry.Registry
Sink Sink
// Salt must match the deployment's Config.Analytics.HashSalt so re-projected
// Hash fields hash identically to how the live applier wrote them.
Salt string
}
Reprojector retroactively applies each analytics-marked entity's CURRENT redaction allow-list to already-stored facts and events. It is the privacy correction the version gate otherwise blocks: tightening an entity's `Include` (or dropping a field) does not re-redact existing rows on its own, because a same-version backfill is a no-op. Reprojection rewrites the stored payloads in place through the current spec.
It can only NARROW: it re-projects the already-stored (possibly wider) payload, so it removes fields that should no longer be co-located. Widening (bringing a previously-stripped field back) is not possible from stored data — that needs a forced backfill from the source database, out of scope here.
func (*Reprojector) AllTenants ¶
func (r *Reprojector) AllTenants(ctx context.Context, tenants []string, concurrency int) (map[string]int64, error)
AllTenants reprojects each tenant with bounded concurrency. One tenant's failure is recorded (first error returned) but does not abort the others. Concurrency <= 0 defaults to 4.
type Sink ¶
type Sink interface {
UpsertFacts(ctx context.Context, facts []Fact) error
AppendEvents(ctx context.Context, events []Event) error
Watermark(ctx context.Context, tenantID, aggregate, aggID string) (int64, error)
SetWatermark(ctx context.Context, ws []Watermark) error
// AllWatermarks returns every applied watermark for a tenant in one read —
// the reconciler compares these against the source's current versions to
// find drift (facts a skipped event left missing or stale) without a
// per-aggregate round-trip.
AllWatermarks(ctx context.Context, tenantID string) ([]Watermark, error)
// LagByTenant reports per-tenant read-model freshness: for each tenant with
// at least one fact, now() - (that tenant's newest fact commit time), in
// seconds. An empty map means the sink holds no facts. Per-tenant (rather
// than one fleet-wide number) so a single stalled tenant cannot hide behind
// others still flowing; the poller derives the worst-case gauge and the
// tenants-behind count from it without emitting a per-tenant metric series.
LagByTenant(ctx context.Context) (map[string]float64, error)
// ReprojectTenant re-writes stored fact AND event payloads for a tenant in
// place, applying transform to each. aggregate "" means every aggregate.
// It returns the number of rows whose payload actually changed. Used to
// retroactively apply a tightened redaction allow-list to already-stored
// data (a privacy correction): the transform re-projects each stored payload
// through the entity's current AnalyticsSpec. Idempotent — a second run with
// the same transform rewrites nothing.
ReprojectTenant(ctx context.Context, tenantID, aggregate string, transform func(payload json.RawMessage) (json.RawMessage, error)) (rowsRewritten int64, err error)
// PruneEvents deletes append-only history events committed strictly before
// olderThan, across all tenants, and returns the count removed. It is the
// retention control for the unbounded event log; facts (latest state) are
// never pruned — only the history. Idempotent.
PruneEvents(ctx context.Context, olderThan time.Time) (rowsDeleted int64, err error)
// MaintainPartitions is the partition-management hook for sinks that
// range-partition the event log: it creates upcoming partitions and, when
// retention > 0, drops partitions entirely older than retention (instant
// reclaim). Sinks that do not partition return (0, 0, nil).
MaintainPartitions(ctx context.Context, retention time.Duration) (created, dropped int, err error)
// PurgeTenant hard-deletes ALL of one tenant's co-located data — facts,
// events, and watermarks — and returns the number of rows removed. It is the
// erasure primitive for tenant offboarding and right-to-be-forgotten
// requests: the ONE deliberately destructive operation on the sink, and the
// only supported way to remove a tenant from the shared store. Idempotent
// (purging an absent tenant deletes nothing and returns 0). Never invoked
// automatically — an explicit operator step, like dropping a tenant database.
PurgeTenant(ctx context.Context, tenantID string) (rowsDeleted int64, err error)
Close() error
}
Sink persists analytics records. All methods are idempotent under at-least-once delivery: UpsertFacts version-gates (older/equal versions are no-ops), AppendEvents dedupes on the natural key, SetWatermark advances monotonically. Implemented by adapters/pganalytics and fabriqtest.FakeAnalyticsSink (one conformance suite gates both).
type SnapshotFunc ¶
SnapshotFunc streams a tenant's current-state envelopes (satisfied by postgres.Adapter.SnapshotEntities and the catalog DynamicSet-routed snapshot). Backfill drives the SAME applier as live consume, so the two agree; the sink's version gate makes a re-run a no-op and lets live traffic run concurrently.