profile

package
v0.0.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Overview

Package profile summarizes a stream of canonical records: how many there are, how often each attribute is non-empty, and which values occur most.

Index

Constants

View Source
const (
	DefaultTopN        = 10
	DefaultMaxDistinct = 100_000
	// MaxColumns bounds how many distinct attribute paths a profile reports, so a map column with
	// unbounded keys can't grow it without limit.
	MaxColumns = 4096
)

Defaults for the knobs Config leaves at zero.

Variables

This section is empty.

Functions

This section is empty.

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector aggregates the records one decode worker sees. It implements formatio.RecordEncoder so the pool drives it exactly as it would an encoder; it never produces a document.

func (*Collector) EncodeBatch

func (c *Collector) EncodeBatch(batch []record.Record) ([][]byte, error)

EncodeBatch folds batch into the collector's counts and returns no documents.

type Column

type Column struct {
	Name string
	// TopValues are the most frequent non-empty values, most frequent first.
	TopValues []ValueCount
	// NonEmpty counts the records where the attribute is present and neither null nor empty.
	NonEmpty int64
	// Pct is NonEmpty as a percentage (0–100) of all records.
	Pct float64
	// Distinct is how many distinct non-empty values were counted; a lower bound when
	// DistinctExact is false.
	Distinct int64
	// DistinctExact is false once any worker exceeded Config.MaxDistinct for this attribute:
	// TopValues counts are then estimates (see ValueCount.MaxError) and Distinct a lower bound.
	DistinctExact bool
}

Column summarizes one attribute path.

type Config

type Config struct {
	// Columns, when non-empty, restricts the profile to these attribute paths and everything
	// nested under them.
	Columns []string
	// TopN is how many most-frequent values to keep per attribute.
	TopN int
	// MaxDistinct is how many distinct values each decode worker tracks per attribute exactly,
	// before it falls back to approximate heavy-hitter counting.
	MaxDistinct int
}

Config tunes a profile run; zero fields take the defaults above.

type Profile

type Profile struct {
	Columns []Column
	Records int64
	// ColumnsTruncated is true when more than MaxColumns attribute paths were seen and the rest
	// were left out.
	ColumnsTruncated bool
}

Profile is a finished summary.

type Set

type Set struct {
	// contains filtered or unexported fields
}

Set hands out one Collector per decode worker and merges them into a Profile once the run ends.

func NewSet

func NewSet(cfg Config) *Set

NewSet returns an empty Set using cfg, with zero fields defaulted.

func (*Set) Merge

func (s *Set) Merge() Profile

Merge combines every registered collector into one Profile. It consumes the collectors, so it must be called once, after the run has finished.

func (*Set) NewCollector

func (s *Set) NewCollector() (formatio.RecordEncoder, error)

NewCollector builds a worker-local Collector and registers it for the final merge.

type ValueCount

type ValueCount struct {
	Value string
	Count int64
	// MaxError bounds how far Count may be from the true count; zero when the column is exact.
	MaxError int64
	// Pct is Count as a percentage (0–100) of all records.
	Pct float64
}

ValueCount is one value and how often it occurred.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL