Documentation
¶
Overview ¶
Package profile summarizes a stream of canonical records: how many there are, how often each attribute is non-empty, and which values occur most.
Index ¶
Constants ¶
const ( DefaultTopN = 10 DefaultMaxDistinct = 100_000 // MaxColumns bounds how many distinct attribute paths a profile reports, so a map column with // unbounded keys can't grow it without limit. MaxColumns = 4096 )
Defaults for the knobs Config leaves at zero.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector aggregates the records one decode worker sees. It implements formatio.RecordEncoder so the pool drives it exactly as it would an encoder; it never produces a document.
type Column ¶
type Column struct {
Name string
// TopValues are the most frequent non-empty values, most frequent first.
TopValues []ValueCount
// NonEmpty counts the records where the attribute is present and neither null nor empty.
NonEmpty int64
// Pct is NonEmpty as a percentage (0–100) of all records.
Pct float64
// Distinct is how many distinct non-empty values were counted; a lower bound when
// DistinctExact is false.
Distinct int64
// DistinctExact is false once any worker exceeded Config.MaxDistinct for this attribute:
// TopValues counts are then estimates (see ValueCount.MaxError) and Distinct a lower bound.
DistinctExact bool
}
Column summarizes one attribute path.
type Config ¶
type Config struct {
// Columns, when non-empty, restricts the profile to these attribute paths and everything
// nested under them.
Columns []string
// TopN is how many most-frequent values to keep per attribute.
TopN int
// MaxDistinct is how many distinct values each decode worker tracks per attribute exactly,
// before it falls back to approximate heavy-hitter counting.
MaxDistinct int
}
Config tunes a profile run; zero fields take the defaults above.
type Profile ¶
type Profile struct {
Columns []Column
Records int64
// ColumnsTruncated is true when more than MaxColumns attribute paths were seen and the rest
// were left out.
ColumnsTruncated bool
}
Profile is a finished summary.
type Set ¶
type Set struct {
// contains filtered or unexported fields
}
Set hands out one Collector per decode worker and merges them into a Profile once the run ends.
func (*Set) Merge ¶
Merge combines every registered collector into one Profile. It consumes the collectors, so it must be called once, after the run has finished.
func (*Set) NewCollector ¶
func (s *Set) NewCollector() (formatio.RecordEncoder, error)
NewCollector builds a worker-local Collector and registers it for the final merge.