compact

package
v0.2.37 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: GPL-3.0 Imports: 8 Imported by: 0

Documentation

Overview

Package compact implements the core of span-compact: it deduplicates NDJSON streams on a user-defined field, keeping one record per key chosen by a selectable strategy. Built for the 10M-100M record range: uses an external sort instead of in-memory hash maps, so memory usage stays bounded.

Strategies: 

first    keep the first record seen for a key
last     keep the last record seen
random   keep a uniformly random record (reservoir over the group)
min      keep the record with the smallest -sort-key value
max      keep the record with the largest -sort-key value

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run 

func Run(cfg Config, in io.Reader, out io.Writer) error

Run wires the input stream through an external sort and emits the first record of each sorted group, which (given the per-strategy sort key) is the winning record.

Types

type Config 

type Config struct {
	KeyField    string
	SortField   string
	Strategy    string
	NumericSort bool
	SortMem     string
	SortTmp     string
}

Config holds the tunables for a compact run.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns the default configuration.

func (Config) Validate 

func (c Config) Validate() error

Validate checks that the configured strategy (and its prerequisites) are valid.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.