compaction

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 23, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Overview

Package compaction merges small Parquet files within a partition into larger files, reducing S3 list overhead and scan file-open costs.

Index

Constants

View Source
const DefaultCompactionInterval = 5 * time.Minute
View Source
const DefaultDeleteGrace = 30 * time.Minute

DefaultDeleteGrace keeps compacted-away bytes alive long enough for any in-flight query dispatched against the old manifest to finish reading them.

View Source
const DefaultGCMinAge = 30 * time.Minute

DefaultGCMinAge is the minimum age before delete markers are eligible for GC. Set to 30 minutes to safely exceed the duration of long-running analytical queries, preventing GC from rewriting files that are being scanned.

Variables

This section is empty.

Functions

This section is empty.

Types

type BackgroundCompactor

type BackgroundCompactor struct {
	// contains filtered or unexported fields
}

BackgroundCompactor runs periodic compaction sweeps across all tables.

func NewBackgroundCompactor

func NewBackgroundCompactor(cat *catalog.Catalog, cfg BackgroundConfig, logger *slog.Logger) *BackgroundCompactor

NewBackgroundCompactor creates a background compactor.

func (*BackgroundCompactor) Start

func (bc *BackgroundCompactor) Start(ctx context.Context)

Start launches the background compaction loop.

type BackgroundConfig

type BackgroundConfig struct {
	// Enabled controls whether background compaction runs. Default: true.
	Enabled bool
	// Interval between compaction sweeps. Zero uses DefaultCompactionInterval.
	Interval time.Duration
	// Compaction controls compaction trigger thresholds.
	Compaction Config
	// GCMinAge is the minimum age before delete markers are garbage collected
	// and their files are force-rewritten. Zero uses DefaultGCMinAge.
	GCMinAge time.Duration
}

BackgroundConfig controls the periodic compaction loop.

type Compactor

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor merges small Parquet files per partition.

func New

func New(cat *catalog.Catalog, logger *slog.Logger, cfg Config) *Compactor

New creates a compactor.

func (*Compactor) CompactTable

func (c *Compactor) CompactTable(ctx context.Context, tableName string) (*Result, error)

CompactTable runs compaction for all partitions of a table. When a partition has more files than can be merged in one pass, multiple passes run back-to-back until the partition is fully compacted (no 5-minute wait between passes).

func (*Compactor) FlushDeferredDeletes

func (c *Compactor) FlushDeferredDeletes(ctx context.Context) int

FlushDeferredDeletes physically deletes every pending file older than DeleteGrace. Called from the background sweep; safe to call any time. Returns the number of files deleted.

func (*Compactor) ForceCompactFile

func (c *Compactor) ForceCompactFile(ctx context.Context, tableName string, filePath string, gcIndices map[int64]bool) error

ForceCompactFile rewrites a single data file, applying any pending delete markers for that file. Used by delete marker GC to physically purge deleted rows from files whose markers have aged out.

Safety invariants:

  • Write-before-delete: the new file is written to the object store before the old file is removed. On partial failure, the old file may become an orphan in S3, but data is never lost.
  • Scoped marker removal: only the specific row indices that were applied during the rewrite are removed from the manifest. Concurrent DELETEs that add new indices between GC scan and rewrite are preserved.
  • Atomic manifest swap: old file removal, new file addition, and marker cleanup happen in a single CAS operation via SwapFileForGC.
  • Per-file lock: prevents double GC rewrite if two sweeps overlap.

type Config

type Config struct {
	// MinFiles is the minimum file count per partition to trigger compaction.
	MinFiles int
	// MaxFileSizeBytes is the average size below which compaction triggers.
	MaxFileSizeBytes int64
	// MaxFilesPerPass caps the number of files merged in one compaction pass
	// to bound memory usage.
	MaxFilesPerPass int
	// DeleteGrace is how long a compacted-away file stays physically present
	// in the object store after its manifest entry is removed. In-flight
	// tasks hold file lists resolved at DISPATCH time; deleting the bytes
	// the instant the manifest swaps races every running query against the
	// compactor (observed 2026-06-11: first successful mid-benchmark
	// compaction at SF10 deleted chunks under three dispatched scan tasks →
	// "object not found" ×5 → circuit breaker open → every later query
	// failed). Mirrors DefaultGCMinAge's reasoning. Zero uses
	// DefaultDeleteGrace; negative deletes immediately (tests).
	DeleteGrace time.Duration
}

Config controls compaction trigger thresholds and limits.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns production defaults.

type Result

type Result struct {
	Table               string
	PartitionsCompacted int
	FilesRemoved        int
	FilesCreated        int
	RowsMerged          int64
	BytesBefore         int64
	BytesAfter          int64
}

Result summarizes one compaction pass for a table.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL