compaction

package
v0.18.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 29, 2026 License: AGPL-3.0 Imports: 14 Imported by: 0

Documentation

Overview

Package compaction merges small Parquet files within a partition into larger files, reducing S3 list overhead and scan file-open costs.

Index

Constants

View Source
const DefaultCompactionInterval = 5 * time.Minute
View Source
const DefaultDeleteGrace = 30 * time.Minute

DefaultDeleteGrace keeps compacted-away bytes alive long enough for any in-flight query dispatched against the old manifest to finish reading them.

View Source
const DefaultGCMinAge = 30 * time.Minute

DefaultGCMinAge is the minimum age before delete markers are eligible for GC. Set to 30 minutes to safely exceed the duration of long-running analytical queries, preventing GC from rewriting files that are being scanned.

Variables

This section is empty.

Functions

This section is empty.

Types

type BackgroundCompactor

type BackgroundCompactor struct {
	// contains filtered or unexported fields
}

BackgroundCompactor runs periodic compaction sweeps across all tables.

func NewBackgroundCompactor

func NewBackgroundCompactor(cat *catalog.Catalog, cfg BackgroundConfig, logger *slog.Logger) *BackgroundCompactor

NewBackgroundCompactor creates a background compactor.

func (*BackgroundCompactor) Start

func (bc *BackgroundCompactor) Start(ctx context.Context)

Start launches the background compaction loop.

type BackgroundConfig

type BackgroundConfig struct {
	// Enabled controls whether background compaction runs. Default: true.
	Enabled bool
	// Interval between compaction sweeps. Zero uses DefaultCompactionInterval.
	Interval time.Duration
	// Compaction controls compaction trigger thresholds.
	Compaction Config
	// GCMinAge is the minimum age before delete markers are garbage collected
	// and their files are force-rewritten. Zero uses DefaultGCMinAge.
	GCMinAge time.Duration
	// DropGrace is the minimum age before a dropped table's data files are
	// physically deleted (catalog.Catalog.FlushDroppedTableFiles). Zero
	// uses catalog.DefaultDropTableGrace. Only consulted when
	// ReclaimDroppedTables is true.
	DropGrace time.Duration
	// ReclaimDroppedTables controls whether the sweep calls
	// catalog.Catalog.FlushDroppedTableFiles at all. Default: false.
	//
	// Deliberately opt-in rather than tied to Enabled (#494 review): this
	// *Catalog is not the only one a DROP can go through. An embedded
	// wadjet.DB and a standalone pgwire DB each own a separate *Catalog
	// from the one a BackgroundCompactor sweeps (cmd/wadjet/main.go's
	// standalone mode is the concrete case — its pgwire server opens its
	// own wadjet.DB), so turning this on unconditionally would reclaim a
	// DROPped table's files for queries issued through the compactor's
	// catalog while silently never reclaiming ones issued through psql or
	// the embedded API against the others. An explicit, honest default of
	// "not reclaimed anywhere yet" beats an inconsistent "reclaimed here,
	// not there" that looks like a bug until someone reads the wiring.
	// Turning it on is safe wherever it runs: FlushDroppedTableFiles's
	// live-manifest guard and table-prefix scoping hold regardless of
	// which *Catalog instance calls it.
	ReclaimDroppedTables bool
}

BackgroundConfig controls the periodic compaction loop.

type CompactionFailed added in v0.18.1

type CompactionFailed struct {
	Table     string
	Failures  []PartitionFailure
	Compacted int
}

CompactionFailed is the aggregate error CompactTable and RewriteTable return when at least one partition's merge failed.

One error for the whole table, not the first failure, because a merge failure is scoped to the partition whose files could not be read: #435 correctly made a failed merge visible to the caller, but by RETURNING at the first one, so a single drifted partition froze compaction of every other partition in the table — and, since the background sweep `continue`s to the next table on an error from here, the table's delete-marker GC with it.

Compacted lets a caller tell a partial run from a total one without inspecting the Result: zero means nothing in this table compacted, non-zero means the named partitions failed while the rest went through.

func (*CompactionFailed) Error added in v0.18.1

func (e *CompactionFailed) Error() string

func (*CompactionFailed) Partial added in v0.18.1

func (e *CompactionFailed) Partial() bool

Partial reports whether some partitions compacted despite the failures.

func (*CompactionFailed) Unwrap added in v0.18.1

func (e *CompactionFailed) Unwrap() []error

Unwrap exposes the individual causes to errors.Is and errors.As.

type Compactor

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor merges small Parquet files per partition.

func New

func New(cat *catalog.Catalog, logger *slog.Logger, cfg Config) *Compactor

New creates a compactor.

func (*Compactor) CompactTable

func (c *Compactor) CompactTable(ctx context.Context, tableName string) (*Result, error)

CompactTable runs compaction for all partitions of a table. When a partition has more files than can be merged in one pass, multiple passes run back-to-back until the partition is fully compacted (no 5-minute wait between passes).

A partition whose merge fails does not stop the others: the failure is recorded in Result.Failed, the partition is skipped for the rest of the call, and the aggregate *CompactionFailed comes back at the end. See that type for why the first-failure return this replaced was too blunt.

func (*Compactor) FlushDeferredDeletes

func (c *Compactor) FlushDeferredDeletes(ctx context.Context) int

FlushDeferredDeletes physically deletes every pending file older than DeleteGrace. Called from the background sweep; safe to call any time. Returns the number of files deleted.

func (*Compactor) ForceCompactFile

func (c *Compactor) ForceCompactFile(ctx context.Context, tableName string, filePath string, gcIndices map[int64]bool) error

ForceCompactFile rewrites a single data file, applying any pending delete markers for that file. Used by delete marker GC to physically purge deleted rows from files whose markers have aged out.

Safety invariants:

  • Write-before-delete: the new file is written to the object store before the old file is removed. On partial failure, the old file may become an orphan in S3, but data is never lost.
  • Scoped marker removal: only the specific row indices that were applied during the rewrite are removed from the manifest. Concurrent DELETEs that add new indices between GC scan and rewrite are preserved.
  • Atomic manifest swap: old file removal, new file addition, and marker cleanup happen in a single CAS operation via SwapFileForGC.
  • Per-file lock: prevents double GC rewrite if two sweeps overlap.

func (*Compactor) RewriteTable added in v0.18.1

func (c *Compactor) RewriteTable(ctx context.Context, tableName string) (*Result, error)

RewriteTable rewrites EVERY file of every partition of a table exactly once, through the current writer, and replaces the originals.

This is the format-migration mode, and it is deliberately not compaction. shouldCompact's floors — two files, MinFiles, an average size under MaxFileSizeBytes — all answer "is this partition worth merging", which is the right question for a background sweep and the wrong one for a migration: a partition holding ONE 512 MB file is exactly the file that has to be rewritten, and it is the one shape compaction will never touch. So a rewrite is exempt from the floors and admits a 1 -> 1 pass.

It terminates structurally rather than by CompactTable's progress rule. The file list is read from the manifest ONCE, split into memory-bounded groups, and each group is written once; nothing re-reads the manifest, so no output of this call can become an input to it. "1 removed, 1 created" is progress here, which is precisely why the progress rule cannot apply.

Its use is ADR-0018's DECIMAL(p > 18) migration: files written before #429 annotate a wide DECIMAL over an INT64 leaf, and no reader outside wadjet can open them. One rewrite through the current writer produces a FLBA(16) leaf with byte-identical unscaled values. Every other type round-trips unchanged (that is the compaction gate's property), so running it over a table that needs nothing costs the rewrite and changes no value.

Like CompactTable, a partition whose merge fails does not stop the others; the aggregate is *CompactionFailed.

type Config

type Config struct {
	// MinFiles is the minimum file count per partition to trigger compaction.
	MinFiles int
	// MaxFileSizeBytes is the average size below which compaction triggers.
	MaxFileSizeBytes int64
	// MaxFilesPerPass caps the number of files merged in one compaction pass
	// to bound memory usage.
	MaxFilesPerPass int
	// DeleteGrace is how long a compacted-away file stays physically present
	// in the object store after its manifest entry is removed. In-flight
	// tasks hold file lists resolved at DISPATCH time; deleting the bytes
	// the instant the manifest swaps races every running query against the
	// compactor (observed 2026-06-11: first successful mid-benchmark
	// compaction at SF10 deleted chunks under three dispatched scan tasks →
	// "object not found" ×5 → circuit breaker open → every later query
	// failed). Mirrors DefaultGCMinAge's reasoning. Zero uses
	// DefaultDeleteGrace; negative deletes immediately (tests).
	DeleteGrace time.Duration
}

Config controls compaction trigger thresholds and limits.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns production defaults.

type PartitionFailure added in v0.18.1

type PartitionFailure struct {
	Partition string
	Err       error
}

PartitionFailure is one partition whose merge failed, and why.

type Result

type Result struct {
	Table string
	// PartitionsCompacted counts successful MERGES, not distinct partitions:
	// a partition compacted over three passes, or rewritten as three
	// memory-bounded groups, counts three times.
	PartitionsCompacted int
	FilesRemoved        int
	FilesCreated        int
	RowsMerged          int64
	BytesBefore         int64
	BytesAfter          int64
	Failed              []PartitionFailure

	// PassLimitReached reports that the multi-pass loop stopped at
	// maxCompactionPasses with the table still shrinking. It is a flag on a
	// SUCCESSFUL result rather than an error: every one of those passes had
	// to remove more files than it created to get there (the progress rule
	// in CompactTable), so the work is real and committed, and calling the
	// call a failure would discard a correct result and — through the
	// background sweep's `continue` on error — skip the table's
	// delete-marker GC as well. The next sweep picks the table up where
	// this one left off.
	PassLimitReached bool
}

Result summarizes one compaction pass for a table.

Failed names the partitions whose merge did not run. It is part of the result rather than only of the log because the counters below count only the partitions that DID compact, so a caller reading them alone cannot tell a clean run from one where a partition failed on every pass (#435).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL