Documentation
¶
Overview ¶
Package compaction merges small Parquet files within a partition into larger files, reducing S3 list overhead and scan file-open costs.
Index ¶
- Constants
- type BackgroundCompactor
- type BackgroundConfig
- type CompactionFailed
- type Compactor
- func (c *Compactor) CompactTable(ctx context.Context, tableName string) (*Result, error)
- func (c *Compactor) FlushDeferredDeletes(ctx context.Context) int
- func (c *Compactor) ForceCompactFile(ctx context.Context, tableName string, filePath string, ...) error
- func (c *Compactor) RewriteTable(ctx context.Context, tableName string) (*Result, error)
- type Config
- type PartitionFailure
- type Result
Constants ¶
const DefaultCompactionInterval = 5 * time.Minute
const DefaultDeleteGrace = 30 * time.Minute
DefaultDeleteGrace keeps compacted-away bytes alive long enough for any in-flight query dispatched against the old manifest to finish reading them.
const DefaultGCMinAge = 30 * time.Minute
DefaultGCMinAge is the minimum age before delete markers are eligible for GC. Set to 30 minutes to safely exceed the duration of long-running analytical queries, preventing GC from rewriting files that are being scanned.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BackgroundCompactor ¶
type BackgroundCompactor struct {
// contains filtered or unexported fields
}
BackgroundCompactor runs periodic compaction sweeps across all tables.
func NewBackgroundCompactor ¶
func NewBackgroundCompactor(cat *catalog.Catalog, cfg BackgroundConfig, logger *slog.Logger) *BackgroundCompactor
NewBackgroundCompactor creates a background compactor.
func (*BackgroundCompactor) Start ¶
func (bc *BackgroundCompactor) Start(ctx context.Context)
Start launches the background compaction loop.
type BackgroundConfig ¶
type BackgroundConfig struct {
// Enabled controls whether background compaction runs. Default: true.
Enabled bool
// Interval between compaction sweeps. Zero uses DefaultCompactionInterval.
Interval time.Duration
// Compaction controls compaction trigger thresholds.
Compaction Config
// GCMinAge is the minimum age before delete markers are garbage collected
// and their files are force-rewritten. Zero uses DefaultGCMinAge.
GCMinAge time.Duration
}
BackgroundConfig controls the periodic compaction loop.
type CompactionFailed ¶ added in v0.18.1
type CompactionFailed struct {
Table string
Failures []PartitionFailure
Compacted int
}
CompactionFailed is the aggregate error CompactTable and RewriteTable return when at least one partition's merge failed.
One error for the whole table, not the first failure, because a merge failure is scoped to the partition whose files could not be read: #435 correctly made a failed merge visible to the caller, but by RETURNING at the first one, so a single drifted partition froze compaction of every other partition in the table — and, since the background sweep `continue`s to the next table on an error from here, the table's delete-marker GC with it.
Compacted lets a caller tell a partial run from a total one without inspecting the Result: zero means nothing in this table compacted, non-zero means the named partitions failed while the rest went through.
func (*CompactionFailed) Error ¶ added in v0.18.1
func (e *CompactionFailed) Error() string
func (*CompactionFailed) Partial ¶ added in v0.18.1
func (e *CompactionFailed) Partial() bool
Partial reports whether some partitions compacted despite the failures.
func (*CompactionFailed) Unwrap ¶ added in v0.18.1
func (e *CompactionFailed) Unwrap() []error
Unwrap exposes the individual causes to errors.Is and errors.As.
type Compactor ¶
type Compactor struct {
// contains filtered or unexported fields
}
Compactor merges small Parquet files per partition.
func (*Compactor) CompactTable ¶
CompactTable runs compaction for all partitions of a table. When a partition has more files than can be merged in one pass, multiple passes run back-to-back until the partition is fully compacted (no 5-minute wait between passes).
A partition whose merge fails does not stop the others: the failure is recorded in Result.Failed, the partition is skipped for the rest of the call, and the aggregate *CompactionFailed comes back at the end. See that type for why the first-failure return this replaced was too blunt.
func (*Compactor) FlushDeferredDeletes ¶
FlushDeferredDeletes physically deletes every pending file older than DeleteGrace. Called from the background sweep; safe to call any time. Returns the number of files deleted.
func (*Compactor) ForceCompactFile ¶
func (c *Compactor) ForceCompactFile(ctx context.Context, tableName string, filePath string, gcIndices map[int64]bool) error
ForceCompactFile rewrites a single data file, applying any pending delete markers for that file. Used by delete marker GC to physically purge deleted rows from files whose markers have aged out.
Safety invariants:
- Write-before-delete: the new file is written to the object store before the old file is removed. On partial failure, the old file may become an orphan in S3, but data is never lost.
- Scoped marker removal: only the specific row indices that were applied during the rewrite are removed from the manifest. Concurrent DELETEs that add new indices between GC scan and rewrite are preserved.
- Atomic manifest swap: old file removal, new file addition, and marker cleanup happen in a single CAS operation via SwapFileForGC.
- Per-file lock: prevents double GC rewrite if two sweeps overlap.
func (*Compactor) RewriteTable ¶ added in v0.18.1
RewriteTable rewrites EVERY file of every partition of a table exactly once, through the current writer, and replaces the originals.
This is the format-migration mode, and it is deliberately not compaction. shouldCompact's floors — two files, MinFiles, an average size under MaxFileSizeBytes — all answer "is this partition worth merging", which is the right question for a background sweep and the wrong one for a migration: a partition holding ONE 512 MB file is exactly the file that has to be rewritten, and it is the one shape compaction will never touch. So a rewrite is exempt from the floors and admits a 1 -> 1 pass.
It terminates structurally rather than by CompactTable's progress rule. The file list is read from the manifest ONCE, split into memory-bounded groups, and each group is written once; nothing re-reads the manifest, so no output of this call can become an input to it. "1 removed, 1 created" is progress here, which is precisely why the progress rule cannot apply.
Its use is ADR-0018's DECIMAL(p > 18) migration: files written before #429 annotate a wide DECIMAL over an INT64 leaf, and no reader outside wadjet can open them. One rewrite through the current writer produces a FLBA(16) leaf with byte-identical unscaled values. Every other type round-trips unchanged (that is the compaction gate's property), so running it over a table that needs nothing costs the rewrite and changes no value.
Like CompactTable, a partition whose merge fails does not stop the others; the aggregate is *CompactionFailed.
type Config ¶
type Config struct {
// MinFiles is the minimum file count per partition to trigger compaction.
MinFiles int
// MaxFileSizeBytes is the average size below which compaction triggers.
MaxFileSizeBytes int64
// MaxFilesPerPass caps the number of files merged in one compaction pass
// to bound memory usage.
MaxFilesPerPass int
// DeleteGrace is how long a compacted-away file stays physically present
// in the object store after its manifest entry is removed. In-flight
// tasks hold file lists resolved at DISPATCH time; deleting the bytes
// the instant the manifest swaps races every running query against the
// compactor (observed 2026-06-11: first successful mid-benchmark
// compaction at SF10 deleted chunks under three dispatched scan tasks →
// "object not found" ×5 → circuit breaker open → every later query
// failed). Mirrors DefaultGCMinAge's reasoning. Zero uses
// DefaultDeleteGrace; negative deletes immediately (tests).
DeleteGrace time.Duration
}
Config controls compaction trigger thresholds and limits.
type PartitionFailure ¶ added in v0.18.1
PartitionFailure is one partition whose merge failed, and why.
type Result ¶
type Result struct {
Table string
// PartitionsCompacted counts successful MERGES, not distinct partitions:
// a partition compacted over three passes, or rewritten as three
// memory-bounded groups, counts three times.
PartitionsCompacted int
FilesRemoved int
FilesCreated int
RowsMerged int64
BytesBefore int64
BytesAfter int64
Failed []PartitionFailure
// PassLimitReached reports that the multi-pass loop stopped at
// maxCompactionPasses with the table still shrinking. It is a flag on a
// SUCCESSFUL result rather than an error: every one of those passes had
// to remove more files than it created to get there (the progress rule
// in CompactTable), so the work is real and committed, and calling the
// call a failure would discard a correct result and — through the
// background sweep's `continue` on error — skip the table's
// delete-marker GC as well. The next sweep picks the table up where
// this one left off.
PassLimitReached bool
}
Result summarizes one compaction pass for a table.
Failed names the partitions whose merge did not run. It is part of the result rather than only of the log because the counters below count only the partitions that DID compact, so a caller reading them alone cannot tell a clean run from one where a partition failed on every pass (#435).
Source Files