Documentation
¶
Overview ¶
Package compaction merges small Parquet files within a partition into larger files, reducing S3 list overhead and scan file-open costs.
Index ¶
Constants ¶
const DefaultCompactionInterval = 5 * time.Minute
const DefaultDeleteGrace = 30 * time.Minute
DefaultDeleteGrace keeps compacted-away bytes alive long enough for any in-flight query dispatched against the old manifest to finish reading them.
const DefaultGCMinAge = 30 * time.Minute
DefaultGCMinAge is the minimum age before delete markers are eligible for GC. Set to 30 minutes to safely exceed the duration of long-running analytical queries, preventing GC from rewriting files that are being scanned.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BackgroundCompactor ¶
type BackgroundCompactor struct {
// contains filtered or unexported fields
}
BackgroundCompactor runs periodic compaction sweeps across all tables.
func NewBackgroundCompactor ¶
func NewBackgroundCompactor(cat *catalog.Catalog, cfg BackgroundConfig, logger *slog.Logger) *BackgroundCompactor
NewBackgroundCompactor creates a background compactor.
func (*BackgroundCompactor) Start ¶
func (bc *BackgroundCompactor) Start(ctx context.Context)
Start launches the background compaction loop.
type BackgroundConfig ¶
type BackgroundConfig struct {
// Enabled controls whether background compaction runs. Default: true.
Enabled bool
// Interval between compaction sweeps. Zero uses DefaultCompactionInterval.
Interval time.Duration
// Compaction controls compaction trigger thresholds.
Compaction Config
// GCMinAge is the minimum age before delete markers are garbage collected
// and their files are force-rewritten. Zero uses DefaultGCMinAge.
GCMinAge time.Duration
}
BackgroundConfig controls the periodic compaction loop.
type Compactor ¶
type Compactor struct {
// contains filtered or unexported fields
}
Compactor merges small Parquet files per partition.
func (*Compactor) CompactTable ¶
CompactTable runs compaction for all partitions of a table. When a partition has more files than can be merged in one pass, multiple passes run back-to-back until the partition is fully compacted (no 5-minute wait between passes).
func (*Compactor) FlushDeferredDeletes ¶
FlushDeferredDeletes physically deletes every pending file older than DeleteGrace. Called from the background sweep; safe to call any time. Returns the number of files deleted.
func (*Compactor) ForceCompactFile ¶
func (c *Compactor) ForceCompactFile(ctx context.Context, tableName string, filePath string, gcIndices map[int64]bool) error
ForceCompactFile rewrites a single data file, applying any pending delete markers for that file. Used by delete marker GC to physically purge deleted rows from files whose markers have aged out.
Safety invariants:
- Write-before-delete: the new file is written to the object store before the old file is removed. On partial failure, the old file may become an orphan in S3, but data is never lost.
- Scoped marker removal: only the specific row indices that were applied during the rewrite are removed from the manifest. Concurrent DELETEs that add new indices between GC scan and rewrite are preserved.
- Atomic manifest swap: old file removal, new file addition, and marker cleanup happen in a single CAS operation via SwapFileForGC.
- Per-file lock: prevents double GC rewrite if two sweeps overlap.
type Config ¶
type Config struct {
// MinFiles is the minimum file count per partition to trigger compaction.
MinFiles int
// MaxFileSizeBytes is the average size below which compaction triggers.
MaxFileSizeBytes int64
// MaxFilesPerPass caps the number of files merged in one compaction pass
// to bound memory usage.
MaxFilesPerPass int
// DeleteGrace is how long a compacted-away file stays physically present
// in the object store after its manifest entry is removed. In-flight
// tasks hold file lists resolved at DISPATCH time; deleting the bytes
// the instant the manifest swaps races every running query against the
// compactor (observed 2026-06-11: first successful mid-benchmark
// compaction at SF10 deleted chunks under three dispatched scan tasks →
// "object not found" ×5 → circuit breaker open → every later query
// failed). Mirrors DefaultGCMinAge's reasoning. Zero uses
// DefaultDeleteGrace; negative deletes immediately (tests).
DeleteGrace time.Duration
}
Config controls compaction trigger thresholds and limits.