compaction

func CollectDeadEqualityDeletes ¶

func CollectDeadEqualityDeletes(
	ctx context.Context,
	fs iceio.IO,
	snap *table.Snapshot,
	rewrittenPaths map[string]struct{},
) ([]iceberg.DataFile, error)

CollectDeadEqualityDeletes walks the given snapshot's manifests and returns equality delete files that are provably dead after a rewrite removes the data files in rewrittenPaths. It preserves the original conservative API, which treats every candidate as potentially global because it has no spec metadata. New callers should use CollectDeadEqualityDeletesWithSpecs.

"Dead" means: no surviving data file in the snapshot (excluding rewrittenPaths) could apply to the eq-delete under the conservative compatibility predicate (see DecideDeadEqualityDeletes).

rewrittenPaths is the union of every old data file path being replaced by a planned rewrite (across all groups). The caller typically computes this from a compaction plan.

The returned files are safe to remove during the same commit that stages the rewrite — they cannot be resurrected by any concurrent commit because (1) any concurrent eq-delete is rejected by the rewrite-validator, and (2) any concurrent data file gets a fresh sequence number strictly greater than every preexisting eq-delete, so it cannot satisfy E.seq > D.seq for any preexisting E.

Two-pass design: the first pass walks delete manifests only to discover candidate eq-deletes; if there are none, the more expensive data-manifest walk is skipped.

func CollectDeadEqualityDeletesWithSpecs ¶ added in v0.7.0

func CollectDeadEqualityDeletesWithSpecs(
	ctx context.Context,
	fs iceio.IO,
	specs PartitionSpecLookup,
	snap *table.Snapshot,
	rewrittenPaths map[string]struct{},
) ([]iceberg.DataFile, error)

CollectDeadEqualityDeletesWithSpecs walks the given snapshot's manifests and uses partition spec metadata to identify dead equality deletes exactly.

func CollectDeadPositionDeletes ¶ added in v0.7.0

func CollectDeadPositionDeletes(
	ctx context.Context,
	fs iceio.IO,
	snap *table.Snapshot,
	rewrittenPaths map[string]struct{},
) ([]iceberg.DataFile, error)

CollectDeadPositionDeletes walks the given snapshot's manifests and returns the position-delete files made dead by a rewrite that removes rewrittenPaths. A dead position delete references only data files the rewrite is removing, so after the rewrite it applies to nothing and is safe to expunge in the same commit.

Scope resolution mirrors Java ContentFileUtil.referencedDataFile and DeleteFileIndex:

• File-scoped — explicit referenced_data_file, or equal file_path lower/upper bounds — is dead iff its single target is in rewrittenPaths.
• Partition-scoped — no single target — is dead iff no surviving (non-rewritten) data file in the same (specID, partition) has a sequence number <= the delete's. Such a survivor is one the delete still applies to, so the delete must be retained for a later dangling-delete pass.

Deletion vectors (Puffin position deletes) are intentionally excluded: they are 1:1 with their data file and expunged per-group via table.CompactionGroupResult.SafeDeletionVectors.

rewrittenPaths is the union of every old data file path being replaced across all rewrite groups. This is the whole-rewrite re-check that table.CollectSafePositionDeletes's caller contract requires whenever multiple groups land in one rewrite snapshot; the result feeds table.RewriteDataFilesOptions.ExtraDeleteFilesToRemove. Like CollectDeadEqualityDeletesWithSpecs, the returned files are safe to remove in the same commit that stages the rewrite: a concurrent commit cannot resurrect them, because any concurrent delete touching a rewritten file is rejected by the rewrite validator and any concurrent data file gets a sequence number strictly greater than every preexisting delete.

Two-pass design: the first pass walks delete manifests to resolve candidates, deciding file-scoped ones immediately; the more expensive data-manifest walk runs only when a partition-scoped candidate needs a survivor check.

func DecideDeadEqualityDeletes ¶

func DecideDeadEqualityDeletes(survey *SurvivorSurvey, candidates []iceberg.ManifestEntry) []iceberg.DataFile

DecideDeadEqualityDeletes conservatively returns equality-delete files that are dead without partition spec metadata. Because a non-empty partition tuple may belong to a void-only spec whose deletes apply globally, this compatibility API assumes every candidate could be global. It may preserve deletes that exact spec-aware matching can prove dead. New callers with table metadata should use DecideDeadEqualityDeletesWithSpecs.

func DecideDeadEqualityDeletesWithSpecs ¶ added in v0.7.0

func DecideDeadEqualityDeletesWithSpecs(
	survey *SurvivorSurvey,
	candidates []iceberg.ManifestEntry,
	specs PartitionSpecLookup,
) ([]iceberg.DataFile, error)

DecideDeadEqualityDeletesWithSpecs is the exact spec-aware predicate for equality-delete cleanup during compaction. Given a survey of surviving data file sequences populated with SurvivorSurvey.AddSurvivorWithSpec, it returns the equality-delete files that no surviving data file could apply to.

The cleanup rule is:

E applies to D iff E.seq > D.seq AND (
    E.spec.isUnpartitioned() ||
    (E.specID == D.specID && E.partition == D.partition)
)

E is dead iff no applicable surviving D has D.seq < E.seq — equivalently, the applicable min-seq is >= E.seq.

Defensive: candidates with sequence number < 0 (sentinel for unset) are skipped — preserved rather than risk dropping an unidentifiable file.

Dedup by file path: the same eq-delete file may appear in multiple manifest entries after manifest merging.