pebble

func MergeFilesInto ¶ added in v0.13.2

func MergeFilesInto(ctx context.Context, dest *enginepkg.Engine, sources []SourceFile, destSyncID string, tmpDir string, opts ...KWayOption) (*v3.SyncStatsRecord, error)

MergeFilesInto folds source syncs into destSyncID using a bounded K-way semantic merge and final SST ingestion. It materializes resource_types, resources, entitlements, and grants plus all derived indexes. Assets are not copied, matching SQLite compaction behavior.

Algorithm (external merge sort with fan-in cfg.fanIn, default 50):

• ≤ fanIn sources merge DIRECTLY into the dest Pebble — one pass, no run files (mergeSourceChunkToPebble).
• Otherwise sources are processed in fanIn-sized chunks, each chunk's buckets streamed into one sorted run file (buildChunkRunFileFromSources); run files are then merged fanIn at a time per round (mergeRunFileGroup) until ≤ fanIn remain, and the final generation is materialized into SSTs and ingested (materializeRunFilesToPebble). Each extra round re-reads and re-writes the full dataset, so fan-in should comfortably exceed the expected source count.
• Winner rule: newest discovered_at per primary key, ties keep the newer source (runRecordIsNewer) — same rule as the overlay and sqlite compactors. Sources are unpacked per chunk and their directories removed asynchronously when the chunk closes, so peak disk is O(fanIn) unpacked sources, not O(len(sources)).

Memory stays bounded regardless of input count or size — there is no in-memory dedup state — at the cost of writing and re-reading every record through run files. The overlay merge routes its oversized buckets here; see the package doc for how the strategies divide up.

On success it returns the dest sync's stats record, accumulated at the final winner-dedupe sites (never at run-file build time — run files keep duplicate keys across chunks until the last merge round), so the caller can persist the stats sidecar without re-scanning the output.

func MergeFilesIntoOverlay ¶ added in v0.13.2

func MergeFilesIntoOverlay(ctx context.Context, dest *enginepkg.Engine, sources []SourceFile, destSyncID string, tmpDir string, opts ...OverlayOption) (*v3.SyncStatsRecord, error)

MergeFilesIntoOverlay is a sqlite-shaped compactor:

• keep a bounded in-memory seen map (suffix hash → discovered_at) for each bucket while scanning sources newest-to-oldest;
• per key, admit the record with the strictly newest discovered_at; ties keep the earliest admission (the newest source). This is the same winner rule as the K-way merge (runRecordIsNewer) and the sqlite attached compactor (`a.discovered_at > m.discovered_at`), enforced via a shallow discovered_at scan of each candidate value;
• write winners through raw Pebble batches so secondary indexes are materialized once;
• let the last source scanned (the oldest — in the production skewed shape, the large base) skip the record write path when its bucket is big enough (overlayWholeSourceWorthIt): the bucket is materialized as SST files filtered against the seen set and ingested wholesale, making the base's cost proportional to bytes copied rather than records decoded; and
• degrade buckets whose seen set would outgrow memory to the K-way run-file path ADAPTIVELY (per-bucket state machine): a pre-source gate cuts at a source boundary and RESUMES via kway with the seen map frozen as the conflict oracle; a source that crosses the soft limit mid-scan may finish inside the buffer (hard limit = soft × bufferFactor) and resume at the boundary; crossing the hard limit mid-source RESTARTS the bucket — its dest keyspace is range-deleted and the whole bucket flows through the blind kway path (keeping SST ingest, total work ≤2×). Planning routes only the provably-safe (stats sum ≤ soft) and provably-doomed (single-source count > hard) buckets up front.

On success it returns the dest sync's stats record, accumulated as winners were written, so the caller can persist the stats sidecar without re-scanning the output.

func MergeInto ¶ added in v0.12.5

func MergeInto(ctx context.Context, dest *enginepkg.Engine, sources []SourceSync, destSyncID string) error

MergeInto folds every source's primary records into dest under destSyncID. Across all inputs (and any records already present under destSyncID), the newest record per logical key (by discovered_at, ties keep the incumbent) survives, and dest's derived indexes are maintained by the keep-newer write path.

destSyncID must already exist in dest. It may be non-empty: the in-place fold compaction merges partial syncs directly into the base sync's keyspace, relying on the Put*RecordsIfNewer semantics to resolve conflicts against pre-existing records. MergeInto binds the dest engine's current sync to destSyncID — the engine's write methods key off the current sync, not any per-record field.

Only the four primary record buckets are copied: resource_types, resources, entitlements, grants. Assets are intentionally NOT copied — this matches the SQLite compaction path, which folds only those four tables and drops assets from the compacted sync; copying assets here would give Pebble compacted syncs asset access the SQLite path does not have. Index keyspaces are not copied verbatim either; the per-record write path maintains them.

Sources are applied in the given order. The dedup keeps the record with the strictly-greater discovered_at; on an equal discovered_at the already-written (earlier source, or pre-existing dest) record is kept. Callers pass sources newest-first so the tie winner matches the SQLite fold.