Documentation
¶
Overview ¶
Package pebble implements the cross-engine compaction primitive for the v3 Pebble-backed storage engine.
The compactor takes a `base` engine and one or more `applied` engines (each representing a sync_run worth of records) and atomically merges `applied`'s data into `base` using pebble.DB.IngestAndExcise:
- For each applied engine, iterate its records in the source engine's key order and accumulate them into an SST file on disk.
- Call base.DB.IngestAndExcise(paths, exciseSpan) with the new SST and the key range covering the applied sync_id. Pebble atomically: (a) excises every key in [exciseSpan.Start, exciseSpan.End) from base — old sync_id rows go away in one shot. (b) ingests the SST as a new L6 file (or flushable, depending on Pebble's choice) under that range.
The net effect is a byte-level merge: zero proto encode/decode per record, zero LSM compaction churn from a record-by-record Put loop.
Bound: each Compact call replaces exactly one sync_id range in the destination. Multi-sync rollup is a higher-level loop that Compact in sequence.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrEmptySync = errors.New("synccompactor/pebble: source has no records under the given sync_id")
ErrEmptySync is returned when Compact is called with a sync_id that has no records in the source engine. The caller can treat this as a no-op or as an error depending on context.
Functions ¶
This section is empty.
Types ¶
type Compactor ¶
type Compactor struct {
// contains filtered or unexported fields
}
Compactor merges sync_run data between two Pebble engines using IngestAndExcise. Reusable across many Compact calls; safe for sequential use only (not concurrent).
func NewCompactor ¶
NewCompactor builds a compactor that writes its merge into base. The tmpDir is where intermediate SST files are written; it must be on the same filesystem as the engine's data directory so Pebble can hard-link (a different FS would force a copy and break atomicity). If tmpDir is empty, os.TempDir is used (acceptable for tests but not production).
func (*Compactor) Compact ¶
Compact merges all records belonging to syncID in `source` into the base engine. Any pre-existing data in base under that syncID is excised before the new data is ingested — base ends up with exactly source's view of that sync.
Atomicity: per-bucket atomic, NOT whole-Compact atomic. Each IngestAndExcise call (one per record-type bucket: grants, by_entitlement index, by_principal index) is atomic from base's perspective — a concurrent reader sees the old-or-new state of that bucket, never a mixture. However, the multi-bucket loop is NOT transactional as a whole: a crash or hard cancellation mid-loop leaves base with new data in some buckets and old data in others (and the same for the DeleteRange-only path used for empty buckets). Recovery is "re-run Compact for the same syncID" — every step is idempotent (excise + ingest the same SSTs again converges to the source's view).
Caller is responsible for quiescing writes to `source` for the duration of the call (an active syncer would invalidate the SST as it's being built).
Source Files