Documentation
¶
Overview ¶
Package drum is the on-disk seen-set and URL index of a partition (spec 2072 doc 04, Stage B): a real Disk Repository with Update Management (IRLbot, Lee et al. 2008) that absorbs the dedup exact set and the store's resident URL index into one on-disk authority whose aux value is the record location (D2). A membership probe and a location lookup are the same probe, the seen-set and the index are maintained once on disk instead of twice resident, and the resident cost is a fixed budget (the in-flight bucket buffers plus a ~0.17 B/url block index) independent of how many distinct URLs the partition has ever seen.
The shape: K=256 in-memory buckets aligned to the store's 256 shards accumulate discovered (URLKey, locEntry) pairs; a bucket flushes to a bucket-NNN.pending file when it fills; a merge k-way-merges the pending runs against the single sorted repository in one sequential sweep, classifying each key unique-or- duplicate (the dedup check) and updating its location in place (the index update). The point read for a cold due URL binary-searches a resident sparse block index to one block and reads it. Pure Go, CGO_ENABLED=0, no internal/.
Index ¶
- type Classification
- type DRUM
- func (d *DRUM) Close() error
- func (d *DRUM) Discover(key meguri.URLKey, off int64, lsn uint64) error
- func (d *DRUM) Insert(key meguri.URLKey, off int64, lsn uint64) error
- func (d *DRUM) Locate(key meguri.URLKey) (off int64, lsn uint64, present bool, err error)
- func (d *DRUM) Merge() ([]Classification, error)
- func (d *DRUM) OpenRepoCursor() (*RepoCursor, error)
- func (d *DRUM) ScanRepo(fn func(key meguri.URLKey, off int64, lsn uint64) error) error
- func (d *DRUM) Tombstone(key meguri.URLKey, off int64, lsn uint64) error
- func (d *DRUM) Unmerged() int64
- type Options
- type RepoCursor
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Classification ¶
Classification is a dedup verdict the discovery path consumes: a key and whether the merge found it genuinely new (Unique) or already in the repository.
type DRUM ¶
type DRUM struct {
// contains filtered or unexported fields
}
DRUM is a partition's on-disk seen-set and index.
func Open ¶
Open opens or recovers a DRUM rooted at dir. It creates the drum/ subdirectory, discards any torn .pending file left by an interrupted flush (recovery re-derives its entries from the log, section 3.6), and loads the resident block index over the existing repository (rebuilding it from the repository if the index file is missing or torn).
func (*DRUM) Close ¶
Close flushes nothing implicitly (a caller checkpoints by calling Merge) and releases the repository file handle. Buffered, unflushed discoveries are re-derivable from the log, so closing without a final Merge loses no data.
func (*DRUM) Discover ¶
Discover buffers a check-and-insert: the discovery wants to know whether key is new (the verdict comes from the next Merge) and to index its record at loc. The pair is routed to bucket key.HostKey&255 and the bucket flushes if it fills.
func (*DRUM) Insert ¶
Insert buffers an insert-only entry, the recovery and handoff path: index the record without producing a dedup verdict (section 2.2).
func (*DRUM) Locate ¶
Locate answers the point read: given a URLKey, return its current locEntry, or report absent. It consults the in-flight overlay (resident buckets and unmerged .pending files, which hold discoveries newer than the repository) before the repository, and returns the highest-LSN location across all of them so a rediscovery whose record moved since the last merge is found at its newest frame (section 4.2). A tombstoned key reports absent.
func (*DRUM) Merge ¶
func (d *DRUM) Merge() ([]Classification, error)
Merge folds every buffered and flushed discovery into the repository in one sequential sweep, returning the dedup verdicts. It flushes all non-empty buckets first so the merge sees a consistent cut of the discovery stream, gathers all .pending runs, runs the merge cycle, swaps the fresh repository and block index in, and drops the consumed pending files.
func (*DRUM) OpenRepoCursor ¶
func (d *DRUM) OpenRepoCursor() (*RepoCursor, error)
OpenRepoCursor opens a cursor over the current repository.
func (*DRUM) ScanRepo ¶
ScanRepo walks the merged repository in URLKey order, calling fn for every live (non-tombstoned) key with its location. It is the checkpoint and count path: the repository is already globally sorted by URLKey, which is the snapshot's row order, so a checkpoint streams straight from this scan with no extra sort or k-way merge. The caller must Merge first so the repository reflects all discoveries; ScanRepo reads only the on-disk repository, never the in-flight buckets.
type Options ¶
type Options struct {
// FlushBytes is the per-bucket serialized size at which a bucket flushes to a
// .pending file. Zero picks a 1 MiB default (~33k entries per bucket).
FlushBytes int
// ReadBuf and WriteBuf size the merge's sequential repository reader and writer.
// Zero picks 1 MiB each.
ReadBuf int
WriteBuf int
}
Options configure a DRUM at open.
type RepoCursor ¶
type RepoCursor struct {
// contains filtered or unexported fields
}
RepoCursor is a pull-based sequential reader over the merged repository in URLKey order, the streaming-checkpoint source (a host store wraps it to read each record's body from the log and feed format.StreamEncodeToFile). It reads only the on-disk repository; the caller Merges first so the repository is current.
func (*RepoCursor) Close ¶
func (c *RepoCursor) Close() error
Source Files