Documentation
¶
Overview ¶
Package store is the durable live state store of a partition (doc 11): the hot mutable form of the frontier, log-structured so a crawl update is an append and the log is its own recovery journal (D11). It reuses the kv/hashlog design (Spec 2070): a resident sharded hash index over an append-only log, the log-as-journal collapse with no separate write-ahead log, group commit, the three-position durability dial, and larger-than-memory residency where the hot tail is in RAM and the cold bulk is on disk addressed by log offset.
meguri adds the frontier-specific layer: the URLRecord and HostRecord codec that fills the log frame value, the string arena the records' references index into, and the checkpoint that folds the live state into a .meguri file (doc 10) plus the durable log frontier (D15). Recovery loads that snapshot, rebuilds the index, and replays the log tail past the snapshot's frontier, per-partition and redo-only: the log is append-only, so there is nothing to roll back.
The single-file promise holds (D20): open a directory, get a partition. The store keeps one active log, one .meguri snapshot, and a two-slot superblock, all pure Go, CGO_ENABLED=0, no internal/ directories.
Index ¶
- Variables
- type Durability
- type Options
- type Store
- func (s *Store) Checkpoint() error
- func (s *Store) CheckpointFrom(part *format.Partition) error
- func (s *Store) CheckpointStreaming(maxPageRows int) error
- func (s *Store) Close() error
- func (s *Store) DeleteURL(key meguri.URLKey) error
- func (s *Store) GetHost(key uint64) (meguri.HostRecord, bool)
- func (s *Store) GetURL(key meguri.URLKey) (meguri.URLRecord, bool)
- func (s *Store) HostCount() int
- func (s *Store) Intern(str string) (uint64, error)
- func (s *Store) InternRobots(blob []byte) (uint64, error)
- func (s *Store) LSN() uint64
- func (s *Store) PutHost(rec *meguri.HostRecord) (uint64, error)
- func (s *Store) PutURL(rec *meguri.URLRecord) (uint64, error)
- func (s *Store) Resident() int
- func (s *Store) Robots(off uint64) []byte
- func (s *Store) Snapshot() *format.Partition
- func (s *Store) Str(off uint64) string
- func (s *Store) URLCount() int
Constants ¶
This section is empty.
Variables ¶
var ErrNoCheckpoint = errors.New("store: no valid checkpoint")
ErrNoCheckpoint marks a superblock with neither slot valid, the fresh-store case where there is nothing to recover from.
var ErrTornFrame = errors.New("store: torn log frame")
ErrTornFrame marks a frame whose checksum fails: the durable tail ends here and replay stops, the torn-write safety net (doc 11 section 5.2).
Functions ¶
This section is empty.
Types ¶
type Durability ¶
type Durability uint8
Durability is the three-position dial (doc 11 section 8), inherited from hashlog (Spec 2070 doc 04 section 5). Honesty per D19: at concurrency 1 under DurabilityFull one update is one device flush, the fsync floor every honest durable store sits on; the win is above conc-1, where group commit amortizes the flush across the concurrent crawl updates a partition is processing.
const ( // DurabilityNone never fsyncs: the speed ceiling and the in-memory benchmark // regime, for a scratch crawl where re-deriving the frontier is cheaper than // paying for durability. A power loss loses everything the OS had not written // back (Spec 2070 doc 04 section 5.2). DurabilityNone Durability = iota // DurabilityNormal fsyncs at checkpoint boundaries, not on every update: a // crash loses at most the updates since the last checkpoint, and a lost crawl // outcome costs one redundant recrawl, not lost data. This is the default for // a durable frontier. DurabilityNormal // DurabilityFull fsyncs before every update returns, via group commit so // concurrent updates share one flush. No acknowledged update is ever lost. DurabilityFull )
type Options ¶
type Options struct {
Durability Durability // None | Normal | Full; default Normal
ResidentBudget int // max resident URL records, 0 = unbounded
PartitionID uint32 // stamped into the .meguri snapshot
CreatedHours uint32 // build time, epoch-hours
// SpillArena turns on the Stage A spilled arena (spec 2072 doc 05): the
// canonical-URL string region lives in a disk-backed spill file read through a
// bounded LRU instead of a fully resident []byte, removing ~70 B/url (~7 GiB at
// 100M) from the held heap. ArenaBudget is the LRU's resident byte ceiling
// (B_arena); 0 with SpillArena on picks a default working-set budget.
SpillArena bool
ArenaBudget int64
// DiskIndex turns on the Stage B on-disk DRUM index (spec 2072 doc 04): the URL
// location index moves out of the resident shards map into a sorted on-disk
// repository, removing the ~80-90 B/url resident index term. MergeBatch is the
// number of buffered discoveries that triggers a merge into the repository; 0
// picks a default.
DiskIndex bool
MergeBatch int
}
Options configure a store at open.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the durable live frontier of one partition.
func Open ¶
Open opens or recovers a partition store rooted at dir. If dir holds a valid checkpoint, Open loads the .meguri snapshot, rebuilds the index, and replays the log tail past the snapshot's frontier (doc 11 section 5). Otherwise it starts a fresh, empty store. Either way the returned store is ready to serve reads and accept updates.
func (*Store) Checkpoint ¶
Checkpoint folds the live store into a .meguri snapshot plus the durable log frontier, then swaps it in (doc 11 section 4). The snapshot is the partition's frontier serialized into doc 10's columnar regions; it is at once the recovery point, the redistribution unit, and the cold archive (D1). After the swap the store writes to a fresh log, so the next recovery loads this snapshot and replays only the updates that follow it.
func (*Store) CheckpointFrom ¶
CheckpointFrom commits an externally built partition snapshot as the store's durable checkpoint, the path the top-level lifecycle uses to persist a frontier it recovered from this store and then advanced. It writes the snapshot, rotates the log, and commits the superblock exactly as Checkpoint does, but takes the partition from the caller (the resident frontier's serialized form) rather than the store's own index. The caller's frontier is the live truth for the rest of the session; the store's in-memory index is not re-synced here because nothing reads it again before the next process reopens from the committed snapshot.
func (*Store) CheckpointStreaming ¶ added in v0.2.0
CheckpointStreaming folds the live store into the durable .meguri snapshot the bounded-memory way (spec 2072 D9, 2071 implementation doc 51): instead of materializing the whole partition as a record slice and a column-major copy (snapshotPartition + EncodeToFile, ~0.9 KB per row of transient that OOMs a 64 GB box at 100M), it streams the URL table through a k-way merge over the 256 sorted shards into format.StreamEncodeToFile, which spills the columns page by page. The resident transient is the shard key copies (a 16 B/url copy of keys already resident in the index) plus one record and one page per column at a time, not the partition.
maxPageRows must be > 0 for the win: it caps the per-column page buffers, so the snapshot is multi-page (still a valid, byte-stable .meguri). The host table, string arena, and seen-set are small and stay materialized in the shell. The resulting file is identical to what Checkpoint would write for the same records at the same page cap; only the path to it is bounded.
func (*Store) DeleteURL ¶
DeleteURL tombstones a key, the rare removal of a URL from the partition (a host moves, doc 11 section 3.4). The everyday "this URL is dead" is a Gone status on a live row, not a delete.
func (*Store) GetHost ¶
func (s *Store) GetHost(key uint64) (meguri.HostRecord, bool)
GetHost returns the current record for a host key.
func (*Store) GetURL ¶
GetURL returns the current record for key, materializing it from disk if it was evicted. The bool is false if the key is absent or tombstoned.
func (*Store) Intern ¶
Intern appends s to the string arena and returns its byte offset, the reference a record's *Ref field carries (doc 11 section 3.3). The append is logged so recovery rebuilds the arena to identical offsets; offsets are stable for the life of the store, so a checkpoint hands the arena to the file with no remap, matching the frontier's no-remap invariant. The empty string returns 0, the none sentinel, without growing the arena.
func (*Store) InternRobots ¶
InternRobots packs a robots blob through the doc 10 section 7 robots modes and interns the packed bytes, returning the reference a HostRecord's RobotsRef carries. The packing picks the smallest of the allow-all sentinel, raw, or codec-compressed, so a host serving an allow-all policy never grows the arena: an empty blob is the allow-all case and returns 0, the none sentinel, which a nil-Rules reader already treats as allow-all (robots.Rules). A non-empty blob is stored once, deduped by the checkpoint's content dictionary when a partition is split or merged.
func (*Store) LSN ¶
LSN reports the next LSN the store will assign, the value a checkpoint records as its durable frontier.
func (*Store) PutHost ¶
func (s *Store) PutHost(rec *meguri.HostRecord) (uint64, error)
PutHost appends an updated host record and repoints the host index. A host update is a same-size bump in the common case (doc 11 section 3.4); the store logs it the same way regardless.
func (*Store) PutURL ¶
PutURL appends an updated URL record and repoints the index at it. This is the dominant write, the point-update-on-crawl of doc 11 section 1.1: one append, one index repoint, the old record left as garbage for a later compaction. It returns the record's LSN.
func (*Store) Resident ¶
Resident reports the current number of resident URL records, the figure the resident budget bounds.
func (*Store) Robots ¶
Robots reads back and unpacks the robots blob at off, the inverse of InternRobots. A zero or out-of-range reference, the allow-all sentinel, or a blob that does not unpack returns nil, which a nil-Rules reader treats as allow-all, so a stale or corrupt reference degrades to the permissive default rather than panicking.
func (*Store) Snapshot ¶
Snapshot streams the live store into a sorted format.Partition without writing anything, the read-only half of a checkpoint. The top-level partition lifecycle (engine.OpenPartition) calls it to recover a resident frontier from a store it just opened, then advances that frontier and folds it back through CheckpointFrom.
Source Files