Documentation
¶
Index ¶
- Constants
- func BuildSharded(outDir string, specs []ShardBuildSpec, pool int) (StoreManifest, []ShardBuildResult, error)
- func CompactSharded(inDir, outDir string, in StoreManifest, deltas []*Delta, ...) (StoreManifest, []ShardCompactResult, error)
- func RecrawlSharded(inDir, outDir string, in StoreManifest, mkOpts func(ShardRef) RecrawlOptions, ...) (StoreManifest, []ShardRecrawlResult, error)
- func WriteStoreManifest(dir string, man StoreManifest) error
- type BuildOptions
- type BuildResult
- type CompactOptions
- type CompactResult
- type Delta
- type DeltaEntry
- type Engine
- func (e *Engine) BaseProbes() uint64
- func (e *Engine) BitsPerURL() float64
- func (e *Engine) Close() error
- func (e *Engine) DueCursor(now uint32) (*format.DueCursor, error)
- func (e *Engine) GetURL(key m.URLKey) (m.URLRecord, bool, error)
- func (e *Engine) HostKeyRange() (lo, hi uint64)
- func (e *Engine) Reader() *format.Reader
- func (e *Engine) Seen(key m.URLKey) (bool, error)
- func (e *Engine) URLCount() int
- type Item
- type OpenOption
- type RecordItem
- type RecordSource
- type RecrawlOptions
- type RecrawlResult
- type ShardBuildResult
- type ShardBuildSpec
- type ShardCompactResult
- type ShardRecrawlResult
- type ShardRef
- type Source
- type Store
- type StoreManifest
Constants ¶
const StoreManifestName = "store.json"
StoreManifestName is the fixed filename of the shard map written beside the shard .meguri files.
Variables ¶
This section is empty.
Functions ¶
func BuildSharded ¶
func BuildSharded(outDir string, specs []ShardBuildSpec, pool int) (StoreManifest, []ShardBuildResult, error)
BuildSharded builds every shard's .meguri from its seed slice with a bounded worker pool, then writes the store manifest under outDir (Spec 2074 doc 07, the parallel build). At most pool shards build at once, so the box runs full but not oversubscribed and the resident scratch is bounded by pool per-shard builds, not by the shard count. Each shard is one worker for its whole build, which is the single-writer-per-shard rule: no two workers ever touch the same shard file.
A shard's output is outDir/shard-NNNNN.meguri. The manifest is written only if every shard built; a failed shard is returned in the per-shard results and aborts the manifest so a partial store is never published.
func CompactSharded ¶
func CompactSharded(inDir, outDir string, in StoreManifest, deltas []*Delta, mkOpts func(ShardRef) CompactOptions, pool int) (StoreManifest, []ShardCompactResult, error)
CompactSharded folds each shard's delta into its base and writes the store's next generation, K shards at a time (Spec 2074 doc 07). deltas is indexed by manifest position; a shard whose delta is nil or empty is not re-encoded, its file is copied through and its generation kept, which is the design's "a cold range never compacts" property. A shard with a delta runs one bounded live.Compact into outDir, so the fold is the same per-shard bounded encode the recrawl proved, and pool is the K knob.
The output manifest keeps the input ranges; a folded shard refreshes its counts and file bytes and bumps its generation, a carried shard keeps its ref verbatim. It is written only if every shard succeeded, so a partial generation is never published.
func RecrawlSharded ¶
func RecrawlSharded(inDir, outDir string, in StoreManifest, mkOpts func(ShardRef) RecrawlOptions, pool int) (StoreManifest, []ShardRecrawlResult, error)
RecrawlSharded folds a crawl outcome into every due row of every shard and writes the store's next generation, K shards at a time (Spec 2074 doc 07, the parallel write and the direct OOM fix). Each shard's fold is one bounded live.Recrawl into outDir with the same filename, so the whole-keyspace encode that overran the box is replaced by K bounded per-shard encodes and K (the pool) is the backpressure knob the monolith lacked. mkOpts fills the recrawl knobs for a shard; the driver sets OutPath and TmpDir. A shard is owned by exactly one worker for its whole fold, the single-writer rule.
The output manifest carries the input ranges unchanged (a fold introduces no keys and moves no host across a boundary) with each shard's FileBytes and counts refreshed and Generation bumped. It is written only if every shard folded, so a partial generation is never published.
func WriteStoreManifest ¶
func WriteStoreManifest(dir string, man StoreManifest) error
WriteStoreManifest writes m to StoreManifestName under dir.
Types ¶
type BuildOptions ¶
type BuildOptions struct {
Path string // output .meguri file
TmpDir string // scratch for sort runs, the arena, and the records temp
ExpectedKeys uint64 // sizes the resident filter; the corpus row count is exact
RunRows int // sort buffer cap in rows (0 = 1<<20)
PageRows int // encoder page row cap (0 = single page, not for scale)
Codec uint8 // format.CodecZstd for the compact file
FPRate float64 // filter false-positive budget (0 = 1%)
NowHours uint32 // epoch-hours stamped as FirstSeen and NextDue
PartitionID uint32
Priority float32 // default URL priority (0 -> 0.5)
Status m.URLStatus // default URL status
Source m.DiscoverySource
CrawlDelay uint16 // host crawl delay, deciseconds (0 -> 100)
}
BuildOptions configures a bulk build.
type BuildResult ¶
BuildResult reports what a bulk build produced.
func BulkLoad ¶
func BulkLoad(src Source, opts BuildOptions) (BuildResult, error)
BulkLoad builds one compact .meguri file from a discovery source in bounded memory (spec 2073 doc 08, the friendly bulk case). It external-sorts the items by URLKey, builds the resident seen-set filter and the host table, writes the string arena host-clustered to a temp file, and runs the streaming columnar encode from a key-ordered records temp, so the durable output is a single file and the transient is one sort buffer plus one encoder page per column. No DRUM, no append log, no permanent arena: the temp files are removed on the way out.
func ImportRecords ¶
func ImportRecords(src RecordSource, opts BuildOptions) (BuildResult, error)
ImportRecords builds a compact .meguri file from a stream of full records, the inverse of a Parquet export (spec 2073 doc 08, the file is the store). It mirrors BulkLoad's bounded-memory shape but carries the whole record rather than stamping a discovery: the strings intern host-clustered into the arena, the resident seen-set filter is solved over the keys, and the streaming columnar encoder writes the single output file. The transient is one record at a time plus the host map, never the whole table.
The input must be in ascending URLKey order (a meguri export is); a key that moves backward is an error rather than a silently misbuilt file, because the encoder and the seen-set both rely on the order. Two disk scratch files carry the rows across the two passes the arena's hosts-before-urls layout needs; both are removed on the way out.
type CompactOptions ¶
type CompactOptions struct {
OutPath string // the new .meguri file; written to a temp then atomically renamed
TmpDir string // scratch for the new arena and the records temp
PageRows int // encoder page row cap (match the base for a stable layout)
Codec uint8 // format.CodecZstd
FPRate float64 // filter FP budget when the base carries no filter to reuse
NowHours uint32 // stamped on delta inserts that leave FirstSeen/NextDue unset
PartitionID uint32
CrawlDelay uint16 // crawl delay for hosts the delta introduces (0 -> 100)
}
CompactOptions configures a Stage 2 compaction.
type CompactResult ¶
type CompactResult struct {
URLCount int
HostCount int
Inserted int // delta keys not present in the base
Updated int // delta keys that replaced a base row
Carried int // base rows copied through unchanged
FileBytes int64
BitsPerURL float64
}
CompactResult reports what a compaction produced.
func Compact ¶
func Compact(basePath string, delta *Delta, opts CompactOptions) (CompactResult, error)
Compact folds a resident Delta into the base .meguri file and writes the next file generation, the write path of spec 2073 doc 08 Stage 2. It merge-joins the base URL table (read in URLKey order through a row cursor) with the sorted delta: a delta key that matches a base row replaces it (a recrawl update), a delta key with no match is inserted, and every untouched base row is carried through. The base's URL and host strings are re-interned into a fresh arena, so the output is one self-contained file with no reference back to the base.
The read side is sequential, not random: the cursor walks base rows in key order and BulkLoad interned strings in that order, so an ArenaSeqReader resolves each carried row's string with about one blob page resident, never the whole arena. This is why compaction is the cursor-based merge the doc argues for over the random point-lookup tail the 100M rediscovery sample measured. The transient is the host table, the resident filter, one blob-page window, the sorted delta, plus one encoder page per column; the durable output is a single file and the temps are removed on the way out.
type Delta ¶
type Delta struct {
// contains filtered or unexported fields
}
Delta is the bounded resident write buffer of spec 2073 doc 08 Stage 2: discoveries and recrawl updates accumulate here, and a compaction folds them into the base file to produce the next file generation. It is sorted by URLKey only at flush, so Put is O(1) and the sort is paid once per compaction. The residency budget is the caller's: Bytes reports the approximate resident cost so the caller compacts before the delta outgrows the box, the same way the base filter is sized ahead.
func (*Delta) Put ¶
func (d *Delta) Put(e DeltaEntry)
Put appends a pending write. A later Put for the same key supersedes an earlier one; the flush keeps the last write per key, so an update after an insert in the same batch folds to a single row.
type DeltaEntry ¶
type DeltaEntry struct {
Rec m.URLRecord // full post-write record; URLRef is assigned by Compact
URL string // canonical URL string, re-interned into the new arena
Host string // host grouping string, interned if the host is new
}
DeltaEntry is one pending write against the live store: a discovery (a key not in the base) or a recrawl update (a key already in the base, re-fetched with fresh crawl state). Rec is the full record to store, so the caller owns the field merge (a recrawler reads the base row, updates NextDue and the crawl counters, and hands the whole post-update record here); Compact re-interns URL and Host into the new file's arena and overwrites Rec.URLRef, so the caller leaves those unset. The string fields travel with the entry because the base arena is not addressable by the new file.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is the file-backed live store of spec 2073 doc 08: the mapped .meguri file is the durable state, and the engine reads dedup, lookup, and schedule straight off it. The base is the mapped file (read-only, reclaimable page cache), the filter is the resident one-sided membership filter (the ribbon snapshot the seal writes, or the blocked Bloom an older file carries) that answers "is this new" without touching the file, and the file's sorted keys are the exact set a hit is confirmed against. There is no DRUM, no append log, and no spilled arena; the only resident per-URL cost is the filter.
This is the read and dedup half. The write half (a bounded delta plus compaction) layers on top and is what turns a discovery into a new file generation.
func Open ¶
func Open(path string, opts ...OpenOption) (*Engine, error)
Open maps the .meguri file and loads the resident filter from its seen-set region. The file is mapped, not read, so a multi-gigabyte base costs reclaimable page cache and not heap (the goal-box property). If the file has no seen-set region the filter is left nil and dedup falls through to the file for every key, which is correct but slower; a file BulkLoad wrote always carries the region.
func (*Engine) BaseProbes ¶
BaseProbes is the number of dedup decisions that fell through the resident filter to the mapped base, the slow-path count. On a fresh discovery stream it is the filter's false positives; on a recrawl it is the rediscoveries. It is the honest split between the resident fast path and the file-touching slow path, the metric the residency claim of doc 08 turns on.
func (*Engine) BitsPerURL ¶
BitsPerURL reports the resident filter cost per URL, the residency budget term.
func (*Engine) DueCursor ¶
DueCursor opens a bounded due-dispatch scan over the base file for the rows due at or before now, the scheduler read of doc 08 Stage 3. It streams due keys off the mapped file in caller-capped batches rather than materializing the whole next_due column, so a scheduler pulls the next work at 100M without a multi-gigabyte transient. A now the file's footer proves nothing is due for yields an empty cursor without decoding a column.
func (*Engine) GetURL ¶
GetURL returns the record for key and whether it was found. It takes the same filter-then-base path as Seen, so an absent key that the filter rules out costs no file access.
func (*Engine) HostKeyRange ¶
HostKeyRange is the partition's host-key span, the redistribution and routing key range.
func (*Engine) Reader ¶
Reader exposes the underlying file reader for the schedule and recrawl read paths (DueByWheel, Schedule, URLRows), which read the file directly.
func (*Engine) Seen ¶
Seen reports whether key is already in the store, the dedup decision. A filter miss is an authoritative "new", returned without faulting a single file page, which is the common case on a discovery stream and why intake does not thrash the map. A filter hit is confirmed against the mapped base by default: a true positive is a rediscovery, a false positive decodes one page and finds nothing. Under WithTrustFilter the confirm is skipped and a filter hit returns seen directly, so the pass never faults a file page for a hit and accepts the filter's one-sided false positives as its drop rate (M3, doc 03).
type Item ¶
type Item struct {
Key m.URLKey
URL string
Host string // host grouping string, interned once per distinct host
}
Item is one URL on its way into a bulk build: its key, its canonical string, and its host grouping. The caller owns canonicalization and keying (the scale harness and the engine both have it), so the loader stays a pure sort-and-encode.
type OpenOption ¶
type OpenOption func(*Engine)
OpenOption configures an Engine at Open time.
func WithTrustFilter ¶
func WithTrustFilter() OpenOption
WithTrustFilter makes Seen treat a filter hit as an authoritative "seen" and skip the base-confirm decode. This is the M3 dedup default (doc 03): the confirm exists only to recover the filter's one-sided false positives, and at the ribbon's measured rate that decode is more than half the pass paid to rescue at most one in ten thousand keys. A trusted run trades that: a false positive drops one genuinely-new URL as a duplicate, at the filter's ~2^-r rate, and never faults a file page for a hit. Correct only where the caller accepts the filter's false-positive rate as its drop rate; the scale dedup pass does, exact callers (GetURL, the write path) do not use it. With no filter present the flag is inert and Seen still falls through to the base.
type RecordItem ¶
RecordItem is one fully-formed URL row on its way into a rebuild: the complete record plus the strings its refs resolve to. It is what an import from Parquet yields, in contrast to bulkload's Item, which carries only a discovery (key, url, host) and lets the build stamp the state. Here every field is already set, so the rebuild is lossless.
type RecordSource ¶
type RecordSource interface {
Next() (RecordItem, bool, error)
}
RecordSource yields RecordItems in ascending URLKey order. ImportRecords does not sort, so the caller must present the rows key-ordered; a meguri Parquet export always is, because it was written in key order.
type RecrawlOptions ¶
type RecrawlOptions struct {
OutPath string // the new .meguri file; written to a temp then atomically renamed
TmpDir string // scratch for the new arena and the records temp
PageRows int // encoder page row cap (match the base for a stable layout)
Codec uint8 // format.CodecZstd
NowHours uint32 // a row with 0 < NextDue <= NowHours is due and gets an outcome folded
PartitionID uint32 // partition id stamped on the new file
FPRate float64 // filter FP budget when the base carries no filter to reuse
CrawlDelay uint16 // crawl delay for hosts the base carries with none (0 -> 100)
Params freshness.Params // recrawler math parameters (freshness.DefaultParams when zero)
Tau float64 // the water level the allocation reschedules against
ChangeRate float64 // probability an outcome is a real content change, the rest are 304 no-change
Seed uint64 // deterministic outcome draw seed
}
RecrawlOptions configures a Stage 3 recrawl fold.
type RecrawlResult ¶
type RecrawlResult struct {
URLCount int
HostCount int
Recrawled int // due rows that had an outcome folded and were rescheduled
Carried int // rows not due, copied through unchanged
Changed int // outcomes classified as a real content change
NoChange int // outcomes that were a 304 or a cosmetic no-change
FileBytes int64 // the new generation's size on disk
MeanLambda float64 // mean estimated change rate over the recrawled rows, a convergence sanity
BitsPerURL float64
}
RecrawlResult reports what a recrawl fold produced.
func Recrawl ¶
func Recrawl(basePath string, opts RecrawlOptions) (RecrawlResult, error)
Recrawl folds a crawl outcome into every due row of the base .meguri file and writes the next file generation, the "update a URL after it is fetched" need of spec 2073 doc 08. It streams the base URL table in key order through a row cursor, and for a row whose NextDue is at or before now it draws a typed outcome, advances the row's change-rate counters exactly as the frontier's markCrawled does, re-estimates the URL's Poisson change rate, and sets the next due time from the water-filling allocation. A row that is not due is carried through unchanged. Every row's URL string is re-interned into a fresh arena, so the output is one self-contained generation swapped in with an atomic rename.
The read is sequential, the same cursor walk compaction uses: the dispatch order the scheduler emits is the file's stored key order, so folding outcomes back in reads the base once front to back with about one blob page resident, never the random point-lookup tail a per-key GetURL would pay. The outcomes here are typed feedback values drawn to a change rate, not live fetches, so this measures the recrawler fold and its residency at scale, exactly the honest framing doc 05 sets for the recrawl benchmark.
type ShardBuildResult ¶
type ShardBuildResult struct {
Index int
Result BuildResult
Err error
}
ShardBuildResult pairs a shard's build result with its index and any error, so the driver can report per-shard outcomes without ordering assumptions.
type ShardBuildSpec ¶
type ShardBuildSpec struct {
Index int
HostLo uint64
HostHi uint64
NewSource func() (Source, error)
Opts BuildOptions
}
ShardBuildSpec describes one shard to build: its index and hostkey range, a factory that opens a fresh Source over that shard's seed slice, and the BulkLoad options. The driver sets Opts.Path, so a caller fills only the build knobs (codec, filter budget, expected keys, stamps). NewSource is a factory, not a Source, so the driver opens the seed only when a worker picks the shard up, keeping at most pool sources live at once.
type ShardCompactResult ¶
type ShardCompactResult struct {
Index int
Result CompactResult
Skipped bool // true when the shard had no delta and its file was copied unchanged
Err error
}
ShardCompactResult pairs a shard's compaction result with its manifest index, whether the shard was folded or carried cold, and any error.
type ShardRecrawlResult ¶
type ShardRecrawlResult struct {
Index int
Result RecrawlResult
Err error
}
ShardRecrawlResult pairs a shard's recrawl fold result with its manifest index and any error, so the driver reports per-shard outcomes without ordering assumptions.
type ShardRef ¶
type ShardRef struct {
Index int `json:"index"`
Path string `json:"path"` // .meguri filename relative to the store dir
HostLo uint64 `json:"host_lo"`
HostHi uint64 `json:"host_hi"`
URLCount int `json:"url_count"`
HostCount int `json:"host_count"`
FileBytes int64 `json:"file_bytes"`
Generation int `json:"generation"`
}
ShardRef is one shard's entry in the store manifest: where its .meguri lives, the hostkey range it owns, and the summary stats a build recorded. Generation bumps each time the shard is rewritten (compaction, recrawl fold) so the manifest names the current file and an interrupted swap is detectable.
type Source ¶
Source yields discovered items for a bulk build. Next returns ok false at the end. It is pulled once, streaming, so the corpus is never resident.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an opened sharded live store: the manifest plus one live Engine per shard, mapped read-only. It routes a key to its shard's engine for dedup, lookup, and schedule, and fans a whole-store read out over all shards. Each shard's mmap is reclaimable page cache, so the resident cost is the sum of the per-shard filters, not the shard bodies.
func OpenStore ¶
OpenStore reads the manifest under dir and opens every shard engine. If a shard fails to open, the already-opened shards are closed and the error returned, so a partial store never leaks maps.
func (*Store) BaseProbes ¶
BaseProbes sums the slow-path (base-touching) dedup decisions across all shards, the store-wide analogue of Engine.BaseProbes.
func (*Store) Manifest ¶
func (s *Store) Manifest() StoreManifest
Manifest returns the store's shard map.
type StoreManifest ¶
StoreManifest is the shard map of a sharded live store (Spec 2074 doc 07): the ordered list of hostkey-range shards that make up the store, each a self-contained .meguri. It is small and read on open, so a driver learns the whole shard set and each shard's range and stats without touching a shard body. The ranges tile the uint64 hostkey space with no gap and no overlap; shard k owns [HostLo, HostHi) and the last shard's HostHi is the max. A store of one shard is the single-file engine's N=1 special case.
func ReadStoreManifest ¶
func ReadStoreManifest(dir string) (StoreManifest, error)
ReadStoreManifest reads the shard map from dir.
func (StoreManifest) Route ¶
func (man StoreManifest) Route(hostKey uint64) int
Route returns the index of the shard that owns hostKey, a binary search over the shard ranges. It assumes the shards tile the space in ascending order, which the build guarantees.
Source Files