dedup

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package dedup is the seen-set and the content-dedup machinery: the layer that keeps the same URL discovered a thousand times to one frontier entry, and the same content served under a thousand URLs to one crawl (doc 08, D5, D7, D17).

The seen-set is two tiers. A resident approximate filter (a one-sided blocked Bloom filter, around eleven bits per URL at the 1% default) answers "definitely not seen" authoritatively; a "probably seen" hit is confirmed against an exact on-disk key set consulted through DRUM batching, so a false positive costs a batched sequential confirm, never a dropped page, and a false negative never happens. SeenSet wraps the two and exposes the idempotent check-and-insert that makes discovery tolerate at-least-once delivery.

Canonicalization (the eleven steps of doc 03), the registrable-domain grouping over the embedded Public Suffix List, and the 128-bit URLKey derivation feed the seen-set: a URL is canonicalized once at discovery, before it is keyed.

Content dedup is a 64-bit exact content fingerprint and a 64-bit Charikar simhash with Hamming threshold 3, computed at fetch time, used to tell a real edit from a rotating ad and to collapse the same content under different URLs. The trap defense is the per-host budget and depth cap, soft-404 detection by content fingerprint, and near-dup recovery of the budget a pure cap would waste.

This is the M2 milestone.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Admit 

func Admit(depth uint16, h *meguri.HostRecord, passesHeuristics bool) meguri.URLStatus

Admit is the blunt always-correct trap defense of doc 08, section 8.2 (D17): per-host url_budget and depth_cap decide whether a new discovery enters the frontier or is parked in Trapped. It does not need to understand why a host is exploding, only that it is. A zero budget or zero depth cap means unlimited (doc 09 sets the real numbers); a flagged host's discoveries must additionally pass the pattern heuristics.

Trapped is a parked state, not a dropped URL: the caller keeps the row, so the URL is in the seen-set and dedups rediscoveries, and it can be released back to Scheduled if the host's budget is raised or the flag is cleared.

func BuildRibbonFilter 

func BuildRibbonFilter(keys []meguri.URLKey, opts ...RibbonOption) ([]byte, error)

BuildRibbonFilter builds the cold ribbon form over a key set and returns its serialized kind-1 blob, the static snapshot a frozen partition carries instead of the mutable blocked-Bloom filter. UnmarshalFilter reads it back into a ResidentFilter that answers MaybeContains behind the same one-sided contract.

func BuildShardedRibbonFilter added in v0.2.0 

func BuildShardedRibbonFilter(shardKeys [][]meguri.URLKey, opts ...RibbonOption) ([]byte, error)

BuildShardedRibbonFilter builds the cold ribbon form over keys already partitioned into shards (by RibbonShardIndex with count len(shardKeys)) and returns the serialized blob a frozen partition carries. A single shard emits the kind-1 blob BuildRibbonFilter would, so small seals stay byte-for-byte compatible with the single form; more than one shard emits the kind-2 sharded blob. UnmarshalFilter reads either back into a ResidentFilter behind the same one-sided contract.

func BuildShardedRibbonFilterDisk added in v0.2.0 

func BuildShardedRibbonFilterDisk(shardCount int, n uint64, load func(i int) ([]meguri.URLKey, error), opts ...RibbonOption) ([]byte, error)

BuildShardedRibbonFilterDisk builds the kind-2 blob solving each shard from keys the loader streams back, instead of from a slice already resident. It is the large-seal path: the seal spills each shard's keys to a temp file at collection and passes a load that reads shard i back, so the solve holds at most a few shards' keys in memory at once rather than the whole key set. shardCount must be greater than one (the seal uses the single kind-1 ribbon below the shard threshold); n is the total distinct key count the blob records for the query path. load(i) returns shard i's keys and need not retain them after it returns.

func CanonicalKey 

func CanonicalKey(raw, base string, g meguri.HostGrouping, pol *CanonPolicy) (meguri.URLKey, string, string, bool)

CanonicalKey is the discovery-path convenience: canonicalize a raw link and derive its URLKey in one call. ok is false when the link is not a frontier URL.

func Canonicalize 

func Canonicalize(raw, base string, pol *CanonPolicy) (string, bool)

Canonicalize runs the eleven-step canonicalization of doc 03 over a raw link found on the page at base (base may be empty when raw is already absolute). It returns the canonical URL string and ok=true, or ok=false when the link is not a frontier URL (a non-http(s) scheme, an unparseable reference, or an invalid host). The function is pure: same input, same output, no clock, no network.

The asymmetry of doc 03 is the rule throughout: a missed merge wastes a little space by carrying two records for one page, a wrong merge silently drops a real page, so every borderline rule errs toward keeping URLs distinct. Trailing slash, path case, index files, duplicate slashes, and encoded delimiters are all kept by default and folded only when the host's policy says so.

func ContentFP 

func ContentFP(body []byte) uint64

ContentFP is the 64-bit exact content fingerprint: xxHash64 over the body (doc 08, section 7.1). Any byte change flips it, so it is the all-or-nothing change signal, where the simhash is the graded one. The caller normalizes the body (the fetcher computes both once at extraction); NormalizeBody is the default normalization.

func FlagTrapSuspect 

func FlagTrapSuspect(h *meguri.HostRecord)

FlagTrapSuspect sets the trap-suspect bit on a host, the precise defense that tightens admission (doc 08, section 8.4). It is sticky and serialized with the HostRecord, so it survives a checkpoint and reload.

func HostGroupKey 

func HostGroupKey(host string, g meguri.HostGrouping) string

HostGroupKey returns the bytes the HostKey hashes for a canonical host, per the grouping mode (doc 03's hostGroupKey).

func IsTrapSuspect 

func IsTrapSuspect(h *meguri.HostRecord) bool

IsTrapSuspect reports whether the host is flagged.

func Key 

func Key(canonURL string, g meguri.HostGrouping) (meguri.URLKey, string, bool)

Key derives the 128-bit URLKey of a canonical URL under a grouping mode, and returns the host group key alongside it. The HostKey hashes the group key, the PathKey hashes the canonical-URL remainder "scheme://full_host[:port]path[?query]" (doc 03's PathKey byte range). ok is false only when the canonical URL has no host.

func Near 

func Near(a, b uint64) bool

Near reports whether two simhashes are within the near-dup threshold, a popcount of their XOR.

func NormalizeBody 

func NormalizeBody(body []byte) []byte

NormalizeBody collapses runs of whitespace to a single space and trims the ends, so trivial reflowing does not move the exact fingerprint. It is the minimal normalization the fingerprint runs over; richer boilerplate stripping is the fetcher's job before it fills the Outcome.

func RegistrableDomain 

func RegistrableDomain(host string) string

RegistrableDomain returns the registrable domain (public-suffix-plus-one) of a host, the default host group key (doc 03). It is computed by the embedded Public Suffix List: the longest matching public suffix plus the one label to its left. A host that is itself a public suffix, or has no label above the suffix, returns the host unchanged.

func RibbonBitsForFPR added in v0.2.0 

func RibbonBitsForFPR(fp float64) int

RibbonBitsForFPR maps a target false-positive rate to the ribbon fingerprint width that meets it. A ribbon's realized FPR is exactly 2^-r for the r-bit fingerprint compare, independent of the key distribution, so r is the smallest width with 2^-r <= fp, that is ceil(-log2(fp)). It is clamped to the 1..16 a uint16 slot holds. The region's bits per key is then r/load, so a 1e-4 target maps to r=14 and about 15.6 bits per key, a third under the blocked-Bloom's 22.

func RibbonShardCount added in v0.2.0 

func RibbonShardCount(n int) int

RibbonShardCount picks the power-of-two shard count for a seal of n keys: one shard when n is at or below the target so small seals stay a single ribbon, otherwise the smallest power of two that keeps each shard near the target, capped at maxRibbonShards. A seal path shards its keys at collection with this count and the blob records it, so the query path routes with the same count.

func RibbonShardIndex added in v0.2.0 

func RibbonShardIndex(key meguri.URLKey, shards int) int

RibbonShardIndex routes a key to one of shards (which must be a power of two) by the high log2(shards) bits of its shard hash, so the split is even and the index is a single shift. It is the one place the routing lives; a seal and its query must call it with the same shard count.

func Simhash 

func Simhash(features []WeightedFeature) uint64

Simhash computes the 64-bit Charikar simhash of a document's weighted features (doc 08, section 7.2). Each feature is hashed to 64 bits; a 64-entry signed accumulator adds the weight where the feature-hash bit is 1 and subtracts it where the bit is 0; the simhash bit i is 1 iff accumulator i is positive. The Hamming distance between two simhashes tracks the dissimilarity of the two documents, which is the graded change signal the freshness model wants.

func SimhashText 

func SimhashText(text string) uint64

SimhashText is the convenience the fetcher uses: tokenize a body into terms, weight each by its frequency, and simhash the result. Tokens are maximal runs of letters and digits, lowercased, so cosmetic markup and case do not move the signature. An empty body simhashes to zero, which the change classifier reads as no near-dup signal.

Types

type CanonPolicy 

type CanonPolicy struct {
	QueryAllow    []string // if non-empty, only these query keys survive (allowlist mode)
	QueryDeny     []string // extra deny-list keys beyond the global defaults
	FoldTrailing  bool     // collapse /p and /p/ to /p
	FoldIndex     bool     // collapse /dir/ and /dir/index.html
	FoldCase      bool     // lowercase the path and query (case-insensitive host)
	CollapseSlash bool     // collapse duplicate path slashes
	Version       uint16   // policy version, recorded with the partition
}

CanonPolicy is the per-host canonicalization configuration (doc 03). It is versioned and pinned for a campaign: a change to it changes identity for the host, so it is a campaign-boundary change, never a mid-flight mutation. A nil policy means the global defaults: the tracking deny-list, no folding, case preserved.

type ChangeKind 

type ChangeKind uint8

ChangeKind is the verdict comparing a fresh fetch against a URL's stored signals (doc 08, section 7.4). 

const (
	// NoChange: the body is byte-identical (content_fp equal). The estimator
	// counts a no-change observation; nochange_streak increments.
	NoChange ChangeKind = iota
	// CosmeticChange: the body differs but the simhash is within Hamming 3, a
	// rotating ad or a live timestamp. It is a no-change for the rate estimator,
	// so it does not poison lambda (doc 08, section 7.5).
	CosmeticChange
	// RealChange: the body differs and the simhash moved more than Hamming 3, a
	// meaningful edit. change_count increments, last_changed is set.
	RealChange
)

func ClassifyChange 

func ClassifyChange(oldFP, newFP, oldSimhash, newSimhash uint64) ChangeKind

ClassifyChange compares a fresh fetch's content fingerprint and simhash against the URL's stored ones and returns the change verdict (doc 08, section 7.4). This is the join with the freshness model (doc 06): it defines what "saw a change" means for the change-rate estimator, so a cosmetic-only change is a no-change observation and lambda reflects the rate the meaningful content moves.

type Classification 

type Classification struct {
	Key    meguri.URLKey
	Unique bool
}

Classification is the verdict the DRUM merge returns for one checked key: a rediscovery (Unique false, the key was already present) or a genuinely new URL (Unique true, the key was folded into the bucket).

type Filter added in v0.2.0 

type Filter struct {
	// contains filtered or unexported fields
}

Filter is the resident approximate seen-set tier on its own, the doc 04 blocked Bloom without the exact set behind it. The file-backed engine (spec 2073 doc 08) uses it as the negative-fast dedup tier in front of the mapped .meguri file: a miss is an authoritative "this URL is new" that never touches the file, and the file's sorted keys are the exact set a positive is confirmed against. SeenSet pairs the same filter with a resident exact set, which is O(distinct URLs) and is the term doc 08 moves onto disk; Filter is the filter alone, for a caller whose exact set is the mapped file.

func LoadFilter added in v0.2.0 

func LoadFilter(b []byte) (*Filter, error)

LoadFilter restores a Filter from Marshal bytes, the recovery path that reads the seen-set region of a mapped file straight into the resident tier.

func NewFilter added in v0.2.0 

func NewFilter(capacity uint64, fpRate float64) *Filter

NewFilter builds an empty resident filter sized for capacity keys at the given false-positive rate. The rate is a throughput knob, not a correctness one: a higher rate means more confirmations against the file, never a dropped key, because the filter is one-sided and has no false negatives.

func (*Filter) Add added in v0.2.0 

func (r *Filter) Add(key meguri.URLKey)

Add records a key as seen.

func (*Filter) BitsPerURL added in v0.2.0 

func (r *Filter) BitsPerURL() float64

BitsPerURL reports the resident filter cost per added key, the budget the residency gate checks.

func (*Filter) Len added in v0.2.0 

func (r *Filter) Len() uint64

Len reports the number of keys added.

func (*Filter) Marshal added in v0.2.0 

func (r *Filter) Marshal() []byte

Marshal serializes the filter for the .meguri seen-set region, so a reload restores it without re-adding every key.

func (*Filter) MaybeContains added in v0.2.0 

func (r *Filter) MaybeContains(key meguri.URLKey) bool

MaybeContains reports the filter's verdict. A false is authoritative: the key was never added, so it is new and no file lookup is needed. A true is the filter's "probably", which on an unseen key is a false positive the caller confirms against the exact set on disk.

type NearDup 

type NearDup struct {
	// contains filtered or unexported fields
}

NearDup is the near-duplicate index of doc 08, section 7.3: it answers "is there an existing page whose simhash is within Hamming 3 of this one" over a growing repository, the mirror-collapse and soft-404 query. It is Manku, Jain, and Das Sarma's permuted-table approach: nearBlocks sorted tables, each rotated so a different 16-bit block leads, queried by an exact-match probe on the leading block followed by a Hamming check on the survivors. This is exact recall: a true near-dup is never missed, by the pigeonhole argument above.

func NewNearDup 

func NewNearDup() *NearDup

NewNearDup returns an empty near-dup index.

func (*NearDup) Add 

func (n *NearDup) Add(sim uint64, key meguri.URLKey)

Add stores a simhash and its key. The tables are marked dirty so the next Find re-sorts them; batching inserts keeps the amortized cost low.

func (*NearDup) Find 

func (n *NearDup) Find(sim uint64) (meguri.URLKey, bool)

Find returns the key of a stored page whose simhash is within Hamming 3 of sim, and ok=true, or ok=false when none is near. When several are near, it returns the first found, which is enough for the collapse decision.

func (*NearDup) Len 

func (n *NearDup) Len() int

Len reports the number of distinct simhashes stored.

type PatternKind 

type PatternKind uint8

PatternKind names the trap signature a URL bears, the structural tell that a host is generating frontier entries faster than it serves distinct content (doc 08, section 8.4). PatternNone is a URL with no trap signature. 

const (
	PatternNone     PatternKind = iota
	PatternCalendar             // an advancing date past the crawl horizon, the infinite-calendar walk
	PatternFaceted              // stacked filter parameters, the combinatorial facet explosion
	PatternSession              // a high-entropy session id, a fresh identity per visit for one page
)

func (PatternKind) String 

func (k PatternKind) String() string

String renders a PatternKind for logs and test failures.

type ResidentFilter 

type ResidentFilter struct {
	// contains filtered or unexported fields
}

ResidentFilter is a reconstructed read-only seen-set filter: it answers the one-sided membership probe a discovery path uses to short-circuit "definitely not seen", and reports its resident cost, but it does not insert. A recovery loads it from the .meguri seen-set region and pairs it with the exact set rebuilt from the urlkey column. The blocked-Bloom and ribbon forms ride behind the same membership contract.

func UnmarshalFilter 

func UnmarshalFilter(b []byte) (*ResidentFilter, error)

UnmarshalFilter reconstructs a resident filter from a filter blob, dispatching on the kind byte: the blocked-Bloom form MarshalFilter wrote, or the ribbon snapshot BuildRibbonFilter wrote. The reconstructed filter answers MaybeContains identically to the original for every key, the property the round-trip gate asserts.

func (*ResidentFilter) BitsPerURL 

func (r *ResidentFilter) BitsPerURL() float64

BitsPerURL reports the reconstructed filter's resident cost per held key.

func (*ResidentFilter) Len 

func (r *ResidentFilter) Len() uint64

Len reports the number of keys the filter was built over.

func (*ResidentFilter) MaybeContains 

func (r *ResidentFilter) MaybeContains(key meguri.URLKey) bool

MaybeContains is the one-sided probe: false is authoritative (the key was never added), true is the filter's "probably", confirmed against the exact set by the caller.

type RibbonOption 

type RibbonOption func(*ribbonConfig)

RibbonOption configures a ribbon snapshot build.

func WithRibbonBits 

func WithRibbonBits(bits int) RibbonOption

WithRibbonBits sets the fingerprint width in bits, the bits-per-url versus false-positive-rate knob (a false-positive rate of 2^-bits). It is clamped to the 1..16 a uint16 slot holds.

type SeenOption 

type SeenOption func(*seenConfig)

SeenOption configures a SeenSet.

func WithBuckets 

func WithBuckets(n int) SeenOption

WithBuckets sets the number of DRUM buckets the exact set shards into.

func WithCapacity 

func WithCapacity(n uint64) SeenOption

WithCapacity sizes the resident filter for an expected key count.

func WithFPRate 

func WithFPRate(p float64) SeenOption

WithFPRate sets the filter false-positive budget (the confirm-rate knob).

type SeenSet 

type SeenSet struct {
	// contains filtered or unexported fields
}

SeenSet is the two-tier dedup authority of doc 08 (D5): a resident approximate filter in front of an exact key set, so the same URL discovered a thousand times becomes one frontier entry. The filter answers "definitely not seen" authoritatively and "probably seen" provisionally; a provisional hit is confirmed against the exact set, so a false positive costs a confirm, not a dropped page, and a false negative never happens because the filter is one-sided. This is what makes discovery idempotent (doc 08, section 9.3): a key delivered twice creates at most one record.

func NewSeenSet 

func NewSeenSet(opts ...SeenOption) *SeenSet

NewSeenSet builds an empty seen-set.

func (*SeenSet) BitsPerURL 

func (s *SeenSet) BitsPerURL() float64

BitsPerURL reports the resident filter cost per held key, the budget the gate checks against the bits-per-URL table (doc 08, section 3.2).

func (*SeenSet) Contains 

func (s *SeenSet) Contains(key meguri.URLKey) bool

Contains reports membership without inserting, going straight to the exact authority through the filter. A filter miss is an authoritative no without touching the exact set.

func (*SeenSet) Insert 

func (s *SeenSet) Insert(key meguri.URLKey)

Insert adds a key known to be new (a recovery rebuild from a key column), folding it into both tiers without classification.

func (*SeenSet) InsertBatch added in v0.2.0 

func (s *SeenSet) InsertBatch(keys []meguri.URLKey)

InsertBatch folds a batch of keys into both tiers in one DRUM pass per bucket, the insert-only scale form of Insert: Merge is to Seen what this is to Insert. It routes the keys to buckets by HostKey prefix and merges each bucket's run against its sorted run once, so n inserts cost one sorted merge per bucket rather than n shifts into the middle of a sorted slice (the O(n^2) the single- key add pays at scale, doc 08 section 4.3). The caller owns classification: this returns nothing, so it suits a seed loop that has already decided every key is new (it deduplicates again internally, so a repeat is harmless, not doubled).

func (*SeenSet) Len 

func (s *SeenSet) Len() int

Len reports the number of distinct keys the exact set holds.

func (*SeenSet) MarshalFilter 

func (s *SeenSet) MarshalFilter() []byte

MarshalFilter serializes the resident filter to a portable blob (doc 10 section 6). It captures only the approximate tier; the exact set is rebuilt from the urlkey column on reload. The bytes are deterministic for a given filter state, so a checkpoint that did not change the filter writes the same region.

func (*SeenSet) MarshalRibbon 

func (s *SeenSet) MarshalRibbon(opts ...RibbonOption) ([]byte, error)

MarshalRibbon freezes the seen-set's exact keys into a ribbon snapshot blob, the read-mostly form an engine writes when a partition goes cold. The keys come from the exact set, so the snapshot holds exactly what the set holds.

func (*SeenSet) MaybeContains 

func (s *SeenSet) MaybeContains(key meguri.URLKey) bool

MaybeContains reports the filter's verdict alone, with no exact confirm. A false is authoritative (the one-sided filter never drops a key it has seen), a true is the filter's "probably", which on a key the set never held is a false positive. It is the probe the benchmark uses to measure the achieved filter false-positive rate at the resident bits-per-URL (doc 14, section 3.7); the discovery path uses Seen and Contains, which confirm against the exact set.

func (*SeenSet) Merge 

func (s *SeenSet) Merge(keys []meguri.URLKey) []Classification

Merge classifies a whole batch of discovered keys at once through the DRUM path, the scale form of Seen: it routes the keys to buckets by HostKey prefix and merges each bucket in one sequential pass, returning a Unique verdict per key. The filter is updated for every unique key so later single-key Seen calls see them. Keys that repeat within the batch dedup against the first occurrence.

func (*SeenSet) Seen 

func (s *SeenSet) Seen(key meguri.URLKey) bool

Seen reports whether the key was already in the set, inserting it if new. It is the check-and-insert that makes discovery idempotent: true means a rediscovery (dedup, no new record), false means a genuinely new URL (now recorded). This is onDiscovery's core minus the link-signal crediting (doc 08, section 9.3).

The filter is consulted first. A miss is authoritative, the key is new, so it is added to both tiers. A hit is confirmed against the exact set: a true positive is a rediscovery, a false positive falls through to the new-URL path.

type SoftDetector 

type SoftDetector struct {
	// contains filtered or unexported fields
}

SoftDetector recognizes soft-404 farms by content fingerprint: a server that returns 200 OK with the same "not found" body for any URL defeats the 404 stop signal, but all those URLs share one content_fp, so counting distinct keys per fingerprint per host surfaces the template (doc 08, section 8.6). It also feeds HostFlagTrapSuspect: a host serving one boilerplate for many URLs is a trap suspect.

func NewSoftDetector 

func NewSoftDetector() *SoftDetector

NewSoftDetector returns a soft-404 detector at the default threshold.

func (*SoftDetector) IsTemplate 

func (d *SoftDetector) IsTemplate(hostKey, fp uint64) bool

IsTemplate reports whether fp is a confirmed soft-404 template on the host, without recording an observation.

func (*SoftDetector) Observe 

func (d *SoftDetector) Observe(hostKey, fp uint64, key meguri.URLKey) bool

Observe records that key on host returned content fingerprint fp, and reports whether fp is now a recognized soft-404 template. Once a fingerprint crosses the threshold of distinct keys, it is a template, and every subsequent URL on the host returning it should be treated as Gone rather than Crawled, restoring the stop signal the soft-404 tried to defeat.

func (*SoftDetector) WithThreshold 

func (d *SoftDetector) WithThreshold(n int) *SoftDetector

WithThreshold sets how many distinct keys sharing a fingerprint mark a soft-404 template.

type TrapDetector 

type TrapDetector struct {
	// contains filtered or unexported fields
}

TrapDetector recognizes calendar, faceted, and session-id trap signatures in a host's URL stream and decides, for a flagged host, whether a single discovery passes the pattern heuristics (doc 08, section 8.4). It is the precise defense layered on the blunt per-host budget and depth cap: the cap bounds every host, and the detector catches the trap hosts earlier and tighter.

The detector is pure given its inputs: the only clock it sees is the epoch-hours the caller threads in, which it converts to a year-month to place the calendar horizon. Same URLs, same now, same verdict, so a checkpoint replay is exact.

It accumulates per host: each kind has its own count of distinct trap-signature URLs, and a host crosses into suspect for a kind when that count passes the threshold. Suspicion is sticky for the life of the detector; the HostRecord flag the caller sets from it is what survives a checkpoint (doc 08, section 8.4).

func NewTrapDetector 

func NewTrapDetector(opts ...TrapOption) *TrapDetector

NewTrapDetector returns a detector at the default thresholds (doc 09 may retune them per campaign).

func (*TrapDetector) Flags 

func (d *TrapDetector) Flags(hostKey uint64) (calendar, faceted, session bool)

Flags reports which trap kinds the host is flagged for, so a caller can see why a host became a suspect (and a precision gate can tell a structural calendar or facet flag from a session-token explosion).

func (*TrapDetector) Observe 

func (d *TrapDetector) Observe(hostKey uint64, key meguri.URLKey, canonURL string, now uint32) (PatternKind, bool)

Observe folds one discovery into the host's accumulation and reports the URL's trap signature and whether the host became a suspect (for any kind) on this observation, so the caller flags the HostRecord exactly once. A URL with no trap signature still counts as an observation but moves no counter.

The host key the URL belongs to is passed explicitly rather than re-derived, so the detector shares the frontier's host grouping without re-parsing.

func (*TrapDetector) Passes 

func (d *TrapDetector) Passes(hostKey uint64, canonURL string, now uint32) bool

Passes is the admission heuristic feeding Admit's passesHeuristics argument: it reports whether a discovery on a flagged host may be scheduled. A host that is not a suspect passes everything (admission is unchanged until a host is flagged). A flagged host passes a URL only when the URL does not violate one of the rules the host was flagged for, so the calendar URLs of a calendar-flagged host are parked while its real article URLs are admitted (doc 08, section 8.4).

func (*TrapDetector) Suspect 

func (d *TrapDetector) Suspect(hostKey uint64) bool

Suspect reports whether the host has crossed the threshold on any trap kind.

func (*TrapDetector) Violation 

func (d *TrapDetector) Violation(canonURL string, now uint32) PatternKind

Violation classifies a single canonical URL: it returns the trap signature the URL bears, or PatternNone. It is the pure per-URL rule both the accumulation and the admission test are built on, so a URL the detector would count toward a flag is exactly a URL the detector would park on a flagged host. now is epoch-hours, the campaign clock the calendar horizon is measured from.

The order is calendar, then session, then faceted: a URL that both carries a session id and stacks facets is reported by its strongest signal first, but every rule is checked independently on a flagged host, so the order only names the URL, it does not weaken any rule.

type TrapOption 

type TrapOption func(*TrapDetector)

TrapOption configures a TrapDetector.

func WithCalendarHorizon 

func WithCalendarHorizon(months int) TrapOption

WithCalendarHorizon sets how many months past the current month a dated URL may point before the calendar rule parks it.

func WithFacetLimit 

func WithFacetLimit(n int) TrapOption

WithFacetLimit sets how many stacked facet parameters mark a faceted trap.

func WithFlagThreshold 

func WithFlagThreshold(n int) TrapOption

WithFlagThreshold sets how many distinct trap-signature URLs of one kind flag a host a suspect for that kind.

type WeightedFeature 

type WeightedFeature struct {
	Token  string
	Weight int64
}

WeightedFeature is one feature of a document for the simhash: a token and its weight, typically a term and its frequency (doc 08, section 7.2).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.