Documentation
¶
Overview ¶
Package politeness enforces the rules that keep a crawl from hammering a host: the per-host and per-IP token buckets, the crawl-delay derived from robots.txt and adaptive backoff, and the gate that decides when a host's next fetch is eligible. It reads and writes the politeness fields of meguri.HostRecord and is consulted by the frontier scheduler before every dispatch.
This is part of the M3 milestone (politeness, DNS, robots, conditional GET). The package is a placeholder until then.
Package politeness is meguri's rate brain: the policy that decides how fast a single host may be crawled and keeps two independent throttles in step, one per host group and one per resolved IP. It owns no network and no state beyond the shared per-IP table; the frontier reads its functions to compute a host's next-eligible instant and to fold one fetch outcome into the host's adaptive interval (doc 07). The split is deliberate: meguri owns the politeness policy, ami owns the fetch mechanism.
The model is two one-token buckets reduced to timestamps. A bucket has no burst: a fetch may start only at or after its next-eligible instant, and the instant advances by one interval measured start-to-start. A host is dispatched only when BOTH its host bucket and its IP bucket permit, and dispatching spends both. The per-IP bucket is what stops meguri from hammering one machine through a hundred vhosts that share an address.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
Default time.Duration // baseline interval for a host with no signal yet
Floor time.Duration // hard minimum; never fetch one host faster than this
Ceiling time.Duration // hard maximum; a host pinned here is flagged dead
IPFloor time.Duration // per-IP minimum when many hosts share one address
Backoff float64 // multiplicative widen on 429/5xx, >= 1
SoftWiden float64 // multiplicative widen on sharply rising latency, >= 1
Narrow time.Duration // additive narrow toward the floor on a healthy fetch
LatencyRise float64 // latency ratio that counts as "rising", e.g. 1.5
}
Config is the politeness policy: the interval band a host's rate must stay in and the AIMD reaction constants that turn fetch outcomes into a crawl rate. The defaults come from doc 07: a one-second baseline, a 250ms floor that is the physical wall meguri refuses to cross (D19), and a ceiling past which a host is treated as dead.
func (Config) Adapt ¶
Adapt is the AIMD step: it maps the current adaptive interval and one fetch signal to the next interval, before clamping. The reactions, in order:
- a 429 or 5xx multiplies the interval by Backoff and honors Retry-After as a floor, the multiplicative increase that backs a struggling host off fast.
- sharply rising latency (Latency over PrevLatency*LatencyRise) softly widens by SoftWiden, easing off before the host starts erroring.
- an otherwise healthy fetch narrows the interval by Narrow, the additive decrease that creeps a recovered host back toward the floor slowly.
The result is unclamped; the caller applies HostInterval to bound it into the host's legal band.
func (Config) AtCeiling ¶
AtCeiling reports whether the clamped interval has been pinned to the ceiling, the condition that, sustained over a fetch run, marks a host dead.
func (Config) HostInterval ¶
HostInterval clamps a host's adaptive interval into the legal band. The lower bound is the larger of the global floor and the host's published crawl-delay (from robots or config), so a site asking to be crawled slower is always honored and meguri never dips under its own physical floor. The upper bound is the ceiling.
type IPTable ¶
type IPTable struct {
// contains filtered or unexported fields
}
IPTable is the shared per-IP politeness state. Many host groups can resolve to one address (a CDN, shared hosting), and hammering that machine through a hundred vhosts is the rudeness the per-host bucket alone cannot catch. The table reduces each IP to a single next-eligible instant, advanced every time a host on that IP dispatches, and stays bounded by the in-flight IP working set because idle buckets are evicted.
It is safe for concurrent use: the engine dispatches many hosts at once and several of them may share an address.
func NewIPTable ¶
NewIPTable returns an empty table whose per-IP interval never drops below floor, regardless of how fast an individual host on that IP is allowed to go.
func (*IPTable) EligibleAt ¶
EligibleAt returns the unix-second instant ip may next be fetched, or 0 when the IP is unknown or unresolved (the zero address).
func (*IPTable) Evict ¶
Evict drops IP buckets untouched at or before cutoff, keeping the table sized to the active IP working set rather than every address ever resolved. It returns the number evicted.
func (*IPTable) Spend ¶
Spend records a dispatch to ip at unix-second now using the host's interval, pushing the IP's next-eligible forward by at least the per-IP floor. A fetch is only dispatched when the IP already permits, so now is at or past the old next-eligible and the advance is monotonic. It is a no-op for the zero IP.
type Signal ¶
type Signal struct {
Status int // HTTP status, 0 for a transport error
RetryAfter time.Duration // Retry-After header value, 0 if none
Latency time.Duration // this fetch's round-trip time
PrevLatency time.Duration // the host's smoothed latency before this fetch
}
Signal is the part of a fetch Outcome the rate controller reads.
Source Files