Documentation
¶
Overview ¶
Package safety holds the guards that stand between an agent-supplied URL and the network: address filtering, robots.txt compliance, and per-domain rate limiting.
The threat model is specific. When a CLI user types a URL, it is their own machine and their own intent. When an agent calls `distill`, the URL may have come from a web page the agent was reading, which means it is attacker influenced in the general case. A fetcher that will retrieve any URL it is handed is a server-side request forgery primitive, and one running inside a cloud environment is a credential disclosure primitive, because instance metadata services answer unauthenticated HTTP on a well-known address.
Index ¶
Constants ¶
const RobotsTTL = 24 * time.Hour
RobotsTTL is how long a stored robots.txt is trusted. A day is well inside the interval at which sites change these files and well outside the interval at which one tool re-reads a site.
const UserAgent = "sieve"
UserAgent is the token sieve identifies itself with, in robots.txt and in the User-Agent header. It carries a contact URL because a site operator who wants this traffic to stop should not have to guess who to ask.
Variables ¶
var ErrBlocked = errors.New("blocked by safety policy")
ErrBlocked is the class of error returned when a URL is refused. Callers distinguish it from a network failure, because "the site is down" and "we declined to connect" are different facts about the world.
var ErrUnreachable = errors.New("host unreachable")
ErrUnreachable is returned when the host could not be reached at all, as distinct from being refused.
The guard is the first thing in the pipeline to touch the network, so a flaky resolver surfaces here -- and reporting that as a policy refusal tells the operator that sieve declined to visit a site it was in fact willing to visit, which sends them looking for a block that does not exist. The two cases also want opposite responses: a refusal should be respected, and a resolver timeout should be retried.
Functions ¶
This section is empty.
Types ¶
type Guard ¶
type Guard struct {
// contains filtered or unexported fields
}
Guard vets URLs before the browser is allowed to open them.
func (*Guard) Check ¶
Check vets one URL. It is safe for concurrent use and is called for the initial navigation and again for every redirect hop, because a URL that passed once says nothing about where it points after three hops. Checking only the first request is the most common way an SSRF filter is defeated: the attacker supplies a benign public host that answers 302 to 169.254.169.254.
func (*Guard) ForPage ¶
ForPage returns a guard with the same policy and a fresh redirect budget.
The budget belongs to one page's redirect chain, not to the process. A long-lived server that shares a single guard across pages is counting every hop every page has ever taken, so it works for a handful of URLs and then refuses everything: the MCP server answered four pages and rejected the next hundred and eighty with "more than 8 redirects", including pages that redirect exactly once. The CLI never showed it because each run is a new process.
This is a copy rather than a Reset because pages can be in flight at the same time, and resetting a shared counter mid-chain would clear a budget another page is still spending. Nothing here is per-URL state except the counters, so the copy is cheap and the policy is identical.
type GuardConfig ¶
type GuardConfig struct {
// AllowPrivate disables the private-address check. It exists for
// distilling a site on localhost during development and must never be on
// for agent-supplied URLs.
AllowPrivate bool
// AllowedSchemes defaults to http and https.
AllowedSchemes []string
// AllowHosts, when non-empty, is an allowlist: nothing else is fetched.
AllowHosts []string
// DenyHosts is applied after AllowHosts.
DenyHosts []string
// MaxRedirects bounds a redirect chain.
MaxRedirects int
// Resolver is injectable for tests.
Resolver *net.Resolver
// Fallback is consulted only to confirm a negative, and only when the
// primary resolver has already refused. It exists because a platform
// resolver can report a live domain as nonexistent, and one mechanism's
// confident wrong answer should not end a run. Nil means Go's own DNS
// client, which is independent of the default resolver's getaddrinfo path.
Fallback *net.Resolver
// LookupTimeout bounds name resolution.
LookupTimeout time.Duration
}
GuardConfig configures address filtering.
func DefaultGuardConfig ¶
func DefaultGuardConfig() GuardConfig
DefaultGuardConfig is the policy for agent-supplied URLs.
type Limiter ¶
type Limiter struct {
// contains filtered or unexported fields
}
Limiter enforces politeness towards a single site: how many pages may be rendered at once, and how often.
This is not rate limiting for our own benefit. A distiller that opens eight headless tabs against a small studio's site and hammers it is indistinguishable from an attack, and the crawl-delay a site publishes is a request that should be honoured rather than logged.
func NewLimiter ¶
NewLimiter builds a limiter. concurrency is per host, not global.
func (*Limiter) Acquire ¶
Acquire blocks until it is polite to fetch from this host, and returns the function that releases the slot.
type Robots ¶
type Robots struct {
// CrawlDelay is the delay the most specific matching group asked for.
CrawlDelay time.Duration
// Sitemaps are advertised sitemap URLs, useful for a bounded crawl.
Sitemaps []string
// Missing is true when the file did not exist, which means everything is
// allowed. It is recorded so the artifact can say which it was.
Missing bool
// contains filtered or unexported fields
}
Robots is a parsed robots.txt for one origin.
func FetchRobots ¶
FetchRobots retrieves and parses robots.txt for a URL's origin.
The status codes follow RFC 9309, which divides them in a way that is worth stating because the obvious reading is backwards.
"Unavailable" is the 4xx range, and it means the rules do not exist for us: the standard says a crawler may then access any resource, and Google documents the same behaviour. That covers 401 and 403 as well as 404. sieve used to treat those two as a refusal of the entire site, reasoning that a site which will not show its rules has not invited us in -- which sounds careful and is not: sciencedirect.com returns 403 on robots.txt to user agents it does not recognise while serving its articles to anyone, and sieve declined the whole domain over it.
"Unreachable" is the 5xx range, and that is the one that means stop. A server that cannot answer might be hiding a Disallow we are obliged to honour, so the standard requires assuming a full disallow. That is the opposite of the 4xx case and it is the direction the caution belongs in.
A transport error is neither: we could not ask, nobody refused, and the request itself will fail in a moment anyway.
type RobotsCache ¶
type RobotsCache struct {
// contains filtered or unexported fields
}
RobotsCache holds one parsed robots.txt per origin.
It can be persisted between runs, and on this corpus that matters more than it sounds: asking permission is the one thing that must happen before anything else, it costs a round trip to a cold host, and several of these sites take well over a second to answer. That was a second and a half of a ten-second budget, spent before a byte of the page had been requested, to re-learn something that had not changed since the last run an hour earlier.
Persisting it does not weaken the check. The stored copy is what the site said, it expires, and a run that finds no fresh copy asks again.
func NewRobotsCache ¶
func NewRobotsCache(client *http.Client) *RobotsCache
NewRobotsCache builds a cache.
func (*RobotsCache) Restore ¶
func (c *RobotsCache) Restore(in map[string]StoredRobots)
Restore loads a persisted cache, discarding anything past its TTL.
func (*RobotsCache) Snapshot ¶
func (c *RobotsCache) Snapshot() map[string]StoredRobots
Snapshot exports the cache for persistence.
type StoredRobots ¶
type StoredRobots struct {
FetchedAt time.Time `json:"fetched_at"`
Body string `json:"body"`
// Missing records that the site had no robots.txt, which is itself an
// answer and is worth not asking for twice.
Missing bool `json:"missing,omitempty"`
}
StoredRobots is the persisted form: the raw file as served, per origin, with the time it was read.
Source Files