Help us shape the future of Go! Take the 2026 Go Developer Survey

fetch

package
v0.37.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 21, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Overview

Package fetch is autoupdate's HTTP layer: the retrying client with its circuit breaker, per-host policies and body caps, the rate limiter, the response and fetch caches, header allow-listing with credential scoping, and authenticated distfile fetching driven by a record's meta.fetch_* keys, including the spec parser that --lint reuses.

It imports no other autoupdate package except statefile; the registry imports it to validate meta.fetch_* the way the fetch parses it.

Index

Constants

View Source 
const (

	// MetaFetchMethod is the meta key fetch_method: "post" (default) or "get".
	MetaFetchMethod = "fetch_method"
	// MetaFetchURL is the meta key fetch_url: form action / endpoint.
	MetaFetchURL = "fetch_url"
	// MetaFetchSerialEnv is the meta key fetch_serial_env: env var holding the serial; optional, but only together with the field.
	MetaFetchSerialEnv = "fetch_serial_env"
	// MetaFetchSerialField is the meta key fetch_serial_field: form field the serial goes into; optional, but only together with the env var.
	MetaFetchSerialField = "fetch_serial_field"
	// MetaFetchForm is the meta key fetch_form: other form fields, urlencoded.
	MetaFetchForm = "fetch_form"
	// MetaFetchFilename is the meta key fetch_filename: dest name; {version} is substituted.
	MetaFetchFilename = "fetch_filename"
)

meta keys that drive an authenticated fetch. fetch_url is the trigger: a package without it is fetched the normal way (pkgdev from SRC_URI).

View Source 
const (

	// FetchBodyJSON is the fetch_body value that sends the form as a JSON object.
	FetchBodyJSON = "json"

	// FetchResponseURL is the fetch_response value saying the first reply's body
	// IS the download URL.
	FetchResponseURL = "url"

	// VersionPlaceholder is substituted in fetch_filename, fetch_id_url and
	// fetch_id_pattern; idPlaceholder is substituted in fetch_url once the id
	// lookup has answered.
	VersionPlaceholder = "{version}"
)

The two encodings fetch_body selects, and the two things fetch_response says the first reply is. They are constants rather than bare literals because the parser, the validator and the request builder each compare against them, and a literal in three places is a literal that can be misspelled in one.

View Source 
const (

	// TemplateNotHTTP means the template is not an absolute http(s) URL.
	TemplateNotHTTP templateFault
	// TemplatePlaceholderInHost means a placeholder sits in the scheme or authority.
	TemplatePlaceholderInHost
)
View Source 
const (
	// DefaultFetchCachePerBody is the largest response body a run will retain
	// for reuse, in bytes: 2 MiB (2097152).
	DefaultFetchCachePerBody int64 = 2 << 20

	// DefaultFetchCacheBudget is the total number of bytes a run will retain
	// across all identities before admission stops, in bytes: 64 MiB
	// (67108864).
	DefaultFetchCacheBudget int64 = 64 << 20
)

The retention limits a real run uses, in bytes. They bound MEMORY, never correctness: a body over a limit is still fetched and returned to its caller unchanged, it is simply not kept.

Both figures come from measuring the registry:

  • 2 MiB per body — the most widely shared body, the 170-way GStreamer tags listing, measures 33.7 KB, some sixty times under the cap. The largest payload, www-misc/warsaw at ~8.2 MB, is declared by exactly ONE record, so retaining it would save no fetch while costing more than the rest combined.
  • 64 MiB total — the 230 distinct URLs at tens of KB each land near ~10 MB, so this is a ceiling on pathology, not a working figure.

They are comfortable rather than tuned: confirm them against a real run's counters (Oversize and BudgetFull staying at zero) before treating them as settled.

View Source 
const (
	// DefaultMaxRetries is the default number of retry attempts.
	DefaultMaxRetries = 3
	// DefaultRetryBaseDelay is the default base delay before the first retry.
	DefaultRetryBaseDelay = 1 * time.Second
	// DefaultHTTPTimeout is the default timeout for each HTTP request.
	DefaultHTTPTimeout = 30 * time.Second

	// DefaultBreakerMaxFailures is the number of consecutive failures before the circuit opens.
	DefaultBreakerMaxFailures = 5
	// DefaultBreakerTimeout is how long the circuit stays open before allowing a probe.
	DefaultBreakerTimeout = 30 * time.Second
	// DefaultBreakerMaxRequests is the number of probes allowed in half-open state.
	DefaultBreakerMaxRequests = 3
	// DefaultBreakerInterval is the window over which failures are counted.
	DefaultBreakerInterval = 60 * time.Second
)
View Source 
const (
	// DefaultLLMInterval is the minimum interval between LLM requests (5 per minute).
	DefaultLLMInterval = 12 * time.Second
	// DefaultHTTPInterval is the minimum interval between HTTP requests per domain (10 per minute).
	DefaultHTTPInterval = 6 * time.Second
	// DefaultLLMBurst is the burst size for LLM rate limiting.
	DefaultLLMBurst = 1
	// DefaultHTTPBurst is the burst size for HTTP rate limiting per domain.
	DefaultHTTPBurst = 1
	// DefaultMaxDomains is the maximum number of HTTP domain limiters to keep in memory.
	DefaultMaxDomains = 256
	// DefaultCleanupAge is the age after which an unused domain entry can be evicted.
	DefaultCleanupAge = 1 * time.Hour
)
View Source 
const DefaultCacheTTL = time.Hour

DefaultCacheTTL is the default time-to-live for cache entries (1 hour)

View Source 
const KeySeparator = "\x00"

KeySeparator joins the raw URL to the header digest inside a cache key. A NUL byte is used because no URL can contain one — net/url rejects it outright ("invalid control character in URL"), so no URL can spell the boundary itself and impersonate another identity's header term.

View Source 
const MaxRetryAfter = 60 * time.Second

MaxRetryAfter is the longest Retry-After the client waits out. A longer one fails the request at once instead of holding a check for minutes.

Variables

View Source 
var (
	// ErrAuthFetchSecretMissing is returned when the configured serial cannot be
	// resolved from the environment or the secrets file.
	ErrAuthFetchSecretMissing = errors.New("authenticated fetch: serial not found")
	// ErrAuthFetchFailed is returned when the download itself fails (bad config,
	// transport error, non-OK status, or an HTML "form re-shown" response).
	ErrAuthFetchFailed = errors.New("authenticated distfile fetch failed")
)

Error variables for authenticated fetching.

View Source 
var (
	// ErrMetaFetchURLRequired is returned when a [meta] block configures an
	// authenticated fetch but its trigger, fetch_url, is missing or blank.
	ErrMetaFetchURLRequired = errors.New("meta: fetch_url is required when any fetch_* key is present (it is the trigger; without it the authenticated download is silently skipped)")
	// ErrInvalidMetaFetchMethod is returned when meta.fetch_method holds
	// something other than the two HTTP methods the fetcher can build.
	ErrInvalidMetaFetchMethod = errors.New(`invalid meta.fetch_method: must be "post" (default) or "get"`)
	// ErrInvalidMetaFetchBody is returned when meta.fetch_body names an encoding
	// the request builder cannot produce.
	ErrInvalidMetaFetchBody = errors.New(`invalid meta.fetch_body: must be "form" (default) or "json"`)
	// ErrInvalidMetaFetchResponse is returned when meta.fetch_response names
	// something other than the two things the first reply can be.
	ErrInvalidMetaFetchResponse = errors.New(`invalid meta.fetch_response: must be "file" (default) or "url"`)
	// ErrUnknownMetaFetchKey is returned for a fetch_*-prefixed key the applier
	// does not read — almost always a misspelling of one of the twelve.
	ErrUnknownMetaFetchKey = errors.New("meta: unknown authenticated-fetch key")
)

Validation errors for the meta.fetch_* sub-schema. They are checked at lint time (the registry load does not validate), unlike the Err* pair above, which report a download that actually failed.

View Source 
var (
	// ErrCacheCorrupted is returned when the cache file cannot be parsed
	ErrCacheCorrupted = errors.New("cache file is corrupted")
	// ErrCacheMiss is returned when a cache entry is not found or expired
	ErrCacheMiss = errors.New("cache miss")
)

Error variables for cache errors

View Source 
var (
	// ErrMaxRetriesExceeded is returned when all retry attempts have failed
	ErrMaxRetriesExceeded = errors.New("max retries exceeded")
	// ErrRequestTimeout is returned when a request times out
	ErrRequestTimeout = errors.New("request timeout")
	// ErrRetryAfterTooLong is returned, without waiting, when an upstream's
	// Retry-After asks for more than MaxRetryAfter or for more than the time
	// left before the context deadline.
	ErrRetryAfterTooLong = errors.New("retry-after too long")
	// ErrResponseTooLarge is returned when an HTTP response body exceeds the
	// MaxBodyBytes cap. It is httpx's sentinel, assigned rather than copied,
	// so errors.Is matches it on the provider paths too.
	ErrResponseTooLarge = httpx.ErrResponseTooLarge
	// ErrCredentialHostMismatch is returned, before any network I/O, when a
	// header references a credential variable that is bound to hosts other
	// than the one the request goes to.
	ErrCredentialHostMismatch = errors.New("credential header bound to another host")
)

Error variables for HTTP client errors

View Source 
var (
	// ErrRateLimitExceeded is returned when a rate limit is exceeded and context is cancelled
	ErrRateLimitExceeded = errors.New("rate limit exceeded")
)

Error variables for rate limiting errors

Functions

func BackoffCeilingFor 

func BackoffCeilingFor(rc RetryConfig, attempt int) time.Duration

BackoffCeilingFor is the single source of the exponential backoff cap: BaseDelay×2^(attempt-1), capped at MaxDelay (1s, 2s, 4s, 4s… by default), and 0 for attempt <= 0. The retry wait draws below it and deriveOpTimeout sums it, so the operation budget always covers the longest possible waits.

func BodyKey 

func BodyKey(rawURL string, headers map[string]string) string

BodyKey derives the identity two upstream reads must share before either may be handed the other's bytes: rawURL + NUL + hex(sha256(canonicalHeaders(headers))). Two reads share a key only when the requests they would issue are byte-identical. The URL is carried verbatim, so identity follows the URL actually requested and no registry edit can leave the key disagreeing with it.

Collapsing wrongly hands one entry another entry's bytes. Range is the sharp edge: the checker accepts a 206, and a 206 fragment handed to a caller that asked for the whole file looks like a successful read of a truncated document. Keying on the declared headers makes that impossible by construction.

The header term is hashed so a key is safe to log at DEBUG by construction (today ${VAR} placeholders are still unexpanded here). Deliberately absent from the key: the client's default headers (fixed per Checker), the GitHub token (a function of the URL, which is in the key) and ${VAR} expansions (the environment is fixed mid-run). A per-package User-Agent, per-package tokens or live expansion would force the key onto the effective header set applyHeaders produces. A nil map is a valid "no headers declared".

func CheckCredentialBinding 

func CheckCredentialBinding(rawURL string, headers map[string]string, scope CredentialScope) error

CheckCredentialBinding returns an error wrapping ErrCredentialHostMismatch when a header in headers references an allow-listed variable that is not bound to rawURL's host. It is a pure check: it reads no environment variable and performs no I/O, and its error never carries a variable's value.

func ClassifyBodyReadError 

func ClassifyBodyReadError(err error) error

ClassifyBodyReadError maps an error returned while reading an HTTP response body to a domain error. When the read tripped an http.MaxBytesReader cap the standard library yields an *http.MaxBytesError; this is translated into an error wrapping ErrResponseTooLarge. Any other non-nil error is returned unchanged, and a nil error yields nil. It delegates to httpx.ClassifyBodyReadError, which owns the shared sentinel.

func ContainsCRLF 

func ContainsCRLF(s string) bool

ContainsCRLF reports whether s contains a carriage return or line feed. Such characters in a header name are a header/CRLF-injection vector and are always rejected, independently of canonicalisation.

func ExtractDomain 

func ExtractDomain(rawURL string) (string, error)

ExtractDomain extracts the domain from a URL string.

func IsAllowedHeaderName 

func IsAllowedHeaderName(name string) bool

IsAllowedHeaderName reports whether the given header name is eligible for environment-variable expansion of its value.

The name is trimmed of surrounding whitespace and canonicalised with textproto.CanonicalMIMEHeaderKey before the allow-list lookup, so callers may pass values with arbitrary casing or padding. Any name containing a CR or LF byte is rejected outright (defence against CRLF/header injection) BEFORE canonicalisation, because textproto.CanonicalMIMEHeaderKey returns its input unchanged when it contains invalid bytes.

func ParseAuthFetchSpec 

func ParseAuthFetchSpec(meta map[string]string) (*authFetchSpec, bool, error)

ParseAuthFetchSpec extracts an authFetchSpec from a package's meta map.

It returns (nil, false, nil) when the package defines no authenticated fetch (no fetch_url) — the common case, handled by the normal manifest path. When fetch_url IS present it returns a non-nil error if any required companion field is missing or malformed, so a half-written config fails loudly rather than silently falling back to a fetch that cannot work.

func ReadBodyForStatus 

func ReadBodyForStatus(resp *http.Response, accepted ...int) ([]byte, error)

ReadBodyForStatus validates an HTTP response status against the accepted set supplied by the caller and, when the status is accepted, reads the body in full and classifies an overflow. A status outside the set yields "HTTP request returned status %d" without touching the body.

The accepted statuses are the caller's to choose: Checker.fetchContent passes http.StatusOK plus http.StatusPartialContent when the request actually sent carried a Range header (sentRequestDeclaresRange in checker.go reads the request recorded on the response, not the record's declared header map); Analyzer.fetchContentFromURL passes http.StatusOK alone. Both list codes explicitly rather than accepting all of 2xx, since 204 and 205 have an empty body by definition.

It imposes no cap of its own: the GET helpers already bound the body with httpx.MaxBodyBytes, so the bound has a single source, and an overflow is translated into ErrResponseTooLarge via classifyBodyReadError. The LLM path keeps its own per-client, raisable cap (readCappedBody in llm.go) and deliberately does not route through here.

func SubstituteEnvVars 

func SubstituteEnvVars(log *slog.Logger, value, headerName string) string

SubstituteEnvVars replaces ${VAR_NAME} patterns in a header value with the corresponding environment variable values, subject to a strict allow-list.

A ${VAR} reference is expanded ONLY when BOTH of the following hold:

  • headerName is an allow-listed header (see isAllowedHeaderName), AND
  • VAR is an allow-listed environment variable (see isAllowedEnvVar).

Substitution is single-pass: a value produced by expanding one ${VAR} is never re-scanned, so a secret whose own value contains ${OTHER} cannot trigger a second expansion.

On any denial — header not allow-listed, variable not allow-listed, or an allow-listed variable that is unset/empty — the literal ${VAR} text is passed through unchanged and a Warn-level line is emitted identifying the header and variable. The line goes to log; nil discards it.

func URLTemplateFault 

func URLTemplateFault(template string) templateFault

URLTemplateFault applies the rule checkFetchURLTemplate documents — an absolute http(s) URL whose scheme and authority hold no placeholder — and says which half failed, so each caller words its own error.

func ValidateMetaFetch 

func ValidateMetaFetch(pkg string, meta map[string]string) error

ValidateMetaFetch checks the fetch_* sub-schema hiding inside a package's free-form [meta] map. It lives beside parseAuthFetchSpec on purpose: these rules only stay correct while they mirror that consumer, and holding the two in different files is what let the schema go undocumented.

Its own rules come first and cover what the parser cannot report: the silent case. A [meta] block whose trigger is missing or misspelled does not read as broken, it reads as "no authenticated fetch", and pkgdev is then sent to digest a distfile that exists on no public mirror.

Once those pass and fetch_url is set, it runs parseAuthFetchSpec itself and returns the parser's error unchanged. The parser is the one place a record is refused — a missing filename, a half-declared serial, a variable outside the BENTOO_FETCH_ namespace — so --lint, the sweep and `bentoo distfile` refuse the same records with the same text instead of lint approving what the fetch then rejects. Running it here costs nothing on the load path: LoadPackagesConfig does not validate, and LintPackagesConfig reports one message per record, so a broken record is one issue for that package and never blocks the others. The parser only reads the map: it resolves no variable and does no I/O.

Types

type BodyCache 

type BodyCache struct {
	// contains filtered or unexported fields
}

BodyCache deduplicates response bodies within the lifetime of one Checker — that is, one command invocation. It is not persisted and has no TTL: a run is short enough that an upstream endpoint is treated as immutable for its duration, which is the same assumption every concurrent HTTP client makes.

It knows nothing about HTTP, packages or the registry: it is handed a key and a function that produces bytes. That is what keeps its concurrency contract provable without a server, and what keeps the two fetch paths that must never be cached — multi-megabyte distfiles and the analyzer's one-shot reads — out of reach by construction, since neither goes through a Checker's fetch door.

func NewDefaultBodyCache 

func NewDefaultBodyCache() *BodyCache

NewDefaultBodyCache builds the cache a real run uses, at the limits above.

It exists so the two places that install one — NewChecker's struct literal and WithFetchCache(true) — cannot drift apart. "Enabled explicitly" has to mean exactly "enabled by default"; with the limits spelled out at both sites that would hold only by coincidence, and a future edit to one argument list would make an option that claims to be the default quietly install something else.

func (*BodyCache) Do 

func (c *BodyCache) Do(ctx context.Context, key string, fetch func() ([]byte, error)) ([]byte, error)

Do returns the body for key, fetching it at most once across concurrent callers: the first caller to arrive leads and calls fetch, callers arriving while it is in flight wait for its result, and later callers are served the retained body with no fetch and no rate-limit token.

THIS FUNCTION TAKES NO LOCK, and the mutex is never held across fetch or the wait: a lock held there would pass every test while serialising all records behind one network round trip. Each locking helper (classify, publish, countRefetch, retain) is short and holds neither a fetch nor a wait. The wait is bounded by the parent ctx, never a per-operation timeout: queue time charged to an HTTP deadline once failed packages before any request was issued.

An error is neither cached nor shared: a waiter released by a failed leader fetches on its own. No outcome is logged — a cold cache misses on every read; the counters are the observability surface, and the caller logs its failures.

THE RETURNED SLICE IS SHARED AND READ-ONLY: a mutation would silently make one package report another's version, and copying per waiter would restore the memory cost the cache removes.

func (*BodyCache) Snapshot 

func (c *BodyCache) Snapshot() BodyCacheStats

Snapshot returns the per-run counters by value. The copy is deliberate and is taken under the mutex: handing out a pointer into live state would let a reader observe counters torn mid-update, and would let a caller mutate the figures the run is reporting.

type BodyCacheStats 

type BodyCacheStats struct {
	Hits       int // served from a retained body
	Joins      int // waited on an in-flight fetch
	Misses     int // became a leader and fetched
	Refetches  int // a leader failed, so a waiter fetched on its own
	Oversize   int // fetched, returned, not retained (per-body cap)
	BudgetFull int // fetched, returned, not retained (total budget)
}

BodyCacheStats is DEBUG-level observability, and the assertion surface tests use to prove a request was or was not issued. Counts are per bodyCache, and a bodyCache lives exactly as long as one Checker, so these are per-run figures that are never carried across invocations.

type Cache 

type Cache struct {
	// Entries holds all cached version entries, keyed by package name
	Entries map[string]CacheEntry `json:"entries"`
	// Preconditions holds the host preconditions a build gate found unmet, keyed
	// by package exactly as Entries is. Every Cache carries it, including the
	// checker's, so that any writer round-trips the map instead of dropping it:
	// a save from an object that did not know the field would erase every record
	// on disk.
	Preconditions map[string]PreconditionRecord `json:"preconditions,omitempty"`
	// TTL is the time-to-live for cache entries
	TTL time.Duration
	// contains filtered or unexported fields
}

Cache manages version query caching with TTL-based expiration. It persists cache entries to disk and supports concurrent access.

func NewCache 

func NewCache(configDir string, opts ...CacheOption) (*Cache, error)

NewCache creates or loads a cache from disk. If the cache file exists, it loads existing entries. If the cache file doesn't exist or is corrupted, it creates a new empty cache. The configDir should be the bentoo config directory (e.g., ~/.config/bentoo/autoupdate).

func (*Cache) Cleanup 

func (c *Cache) Cleanup() error

Cleanup removes all expired entries from the cache. It automatically saves the cache to disk after cleanup.

Preconditions are not swept here and must not be: the TTL is about how stale an upstream version answer is, and expiring a host diagnosis on that clock would hand the package back to the same failing build an hour later.

func (*Cache) Clear 

func (c *Cache) Clear() error

Clear removes all entries from the cache. It automatically saves the cache to disk after clearing.

Recorded preconditions go with them, deliberately: clearing the cache is what an operator does to unstick a run, and a "clear" that left a package held back by a diagnosis they cannot see would be the one state nothing can get out of.

func (*Cache) Delete 

func (c *Cache) Delete(pkg string) error

Delete removes a package from the cache. It automatically saves the cache to disk after deletion.

func (*Cache) DeletePrecondition 

func (c *Cache) DeletePrecondition(pkg string) error

DeletePrecondition drops the recorded precondition for pkg and persists the cache. Deleting a package that has no record is not an error: the caller's intent — "this package must not be held back" — is already satisfied.

func (*Cache) Get 

func (c *Cache) Get(pkg string) (string, bool)

Get retrieves a cached version if it exists and is not expired. Returns the version and true if found and valid, empty string and false otherwise.

func (*Cache) GetEntry 

func (c *Cache) GetEntry(pkg string) (CacheEntry, bool)

GetEntry retrieves the full cache entry for a package. Returns the entry and true if found, zero value and false otherwise. This does not check TTL - use Get for TTL-aware retrieval.

func (*Cache) GetWithForce 

func (c *Cache) GetWithForce(pkg string, force bool) (string, bool)

GetWithForce retrieves a cached version, optionally ignoring the cache. If force is true, always returns cache miss. Returns the version and true if found and valid (and not forced), empty string and false otherwise.

func (*Cache) Len 

func (c *Cache) Len() int

Len returns the number of entries in the cache.

func (*Cache) Precondition 

func (c *Cache) Precondition(pkg string) (PreconditionRecord, bool)

Precondition returns the recorded unmet precondition for pkg. Unlike Get, it applies no TTL — see PreconditionRecord.RecordedAt for why a clock is the wrong thing to expire this on.

func (*Cache) Save 

func (c *Cache) Save() error

Save persists the cache to disk. This is thread-safe and can be called concurrently.

func (*Cache) Set 

func (c *Cache) Set(pkg, version, source string) error

Set stores a version in the cache with the current timestamp. It automatically saves the cache to disk after setting.

func (*Cache) SetPrecondition 

func (c *Cache) SetPrecondition(pkg, path string) error

SetPrecondition records the host precondition a build gate found unmet for pkg, replacing any earlier record for the same package, and persists the cache.

The path is expected to have come from extractUnmetPrecondition, which is where the "is this answer usable" judgement lives; this method is the storage and nothing more. It does not touch the package's version entry, so a package with a precondition and no cached version stays a cache MISS for version lookups — which is correct: nothing here is evidence about upstream.

type CacheEntry 

type CacheEntry struct {
	// Version is the cached version string
	Version string `json:"version"`
	// Timestamp is when this entry was cached
	Timestamp time.Time `json:"timestamp"`
	// Source is the URL that was queried to get this version
	Source string `json:"source"`
}

CacheEntry represents a cached version query result. It stores the version, when it was cached, and the source URL.

type CacheOption 

type CacheOption func(*Cache)

CacheOption is a functional option for configuring Cache

func WithNowFunc 

func WithNowFunc(fn func() time.Time) CacheOption

WithNowFunc sets a custom time function for testing

func WithTTL 

func WithTTL(ttl time.Duration) CacheOption

WithTTL sets a custom TTL for the cache

type Clock 

type Clock interface {
	Now() time.Time
	Sleep(d time.Duration)
}

Clock interface allows mocking time for testing

type CredentialScope 

type CredentialScope struct {
	OwnHosts []string
}

CredentialScope carries the hostnames a BENTOO_* credential may be sent to: the hosts of the package's own url and base_url, or, for a caller with no package (GetWithHeadersContext), the request's own host.

type PreconditionRecord 

type PreconditionRecord struct {
	// Path is the absolute path the failed build named and could not read. It is
	// never relative and never "/" — extractUnmetPrecondition refuses both,
	// because an answer that resolves differently depending on who asks is worse
	// than no answer at all.
	Path string `json:"path"`
	// RecordedAt is when the failure was observed. It is deliberately NOT an
	// expiry. The version TTL measures how stale an upstream answer is, which is
	// no evidence whatsoever about whether a key file has appeared on disk; a
	// record that timed out on a clock would simply restore the wasteful retry
	// loop it exists to stop. What ends a record is the path becoming readable.
	RecordedAt time.Time `json:"recorded_at"`
}

PreconditionRecord is what a build gate learned about THIS HOST while failing: a path the build needed and could not have. It is a fact about the machine, not about the package's upstream, and it is what lets a second run decline a gate that can only fail the same way again.

It lives here — in cache.json, under ~/.config/bentoo — and it may live nowhere else. The obvious alternative, the package's own registry entry, is in packages.toml, and that file sits in an overlay that auto-commits and pushes within minutes: a note about one workstation's unreadable key would be published to every consumer of the overlay as though it were a property of the package.

type RateLimiter 

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter manages request rate limiting for LLM and HTTP requests. It enforces: - LLM rate limiting: 5 requests per minute - HTTP rate limiting: 10 requests per minute per domain (bounded to maxDomains entries)

func NewRateLimiter 

func NewRateLimiter(opts ...RateLimiterOption) *RateLimiter

NewRateLimiter creates a new rate limiter with default settings. LLM requests are limited to 5 per minute. HTTP requests are limited to 10 per minute per domain, with a maximum of DefaultMaxDomains tracked concurrently (LRU eviction when full).

func (*RateLimiter) AllowHTTP 

func (r *RateLimiter) AllowHTTP(domain string) bool

AllowHTTP reports whether an HTTP request to the domain may happen now. It does not block or consume a token.

func (*RateLimiter) AllowLLM 

func (r *RateLimiter) AllowLLM() bool

AllowLLM reports whether an LLM request may happen now. It does not block or consume a token.

func (*RateLimiter) DomainCount 

func (r *RateLimiter) DomainCount() int

DomainCount returns the number of domains being tracked for HTTP rate limiting.

func (*RateLimiter) HTTPLimit 

func (r *RateLimiter) HTTPLimit(domain string) rate.Limit

HTTPLimit returns the current HTTP rate limit for a domain (requests per second).

func (*RateLimiter) LLMLimit 

func (r *RateLimiter) LLMLimit() rate.Limit

LLMLimit returns the current LLM rate limit (requests per second).

func (*RateLimiter) ReserveHTTP 

func (r *RateLimiter) ReserveHTTP(domain string) *rate.Reservation

ReserveHTTP returns a Reservation that indicates how long the caller must wait before an HTTP request to the domain can proceed.

func (*RateLimiter) ReserveLLM 

func (r *RateLimiter) ReserveLLM() *rate.Reservation

ReserveLLM returns a Reservation that indicates how long the caller must wait before an LLM request can proceed.

func (*RateLimiter) Reset 

func (r *RateLimiter) Reset()

Reset clears all HTTP domain limiters and resets the LLM limiter. Useful for testing.

func (*RateLimiter) SetHTTPLimit 

func (r *RateLimiter) SetHTTPLimit(domain string, limit rate.Limit, burst int)

SetHTTPLimit sets a custom HTTP rate limit for a domain (for testing).

func (*RateLimiter) SetLLMLimit 

func (r *RateLimiter) SetLLMLimit(limit rate.Limit, burst int)

SetLLMLimit sets a custom LLM rate limit (for testing).

func (*RateLimiter) WaitHTTP 

func (r *RateLimiter) WaitHTTP(ctx context.Context, domain string) error

WaitHTTP waits for HTTP rate limit for a specific domain before proceeding. It blocks until a token is available or the context is cancelled. Returns ErrRateLimitExceeded if the context is cancelled while waiting.

func (*RateLimiter) WaitHTTPForURL 

func (r *RateLimiter) WaitHTTPForURL(ctx context.Context, rawURL string) error

WaitHTTPForURL waits for HTTP rate limit for a URL's domain before proceeding. It extracts the domain from the URL and applies rate limiting.

func (*RateLimiter) WaitLLM 

func (r *RateLimiter) WaitLLM(ctx context.Context) error

WaitLLM waits for LLM rate limit before proceeding. It blocks until a token is available or the context is cancelled. Returns ErrRateLimitExceeded if the context is cancelled while waiting.

type RateLimiterOption 

type RateLimiterOption func(*RateLimiter)

RateLimiterOption configures a RateLimiter

func WithCleanupAge 

func WithCleanupAge(d time.Duration) RateLimiterOption

WithCleanupAge sets the age after which an unused domain entry is eligible for eviction.

func WithClock 

func WithClock(clock Clock) RateLimiterOption

WithClock sets a custom clock for testing

func WithHTTPInterval 

func WithHTTPInterval(interval time.Duration, burst int) RateLimiterOption

WithHTTPInterval sets the fallback per-domain HTTP rate limit (minimum interval between requests and burst size) used for any host that has no specific policy. interval must be > 0 and burst >= 1; invalid values are ignored, preserving the DefaultHTTPInterval/DefaultHTTPBurst defaults.

func WithHostPolicy 

func WithHostPolicy(host string, interval time.Duration, burst int) RateLimiterOption

WithHostPolicy overrides the HTTP rate limit for a single host. It takes precedence over WithHTTPInterval (and the default) for that host. interval must be > 0 and burst >= 1; invalid values are ignored. A later call for the same host overwrites an earlier one.

func WithMaxDomains 

func WithMaxDomains(n int) RateLimiterOption

WithMaxDomains sets the maximum number of HTTP domain limiters to keep in memory.

func WithTunedHostPolicies 

func WithTunedHostPolicies() RateLimiterOption

WithTunedHostPolicies loads the built-in per-host tuning (defaultHostPolicies), raising the rate limit for high-volume, high-headroom hosts (GitHub, GitLab) while leaving every other host at the conservative fallback interval. Entries already set by an earlier WithHostPolicy are preserved (built-ins never overwrite an explicit override).

type RetryConfig 

type RetryConfig struct {
	// MaxRetries is the maximum number of retry attempts (default: 3)
	MaxRetries int
	// BaseDelay is the initial delay before first retry (default: 1s)
	BaseDelay time.Duration
	// MaxDelay is the maximum delay between retries (default: 4s)
	MaxDelay time.Duration
	// Timeout is the timeout for each individual request (default: 30s)
	Timeout time.Duration
}

RetryConfig holds configuration for retry behavior.

func DefaultRetryConfig 

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns the default retry configuration. Uses exponential backoff with delays of 1s, 2s, 4s.

type RetryableHTTPClient 

type RetryableHTTPClient struct {
	// contains filtered or unexported fields
}

RetryableHTTPClient wraps an HTTP client with retry logic and an optional circuit breaker. It implements exponential backoff for failed requests and prevents cascading failures via a circuit breaker that opens after repeated failures.

func NewRetryableHTTPClient 

func NewRetryableHTTPClient() *RetryableHTTPClient

NewRetryableHTTPClient creates a new HTTP client with retry support and circuit breaker. Uses the default retry configuration.

func NewRetryableHTTPClientWithConfig 

func NewRetryableHTTPClientWithConfig(config RetryConfig) *RetryableHTTPClient

NewRetryableHTTPClientWithConfig creates a new HTTP client with custom retry configuration. The circuit breaker is enabled by default.

func (*RetryableHTTPClient) ClearRecordedDelays 

func (c *RetryableHTTPClient) ClearRecordedDelays()

ClearRecordedDelays clears the recorded delays.

func (*RetryableHTTPClient) Config 

func (c *RetryableHTTPClient) Config() RetryConfig

Config returns the current retry configuration.

func (*RetryableHTTPClient) Do 

func (c *RetryableHTTPClient) Do(req *http.Request) (*http.Response, error)

Do executes an HTTP request with retry logic. It retries on network errors and 5xx server errors with exponential backoff. Returns the response and any error encountered after all retries are exhausted.

func (*RetryableHTTPClient) DoWithContext 

func (c *RetryableHTTPClient) DoWithContext(ctx context.Context, req *http.Request) (*http.Response, error)

DoWithContext executes an HTTP request with retry logic, context support, and circuit breaker protection. Each individual attempt is wrapped by the circuit breaker so that consecutive failures cause the circuit to open and subsequent requests fail fast.

func (*RetryableHTTPClient) Download added in v0.36.0 

func (c *RetryableHTTPClient) Download(ctx context.Context, url string) (*http.Response, error)

Download performs one GET for a file whose transfer may outlast the per-request timeout: http.Client.Timeout also bounds reading the body, so a distfile larger than http_timeout allows would be cut off mid-transfer. The request runs on a copy of the client with that timeout removed, so ctx alone bounds the transfer; the transport's header wait still bounds a silent server. It neither retries nor goes through the circuit breakers, and it sends the client's default headers. The caller closes the body.

func (*RetryableHTTPClient) DownloadRequest added in v0.36.0 

func (c *RetryableHTTPClient) DownloadRequest(req *http.Request) (*http.Response, error)

DownloadRequest is Download for a request the caller built, so it can carry headers of its own; the request's context bounds the transfer. A default header the request already sets is left as the caller set it.

func (*RetryableHTTPClient) Get 

func (c *RetryableHTTPClient) Get(ctx context.Context, url string) (*http.Response, error)

Get performs an HTTP GET request with retry logic. The request is bound to ctx, so cancelling it aborts the attempt in flight and any retry wait.

func (*RetryableHTTPClient) GetDefaultHeaders 

func (c *RetryableHTTPClient) GetDefaultHeaders() map[string]string

GetDefaultHeaders returns the configured default headers.

func (*RetryableHTTPClient) GetGitHubToken 

func (c *RetryableHTTPClient) GetGitHubToken() string

GetGitHubToken returns the configured GitHub token.

func (*RetryableHTTPClient) GetRecordedDelays 

func (c *RetryableHTTPClient) GetRecordedDelays() []time.Duration

GetRecordedDelays returns the delays that were recorded during requests. Only populated when using a custom delay function that records delays.

func (*RetryableHTTPClient) GetWithContext 

func (c *RetryableHTTPClient) GetWithContext(ctx context.Context, url string) (*http.Response, error)

GetWithContext performs an HTTP GET request with retry logic and context support.

The returned response body is wrapped in an http.MaxBytesReader bounded by httpx.MaxBodyBytes (10 MiB). A subsequent read that exceeds the cap yields an *http.MaxBytesError; callers should pass such read errors through classifyBodyReadError so the overflow surfaces as ErrResponseTooLarge.

func (*RetryableHTTPClient) GetWithHeaders 

func (c *RetryableHTTPClient) GetWithHeaders(url string, headers map[string]string) (*http.Response, error)

GetWithHeaders performs an HTTP GET request with custom headers and retry logic. Headers are processed for environment variable substitution using ${VAR_NAME} syntax. If the URL is a GitHub API URL and a GitHub token is configured, it will be included.

Callers that need cancellation should use GetWithHeadersContext directly.

func (*RetryableHTTPClient) GetWithHeadersContext 

func (c *RetryableHTTPClient) GetWithHeadersContext(ctx context.Context, url string, headers map[string]string) (*http.Response, error)

GetWithHeadersContext performs an HTTP GET request with custom headers, context, and retry logic. Headers are processed for environment variable substitution using ${VAR_NAME} syntax. If the URL is a GitHub API URL and a GitHub token is configured, it will be included.

The returned response body is wrapped in an http.MaxBytesReader bounded by httpx.MaxBodyBytes (10 MiB), exactly as GetWithContext does. A subsequent read that exceeds the cap yields an *http.MaxBytesError; callers should pass such read errors through classifyBodyReadError so the overflow surfaces as ErrResponseTooLarge. The cap holds even when a caller sent a Range header: a Range is only a request, and a server that ignores it and streams the full body is bounded here rather than at the caller's read.

A header that references a credential variable bound to another host is refused before any network I/O with an error wrapping ErrCredentialHostMismatch. With no package to consult, the request URL's own host is taken as the package host for BENTOO_* variables.

func (*RetryableHTTPClient) GetWithHeadersScopedContext 

func (c *RetryableHTTPClient) GetWithHeadersScopedContext(ctx context.Context, url string, headers map[string]string, scope CredentialScope) (*http.Response, error)

GetWithHeadersScopedContext is GetWithHeadersContext with the package's credential scope supplied by the caller (the checker passes the hosts of the package's url and base_url).

func (*RetryableHTTPClient) SetDefaultHeaders 

func (c *RetryableHTTPClient) SetDefaultHeaders(headers map[string]string)

SetDefaultHeaders sets default headers that will be applied to all requests. These headers are applied before any request-specific headers. Provided headers are merged into existing defaults so built-in headers (for example User-Agent) are preserved unless explicitly overridden.

func (*RetryableHTTPClient) SetDelayFunc 

func (c *RetryableHTTPClient) SetDelayFunc(fn func(time.Duration))

SetDelayFunc sets a custom delay function (useful for testing). The function receives the delay duration that would normally be waited, and runs in place of the wait; the context is checked once it returns.

func (*RetryableHTTPClient) SetGitHubToken 

func (c *RetryableHTTPClient) SetGitHubToken(token string)

SetGitHubToken sets the GitHub API token for authentication. When set, requests to GitHub API will include the Authorization header.

func (*RetryableHTTPClient) SetHTTP1FallbackClient 

func (c *RetryableHTTPClient) SetHTTP1FallbackClient(client *http.Client)

SetHTTP1FallbackClient sets the client used for the HTTP/1.1 fallback retry. Passing nil disables the fallback.

func (*RetryableHTTPClient) SetHTTPClient 

func (c *RetryableHTTPClient) SetHTTPClient(client *http.Client)

SetHTTPClient sets a custom underlying HTTP client (useful for testing).

The HTTP/1.1 fallback is disabled by this call: a caller that supplies its own client (a test transport, a stub, a proxy-aware client) owns the transport entirely, and silently re-issuing a request through a bentoolkit-built transport would bypass it. Re-enable the fallback explicitly with SetHTTP1FallbackClient if it is wanted.

func (*RetryableHTTPClient) SetJitterFunc 

func (c *RetryableHTTPClient) SetJitterFunc(fn func(ceiling time.Duration) time.Duration)

SetJitterFunc replaces the source that draws each retry wait from [0, ceiling] (useful for testing). Passing nil restores defaultJitter.

func (*RetryableHTTPClient) SetLogger 

func (c *RetryableHTTPClient) SetLogger(l *slog.Logger)

SetLogger sets the logger the client reports its diagnostics to. Passing nil discards them.

func (*RetryableHTTPClient) SetRequestTimeout 

func (c *RetryableHTTPClient) SetRequestTimeout(d time.Duration)

SetRequestTimeout sets the per-request timeout (the cap on a single attempt) on both the retry config and the underlying http.Client. Because the standard library resets this timer on every Do call, each retry attempt gets a fresh budget — so as long as the caller's per-operation deadline is larger than this value, the retry loop can actually run its attempts instead of the first slow request consuming the whole operation budget. A non-positive duration is ignored so callers can pass an unresolved value safely.

The header wait of each *http.Transport is raised to d when d exceeds httpx.DefaultResponseHeaderTimeout, and never lowered, so a user's larger http_timeout is not silently capped by the transport default. Other RoundTrippers, and the shared http.DefaultTransport, are left alone.

func (*RetryableHTTPClient) Wait added in v0.36.0 

func (c *RetryableHTTPClient) Wait(ctx context.Context, d time.Duration) error

Wait blocks for d the way the client waits between its own retries: the delay function set with SetDelayFunc runs in its place when there is one, and ctx's error is returned as soon as ctx is done.

func (*RetryableHTTPClient) WithCircuitBreaker 

func (c *RetryableHTTPClient) WithCircuitBreaker(enabled bool) *RetryableHTTPClient

WithCircuitBreaker enables or disables the per-host circuit breakers on this client. Either way every breaker built so far is dropped, so enabling starts each host from a closed breaker. Pass false to disable circuit breaker behavior entirely.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.