Documentation
¶
Overview ¶
Package httpx is the shared HTTP layer for every outbound request.
It provides the token bucket (golang.org/x/time/rate, per ADR-004), retry with backoff on 429 and 5xx, and an on-disk response cache so that re-running an expansion does not re-pay for work already done.
Every package under sources/ dials through here, and nothing else makes a raw http.Client call. A limiter that only some callers respect is how you get rate limited on somebody else's free tier.
Build one Client per upstream service, once, and share it: each service has its own rate limit, and a limiter is only a limit if every request to that service waits on the same one.
The response cache is a directory of files, not the separate SQLite file ADR-004 proposed. See DirCache for why.
Index ¶
Constants ¶
const ( // DefaultMaxRetries is §7's "retry three times, then mark the batch failed // and continue". DefaultMaxRetries = 3 // DefaultMaxRetryWait is the longest Retry-After this client will sleep // through. A 429 asking for longer is not a burst to wait out: it is the // daily allowance spent, with a reset hours away, and blocking a CLI for // that long looks like a hang. The request fails as transient instead, with // the wait attached for the caller to report. DefaultMaxRetryWait = 60 * time.Second // DefaultTimeout bounds one attempt, not the whole call with its retries. DefaultTimeout = 30 * time.Second // DefaultCacheMaxAge is how long a cached response is served. The cache // exists so a debugging loop does not refetch (ADR-004), not to be a // mirror; a week keeps citation counts from drifting far. DefaultCacheMaxAge = 7 * 24 * time.Hour )
Variables ¶
This section is empty.
Functions ¶
func RetryAfter ¶
RetryAfter returns the server's requested wait carried by err, if any.
The errs package deliberately keeps this off ErrTransient: a caller deciding whether to retry should not have to understand when. This is the when, for the one caller that reports it — "OpenAlex asked us to wait 6h; your daily allowance is spent".
func StatusCode ¶
StatusCode returns the HTTP status carried by err, or 0 if there is none.
Types ¶
type Cache ¶
type Cache interface {
// Get returns the body stored for key, if there is one no older than
// maxAge. A maxAge of zero means any age.
Get(key string, maxAge time.Duration) (body []byte, ok bool, err error)
Put(key string, body []byte) error
}
Cache stores successful response bodies by request key, so that re-running an expansion while debugging does not re-pay for requests already made (ADR-004).
A cache must never be the reason a request fails. The client treats a Get error as a miss and ignores a Put error: a full disk or an unreadable file costs a refetch, not a failed expansion.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is the only way anything in Filiation makes an HTTP request. See the package documentation for why.
It is safe for concurrent use: the limiter is shared, so concurrent callers queue on the bucket rather than multiplying the rate.
func (*Client) Get ¶
Get fetches rawURL and returns the response body.
A 2xx is returned, and cached if a cache is set. A 429, any 5xx or a network failure is retried with backoff, and if it never succeeds the error wraps errs.ErrTransient — the expander's signal to skip the batch and carry on (§7). Any other status fails at once with a *StatusError, which a source inspects: a 404 from OpenAlex means the work is unresolved, not that the network is down.
Cancelling ctx stops the call at the next wait, whether that is the token bucket, a backoff or the request itself.
type DirCache ¶
type DirCache struct {
// contains filtered or unexported fields
}
DirCache is a Cache kept as one file per response under a directory.
ADR-004 specified a separate SQLite file. A directory was chosen instead, for two reasons. Every SQL statement in Filiation lives in internal/store, and a second database owned by this package would break that rule for a key-value lookup that needs no SQL. And every connection in the WASM driver is a separate instance, which is a real cost to pay for a cache. A directory keeps what the ADR wanted — the cache is separate from the library and can be deleted at any time without touching the user's data — and a person can inspect it with ls.
Each file starts with the key on its own line, then the body. The key is checked on read, so the file name being a hash is never trusted on its own.
func NewDirCache ¶
NewDirCache opens, creating if needed, a cache rooted at dir.
func (*DirCache) Clear ¶
Clear deletes every cached response. It is `fil cache clear`, which ADR-004 requires from day one: a stale cache can hide a bug, and the user needs a way to rule it out.
type Options ¶
type Options struct {
// RatePerSecond is the token bucket's rate. Required. For OpenAlex it is 5,
// against a measured first 429 at ~11 req/s (spike 3).
RatePerSecond float64
// Burst is the bucket size. Zero means 1, which spaces requests evenly
// rather than letting a burst run into the limit.
Burst int
// Contact is the user's email. Optional (D11 measured no benefit from it),
// but it is what OpenAlex asks for and how they reach a misbehaving client.
// It goes into the User-Agent, and into the query as ContactParam if that
// is set.
Contact string
// ContactParam is the query parameter the upstream wants the contact in:
// "mailto" for OpenAlex, "email" for Unpaywall. Empty sends it in the
// User-Agent only.
ContactParam string
// UserAgent names the tool. Empty means "fil".
UserAgent string
// MaxRetries is how many times a transient failure is retried. Zero means
// DefaultMaxRetries; negative means never retry.
MaxRetries int
// MaxRetryWait caps how long a Retry-After is honoured. Zero means
// DefaultMaxRetryWait.
MaxRetryWait time.Duration
// Timeout bounds each attempt. Zero means DefaultTimeout.
Timeout time.Duration
// Cache, if set, serves repeated GETs without a request. Nil disables
// caching, which is what --no-cache does.
Cache Cache
// CacheMaxAge is how old a cached response may be. Zero means
// DefaultCacheMaxAge.
CacheMaxAge time.Duration
// Transport replaces the network. Tests pass a stub here so that no unit
// test touches the network; production leaves it nil.
Transport http.RoundTripper
}
Options configures one Client. Each upstream service gets its own Client, because each has its own rate limit: a token bucket shared between OpenAlex and Unpaywall would throttle one for the other's sake.
type Quota ¶
type Quota struct {
Limit int
Remaining int
Reset time.Duration // until the allowance refills
Observed time.Time
}
Quota is the allowance the upstream last reported, from OpenAlex's X-RateLimit-* headers.
It is read from responses rather than compiled in because OpenAlex has already changed its pricing once: the documented allowance was 100,000 credits a day and the measured one is 1,000 (D11).
type StatusError ¶
type StatusError struct {
URL string // without the contact parameter
StatusCode int
// RetryAfter is the server's requested wait, zero if it sent none.
RetryAfter time.Duration
// Body is the start of the response, because OpenAlex explains a 400 in it
// — "101 IDs is over the limit of 100" is only ever said there.
Body string
}
StatusError is a response with a status other than 2xx.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
func (*StatusError) Is ¶
func (e *StatusError) Is(target error) bool
Is makes a 429 or 5xx match errs.ErrTransient, so callers can test for it without knowing this type exists.
func (*StatusError) Transient ¶
func (e *StatusError) Transient() bool
Transient reports whether this status may not recur: rate limited, or a server-side failure.
Source Files