Documentation
¶
Overview ¶
Package fetch provides streaming artifact downloading with retry, circuit breaking, and URL resolution for package registries.
Index ¶
- Variables
- type Artifact
- type ArtifactInfo
- type CircuitBreakerFetcher
- func (cbf *CircuitBreakerFetcher) Fetch(ctx context.Context, fetchURL string) (*Artifact, error)
- func (cbf *CircuitBreakerFetcher) FetchObserved(ctx context.Context, fetchURL string) (*ObservedArtifact, error)
- func (cbf *CircuitBreakerFetcher) FetchObservedWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*ObservedArtifact, error)
- func (cbf *CircuitBreakerFetcher) FetchWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*Artifact, error)
- func (cbf *CircuitBreakerFetcher) GetBreakerState() map[string]string
- func (cbf *CircuitBreakerFetcher) Head(ctx context.Context, headURL string) (size int64, contentType string, err error)
- type FetchObservation
- type Fetcher
- func (f *Fetcher) Close() error
- func (f *Fetcher) Fetch(ctx context.Context, url string) (*Artifact, error)
- func (f *Fetcher) FetchObserved(ctx context.Context, url string) (*ObservedArtifact, error)
- func (f *Fetcher) FetchObservedWithHeaders(ctx context.Context, url string, headers http.Header) (*ObservedArtifact, error)
- func (f *Fetcher) FetchWithHeaders(ctx context.Context, url string, headers http.Header) (*Artifact, error)
- func (f *Fetcher) Head(ctx context.Context, url string) (size int64, contentType string, err error)
- type FetcherInterface
- type ObservedArtifact
- type Option
- func WithAllowPrivateHosts(hosts ...string) Option
- func WithAuthFunc(fn func(url string) (headerName, headerValue string)) Option
- func WithBaseDelay(d time.Duration) Option
- func WithHTTPClient(c *http.Client) Option
- func WithMaxArtifactSize(maxSize int64) Option
- func WithMaxRetries(n int) Option
- func WithUserAgent(ua string) Option
- type Registry
- type ResolveOptions
- type Resolver
- type ResolverOption
Constants ¶
This section is empty.
Variables ¶
var ( ErrNotFound = errors.New("artifact not found") ErrRateLimited = errors.New("rate limited by upstream") ErrUpstreamDown = errors.New("upstream registry unavailable") )
var ( ErrUnsupportedEcosystem = errors.New("unsupported ecosystem") ErrNoDownloadURL = errors.New("no download URL available") ErrNoMatchingArtifact = errors.New("no artifact matches the requested file") ErrUnsafeURL = errors.New("unsafe download URL from registry metadata") )
var ErrArtifactTooLarge = errors.New("artifact exceeds size limit")
ErrArtifactTooLarge reports a declared or streamed artifact over the limit.
Functions ¶
This section is empty.
Types ¶
type Artifact ¶
type Artifact struct {
Body io.ReadCloser
Size int64 // -1 if unknown
ContentType string
ETag string
}
Artifact contains the response from fetching an upstream artifact.
type ArtifactInfo ¶
type ArtifactInfo struct {
URL string
Filename string
Integrity string // sha256-... or sha512-...
Size int64 // Zero when the registry does not publish a size.
}
ArtifactInfo contains information about a downloadable artifact.
type CircuitBreakerFetcher ¶
type CircuitBreakerFetcher struct {
// contains filtered or unexported fields
}
CircuitBreakerFetcher wraps a Fetcher with per-registry circuit breakers.
func NewCircuitBreakerFetcher ¶
func NewCircuitBreakerFetcher(f *Fetcher) *CircuitBreakerFetcher
NewCircuitBreakerFetcher creates a new circuit breaker wrapper for a fetcher.
func (*CircuitBreakerFetcher) Fetch ¶
Fetch wraps the underlying fetcher's Fetch with circuit breaker logic.
func (*CircuitBreakerFetcher) FetchObserved ¶
func (cbf *CircuitBreakerFetcher) FetchObserved(ctx context.Context, fetchURL string) (*ObservedArtifact, error)
FetchObserved wraps the underlying fetcher's FetchObserved with circuit breaker logic.
func (*CircuitBreakerFetcher) FetchObservedWithHeaders ¶
func (cbf *CircuitBreakerFetcher) FetchObservedWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*ObservedArtifact, error)
FetchObservedWithHeaders wraps the underlying fetcher's FetchObservedWithHeaders with circuit breaker logic.
func (*CircuitBreakerFetcher) FetchWithHeaders ¶
func (cbf *CircuitBreakerFetcher) FetchWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*Artifact, error)
FetchWithHeaders wraps the underlying fetcher's FetchWithHeaders with circuit breaker logic.
func (*CircuitBreakerFetcher) GetBreakerState ¶
func (cbf *CircuitBreakerFetcher) GetBreakerState() map[string]string
GetBreakerState returns the current state of circuit breakers (for health checks).
type FetchObservation ¶
type FetchObservation struct {
RequestedURL string
FinalURL string
ResponseTime time.Duration // Time until the final response headers arrived.
StatusCode int
Headers http.Header // Allow-listed response headers only.
DeclaredSize int64
MediaType string
ByteCount int64
Digests map[string]string
Complete bool
}
FetchObservation contains metadata about a successful artifact response. ByteCount and Digests are populated, and Complete is set to true, only after the response body reaches EOF.
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher downloads artifacts from upstream registries.
func NewFetcher ¶
NewFetcher creates a new Fetcher with the given options. Callers should invoke Close when done to release the DNS refresh goroutine. Under TinyGo, requests require WithHTTPClient.
func (*Fetcher) Close ¶
Close stops the Fetcher's background DNS refresh goroutine. It is safe to call Close more than once.
func (*Fetcher) Fetch ¶
Fetch downloads an artifact from the given URL. The caller must close the returned Artifact.Body when done.
func (*Fetcher) FetchObserved ¶
FetchObserved downloads an artifact and records metadata about the response. The caller must read the body to EOF before treating the observation as complete.
func (*Fetcher) FetchObservedWithHeaders ¶
func (f *Fetcher) FetchObservedWithHeaders(ctx context.Context, url string, headers http.Header) (*ObservedArtifact, error)
FetchObservedWithHeaders downloads an artifact with additional HTTP headers and records metadata about the response. Request headers are not copied into the observation.
type FetcherInterface ¶
type FetcherInterface interface {
Fetch(ctx context.Context, url string) (*Artifact, error)
FetchWithHeaders(ctx context.Context, url string, headers http.Header) (*Artifact, error)
Head(ctx context.Context, url string) (size int64, contentType string, err error)
}
FetcherInterface defines the interface for artifact fetchers.
type ObservedArtifact ¶
type ObservedArtifact struct {
*Artifact
Observation *FetchObservation
}
ObservedArtifact contains an artifact and its fetch observation.
type Option ¶
type Option func(*Fetcher)
Option configures a Fetcher.
func WithAllowPrivateHosts ¶
WithAllowPrivateHosts permits the named hosts to resolve to private IP addresses. Loopback and link-local addresses remain blocked.
func WithAuthFunc ¶
WithAuthFunc sets a function that returns auth headers for a given URL. The function receives the request URL and returns a header name and value. Return empty strings to skip authentication for that URL.
func WithBaseDelay ¶
WithBaseDelay sets the base delay for exponential backoff.
func WithHTTPClient ¶
WithHTTPClient sets a custom HTTP client. To allow private hosts with a custom client, construct it with safehttp.New and the desired options before applying transport wrappers and passing it here.
func WithMaxArtifactSize ¶ added in v0.10.0
WithMaxArtifactSize limits declared and streamed artifact bodies. Zero or negative values leave downloads unlimited. Observations remain incomplete when the limit is exceeded. HEAD requests still report the upstream size.
func WithMaxRetries ¶
WithMaxRetries sets the maximum retry attempts.
type Registry ¶
type Registry interface {
Ecosystem() string
FetchVersions(ctx context.Context, name string) ([]registries.Version, error)
URLs() client.URLBuilder
}
Registry provides package metadata and URL information for artifact resolution. This interface is satisfied by registries.Registry implementations.
type ResolveOptions ¶
type ResolveOptions struct {
Filename string
Integrity string
// Qualifiers are PURL qualifiers. Those in selectionQualifiers pick among
// the artifacts a registry lists for a version; an artifact that does not
// record a qualifier is not ruled out by it. repository_url and channel
// choose the registry, which is up to the caller, and are ignored here.
Qualifiers map[string]string
}
ResolveOptions narrows versions that publish several artifact files.
type Resolver ¶
type Resolver struct {
// contains filtered or unexported fields
}
Resolver determines download URLs for package artifacts.
func NewResolver ¶
func NewResolver(opts ...ResolverOption) *Resolver
NewResolver creates a new URL resolver.
func (*Resolver) RegisterRegistry ¶
RegisterRegistry adds a registry for URL resolution. Ecosystem names are matched by PURL type, so "rubygems" and "gem" are the same ecosystem.
func (*Resolver) Resolve ¶
func (r *Resolver) Resolve(ctx context.Context, ecosystem, name, version string) (*ArtifactInfo, error)
Resolve returns the download URL and filename for a package artifact.
func (*Resolver) ResolveWithOptions ¶
func (r *Resolver) ResolveWithOptions( ctx context.Context, ecosystem string, name string, version string, options ResolveOptions, ) (*ArtifactInfo, error)
ResolveWithOptions returns the download metadata matching filename and registry-native integrity constraints when they are populated.
type ResolverOption ¶ added in v0.11.0
type ResolverOption func(*Resolver)
ResolverOption configures a Resolver.
func WithAddressPolicy ¶ added in v0.11.1
func WithAddressPolicy(opts safehttp.Options) ResolverOption
WithAddressPolicy sets which addresses the hosts of download URLs taken from registry metadata may resolve to. Pass the same options as the safehttp client that downloads them, so the resolver and the client agree. The default is safehttp's strict policy.
func WithBaseURLs ¶ added in v0.11.0
func WithBaseURLs(baseURLs map[string]string) ResolverOption
WithBaseURLs overrides the base URL used for ecosystems resolved without a registered registry, keyed by ecosystem. Ecosystems not in the map keep the public registry host.