fetch

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 30 Imported by: 1

Documentation

Overview

Package fetch provides streaming artifact downloading with retry, circuit breaking, and URL resolution for package registries.

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrNotFound     = errors.New("artifact not found")
	ErrRateLimited  = errors.New("rate limited by upstream")
	ErrUpstreamDown = errors.New("upstream registry unavailable")
)
View Source
var (
	ErrUnsupportedEcosystem = errors.New("unsupported ecosystem")
	ErrNoDownloadURL        = errors.New("no download URL available")
	ErrNoMatchingArtifact   = errors.New("no artifact matches the requested file")
	ErrUnsafeURL            = errors.New("unsafe download URL from registry metadata")
)
View Source
var ErrArtifactTooLarge = errors.New("artifact exceeds size limit")

ErrArtifactTooLarge reports a declared or streamed artifact over the limit.

Functions

This section is empty.

Types

type Artifact

type Artifact struct {
	Body        io.ReadCloser
	Size        int64 // -1 if unknown
	ContentType string
	ETag        string
}

Artifact contains the response from fetching an upstream artifact.

type ArtifactInfo

type ArtifactInfo struct {
	URL       string
	Filename  string
	Integrity string // sha256-... or sha512-...
	Size      int64  // Zero when the registry does not publish a size.
}

ArtifactInfo contains information about a downloadable artifact.

type CircuitBreakerFetcher

type CircuitBreakerFetcher struct {
	// contains filtered or unexported fields
}

CircuitBreakerFetcher wraps a Fetcher with per-registry circuit breakers.

func NewCircuitBreakerFetcher

func NewCircuitBreakerFetcher(f *Fetcher) *CircuitBreakerFetcher

NewCircuitBreakerFetcher creates a new circuit breaker wrapper for a fetcher.

func (*CircuitBreakerFetcher) Fetch

func (cbf *CircuitBreakerFetcher) Fetch(ctx context.Context, fetchURL string) (*Artifact, error)

Fetch wraps the underlying fetcher's Fetch with circuit breaker logic.

func (*CircuitBreakerFetcher) FetchObserved

func (cbf *CircuitBreakerFetcher) FetchObserved(ctx context.Context, fetchURL string) (*ObservedArtifact, error)

FetchObserved wraps the underlying fetcher's FetchObserved with circuit breaker logic.

func (*CircuitBreakerFetcher) FetchObservedWithHeaders

func (cbf *CircuitBreakerFetcher) FetchObservedWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*ObservedArtifact, error)

FetchObservedWithHeaders wraps the underlying fetcher's FetchObservedWithHeaders with circuit breaker logic.

func (*CircuitBreakerFetcher) FetchWithHeaders

func (cbf *CircuitBreakerFetcher) FetchWithHeaders(ctx context.Context, fetchURL string, headers http.Header) (*Artifact, error)

FetchWithHeaders wraps the underlying fetcher's FetchWithHeaders with circuit breaker logic.

func (*CircuitBreakerFetcher) GetBreakerState

func (cbf *CircuitBreakerFetcher) GetBreakerState() map[string]string

GetBreakerState returns the current state of circuit breakers (for health checks).

func (*CircuitBreakerFetcher) Head

func (cbf *CircuitBreakerFetcher) Head(ctx context.Context, headURL string) (size int64, contentType string, err error)

Head wraps the underlying fetcher's Head with circuit breaker logic.

type FetchObservation

type FetchObservation struct {
	RequestedURL string
	FinalURL     string
	ResponseTime time.Duration // Time until the final response headers arrived.
	StatusCode   int
	Headers      http.Header // Allow-listed response headers only.
	DeclaredSize int64
	MediaType    string
	ByteCount    int64
	Digests      map[string]string
	Complete     bool
}

FetchObservation contains metadata about a successful artifact response. ByteCount and Digests are populated, and Complete is set to true, only after the response body reaches EOF.

func (*FetchObservation) Artifact

func (observation *FetchObservation) Artifact(packageURL, filename string) (artifacts.Artifact, error)

Artifact converts a completed observation into a shared artifact value.

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher downloads artifacts from upstream registries.

func NewFetcher

func NewFetcher(opts ...Option) *Fetcher

NewFetcher creates a new Fetcher with the given options. Callers should invoke Close when done to release the DNS refresh goroutine. Under TinyGo, requests require WithHTTPClient.

func (*Fetcher) Close

func (f *Fetcher) Close() error

Close stops the Fetcher's background DNS refresh goroutine. It is safe to call Close more than once.

func (*Fetcher) Fetch

func (f *Fetcher) Fetch(ctx context.Context, url string) (*Artifact, error)

Fetch downloads an artifact from the given URL. The caller must close the returned Artifact.Body when done.

func (*Fetcher) FetchObserved

func (f *Fetcher) FetchObserved(ctx context.Context, url string) (*ObservedArtifact, error)

FetchObserved downloads an artifact and records metadata about the response. The caller must read the body to EOF before treating the observation as complete.

func (*Fetcher) FetchObservedWithHeaders

func (f *Fetcher) FetchObservedWithHeaders(ctx context.Context, url string, headers http.Header) (*ObservedArtifact, error)

FetchObservedWithHeaders downloads an artifact with additional HTTP headers and records metadata about the response. Request headers are not copied into the observation.

func (*Fetcher) FetchWithHeaders

func (f *Fetcher) FetchWithHeaders(ctx context.Context, url string, headers http.Header) (*Artifact, error)

FetchWithHeaders downloads an artifact from the given URL with additional HTTP headers. The caller must close the returned Artifact.Body when done.

func (*Fetcher) Head

func (f *Fetcher) Head(ctx context.Context, url string) (size int64, contentType string, err error)

Head checks if an artifact exists and returns its metadata without downloading.

type FetcherInterface

type FetcherInterface interface {
	Fetch(ctx context.Context, url string) (*Artifact, error)
	FetchWithHeaders(ctx context.Context, url string, headers http.Header) (*Artifact, error)
	Head(ctx context.Context, url string) (size int64, contentType string, err error)
}

FetcherInterface defines the interface for artifact fetchers.

type ObservedArtifact

type ObservedArtifact struct {
	*Artifact
	Observation *FetchObservation
}

ObservedArtifact contains an artifact and its fetch observation.

type Option

type Option func(*Fetcher)

Option configures a Fetcher.

func WithAllowPrivateHosts

func WithAllowPrivateHosts(hosts ...string) Option

WithAllowPrivateHosts permits the named hosts to resolve to private IP addresses. Loopback and link-local addresses remain blocked.

func WithAuthFunc

func WithAuthFunc(fn func(url string) (headerName, headerValue string)) Option

WithAuthFunc sets a function that returns auth headers for a given URL. The function receives the request URL and returns a header name and value. Return empty strings to skip authentication for that URL.

func WithBaseDelay

func WithBaseDelay(d time.Duration) Option

WithBaseDelay sets the base delay for exponential backoff.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient sets a custom HTTP client. To allow private hosts with a custom client, construct it with safehttp.New and the desired options before applying transport wrappers and passing it here.

func WithMaxArtifactSize added in v0.10.0

func WithMaxArtifactSize(maxSize int64) Option

WithMaxArtifactSize limits declared and streamed artifact bodies. Zero or negative values leave downloads unlimited. Observations remain incomplete when the limit is exceeded. HEAD requests still report the upstream size.

func WithMaxRetries

func WithMaxRetries(n int) Option

WithMaxRetries sets the maximum retry attempts.

func WithUserAgent

func WithUserAgent(ua string) Option

WithUserAgent sets the User-Agent header.

type Registry

type Registry interface {
	Ecosystem() string
	FetchVersions(ctx context.Context, name string) ([]registries.Version, error)
	URLs() client.URLBuilder
}

Registry provides package metadata and URL information for artifact resolution. This interface is satisfied by registries.Registry implementations.

type ResolveOptions

type ResolveOptions struct {
	Filename  string
	Integrity string
	// Qualifiers are PURL qualifiers. Those in selectionQualifiers pick among
	// the artifacts a registry lists for a version; an artifact that does not
	// record a qualifier is not ruled out by it. repository_url and channel
	// choose the registry, which is up to the caller, and are ignored here.
	Qualifiers map[string]string
}

ResolveOptions narrows versions that publish several artifact files.

type Resolver

type Resolver struct {
	// contains filtered or unexported fields
}

Resolver determines download URLs for package artifacts.

func NewResolver

func NewResolver(opts ...ResolverOption) *Resolver

NewResolver creates a new URL resolver.

func (*Resolver) RegisterRegistry

func (r *Resolver) RegisterRegistry(reg Registry)

RegisterRegistry adds a registry for URL resolution. Ecosystem names are matched by PURL type, so "rubygems" and "gem" are the same ecosystem.

func (*Resolver) Resolve

func (r *Resolver) Resolve(ctx context.Context, ecosystem, name, version string) (*ArtifactInfo, error)

Resolve returns the download URL and filename for a package artifact.

func (*Resolver) ResolveWithOptions

func (r *Resolver) ResolveWithOptions(
	ctx context.Context,
	ecosystem string,
	name string,
	version string,
	options ResolveOptions,
) (*ArtifactInfo, error)

ResolveWithOptions returns the download metadata matching filename and registry-native integrity constraints when they are populated.

type ResolverOption added in v0.11.0

type ResolverOption func(*Resolver)

ResolverOption configures a Resolver.

func WithAddressPolicy added in v0.11.1

func WithAddressPolicy(opts safehttp.Options) ResolverOption

WithAddressPolicy sets which addresses the hosts of download URLs taken from registry metadata may resolve to. Pass the same options as the safehttp client that downloads them, so the resolver and the client agree. The default is safehttp's strict policy.

func WithBaseURLs added in v0.11.0

func WithBaseURLs(baseURLs map[string]string) ResolverOption

WithBaseURLs overrides the base URL used for ecosystems resolved without a registered registry, keyed by ecosystem. Ecosystems not in the map keep the public registry host.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL