mirror

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2026 License: GPL-3.0 Imports: 30 Imported by: 0

Documentation

Overview

Package mirror provides selective package mirroring for pre-populating the proxy cache.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ParsePlatforms added in v0.10.0

func ParsePlatforms(values []string) ([]handler.Platform, error)

ParsePlatforms parses os/architecture[/variant] values.

Types

type BottleResolver added in v0.11.0

type BottleResolver interface {
	BottleImage(ctx context.Context, formula, version string) (handler.ImageRef, error)
}

BottleResolver resolves a Homebrew formula version to the image its bottles are published under.

type ChartMirrorer added in v0.11.0

type ChartMirrorer interface {
	MirrorChart(ctx context.Context, repository, chart, version string) (*handler.CacheResult, error)
}

ChartMirrorer caches Helm charts from the configured repositories the way the Helm handler serves them.

type ImageMirrorer added in v0.10.0

type ImageMirrorer interface {
	MirrorImage(ctx context.Context, ref handler.ImageRef, platforms []handler.Platform) (handler.ImageResult, error)
}

ImageMirrorer fetches container images into the cache.

type Job

type Job struct {
	ID        string    `json:"id"`
	State     JobState  `json:"state"`
	Progress  Progress  `json:"progress"`
	CreatedAt time.Time `json:"created_at"`
	Error     string    `json:"error,omitempty"`
	// contains filtered or unexported fields
}

Job represents an async mirror operation.

type JobRequest

type JobRequest struct {
	PURLs    []string        `json:"purls,omitempty"`
	SBOM     json.RawMessage `json:"sbom,omitempty"`
	Registry string          `json:"registry,omitempty"`
	// Platforms limit which platforms of container images are mirrored,
	// as os/architecture[/variant].
	Platforms []string `json:"platforms,omitempty"`
	// Latest mirrors only the newest version of PURLs without a version.
	Latest bool `json:"latest,omitempty"`
}

JobRequest is the JSON body for starting a mirror job via the API.

type JobState

type JobState string

JobState represents the current state of a mirror job.

const (
	JobStatePending  JobState = "pending"
	JobStateRunning  JobState = "running"
	JobStateComplete JobState = "complete"
	JobStateFailed   JobState = "failed"
	JobStateCanceled JobState = "canceled"
)

type JobStore

type JobStore struct {
	// contains filtered or unexported fields
}

JobStore manages in-memory mirror jobs.

func NewJobStore

func NewJobStore(ctx context.Context, m *Mirror) *JobStore

NewJobStore creates a new job store. The parent context is used as the base for all job contexts so that jobs are canceled when the server shuts down.

func (*JobStore) Cancel

func (js *JobStore) Cancel(id string) bool

Cancel cancels a running job.

func (*JobStore) Cleanup

func (js *JobStore) Cleanup()

Cleanup removes completed/failed/canceled jobs older than jobTTL.

func (*JobStore) Create

func (js *JobStore) Create(req JobRequest) (string, error)

Create starts a new mirror job and returns its ID.

func (*JobStore) Get

func (js *JobStore) Get(id string) *Job

Get returns a snapshot of a job by ID. The returned copy is safe to serialize without holding the lock.

func (*JobStore) StartCleanup

func (js *JobStore) StartCleanup(ctx context.Context)

StartCleanup runs periodic cleanup of old jobs until the context is canceled.

func (*JobStore) Wait added in v0.11.0

func (js *JobStore) Wait()

Wait blocks until every job has returned. Cancel the parent context first to stop running jobs.

type LatestFinder added in v0.11.0

type LatestFinder interface {
	LatestVersion(ctx context.Context, ecosystem, name string) (string, error)
}

LatestFinder picks the latest version of a package, for unversioned PURLs when only the latest is mirrored. Listers without it fall back to registries.SelectLatestVersion over ListVersions.

type Mirror

type Mirror struct {

	// Images mirrors docker and oci items. Without it they fail like any
	// other unsupported ecosystem.
	Images ImageMirrorer

	// Charts mirrors helm charts. Without it they fail.
	Charts ChartMirrorer

	// Bottles resolves Homebrew formula versions to the images their
	// bottles are published under, which Images mirrors. Without it brew
	// PURLs fail.
	Bottles BottleResolver

	// Versions lists the versions of unversioned PURLs in mirror jobs and
	// schedules. Nil lists them from the public registries.
	Versions VersionLister

	// JuliaUpstream is the Pkg server Julia packages are fetched from. Empty
	// uses https://pkg.julialang.org.
	JuliaUpstream string

	// RefusePackages, when set, fails every item Images does not take with
	// this reason instead of fetching it, so nothing is cached.
	RefusePackages string
	// contains filtered or unexported fields
}

Mirror pre-populates the proxy cache from various input sources.

func New

func New(proxy *handler.Proxy, db *database.DB, store storage.Storage, logger *slog.Logger, workers int) *Mirror

New creates a new Mirror with the given dependencies.

func (*Mirror) Run

func (m *Mirror) Run(ctx context.Context, source Source, onProgress ...ProgressFunc) (*Progress, error)

Run mirrors all packages from the source using a bounded worker pool. It returns the final progress when complete. If onProgress is non-nil, it is called with progress snapshots as work proceeds.

func (*Mirror) RunDryRun

func (m *Mirror) RunDryRun(ctx context.Context, source Source) ([]PackageVersion, error)

RunDryRun enumerates what would be mirrored without downloading.

type MirrorError

type MirrorError struct {
	Ecosystem string `json:"ecosystem"`
	Name      string `json:"name"`
	Version   string `json:"version"`
	Error     string `json:"error"`
}

MirrorError records a single failed mirror attempt.

type PURLSource

type PURLSource struct {
	PURLs []string
	// Versions lists the versions of unversioned PURLs. Nil lists them from
	// the public registries.
	Versions VersionLister
	// Latest mirrors only the newest version of unversioned PURLs.
	Latest bool
	// Platforms limit which platforms of container images are mirrored.
	Platforms []string
}

PURLSource yields packages from PURL strings. Versioned PURLs produce a single item. Unversioned PURLs produce one item per version Versions lists, or only the newest with Latest.

func (*PURLSource) Enumerate

func (s *PURLSource) Enumerate(ctx context.Context, fn func(PackageVersion) error) error

type PackageFinder added in v0.11.0

type PackageFinder interface {
	FindPackage(ctx context.Context, ecosystem, name string) (*registries.Package, error)
}

PackageFinder finds a package's registry entry, which for Julia gives the UUID the package is registered under.

type PackageVersion

type PackageVersion struct {
	Ecosystem string
	Name      string
	Version   string
	// Qualifiers are the PURL qualifiers. Images use repository_url, tag and
	// arch; PyPI uses file_name, and conda subdir, build and type, to select
	// which files of the version are mirrored. Julia uses uuid to identify
	// the package.
	Qualifiers map[string]string
	// Platforms limit which platforms of an image index are mirrored.
	Platforms []string
}

PackageVersion identifies a specific package version to mirror.

func (PackageVersion) String

func (pv PackageVersion) String() string

type Progress

type Progress struct {
	Total     int64         `json:"total"`
	Completed int64         `json:"completed"`
	Skipped   int64         `json:"skipped"`
	Failed    int64         `json:"failed"`
	Bytes     int64         `json:"bytes"`
	Errors    []MirrorError `json:"errors,omitempty"`
	StartedAt time.Time     `json:"started_at"`
	Phase     string        `json:"phase"`
}

Progress tracks the state of a mirror operation.

type ProgressFunc

type ProgressFunc func(Progress)

ProgressFunc is called periodically with a snapshot of the current progress.

type RegistryVersions added in v0.11.0

type RegistryVersions struct {
	// Client makes the requests. Nil uses registries.DefaultClient.
	Client *registries.Client
	// URL returns the registry base URL to list a package from, by PURL
	// type. Nil, or an empty URL, uses the public registry.
	URL func(purlType, name string) (string, error)
}

RegistryVersions lists versions through the registries clients.

func (RegistryVersions) FindPackage added in v0.11.0

func (r RegistryVersions) FindPackage(ctx context.Context, ecosystem, name string) (*registries.Package, error)

FindPackage implements PackageFinder.

func (RegistryVersions) FindVersion added in v0.11.0

func (r RegistryVersions) FindVersion(ctx context.Context, ecosystem, name, version string) (registries.Version, error)

FindVersion implements VersionFinder.

func (RegistryVersions) LatestVersion added in v0.11.0

func (r RegistryVersions) LatestVersion(ctx context.Context, ecosystem, name string) (string, error)

LatestVersion implements LatestFinder with registries.FetchLatestVersion: the version the registry advertises as latest where it has one, such as npm's dist-tags.latest, and otherwise the newest version by publish time that isn't yanked, retracted or deprecated.

func (RegistryVersions) ListVersions added in v0.11.0

func (r RegistryVersions) ListVersions(ctx context.Context, ecosystem, name string) ([]string, error)

ListVersions implements VersionLister.

type SBOMSource

type SBOMSource struct {
	Data []byte
	Name string
	// Versions lists the versions of unversioned PURLs. Nil lists them from
	// the public registries.
	Versions VersionLister
	// Latest mirrors only the newest version of unversioned PURLs.
	Latest bool
	// Platforms limit which platforms of container images are mirrored.
	Platforms []string
}

SBOMSource extracts package versions from CycloneDX or SPDX SBOM data.

func (*SBOMSource) Enumerate

func (s *SBOMSource) Enumerate(ctx context.Context, fn func(PackageVersion) error) error

type Schedule added in v0.10.0

type Schedule struct {
	PURLs     []string
	Platforms []string
	Interval  time.Duration
}

Schedule mirrors a set of image PURLs every Interval.

type Scheduler added in v0.10.0

type Scheduler struct {

	// Claim, when set, reports whether this instance should run a schedule
	// that no instance sharing the database started within minGap. Each
	// instance keeps its own timer, so without it every instance would run
	// every schedule.
	Claim func(key string, minGap time.Duration) (bool, error)
	// Release, when set, drops the claim of a run that was interrupted, so
	// the next attempt runs it rather than waiting an interval.
	Release func(key string) error
	// contains filtered or unexported fields
}

Scheduler runs schedules in the background until its context ends.

func NewScheduler added in v0.10.0

func NewScheduler(m *Mirror, schedules []Schedule, logger *slog.Logger) *Scheduler

NewScheduler creates a scheduler that runs the schedules through m.

func (*Scheduler) Start added in v0.10.0

func (s *Scheduler) Start(ctx context.Context)

Start starts one goroutine per schedule.

func (*Scheduler) Wait added in v0.10.0

func (s *Scheduler) Wait()

Wait blocks until every schedule has stopped after its context ended.

type Source

type Source interface {
	Enumerate(ctx context.Context, fn func(PackageVersion) error) error
}

Source produces PackageVersion items for mirroring.

type UnavailableImages added in v0.10.0

type UnavailableImages struct {
	Reason string
}

UnavailableImages refuses every image with the given reason.

func (UnavailableImages) MirrorImage added in v0.10.0

type VersionFinder added in v0.11.0

type VersionFinder interface {
	FindVersion(ctx context.Context, ecosystem, name, version string) (registries.Version, error)
}

VersionFinder finds one version of a package, with the files it publishes, for ecosystems the mirror fetches from the version's metadata.

type VersionLister added in v0.11.0

type VersionLister interface {
	ListVersions(ctx context.Context, ecosystem, name string) ([]string, error)
}

VersionLister lists the versions of a package, for PURLs without one.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL