common

package
v0.0.11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 20, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MainComponent  = "main"
	DebugComponent = "debug"
)

Variables

View Source 
var (
	ErrEmptyPattern           = errors.New("empty pattern")
	ErrNoSegments             = errors.New("pattern must contain at least one segment (* or #)")
	ErrExpectedDelimiter      = errors.New("expected delimiter between segments")
	ErrAmountMismatch         = errors.New("amount count does not match tracked segment count")
	ErrVersionNotMatchPattern = errors.New("version does not match pattern")
	ErrNoMatchingPattern      = errors.New("item version does not match any retention pattern")
)

Errors

Functions 

func EnsureHardlink(src, dst string) error

EnsureHardlink creates a hardlink from src to dst with force behavior If dst exists and points to a different file, it will be removed first If dst exists and points to the same file (same inode), nothing is done This function is safe for concurrent use when multiple goroutines might try to create the same hardlink simultaneously.

func MatchesGlobPatterns 

func MatchesGlobPatterns(patterns []string, value string) bool

MatchesGlobPatterns checks if a value matches the given glob patterns. Empty patterns list means match all. Patterns support wildcards (* and ?). Patterns prefixed with ! are negations and exclude matching values. Negations are evaluated after positive matches.

func MkdirAll added in v0.0.11 

func MkdirAll(path string, perm os.FileMode) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. Unlike os.MkdirAll, it returns success if the path exists as a symlink to a directory.

Types

type CompressionFormat 

type CompressionFormat string

CompressionFormat represents a supported compression format 

const (
	CompressionNone  CompressionFormat = ""
	CompressionGzip  CompressionFormat = "gz"
	CompressionBzip2 CompressionFormat = "bz2"
	CompressionXZ    CompressionFormat = "xz"
)

func DetectCompressionFormat 

func DetectCompressionFormat(filename string) CompressionFormat

DetectCompressionFormat returns the compression format based on file extension

func (CompressionFormat) Extension 

func (f CompressionFormat) Extension() string

Extension returns the file extension for the compression format

type DeCompressResult 

type DeCompressResult string

DeCompressResult contains the outcome of a single download job

func (*DeCompressResult) Destination 

func (r *DeCompressResult) Destination() string

type DeCompressor 

type DeCompressor struct {
	// contains filtered or unexported fields
}

DeCompressor handles parallel compression/decompression operations

func NewDeCompressor 

func NewDeCompressor(pool pond.ResultPool[Result]) *DeCompressor

NewDeCompressor creates and initializes a new decompressor with the provided worker pool

func (*DeCompressor) Compress 

func (d *DeCompressor) Compress(ctx context.Context, sourcePath string, formats ...CompressionFormat) pond.ResultTaskGroup[Result]

Compress compresses a file into multiple formats in parallel using a task group Destination paths are automatically derived by appending the compression format extension. Returns a task group that can be waited on. Call Wait() to get results and any error. This method is thread-safe and can be called concurrently.

func (*DeCompressor) Decompress 

func (d *DeCompressor) Decompress(ctx context.Context, sourcePaths ...string) pond.ResultTaskGroup[Result]

Decompress decompresses one or more files in parallel using a task group Destination paths are automatically derived by removing the compression extension. Returns a task group that can be waited on. Call Wait() to get results and any error. This method is thread-safe and can be called concurrently.

type DownloadRequest 

type DownloadRequest struct {
	URL         string // URL to download
	Destination string // Full file path where file will be saved
	Checksum    string // Optional SHA256 checksum (hex-encoded) for verification during download
}

DownloadRequest represents one or more files to download to a destination

type DownloadResult 

type DownloadResult struct {
	*DownloadRequest       // The request that was downloaded
	Size             int64 // Bytes downloaded
}

DownloadResult contains the outcome of a single download job

func (*DownloadResult) Destination 

func (d *DownloadResult) Destination() string

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader handles parallel downloads with configurable settings

func NewDownloader 

func NewDownloader(pool pond.ResultPool[Result], httpClient *http.Client, decompressor *DeCompressor) *Downloader

NewDownloader creates and initializes a new download manager with the provided worker pool

func (*Downloader) Download 

func (m *Downloader) Download(ctx context.Context, requests ...*DownloadRequest) pond.ResultTaskGroup[Result]

Download downloads one or more files in parallel using a task group Returns a task group that can be waited on. Call Wait() to get results and any error. This method is thread-safe and can be called concurrently. If multiple goroutines request the same destination file with the same URL and checksum, only one download occurs and all waiters share the result.

func (*Downloader) DownloadAndDecompress 

func (m *Downloader) DownloadAndDecompress(ctx context.Context, requests ...*DownloadRequest) pond.ResultTaskGroup[Result]

DownloadAndDecompress downloads compressed files and automatically decompresses them The compression format is detected from the file extension (.gz, .xz, .bz2). Original compressed files are kept. Returns results with paths pointing to decompressed files. Returns an error if the file has an unsupported or no compression extension. This method is thread-safe and can be called concurrently.

type FileForTrust 

type FileForTrust struct {
	Path         string
	Distribution string
	Hash         string // SHA256 hash
	Source       string // Source package name for grouping
	Redirect     string // Relative redirect suffix (original file source) relative to the feed base URL
}

FileForTrust represents a file to be moved to trusted storage with its metadata

type GenericRetentionCollector 

type GenericRetentionCollector[T any] struct {
	// contains filtered or unexported fields
}

GenericRetentionCollector collects items with retention filtering per source package Structure: dist -> component -> packageName -> arch -> RetentionFilter Thread-safe for concurrent Add() and read operations

func NewGenericRetentionCollector 

func NewGenericRetentionCollector[T any](
	retentionPolicies []RetentionPolicy,
	getMetadata func(T) (string, string, string, string),
) *GenericRetentionCollector[T]

NewGenericRetentionCollector creates a new collector with retention policies getMetadata should return: sourceName, packageName, arch, version Always uses NoMatchKeep behavior for items that don't match any retention pattern

func NewPackageRetentionCollector 

func NewPackageRetentionCollector(
	retentionPolicies []RetentionPolicy,
) *GenericRetentionCollector[*deb.Package]

NewPackageRetentionCollector creates a collector for *deb.Package items Package grouping: by package name and architecture Always uses NoMatchKeep behavior

func (*GenericRetentionCollector[T]) Add 

func (c *GenericRetentionCollector[T]) Add(dist, component string, item T) error

Add adds an item to the collector with retention filtering for a specific component Thread-safe for concurrent calls

func (*GenericRetentionCollector[T]) ForEachKept 

func (c *GenericRetentionCollector[T]) ForEachKept(fn func(dist, component, packageName, arch string, item T) error) error

ForEachKept iterates through all kept items with their key information The callback receives: dist, component, packageName, arch, and the item Thread-safe for concurrent access

func (*GenericRetentionCollector[T]) Kept 

func (c *GenericRetentionCollector[T]) Kept() ([]T, error)

Kept returns all items that passed both source filtering and retention policies Thread-safe for concurrent access

type NoMatchBehavior 

type NoMatchBehavior int

NoMatchBehavior defines how to handle items that don't match any retention pattern 

const (
	NoMatchKeep   NoMatchBehavior = iota // Keep items that don't match any pattern (default, safe)
	NoMatchIgnore                        // Ignore/drop items that don't match any pattern
	NoMatchError                         // Return error when item doesn't match any pattern
)

type PackageOptions 

type PackageOptions struct {
	// Primary if set indicates the primary package to use for distribution sorting
	Primary string `yaml:"primary,omitempty"`
	// Debug indicates whether to include debug packages
	Debug bool `yaml:"debug"`
	// Source indicates whether to include source packages
	Source bool `yaml:"source"`
}

PackageOptions controls which package types to include.

type RepositoryOptions added in v0.0.4 

type RepositoryOptions struct {
	// Packages controls which package types are included
	Packages PackageOptions `yaml:"packages,omitempty"`
	// Distributions to fetch and process
	Distributions []string `yaml:"distributions,omitempty"`
	// Architectures to fetch and process
	Architectures []string `yaml:"architectures,omitempty"`
	// Retention policies for version filtering - which versions to keep
	Retention []RetentionPolicy `yaml:"retention,omitempty"`
}

RepositoryConfig options which can be relevant for feeds to download only requested packages

type Result 

type Result interface {
	Destination() string
}

type RetentionFilter 

type RetentionFilter[T any] struct {
	// contains filtered or unexported fields
}

RetentionFilter applies retention rules to items. Thread-safe for concurrent Add() and Kept().

func NewRetentionFilter 

func NewRetentionFilter[T any](rules []RetentionRule, getVersion func(T) string, noMatchBehavior NoMatchBehavior) (*RetentionFilter[T], error)

NewRetentionFilter creates a retention filter

func (*RetentionFilter[T]) Add 

func (f *RetentionFilter[T]) Add(item T) error

Add adds an item. Thread-safe.

func (*RetentionFilter[T]) Filter 

func (f *RetentionFilter[T]) Filter(items []T) ([]T, error)

Filter returns items to keep based on retention rules

func (*RetentionFilter[T]) Kept 

func (f *RetentionFilter[T]) Kept() ([]T, error)

Kept returns filtered items from Add() calls. Thread-safe.

type RetentionPolicy 

type RetentionPolicy struct {
	RetentionRule `yaml:",inline"`
	FromSources   []string `yaml:"from_sources,omitempty"` // Optional: source name patterns (supports glob), empty = applies to all sources
}

RetentionPolicy defines retention rules with optional source filtering

type RetentionRule 

type RetentionRule struct {
	Pattern string `yaml:"pattern"`
	Amount  []int  `yaml:"amount"`
}

RetentionRule defines pattern-based version retention policy

func FilterBySource 

func FilterBySource(policies []RetentionPolicy, sourceName string) []RetentionRule

FilterBySource returns rules matching sourceName. Empty FromSources matches all. Supports glob patterns (* and ?) with ! prefix for negations (evaluated after positive matches).

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage handles file storage and downloads in downloads/, trusted/, and public/ directories

func NewStorage 

func NewStorage(downloader *Downloader, downloadDir, trustedDir string, pathParts ...string) *Storage

NewStorage creates a new storage manager downloadDir and trustedDir should be absolute paths to the base directories

func (*Storage) Download 

func (m *Storage) Download(ctx context.Context, requests ...*DownloadRequest) pond.ResultTaskGroup[Result]

Download downloads files to the downloads directory (destinations are relative paths)

func (*Storage) DownloadAndDecompress 

func (m *Storage) DownloadAndDecompress(ctx context.Context, requests ...*DownloadRequest) pond.ResultTaskGroup[Result]

DownloadAndDecompress downloads and decompresses files based on extension (destinations are relative paths)

func (*Storage) FileExistsOrDownload 

func (m *Storage) FileExistsOrDownload(ctx context.Context, hashMethod, expectedHash, downloadURL string, pathParts ...string) (string, error)

FileExistsOrDownload returns path to file with expected hash, downloading if necessary

func (*Storage) GetDownloadPath 

func (m *Storage) GetDownloadPath(pathParts ...string) string

GetDownloadPath returns the full path for a file in the download directory

func (*Storage) GetTrustedPath 

func (m *Storage) GetTrustedPath(pathParts ...string) string

GetTrustedPath returns the full path for a file in the trusted directory (public method)

func (*Storage) LinkFilesToTrusted 

func (m *Storage) LinkFilesToTrusted(ctx context.Context, files []*FileForTrust) error

LinkFilesToTrusted creates hardlinks from downloads to trusted with distribution-based organization

func (*Storage) Scope 

func (m *Storage) Scope(pathParts ...string) *Storage

Scope creates a new Storage instance scoped to additional path parts

func (*Storage) UncompressedFileExistsOrDownloadAndDecompress 

func (m *Storage) UncompressedFileExistsOrDownloadAndDecompress(ctx context.Context, hashMethod, uncompressedHash, compressedHash, downloadURL string, compressionFormat CompressionFormat, pathParts ...string) (string, error)

UncompressedFileExistsOrDownloadAndDecompress returns path to uncompressed file with expected hash, downloading and decompressing if necessary

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.