download

package
v0.13.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 29, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AvailableSpace

func AvailableSpace(path string) (uint64, error)

AvailableSpace returns available bytes on the filesystem containing path.

func CheckDiskSpace

func CheckDiskSpace(dir string, neededBytes uint64, minFreeGB int) error

CheckDiskSpace verifies there's enough space for the download. It checks that available space on dir is at least neededBytes plus minFreeGB gigabytes.

func FormatBytes

func FormatBytes(bytes uint64) string

FormatBytes formats bytes as a human-readable string using 1024-based units.

Types

type Config

type Config struct {
	OutputDir      string
	Parallel       int
	CheckDiskSpace bool
	MinFreeSpaceGB int
	MaxRetries     int
}

Config holds configuration for the Downloader.

type DownloadError

type DownloadError struct {
	URL   string
	Error string
}

DownloadError records which URL failed and why.

type Downloader

type Downloader struct {

	// OnProgress is called periodically during downloads if set.
	// Arguments: filename, bytesWritten, totalBytes (-1 if unknown).
	OnProgress func(filename string, written, total int64)
	// contains filtered or unexported fields
}

Downloader manages parallel HTTP downloads with resume and retry support.

func New

func New(cfg Config) *Downloader

New creates a Downloader using cfg. Defaults are applied for zero values.

func (*Downloader) DownloadAll

func (d *Downloader) DownloadAll(urls []string) Result

DownloadAll downloads all urls in parallel using a goroutine pool limited by cfg.Parallel. It returns a Result with counts and any per-URL errors.

func (*Downloader) DownloadAllFiles

func (d *Downloader) DownloadAllFiles(tasks []FileTask) Result

DownloadAllFiles downloads tasks in parallel. Each FileTask carries a URL, filename, and optional MD5 for verification. This is the preferred method when callers have file metadata (e.g., from an index).

func (*Downloader) DownloadFile

func (d *Downloader) DownloadFile(url, filename string) error

DownloadFile fetches url and saves it as filename inside the output directory. It skips the download when the final file already exists, and resumes from a .part file when one is present. Retries are performed with exponential backoff for 429 and 5xx responses; 4xx (except 429) fail immediately.

func (*Downloader) DownloadFileVerified

func (d *Downloader) DownloadFileVerified(url, filename, expectedMD5 string) error

DownloadFileVerified is like DownloadFile but also verifies the MD5 checksum of the downloaded file when expectedMD5 is non-empty. If verification fails, the file is removed and an error is returned.

func (*Downloader) WriteManifest

func (d *Downloader) WriteManifest(queryDesc string, result Result) error

WriteManifest writes a manifest.json file summarising the download batch.

type FileTask

type FileTask struct {
	URL      string
	Filename string // saved as this name; falls back to filepath.Base(URL)
	MD5      string // expected MD5 hex digest; empty to skip verification
}

FileTask describes a single file to download, with optional metadata for integrity verification and proper naming.

type Manifest

type Manifest struct {
	Query      string    `json:"query"`
	Timestamp  time.Time `json:"timestamp"`
	Total      int       `json:"total_files"`
	Completed  int       `json:"completed"`
	Failed     int       `json:"failed"`
	TotalBytes int64     `json:"total_bytes"`
}

Manifest is written as manifest.json after a batch download.

type Result

type Result struct {
	Total     int
	Completed int
	Failed    int
	Bytes     int64
	Errors    []DownloadError
}

Result summarises the outcome of a DownloadAll call.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL