Documentation
¶
Index ¶
- func AvailableSpace(path string) (uint64, error)
- func CheckDiskSpace(dir string, neededBytes uint64, minFreeGB int) error
- func FormatBytes(bytes uint64) string
- type Config
- type DownloadError
- type Downloader
- func (d *Downloader) DownloadAll(urls []string) Result
- func (d *Downloader) DownloadAllFiles(tasks []FileTask) Result
- func (d *Downloader) DownloadFile(url, filename string) error
- func (d *Downloader) DownloadFileVerified(url, filename, expectedMD5 string) error
- func (d *Downloader) WriteManifest(queryDesc string, result Result) error
- type FileTask
- type Manifest
- type Result
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AvailableSpace ¶
AvailableSpace returns available bytes on the filesystem containing path.
func CheckDiskSpace ¶
CheckDiskSpace verifies there's enough space for the download. It checks that available space on dir is at least neededBytes plus minFreeGB gigabytes.
func FormatBytes ¶
FormatBytes formats bytes as a human-readable string using 1024-based units.
Types ¶
type Config ¶
type Config struct {
OutputDir string
Parallel int
CheckDiskSpace bool
MinFreeSpaceGB int
MaxRetries int
}
Config holds configuration for the Downloader.
type DownloadError ¶
DownloadError records which URL failed and why.
type Downloader ¶
type Downloader struct {
// OnProgress is called periodically during downloads if set.
// Arguments: filename, bytesWritten, totalBytes (-1 if unknown).
OnProgress func(filename string, written, total int64)
// contains filtered or unexported fields
}
Downloader manages parallel HTTP downloads with resume and retry support.
func New ¶
func New(cfg Config) *Downloader
New creates a Downloader using cfg. Defaults are applied for zero values.
func (*Downloader) DownloadAll ¶
func (d *Downloader) DownloadAll(urls []string) Result
DownloadAll downloads all urls in parallel using a goroutine pool limited by cfg.Parallel. It returns a Result with counts and any per-URL errors.
func (*Downloader) DownloadAllFiles ¶
func (d *Downloader) DownloadAllFiles(tasks []FileTask) Result
DownloadAllFiles downloads tasks in parallel. Each FileTask carries a URL, filename, and optional MD5 for verification. This is the preferred method when callers have file metadata (e.g., from an index).
func (*Downloader) DownloadFile ¶
func (d *Downloader) DownloadFile(url, filename string) error
DownloadFile fetches url and saves it as filename inside the output directory. It skips the download when the final file already exists, and resumes from a .part file when one is present. Retries are performed with exponential backoff for 429 and 5xx responses; 4xx (except 429) fail immediately.
func (*Downloader) DownloadFileVerified ¶
func (d *Downloader) DownloadFileVerified(url, filename, expectedMD5 string) error
DownloadFileVerified is like DownloadFile but also verifies the MD5 checksum of the downloaded file when expectedMD5 is non-empty. If verification fails, the file is removed and an error is returned.
func (*Downloader) WriteManifest ¶
func (d *Downloader) WriteManifest(queryDesc string, result Result) error
WriteManifest writes a manifest.json file summarising the download batch.
type FileTask ¶
type FileTask struct {
URL string
Filename string // saved as this name; falls back to filepath.Base(URL)
MD5 string // expected MD5 hex digest; empty to skip verification
}
FileTask describes a single file to download, with optional metadata for integrity verification and proper naming.