core

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2025 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package core provides the core implementation of the gdl download functionality.

Index

Constants

View Source 
const DefaultChunkSize = 32 * 1024 // 32KB

DefaultChunkSize is the default size for reading chunks during download.

View Source 
const DefaultTimeout = 30 * time.Minute

DefaultTimeout is the default timeout for download operations.

View Source 
const DefaultUserAgent = "gdl/1.0"

DefaultUserAgent is the default User-Agent string used for HTTP requests.

Variables

View Source 
var GlobalBufferPool = NewBufferPool()

GlobalBufferPool is a shared buffer pool for the entire application

Functions

func ApplyPlatformOptimizations added in v1.3.0 

func ApplyPlatformOptimizations(transport *http.Transport)

ApplyPlatformOptimizations applies platform-specific optimizations to HTTP transport

func GetOptimalBufferSize added in v1.3.0 

func GetOptimalBufferSize() int

GetOptimalBufferSize returns the optimal buffer size for Linux

func GetOptimalChunkSizePlatform added in v1.3.0 

func GetOptimalChunkSizePlatform(fileSize int64) int

GetOptimalChunkSizePlatform returns optimal chunk size for current platform

func GetOptimalConcurrency added in v1.3.0 

func GetOptimalConcurrency() int

GetOptimalConcurrency returns the optimal concurrency for Linux

func GetPlatformName added in v1.3.0 

func GetPlatformName() string

GetPlatformName returns the platform name

func GetPlatformString added in v1.3.0 

func GetPlatformString() string

GetPlatformString returns a string describing the platform

func PlatformOptimizations added in v1.3.0 

func PlatformOptimizations() *http.Transport

PlatformOptimizations returns Linux-specific optimizations

func PlatformSpecificInit added in v1.3.0 

func PlatformSpecificInit()

PlatformSpecificInit performs any platform-specific initialization

func SendFile added in v1.3.0 

func SendFile(dst *os.File, src *os.File, offset, count int64) (int64, error)

SendFile uses the sendfile system call for true zero-copy on Linux This is a low-level implementation for maximum performance

func SendFileLinux added in v1.3.0 

func SendFileLinux(dst *os.File, src *os.File, offset, count int64) (int64, error)

SendFileLinux uses the sendfile system call for true zero-copy on Linux

func SetPlatformSocketOptions added in v1.3.0 

func SetPlatformSocketOptions(fd uintptr) error

SetPlatformSocketOptions sets Linux-specific socket options

func ShouldUseZeroCopyPlatform added in v1.3.0 

func ShouldUseZeroCopyPlatform(fileSize int64) bool

ShouldUseZeroCopyPlatform checks if zero-copy should be used on current platform

func UseSendfile added in v1.3.0 

func UseSendfile() bool

UseSendfile checks if sendfile should be used on Linux

Types

type BufferPool added in v1.3.0 

type BufferPool struct {
	// contains filtered or unexported fields
}

BufferPool manages reusable byte buffers to reduce memory allocations

func NewBufferPool added in v1.3.0 

func NewBufferPool() *BufferPool

NewBufferPool creates a new buffer pool with various sizes

func (*BufferPool) Get added in v1.3.0 

func (bp *BufferPool) Get(size int64) []byte

Get returns a buffer appropriate for the given size

func (*BufferPool) GetForFileSize added in v1.3.0 

func (bp *BufferPool) GetForFileSize(fileSize int64) []byte

GetForFileSize returns an appropriate buffer based on file size

func (*BufferPool) GetSized added in v1.3.0 

func (bp *BufferPool) GetSized(minSize int) []byte

GetSized returns a buffer of at least the requested size

func (*BufferPool) Put added in v1.3.0 

func (bp *BufferPool) Put(buf []byte)

Put returns a buffer to the pool for reuse

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

func NewDownloader 

func NewDownloader() *Downloader

NewDownloader creates a new Downloader instance with default settings.

func NewDownloaderWithClient 

func NewDownloaderWithClient(client *http.Client) *Downloader

NewDownloaderWithClient creates a new Downloader instance with a custom HTTP client.

func (*Downloader) Download 

func (d *Downloader) Download(
	ctx context.Context,
	url, destination string,
	options *types.DownloadOptions,
) (*types.DownloadStats, error)

func (*Downloader) DownloadToWriter 

func (d *Downloader) DownloadToWriter(
	ctx context.Context,
	url string,
	writer io.Writer,
	options *types.DownloadOptions,
) (*types.DownloadStats, error)

DownloadToWriter downloads a file from the given URL and writes it to the provided writer. It implements the types.Downloader interface.

func (*Downloader) GetFileInfo 

func (d *Downloader) GetFileInfo(ctx context.Context, url string) (*types.FileInfo, error)

GetFileInfo retrieves information about a file without downloading it. It implements the types.Downloader interface.

func (*Downloader) WithLogging 

func (d *Downloader) WithLogging(enabled bool) *Downloader

WithLogging enables or disables enhanced logging with error context.

func (*Downloader) WithRetryStrategy 

func (d *Downloader) WithRetryStrategy(manager *retry.RetryManager) *Downloader

WithRetryStrategy configures the retry strategy for downloads.

func (*Downloader) WithSpaceChecker 

func (d *Downloader) WithSpaceChecker(checker *storage.SpaceChecker) *Downloader

WithSpaceChecker configures the disk space checker.

type LightweightDownloader added in v1.3.0 

type LightweightDownloader struct {
	// contains filtered or unexported fields
}

LightweightDownloader is an optimized downloader for small files with minimal overhead and reduced memory usage

func NewLightweightDownloader added in v1.3.0 

func NewLightweightDownloader() *LightweightDownloader

NewLightweightDownloader creates a new lightweight downloader optimized for small files

func (*LightweightDownloader) Download added in v1.3.0 

func (ld *LightweightDownloader) Download(ctx context.Context, url string, writer io.Writer) (int64, error)

Download performs a lightweight download optimized for small files

func (*LightweightDownloader) DownloadWithOptions added in v1.3.0 

func (ld *LightweightDownloader) DownloadWithOptions(ctx context.Context, url string, writer io.Writer, userAgent string) (int64, error)

DownloadWithOptions performs a lightweight download with custom user agent

func (*LightweightDownloader) DownloadWithProgress added in v1.3.0 

func (ld *LightweightDownloader) DownloadWithProgress(ctx context.Context, url string, writer io.Writer, progressFunc func(downloaded, total int64)) (int64, error)

DownloadWithProgress performs a lightweight download with progress callback

func (*LightweightDownloader) DownloadWithProgressAndOptions added in v1.3.0 

func (ld *LightweightDownloader) DownloadWithProgressAndOptions(ctx context.Context, url string, writer io.Writer, progressFunc func(downloaded, total int64), userAgent string) (int64, error)

DownloadWithProgressAndOptions performs a lightweight download with progress callback and user agent

type PlatformInfo added in v1.3.0 

type PlatformInfo struct {
	OS            string
	Arch          string
	NumCPU        int
	IsARM         bool
	IsServerGrade bool
	Optimizations PlatformOptimizationSet
}

PlatformInfo contains information about the current platform

func DetectPlatform added in v1.3.0 

func DetectPlatform() *PlatformInfo

DetectPlatform detects the current platform and returns optimization settings

type PlatformOptimizationSet added in v1.3.0 

type PlatformOptimizationSet struct {
	BufferSize      int
	Concurrency     int
	MaxConnections  int
	UseSendfile     bool
	UseZeroCopy     bool
	EnableHTTP2     bool
	ConnectionReuse bool
}

PlatformOptimizationSet contains platform-specific optimizations

type PooledBuffer added in v1.3.0 

type PooledBuffer struct {
	// contains filtered or unexported fields
}

PooledBuffer provides a convenient wrapper for pooled buffers

func NewPooledBuffer added in v1.3.0 

func NewPooledBuffer(size int64) *PooledBuffer

NewPooledBuffer gets a buffer from the pool

func (*PooledBuffer) Bytes added in v1.3.0 

func (pb *PooledBuffer) Bytes() []byte

Bytes returns the underlying byte slice

func (*PooledBuffer) Release added in v1.3.0 

func (pb *PooledBuffer) Release()

Release returns the buffer to the pool

func (*PooledBuffer) Resize added in v1.3.0 

func (pb *PooledBuffer) Resize(newSize int)

Resize adjusts the buffer size if needed

type ZeroCopyDownloader added in v1.3.0 

type ZeroCopyDownloader struct {
	// contains filtered or unexported fields
}

ZeroCopyDownloader provides zero-copy optimized downloads for large files

func NewZeroCopyDownloader added in v1.3.0 

func NewZeroCopyDownloader() *ZeroCopyDownloader

NewZeroCopyDownloader creates a new zero-copy optimized downloader

func (*ZeroCopyDownloader) Download added in v1.3.0 

func (zd *ZeroCopyDownloader) Download(ctx context.Context, url string, dest string) (int64, error)

Download performs a zero-copy download using platform-specific optimizations

func (*ZeroCopyDownloader) DownloadWithProgress added in v1.3.0 

func (zd *ZeroCopyDownloader) DownloadWithProgress(
	ctx context.Context,
	url string,
	dest string,
	progressFunc func(downloaded, total int64),
) (int64, error)

DownloadWithProgress performs zero-copy download with progress reporting

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.