gdl

gdl - Go Downloader

A fast, concurrent, and feature-rich file downloader library and CLI tool written in Go.

✨ Features

• 🚀 Optimized Performance: Smart defaults with adaptive concurrency based on file size
• ⚡ Bandwidth Control: Rate limiting with human-readable formats (1MB/s, 500k, etc.)
• 📊 Progress Tracking: Real-time progress with multiple display formats
• 🔄 Resume Support: Automatic resume of interrupted downloads
• 🌐 Protocol Support: HTTP/HTTPS with custom headers and proxy support
• 🛡️ Error Handling: Comprehensive error handling with smart retry logic
• ⚡ Cross-Platform: Works on Linux, macOS, Windows, and ARM (including Apple Silicon, Raspberry Pi, and ARM servers)
• 🔧 Dual Interface: Both library API and command-line tool
• 📱 User-Friendly: Interactive prompts and helpful error messages
• 🔌 Plugin System: Extensible plugin architecture for custom functionality
• 🎯 Event-Driven: Hook into download lifecycle events
• 📊 Performance Monitoring: Metrics collection and aggregation for production use
• 🔐 Security: Built-in security constraints and validation

📈 Performance

gdl uses intelligent optimization to provide the best download speeds:

Benchmark-Optimized Smart Defaults

• Small files (<1MB): Lightweight mode - minimal overhead, 60-90% of curl speed ⚡
• Small-medium files (1-10MB): 2 concurrent connections
• Medium files (10-100MB): 4 concurrent connections
• Large files (100MB-1GB): 8 concurrent connections
• Very large files (>1GB): 16 concurrent connections

Real-World Performance

Based on benchmarks against curl and wget:

Performance as % of curl speed (baseline = 100%). Higher is better.

Technical Optimizations

• Zero-Copy I/O: Linux sendfile for files >10MB reduces CPU by 20-30%
• Buffer Pooling: 4-tier memory pool (8KB/64KB/1MB/4MB) reduces allocations by 50-90%
• Advanced Connection Pool: DNS caching, TLS session resumption, CDN optimization
• Lightweight Mode: Minimal overhead HTTP client for files <1MB (3-6x faster)
• Optimized HTTP Client: Enhanced connection pooling and HTTP/2 support
• Adaptive Chunk Sizing: Dynamic buffer sizes (8KB-1MB) based on file size
• Memory Efficiency: 50-90% less memory usage with advanced pooling
• CI Performance Testing: Automated regression detection with 10% threshold

📦 Installation

As a CLI tool

Go Install

go install github.com/forest6511/gdl/cmd/gdl@latest

Homebrew (macOS/Linux)

brew tap forest6511/tap
brew install forest6511/tap/gdl

Note: Use the full tap name forest6511/tap/gdl to avoid conflicts with the GNOME gdl package.

Binary Downloads

Download pre-built binaries from GitHub Releases

As a library

go get github.com/forest6511/gdl

🚀 Quick Start

CLI Usage

# Simple download (uses smart defaults)
gdl https://example.com/file.zip

# Override smart defaults with custom settings
gdl --concurrent 8 --chunk-size 2MB -o myfile.zip https://example.com/file.zip

# With bandwidth limiting
gdl --max-rate 1MB/s https://example.com/large-file.zip
gdl --max-rate 500k https://example.com/file.zip  # Smart concurrency still applies

# With custom headers and resume
gdl -H "Authorization: Bearer token" --resume https://example.com/file.zip

# Using plugins
gdl plugin install oauth2-auth
gdl plugin list
gdl --plugin oauth2-auth https://secure-api.example.com/file.zip

Library Usage

package main

import (
    "bytes"
    "context"
    "fmt"
    "github.com/forest6511/gdl"
)

func main() {
    // Simple download using Download function
    stats, err := gdl.Download(context.Background(), 
        "https://example.com/file.zip", "file.zip")
    if err != nil {
        panic(err)
    }
    fmt.Printf("Downloaded %d bytes in %v\n", stats.BytesDownloaded, stats.Duration)
    
    // Download with progress callback and bandwidth limiting using DownloadWithOptions
    options := &gdl.Options{
        MaxConcurrency: 4,
        MaxRate: 1024 * 1024, // 1MB/s rate limit
        ProgressCallback: func(p gdl.Progress) {
            fmt.Printf("Progress: %.1f%% Speed: %.2f MB/s\n", 
                p.Percentage, float64(p.Speed)/1024/1024)
        },
    }
    
    stats, err = gdl.DownloadWithOptions(context.Background(),
        "https://example.com/file.zip", "file.zip", options)
    if err == nil {
        fmt.Printf("Download completed successfully! Average speed: %.2f MB/s\n", 
            float64(stats.AverageSpeed)/1024/1024)
    }
    
    // Download to memory using DownloadToMemory
    data, stats, err := gdl.DownloadToMemory(context.Background(),
        "https://example.com/small-file.txt")
    if err == nil {
        fmt.Printf("Downloaded %d bytes to memory in %v\n", len(data), stats.Duration)
    }
    
    // Download to any io.Writer using DownloadToWriter
    var buffer bytes.Buffer
    stats, err = gdl.DownloadToWriter(context.Background(),
        "https://example.com/data.json", &buffer)
    if err == nil {
        fmt.Printf("Downloaded to buffer: %d bytes\n", stats.BytesDownloaded)
    }
        
    // Resume a partial download using DownloadWithResume
    stats, err = gdl.DownloadWithResume(context.Background(),
        "https://example.com/large-file.zip", "large-file.zip")
    if err == nil && stats.Resumed {
        fmt.Printf("Successfully resumed download: %d bytes\n", stats.BytesDownloaded)
    }
        
    // Get file information without downloading using GetFileInfo
    fileInfo, err := gdl.GetFileInfo(context.Background(),
        "https://example.com/file.zip")
    if err == nil {
        fmt.Printf("File size: %d bytes\n", fileInfo.Size)
    }
    
    // Using the extensible Downloader with plugins
    downloader := gdl.NewDownloader()
    
    // Register custom protocol handler
    err = downloader.RegisterProtocol(customProtocolHandler)
    
    // Use middleware
    downloader.UseMiddleware(rateLimitingMiddleware)
    
    // Register event listeners
    downloader.On(events.EventDownloadStarted, func(event events.Event) {
        fmt.Printf("Download started: %s\n", event.Data["url"])
    })
    
    // Download with plugins and middleware
    stats, err = downloader.Download(context.Background(),
        "https://example.com/file.zip", "file.zip", options)
    if err == nil {
        fmt.Printf("Plugin-enhanced download completed: %d bytes\n", stats.BytesDownloaded)
    }
}

📚 Complete API Documentation

• 📋 API Reference - Complete library API documentation
• 📁 Directory Structure - Complete project organization
• 🔧 Maintenance Guide - Development and maintenance procedures
• 🚨 Error Handling - Error types and handling strategies
• 🔌 Plugin Development - Plugin development guide
• 🚀 Extending Guide - Extension points and customization
• 📦 Go Package Docs - Generated Go documentation

📋 Complete CLI Reference

• ⚙️ CLI Reference - Comprehensive CLI usage guide

📖 Examples

Complete working examples are available in the examples/ directory:

• Basic Download - Simple download operations
• Concurrent Downloads - Parallel download optimization
• Progress Tracking - Real-time progress monitoring
• Resume Support - Interrupt and resume downloads
• Error Handling - Robust error recovery
• Production Usage - Production-ready patterns with monitoring
• CLI Examples - Command-line usage patterns
• Integration Tests - Feature verification
• Plugin Examples - Custom plugin development
• Extension Examples - System extension patterns

Running Examples

# Core functionality examples
cd examples/01_basic_download && go run main.go
cd examples/02_concurrent_download && go run main.go
cd examples/03_progress_tracking && go run main.go
cd examples/04_resume_functionality && go run main.go
cd examples/05_error_handling && go run main.go
cd examples/06_production_usage && go run main.go

# Interface examples
cd examples/cli
chmod +x *.sh
./basic_cli_examples.sh
./advanced_cli_examples.sh

# Integration demo
cd examples/integration
go run feature_demo.go

# Plugin examples
cd examples/plugins/auth/oauth2
go build -buildmode=plugin -o oauth2.so
cd examples/plugins/storage/s3
go build -buildmode=plugin -o s3.so

🔄 Feature Parity Matrix

Legend

• ✅ Fully supported - Feature is available and fully functional
• ❌ Not applicable - Feature doesn't make sense in this context

Key Concepts: Resume vs. Retry

The terms "Resume" and "Retry" sound similar but handle different situations. Understanding the difference is key to using gdl effectively.

Scenario: Combining Resume and Retry

1. Day 1: You start downloading a 10GB file. It gets to 5GB, and you stop the program (e.g., with Ctrl+C). (-> Interruption)
2. Day 2: You run the same command again with resume enabled. gdl detects the incomplete 5GB file and starts downloading from that point. (-> This is a Resumed download)
3. During the download of the remaining 5GB, your network connection briefly drops, causing a timeout error.
4. gdl automatically waits a moment and re-attempts the failed request. (-> This is a Retry)
5. The download then completes successfully.

In this case, the final DownloadStats would be Resumed: true and Retries: 1.

🏗️ Architecture

For the complete project structure, see Directory Structure.

Key Components

• Core Engine (internal/core): Main download orchestration
• Concurrency Manager (internal/concurrent): Parallel download coordination
• Rate Limiter (pkg/ratelimit): Bandwidth throttling and rate control
• Resume Engine (internal/resume): Download resumption and partial file handling
• Progress System (pkg/progress): Real-time progress tracking
• Error Framework (pkg/errors): Comprehensive error handling
• Network Layer (internal/network): HTTP client and diagnostics
• Storage Layer (internal/storage): File system operations
• Plugin System (pkg/plugin): Extensible plugin architecture
• Event System (pkg/events): Download lifecycle events
• Middleware Layer (pkg/middleware): Request/response processing
• Protocol Registry (pkg/protocols): Custom protocol handlers
• Monitoring System (pkg/monitoring): Performance metrics and analytics
• CLI Interface (cmd/gdl): Command-line tool implementation

Plugin Architecture

┌─────────────────────────────────────────────────────────────┐
│                       gdl Core                              │
├─────────────────────────────────────────────────────────────┤
│                 Plugin Manager                              │
├─────────────────────────────────────────────────────────────┤
│  Auth Plugins  │ Protocol Plugins │ Storage Plugins         │
├────────────────┼──────────────────┼─────────────────────────┤
│ Transform      │ Hook Plugins     │ Custom Plugins          │
│ Plugins        │                  │                         │
└─────────────────────────────────────────────────────────────┘

Plugin Types:

• Authentication Plugins: OAuth2, API keys, custom auth schemes
• Protocol Plugins: FTP, S3, custom protocols
• Storage Plugins: Cloud storage, databases, custom backends
• Transform Plugins: Compression, encryption, format conversion
• Hook Plugins: Pre/post processing, logging, analytics

🧪 Testing

The project includes comprehensive testing:

Quick Development Testing

# Run all tests (basic development)
go test ./...

# Run tests with race detection (recommended)
go test -race ./...

# Run with coverage
go test -coverprofile=coverage.out ./...

# Run benchmarks
go test -bench=. ./...

# Run performance benchmarks with optimization comparisons
go test -bench=BenchmarkDownloadWith -benchmem ./internal/core/

# Check for performance regressions
./scripts/performance_check.sh

Pre-Push Validation (Recommended)

# Complete local CI validation (recommended before pushing)
./scripts/local-ci-check.sh

# OR use Makefile targets
make pre-push           # Format + all CI checks
make ci-check          # All CI checks without formatting

Cross-Platform Testing

# Test all platforms locally (requires: brew install act)
make test-ci-all       # Ubuntu + Windows + macOS
make test-ci-ubuntu    # Ubuntu only
make test-ci-windows   # Windows only  
make test-ci-macos     # macOS only

# Quick cross-compilation check
make test-cross-compile

When to Use What

Test Coverage

• Unit tests: All packages have >90% coverage
• Integration tests: Real HTTP download scenarios
• CLI tests: Command-line interface functionality
• Benchmark tests: Performance regression detection
• Race detection: Concurrent safety verification

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Development Setup

# Clone the repository
git clone https://github.com/forest6511/gdl.git
cd gdl

# Install dependencies
go mod download

# Install golangci-lint (if not already installed)
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

# Install act for local CI testing (optional but recommended)
brew install act

# Verify everything works (CI equivalent check)
make ci-check

# Run tests
go test ./...

# Build CLI
go build -o gdl ./cmd/gdl/

Pre-commit Checks

Before committing and pushing changes, always run these checks locally to avoid CI failures:

# Run lint checks (essential before commit/push)
golangci-lint run

# Run tests with race detection
go test -race ./...

# Run tests with coverage
go test -coverprofile=coverage.out ./...

🔄 CI Compatibility

To prevent "works locally but fails in CI" issues, use these CI-equivalent commands:

# RECOMMENDED: Run complete pre-push validation
make pre-push        # Formats code AND runs all CI checks

# Alternative: Run CI checks without formatting
make ci-check        # All CI checks locally

# Cross-platform testing with act (requires: brew install act)
make test-ci-all     # Test Ubuntu, Windows, macOS locally
make test-ci-ubuntu  # Test Ubuntu CI locally
make test-ci-windows # Test Windows CI locally  
make test-ci-macos   # Test macOS CI locally

# Fix formatting issues automatically
make fix-and-commit  # Auto-fix formatting and create commit if needed

# Quick cross-compilation check
make test-cross-compile  # Fast Windows/macOS build verification

# Individual commands
make ci-format       # Format code to CI standards
make ci-vet          # go vet (excluding examples)
make ci-test-core    # core library tests with race detection

⚠️ Important: If the pre-commit hook reformats your code, it will stop the commit. Add the changes and commit again:

git add .
git commit

A pre-commit hook is automatically installed that runs CI-equivalent checks on every commit, preventing most CI failures.

Important: Always run make ci-check locally before pushing to ensure all checks pass. This prevents CI pipeline failures and maintains code quality standards.

🔧 Developer Tools

Git Hooks

# Setup commit message validation and pre-commit checks
./scripts/setup-git-hooks.sh

Release Management

# Prepare a new release
./scripts/prepare-release.sh v0.10.0
# Edit CHANGELOG.md with release notes
./scripts/prepare-release.sh --release v0.10.0

Local Testing

# Test GitHub Actions locally with act
act push -j quick-checks          # Fast validation
act push -W .github/workflows/main.yml --dryrun  # Full CI dry run

📚 Documentation

• Contributing Guide - Development guidelines and workflow
• Release Setup - Release management and distribution
• Local Testing - GitHub Actions testing with act
• API Reference - Library API documentation
• CLI Reference - Command-line usage
• Plugin Development - Plugin system guide
• Maintenance - Development and maintenance procedures

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Go community for excellent libraries and tools
• Contributors who helped improve this project
• Users who provided feedback and bug reports

Made with ❤️ in Go