Documentation
¶
Overview ¶
Package parallel provides an http.RoundTripper that transparently parallelizes GET requests using concurrent byte-range requests for better throughput.
───────────────────────────── How it works ─────────────────────────────
- For non-GET requests, the transport passes them through unmodified to the underlying transport.
- For GET requests, it first performs a HEAD request to check if the server supports byte ranges and to determine the total response size.
- If the HEAD request indicates range support and known size, the transport generates multiple concurrent GET requests with specific byte-range headers.
- Subranges are written to temporary files and stitched together in a custom Response.Body that's transparent to the caller.
- Per-host and per-request concurrency limits are enforced using semaphores.
───────────────────────────── Notes & caveats ───────────────────────────
- Only works with servers that support "Accept-Ranges: bytes" and provide Content-Length or Content-Range headers with total size information.
- Content-Encoding (compression) is not compatible with byte ranges, so compressed responses fall back to single-threaded behavior.
- Temporary files are created for each subrange and cleaned up automatically.
- The transport respects per-host concurrency limits to avoid overwhelming servers.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
type Option func(*ParallelTransport)
Option configures a ParallelTransport.
func WithMaxConcurrentPerHost ¶
WithMaxConcurrentPerHost sets the maximum concurrent requests per hostname. Default concurrency limits are applied if not specified.
func WithMaxConcurrentPerRequest ¶
WithMaxConcurrentPerRequest sets the maximum concurrent subrange requests for a single request. Default: 4.
func WithMinChunkSize ¶
WithMinChunkSize sets the minimum size in bytes for each subrange chunk. Requests smaller than this will not be parallelized. Default: 1MB.
func WithTempDir ¶
WithTempDir sets the directory for temporary files. If empty, os.TempDir() is used.
type ParallelTransport ¶
type ParallelTransport struct {
// contains filtered or unexported fields
}
ParallelTransport wraps another http.RoundTripper and parallelizes GET requests using concurrent byte-range requests when possible.
func New ¶
func New(base http.RoundTripper, opts ...Option) *ParallelTransport
New returns a ParallelTransport wrapping base. If base is nil, http.DefaultTransport is used. Options configure parallelization behavior.
Source Files