rangeio

package
v3.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2025 License: AGPL-3.0 Imports: 18 Imported by: 0

Documentation

Overview

Package rangeio provides basic interfaces and utilities for reading ranges of data.

Index

Constants

This section is empty.

Variables

View Source
var DefaultConfig = Config{

	MaxParallelism: 10,

	CoalesceSize: 1 << 20,

	MaxRangeSize: 8 * (1 << 20),

	MinRangeSize: 1 << 20,
}

DefaultConfig holds the default values for Config.

Functions

func ReadRanges

func ReadRanges(ctx context.Context, r Reader, ranges []Range) error

ReadRanges reads the set of ranges from the provided Reader, populating Data for each element in ranges.

ReadRanges makes a copy of ranges and optimizes them for performance: coalescing ranges that are close together and splitting ranges that are too large. The optimized set of ranges are read in parallel.

The optimization behavior of ReadRanges can be controlled by providing a context injected with custom configuration by WithConfig. If there is no custom configuration in the context, DefaultConfig is used.

ReadRanges returns an error if any call to r.ReadRange returns an error. ReadRanges only returns io.EOF if one of the ranges is beyond the end of the input source.

func WithConfig

func WithConfig(ctx context.Context, config *Config) context.Context

WithConfig creates a new context that contains the provided Config. Calls to ReadRanges with this context will use the config.

Types

type Config

type Config struct {
	// MaxParallelism is the maximum number of goroutines that may be used to
	// read ranges in parallel. If MaxParallelism <= 0, [runtime.NumCPU] is
	// used.
	//
	// Ranges are split into smaller ranges (no smaller than MinRangeSize) to
	// get as close as possible to MaxParallelism.
	//
	// If MaxParallelism is 1, ReadRanges will read each range sequentially.
	MaxParallelism int `yaml:"max_parallelism" category:"experimental"`

	// CoalesceSize determines the maximum size (in bytes) of a gap between each
	// pair of ranges that causes them to be coalesced into a single range.
	CoalesceSize int `yaml:"coalesce_size" category:"experimental"`

	// MaxRangeSize determines the maximum size (in bytes) of a range. Ranges
	// won't be coalesced if they exceed this size, and existing ranges will be
	// split if they exceed this size (down to MinRangeSize).
	MaxRangeSize int `yaml:"max_range_size" category:"experimental"`

	// MinRangeSize determines the minimum size (in bytes) of a range. When a
	// range is split, it won't be split into units smaller than MinRangeSize.
	MinRangeSize int `yaml:"min_range_size" category:"experimental"`
}

Config configures the behavior of ReadRanges.

func (*Config) IsZero

func (cfg *Config) IsZero() bool

func (*Config) RegisterFlags

func (cfg *Config) RegisterFlags(prefix string, fs *flag.FlagSet)

type Range

type Range struct {
	// Data to read into; exactly len(Data) bytes will be read, or an error will
	// be returned.
	Data []byte

	// Offset to start reading from.
	Offset int64
}

Range represents a range of data to be read.

func (Range) Len

func (r Range) Len() int64

Len returns the length of the range.

type Reader

type Reader interface {
	// ReadRange reads len(r.Data) bytes into r.Data starting at r.Offset in the
	// underlying input source.
	//
	// It returns the number of bytes read (0 <= n <= len(r.Data)) and any error
	// encountered.
	//
	// When ReadRange returns n < len(r.Data), it returns a non-nil error
	// explaining why more bytes were not returned. The error must be [io.EOF]
	// when reading beyond the end of the input source.
	//
	// ReadRange may use all of r.Data as scratch space during the call, even if
	// less than len(r.Data) bytes are read. If some data is available but not
	// len(r.Data) bytes, ReadRange blocks until either all the data is
	// available or an error occurs.
	//
	// Implementations are recommended but not required to immediately respond
	// to the cancellation of ctx; for example, cancellation may not occur
	// immediately when using disk-based I/O.
	//
	// If the len(r.Data) bytes returned by ReadRange are at the end of the
	// input source, ReadRange may return either err == [io.EOF] or err == nil.
	//
	// If ReadRange is reading from an input source with a seek offset,
	// ReadRange should not affect nor be affected by the underlying seek
	// offset.
	//
	// It is safe to call ReadRange concurrently from multiple goroutines.
	//
	// Implementations must not retain r.Data after the call returns.
	ReadRange(ctx context.Context, r Range) (int, error)
}

Reader is the interface that wraps the basic ReadRange method. Reader is similar to io.ReaderAt, but allows providing a context.Context for canceling the operation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL