stream

package
v1.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 8 Imported by: 1

Documentation

Overview

Package stream provides a managed streaming engine for Trove with backpressure, chunked transfers, buffer pooling, and concurrency control.

Index

Constants

This section is empty.

Variables

View Source
var ErrBackpressure = errors.New("stream: backpressure limit reached")

ErrBackpressure is returned when a non-blocking send would block.

Functions

This section is empty.

Types

type Backpressure

type Backpressure int

Backpressure identifies the backpressure strategy used when a stream's consumer cannot keep up with the producer.

const (
	// BackpressureBlock blocks the sender until the consumer catches up.
	BackpressureBlock Backpressure = iota

	// BackpressureDrop drops the oldest unacknowledged chunks.
	BackpressureDrop

	// BackpressureBuffer spills chunks to a temporary buffer when
	// the in-memory channel fills.
	BackpressureBuffer

	// BackpressureAdaptive dynamically adjusts chunk size based on
	// measured throughput vs. consumption rate.
	BackpressureAdaptive
)

func (Backpressure) String

func (b Backpressure) String() string

String returns the strategy name.

type BackpressureHandler

type BackpressureHandler interface {
	// Send delivers a chunk to the target channel, applying the strategy
	// when the channel is full. Returns ErrBackpressure or context errors.
	Send(ctx context.Context, ch chan *Chunk, chunk *Chunk) error
}

BackpressureHandler defines how a stream reacts when its chunk channel is full. Each strategy implements Send differently.

func NewAdaptiveHandler

func NewAdaptiveHandler() BackpressureHandler

NewAdaptiveHandler creates a handler that blocks (like BackpressureBlock) but signals that chunk sizes should be adjusted. Future phases will integrate throughput-based auto-tuning.

func NewBackpressureHandler

func NewBackpressureHandler(mode Backpressure, pool *ChunkPool) BackpressureHandler

NewBackpressureHandler constructs the appropriate handler for the strategy.

func NewBlockHandler

func NewBlockHandler() BackpressureHandler

NewBlockHandler creates a handler that blocks until the channel has room or the context is cancelled.

func NewBufferHandler

func NewBufferHandler(maxOverflow int) BackpressureHandler

NewBufferHandler creates a handler that spills chunks to an internal slice when the channel is full, up to maxOverflow extra chunks.

func NewDropHandler

func NewDropHandler(pool *ChunkPool) BackpressureHandler

NewDropHandler creates a handler that drops the oldest chunk in the channel when full, returning its buffer to the pool.

type Checksum

type Checksum struct {
	Algorithm string // "sha256", "blake3", "xxhash"
	Value     []byte
}

Checksum holds an integrity hash for a chunk.

type Chunk

type Chunk struct {
	// Index is the sequential chunk number within the stream (0-based).
	Index int

	// Offset is the byte offset of this chunk within the full object.
	Offset int64

	// Data holds the chunk payload. The backing slice comes from a ChunkPool
	// and must be returned after processing.
	Data []byte

	// Size is the number of valid bytes in Data (may be less than cap(Data)
	// for the final chunk).
	Size int

	// Checksum holds an optional integrity hash for this chunk.
	Checksum Checksum

	// Final marks this as the last chunk in the stream.
	Final bool
}

Chunk is a unit of data in a stream. Buffers are drawn from a ChunkPool and MUST be returned via ChunkPool.Put when no longer needed.

type ChunkAck

type ChunkAck struct {
	Index int
	ETag  string
	Err   error
}

ChunkAck is the backend's acknowledgement of a received chunk.

type ChunkHook

type ChunkHook func(*Chunk, ChunkAck)

ChunkHook is called after each chunk is sent/acknowledged.

type ChunkPool

type ChunkPool struct {
	// contains filtered or unexported fields
}

ChunkPool manages a pool of reusable byte buffers for zero-copy chunk allocation. It wraps sync.Pool with fixed-size buffers and tracks allocation metrics.

func NewChunkPool

func NewChunkPool(chunkSize int64) *ChunkPool

NewChunkPool creates a ChunkPool that allocates buffers of the given size.

func (*ChunkPool) ChunkSize

func (cp *ChunkPool) ChunkSize() int64

ChunkSize returns the size of buffers allocated by this pool.

func (*ChunkPool) Get

func (cp *ChunkPool) Get() []byte

Get returns a buffer from the pool. The caller MUST return it via Put.

func (*ChunkPool) Put

func (cp *ChunkPool) Put(buf []byte)

Put returns a buffer to the pool for reuse. The buffer is reset to its original capacity (zero-length but full cap).

func (*ChunkPool) Stats

func (cp *ChunkPool) Stats() ChunkPoolStats

Stats returns allocation statistics.

type ChunkPoolStats

type ChunkPoolStats struct {
	Allocated int64 `json:"allocated"`
	Released  int64 `json:"released"`
	ChunkSize int64 `json:"chunk_size"`
}

ChunkPoolStats holds chunk pool allocation metrics.

type CompleteHook

type CompleteHook func(*driver.ObjectInfo)

CompleteHook is called when a stream completes successfully.

type ControlMsg

type ControlMsg struct {
	Type   ControlType
	Offset int64 // For CtrlSeek.
}

ControlMsg carries a control command for a stream.

type ControlType

type ControlType int

ControlType identifies a stream control command.

const (
	CtrlPause  ControlType = iota // Pause the stream.
	CtrlResume                    // Resume a paused stream.
	CtrlCancel                    // Cancel the stream.
	CtrlSeek                      // Seek to a byte offset.
)

type Direction

type Direction int

Direction indicates the data flow of a stream.

const (
	// DirectionUpload streams data from client to storage.
	DirectionUpload Direction = iota
	// DirectionDownload streams data from storage to client.
	DirectionDownload
	// DirectionBiDi allows simultaneous read and write (sync, mirroring).
	DirectionBiDi
)

func (Direction) String

func (d Direction) String() string

String returns a human-readable direction name.

type ErrorHook

type ErrorHook func(error)

ErrorHook is called when a stream encounters an error.

type Metrics

type Metrics struct {
	BytesSent atomic.Int64
	BytesRecv atomic.Int64
	Chunks    atomic.Int64
	Retries   atomic.Int64
	StartTime time.Time
}

Metrics tracks per-stream transfer metrics.

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a Metrics with the start time set to now.

func (*Metrics) StreamSnapshot

func (sm *Metrics) StreamSnapshot() Snapshot

StreamSnapshot returns a point-in-time copy of stream metrics.

func (*Metrics) Throughput

func (sm *Metrics) Throughput() float64

Throughput returns the average throughput in bytes/sec since start.

type MetricsSnapshot

type MetricsSnapshot struct {
	ActiveStreams int64         `json:"active_streams"`
	TotalBytes    int64         `json:"total_bytes"`
	TotalChunks   int64         `json:"total_chunks"`
	FailedStreams int64         `json:"failed_streams"`
	RetryCount    int64         `json:"retry_count"`
	AvgThroughput float64       `json:"avg_throughput_bps"`
	P99Latency    time.Duration `json:"p99_latency"`
}

MetricsSnapshot is a serializable point-in-time copy of pool metrics.

type Option

type Option func(*streamConfig)

Option configures a Stream.

func WithBackpressure

func WithBackpressure(mode Backpressure) Option

WithBackpressure sets the backpressure strategy for this stream.

func WithChannelSize

func WithChannelSize(n int) Option

WithChannelSize sets the chunk channel buffer size.

func WithChunkSize

func WithChunkSize(size int64) Option

WithChunkSize sets the chunk size for this stream.

func WithOnChunk

func WithOnChunk(fn ChunkHook) Option

WithOnChunk registers a chunk callback.

func WithOnComplete

func WithOnComplete(fn CompleteHook) Option

WithOnComplete registers a completion callback.

func WithOnError

func WithOnError(fn ErrorHook) Option

WithOnError registers an error callback.

func WithOnProgress

func WithOnProgress(fn ProgressHook) Option

WithOnProgress registers a progress callback.

func WithPool

func WithPool(name string) Option

WithPool assigns this stream to a named pool.

func WithResumable

func WithResumable() Option

WithResumable enables resumable upload/download with offset tracking.

type Pool

type Pool struct {
	ID      id.ID
	Name    string
	Config  PoolConfig
	Metrics *PoolMetrics
	// contains filtered or unexported fields
}

Pool manages a set of concurrent streams with shared resources, concurrency limits, and bandwidth throttling.

func NewPool

func NewPool(name string, cfg PoolConfig) *Pool

NewPool creates a new streaming pool.

func (*Pool) Acquire

func (p *Pool) Acquire(ctx context.Context, dir Direction, bucket, key string, opts ...Option) (*Stream, error)

Acquire reserves a concurrency slot and creates a new stream within the pool. Blocks until a slot is available or the context is cancelled.

func (*Pool) ActiveCount

func (p *Pool) ActiveCount() int

ActiveCount returns the number of currently active streams.

func (*Pool) ChunkPool

func (p *Pool) ChunkPool() *ChunkPool

ChunkPool returns the pool's shared chunk buffer pool.

func (*Pool) Close

func (p *Pool) Close() error

Close gracefully shuts down the pool. It cancels the pool context, waits for all active streams to be released, and returns.

func (*Pool) Get

func (p *Pool) Get(streamID id.ID) *Stream

Get retrieves an active stream by ID.

func (*Pool) Range

func (p *Pool) Range(fn func(s *Stream) bool)

Range iterates over all active streams. The callback should not block.

func (*Pool) Release

func (p *Pool) Release(s *Stream)

Release returns a stream's concurrency slot to the pool. The stream is removed from the active set.

type PoolConfig

type PoolConfig struct {
	// MaxStreams is the maximum number of concurrent streams.
	MaxStreams int `json:"max_streams" yaml:"max_streams"`

	// MaxBandwidth limits total throughput in bytes/sec (0 = unlimited).
	MaxBandwidth int64 `json:"max_bandwidth" yaml:"max_bandwidth"`

	// ChunkSize is the default chunk size for streams in this pool.
	ChunkSize int64 `json:"chunk_size" yaml:"chunk_size"`

	// BufferCount is the initial number of buffers in the chunk pool.
	BufferCount int `json:"buffer_count" yaml:"buffer_count"`

	// StreamTimeout is the maximum lifetime of a single stream.
	StreamTimeout time.Duration `json:"stream_timeout" yaml:"stream_timeout"`

	// IdleTimeout closes a stream that has been idle for this duration.
	IdleTimeout time.Duration `json:"idle_timeout" yaml:"idle_timeout"`

	// BackpressureMode is the default backpressure strategy for streams.
	BackpressureMode Backpressure `json:"backpressure" yaml:"backpressure"`
}

PoolConfig configures a streaming pool.

func DefaultPoolConfig

func DefaultPoolConfig() PoolConfig

DefaultPoolConfig returns a PoolConfig with sensible defaults.

type PoolMetrics

type PoolMetrics struct {
	ActiveStreams atomic.Int64
	TotalBytes    atomic.Int64
	TotalChunks   atomic.Int64
	FailedStreams atomic.Int64
	RetryCount    atomic.Int64
	// contains filtered or unexported fields
}

PoolMetrics tracks streaming performance for a Pool.

func (*PoolMetrics) AvgThroughput

func (m *PoolMetrics) AvgThroughput() float64

AvgThroughput returns the rolling average throughput (bytes/sec).

func (*PoolMetrics) P99Latency

func (m *PoolMetrics) P99Latency() time.Duration

P99Latency returns the P99 chunk latency.

func (*PoolMetrics) RecordBytes

func (m *PoolMetrics) RecordBytes(n int64)

RecordBytes adds n bytes to the cumulative total.

func (*PoolMetrics) RecordChunk

func (m *PoolMetrics) RecordChunk()

RecordChunk increments the chunk counter.

func (*PoolMetrics) RecordFailure

func (m *PoolMetrics) RecordFailure()

RecordFailure increments the failed stream counter.

func (*PoolMetrics) RecordRetry

func (m *PoolMetrics) RecordRetry()

RecordRetry increments the retry counter.

func (*PoolMetrics) SetAvgThroughput

func (m *PoolMetrics) SetAvgThroughput(bps float64)

SetAvgThroughput stores the rolling average throughput (bytes/sec).

func (*PoolMetrics) SetP99Latency

func (m *PoolMetrics) SetP99Latency(d time.Duration)

SetP99Latency stores the P99 chunk latency.

func (*PoolMetrics) Snapshot

func (m *PoolMetrics) Snapshot() MetricsSnapshot

Snapshot returns a point-in-time copy of all metrics.

type Progress

type Progress struct {
	StreamID  id.ID
	BytesSent int64
	BytesRecv int64
	TotalSize int64
	Percent   int
	Speed     float64 // bytes/sec
}

Progress holds a point-in-time progress report for a stream.

type ProgressHook

type ProgressHook func(Progress)

ProgressHook is called periodically with transfer progress.

type Snapshot

type Snapshot struct {
	BytesSent  int64         `json:"bytes_sent"`
	BytesRecv  int64         `json:"bytes_recv"`
	Chunks     int64         `json:"chunks"`
	Retries    int64         `json:"retries"`
	Throughput float64       `json:"throughput_bps"`
	Elapsed    time.Duration `json:"elapsed"`
}

Snapshot is a serializable copy of per-stream metrics.

type State

type State string

State represents the lifecycle state of a Stream.

const (
	StateIdle       State = "idle"
	StateActive     State = "active"
	StatePaused     State = "paused"
	StateCompleting State = "completing"
	StateCompleted  State = "completed"
	StateFailed     State = "failed"
	StateCancelled  State = "cancelled"
)

func (State) IsTerminal

func (s State) IsTerminal() bool

IsTerminal reports whether this state is a final state.

type Stream

type Stream struct {
	ID        id.ID
	Direction Direction
	Bucket    string
	Key       string
	// contains filtered or unexported fields
}

Stream represents a single managed data channel for transferring an object between a client and storage. It provides chunked transfer, backpressure, resumability, and lifecycle hooks.

func NewStream

func NewStream(
	ctx context.Context,
	dir Direction,
	bucket, key string,
	chunkPool *ChunkPool,
	opts ...Option,
) *Stream

NewStream creates a new stream in the Idle state.

func (*Stream) Ack

func (s *Stream) Ack(ack ChunkAck)

Ack sends an acknowledgement for a processed chunk.

func (*Stream) BackpressureMode

func (s *Stream) BackpressureMode() Backpressure

Config returns the stream's backpressure mode.

func (*Stream) ChunkPool

func (s *Stream) ChunkPool() *ChunkPool

ChunkPool returns the chunk pool used by this stream.

func (*Stream) Close

func (s *Stream) Close()

Close releases all stream resources. Safe to call multiple times.

func (*Stream) Complete

func (s *Stream) Complete(info *driver.ObjectInfo) error

Complete transitions the stream through completing → completed. It closes the chunk channel to signal EOF to the reader and fires completion hooks.

func (*Stream) Control

func (s *Stream) Control(msg ControlMsg) error

Control sends a control command to the stream.

func (*Stream) ControlChan

func (s *Stream) ControlChan() <-chan ControlMsg

ControlChan returns the read-only control channel for consuming control messages in the stream's processing loop.

func (*Stream) Done

func (s *Stream) Done() <-chan struct{}

Done returns a channel that is closed when the stream's context is cancelled (either by completion, failure, or explicit cancel).

func (*Stream) Err

func (s *Stream) Err() error

Err returns the last error, if any.

func (*Stream) Fail

func (s *Stream) Fail(err error)

Fail transitions the stream to the failed state with an error.

func (*Stream) Metrics

func (s *Stream) Metrics() *Metrics

Metrics returns the stream's transfer metrics.

func (*Stream) Offset

func (s *Stream) Offset() int64

Offset returns the current byte offset of the transfer.

func (*Stream) Read

func (s *Stream) Read(ctx context.Context) (*Chunk, error)

Read receives the next chunk from the stream. Blocks until a chunk is available or the context is cancelled.

func (*Stream) ReadAck

func (s *Stream) ReadAck(ctx context.Context) (ChunkAck, error)

ReadAck receives the next ack. Blocks until available or context cancelled.

func (*Stream) Resumable

func (s *Stream) Resumable() bool

Resumable reports whether this stream supports resume from offset.

func (*Stream) SetTotalSize

func (s *Stream) SetTotalSize(n int64)

SetTotalSize sets the expected total size of the transfer. This enables percentage-based progress reporting.

func (*Stream) Start

func (s *Stream) Start() error

Start activates the stream (idle → active).

func (*Stream) State

func (s *Stream) State() State

State returns the current stream state.

func (*Stream) TotalSize

func (s *Stream) TotalSize() int64

TotalSize returns the expected total size of the transfer, or -1 when nobody has set it.

func (*Stream) Write

func (s *Stream) Write(chunk *Chunk) error

Write sends a chunk into the stream. Applies the configured backpressure strategy if the chunk channel is full.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL