Documentation
¶
Overview ¶
Package stream provides a managed streaming engine for Trove with backpressure, chunked transfers, buffer pooling, and concurrency control.
Index ¶
- Variables
- type Backpressure
- type BackpressureHandler
- type Checksum
- type Chunk
- type ChunkAck
- type ChunkHook
- type ChunkPool
- type ChunkPoolStats
- type CompleteHook
- type ControlMsg
- type ControlType
- type Direction
- type ErrorHook
- type Metrics
- type MetricsSnapshot
- type Option
- func WithBackpressure(mode Backpressure) Option
- func WithChannelSize(n int) Option
- func WithChunkSize(size int64) Option
- func WithOnChunk(fn ChunkHook) Option
- func WithOnComplete(fn CompleteHook) Option
- func WithOnError(fn ErrorHook) Option
- func WithOnProgress(fn ProgressHook) Option
- func WithPool(name string) Option
- func WithResumable() Option
- type Pool
- func (p *Pool) Acquire(ctx context.Context, dir Direction, bucket, key string, opts ...Option) (*Stream, error)
- func (p *Pool) ActiveCount() int
- func (p *Pool) ChunkPool() *ChunkPool
- func (p *Pool) Close() error
- func (p *Pool) Get(streamID id.ID) *Stream
- func (p *Pool) Range(fn func(s *Stream) bool)
- func (p *Pool) Release(s *Stream)
- type PoolConfig
- type PoolMetrics
- func (m *PoolMetrics) AvgThroughput() float64
- func (m *PoolMetrics) P99Latency() time.Duration
- func (m *PoolMetrics) RecordBytes(n int64)
- func (m *PoolMetrics) RecordChunk()
- func (m *PoolMetrics) RecordFailure()
- func (m *PoolMetrics) RecordRetry()
- func (m *PoolMetrics) SetAvgThroughput(bps float64)
- func (m *PoolMetrics) SetP99Latency(d time.Duration)
- func (m *PoolMetrics) Snapshot() MetricsSnapshot
- type Progress
- type ProgressHook
- type Snapshot
- type State
- type Stream
- func (s *Stream) Ack(ack ChunkAck)
- func (s *Stream) BackpressureMode() Backpressure
- func (s *Stream) ChunkPool() *ChunkPool
- func (s *Stream) Close()
- func (s *Stream) Complete(info *driver.ObjectInfo) error
- func (s *Stream) Control(msg ControlMsg) error
- func (s *Stream) ControlChan() <-chan ControlMsg
- func (s *Stream) Done() <-chan struct{}
- func (s *Stream) Err() error
- func (s *Stream) Fail(err error)
- func (s *Stream) Metrics() *Metrics
- func (s *Stream) Offset() int64
- func (s *Stream) Read(ctx context.Context) (*Chunk, error)
- func (s *Stream) ReadAck(ctx context.Context) (ChunkAck, error)
- func (s *Stream) Resumable() bool
- func (s *Stream) SetTotalSize(n int64)
- func (s *Stream) Start() error
- func (s *Stream) State() State
- func (s *Stream) TotalSize() int64
- func (s *Stream) Write(chunk *Chunk) error
Constants ¶
This section is empty.
Variables ¶
var ErrBackpressure = errors.New("stream: backpressure limit reached")
ErrBackpressure is returned when a non-blocking send would block.
Functions ¶
This section is empty.
Types ¶
type Backpressure ¶
type Backpressure int
Backpressure identifies the backpressure strategy used when a stream's consumer cannot keep up with the producer.
const ( // BackpressureBlock blocks the sender until the consumer catches up. BackpressureBlock Backpressure = iota // BackpressureDrop drops the oldest unacknowledged chunks. BackpressureDrop // BackpressureBuffer spills chunks to a temporary buffer when // the in-memory channel fills. BackpressureBuffer // BackpressureAdaptive dynamically adjusts chunk size based on // measured throughput vs. consumption rate. BackpressureAdaptive )
func (Backpressure) String ¶
func (b Backpressure) String() string
String returns the strategy name.
type BackpressureHandler ¶
type BackpressureHandler interface {
// Send delivers a chunk to the target channel, applying the strategy
// when the channel is full. Returns ErrBackpressure or context errors.
Send(ctx context.Context, ch chan *Chunk, chunk *Chunk) error
}
BackpressureHandler defines how a stream reacts when its chunk channel is full. Each strategy implements Send differently.
func NewAdaptiveHandler ¶
func NewAdaptiveHandler() BackpressureHandler
NewAdaptiveHandler creates a handler that blocks (like BackpressureBlock) but signals that chunk sizes should be adjusted. Future phases will integrate throughput-based auto-tuning.
func NewBackpressureHandler ¶
func NewBackpressureHandler(mode Backpressure, pool *ChunkPool) BackpressureHandler
NewBackpressureHandler constructs the appropriate handler for the strategy.
func NewBlockHandler ¶
func NewBlockHandler() BackpressureHandler
NewBlockHandler creates a handler that blocks until the channel has room or the context is cancelled.
func NewBufferHandler ¶
func NewBufferHandler(maxOverflow int) BackpressureHandler
NewBufferHandler creates a handler that spills chunks to an internal slice when the channel is full, up to maxOverflow extra chunks.
func NewDropHandler ¶
func NewDropHandler(pool *ChunkPool) BackpressureHandler
NewDropHandler creates a handler that drops the oldest chunk in the channel when full, returning its buffer to the pool.
type Chunk ¶
type Chunk struct {
// Index is the sequential chunk number within the stream (0-based).
Index int
// Offset is the byte offset of this chunk within the full object.
Offset int64
// Data holds the chunk payload. The backing slice comes from a ChunkPool
// and must be returned after processing.
Data []byte
// Size is the number of valid bytes in Data (may be less than cap(Data)
// for the final chunk).
Size int
// Checksum holds an optional integrity hash for this chunk.
Checksum Checksum
// Final marks this as the last chunk in the stream.
Final bool
}
Chunk is a unit of data in a stream. Buffers are drawn from a ChunkPool and MUST be returned via ChunkPool.Put when no longer needed.
type ChunkPool ¶
type ChunkPool struct {
// contains filtered or unexported fields
}
ChunkPool manages a pool of reusable byte buffers for zero-copy chunk allocation. It wraps sync.Pool with fixed-size buffers and tracks allocation metrics.
func NewChunkPool ¶
NewChunkPool creates a ChunkPool that allocates buffers of the given size.
func (*ChunkPool) Put ¶
Put returns a buffer to the pool for reuse. The buffer is reset to its original capacity (zero-length but full cap).
func (*ChunkPool) Stats ¶
func (cp *ChunkPool) Stats() ChunkPoolStats
Stats returns allocation statistics.
type ChunkPoolStats ¶
type ChunkPoolStats struct {
Allocated int64 `json:"allocated"`
Released int64 `json:"released"`
ChunkSize int64 `json:"chunk_size"`
}
ChunkPoolStats holds chunk pool allocation metrics.
type CompleteHook ¶
type CompleteHook func(*driver.ObjectInfo)
CompleteHook is called when a stream completes successfully.
type ControlMsg ¶
type ControlMsg struct {
Type ControlType
Offset int64 // For CtrlSeek.
}
ControlMsg carries a control command for a stream.
type ControlType ¶
type ControlType int
ControlType identifies a stream control command.
const ( CtrlPause ControlType = iota // Pause the stream. CtrlResume // Resume a paused stream. CtrlCancel // Cancel the stream. CtrlSeek // Seek to a byte offset. )
type Metrics ¶
type Metrics struct {
BytesSent atomic.Int64
BytesRecv atomic.Int64
Chunks atomic.Int64
Retries atomic.Int64
StartTime time.Time
}
Metrics tracks per-stream transfer metrics.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics creates a Metrics with the start time set to now.
func (*Metrics) StreamSnapshot ¶
StreamSnapshot returns a point-in-time copy of stream metrics.
func (*Metrics) Throughput ¶
Throughput returns the average throughput in bytes/sec since start.
type MetricsSnapshot ¶
type MetricsSnapshot struct {
ActiveStreams int64 `json:"active_streams"`
TotalBytes int64 `json:"total_bytes"`
TotalChunks int64 `json:"total_chunks"`
FailedStreams int64 `json:"failed_streams"`
RetryCount int64 `json:"retry_count"`
AvgThroughput float64 `json:"avg_throughput_bps"`
P99Latency time.Duration `json:"p99_latency"`
}
MetricsSnapshot is a serializable point-in-time copy of pool metrics.
type Option ¶
type Option func(*streamConfig)
Option configures a Stream.
func WithBackpressure ¶
func WithBackpressure(mode Backpressure) Option
WithBackpressure sets the backpressure strategy for this stream.
func WithChannelSize ¶
WithChannelSize sets the chunk channel buffer size.
func WithChunkSize ¶
WithChunkSize sets the chunk size for this stream.
func WithOnComplete ¶
func WithOnComplete(fn CompleteHook) Option
WithOnComplete registers a completion callback.
func WithOnProgress ¶
func WithOnProgress(fn ProgressHook) Option
WithOnProgress registers a progress callback.
func WithResumable ¶
func WithResumable() Option
WithResumable enables resumable upload/download with offset tracking.
type Pool ¶
type Pool struct {
ID id.ID
Name string
Config PoolConfig
Metrics *PoolMetrics
// contains filtered or unexported fields
}
Pool manages a set of concurrent streams with shared resources, concurrency limits, and bandwidth throttling.
func NewPool ¶
func NewPool(name string, cfg PoolConfig) *Pool
NewPool creates a new streaming pool.
func (*Pool) Acquire ¶
func (p *Pool) Acquire(ctx context.Context, dir Direction, bucket, key string, opts ...Option) (*Stream, error)
Acquire reserves a concurrency slot and creates a new stream within the pool. Blocks until a slot is available or the context is cancelled.
func (*Pool) ActiveCount ¶
ActiveCount returns the number of currently active streams.
func (*Pool) Close ¶
Close gracefully shuts down the pool. It cancels the pool context, waits for all active streams to be released, and returns.
type PoolConfig ¶
type PoolConfig struct {
// MaxStreams is the maximum number of concurrent streams.
MaxStreams int `json:"max_streams" yaml:"max_streams"`
// MaxBandwidth limits total throughput in bytes/sec (0 = unlimited).
MaxBandwidth int64 `json:"max_bandwidth" yaml:"max_bandwidth"`
// ChunkSize is the default chunk size for streams in this pool.
ChunkSize int64 `json:"chunk_size" yaml:"chunk_size"`
// BufferCount is the initial number of buffers in the chunk pool.
BufferCount int `json:"buffer_count" yaml:"buffer_count"`
// StreamTimeout is the maximum lifetime of a single stream.
StreamTimeout time.Duration `json:"stream_timeout" yaml:"stream_timeout"`
// IdleTimeout closes a stream that has been idle for this duration.
IdleTimeout time.Duration `json:"idle_timeout" yaml:"idle_timeout"`
// BackpressureMode is the default backpressure strategy for streams.
BackpressureMode Backpressure `json:"backpressure" yaml:"backpressure"`
}
PoolConfig configures a streaming pool.
func DefaultPoolConfig ¶
func DefaultPoolConfig() PoolConfig
DefaultPoolConfig returns a PoolConfig with sensible defaults.
type PoolMetrics ¶
type PoolMetrics struct {
ActiveStreams atomic.Int64
TotalBytes atomic.Int64
TotalChunks atomic.Int64
FailedStreams atomic.Int64
RetryCount atomic.Int64
// contains filtered or unexported fields
}
PoolMetrics tracks streaming performance for a Pool.
func (*PoolMetrics) AvgThroughput ¶
func (m *PoolMetrics) AvgThroughput() float64
AvgThroughput returns the rolling average throughput (bytes/sec).
func (*PoolMetrics) P99Latency ¶
func (m *PoolMetrics) P99Latency() time.Duration
P99Latency returns the P99 chunk latency.
func (*PoolMetrics) RecordBytes ¶
func (m *PoolMetrics) RecordBytes(n int64)
RecordBytes adds n bytes to the cumulative total.
func (*PoolMetrics) RecordChunk ¶
func (m *PoolMetrics) RecordChunk()
RecordChunk increments the chunk counter.
func (*PoolMetrics) RecordFailure ¶
func (m *PoolMetrics) RecordFailure()
RecordFailure increments the failed stream counter.
func (*PoolMetrics) RecordRetry ¶
func (m *PoolMetrics) RecordRetry()
RecordRetry increments the retry counter.
func (*PoolMetrics) SetAvgThroughput ¶
func (m *PoolMetrics) SetAvgThroughput(bps float64)
SetAvgThroughput stores the rolling average throughput (bytes/sec).
func (*PoolMetrics) SetP99Latency ¶
func (m *PoolMetrics) SetP99Latency(d time.Duration)
SetP99Latency stores the P99 chunk latency.
func (*PoolMetrics) Snapshot ¶
func (m *PoolMetrics) Snapshot() MetricsSnapshot
Snapshot returns a point-in-time copy of all metrics.
type Progress ¶
type Progress struct {
StreamID id.ID
BytesSent int64
BytesRecv int64
TotalSize int64
Percent int
Speed float64 // bytes/sec
}
Progress holds a point-in-time progress report for a stream.
type ProgressHook ¶
type ProgressHook func(Progress)
ProgressHook is called periodically with transfer progress.
type Snapshot ¶
type Snapshot struct {
BytesSent int64 `json:"bytes_sent"`
BytesRecv int64 `json:"bytes_recv"`
Chunks int64 `json:"chunks"`
Retries int64 `json:"retries"`
Throughput float64 `json:"throughput_bps"`
Elapsed time.Duration `json:"elapsed"`
}
Snapshot is a serializable copy of per-stream metrics.
type State ¶
type State string
State represents the lifecycle state of a Stream.
func (State) IsTerminal ¶
IsTerminal reports whether this state is a final state.
type Stream ¶
type Stream struct {
ID id.ID
Direction Direction
Bucket string
Key string
// contains filtered or unexported fields
}
Stream represents a single managed data channel for transferring an object between a client and storage. It provides chunked transfer, backpressure, resumability, and lifecycle hooks.
func NewStream ¶
func NewStream( ctx context.Context, dir Direction, bucket, key string, chunkPool *ChunkPool, opts ...Option, ) *Stream
NewStream creates a new stream in the Idle state.
func (*Stream) BackpressureMode ¶
func (s *Stream) BackpressureMode() Backpressure
Config returns the stream's backpressure mode.
func (*Stream) Close ¶
func (s *Stream) Close()
Close releases all stream resources. Safe to call multiple times.
func (*Stream) Complete ¶
func (s *Stream) Complete(info *driver.ObjectInfo) error
Complete transitions the stream through completing → completed. It closes the chunk channel to signal EOF to the reader and fires completion hooks.
func (*Stream) Control ¶
func (s *Stream) Control(msg ControlMsg) error
Control sends a control command to the stream.
func (*Stream) ControlChan ¶
func (s *Stream) ControlChan() <-chan ControlMsg
ControlChan returns the read-only control channel for consuming control messages in the stream's processing loop.
func (*Stream) Done ¶
func (s *Stream) Done() <-chan struct{}
Done returns a channel that is closed when the stream's context is cancelled (either by completion, failure, or explicit cancel).
func (*Stream) Read ¶
Read receives the next chunk from the stream. Blocks until a chunk is available or the context is cancelled.
func (*Stream) ReadAck ¶
ReadAck receives the next ack. Blocks until available or context cancelled.
func (*Stream) SetTotalSize ¶
SetTotalSize sets the expected total size of the transfer. This enables percentage-based progress reporting.