Documentation
¶
Index ¶
- Variables
- func BuildSegmentRange(ctx context.Context, start, end int64, ml SegmentLoader, startSegIdx int, ...) *segmentRange
- func CheckMetadataIntegrity(fileSize int64, ml SegmentLoader) error
- func GetSegmentsInRange(ctx context.Context, start, end int64, ml SegmentLoader) *segmentRange
- func GetSegmentsInRangeFromIndex(ctx context.Context, start, end int64, ml SegmentLoader, startSegmentIndex int, ...) *segmentRange
- func IsArticleNotFound(err error) bool
- func NewLazySegmentRange(ctx context.Context, start, end int64, ml SegmentLoader, startSegIdx int, ...) *segmentRange
- func SelectSegmentsForValidation(segments []*metapb.SegmentData, samplePercentage int) []*metapb.SegmentData
- type ConnBudget
- type DataCorruptionError
- type HoleHooks
- type MetricsTracker
- type MissingSegment
- type ReaderOption
- type Segment
- type SegmentLoader
- type SegmentStore
- type UsenetReader
- type ValidationResult
Constants ¶
This section is empty.
Variables ¶
var ( ErrBufferNotReady = errors.New("buffer not ready") ErrSegmentLimit = errors.New("segment limit reached") )
Functions ¶
func BuildSegmentRange ¶
func BuildSegmentRange(ctx context.Context, start, end int64, ml SegmentLoader, startSegIdx int, startFilePos int64, endSegIdx int, endFilePos int64, ) *segmentRange
BuildSegmentRange returns a lazy segment range when index bounds are provided (endSegIdx >= 0), giving O(1) initialization. Otherwise it falls back to the eager O(N) path via GetSegmentsInRangeFromIndex.
func CheckMetadataIntegrity ¶
func CheckMetadataIntegrity(fileSize int64, ml SegmentLoader) error
CheckMetadataIntegrity verifies that the segments provided by the loader cover the entire file range [0, fileSize-1] without any gaps. Returns an error if any part of the file is not covered by segment data.
func GetSegmentsInRange ¶
func GetSegmentsInRange(ctx context.Context, start, end int64, ml SegmentLoader) *segmentRange
GetSegmentsInRange returns a segmentRange representing the requested byte range [start,end] across the underlying ordered segments provided by the SegmentLoader. Behaviour / rules:
- start and end are inclusive; caller guarantees 0 <= start <= end < filesize (filesize not passed here)
- Each loader Segment indicates:
- Start: offset (in bytes) within the physical segment where valid data for the file begins (can be > 0)
- End: offset (in bytes) within the physical segment where valid data for the file ends (inclusive)
- Size: full physical segment size (in bytes) for downloading Therefore the usable data length contributed to the logical file by a loader segment is (End - Start + 1).
- We build output *segment objects (internal) with Start & End (inclusive) relative to the physical segment so the reader will skip Start bytes and read up to End+1 bytes.
- First and last returned segments are trimmed so the concatenation of (End-Start+1) bytes across returned segments equals the requested range length (unless the range lies fully outside available data, in which case zero segments are returned).
func GetSegmentsInRangeFromIndex ¶
func GetSegmentsInRangeFromIndex(ctx context.Context, start, end int64, ml SegmentLoader, startSegmentIndex int, startFilePos int64) *segmentRange
GetSegmentsInRangeFromIndex is like GetSegmentsInRange but allows skipping directly to a known segment index with its corresponding file position. This enables O(1) skip to the correct segment when a pre-built index provides the starting segment via binary search.
Parameters:
- startSegmentIndex: the segment index to start iteration from (use 0 for beginning)
- startFilePos: the cumulative file offset at the start of startSegmentIndex (use 0 for beginning)
func IsArticleNotFound ¶ added in v0.3.0
IsArticleNotFound reports whether err stems from an article missing on all providers (permanent, never retried) — the only failure the hole model treats as a hole.
func NewLazySegmentRange ¶
func NewLazySegmentRange(ctx context.Context, start, end int64, ml SegmentLoader, startSegIdx int, startFilePos int64, endSegIdx int, endFilePos int64, ) *segmentRange
NewLazySegmentRange creates a segmentRange that defers segment object creation until GetSegment() is called by the download manager. This is O(1) regardless of segment count, compared to O(N) for the eager GetSegmentsInRangeFromIndex path.
Parameters:
- start, end: inclusive byte range in file coordinates
- ml: segment loader for on-demand segment info retrieval
- startSegIdx: loader index of the first segment covering 'start'
- startFilePos: cumulative file offset at the start of startSegIdx's usable data
- endSegIdx: loader index of the last segment covering 'end'
- endFilePos: cumulative file offset at the start of endSegIdx's usable data
func SelectSegmentsForValidation ¶
func SelectSegmentsForValidation(segments []*metapb.SegmentData, samplePercentage int) []*metapb.SegmentData
SelectSegmentsForValidation is the exported form of the sampling selector. It returns the subset of segments that should be validated based on samplePercentage, applying the same first-3 / last-2 / random-middle strategy used internally.
Types ¶
type ConnBudget ¶ added in v0.3.0
type ConnBudget interface {
AcquireImportConnection(ctx context.Context) (release func(), err error)
}
ConnBudget grants connection tokens for import segment fetches. Implemented by pool.Manager (AcquireImportConnection).
type DataCorruptionError ¶
type DataCorruptionError struct {
UnderlyingErr error
BytesRead int64
NoRetry bool
// FileOffset is the absolute file-coordinate position where the failure
// surfaced (-1 when unknown), enabling playback-impact classification.
FileOffset int64
// SegmentID is the message ID of the failing segment, when known.
SegmentID string
}
func (*DataCorruptionError) Error ¶
func (e *DataCorruptionError) Error() string
func (*DataCorruptionError) Unwrap ¶
func (e *DataCorruptionError) Unwrap() error
type HoleHooks ¶ added in v0.3.0
type HoleHooks struct {
// OnHole returns the pad/fail decision for a missing segment, identified
// by its index in the file's segment space.
OnHole func(segIndex int, segID string) holes.Decision
// KnownHoles reports segments already known missing: those are
// zero-filled immediately, without any fetch (replay pre-pad).
KnownHoles func(segIndex int) bool
}
HoleHooks lets the owner of a reader decide, synchronously, what happens when a segment is confirmed missing (ErrArticleNotFound — never retried). The reader stays dumb: it asks, the owner accounts, persists and transitions health status. Segments approved for padding are zero-filled in place so the read loop never sees an error and playback continues. Both callbacks run on download goroutines: they must be concurrency-safe and fast (no network).
type MetricsTracker ¶
type MissingSegment ¶ added in v0.3.0
type MissingSegment struct {
Index int // index into the original segments slice
ID string // Usenet message ID
Start int64 // inclusive file-coordinate byte range
End int64
}
MissingSegment identifies one unavailable segment together with the byte range it covers in file coordinates, enabling playback-impact classification.
type ReaderOption ¶ added in v0.3.0
type ReaderOption func(*UsenetReader)
ReaderOption customizes a UsenetReader.
func WithHoleHooks ¶ added in v0.3.0
func WithHoleHooks(h *HoleHooks) ReaderOption
WithHoleHooks enables zero-filling of confirmed-missing segments under the owner's control. Without it, a missing segment fails the read as always.
func WithImportProfile ¶ added in v0.3.0
func WithImportProfile(budget ConnBudget) ReaderOption
WithImportProfile marks the reader as import-owned: segment fetches use the pool's normal request lane (so they always yield to streaming reads, which use the priority lane) and each fetch is gated by the global import connection budget. Without this option the reader behaves as a streaming reader: priority lane, no budget. A nil budget only switches the lane.
type SegmentLoader ¶
type SegmentStore ¶
type SegmentStore interface {
Get(messageID string) ([]byte, bool)
Put(messageID string, data []byte) error
}
SegmentStore is an optional cache for decoded segment data. Implementations must be safe for concurrent use.
type UsenetReader ¶
type UsenetReader struct {
// contains filtered or unexported fields
}
func NewUsenetReader ¶
func NewUsenetReader( ctx context.Context, poolGetter func() (pool.NntpClient, error), rg *segmentRange, maxPrefetch int, metricsTracker MetricsTracker, streamID string, segmentStore SegmentStore, opts ...ReaderOption, ) (*UsenetReader, error)
func (*UsenetReader) Close ¶
func (b *UsenetReader) Close() error
func (*UsenetReader) GetBufferedOffset ¶
func (b *UsenetReader) GetBufferedOffset() int64
func (*UsenetReader) Interrupt ¶ added in v0.3.0
func (b *UsenetReader) Interrupt()
Interrupt cancels the reader's context and signals any blocked Read to return. Non-blocking and idempotent; safe to call concurrently with Read or Close. The caller is still responsible for invoking Close to release goroutines and resources. Used by callers (like MetadataVirtualFile.Close) that need to abort an in-flight download without taking the file's own lock.
func (*UsenetReader) Read ¶
func (b *UsenetReader) Read(p []byte) (int, error)
Read reads len(p) byte from the Buffer starting at the current offset. It returns the number of bytes read and an error if any. Returns io.EOF error if pointer is at the end of the Buffer.
func (*UsenetReader) Start ¶
func (b *UsenetReader) Start()
Start triggers the background download process manually. This is useful for pre-fetching data before the first Read call.
type ValidationResult ¶
type ValidationResult struct {
TotalChecked int
MissingCount int
MissingIDs []string
MissingSegments []MissingSegment // same 50-entry cap as MissingIDs
}
ValidationResult holds detailed validation results
func ValidateSegmentAvailabilityBatch ¶ added in v0.3.0
func ValidateSegmentAvailabilityBatch( ctx context.Context, perFileIDs [][]string, poolManager pool.Manager, maxConnections int, timeout time.Duration, ) ([]ValidationResult, error)
ValidateSegmentAvailabilityBatch checks pre-sampled segment IDs for many files in a single StatMany sweep. perFileIDs is index-aligned with the returned results: files with an empty ID list yield a zero ValidationResult. IDs are interleaved round-robin across files (every file's first sample, then every file's second, …) so one file with many segments cannot serialize the sweep for the others. An error is returned only for infrastructure failures (pool unavailable); per-segment misses are reported in the per-file results.
func ValidateSegmentAvailabilityDetailed ¶
func ValidateSegmentAvailabilityDetailed( ctx context.Context, segments []*metapb.SegmentData, poolManager pool.Manager, maxConnections int, samplePercentage int, progressTracker progress.ProgressTracker, timeout time.Duration, ) (ValidationResult, error)
ValidateSegmentAvailabilityDetailed validates segments and returns detailed results instead of failing fast on the first error.
Source Files