Documentation
¶
Index ¶
- Variables
- type PieceReader
- type SpanConfig
- type StorageBlockStore
- func (s *StorageBlockStore) AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)
- func (s *StorageBlockStore) Close()
- func (s *StorageBlockStore) DeleteBlock(ctx context.Context, c cid.Cid) error
- func (s *StorageBlockStore) Get(ctx context.Context, c cid.Cid) (blocks.Block, error)
- func (s *StorageBlockStore) GetSize(ctx context.Context, c cid.Cid) (int, error)
- func (s *StorageBlockStore) Has(ctx context.Context, c cid.Cid) (bool, error)
- func (s *StorageBlockStore) HashOnRead(enabled bool)
- func (s *StorageBlockStore) Put(ctx context.Context, block blocks.Block) error
- func (s *StorageBlockStore) PutMany(ctx context.Context, blks []blocks.Block) error
Constants ¶
This section is empty.
Variables ¶
var ( ErrNoCarBlocks = errors.New("no Blocks provided") ErrInvalidStartOffset = errors.New("first block must start at car Header") ErrInvalidEndOffset = errors.New("last block must end at car end") ErrIncontiguousBlocks = errors.New("Blocks must be contiguous") ErrInvalidVarintLength = errors.New("varint read does not match varint length") ErrVarintDoesNotMatchBlockLength = errors.New("varint does not match block length") ErrFileNotProvided = errors.New("file not provided") ErrInvalidWhence = errors.New("invalid whence") ErrNegativeOffset = errors.New("negative offset") ErrOffsetOutOfRange = errors.New("position past end of file") ErrTruncated = errors.New("original file has been truncated") ErrFileHasChanged = errors.New("file has changed") )
Functions ¶
This section is empty.
Types ¶
type PieceReader ¶
type PieceReader struct {
// contains filtered or unexported fields
}
PieceReader is a struct that represents a reader for pieces of data.
Fields:
- ctx: The context in which the PieceReader operates. This can be used to cancel operations or set deadlines.
- fileSize: The size of the file being read.
- header: A byte slice representing the header of the file.
- handlerMap: Handler map from storage ID to the actual RCloneHandler
- carBlocks: A slice of CarBlocks. These represent the blocks of data in the CAR (Content Addressable Archive) format.
- files: A map where the keys are file ID. This represents the files of data being read.
- reader: An io.ReadCloser that is used to read the data and close the reader when done.
- readerFor: A uint64 file ID that represents the current file being read.
- pos: An int64 that represents the current position in the data being read.
- blockIndex: An integer that represents the index of the current block being read.
func NewPieceReader ¶
func NewPieceReader( ctx context.Context, car model.Car, storage model.Storage, carBlocks []model.CarBlock, files []model.File, ) ( *PieceReader, error, )
NewPieceReader is a function that creates a new PieceReader. It takes a context, a Car model, a Storage model, a slice of CarBlock models, and a slice of File models as input. It validates the input data and returns an error if any of it is invalid. The returned PieceReader starts at the beginning of the data (position 0).
Parameters:
- ctx: The context for the new PieceReader. This can be used to cancel operations or set deadlines.
- car: A Car model that represents the CAR (Content Addressable Archive) file being read.
- storage: A Storage model that represents the storage backend holding the data.
- carBlocks: A slice of CarBlock models that represent the blocks of data in the CAR file.
- files: A slice of File models that represent the files of data being read.
Returns:
- A new PieceReader that has been initialized with the provided data, and an error if the initialization failed.
func (*PieceReader) Clone ¶ added in v0.2.14
func (pr *PieceReader) Clone() *PieceReader
Clone is a method on the PieceReader struct that creates a new PieceReader with the same state as the original. The new PieceReader starts at the beginning of the data (position 0).
Returns:
- A new PieceReader that has the same state as the original, but starting at position 0.
func (*PieceReader) Close ¶
func (pr *PieceReader) Close() error
func (*PieceReader) Read ¶
func (pr *PieceReader) Read(p []byte) (n int, err error)
Read is a method on the PieceReader struct that reads data into the provided byte slice.
- It reads data from the current position of the PieceReader and advances the position accordingly.
- If the context of the PieceReader has been cancelled, it returns an error immediately.
- If the end of the file has been reached, it returns io.EOF.
- If the PieceReader is currently at a block boundary, it advances to the next block before reading data.
- If the PieceReader is currently at a varint or CID boundary within a block, it reads the varint or CID data.
- If the PieceReader is currently at a raw block boundary within a block, it reads the raw block data.
- If the PieceReader is currently at an file boundary within a block, it reads the file data.
- If the PieceReader encounters an error while reading data, it returns the error.
Parameters:
- p: The byte slice to read data into.
Returns:
- The number of bytes read, and an error if the read operation failed.
func (*PieceReader) Seek ¶ added in v0.2.13
func (pr *PieceReader) Seek(offset int64, whence int) (int64, error)
Seek is a method on the PieceReader struct that changes the position of the reader. It takes an offset and a 'whence' value as input, similar to the standard io.Seeker interface. The offset is added to the position determined by 'whence'.
- If 'whence' is io.SeekStart, the offset is from the start of the file.
- If 'whence' is io.SeekCurrent, the offset is from the current position.
- If 'whence' is io.SeekEnd, the offset is from the end of the file.
- If the resulting position is negative or beyond the end of the file, an error is returned.
- If a reader is currently open, it is closed before the position is changed.
Parameters:
- offset: The offset to move the position by. Can be negative.
- whence: The position to move the offset from. Must be one of io.SeekStart, io.SeekCurrent, or io.SeekEnd.
Returns:
- The new position after seeking, and an error if the seek operation failed.
type SpanConfig ¶ added in v1.0.0
type SpanConfig struct {
// SpanBlocks is the number of consecutive blocks fetched per backend read.
SpanBlocks int
// PrefetchSpans is the number of spans prefetched ahead once access to a
// file looks sequential.
PrefetchSpans int
// MaxBackendReads caps concurrent backend reads across all requests.
MaxBackendReads int
// CacheBlocks is the block cache capacity in blocks. When <= 0 it is
// derived from the other settings.
CacheBlocks int
// ReadTimeout bounds a single backend span read once it holds a
// connection slot, so a hung backend cannot pin slots indefinitely.
ReadTimeout time.Duration
}
SpanConfig tunes the span-read behavior of StorageBlockStore. Blocks are ~1MiB (pack chunk size), so block counts are roughly MiB.
type StorageBlockStore ¶ added in v1.0.0
type StorageBlockStore struct {
// contains filtered or unexported fields
}
StorageBlockStore is a blockstore backed by the singularity database and rclone storage backends.
DAG nodes (directory structure, file roots) are stored inline in the DB and returned without any storage I/O. File-backed leaf blocks are read from source files via ranged reads covering a span of consecutive blocks; spans land in a bounded LRU cache so the sequential Gets that follow are served from memory. Sequential access additionally prefetches the next spans in parallel, which is what provides per-client throughput beyond the backend's per-connection rate. There are no held-open streams and no global lock: concurrency is bounded only by the backend-read semaphore.
func NewStorageBlockStore ¶ added in v1.0.0
func NewStorageBlockStore(db *gorm.DB, cfg SpanConfig) *StorageBlockStore
func (*StorageBlockStore) AllKeysChan ¶ added in v1.0.0
func (*StorageBlockStore) Close ¶ added in v1.0.0
func (s *StorageBlockStore) Close()
Close cancels in-flight background reads; they exit on context cancellation. rclone handler cleanup is not implemented -- backends may hold connections (e.g. SFTP) but RCloneHandler doesn't expose Shutdown.
func (*StorageBlockStore) DeleteBlock ¶ added in v1.0.0
func (*StorageBlockStore) HashOnRead ¶ added in v1.0.0
func (s *StorageBlockStore) HashOnRead(enabled bool)
Source Files