blockstore

package
v0.42.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 22, 2026 License: Apache-2.0, MIT Imports: 21 Imported by: 214

Documentation

Overview

Package blockstore implements a thin wrapper over a datastore, giving a clean interface for Getting and Putting block objects.

Index

Constants

This section is empty.

Variables

View Source
var BlockPrefix = ds.NewKey("blocks")

BlockPrefix namespaces blockstore datastores

View Source
var ErrHashMismatch = errors.New("block in storage has different hash than requested")

ErrHashMismatch is an error returned when the hash of a block is different than expected.

Functions

This section is empty.

Types

type AllKeysChanWithErrer added in v0.42.0

type AllKeysChanWithErrer interface {
	AllKeysChanWithErr(ctx context.Context) (<-chan cid.Cid, func() error, error)
}

AllKeysChanWithErrer is an optional capability a Blockstore may implement alongside AllKeysChan. AllKeysChanWithErr behaves like Blockstore.AllKeysChan, but additionally returns a function that, once the returned channel has been fully drained, reports any error that terminated enumeration early.

The reported error is nil if enumeration ran to completion without a mid-iteration error or cancellation; a non-nil error means enumeration was truncated and the delivered keys must not be treated as the complete set. Datastore keys that cannot be parsed as a block key are skipped without being treated as an error: such a key cannot correspond to a retrievable block, so omitting it cannot cause a Bloom-filter false negative.

The function blocks until enumeration finishes, so it must be called only after the channel has been drained; calling it earlier deadlocks the caller once the producer fills the channel buffer.

type Blockstore

type Blockstore interface {
	DeleteBlock(context.Context, cid.Cid) error
	Has(context.Context, cid.Cid) (bool, error)
	Get(context.Context, cid.Cid) (blocks.Block, error)

	// GetSize returns the CIDs mapped BlockSize
	GetSize(context.Context, cid.Cid) (int, error)

	// Put puts a given block to the underlying datastore
	Put(context.Context, blocks.Block) error

	// PutMany puts a slice of blocks at the same time using batching
	// capabilities of the underlying datastore whenever possible.
	PutMany(context.Context, []blocks.Block) error

	// AllKeysChan returns a channel from which
	// the CIDs in the Blockstore can be read. It should respect
	// the given context, closing the channel if it becomes Done.
	//
	// AllKeysChan treats the underlying blockstore as a set, and returns that
	// set in full. The only guarantee is that the consumer of AKC will
	// encounter every CID in the underlying set, at least once. If the
	// underlying blockstore supports duplicate CIDs it is up to the
	// implementation to elect to return such duplicates or not. Similarly no
	// guarantees are made regarding CID ordering.
	//
	// When underlying blockstore is operating on Multihash and codec information
	// is not preserved, returned CIDs will use Raw (0x55) codec.
	//
	// If enumeration fails partway (for example an I/O error mid-iteration or a
	// cancelled context), the channel may be closed early without warning.
	// Callers MUST NOT assume the returned error reflects enumeration
	// completeness: it is only guaranteed to cover query setup, not
	// mid-iteration failures. Consumers that require a complete enumeration
	// (such as building a Bloom filter) should check whether the blockstore
	// implements [AllKeysChanWithErrer].
	AllKeysChan(ctx context.Context) (<-chan cid.Cid, error)
}

Blockstore wraps a Datastore block-centered methods and provides a layer of abstraction which allows to add different caching strategies.

func CachedBlockstore

func CachedBlockstore(
	ctx context.Context,
	bs Blockstore,
	opts CacheOpts,
) (cbs Blockstore, err error)

CachedBlockstore returns a blockstore wrapped in an TwoQueueCache and then in a bloom filter cache, if the options indicate it.

When a Bloom filter is configured, the returned Blockstore implements BloomCacheStatus, which can be used to wait for and check the result of the initial Bloom filter build.

func NewBlockstore

func NewBlockstore(d ds.Batching, opts ...Option) Blockstore

NewBlockstore returns a default Blockstore implementation using the provided datastore.Batching backend.

func NewBlockstoreNoPrefix deprecated

func NewBlockstoreNoPrefix(d ds.Batching) Blockstore

NewBlockstoreNoPrefix returns a default Blockstore implementation using the provided datastore.Batching backend. This constructor does not modify input keys in any way

Deprecated: Use NewBlockstore with the NoPrefix option instead.

func NewIdStore

func NewIdStore(bs Blockstore) Blockstore

type BloomCacheStatus added in v0.42.0

type BloomCacheStatus interface {
	// Wait blocks until the initial Bloom filter build finishes, or until ctx
	// is done, and returns that build's error, if any. It reflects only the
	// initial build: a later Rebuild does not update it, so after calling
	// Rebuild use that method's return value and BloomActive instead. A non-nil
	// initial-build error means the filter is not active; the blockstore still
	// returns correct answers, just without Bloom acceleration.
	Wait(ctx context.Context) error

	// BloomActive reports whether the Bloom filter finished building
	// successfully and is being used to answer negative lookups.
	BloomActive() bool

	// Rebuild discards the current Bloom filter and rebuilds it from a full
	// enumeration of the blockstore, returning the build error if any. It is
	// meant to retry after a failed initial build (Wait returned an error).
	// While it runs, the filter is inactive and lookups fall through to the
	// underlying blockstore, so results stay correct but unaccelerated; on a
	// failed rebuild the filter is left inactive.
	//
	// Rebuild serializes with the initial build and concurrent Rebuild calls;
	// it waits for any in-progress build to finish before starting (that wait
	// does not observe ctx), then honors ctx for the enumeration itself.
	//
	// Reliable rebuild while the store is written concurrently assumes the
	// datastore's enumeration reflects all writes that completed before it
	// began; a backend without that property (e.g. a lazy directory walk) may
	// leave a block written during the rebuild as a transient false negative
	// until the next rebuild.
	Rebuild(ctx context.Context) error
}

BloomCacheStatus may be implemented by the Blockstore returned from CachedBlockstore when a Bloom filter is configured (HasBloomFilterSize > 0). It lets callers observe the initial, asynchronous Bloom filter build:

cbs, err := CachedBlockstore(ctx, bs, opts)
if err != nil {
	// handle err
}
if s, ok := cbs.(BloomCacheStatus); ok {
	if err := s.Wait(ctx); err != nil {
		// The filter is not active: the blockstore still answers
		// correctly, but without Bloom-filter acceleration.
	}
}

type CacheOpts

type CacheOpts struct {
	HasBloomFilterSize   int // 1 byte
	HasBloomFilterHashes int // No size, 7 is usually best, consult bloom papers
	HasTwoQueueCacheSize int // 32 bytes
}

CacheOpts wraps options for CachedBlockStore(). Next to each option is it approximate memory usage per unit

func DefaultCacheOpts

func DefaultCacheOpts() CacheOpts

DefaultCacheOpts returns a CacheOpts initialized with default values.

type GCBlockstore

type GCBlockstore interface {
	Blockstore
	GCLocker
}

GCBlockstore is a blockstore that can safely run garbage-collection operations.

func NewGCBlockstore

func NewGCBlockstore(bs Blockstore, gcl GCLocker) GCBlockstore

NewGCBlockstore returns a default implementation of GCBlockstore using the given Blockstore and GCLocker.

type GCLocker

type GCLocker interface {
	// GCLock locks the blockstore for garbage collection. No operations
	// that expect to finish with a pin should occur simultaneously.
	// Reading during GC is safe, and requires no lock.
	GCLock(context.Context) Unlocker

	// PinLock locks the blockstore for sequences of puts expected to finish
	// with a pin (before GC). Multiple put->pin sequences can write through
	// at the same time, but no GC should happen simultaneously.
	// Reading during Pinning is safe, and requires no lock.
	PinLock(context.Context) Unlocker

	// GcRequested returns true if GCLock has been called and is waiting to
	// take the lock
	GCRequested(context.Context) bool
}

GCLocker abstract functionality to lock a blockstore when performing garbage-collection operations.

func NewGCLocker

func NewGCLocker() GCLocker

NewGCLocker returns a default implementation of GCLocker using standard [RW] mutexes.

type Option

type Option struct {
	// contains filtered or unexported fields
}

Option is a default implementation Blockstore option

func NoPrefix

func NoPrefix() Option

NoPrefix avoids wrapping the blockstore into the BlockPrefix namespace ("/blocks"), so keys will not be modified in any way.

func Provider added in v0.34.0

func Provider(provider provider.MultihashProvider) Option

Provider allows performing a StartProvide operation for every block written.

func WriteThrough

func WriteThrough(enabled bool) Option

WriteThrough skips checking if the blockstore already has a block before writing it, when enabled.

type Unlocker

type Unlocker interface {
	Unlock(context.Context)
}

Unlocker represents an object which can Unlock something.

type ValidatingBlockstore added in v0.34.0

type ValidatingBlockstore struct {
	Blockstore
}

ValidatingBlockstore validates blocks on get.

func (*ValidatingBlockstore) AllKeysChanWithErr added in v0.42.0

func (bs *ValidatingBlockstore) AllKeysChanWithErr(ctx context.Context) (<-chan cid.Cid, func() error, error)

func (*ValidatingBlockstore) Get added in v0.34.0

type Viewer

type Viewer interface {
	View(ctx context.Context, cid cid.Cid, callback func([]byte) error) error
}

Viewer can be implemented by blockstores that offer zero-copy access to values.

Callers of View must not mutate or retain the byte slice, as it could be an mmapped memory region, or a pooled byte buffer.

View is especially suitable for deserialising in place.

The callback will only be called iff the query operation is successful (and the block is found); otherwise, the error will be propagated. Errors returned by the callback will be propagated as well.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL