block

package
v0.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package block defines a common pattern for interacting with block reference structures in Hydra and in memory.

Index

Constants

View Source 
const (
	// StoreFeatureNativeBatchPut means PutBlockBatch is implemented natively.
	StoreFeatureNativeBatchPut = StoreFeature_STORE_FEATURE_NATIVE_BATCH_PUT
	// StoreFeatureNativeBatchExists means GetBlockExistsBatch is implemented natively.
	StoreFeatureNativeBatchExists = StoreFeature_STORE_FEATURE_NATIVE_BATCH_EXISTS
	// StoreFeatureSelfBuffered means the store has its own read-through pending
	// buffer, so the world block engine must not wrap it in a BufferedStore.
	// Volume remains the durability barrier owner for volume-backed stores.
	StoreFeatureSelfBuffered = StoreFeature_STORE_FEATURE_SELF_BUFFERED
)
View Source 
const (
	// DecodedBlockCacheNoTransformKey identifies untransformed decoded-cache entries.
	DecodedBlockCacheNoTransformKey = "transform:none"
)
View Source 
const (
	// DefaultDecodedBlockCacheMaxCost is the default decoded-object cache budget.
	DefaultDecodedBlockCacheMaxCost int64 = 1_000_000_000
)
View Source 
const (
	// DefaultHashType is the hash type for new refs when callers do not choose one.
	DefaultHashType = hash.HashType_HashType_SHA256
)
View Source 
const MaxBlockSize = 10485760

MaxBlockSize is the default maximum size in bytes (10MB) for a single serialized block accepted on the wire by transports such as DEX.

This is a sanity / DoS ceiling, not a structural limit on any one block type. Individual block types are naturally bounded well below this value:

  • blob.Blob (root): inline RawData is capped by blob.DefRawHighWaterMark = blob.DefChunkingMaxSize = 786432 (768 KiB). Above that the Blob auto-converts to CHUNKED, where the root only stores a ChunkIndex of references and individual chunk data lives in separate byteslice blocks.
  • blob chunk data (byteslice.ByteSlice): one chunk per block, capped by blob.DefChunkingMaxSize = 786432 (768 KiB).
  • blob.ChunkIndex / IAVL node / other index blocks: hold only references and small metadata, well under 1 MiB in practice.

10 MiB therefore leaves roughly an order of magnitude of headroom over the largest block any current producer will emit, while still bounding the per-message buffer a peer must allocate when receiving a single block.

Variables

View Source 
var (
	StoreFeature_name = map[int32]string{
		0:  "STORE_FEATURE_UNKNOWN",
		1:  "STORE_FEATURE_NATIVE_BATCH_PUT",
		2:  "STORE_FEATURE_NATIVE_BATCH_EXISTS",
		32: "STORE_FEATURE_SELF_BUFFERED",
	}
	StoreFeature_value = map[string]int32{
		"STORE_FEATURE_UNKNOWN":             0,
		"STORE_FEATURE_NATIVE_BATCH_PUT":    1,
		"STORE_FEATURE_NATIVE_BATCH_EXISTS": 2,
		"STORE_FEATURE_SELF_BUFFERED":       32,
	}
)

Enum value maps for StoreFeature.

View Source 
var (
	OverlayMode_name = map[int32]string{
		0: "UPPER_ONLY",
		1: "LOWER_ONLY",
		2: "UPPER_CACHE",
		3: "LOWER_CACHE",
		4: "UPPER_READ_CACHE",
		5: "LOWER_READ_CACHE",
		6: "UPPER_WRITE_CACHE",
		7: "LOWER_WRITE_CACHE",
		8: "UPPER_READBACK_CACHE",
	}
	OverlayMode_value = map[string]int32{
		"UPPER_ONLY":           0,
		"LOWER_ONLY":           1,
		"UPPER_CACHE":          2,
		"LOWER_CACHE":          3,
		"UPPER_READ_CACHE":     4,
		"LOWER_READ_CACHE":     5,
		"UPPER_WRITE_CACHE":    6,
		"LOWER_WRITE_CACHE":    7,
		"UPPER_READBACK_CACHE": 8,
	}
)

Enum value maps for OverlayMode.

View Source 
var (
	// ErrBlockStoreUnavailable is returned when Fetch is called against a nil block store.
	ErrBlockStoreUnavailable = errors.New("block store is unavailable")
	// ErrUnexpectedType is returned if a type assertion failed.
	ErrUnexpectedType = errors.New("block: unexpected object type")
	// ErrNilCursor is returned when a non-nil block cursor is required.
	ErrNilCursor = errors.New("block cursor cannot be nil")
	// ErrNilBlock is returned when a non-nil block is required.
	ErrNilBlock = errors.New("block cannot be nil")
	// ErrEmptyBlock is returned when a non-empty block is required.
	ErrEmptyBlock = errors.New("block data cannot be nil")
	// ErrEmptyBlockRef is returned a ref was required but was empty.
	ErrEmptyBlockRef = errors.New("empty block reference")
	// ErrNotBlock is returned if the object did not implement Block.
	ErrNotBlock = errors.New("object must be a block")
	// ErrNotSubBlock is returned if the block did not implement SubBlock.
	ErrNotSubBlock = errors.New("block must be a sub-block")
	// ErrNotBlockWithSubBlocks is returned if the block did not implement BlockWithSubBlocks.
	ErrNotBlockWithSubBlocks = errors.New("block must implement block with sub-blocks")
	// ErrEmptyChanges is returned if a slice of changes was unexpectedly empty.
	ErrEmptyChanges = errors.New("changes set cannot be empty")
	// ErrNotFound is returned when a block was not found but was required.
	ErrNotFound = errors.New("block not found")
	// ErrNotClonable is returned if a block could not be cloned.
	ErrNotClonable = errors.New("block: unable to clone")
	// ErrBlockRefMismatch is returned if the data does not match the expected ref.
	ErrBlockRefMismatch = errors.New("block: block ref hash mismatch")
	// ErrBufferedStoreFull is returned when a buffered store reaches its memory limits.
	ErrBufferedStoreFull = errors.New("block: buffered store is full")
)

Functions

func ApplySubBlock 

func ApplySubBlock[T SubBlock](r *T, next SubBlock) error

ApplySubBlock applies a sub-block to a field.

func BeginDeferFlush added in v0.52.0 

func BeginDeferFlush(store StoreOps)

BeginDeferFlush opens a deferred GC-flush scope on store when it batches GC ref-graph flushes. Stores without a GC ref-graph are a no-op.

func CloneBlock 

func CloneBlock(blk any) (any, error)

CloneBlock tries to clone an input block.

The block should implement proto.Message or BlockWithClone.

returns ErrUnexpectedType or ErrNotClonable if the block was not clonable.

func CompareNamedSubBlocks 

func CompareNamedSubBlocks[T ComparableNamedSubBlock](a, b []T) (added, removed, changed []T)

CompareNamedSubBlocks compares two sets of ComparableNamedSubBlock. Returns the added, removed, and changed values.

func EndDeferFlush added in v0.52.0 

func EndDeferFlush(ctx context.Context, store StoreOps) error

EndDeferFlush closes a deferred GC-flush scope on store when it batches GC ref-graph flushes. Stores without a GC ref-graph are a no-op.

func IsNamedSubBlocksSorted 

func IsNamedSubBlocksSorted[T NamedSubBlock](namedSubBlocks []T) bool

IsNamedSubBlocksSorted checks if the set of NamedSubBlock is sorted.

func MarshalBlockRefJSON 

func MarshalBlockRefJSON(ref *BlockRef) ([]byte, error)

MarshalBlockRefJSON marshals an BlockRef to JSON.

Returns "null" if the ref is nil.

func NewTransaction 

func NewTransaction(

	store StoreOps,

	transformer Transformer,

	rootRef *BlockRef,

	putOpts *PutOpts,
) (*Transaction, *Cursor)

NewTransaction builds a new transaction with a root cursor.

func RecordDecodedBlockCacheHit added in v0.51.7 

func RecordDecodedBlockCacheHit(ctx context.Context, cloned bool)

RecordDecodedBlockCacheHit records a clone-safe decoded-block cache hit.

func RecordDecodedBlockUncacheable added in v0.51.7 

func RecordDecodedBlockUncacheable(ctx context.Context)

RecordDecodedBlockUncacheable records a decoded-block cache bypass without an exact key.

func RecordDecodedBlockUncloneable added in v0.51.7 

func RecordDecodedBlockUncloneable(ctx context.Context)

RecordDecodedBlockUncloneable records a decoded-block cache candidate that cannot be cloned.

func RecordResourceGetBlock added in v0.51.7 

func RecordResourceGetBlock(ctx context.Context, ref *BlockRef, found bool, bytes int)

RecordResourceGetBlock records a Resource SDK GetBlock call.

func SortNamedSubBlocks 

func SortNamedSubBlocks[T NamedSubBlock](namedSubBlocks []T)

SortNamedSubBlocks sorts a set of NamedSubBlock.

func UnmarshalBlock 

func UnmarshalBlock[T Block](ctx context.Context, bcs *Cursor, ctor func() Block) (T, error)

UnmarshalBlock unmarshals the block from the cursor & type-asserts it. Returns ErrUnexpectedType if the type returned was not T. Incorrect type happens if the cursor already contains a block w/ different type. If bcs == nil, returns empty, nil. If unmarshal() returns nil, returns empty, nil.

func WithDecodedBlockCache added in v0.51.7 

func WithDecodedBlockCache(ctx context.Context, cache *DecodedBlockCache) context.Context

WithDecodedBlockCache attaches an owner-provided decoded-block cache to ctx.

func WithReadOperationStore added in v0.51.7 

func WithReadOperationStore(ctx context.Context, store StoreOps) context.Context

WithReadOperationStore returns a context that routes block fetches through store.

Types

type AliasIdentityToken added in v0.52.0 

type AliasIdentityToken = aliasidentity.Token

AliasIdentityToken is an opaque per-instance token used to detect in-memory block aliases while marshaling a transaction.

type Block 

type Block interface {
	// MarshalBlock marshals the block to binary.
	// This is the initial step of marshaling, before transformations.
	MarshalBlock() ([]byte, error)
	// UnmarshalBlock unmarshals the block to the object.
	// This is the final step of decoding, after transformations.
	UnmarshalBlock(data []byte) error
}

Block defines an in-memory decoded block structure. A block should contain a minimal amount of data with some pointers to other blocks.

func CastToBlock 

func CastToBlock(sb any) (Block, error)

CastToBlock casts a object to a block or returns an error.

func NewBlockRefBlock 

func NewBlockRefBlock() Block

NewBlockRefBlock constructs a new block reference block.

type BlockRef 

type BlockRef struct {

	// Hash is the hash of the object.
	Hash *hash.Hash `protobuf:"bytes,1,opt,name=hash,proto3" json:"hash,omitempty"`
	// contains filtered or unexported fields
}

BlockRef is a block content ID reference.

protobuf-go-lite:disable-json

func BuildBlockRef 

func BuildBlockRef(data []byte, putOpts *PutOpts) (*BlockRef, error)

BuildBlockRef builds a block ref from put opts by hashing the data. If putOpts are nil, uses default hash type.

func CloneBlockRefs 

func CloneBlockRefs(refs []*BlockRef) []*BlockRef

CloneBlockRefs returns a deep copy of refs, preserving nil entries. Returns nil for empty input.

func ExtractBlockRefs 

func ExtractBlockRefs(blk any) ([]*BlockRef, error)

ExtractBlockRefs extracts all outgoing BlockRefs from a block by walking BlockWithRefs and recursing into BlockWithSubBlocks.

This captures refs at all nesting levels: direct refs on the block, refs inside sub-blocks (e.g., BlockRefSlice), and refs inside nested sub-blocks.

func NewBlockRef 

func NewBlockRef(hash *hash.Hash) *BlockRef

NewBlockRef constructs a new block reference.

func PutBlock 

func PutBlock(ctx context.Context, bk StoreOps, b Block) (*BlockRef, bool, error)

PutBlock marshals & puts a block into a bucket.

func UnmarshalBlockRefB58 

func UnmarshalBlockRefB58(ref string) (*BlockRef, error)

UnmarshalBlockRefB58 unmarshals a b58 string block ref.

func UnmarshalBlockRefBlock 

func UnmarshalBlockRefBlock(ctx context.Context, bcs *Cursor) (*BlockRef, error)

UnmarshalBlockRefBlock unmarshals a BlockRef from a block cursor.

func UnmarshalBlockRefJSON 

func UnmarshalBlockRefJSON(data []byte) (*BlockRef, error)

UnmarshalBlockRefJSON attempts to unmarshal an BlockRef from JSON.

func (*BlockRef) ApplyBlockRef 

func (b *BlockRef) ApplyBlockRef(id uint32, ptr *BlockRef) error

ApplyBlockRef applies a ref change with a field id. The reference may be nil if the child block is nil.

func (*BlockRef) Clone 

func (b *BlockRef) Clone() *BlockRef

Clone clones the block ref.

func (*BlockRef) CloneMessageVT 

func (m *BlockRef) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*BlockRef) CloneVT 

func (m *BlockRef) CloneVT() *BlockRef

func (*BlockRef) EqualMessageVT 

func (this *BlockRef) EqualMessageVT(thatMsg any) bool

func (*BlockRef) EqualVT 

func (this *BlockRef) EqualVT(that *BlockRef) bool

func (*BlockRef) EqualsRef 

func (b *BlockRef) EqualsRef(oref *BlockRef) bool

EqualsRef checks if two refs are equal.

func (*BlockRef) GetBlockRefCtor 

func (b *BlockRef) GetBlockRefCtor(id uint32) Ctor

GetBlockRefCtor returns the constructor for the block at the ref id. Return nil to indicate invalid ref ID or unknown.

func (*BlockRef) GetBlockRefs 

func (b *BlockRef) GetBlockRefs() (map[uint32]*BlockRef, error)

GetBlockRefs returns all block references by ID. May return nil, and values may also be nil. Note: this does not include pending references (in a cursor)

func (*BlockRef) GetEmpty 

func (b *BlockRef) GetEmpty() bool

GetEmpty returns if the ref is empty.

func (*BlockRef) GetHash 

func (x *BlockRef) GetHash() *hash.Hash

func (*BlockRef) LessThan 

func (b *BlockRef) LessThan(other *BlockRef) bool

LessThan checks if the ref is less than another. 1. Empty is sorted to the end. 2. Hash type is sorted. 3. Hash itself is sorted in bytes ordering

func (*BlockRef) MarshalBlock 

func (b *BlockRef) MarshalBlock() ([]byte, error)

MarshalBlock marshals the block to binary. This is the initial step of marshaling, before transformations.

func (*BlockRef) MarshalJSON 

func (b *BlockRef) MarshalJSON() ([]byte, error)

MarshalJSON marshals the BlockRef to JSON.

func (*BlockRef) MarshalKey 

func (b *BlockRef) MarshalKey() ([]byte, error)

MarshalKey marshals the block ref for use as a key. The format should be reproducible and identical between versions.

func (*BlockRef) MarshalLog 

func (b *BlockRef) MarshalLog() string

MarshalLog marshals the reference for logging. If nil or empty returns <nil>

func (*BlockRef) MarshalProtoJSON 

func (b *BlockRef) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the BlockRef message to JSON.

func (*BlockRef) MarshalProtoText 

func (x *BlockRef) MarshalProtoText() string

func (*BlockRef) MarshalString 

func (b *BlockRef) MarshalString() string

MarshalString marshals the reference to a string form.

func (*BlockRef) MarshalToSizedBufferVT 

func (m *BlockRef) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*BlockRef) MarshalToVT 

func (m *BlockRef) MarshalToVT(dAtA []byte) (int, error)

func (*BlockRef) MarshalVT 

func (m *BlockRef) MarshalVT() (dAtA []byte, err error)

func (*BlockRef) ParseFromB58 

func (b *BlockRef) ParseFromB58(ref string) error

ParseFromB58 parses the object ref from a base58 string.

func (*BlockRef) ProtoMessage 

func (*BlockRef) ProtoMessage()

func (*BlockRef) Reset 

func (x *BlockRef) Reset()

func (*BlockRef) SizeVT 

func (m *BlockRef) SizeVT() (n int)

func (*BlockRef) String 

func (x *BlockRef) String() string

func (*BlockRef) UnmarshalBlock 

func (b *BlockRef) UnmarshalBlock(data []byte) error

UnmarshalBlock unmarshals the block to the object. This is the final step of decoding, after transformations.

func (*BlockRef) UnmarshalJSON 

func (b *BlockRef) UnmarshalJSON(dat []byte) error

UnmarshalJSON unmarshals the BlockRef from JSON.

func (*BlockRef) UnmarshalProtoJSON 

func (b *BlockRef) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the BlockRef message from JSON.

func (*BlockRef) UnmarshalVT 

func (m *BlockRef) UnmarshalVT(dAtA []byte) error

func (*BlockRef) Validate 

func (b *BlockRef) Validate(allowEmpty bool) error

Validate validates the block ref.

func (*BlockRef) VerifyData 

func (b *BlockRef) VerifyData(data []byte, errDetails bool) error

VerifyData checks the given data matches the block ref. If errDetails is set, wraps the error with the unexpected and expected refs if mismatch.

type BlockStat 

type BlockStat struct {
	// Ref is the block reference.
	Ref *BlockRef
	// Size is the block data size in bytes.
	// May be -1 if the size is unknown.
	Size int64
}

BlockStat contains metadata about a stored block.

type BlockWithAliasIdentity added in v0.52.0 

type BlockWithAliasIdentity interface {
	BlockAliasIdentity() *AliasIdentityToken
}

BlockWithAliasIdentity exposes stable in-memory identity for blocks that can appear both as handles and inline sub-blocks.

type BlockWithAttributes 

type BlockWithAttributes interface {
	// GetBlockGraphAttributes returns the block graph attributes.
	GetBlockGraphAttributes() []encoding.Attribute
}

BlockWithAttributes returns a block with graph attributes.

type BlockWithClone 

type BlockWithClone interface {
	CloneBlock() (Block, error)
}

BlockWithClone defines a block with a clone function. The clone should share nothing with the original.

type BlockWithCloneVT 

type BlockWithCloneVT[T Block] interface {
	// CloneVT clones the block object with VTprotobuf.
	CloneVT() T
}

BlockWithCloneVT defines a block with a VTProtobuf clone function.

type BlockWithPreWriteHook 

type BlockWithPreWriteHook interface {
	// BlockPreWriteHook is called when writing the block.
	BlockPreWriteHook() error
}

BlockWithPreWriteHook is a block with a function called when writing. This can also be applied to a sub-block.

type BlockWithRefs 

type BlockWithRefs interface {
	// ApplyBlockRef applies a ref change with a field id.
	// The reference may be nil if the child block is nil.
	ApplyBlockRef(id uint32, ptr *BlockRef) error
	// GetBlockRefs returns all block references by ID.
	// May return nil, and values may also be nil.
	// Note: this does not include pending references (in a cursor)
	GetBlockRefs() (map[uint32]*BlockRef, error)
	// GetBlockRefCtor returns the constructor for the block at the ref id.
	// Return nil to indicate invalid ref ID or unknown.
	GetBlockRefCtor(id uint32) Ctor
}

BlockWithRefs has references keyed by ID. Each field can contain a reference.

type BlockWithSubBlocks 

type BlockWithSubBlocks interface {
	// ApplySubBlock applies a sub-block change with a field id.
	ApplySubBlock(id uint32, next SubBlock) error
	// GetSubBlocks returns all constructed sub-blocks by ID.
	// May return nil, and values may also be nil.
	GetSubBlocks() map[uint32]SubBlock
	// GetSubBlockCtor returns a function which creates or returns the existing
	// sub-block at reference id. Can return nil to indicate invalid reference id.
	GetSubBlockCtor(id uint32) SubBlockCtor
}

BlockWithSubBlocks is a block containing sub-blocks as fields.

type BufferedStore 

type BufferedStore struct {
	// contains filtered or unexported fields
}

BufferedStore buffers PutBlock calls in memory and drains them explicitly on Sync or when a caller must free capacity.

func NewBufferedStore 

func NewBufferedStore(ctx context.Context, inner StoreOps) *BufferedStore

NewBufferedStore constructs a buffered store around an inner store.

func NewBufferedStoreWithSettings 

func NewBufferedStoreWithSettings(
	_ context.Context,
	inner StoreOps,
	settings *BufferedStoreSettings,
) *BufferedStore

NewBufferedStoreWithSettings constructs a buffered store with explicit settings.

func (*BufferedStore) BeginDeferFlush 

func (s *BufferedStore) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to the inner store.

func (*BufferedStore) BeginReadOperation added in v0.51.7 

func (s *BufferedStore) BeginReadOperation(context.Context) (StoreOps, func(), error)

BeginReadOperation returns the buffered store for a read scope.

func (*BufferedStore) EndDeferFlush 

func (s *BufferedStore) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to the inner store. Buffered blocks are never drained here; only Sync drains.

func (*BufferedStore) GetBlock 

func (s *BufferedStore) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block by reference.

func (*BufferedStore) GetBlockExists 

func (s *BufferedStore) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists.

func (*BufferedStore) GetBlockExistsBatch 

func (s *BufferedStore) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch checks if blocks exist.

func (*BufferedStore) GetHashType 

func (s *BufferedStore) GetHashType() hash.HashType

GetHashType returns the preferred hash type.

func (*BufferedStore) GetSupportedFeatures 

func (s *BufferedStore) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (*BufferedStore) PutBlock 

func (s *BufferedStore) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock buffers a block in memory and drains synchronously only when backpressure requires capacity.

func (*BufferedStore) PutBlockBatch 

func (s *BufferedStore) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) error

PutBlockBatch loops through PutBlock and RmBlock using the buffered store.

func (*BufferedStore) RmBlock 

func (s *BufferedStore) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block by reference.

func (*BufferedStore) StatBlock 

func (s *BufferedStore) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data.

func (*BufferedStore) Sync added in v0.52.0 

func (s *BufferedStore) Sync(ctx context.Context) (bool, error)

Sync drains buffered blocks, then forwards the durability barrier to inner. Draining is owned solely by Sync (and by backpressure inside PutBlock).

type BufferedStoreSettings 

type BufferedStoreSettings struct {
	MaxPendingEntries int
	MaxPendingBytes   int
	DrainBatchEntries int
}

BufferedStoreSettings configures buffered block writeback behavior.

func DefaultBufferedStoreSettings 

func DefaultBufferedStoreSettings() *BufferedStoreSettings

DefaultBufferedStoreSettings returns the default buffered store settings.

type ComparableNamedSubBlock 

type ComparableNamedSubBlock interface {
	NamedSubBlock

	// Equals compares the block to the other block for equality.
	Equals(ot ComparableNamedSubBlock) bool
}

ComparableNamedSubBlock is a NamedSubBlock that has an Equals function.

type ComparableSubBlock 

type ComparableSubBlock interface {
	comparable
	SubBlock
}

ComparableSubBlock is the type constraint for SubBlock.

type Ctor 

type Ctor func() Block

Ctor is a block constructor.

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor tracks traversal of a block reference DAG structure with an associated Transaction. Manages interacting with block handles, the transaction cache, the decoder and marshaller, and the transformers.

func (*Cursor) ClearAllRefs 

func (c *Cursor) ClearAllRefs()

ClearAllRefs clears all references. Cursors that were generated by Follow() will be detached.

Note: does not clear sub-blocks from the parent object. Note; does not clear the BlockRef from the parent object.

func (*Cursor) ClearRef 

func (c *Cursor) ClearRef(refID uint32)

ClearRef removes the reference handle to the given ref ID. Noop if FollowRef has not been previously called with refID. Cursors that were generated by Follow() will be detached.

Note: does not clear sub-blocks from the parent object. Note; does not clear the BlockRef from the parent object.

func (*Cursor) CloneBlock 

func (c *Cursor) CloneBlock() (any, error)

CloneBlock tries to clone the contained block.

returns ErrUnexpectedType or ErrNotClonable if the block was not clonable.

func (*Cursor) CopyToRecursive 

func (c *Cursor) CopyToRecursive(targetPos *Cursor, cloneBlocks, markDirty bool)

CopyToRecursive copies the cursor and referenced positions to another cursor. The cursor can use a different block transaction. Note: the same block refs will be reused (underlying data is not copied). If target tx == nil: is equivalent to DetachRecursive(false, cloneBlocks) If markDirty is set, marks all target positions as dirty.

func (*Cursor) Detach 

func (c *Cursor) Detach(keepRefs bool) *Cursor

Detach clones the cursor position.

If keepRefs is set, adds the new location as a parent of all previously referenced blocks.

Note: does not copy/clone the Block object.

func (*Cursor) DetachRecursive 

func (c *Cursor) DetachRecursive(detachTx, cloneBlocks, markDirty bool) *Cursor

DetachRecursive clones the cursor position and all referenced positions.

Note: if !cloneBlocks, does not copy/clone the Block objects.

func (*Cursor) DetachTransaction 

func (c *Cursor) DetachTransaction() *Cursor

DetachTransaction creates a new ephemeral transaction rooted at the cursor.

func (*Cursor) Fetch 

func (c *Cursor) Fetch(ctx context.Context) ([]byte, bool, error)

Fetch fetches the block data into memory. Fetching is performed using a block lookup. Returns the transformed decoded version of the data. Returns data, dataIsSet, err. Returns nil, false, ErrBlockStoreUnavailable if the block store is unset. Returns nil, false, nil if the reference is empty. Returns nil, false, ErrNotFound if not found (block unavailable).

func (*Cursor) FollowRef 

func (c *Cursor) FollowRef(
	refID uint32,
	blkRef *BlockRef,
) *Cursor

FollowRef follows a block reference, returning a cursor pointing to the next block and enqueuing the block for fetching. Does not wait for the block to be fetched to return. If the reference is empty, will create a new block.

func (*Cursor) FollowSubBlock 

func (c *Cursor) FollowSubBlock(refID uint32) *Cursor

FollowSubBlock follows a sub-block reference, returning a cursor pointing to the same block but at a sub-block inside a field. The block is constructed or retrieved using the BlockWithSubBlocks interface.

Once FollowSubBlock has been called, the field will be overwritten if dirty. If ClearRef is called on the parent then this relation is removed.

Note: there may already be a reference with the same ID, which would be returned. The cursor must have the block decoded or set with SetBlock. The cursor block blk must be a BlockWithSubBlocks. If these conditions are not met, returns nil

func (*Cursor) GetAllRefs 

func (c *Cursor) GetAllRefs(existingOnly bool) (map[uint32]*Cursor, error)

GetBlockRefs returns cursors to all references.

If existingOnly, only returns references that have already been traversed. If !existingOnly uses GetSubBlocks and/or GetBlockRefs to list all references. If the position blk is empty, returns an empty map.

func (*Cursor) GetBlock 

func (c *Cursor) GetBlock() (any, bool)

GetBlock returns the current loaded block at the position. May be nil if Fetch or Unmarshal or SetBlock have not been called. Returns isSubBlock.

func (*Cursor) GetBlockStore 

func (c *Cursor) GetBlockStore() (StoreOps, bool)

GetBlockStore returns the block store used for the transaction.

func (*Cursor) GetExistingRef 

func (c *Cursor) GetExistingRef(refID uint32) *Cursor

GetExistingRef checks if the reference has been traversed already. Returns nil if no handle exists for that ref.

func (*Cursor) GetRef 

func (c *Cursor) GetRef() *BlockRef

GetRef returns the current cursor reference.

func (*Cursor) GetTransaction 

func (c *Cursor) GetTransaction() *Transaction

GetTransaction returns the cursor's associated transaction, may be nil.

func (*Cursor) IsDirty 

func (c *Cursor) IsDirty() bool

IsDirty indicates if the position or any sub-positions were changed.

func (*Cursor) IsSubBlock 

func (c *Cursor) IsSubBlock() bool

IsSubBlock indicates if the cursor is currently at a sub-block position.

func (*Cursor) MarkDirty 

func (c *Cursor) MarkDirty()

MarkDirty marks the cursor location dirty, so that it will be re-written.

Note: if cursor is ephemeral (no transaction) this is no-op.

func (*Cursor) Parents 

func (c *Cursor) Parents() []*Cursor

Parents returns new cursors pointing to the parent blocks. Note: the parent list is completely dependent on the order the graph was traversed. Note: returns nil if the cursor is ephemeral (with Detach call).

func (*Cursor) SetAsSubBlock 

func (c *Cursor) SetAsSubBlock(refID uint32, parent *Cursor) error

SetAsSubBlock sets the cursor position as a sub-block of another block.

Clears any existing parent references. Immediately calls ApplySubBlock on the parent block.

May return ErrNotSubBlock or ErrUnexpectedType if the parent is not a block with sub-blocks.

func (*Cursor) SetBlock 

func (c *Cursor) SetBlock(b any, dirty bool)

SetBlock sets a block at the location, and marks the block as dirty. If the location is a Block, b should implement Block interface. If it is a SubBlock, b should implement the SubBlock interface. If dirty is set, sets the block as dirty.

Clears BlockPreWrite.

func (*Cursor) SetBlockStore 

func (c *Cursor) SetBlockStore(store StoreOps)

SetBlockStore sets the store to read from for this cursor and all sub-cursors. If nil, will use the default bucket attached to the block transaction.

func (*Cursor) SetPreWriteHook 

func (c *Cursor) SetPreWriteHook(h func(b any) error)

SetPreWriteHook sets a hook for final transforms to the block.

Note: this should not call any cursor functions that will be locked during the Write process.

Also valid for sub-blocks.

func (*Cursor) SetRef 

func (c *Cursor) SetRef(refID uint32, cursor *Cursor)

SetRef sets a block reference to the handle at the cursor. Adds c to the list of parents for cursor. The cursors must be from the same transaction. Note: this should only be used if refID and cursor are not sub-blocks.

func (*Cursor) SetRefAtCursor 

func (c *Cursor) SetRefAtCursor(ref *BlockRef, clearBlock bool)

SetRefAtCursor sets the reference at the cursor location. If ref is not equal to the existing ref, and clearBlock is set, blk is set to nil.

func (*Cursor) Unmarshal 

func (c *Cursor) Unmarshal(ctx context.Context, ctor func() Block) (Block, error)

Unmarshal fetches and unmarshals the data to a block. If already unmarshaled, returns existing data. If a sub-block, the sub-block must implement Block. If a sub-block, will return the sub-block value or nil. ctor is ignored if the cursor is a sub-block. Returns nil, nil if ctor is nil and the block is nil. Returns nil, nil if the cursor is nil. Returns value from ctor() without calling Unmarshal if empty. Returns nil, block.ErrNotFound if not found.

type DecodedBlockCache added in v0.51.7 

type DecodedBlockCache struct {
	// contains filtered or unexported fields
}

DecodedBlockCache owns shared decoded block reuse.

func NewDecodedBlockCache added in v0.51.7 

func NewDecodedBlockCache() *DecodedBlockCache

NewDecodedBlockCache constructs a decoded-block cache with default options.

func NewDecodedBlockCacheWithOptions added in v0.51.7 

func NewDecodedBlockCacheWithOptions(opts DecodedBlockCacheOptions) (*DecodedBlockCache, error)

NewDecodedBlockCacheWithOptions constructs a decoded-block cache owner.

func (*DecodedBlockCache) Close added in v0.51.7 

func (c *DecodedBlockCache) Close()

Close releases the cache goroutine.

func (*DecodedBlockCache) InvalidateAll added in v0.51.7 

func (c *DecodedBlockCache) InvalidateAll(ctx context.Context)

InvalidateAll removes every decoded cache entry owned by c.

func (*DecodedBlockCache) InvalidateRef added in v0.51.7 

func (c *DecodedBlockCache) InvalidateRef(ctx context.Context, ref *BlockRef)

InvalidateRef removes decoded cache entries for ref.

func (*DecodedBlockCache) Lookup added in v0.51.7 

func (c *DecodedBlockCache) Lookup(ctx context.Context, front *decodedBlockFrontCache, key decodedBlockCacheKey) (Block, bool, error)

func (*DecodedBlockCache) MaxCost added in v0.51.7 

func (c *DecodedBlockCache) MaxCost() int64

MaxCost returns the configured decoded-cache budget.

func (*DecodedBlockCache) Snapshot added in v0.51.7 

func (c *DecodedBlockCache) Snapshot() DecodedBlockCacheSnapshot

Snapshot returns shared decoded-cache metrics.

func (*DecodedBlockCache) Store added in v0.51.7 

func (c *DecodedBlockCache) Store(
	ctx context.Context,
	front *decodedBlockFrontCache,
	token decodedBlockCacheStoreToken,
	key decodedBlockCacheKey,
	ref *BlockRef,
	blk Block,
	data []byte,
) error

func (*DecodedBlockCache) Wait added in v0.51.7 

func (c *DecodedBlockCache) Wait()

Wait blocks until buffered cache writes have reached Ristretto.

type DecodedBlockCacheFreshener added in v0.51.7 

type DecodedBlockCacheFreshener interface {
	// EnsureDecodedBlockCacheFresh updates owner state before decoded-cache lookup.
	EnsureDecodedBlockCacheFresh(ctx context.Context) error
}

DecodedBlockCacheFreshener is implemented by stores whose decoded-cache hits depend on external freshness state that ordinary block reads would refresh.

type DecodedBlockCacheOptions added in v0.51.7 

type DecodedBlockCacheOptions struct {
	// MaxCost is the Ristretto cache budget.
	MaxCost int64
	// NumCounters controls Ristretto admission counter capacity.
	NumCounters int64
	// BufferItems controls Ristretto get-buffer size.
	BufferItems int64
	// Disabled bypasses decoded-object retention while preserving reads.
	Disabled bool
}

DecodedBlockCacheOptions configures a decoded-block cache owner.

func DefaultDecodedBlockCacheOptions added in v0.51.7 

func DefaultDecodedBlockCacheOptions() DecodedBlockCacheOptions

DefaultDecodedBlockCacheOptions returns the production decoded-cache options.

type DecodedBlockCacheSnapshot added in v0.51.7 

type DecodedBlockCacheSnapshot struct {
	// MaxCost is the configured shared-cache budget.
	MaxCost int64
	// RemainingCost is the current unused budget reported by Ristretto.
	RemainingCost int64
	// RetainedCost is the current retained cost estimate.
	RetainedCost int64
	// Hits is the number of shared-cache hits.
	Hits uint64
	// Misses is the number of shared-cache misses.
	Misses uint64
	// Stores is the number of keys admitted by Ristretto.
	Stores uint64
	// Rejections is the number of set calls dropped or rejected by Ristretto.
	Rejections uint64
	// Evictions is the number of keys evicted by Ristretto.
	Evictions uint64
	// CostAdded is the sum of costs admitted by Ristretto.
	CostAdded uint64
	// CostEvicted is the sum of costs evicted by Ristretto.
	CostEvicted uint64
}

DecodedBlockCacheSnapshot is a point-in-time decoded-cache metrics snapshot.

type DecodedBlockCacheTransformer added in v0.51.7 

type DecodedBlockCacheTransformer interface {
	DecodedBlockCacheTransformKey() string
}

DecodedBlockCacheTransformer identifies a transform boundary for decoded-block caching.

type DecodedBlockCacheable added in v0.51.7 

type DecodedBlockCacheable interface {
	DecodedBlockCacheTypeKey() string
}

DecodedBlockCacheable identifies a block type for decoded-block caching.

type DeferFlusher added in v0.52.0 

type DeferFlusher interface {
	// BeginDeferFlush opens a deferred GC-flush scope. Supports nesting.
	BeginDeferFlush()
	// EndDeferFlush closes a scope; the outermost close flushes pending edges.
	EndDeferFlush(ctx context.Context) error
}

DeferFlusher batches deferred GC ref-graph flushes across a write scope.

This is an in-process concern owned by GCStoreOps, distinct from the durability surface (StoreOps.Sync). While a scope is open, accumulated ref-graph edge operations are not applied; the outermost EndDeferFlush applies them in one batch. In-process store wrappers forward the scope to their inner store so a GCStoreOps nested behind wrappers still batches. It is never carried over the block RPC wire.

type NamedSubBlock 

type NamedSubBlock interface {
	// SubBlock indicates this is a sub-block.
	SubBlock
	// GetName returns the name of the block.
	GetName() string
}

NamedSubBlock is a sub-block with a name attached.

type NopStoreOps 

type NopStoreOps struct{}

NopStoreOps implements StoreOps defaults for tests.

Production wrappers must not embed this type. Test mocks may embed it and override only the methods exercised by the test.

func (NopStoreOps) BeginReadOperation added in v0.51.7 

func (n NopStoreOps) BeginReadOperation(context.Context) (StoreOps, func(), error)

BeginReadOperation returns the no-op store for a read scope.

func (NopStoreOps) GetBlock 

func (NopStoreOps) GetBlock(context.Context, *BlockRef) ([]byte, bool, error)

GetBlock returns a missing block.

func (NopStoreOps) GetBlockExists 

func (NopStoreOps) GetBlockExists(context.Context, *BlockRef) (bool, error)

GetBlockExists returns false.

func (NopStoreOps) GetBlockExistsBatch 

func (n NopStoreOps) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch loops calling GetBlockExists per ref.

func (NopStoreOps) GetHashType 

func (NopStoreOps) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store.

func (NopStoreOps) GetSupportedFeatures 

func (NopStoreOps) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (NopStoreOps) PutBlock 

func (NopStoreOps) PutBlock(context.Context, []byte, *PutOpts) (*BlockRef, bool, error)

PutBlock returns ErrBlockStoreUnavailable.

func (NopStoreOps) PutBlockBatch 

func (n NopStoreOps) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) error

PutBlockBatch loops calling PutBlock or RmBlock per entry.

func (NopStoreOps) RmBlock 

func (NopStoreOps) RmBlock(context.Context, *BlockRef) error

RmBlock returns nil.

func (NopStoreOps) StatBlock 

func (NopStoreOps) StatBlock(context.Context, *BlockRef) (*BlockStat, error)

StatBlock returns nil.

func (NopStoreOps) Sync added in v0.52.0 

func (NopStoreOps) Sync(context.Context) (bool, error)

Sync reports no durability fence because the no-op store holds no writes.

type OverlayMode 

type OverlayMode int32

OverlayMode controls the mode for the block store overlay. 

const (
	// UPPER_ONLY uses the upper store only for reads and writes.
	// reads go to the upper store only.
	// writes go to the upper store only.
	// removes go to the upper store only.
	// the lower store is not used.
	OverlayMode_UPPER_ONLY OverlayMode = 0
	// LOWER_ONLY uses the lower store only for reads and writes.
	// reads go to the lower store only.
	// writes go to the lower store only.
	// removes go to the lower store only.
	// the upper store is not used.
	OverlayMode_LOWER_ONLY OverlayMode = 1
	// UPPER_CACHE uses the upper store as a cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to both stores.
	// removes go to both stores.
	// reads from lower are written back to upper.
	OverlayMode_UPPER_CACHE OverlayMode = 2
	// LOWER_CACHE uses the lower store as a cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to both stores.
	// removes go to both stores.
	// reads from upper are written back to lower.
	OverlayMode_LOWER_CACHE OverlayMode = 3
	// UPPER_READ_CACHE uses the upper store as a cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to the lower store only.
	// removes go to the lower store only.
	// reads from lower are not written back to upper.
	OverlayMode_UPPER_READ_CACHE OverlayMode = 4
	// LOWER_READ_CACHE uses the lower store as a cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to the upper store only.
	// removes go to both stores.
	// reads from upper are not written back to lower.
	OverlayMode_LOWER_READ_CACHE OverlayMode = 5
	// UPPER_WRITE_CACHE uses the upper store as a write cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to the upper store only.
	// removes go to both stores.
	// reads from lower are not written back to upper.
	OverlayMode_UPPER_WRITE_CACHE OverlayMode = 6
	// LOWER_WRITE_CACHE uses the lower store as a write cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to the lower store only.
	// removes go to both stores.
	// reads from upper are not written back to lower.
	OverlayMode_LOWER_WRITE_CACHE OverlayMode = 7
	// UPPER_READBACK_CACHE treats the lower store as read-only and immutable
	// (e.g. remote packfiles) while caching reads back into the upper store.
	// reads go to the upper store first, then the lower store.
	// reads from lower are written back to upper.
	// writes go to the upper store only.
	// removes go to the upper store only (lower lifecycle is external, e.g.
	// packfile garbage collection / repack).
	OverlayMode_UPPER_READBACK_CACHE OverlayMode = 8
)

func (OverlayMode) Enum 

func (x OverlayMode) Enum() *OverlayMode

func (OverlayMode) MarshalJSON 

func (x OverlayMode) MarshalJSON() ([]byte, error)

MarshalJSON marshals the OverlayMode to JSON.

func (OverlayMode) MarshalProtoJSON 

func (x OverlayMode) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the OverlayMode to JSON.

func (OverlayMode) MarshalProtoText 

func (x OverlayMode) MarshalProtoText() string

func (OverlayMode) MarshalText 

func (x OverlayMode) MarshalText() ([]byte, error)

MarshalText marshals the OverlayMode to text.

func (OverlayMode) String 

func (x OverlayMode) String() string

func (*OverlayMode) UnmarshalJSON 

func (x *OverlayMode) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the OverlayMode from JSON.

func (*OverlayMode) UnmarshalProtoJSON 

func (x *OverlayMode) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the OverlayMode from JSON.

func (*OverlayMode) UnmarshalText 

func (x *OverlayMode) UnmarshalText(b []byte) error

UnmarshalText unmarshals the OverlayMode from text.

type PutBatchEntry 

type PutBatchEntry struct {
	// Ref is the expected content-addressed block reference.
	Ref *BlockRef
	// Data is the encoded block payload.
	Data []byte
	// Refs are outgoing block references recorded with this write.
	Refs []*BlockRef
	// Tombstone marks the block ref as deleted.
	Tombstone bool
}

PutBatchEntry describes one durable block write in a lower-layer batch.

type PutOpts 

type PutOpts struct {

	// HashType is the hash type to use.
	// If unset (0 value) will use default for the store.
	HashType hash.HashType `protobuf:"varint,1,opt,name=hash_type,json=hashType,proto3" json:"hashType,omitempty"`
	// ForceBlockRef forces the block ref to equal the given ref.
	// Can be unset to indicate none (allow any).
	//
	// If the generated BlockRef does not match, the storage operation is aborted
	// and returns block.ErrBlockRefMismatch.
	ForceBlockRef *BlockRef `protobuf:"bytes,2,opt,name=force_block_ref,json=forceBlockRef,proto3" json:"forceBlockRef,omitempty"`
	// Refs are outgoing block references recorded with this write.
	// GC-aware stores consume these refs as part of the put. Stores that do not
	// participate in GC ignore this field.
	Refs []*BlockRef `protobuf:"bytes,3,rep,name=refs,proto3" json:"refs,omitempty"`
	// Sync requests a durability fence before PutBlock returns. When false, the
	// store may enqueue the write and make it immediately readable through its
	// read-through path without waiting for durability. When true, PutBlock
	// enqueues the write, then calls the store Sync barrier so this write and all
	// prior buffered writes for the store are durable before return.
	Sync bool `protobuf:"varint,4,opt,name=sync,proto3" json:"sync,omitempty"`
	// contains filtered or unexported fields
}

PutOpts are options that can be passed to PutBlock.

func PutOptsWithoutSync added in v0.52.0 

func PutOptsWithoutSync(opts *PutOpts) (*PutOpts, bool)

PutOptsWithoutSync returns opts with Sync cleared, plus whether Sync was set. Wrappers call inner stores with Sync cleared, then apply their own durability boundary after wrapper-local bookkeeping.

func (*PutOpts) CloneMessageVT 

func (m *PutOpts) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*PutOpts) CloneVT 

func (m *PutOpts) CloneVT() *PutOpts

func (*PutOpts) EqualMessageVT 

func (this *PutOpts) EqualMessageVT(thatMsg any) bool

func (*PutOpts) EqualVT 

func (this *PutOpts) EqualVT(that *PutOpts) bool

func (*PutOpts) GetForceBlockRef 

func (x *PutOpts) GetForceBlockRef() *BlockRef

func (*PutOpts) GetHashType 

func (x *PutOpts) GetHashType() hash.HashType

func (*PutOpts) GetRefs 

func (x *PutOpts) GetRefs() []*BlockRef

func (*PutOpts) GetSync added in v0.52.0 

func (x *PutOpts) GetSync() bool

func (*PutOpts) MarshalB58 

func (o *PutOpts) MarshalB58() string

MarshalB58 marshals the put opts to a base58 string form.

func (*PutOpts) MarshalBlock 

func (o *PutOpts) MarshalBlock() ([]byte, error)

MarshalBlock marshals the block to binary. This is the initial step of marshaling, before transformations.

func (*PutOpts) MarshalJSON 

func (x *PutOpts) MarshalJSON() ([]byte, error)

MarshalJSON marshals the PutOpts to JSON.

func (*PutOpts) MarshalProtoJSON 

func (x *PutOpts) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the PutOpts message to JSON.

func (*PutOpts) MarshalProtoText 

func (x *PutOpts) MarshalProtoText() string

func (*PutOpts) MarshalString 

func (o *PutOpts) MarshalString() string

MarshalString marshals the put opts to b58 string.

func (*PutOpts) MarshalToSizedBufferVT 

func (m *PutOpts) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*PutOpts) MarshalToVT 

func (m *PutOpts) MarshalToVT(dAtA []byte) (int, error)

func (*PutOpts) MarshalVT 

func (m *PutOpts) MarshalVT() (dAtA []byte, err error)

func (*PutOpts) ProtoMessage 

func (*PutOpts) ProtoMessage()

func (*PutOpts) Reset 

func (x *PutOpts) Reset()

func (*PutOpts) SelectHashType 

func (o *PutOpts) SelectHashType(defHashType hash.HashType) hash.HashType

SelectHashType selects the hash type to use for the operation. The given hash type should be the default value to use.

func (*PutOpts) SizeVT 

func (m *PutOpts) SizeVT() (n int)

func (*PutOpts) String 

func (x *PutOpts) String() string

func (*PutOpts) UnmarshalB58 

func (o *PutOpts) UnmarshalB58(ref string) error

UnmarshalB58 unmarshals the put opts from base58 string form.

func (*PutOpts) UnmarshalBlock 

func (o *PutOpts) UnmarshalBlock(data []byte) error

UnmarshalBlock unmarshals the block to the object. This is the final step of decoding, after transformations.

func (*PutOpts) UnmarshalJSON 

func (x *PutOpts) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the PutOpts from JSON.

func (*PutOpts) UnmarshalProtoJSON 

func (x *PutOpts) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the PutOpts message from JSON.

func (*PutOpts) UnmarshalVT 

func (m *PutOpts) UnmarshalVT(dAtA []byte) error

func (*PutOpts) Validate 

func (o *PutOpts) Validate() error

Validate validates the put opts.

type ReadCounter added in v0.51.7 

type ReadCounter struct {
	// contains filtered or unexported fields
}

ReadCounter records aggregate block reads for one logical operation.

func WithReadCounter added in v0.51.7 

func WithReadCounter(ctx context.Context) (context.Context, *ReadCounter)

WithReadCounter returns a context that records aggregate block reads.

func (*ReadCounter) Snapshot added in v0.51.7 

func (c *ReadCounter) Snapshot() ReadCounterSnapshot

Snapshot returns the current counter values.

type ReadCounterSnapshot added in v0.51.7 

type ReadCounterSnapshot struct {
	// BlockReadCount is the number of block reads that returned without store error.
	BlockReadCount uint64
	// BlockReadBytes is the number of bytes returned by found block reads.
	BlockReadBytes uint64
	// BlockReadMissCount is the number of block reads that returned not found without store error.
	BlockReadMissCount uint64
	// ResourceGetBlockCount is the number of Resource SDK GetBlock calls.
	ResourceGetBlockCount uint64
	// ResourceGetBlockRefCount is the number of non-empty refs passed to Resource SDK GetBlock.
	ResourceGetBlockRefCount uint64
	// ResourceGetBlockBytes is the number of bytes returned by found Resource SDK GetBlock calls.
	ResourceGetBlockBytes uint64
	// ResourceGetBlockMissCount is the number of Resource SDK GetBlock calls that missed.
	ResourceGetBlockMissCount uint64
	// DecodedBlockUnmarshalCount is the number of decoded block unmarshals.
	DecodedBlockUnmarshalCount uint64
	// DecodedBlockUnmarshalBytes is the number of bytes passed to decoded block unmarshalling.
	DecodedBlockUnmarshalBytes uint64
	// DecodedBlockCacheAttemptCount is the number of decoded-block cache lookup attempts.
	DecodedBlockCacheAttemptCount uint64
	// DecodedBlockCacheHitCount is the number of decoded-block cache hits.
	DecodedBlockCacheHitCount uint64
	// DecodedBlockCacheMissCount is the number of decoded-block cache misses.
	DecodedBlockCacheMissCount uint64
	// DecodedBlockCloneCount is the number of clone-on-hit decoded block returns.
	DecodedBlockCloneCount uint64
	// DecodedBlockUncloneableCount is the number of cache candidates that could not be cloned.
	DecodedBlockUncloneableCount uint64
	// DecodedBlockUncacheableCount is the number of decoded-block cache bypasses for missing exact keys.
	DecodedBlockUncacheableCount uint64
	// DecodedBlockStoreAttemptCount is the number of decoded-block cache store submissions.
	DecodedBlockStoreAttemptCount uint64
	// DecodedBlockStoreAcceptedCount is the number of store submissions accepted by the cache buffer.
	DecodedBlockStoreAcceptedCount uint64
	// DecodedBlockStoreRejectedCount is the number of store submissions rejected or dropped immediately.
	DecodedBlockStoreRejectedCount uint64
	// DecodedBlockStoreCost is the cost submitted for decoded-block cache retention.
	DecodedBlockStoreCost uint64
}

ReadCounterSnapshot is a point-in-time copy of block read counters.

type StoreFeature 

type StoreFeature int32

StoreFeature is a bitmask of native block store capabilities. Each non-zero enum value is a single bit and values are combined with bitwise OR. The values are intentionally not sequential. 

const (
	// STORE_FEATURE_UNKNOWN indicates no native capabilities are reported.
	StoreFeature_STORE_FEATURE_UNKNOWN StoreFeature = 0
	// STORE_FEATURE_NATIVE_BATCH_PUT means PutBlockBatch is implemented natively.
	StoreFeature_STORE_FEATURE_NATIVE_BATCH_PUT StoreFeature = 1
	// STORE_FEATURE_NATIVE_BATCH_EXISTS means GetBlockExistsBatch is implemented natively.
	StoreFeature_STORE_FEATURE_NATIVE_BATCH_EXISTS StoreFeature = 2
	// STORE_FEATURE_SELF_BUFFERED means the store has its own read-through
	// pending buffer, so the world block engine must not wrap it in a
	// BufferedStore. Volume remains the durability barrier owner for
	// volume-backed stores.
	StoreFeature_STORE_FEATURE_SELF_BUFFERED StoreFeature = 32
)

func (StoreFeature) Enum 

func (x StoreFeature) Enum() *StoreFeature

func (StoreFeature) Has 

func (s StoreFeature) Has(feat StoreFeature) bool

Has reports whether this feature bitset contains every requested feature.

func (StoreFeature) MarshalJSON 

func (x StoreFeature) MarshalJSON() ([]byte, error)

MarshalJSON marshals the StoreFeature to JSON.

func (StoreFeature) MarshalProtoJSON 

func (x StoreFeature) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the StoreFeature to JSON.

func (StoreFeature) MarshalProtoText 

func (x StoreFeature) MarshalProtoText() string

func (StoreFeature) MarshalText 

func (x StoreFeature) MarshalText() ([]byte, error)

MarshalText marshals the StoreFeature to text.

func (StoreFeature) String 

func (x StoreFeature) String() string

func (*StoreFeature) UnmarshalJSON 

func (x *StoreFeature) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the StoreFeature from JSON.

func (*StoreFeature) UnmarshalProtoJSON 

func (x *StoreFeature) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the StoreFeature from JSON.

func (*StoreFeature) UnmarshalText 

func (x *StoreFeature) UnmarshalText(b []byte) error

UnmarshalText unmarshals the StoreFeature from text.

type StoreOps 

type StoreOps interface {
	// GetHashType returns the preferred hash type for the store.
	// This should return as fast as possible (called frequently).
	// If 0 is returned, uses a default defined by Hydra.
	GetHashType() hash.HashType
	// GetSupportedFeatures returns the native feature bitmask for the store.
	// This should return as fast as possible (called frequently) and remain
	// static for the lifetime of the store.
	GetSupportedFeatures() StoreFeature
	// BeginReadOperation opens a read scope for a bounded logical operation.
	// The returned store must be released after use. Implementations without
	// native read-operation state may return themselves with a no-op release.
	BeginReadOperation(ctx context.Context) (StoreOps, func(), error)
	// PutBlock puts a block into the store. By default this may enqueue a
	// buffered, read-through write without waiting for durability. Set
	// PutOpts.Sync to fence this write and all prior buffered writes before return.
	// The ref should not be modified after return.
	// The second return value can optionally indicate if the block already existed.
	// If the hash type is unset, use the type from GetHashType().
	PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)
	// PutBlockBatch writes a batch of block operations.
	// Implementations without native batching fall back internally.
	PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) error
	// GetBlock gets a block with the given reference.
	// The ref should not be modified or retained by GetBlock.
	// Returns data, found, error.
	// Returns nil, false, nil if not found.
	// Note: the block may not be in the specified bucket.
	GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)
	// GetBlockExists checks if a block exists with a cid reference.
	// The ref should not be modified or retained by GetBlock.
	// Note: the block may not be in the specified bucket.
	GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)
	// GetBlockExistsBatch checks whether each block reference exists.
	// Implementations without native batching fall back internally.
	GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)
	// RmBlock deletes a block from the bucket.
	// Does not return an error if the block was not present.
	// In some cases, will return before confirming delete.
	RmBlock(ctx context.Context, ref *BlockRef) error
	// StatBlock returns metadata about a block without reading its data.
	// Returns nil, nil if the block does not exist.
	StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)
	// Sync is the sole durability barrier: it drains any buffered writes and
	// blocks until every write issued before the call is durable, like POSIX
	// sync(2). The boolean reports whether a fence was applied: true when the
	// barrier completed or the store is always-durable, false when the store
	// provides no durability fence. A nil error with false means "no fence
	// available", not a failure; a non-nil error means a real sync failure.
	Sync(ctx context.Context) (bool, error)
}

StoreOps can read/write blocks.

func NewStoreRW 

func NewStoreRW(readHandle, writeHandle StoreOps) StoreOps

NewStoreRW constructs a new Store handle using a read handle and an optional write handle. If the write handle is not nil, the write (put and delete) calls will go to it. Otherwise, all calls are sent to the read handle.

type StoreOverlay 

type StoreOverlay struct {
	// contains filtered or unexported fields
}

StoreOverlay layers an upper block store over a lower store.

ctx is used for writeback requests

func NewOverlay 

func NewOverlay(
	ctx context.Context,
	le *logrus.Entry,
	lower,
	upper StoreOps,
	mode OverlayMode,
	writebackTimeout time.Duration,
	writebackPutOpts *PutOpts,
) *StoreOverlay

NewOverlay constructs a new overlay store.

ctx is used for writeback requests

func (*StoreOverlay) BeginDeferFlush 

func (o *StoreOverlay) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to upper and lower stores.

func (*StoreOverlay) BeginReadOperation added in v0.51.7 

func (o *StoreOverlay) BeginReadOperation(ctx context.Context) (StoreOps, func(), error)

BeginReadOperation opens read scopes for the overlay stores.

func (*StoreOverlay) EndDeferFlush 

func (o *StoreOverlay) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to upper and lower stores.

func (*StoreOverlay) GetBlock 

func (o *StoreOverlay) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block with the given reference. The ref should not be modified or retained by GetBlock. Returns data, found, error. Returns nil, false, nil if not found. Note: the block may not be in the specified bucket.

func (*StoreOverlay) GetBlockExists 

func (o *StoreOverlay) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreOverlay) GetBlockExistsBatch 

func (o *StoreOverlay) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch checks block existence using the same read policy as GetBlockExists.

func (*StoreOverlay) GetHashType 

func (o *StoreOverlay) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store. This should return as fast as possible (called frequently). If 0 is returned, uses a default defined by Hydra.

func (*StoreOverlay) GetSupportedFeatures 

func (o *StoreOverlay) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the overlay.

func (*StoreOverlay) PutBlock 

func (o *StoreOverlay) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock puts a block into the store. The ref should not be modified after return. The second return value can optionally indicate if the block already existed. If the hash type is unset, use the type from GetHashType().

func (*StoreOverlay) PutBlockBatch 

func (o *StoreOverlay) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) error

PutBlockBatch writes a batch of blocks using the same target-store policy as PutBlock.

func (*StoreOverlay) RmBlock 

func (o *StoreOverlay) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block from the bucket. Does not return an error if the block was not present. In some cases, will return before confirming delete.

func (*StoreOverlay) StatBlock 

func (o *StoreOverlay) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data. Returns nil, nil if the block does not exist.

func (*StoreOverlay) Sync added in v0.52.0 

func (o *StoreOverlay) Sync(ctx context.Context) (bool, error)

Sync fences both overlay stores; it reports fenced only when both did.

type StoreRW 

type StoreRW struct {
	// contains filtered or unexported fields
}

StoreRW combines a read and write store together.

func (*StoreRW) BeginDeferFlush 

func (b *StoreRW) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to the write handle.

func (*StoreRW) BeginReadOperation added in v0.51.7 

func (b *StoreRW) BeginReadOperation(ctx context.Context) (StoreOps, func(), error)

BeginReadOperation opens a read scope on the read handle.

func (*StoreRW) EndDeferFlush 

func (b *StoreRW) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to the write handle.

func (*StoreRW) EnsureDecodedBlockCacheFresh added in v0.51.7 

func (b *StoreRW) EnsureDecodedBlockCacheFresh(ctx context.Context) error

EnsureDecodedBlockCacheFresh forwards decoded-cache freshness to the read owner.

func (*StoreRW) GetBlock 

func (b *StoreRW) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreRW) GetBlockExists 

func (b *StoreRW) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreRW) GetBlockExistsBatch 

func (b *StoreRW) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch forwards batched existence probes to the read handle when supported.

func (*StoreRW) GetHashType 

func (b *StoreRW) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store. This should return as fast as possible (called frequently). If 0 is returned, uses a default defined by Hydra.

func (*StoreRW) GetSupportedFeatures 

func (b *StoreRW) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (*StoreRW) PutBlock 

func (b *StoreRW) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock puts a block into the store. The ref should not be modified after return.

func (*StoreRW) PutBlockBatch 

func (b *StoreRW) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) error

PutBlockBatch forwards to the write handle if it supports batched writes.

func (*StoreRW) RmBlock 

func (b *StoreRW) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block from the bucket. Does not return an error if the block was not present. In some cases, will return before confirming delete.

func (*StoreRW) StatBlock 

func (b *StoreRW) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data. Returns nil, nil if the block does not exist.

func (*StoreRW) Sync added in v0.52.0 

func (b *StoreRW) Sync(ctx context.Context) (bool, error)

Sync forwards the durability barrier to the write handle.

type SubBlock 

type SubBlock interface {
	BlockWithAliasIdentity
	// IsNil checks if the object is nil.
	IsNil() bool
}

SubBlock is a object contained inside a Block. May optionally implement Block or other Block interfaces.

type SubBlockCtor 

type SubBlockCtor func(create bool) SubBlock

SubBlockCtor constructs a sub-block. If create == false, returns nil if the field is not set.

func NewSubBlockCtor 

func NewSubBlockCtor[T ComparableSubBlock](r *T, ctor func() T) SubBlockCtor

NewSubBlockCtor constructs a new SubBlock constructor. returns nil if r is nil usage: block.NewSubBlockCtor(r, func() *ChangeLogLL { return &ChangeLogLL{} })

type Transaction 

type Transaction struct {
	// contains filtered or unexported fields
}

Transaction tracks refs traversed between blocks, batching writes and propagating changes through the merkle graph.

The decoded object form of the block can be stored / attached to a block handle. Changes are written to storage with a topological reference sort.

Empty blocks are not written to storage: they are instead represented with a nil BlockRef. SetBlockRef should handle nil BlockRef objects correctly.

func (*Transaction) GetBlockGraph 

func (t *Transaction) GetBlockGraph() graph.Graph

GetBlockGraph returns a handle to the internal block graph state. Do not modify this, used for analysis.

func (*Transaction) GetPutOpts 

func (t *Transaction) GetPutOpts() *PutOpts

GetPutOpts returns the transaction's put options.

func (*Transaction) GetStoreOps 

func (t *Transaction) GetStoreOps() StoreOps

GetStoreOps returns the transaction's store operations.

func (*Transaction) GetTransformer 

func (t *Transaction) GetTransformer() Transformer

GetTransformer returns the transaction's block transformer.

func (*Transaction) SetBufferedStoreSettings 

func (t *Transaction) SetBufferedStoreSettings(s *BufferedStoreSettings)

SetBufferedStoreSettings overrides the BufferedStore settings used to wrap the write store inside WriteAtRoot. Pass nil to reset to defaults. This must be called before Write/WriteAtRoot begins committing for the override to take effect on that commit.

func (*Transaction) SetDecodedBlockCache added in v0.51.7 

func (t *Transaction) SetDecodedBlockCache(cache *DecodedBlockCache)

SetDecodedBlockCache sets the lifecycle-owned decoded-block cache borrowed by this transaction.

func (*Transaction) SetRoot 

func (t *Transaction) SetRoot(cursor *Cursor) error

SetRoot sets the root of the transaction to a different position. Clears all parent blocks from the new root.

func (*Transaction) SetStoreOps 

func (t *Transaction) SetStoreOps(store StoreOps)

SetStoreOps replaces the transaction's store implementation. Used to swap in GCStoreOps after the RefGraph is available.

func (*Transaction) Write 

func (t *Transaction) Write(ctx context.Context, clearTree bool) (
	res *BlockRef,
	rcursor *Cursor,
	rerr error,
)

Write writes the dirty blocks to the store, propagating reference changes up the tree. Clears the blocks cache if clearTree is set, otherwise the updated references are written to the cursor tree. The final block in the event list will be the new root. The new root cursor is returned. Blocks that are not referenced by the root directly or indirectly are "cut" and removed.

Note: after Write with clearTree, use the new returned rcursor only.

func (*Transaction) WriteAtRoot 

func (t *Transaction) WriteAtRoot(ctx context.Context, clearTree bool, subRoot *Cursor) (
	res *BlockRef,
	rcursor *Cursor,
	rerr error,
)

WriteAtRoot writes dirty blocks to the store starting from a sub-tree root. If subRoot is nil, writes from the transaction root (same as Write). If subRoot is non-nil, writes only the sub-tree rooted at that cursor. Blocks outside the sub-tree are not touched. After writing, the sub-tree nodes are non-dirty with refs set, and their block data is freed if clearTree is set. The parent transaction's Write() will skip these nodes.

type Transformer 

type Transformer interface {
	// EncodeBlock encodes the block according to the config.
	// May reuse the same byte slice if possible.
	EncodeBlock([]byte) ([]byte, error)
	// DecodeBlock decodes the block according to the config.
	// May reuse the same byte slice if possible.
	DecodeBlock([]byte) ([]byte, error)
}

Transformer encodes and decodes blocks for storage.

Source Files

View all Source files

Directories

Path Synopsis
Package block_copy provides functions for copying block DAGs between stores.
 Package block_copy provides functions for copying block DAGs between stores.
gc
Package block_gc implements garbage collection for Hydra block stores.
 Package block_gc implements garbage collection for Hydra block stores.
gcgraph
Package gcgraph implements the GC reference graph on OPFS.
 Package gcgraph implements the GC reference graph on OPFS.
rpc
rpc/client
rpc/server
wal
rpc
client
server
bucket
controller
http
http/lookup
http/server
http/test
inmem
kvfile
kvfile/http
kvtx
overlay
redis
ristretto
rpc
rpc/lookup
rpc/server
rpc/server/bucket
s3
s3/lookup
test
vlogger
all
blockenc
chksum
e2e
gzip
lz4
s2
snappy
viz
dot
dot/demo command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.