Help us shape the future of Go! Take the 2026 Go Developer Survey

block

package
v0.57.30 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package block defines a common pattern for interacting with block reference structures in Hydra and in memory.

Index

Constants

View Source 
const (
	// StoreFeatureNativeBatchPut means PutBlockBatch is implemented natively.
	StoreFeatureNativeBatchPut = StoreFeature_STORE_FEATURE_NATIVE_BATCH_PUT
	// StoreFeatureNativeBatchExists means GetBlockExistsBatch is implemented natively.
	StoreFeatureNativeBatchExists = StoreFeature_STORE_FEATURE_NATIVE_BATCH_EXISTS
	// StoreFeatureSelfBuffered means the store has its own read-through pending
	// buffer, so the world block engine must not wrap it in a BufferedStore.
	// Volume.Sync remains the durability barrier for volume-backed stores.
	StoreFeatureSelfBuffered = StoreFeature_STORE_FEATURE_SELF_BUFFERED
)
View Source 
const (
	// DecodedBlockCacheNoTransformKey identifies untransformed decoded-cache entries.
	DecodedBlockCacheNoTransformKey = "transform:none"
)
View Source 
const (
	// DefaultDecodedBlockCacheMaxCost is the default decoded-object cache
	// budget. The process-wide pool uses it for every block store together.
	DefaultDecodedBlockCacheMaxCost int64 = 256 << 20
)
View Source 
const DefaultGraphCopyReads = 512

DefaultGraphCopyReads is the number of source reads a graph copy keeps in flight when the caller sets none. A peer source serves one block per round trip, so the copy runs at most this many blocks per round trip: about 13,600 blocks a second at a 4.7 ms ping. More reads raise that ceiling and the source's lookup load with it.

View Source 
const (
	// DefaultHashType is the hash type for new refs when callers do not choose one.
	//
	// SHA256 matches hash.RecommendedHashType: SubtleCrypto in browsers
	// supports only the SHA family, and the goscript build sums only SHA-256
	// and SHA-1. Peers must compute identical refs for identical content, so
	// every build target must agree on this default for content addressing to
	// converge. A ref embeds its hash type in hash.Hash, so blocks written
	// under a previous default remain readable if this changes; the same
	// content would then hash to a new ref, and old and new writers would no
	// longer deduplicate or find each other's blocks.
	DefaultHashType = hash.HashType_HashType_SHA256
)
View Source 
const MaxBlockSize = 10485760

MaxBlockSize is the default maximum size in bytes (10MB) for a single serialized block accepted on the wire by transports such as DEX.

This is a sanity / DoS ceiling, not a structural limit on any one block type. Individual block types are naturally bounded well below this value:

  • blob.Blob (root): inline RawData is capped by blob.DefRawHighWaterMark = blob.DefChunkingMaxSize = 786432 (768 KiB). Above that the Blob auto-converts to CHUNKED, where the root only stores a ChunkIndex of references and individual chunk data lives in separate byteslice blocks.
  • blob chunk data (byteslice.ByteSlice): one chunk per block, capped by blob.DefChunkingMaxSize = 786432 (768 KiB).
  • blob.ChunkIndex / IAVL node / other index blocks: hold only references and small metadata, well under 1 MiB in practice.

10 MiB therefore leaves roughly an order of magnitude of headroom over the largest block any current producer will emit, while still bounding the per-message buffer a peer must allocate when receiving a single block.

Variables

View Source 
var (
	StoreFeature_name = map[int32]string{
		0:  "STORE_FEATURE_UNKNOWN",
		1:  "STORE_FEATURE_NATIVE_BATCH_PUT",
		2:  "STORE_FEATURE_NATIVE_BATCH_EXISTS",
		32: "STORE_FEATURE_SELF_BUFFERED",
	}
	StoreFeature_value = map[string]int32{
		"STORE_FEATURE_UNKNOWN":             0,
		"STORE_FEATURE_NATIVE_BATCH_PUT":    1,
		"STORE_FEATURE_NATIVE_BATCH_EXISTS": 2,
		"STORE_FEATURE_SELF_BUFFERED":       32,
	}
)

Enum value maps for StoreFeature.

View Source 
var (
	OverlayMode_name = map[int32]string{
		0: "UPPER_ONLY",
		1: "LOWER_ONLY",
		2: "UPPER_CACHE",
		3: "LOWER_CACHE",
		4: "UPPER_READ_CACHE",
		5: "LOWER_READ_CACHE",
		6: "UPPER_WRITE_CACHE",
		7: "LOWER_WRITE_CACHE",
		8: "UPPER_READBACK_CACHE",
	}
	OverlayMode_value = map[string]int32{
		"UPPER_ONLY":           0,
		"LOWER_ONLY":           1,
		"UPPER_CACHE":          2,
		"LOWER_CACHE":          3,
		"UPPER_READ_CACHE":     4,
		"LOWER_READ_CACHE":     5,
		"UPPER_WRITE_CACHE":    6,
		"LOWER_WRITE_CACHE":    7,
		"UPPER_READBACK_CACHE": 8,
	}
)

Enum value maps for OverlayMode.

View Source 
var (
	// ErrBlockStoreUnavailable is returned when Fetch is called against a nil block store.
	ErrBlockStoreUnavailable = errors.New("block store is unavailable")
	// ErrUnexpectedType is returned if a type assertion failed.
	ErrUnexpectedType = errors.New("block: unexpected object type")
	// ErrNilCursor is returned when a non-nil block cursor is required.
	ErrNilCursor = errors.New("block cursor cannot be nil")
	// ErrNilBlock is returned when a non-nil block is required.
	ErrNilBlock = errors.New("block cannot be nil")
	// ErrEmptyBlock is returned when a non-empty block is required.
	ErrEmptyBlock = errors.New("block data cannot be nil")
	// ErrEmptyBlockRef is returned a ref was required but was empty.
	ErrEmptyBlockRef = errors.New("empty block reference")
	// ErrNotBlock is returned if the object did not implement Block.
	ErrNotBlock = errors.New("object must be a block")
	// ErrNotSubBlock is returned if the block did not implement SubBlock.
	ErrNotSubBlock = errors.New("block must be a sub-block")
	// ErrNotBlockWithSubBlocks is returned if the block did not implement BlockWithSubBlocks.
	ErrNotBlockWithSubBlocks = errors.New("block must implement block with sub-blocks")
	// ErrEmptyChanges is returned if a slice of changes was unexpectedly empty.
	ErrEmptyChanges = errors.New("changes set cannot be empty")
	// ErrNotFound is returned when a block was not found but was required.
	ErrNotFound = errors.New("block not found")
	// ErrUnavailable is returned when a read of a block that may exist failed:
	// no source served it, so a later read may succeed. Unlike ErrNotFound, it
	// does not report the block missing.
	ErrUnavailable = errors.New("block unavailable")
	// ErrRefsUnknown is returned when a graph copy reaches a block whose source
	// holds its bytes without its outgoing refs.
	ErrRefsUnknown = errors.New("block refs unknown")
	// ErrReferenceCycle is returned when a write finds a block reachable from itself.
	ErrReferenceCycle = errors.New("block: reference cycle")
	// ErrNotClonable is returned if a block could not be cloned.
	ErrNotClonable = errors.New("block: unable to clone")
	// ErrBlockRefMismatch is returned if the data does not match the expected ref.
	ErrBlockRefMismatch = errors.New("block: block ref hash mismatch")
	// ErrBufferedStoreFull is returned when a buffered store reaches its memory limits.
	ErrBufferedStoreFull = errors.New("block: buffered store is full")
	// ErrAtomicPublicationUnsupported is returned before admission, with no side
	// effects, when a store cannot atomically persist blocks, ownership and metadata.
	// It is the only publication error for which a caller may use a legacy path.
	ErrAtomicPublicationUnsupported = errors.New("atomic publication unsupported")
	// ErrPublicationClosed means the publication writer no longer accepts work.
	ErrPublicationClosed = errors.New("publication writer closed")
	// ErrPublicationTooLarge rejects a publication that cannot fit the writer's
	// bounded admission budget. Large block bodies should be prepared durably first.
	ErrPublicationTooLarge = errors.New("publication exceeds bounded admission budget")
	// ErrPublicationDependency means a predecessor failed or was not submitted to
	// this writer before its dependent publication.
	ErrPublicationDependency = errors.New("publication predecessor did not succeed")
	// ErrStageReleased rejects a write through a stage that was released or
	// reaped, which can no longer own the blocks.
	ErrStageReleased = errors.New("block: stage released")
)

Functions

func ApplySubBlock 

func ApplySubBlock[T SubBlock](r *T, next SubBlock) error

ApplySubBlock applies a sub-block to a field.

func BeginDeferFlush added in v0.52.0 

func BeginDeferFlush(store StoreOps)

BeginDeferFlush opens a deferred GC-flush scope on store when it batches GC ref-graph flushes. Stores without a GC ref-graph are a no-op.

func CloneBlock 

func CloneBlock(blk any) (any, error)

CloneBlock tries to clone an input block.

The block should implement proto.Message or BlockWithClone.

returns ErrUnexpectedType or ErrNotClonable if the block was not clonable.

func CompareNamedSubBlocks 

func CompareNamedSubBlocks[T ComparableNamedSubBlock](a, b []T) (added, removed, changed []T)

CompareNamedSubBlocks compares two sets of ComparableNamedSubBlock. Returns the added, removed, and changed values.

func CopyGraph 

func CopyGraph(ctx context.Context, src, dst StoreOps, root *BlockRef, opts *GraphCopyOptions) error

CopyGraph copies the block DAG under root from src to dst by following the outgoing refs src reports, and writes each block with its refs, so dst rebuilds the same ref graph. It never decodes a block. A src that is a CopySource is read through its copy source.

Each block is written only after every block under it, in post-order, the order a tree write produces. A destination that stages new blocks under a temporary owner hands each child to its parent when the parent lands, and a written parent finds every child present except a hole Missing accepted.

Returns ErrNotFound or ErrUnavailable for a block src cannot serve, unless Missing accepts it, and ErrRefsUnknown for a block src holds without its refs, each wrapped with the block ref. Known, Complete, Visited and Missing run on the calling goroutine, one at a time.

func DisableSharedDecodedBlockCache 

func DisableSharedDecodedBlockCache()

DisableSharedDecodedBlockCache stops the process-wide pool behind NewDecodedBlockCache from serving or retaining blocks, so every read decodes what the store returns. A read operation still reuses the blocks it decoded itself.

func DurableReadGeneration 

func DurableReadGeneration() uint64

DurableReadGeneration returns the removal generation a durable store records before it opens the snapshot that serves a read, and later passes to MarkDurableRead. Reading it after the snapshot opens could admit a block a removal deleted in between.

func EncodeBlockObject 

func EncodeBlockObject(data []byte, refs []*BlockRef) ([]byte, error)

EncodeBlockObject encodes a block's bytes with its outgoing refs.

func EndDeferFlush added in v0.52.0 

func EndDeferFlush(ctx context.Context, store StoreOps) error

EndDeferFlush closes a deferred GC-flush scope on store when it batches GC ref-graph flushes. Stores without a GC ref-graph are a no-op.

func HoldCompleteRoots 

func HoldCompleteRoots(ctx context.Context, stage StoreOps, refs []*BlockRef) ([]bool, error)

HoldCompleteRoots reports which roots have a completion proof and makes the stage own each proved root, so the graph under it survives until the stage is released. A copy that skips a proved subtree holds it this way until the parent that references it lands: a proof read without a hold can lose its subtree to a sweep first. stage is a store OpenStage returned. A store that cannot hold proofs reports none.

func InvalidateAllRemovedBlocks 

func InvalidateAllRemovedBlocks()

InvalidateAllRemovedBlocks records that a durable store removed every block it held, such as when its volume is deleted. Every entry in the pool becomes a miss, in every scope: an entry records the scope that read it, not the store that served it, so the pool cannot tell which entries the removed store backed. Deleting a store is rare, and the cost is one miss per entry.

func InvalidateRemovedBlock 

func InvalidateRemovedBlock(ref *BlockRef)

InvalidateRemovedBlock records that a durable store removed ref. Call it after the removal is durable, such as after the transaction that deleted the block commits; calling it for a removal that did not happen only costs a miss.

func InvalidateRemovedTombstones 

func InvalidateRemovedTombstones(entries []*PutBatchEntry)

InvalidateRemovedTombstones records each tombstone entry of a batch a durable store applied as a removal, as InvalidateRemovedBlock does.

func IsBackground 

func IsBackground(ctx context.Context) bool

IsBackground reports whether ctx is marked as background work.

func IsNamedSubBlocksSorted 

func IsNamedSubBlocksSorted[T NamedSubBlock](namedSubBlocks []T) bool

IsNamedSubBlocksSorted checks if the set of NamedSubBlock is sorted.

func IsNotAvailable 

func IsNotAvailable(err error) bool

IsNotAvailable reports whether err means a block is not available here now: missing from the store, or not served by any source. A reader that needs the block can wait for it and retry, so the failure is not a property of the data that refers to it.

func MarkDerivedRootsComplete 

func MarkDerivedRootsComplete(ctx context.Context, store StoreOps, base *BlockRef, roots ...*BlockRef) error

MarkDerivedRootsComplete records completion proofs for roots written over base, when base is empty or proven complete. A write stores only the blocks it changed, so the subtrees it kept are complete only where base's were: a device that reads base on demand from a peer holds just the changed path. Only a writer that stored every block its write references may derive a proof: a replay of another device's write may point at blocks only their author holds.

func MarkDurableRead 

func MarkDurableRead(ctx context.Context, ref *BlockRef, gen uint64)

MarkDurableRead records on ctx that a durable store served ref from a snapshot opened after DurableReadGeneration returned gen. Only a store whose served bytes are committed may call it, and only for a block it found; a store that calls it must also report its removals with InvalidateRemovedBlock.

func MarkRootComplete 

func MarkRootComplete(ctx context.Context, store StoreOps, ref *BlockRef) error

MarkRootComplete records that a locally constructed DAG has been fenced, or written ahead of the proof on a volume that makes writes durable in order. Its normal block writes must have supplied all outgoing reference metadata.

func MarkRootsComplete 

func MarkRootsComplete(ctx context.Context, store StoreOps, roots []*BlockRef) error

MarkRootsComplete fences a batch of completion proofs in the owning volume. Each root's writes must have supplied all outgoing reference metadata.

func MarkStagedRead 

func MarkStagedRead(ctx context.Context)

MarkStagedRead records on ctx that the block being read came from staged content: a write buffer, a publication being prepared, or any other bytes that are not yet durably stored. It vetoes admission even when a durable store also marked the read.

func MarshalBlockRefJSON 

func MarshalBlockRefJSON(ref *BlockRef) ([]byte, error)

MarshalBlockRefJSON marshals an BlockRef to JSON.

Returns "null" if the ref is nil.

func NewPublicationDependencyError 

func NewPublicationDependencyError(cause error) error

NewPublicationDependencyError wraps a failed or unordered predecessor result with ErrPublicationDependency. Use it to reject a dependent publication.

func NewTransaction 

func NewTransaction(
	store StoreOps,
	transformer Transformer,
	rootRef *BlockRef,
	putOpts *PutOpts,
) (*Transaction, *Cursor)

NewTransaction builds a new transaction with a root cursor. The transformer, root reference, and put options are optional.

func NotFoundError 

func NotFoundError(ref *BlockRef) error

NotFoundError returns ErrNotFound naming ref, so a log of the failure says which block is missing. errors.Is still matches ErrNotFound.

func PinRoot 

func PinRoot(ctx context.Context, store StoreOps, ref *BlockRef) (func(), error)

PinRoot protects a root until release. The caller must already hold a live parent or publication authority while acquiring the pin.

func Prefetch 

func Prefetch(ctx context.Context, store StoreOps, refs []*BlockRef, concurrency int) error

Prefetch reads refs from store in order, with at most concurrency reads in flight, so a read-through store caches them before a program that reads them in the same order needs them. Reads start in list order; a read that fails does not stop the others. With WithReadAhead on ctx, a remote store widens each miss into a range read, so neighbouring blocks of one pack arrive together. Returns the first read error, or the context error when ctx ends first.

func ReadAhead 

func ReadAhead(ctx context.Context) int

ReadAhead returns the requested range-fetch minimum, or zero when unset.

func RecordDecodedBlockCacheHit added in v0.51.7 

func RecordDecodedBlockCacheHit(ctx context.Context, cloned bool)

RecordDecodedBlockCacheHit records a clone-safe decoded-block cache hit.

func RecordDecodedBlockUncacheable added in v0.51.7 

func RecordDecodedBlockUncacheable(ctx context.Context)

RecordDecodedBlockUncacheable records a decoded-block cache bypass without an exact key.

func RecordDecodedBlockUncloneable added in v0.51.7 

func RecordDecodedBlockUncloneable(ctx context.Context)

RecordDecodedBlockUncloneable records a decoded-block cache candidate that cannot be cloned.

func RecordResourceGetBlock added in v0.51.7 

func RecordResourceGetBlock(ctx context.Context, ref *BlockRef, found bool, bytes int)

RecordResourceGetBlock records a Resource SDK GetBlock call.

func RecordWrite 

func RecordWrite(ctx context.Context, bytes int)

RecordWrite records one encoded block submitted for storage on the counter attached to ctx, if any.

func ReleaseRoots 

func ReleaseRoots(ctx context.Context, store StoreOps, refs []*BlockRef) error

ReleaseRoots drops the staging ownership the store holds for roots the caller wrote outside any named root. Each root then survives only through another block's reference, a named root, or a reader pin.

func RootComplete 

func RootComplete(ctx context.Context, store StoreOps, ref *BlockRef) (bool, error)

RootComplete checks a proof whose lifetime is the physical root's lifetime.

func RootsComplete 

func RootsComplete(ctx context.Context, store StoreOps, refs []*BlockRef) ([]bool, error)

RootsComplete checks the proofs of a batch of roots in one transaction. It reports the proof of each root in order; an empty root has none, and a store without root ownership proves nothing.

func SetRetainedRoot 

func SetRetainedRoot(ctx context.Context, store StoreOps, name string, ref *BlockRef) error

SetRetainedRoot advances a named root when the store supports root ownership. The caller serializes this operation with publication of the corresponding head.

func SetRetainedRoots 

func SetRetainedRoots(ctx context.Context, store StoreOps, entries []*PutBatchEntry, roots []NamedRoot) error

SetRetainedRoots writes entries and advances every named root in roots as one change when the store supports root ownership. A root may hold a block entries writes. The caller serializes this operation with publication of the corresponding heads. A store without root ownership writes nothing.

func SetSharedDecodedBlockCacheMaxCost 

func SetSharedDecodedBlockCacheMaxCost(maxCost int64)

SetSharedDecodedBlockCacheMaxCost sets the budget of the process-wide pool behind NewDecodedBlockCache. The default suits a desktop; a server with memory to spare can hold a larger working set. Lowering the budget evicts entries from each shard at its next store. A non-positive budget is ignored.

func SortNamedSubBlocks 

func SortNamedSubBlocks[T NamedSubBlock](namedSubBlocks []T)

SortNamedSubBlocks sorts a set of NamedSubBlock.

func SupportsRootRetention 

func SupportsRootRetention(store StoreOps) bool

SupportsRootRetention reports whether a store implements root ownership.

func UnmarshalBlock 

func UnmarshalBlock[T any, PT Ptr[T]](ctx context.Context, bcs *Cursor) (PT, error)

UnmarshalBlock unmarshals the block at the cursor as a *T.

Returns ErrUnexpectedType if the cursor already holds a block of another type. Returns nil, nil if bcs is nil or the cursor has no block.

func WalkGraph 

func WalkGraph(ctx context.Context, src StoreOps, roots []*BlockRef, reads int, visit func(ref *BlockRef, stored *StoredBlock) error) error

WalkGraph visits every block under roots once by following the outgoing refs src reports, keeping reads source reads in flight. Zero reads selects DefaultGraphCopyReads. It never decodes a block.

Returns ErrNotFound for a missing block and ErrRefsUnknown for a block src holds without its refs, each wrapped with the block ref. visit runs on the calling goroutine, one block at a time.

func WithAccessLogger 

func WithAccessLogger(ctx context.Context, logger *AccessLogger) context.Context

WithAccessLogger returns a context whose block reads record to logger.

func WithBackground 

func WithBackground(ctx context.Context) context.Context

WithBackground marks ctx as background work, such as a fill no person is waiting for. A store that fetches remote data admits a background fetch only while no foreground fetch on that store needs the link. Reads served from local or resident data do not wait.

func WithDecodedBlockCache added in v0.51.7 

func WithDecodedBlockCache(ctx context.Context, cache *DecodedBlockCache) context.Context

WithDecodedBlockCache attaches a decoded-block cache to ctx.

func WithReadAhead 

func WithReadAhead(ctx context.Context, bytes int) context.Context

WithReadAhead requests at least bytes per remote range-cache miss for this operation. Stores may limit the request to an uncovered gap, the end of a file, or their transport and memory bounds. Resident reads need no fetch. The hint does not change a shared store's default policy.

func WithReadOperationStore added in v0.51.7 

func WithReadOperationStore(ctx context.Context, store StoreOps) context.Context

WithReadOperationStore routes the current cursor operation through store. Storage implementations receive a context without this routing override so their own backing cursors remain bound to their underlying stores.

Types

type Access 

type Access struct {

	// Ref is the block read.
	Ref *BlockRef `protobuf:"bytes,1,opt,name=ref,proto3" json:"ref,omitempty"`
	// Size is the stored size of the block in bytes.
	Size uint32 `protobuf:"varint,2,opt,name=size,proto3" json:"size,omitempty"`
	// ElapsedUs is the time from the start of the log to the read, in
	// microseconds.
	ElapsedUs uint64 `protobuf:"varint,3,opt,name=elapsed_us,json=elapsedUs,proto3" json:"elapsedUs,omitempty"`
	// contains filtered or unexported fields
}

Access is the first read of one block.

func (*Access) CloneMessageVT 

func (m *Access) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*Access) CloneVT 

func (m *Access) CloneVT() *Access

func (*Access) EqualMessageVT 

func (this *Access) EqualMessageVT(thatMsg any) bool

func (*Access) EqualVT 

func (this *Access) EqualVT(that *Access) bool

func (*Access) GetElapsedUs 

func (x *Access) GetElapsedUs() uint64

func (*Access) GetRef 

func (x *Access) GetRef() *BlockRef

func (*Access) GetSize 

func (x *Access) GetSize() uint32

func (*Access) MarshalJSON 

func (x *Access) MarshalJSON() ([]byte, error)

MarshalJSON marshals the Access to JSON.

func (*Access) MarshalProtoJSON 

func (x *Access) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the Access message to JSON.

func (*Access) MarshalProtoText 

func (x *Access) MarshalProtoText() string

func (*Access) MarshalToSizedBufferVT 

func (m *Access) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*Access) MarshalToVT 

func (m *Access) MarshalToVT(dAtA []byte) (int, error)

func (*Access) MarshalVT 

func (m *Access) MarshalVT() (dAtA []byte, err error)

func (*Access) ProtoMessage 

func (*Access) ProtoMessage()

func (*Access) Reset 

func (x *Access) Reset()

func (*Access) SizeVT 

func (m *Access) SizeVT() (n int)

func (*Access) String 

func (x *Access) String() string

func (*Access) UnmarshalJSON 

func (x *Access) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the Access from JSON.

func (*Access) UnmarshalProtoJSON 

func (x *Access) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the Access message from JSON.

func (*Access) UnmarshalVT 

func (m *Access) UnmarshalVT(dAtA []byte) error

type AccessLog 

type AccessLog struct {

	// Accesses are the first reads of each block.
	Accesses []*Access `protobuf:"bytes,1,rep,name=accesses,proto3" json:"accesses,omitempty"`
	// contains filtered or unexported fields
}

AccessLog lists the blocks an operation read, once each, in the order of first use.

func (*AccessLog) CloneMessageVT 

func (m *AccessLog) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*AccessLog) CloneVT 

func (m *AccessLog) CloneVT() *AccessLog

func (*AccessLog) EqualMessageVT 

func (this *AccessLog) EqualMessageVT(thatMsg any) bool

func (*AccessLog) EqualVT 

func (this *AccessLog) EqualVT(that *AccessLog) bool

func (*AccessLog) GetAccesses 

func (x *AccessLog) GetAccesses() []*Access

func (*AccessLog) MarshalJSON 

func (x *AccessLog) MarshalJSON() ([]byte, error)

MarshalJSON marshals the AccessLog to JSON.

func (*AccessLog) MarshalProtoJSON 

func (x *AccessLog) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the AccessLog message to JSON.

func (*AccessLog) MarshalProtoText 

func (x *AccessLog) MarshalProtoText() string

func (*AccessLog) MarshalToSizedBufferVT 

func (m *AccessLog) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*AccessLog) MarshalToVT 

func (m *AccessLog) MarshalToVT(dAtA []byte) (int, error)

func (*AccessLog) MarshalVT 

func (m *AccessLog) MarshalVT() (dAtA []byte, err error)

func (*AccessLog) ProtoMessage 

func (*AccessLog) ProtoMessage()

func (*AccessLog) Reset 

func (x *AccessLog) Reset()

func (*AccessLog) SizeVT 

func (m *AccessLog) SizeVT() (n int)

func (*AccessLog) String 

func (x *AccessLog) String() string

func (*AccessLog) UnmarshalJSON 

func (x *AccessLog) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the AccessLog from JSON.

func (*AccessLog) UnmarshalProtoJSON 

func (x *AccessLog) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the AccessLog message from JSON.

func (*AccessLog) UnmarshalVT 

func (m *AccessLog) UnmarshalVT(dAtA []byte) error

type AccessLogger 

type AccessLogger struct {
	// contains filtered or unexported fields
}

AccessLogger records the first read of each block, in order. It is safe for concurrent use.

func GetAccessLogger 

func GetAccessLogger(ctx context.Context) *AccessLogger

GetAccessLogger returns the access logger of the context, or nil.

func NewAccessLogger 

func NewAccessLogger() *AccessLogger

NewAccessLogger starts an empty access log.

func (*AccessLogger) Snapshot 

func (l *AccessLogger) Snapshot() *AccessLog

Snapshot returns the reads logged so far.

type AliasIdentityToken added in v0.52.0 

type AliasIdentityToken = hash.AliasIdentityToken

AliasIdentityToken is an opaque per-instance token used to detect in-memory block aliases while marshaling a transaction.

type AtomicBlockPreparer 

type AtomicBlockPreparer interface {
	// PrepareOwnedBlock durably prepares one oversized body with bucket ownership.
	PrepareOwnedBlock(ctx context.Context, bucketID string, data []byte, opts *PutOpts) (*BlockRef, bool, error)
	// PrepareOwnedBlockBatch durably prepares a batch with bucket ownership.
	// Returns whether each block was stored before the write, like PutBlockBatch.
	PrepareOwnedBlockBatch(ctx context.Context, bucketID string, entries []*PutBatchEntry) ([]bool, error)
}

AtomicBlockPreparer persists construction overflow in the same transaction as its bucket ownership, without changing a World head. Calls are synchronous; the caller retains its buffers until return and no oversized entry is queued. An empty bucketID disables GC bookkeeping. Unsupported is returned before I/O. This capability complements, rather than bypasses, bounded publication admission.

type AtomicHeadUpdate 

type AtomicHeadUpdate struct {
	// ObjectStoreID scopes the metadata key.
	ObjectStoreID string
	// Key is the metadata key relative to the object store.
	Key []byte
	// Replace maps the committed record to its replacement.
	Replace func(ctx context.Context, current []byte, found bool) ([]byte, error)
}

AtomicHeadUpdate describes an in-process metadata compare-and-swap in the same transaction as Entries. Key is relative to ObjectStoreID. Replace must only inspect the supplied committed record and return its replacement; it must not perform I/O or mutate the publication. A failed comparison rejects this publication before any of its entries are applied.

The callback allows transformed metadata to compare decoded references rather than comparing nondeterministically encrypted encodings. It is not a wire API.

type AtomicPublication 

type AtomicPublication struct {
	// Entries are the block writes applied by this publication.
	Entries []*PutBatchEntry
	// After is the earlier publication on which this prepared revision depends.
	// It must belong to this writer and precede this request in admission order.
	After *PublicationReceipt
	// Heads are the metadata compare-and-swaps applied with Entries. At least
	// one is required, and every key is distinct.
	Heads []*AtomicHeadUpdate
	// RootName and Root transfer the published DAG from bucket staging to a
	// durable named root in the same transaction as Heads. Empty RootName keeps
	// the caller's existing retention policy. A submitted named root also stays
	// pinned until the caller releases its PublicationReceipt.
	RootName string
	// Root is the published DAG root RootName holds.
	Root *BlockRef
	// Roots are further named roots set in the same transaction, without a
	// reader pin. They apply only with TrackGC, as RootName does.
	Roots []NamedRoot
	// Ordered lets the physical commit be ordered rather than durable: applied
	// when the receipt resolves, and durable no later than the volume's next
	// durable commit. A group commits ordered only when every member allows it.
	Ordered bool
	// Validate optionally inspects the prepared overlay before mutation.
	Validate func(ctx context.Context, prepared StoreOps) error
	// BucketID scopes GC ownership; populated by the bucket capability.
	BucketID string
	// TrackGC enables GC reference bookkeeping for the entries.
	TrackGC bool
}

AtomicPublication is an immutable, bounded unit of durable publication. Entries and everything they reference must remain unchanged until Done. Validate sees a read-only overlay containing Entries and the physical write transaction, including preceding successful publications in the same group. Validation and head comparison run before any mutations for this publication. BucketID and TrackGC are populated by the bucket capability, not by the World. No independent transaction, reference scan, or application callback may be opened by Validate while the physical transaction is held.

type AtomicPublisher 

type AtomicPublisher interface {
	// AtomicPublicationVolumeID identifies the shared blocks/metadata domain.
	AtomicPublicationVolumeID() string
	// SupportsAtomicPublication reports whether atomic publication is available.
	SupportsAtomicPublication() bool
	// SubmitAtomic admits a publication and returns its completion receipt.
	SubmitAtomic(ctx context.Context, publication *AtomicPublication) (*PublicationReceipt, error)
	// PublishAtomic submits a publication and waits for its durable result.
	PublishAtomic(ctx context.Context, publication *AtomicPublication) error
}

AtomicPublisher is the optional, shared in-process publication capability. SupportsAtomicPublication is stable during the publisher's lifetime. A successful SubmitAtomic transfers an immutable borrow until receipt.Done; admission is NOT durability. Once admitted, cancellation of a waiter does not cancel the publication or imply that it failed. Close must join accepted work. PublishAtomic is SubmitAtomic followed by an uncancelled durability wait after admission, preserving the legacy Commit contract of an unambiguous result.

type Block 

type Block interface {
	// MarshalVT marshals the block to binary.
	// This is the initial step of marshaling, before transformations.
	MarshalVT() ([]byte, error)
	// UnmarshalVT unmarshals the block to the object.
	// This is the final step of decoding, after transformations.
	UnmarshalVT(data []byte) error
}

Block defines an in-memory decoded block structure. A block should contain a minimal amount of data with some pointers to other blocks.

Every protobuf-go-lite message is a Block; other encodings implement the same two methods.

func CastToBlock 

func CastToBlock(sb any) (Block, error)

CastToBlock casts a object to a block or returns an error.

func NewBlockRefBlock 

func NewBlockRefBlock() Block

NewBlockRefBlock constructs a new block reference block.

type BlockObject 

type BlockObject struct {

	// Data is the encoded block.
	Data []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"`
	// Refs are the block's outgoing refs. Empty means a leaf.
	Refs []*BlockRef `protobuf:"bytes,2,rep,name=refs,proto3" json:"refs,omitempty"`
	// contains filtered or unexported fields
}

BlockObject is a block's bytes with its outgoing refs, as stored by backends that keep blocks as opaque objects.

func (*BlockObject) CloneMessageVT 

func (m *BlockObject) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*BlockObject) CloneVT 

func (m *BlockObject) CloneVT() *BlockObject

func (*BlockObject) EqualMessageVT 

func (this *BlockObject) EqualMessageVT(thatMsg any) bool

func (*BlockObject) EqualVT 

func (this *BlockObject) EqualVT(that *BlockObject) bool

func (*BlockObject) GetData 

func (x *BlockObject) GetData() []byte

func (*BlockObject) GetRefs 

func (x *BlockObject) GetRefs() []*BlockRef

func (*BlockObject) MarshalJSON 

func (x *BlockObject) MarshalJSON() ([]byte, error)

MarshalJSON marshals the BlockObject to JSON.

func (*BlockObject) MarshalProtoJSON 

func (x *BlockObject) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the BlockObject message to JSON.

func (*BlockObject) MarshalProtoText 

func (x *BlockObject) MarshalProtoText() string

func (*BlockObject) MarshalToSizedBufferVT 

func (m *BlockObject) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*BlockObject) MarshalToVT 

func (m *BlockObject) MarshalToVT(dAtA []byte) (int, error)

func (*BlockObject) MarshalVT 

func (m *BlockObject) MarshalVT() (dAtA []byte, err error)

func (*BlockObject) ProtoMessage 

func (*BlockObject) ProtoMessage()

func (*BlockObject) Reset 

func (x *BlockObject) Reset()

func (*BlockObject) SizeVT 

func (m *BlockObject) SizeVT() (n int)

func (*BlockObject) String 

func (x *BlockObject) String() string

func (*BlockObject) UnmarshalJSON 

func (x *BlockObject) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the BlockObject from JSON.

func (*BlockObject) UnmarshalProtoJSON 

func (x *BlockObject) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the BlockObject message from JSON.

func (*BlockObject) UnmarshalVT 

func (m *BlockObject) UnmarshalVT(dAtA []byte) error

type BlockRef 

type BlockRef struct {

	// Hash is the hash of the object.
	Hash *hash.Hash `protobuf:"bytes,1,opt,name=hash,proto3" json:"hash,omitempty"`
	// contains filtered or unexported fields
}

BlockRef is a block content ID reference.

protobuf-go-lite:disable-json

func BuildBlockRef 

func BuildBlockRef(data []byte, putOpts *PutOpts) (*BlockRef, error)

BuildBlockRef builds a block ref from put opts by hashing the data. If putOpts are nil, uses default hash type.

func CloneBlockRefs 

func CloneBlockRefs(refs []*BlockRef) []*BlockRef

CloneBlockRefs returns a deep copy of refs, preserving nil entries. Returns nil for empty input.

func ExtractBlockRefs 

func ExtractBlockRefs(blk any) ([]*BlockRef, error)

ExtractBlockRefs extracts all outgoing BlockRefs from a block by walking BlockWithRefs and recursing into BlockWithSubBlocks.

This captures refs at all nesting levels: direct refs on the block, refs inside sub-blocks (e.g., BlockRefSlice), and refs inside nested sub-blocks. References and sub-blocks are visited in field-ID order, preserving sequence order independently of Go map iteration.

func NewBlockRef 

func NewBlockRef(hash *hash.Hash) *BlockRef

NewBlockRef constructs a new block reference.

func PutBlock 

func PutBlock(ctx context.Context, bk StoreOps, b Block) (*BlockRef, bool, error)

PutBlock marshals b and puts it into bk with the refs it holds.

func UnmarshalBlockRefB58 

func UnmarshalBlockRefB58(ref string) (*BlockRef, error)

UnmarshalBlockRefB58 unmarshals a b58 string block ref.

func UnmarshalBlockRefBlock 

func UnmarshalBlockRefBlock(ctx context.Context, bcs *Cursor) (*BlockRef, error)

UnmarshalBlockRefBlock unmarshals a BlockRef from a block cursor.

func UnmarshalBlockRefJSON 

func UnmarshalBlockRefJSON(data []byte) (*BlockRef, error)

UnmarshalBlockRefJSON attempts to unmarshal an BlockRef from JSON.

func (*BlockRef) ApplyBlockRef 

func (b *BlockRef) ApplyBlockRef(id uint32, ptr *BlockRef) error

ApplyBlockRef applies a ref change with a field id. The reference may be nil if the child block is nil.

func (*BlockRef) Clone 

func (b *BlockRef) Clone() *BlockRef

Clone clones the block ref.

func (*BlockRef) CloneMessageVT 

func (m *BlockRef) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*BlockRef) CloneVT 

func (m *BlockRef) CloneVT() *BlockRef

func (*BlockRef) EqualMessageVT 

func (this *BlockRef) EqualMessageVT(thatMsg any) bool

func (*BlockRef) EqualVT 

func (this *BlockRef) EqualVT(that *BlockRef) bool

func (*BlockRef) EqualsRef 

func (b *BlockRef) EqualsRef(oref *BlockRef) bool

EqualsRef checks if two refs are equal.

func (*BlockRef) GetBlockRefCtor 

func (b *BlockRef) GetBlockRefCtor(id uint32) Ctor

GetBlockRefCtor returns the constructor for the block at the ref id. Return nil to indicate invalid ref ID or unknown.

func (*BlockRef) GetBlockRefs 

func (b *BlockRef) GetBlockRefs() (map[uint32]*BlockRef, error)

GetBlockRefs returns all block references by ID. May return nil, and values may also be nil. Note: this does not include pending references (in a cursor)

func (*BlockRef) GetEmpty 

func (b *BlockRef) GetEmpty() bool

GetEmpty returns if the ref is empty.

func (*BlockRef) GetHash 

func (x *BlockRef) GetHash() *hash.Hash

func (*BlockRef) LessThan 

func (b *BlockRef) LessThan(other *BlockRef) bool

LessThan checks if the ref is less than another. 1. Empty is sorted to the end. 2. Hash type is sorted. 3. Hash itself is sorted in bytes ordering

func (*BlockRef) MarshalJSON 

func (b *BlockRef) MarshalJSON() ([]byte, error)

MarshalJSON marshals the BlockRef to JSON.

func (*BlockRef) MarshalKey 

func (b *BlockRef) MarshalKey() ([]byte, error)

MarshalKey marshals the block ref for use as a key. The format should be reproducible and identical between versions.

func (*BlockRef) MarshalLog 

func (b *BlockRef) MarshalLog() string

MarshalLog marshals the reference for logging. If nil or empty returns <nil>

func (*BlockRef) MarshalProtoJSON 

func (b *BlockRef) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the BlockRef message to JSON.

func (*BlockRef) MarshalProtoText 

func (x *BlockRef) MarshalProtoText() string

func (*BlockRef) MarshalString 

func (b *BlockRef) MarshalString() string

MarshalString marshals the reference to a string form.

func (*BlockRef) MarshalToSizedBufferVT 

func (m *BlockRef) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*BlockRef) MarshalToVT 

func (m *BlockRef) MarshalToVT(dAtA []byte) (int, error)

func (*BlockRef) MarshalVT 

func (m *BlockRef) MarshalVT() (dAtA []byte, err error)

func (*BlockRef) ParseFromB58 

func (b *BlockRef) ParseFromB58(ref string) error

ParseFromB58 parses the object ref from a base58 string.

func (*BlockRef) ProtoMessage 

func (*BlockRef) ProtoMessage()

func (*BlockRef) Reset 

func (x *BlockRef) Reset()

func (*BlockRef) SizeVT 

func (m *BlockRef) SizeVT() (n int)

func (*BlockRef) String 

func (x *BlockRef) String() string

func (*BlockRef) UnmarshalJSON 

func (b *BlockRef) UnmarshalJSON(dat []byte) error

UnmarshalJSON unmarshals the BlockRef from JSON.

func (*BlockRef) UnmarshalProtoJSON 

func (b *BlockRef) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the BlockRef message from JSON.

func (*BlockRef) UnmarshalVT 

func (m *BlockRef) UnmarshalVT(dAtA []byte) error

func (*BlockRef) Validate 

func (b *BlockRef) Validate(allowEmpty bool) error

Validate validates the block ref.

func (*BlockRef) VerifyData 

func (b *BlockRef) VerifyData(data []byte, errDetails bool) error

VerifyData checks the given data matches the block ref. If errDetails is set, wraps the error with the unexpected and expected refs if mismatch.

type BlockStat 

type BlockStat struct {
	// Ref is the block reference.
	Ref *BlockRef
	// Size is the block data size in bytes.
	// May be -1 if the size is unknown.
	Size int64
}

BlockStat contains metadata about a stored block.

type BlockWithAliasIdentity added in v0.52.0 

type BlockWithAliasIdentity interface {
	BlockAliasIdentity() *AliasIdentityToken
}

BlockWithAliasIdentity exposes stable in-memory identity for blocks that can appear both as handles and inline sub-blocks.

type BlockWithClone 

type BlockWithClone interface {
	// CloneBlock returns a deep copy of the block.
	CloneBlock() (Block, error)
}

BlockWithClone defines a block with a clone function. The clone should share nothing with the original.

type BlockWithCloneVT 

type BlockWithCloneVT[T Block] interface {
	// CloneVT clones the block object with VTprotobuf.
	CloneVT() T
}

BlockWithCloneVT defines a block with a VTProtobuf clone function.

type BlockWithPreWriteHook 

type BlockWithPreWriteHook interface {
	// BlockPreWriteHook is called when writing the block.
	BlockPreWriteHook() error
}

BlockWithPreWriteHook is a block with a function called when writing. This can also be applied to a sub-block.

type BlockWithRefs 

type BlockWithRefs interface {
	// ApplyBlockRef applies a ref change with a field id.
	// The reference may be nil if the child block is nil.
	ApplyBlockRef(id uint32, ptr *BlockRef) error
	// GetBlockRefs returns all block references by ID.
	//
	// May return nil, and values may also be nil. Pending references in a
	// cursor are not included.
	GetBlockRefs() (map[uint32]*BlockRef, error)
	// GetBlockRefCtor returns the constructor for the block at the ref id.
	// Return nil to indicate invalid ref ID or unknown.
	GetBlockRefCtor(id uint32) Ctor
}

BlockWithRefs has references keyed by ID. Each field can contain a reference.

type BlockWithSubBlocks 

type BlockWithSubBlocks interface {
	// ApplySubBlock applies a sub-block change with a field id.
	ApplySubBlock(id uint32, next SubBlock) error
	// GetSubBlocks returns all constructed sub-blocks by ID.
	// May return nil, and values may also be nil.
	GetSubBlocks() map[uint32]SubBlock
	// GetSubBlockCtor returns a function which creates or returns the existing
	// sub-block at reference id. Can return nil to indicate invalid reference id.
	GetSubBlockCtor(id uint32) SubBlockCtor
}

BlockWithSubBlocks is a block containing sub-blocks as fields.

type BufferedStore 

type BufferedStore struct {
	// contains filtered or unexported fields
}

BufferedStore buffers PutBlock calls in memory and drains them explicitly on Sync or when a caller must free capacity.

func NewBufferedStore 

func NewBufferedStore(ctx context.Context, inner StoreOps) *BufferedStore

NewBufferedStore constructs a buffered store around an inner store.

func NewBufferedStoreWithSettings 

func NewBufferedStoreWithSettings(
	_ context.Context,
	inner StoreOps,
	settings *BufferedStoreSettings,
) *BufferedStore

NewBufferedStoreWithSettings constructs a buffered store with explicit settings.

func (*BufferedStore) BeginDeferFlush 

func (s *BufferedStore) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to the inner store.

func (*BufferedStore) BeginReadOperation added in v0.51.7 

func (s *BufferedStore) BeginReadOperation(context.Context) (StoreOps, func(), error)

BeginReadOperation returns the buffered store for a read scope.

func (*BufferedStore) DrainReachable 

func (s *BufferedStore) DrainReachable(ctx context.Context, roots ...*BlockRef) error

DrainReachable discards pending writes outside roots, then writes the rest to the inner store. It neither drains inner buffered layers nor applies a durability fence. Ownership is as for SyncReachable.

func (*BufferedStore) EndDeferFlush 

func (s *BufferedStore) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to the inner store. Buffered blocks are never drained here; only Sync and Flush drain.

func (*BufferedStore) Flush 

func (s *BufferedStore) Flush(ctx context.Context) error

Flush drains buffered blocks through every buffered layer into the first unbuffered store, without its durability barrier.

func (*BufferedStore) GetBlock 

func (s *BufferedStore) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block by reference.

func (*BufferedStore) GetBlockExists 

func (s *BufferedStore) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists.

func (*BufferedStore) GetBlockExistsBatch 

func (s *BufferedStore) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch checks if blocks exist.

func (*BufferedStore) GetHashType 

func (s *BufferedStore) GetHashType() hash.HashType

GetHashType returns the preferred hash type.

func (*BufferedStore) GetStoredBlock 

func (s *BufferedStore) GetStoredBlock(ctx context.Context, ref *BlockRef) (*StoredBlock, error)

GetStoredBlock gets a block and its references, preferring pending writes.

func (*BufferedStore) GetSupportedFeatures 

func (s *BufferedStore) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (*BufferedStore) HoldCompleteRoots 

func (s *BufferedStore) HoldCompleteRoots(ctx context.Context, refs []*BlockRef) ([]bool, error)

HoldCompleteRoots holds roots already proved complete in the inner stage. It neither drains pending blocks nor creates completion proofs.

func (*BufferedStore) MarkRootsComplete 

func (s *BufferedStore) MarkRootsComplete(ctx context.Context, roots []*BlockRef) error

MarkRootsComplete forwards the durable World proof.

func (*BufferedStore) OpenStage 

func (s *BufferedStore) OpenStage(ctx context.Context) (StoreOps, func(), error)

OpenStage opens a stage on the destination. Reads go through this buffer, so a staged build sees blocks still pending here; writes go to the stage.

func (*BufferedStore) PinRoot 

func (s *BufferedStore) PinRoot(ctx context.Context, ref *BlockRef) (func(), error)

PinRoot acquires a reader pin on ref. A root still pending here is not in the inner store, so no sweep can collect it: the pin waits in the buffer and becomes an inner pin when the drain writes the root. A pin released before then costs the inner store nothing.

func (*BufferedStore) PutBlock 

func (s *BufferedStore) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock buffers a block in memory and drains synchronously only when backpressure requires capacity.

func (*BufferedStore) PutBlockBatch 

func (s *BufferedStore) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) ([]bool, error)

PutBlockBatch buffers verified entries without per-block existence probes. Pending entries deduplicate locally; durable duplicates resolve during drain. A block is reported as existing only when a pending entry already holds it.

func (*BufferedStore) ReleaseRoots 

func (s *BufferedStore) ReleaseRoots(ctx context.Context, refs []*BlockRef) error

ReleaseRoots fences prepared blocks so their staging edges exist before they are released.

func (*BufferedStore) ReleaseUnreached 

func (s *BufferedStore) ReleaseUnreached(ctx context.Context, roots ...*BlockRef) error

ReleaseUnreached drops the staging ownership the inner store holds for each recorded block that roots do not reach through recorded references, then clears the record. Blocks of an abandoned write and blocks a later write replaced then survive only through another owner, such as a parent block written before this store existed.

Call it once the writer is done: blocks still pending are not recorded, and a block released here is collectable by the next sweep. It waits for a drain in progress so the record includes the blocks that drain writes.

func (*BufferedStore) RmBlock 

func (s *BufferedStore) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block by reference.

func (*BufferedStore) RootsComplete 

func (s *BufferedStore) RootsComplete(ctx context.Context, refs []*BlockRef) ([]bool, error)

RootsComplete checks the underlying World proofs.

func (*BufferedStore) SetRetainedRoots 

func (s *BufferedStore) SetRetainedRoots(ctx context.Context, entries []*PutBatchEntry, roots []NamedRoot) error

SetRetainedRoots fences prepared blocks before publishing their durable roots. The drain also turns deferred reader pins into inner pins.

func (*BufferedStore) StatBlock 

func (s *BufferedStore) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data.

func (*BufferedStore) SupportsRootRetention 

func (s *BufferedStore) SupportsRootRetention() bool

SupportsRootRetention reports the destination's root ownership capability.

func (*BufferedStore) Sync added in v0.52.0 

func (s *BufferedStore) Sync(ctx context.Context) (bool, error)

Sync drains buffered blocks, then forwards the durability barrier to inner. Draining is owned by Sync, Flush, and backpressure inside PutBlock.

func (*BufferedStore) SyncReachable 

func (s *BufferedStore) SyncReachable(ctx context.Context, roots ...*BlockRef) (bool, error)

SyncReachable discards pending writes outside roots, then applies the normal durability fence. Reachability follows the outgoing refs of each pending block and, when the store records writes, of each block it already wrote to the inner store. A capacity drain can write a parent before its children arrive, so a store that may drain before this call must record writes.

The caller must exclusively own the buffer and stop all writers first. Use this for a newly constructed snapshot, not a buffer shared with other work. It never removes blocks from the inner store.

func (*BufferedStore) TakePending 

func (s *BufferedStore) TakePending(ctx context.Context) (*PendingBatch, error)

TakePending borrows all currently queued entries without writing or forgetting them. Further writes can prepare the next publication while this borrow is in flight. Both queued and borrowed content count against the existing bounds.

type BufferedStoreSettings 

type BufferedStoreSettings struct {
	// MaxPendingEntries is the maximum queued entries before a drain.
	MaxPendingEntries int
	// MaxPendingBytes is the maximum retained payload bytes before a drain.
	MaxPendingBytes int
	// MaxPendingMetadataBytes bounds retained reference data plus conservative
	// per-entry/reference accounting, independently of payload bytes.
	MaxPendingMetadataBytes int
	// DrainBatchEntries is the number of entries written per drain batch.
	DrainBatchEntries int
	// RecordWrites records every block written to the inner store with its
	// outgoing refs. DrainReachable and SyncReachable follow refs through the
	// recorded blocks, and ReleaseUnreached releases the recorded blocks a
	// final root does not reach.
	RecordWrites bool
	// WithoutPeerWait makes reads of the inner store report ErrUnavailable
	// for a block no connected peer can serve, instead of waiting for a peer
	// to connect. Set it when the reader holds a lock others wait on.
	WithoutPeerWait bool
}

BufferedStoreSettings configures buffered block writeback behavior.

func DefaultBufferedStoreSettings 

func DefaultBufferedStoreSettings() *BufferedStoreSettings

DefaultBufferedStoreSettings returns the default buffered store settings.

type ComparableNamedSubBlock 

type ComparableNamedSubBlock interface {
	NamedSubBlock

	// Equals compares the block to the other block for equality.
	Equals(ot ComparableNamedSubBlock) bool
}

ComparableNamedSubBlock is a NamedSubBlock that has an Equals function.

type ComparableSubBlock 

type ComparableSubBlock interface {
	comparable
	SubBlock
}

ComparableSubBlock is the type constraint for SubBlock.

type CompleteRootHolder 

type CompleteRootHolder interface {
	// HoldCompleteRoots reports which roots have a completion proof and makes
	// the stage own each of them in the same transaction.
	HoldCompleteRoots(context.Context, []*BlockRef) ([]bool, error)
}

CompleteRootHolder is implemented by a stage store whose volume checks completion proofs under the stage's ownership.

type CopySource 

type CopySource interface {
	// GetCopySource returns the store a graph copy reads from.
	GetCopySource() StoreOps
}

CopySource is implemented by a store whose reads fill a cache. A graph copy writes every block itself, so it reads through the returned store, which serves the same blocks without filling.

type Ctor 

type Ctor func() Block

Ctor is a block constructor.

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor tracks traversal of a block reference DAG structure with an associated Transaction. Manages interacting with block handles, the transaction cache, the decoder and marshaller, and the transformers.

func (*Cursor) ClearAllRefs 

func (c *Cursor) ClearAllRefs()

ClearAllRefs clears all references. Cursors that were generated by Follow() will be detached.

Note: does not clear sub-blocks from the parent object. Note; does not clear the BlockRef from the parent object.

func (*Cursor) ClearRef 

func (c *Cursor) ClearRef(refID uint32)

ClearRef removes the reference handle to the given ref ID. Noop if FollowRef has not been previously called with refID. Cursors that were generated by Follow() will be detached.

Note: does not clear sub-blocks from the parent object. Note; does not clear the BlockRef from the parent object.

func (*Cursor) CloneBlock 

func (c *Cursor) CloneBlock() (any, error)

CloneBlock tries to clone the contained block.

returns ErrUnexpectedType or ErrNotClonable if the block was not clonable.

func (*Cursor) Detach 

func (c *Cursor) Detach(keepRefs bool) *Cursor

Detach clones the cursor position.

If keepRefs is set, adds the new location as a parent of all previously referenced blocks.

Note: does not copy/clone the Block object.

func (*Cursor) DetachRecursive 

func (c *Cursor) DetachRecursive(detachTx, cloneBlocks, markDirty bool) *Cursor

DetachRecursive clones the cursor position and all referenced positions.

Note: if !cloneBlocks, does not copy/clone the Block objects.

func (*Cursor) DetachTransaction 

func (c *Cursor) DetachTransaction() *Cursor

DetachTransaction creates a new ephemeral transaction rooted at the cursor. The detached cursor reads through the source transaction's staged writes, so it can load blocks the source staged but has not yet published.

func (*Cursor) DiscardDetached 

func (c *Cursor) DiscardDetached() map[uint32]*Cursor

DiscardDetached unlinks an abandoned cursor position from the positions it referenced, so they stop listing it as a parent. The transaction root and positions still referenced by another parent survive. It returns the traversed references of a discarded position without discarding their targets. Positions cleared by an earlier write have none left. The caller must no longer use the discarded cursor position.

func (*Cursor) Fetch 

func (c *Cursor) Fetch(ctx context.Context) ([]byte, bool, error)

Fetch fetches the block data into memory. Fetching is performed using a block lookup. Returns the transformed decoded version of the data. Returns data, dataIsSet, err. Returns nil, false, ErrBlockStoreUnavailable if the block store is unset. Returns nil, false, nil if the reference is empty. Returns nil, false, ErrNotFound if not found (block unavailable).

func (*Cursor) FollowRef 

func (c *Cursor) FollowRef(
	refID uint32,
	blkRef *BlockRef,
) *Cursor

FollowRef follows a block reference, returning a cursor pointing to the next block and enqueuing the block for fetching. Does not wait for the block to be fetched to return. If the reference is empty, will create a new block.

func (*Cursor) FollowSubBlock 

func (c *Cursor) FollowSubBlock(refID uint32) *Cursor

FollowSubBlock follows a sub-block reference, returning a cursor pointing to the same block but at a sub-block inside a field. The block is constructed or retrieved using the BlockWithSubBlocks interface.

Once FollowSubBlock has been called, the field will be overwritten if dirty. If ClearRef is called on the parent then this relation is removed.

Note: there may already be a reference with the same ID, which would be returned. The cursor must have the block decoded or set with SetBlock. The cursor block blk must be a BlockWithSubBlocks. If these conditions are not met, returns nil

func (*Cursor) GetAllRefs 

func (c *Cursor) GetAllRefs(existingOnly bool) (map[uint32]*Cursor, error)

GetAllRefs returns cursors to all references.

If existingOnly, only returns references that have already been traversed. If !existingOnly uses GetSubBlocks and/or GetBlockRefs to list all references. If the position blk is empty, returns an empty map.

func (*Cursor) GetBlock 

func (c *Cursor) GetBlock() (any, bool)

GetBlock returns the current loaded block at the position. May be nil if Fetch or Unmarshal or SetBlock have not been called. Returns isSubBlock.

func (*Cursor) GetBlockStore 

func (c *Cursor) GetBlockStore() (StoreOps, bool)

GetBlockStore returns the block store used for the transaction.

func (*Cursor) GetExistingRef 

func (c *Cursor) GetExistingRef(refID uint32) *Cursor

GetExistingRef checks if the reference has been traversed already. Returns nil if no handle exists for that ref.

func (*Cursor) GetRef 

func (c *Cursor) GetRef() *BlockRef

GetRef returns the current cursor reference.

func (*Cursor) GetTransaction 

func (c *Cursor) GetTransaction() *Transaction

GetTransaction returns the cursor's associated transaction, may be nil.

func (*Cursor) IsDirty 

func (c *Cursor) IsDirty() bool

IsDirty indicates if the position or any sub-positions were changed.

func (*Cursor) IsSubBlock 

func (c *Cursor) IsSubBlock() bool

IsSubBlock indicates if the cursor is currently at a sub-block position.

func (*Cursor) MarkDirty 

func (c *Cursor) MarkDirty()

MarkDirty marks the cursor location dirty, so that it will be re-written.

Note: if cursor is ephemeral (no transaction) this is no-op.

func (*Cursor) Parents 

func (c *Cursor) Parents() []*Cursor

Parents returns new cursors pointing to the parent blocks. Note: the parent list is completely dependent on the order the graph was traversed. Note: returns nil if the cursor is ephemeral (with Detach call).

func (*Cursor) SetAsSubBlock 

func (c *Cursor) SetAsSubBlock(refID uint32, parent *Cursor) error

SetAsSubBlock sets the cursor position as a sub-block of another block.

Clears any existing parent references. Immediately calls ApplySubBlock on the parent block.

May return ErrNotSubBlock or ErrUnexpectedType if the parent is not a block with sub-blocks.

func (*Cursor) SetBlock 

func (c *Cursor) SetBlock(b any, dirty bool)

SetBlock sets a block at the location, and marks the block as dirty. If the location is a Block, b should implement Block interface. If it is a SubBlock, b should implement the SubBlock interface. If dirty is set, sets the block as dirty.

Clears BlockPreWrite.

func (*Cursor) SetBlockStore 

func (c *Cursor) SetBlockStore(store StoreOps)

SetBlockStore sets the store to read from for this cursor and all sub-cursors. If nil, will use the default bucket attached to the block transaction.

func (*Cursor) SetPreWriteHook 

func (c *Cursor) SetPreWriteHook(h func(b any) error)

SetPreWriteHook sets a hook for final transforms to the block.

Note: this should not call any cursor functions that will be locked during the Write process.

Also valid for sub-blocks.

func (*Cursor) SetRef 

func (c *Cursor) SetRef(refID uint32, cursor *Cursor)

SetRef sets a block reference to the handle at the cursor. Adds c to the list of parents for cursor. The cursors must be from the same transaction. A clean stored block moved under c keeps its ref and is not encoded again. Note: this should only be used if refID and cursor are not sub-blocks.

func (*Cursor) SetRefAtCursor 

func (c *Cursor) SetRefAtCursor(ref *BlockRef, clearBlock bool)

SetRefAtCursor sets the reference at the cursor location. If ref is not equal to the existing ref, and clearBlock is set, blk is set to nil.

func (*Cursor) Unmarshal 

func (c *Cursor) Unmarshal(ctx context.Context, ctor func() Block) (Block, error)

Unmarshal fetches and unmarshals the data to a block. If already unmarshaled, returns existing data. If a sub-block, the sub-block must implement Block. If a sub-block, will return the sub-block value or nil. ctor is ignored if the cursor is a sub-block. Returns nil, nil if ctor is nil and the block is nil. Returns nil, nil if the cursor is nil. Returns value from ctor() without calling Unmarshal if empty. Returns nil, block.ErrNotFound if not found.

type DecodedBlockCache added in v0.51.7 

type DecodedBlockCache struct {
	// contains filtered or unexported fields
}

DecodedBlockCache is one owner's scope in a decoded-block pool. Keys carry the scope prefix, so owners never see each other's blocks. Each entry records the invalidation epochs current when it was stored; Lookup treats an entry with an outdated epoch as a miss, and stale entries age out of the pool's queues.

func NewDecodedBlockCache added in v0.51.7 

func NewDecodedBlockCache() *DecodedBlockCache

NewDecodedBlockCache constructs a scope in the process-wide pool.

func NewDecodedBlockCacheWithOptions added in v0.51.7 

func NewDecodedBlockCacheWithOptions(opts DecodedBlockCacheOptions) (*DecodedBlockCache, error)

NewDecodedBlockCacheWithOptions constructs a scope in a private pool with opts. Close releases the pool.

func (*DecodedBlockCache) Close added in v0.51.7 

func (c *DecodedBlockCache) Close()

Close ends the scope. Its entries in a shared pool become stale and age out.

func (*DecodedBlockCache) InvalidateAll added in v0.51.7 

func (c *DecodedBlockCache) InvalidateAll(ctx context.Context)

InvalidateAll makes every cached entry of this scope stale.

func (*DecodedBlockCache) InvalidateRef added in v0.51.7 

func (c *DecodedBlockCache) InvalidateRef(ctx context.Context, ref *BlockRef)

InvalidateRef makes cached entries for ref stale.

func (*DecodedBlockCache) Lookup added in v0.51.7 

func (c *DecodedBlockCache) Lookup(
	ctx context.Context,
	front *decodedBlockFrontCache,
	key decodedBlockCacheKey,
	share bool,
) (Block, bool, error)

Lookup returns the cached decoded block for the key, recording a cache hit or miss. The returned block is a clone safe for the caller to keep, or with share the shared cached instance, which the caller must not mutate.

func (*DecodedBlockCache) MaxCost added in v0.51.7 

func (c *DecodedBlockCache) MaxCost() int64

MaxCost returns the pool's decoded-cache budget.

func (*DecodedBlockCache) Snapshot added in v0.51.7 

func (c *DecodedBlockCache) Snapshot() DecodedBlockCacheSnapshot

Snapshot returns the pool's decoded-cache metrics. A scope in the shared pool reports the metrics of every scope.

func (*DecodedBlockCache) Store added in v0.51.7 

func (c *DecodedBlockCache) Store(
	ctx context.Context,
	front *decodedBlockFrontCache,
	token decodedBlockCacheStoreToken,
	key decodedBlockCacheKey,
	blk Block,
	size int,
	durableGen uint64,
	share bool,
) error

Store caches the decoded block under the key when the store token is still current. blk must decode stored bytes of length size that the caller verified against the block ref, and that a durable store served from a snapshot opened at removal generation durableGen. With share the caller keeps blk unmutated, so the cache retains blk itself instead of a clone.

type DecodedBlockCacheFreshener added in v0.51.7 

type DecodedBlockCacheFreshener interface {
	// EnsureDecodedBlockCacheFresh refreshes the external freshness state
	// before a decoded-cache lookup.
	EnsureDecodedBlockCacheFresh(ctx context.Context) error
}

DecodedBlockCacheFreshener is implemented by stores whose decoded-cache hits depend on external freshness state that ordinary block reads would refresh.

type DecodedBlockCacheOptions added in v0.51.7 

type DecodedBlockCacheOptions struct {
	// MaxCost is the pool's budget in bytes of decoded blocks.
	MaxCost int64
	// Disabled bypasses decoded-object retention while preserving reads.
	Disabled bool
}

DecodedBlockCacheOptions configures a decoded-block cache pool.

func DefaultDecodedBlockCacheOptions added in v0.51.7 

func DefaultDecodedBlockCacheOptions() DecodedBlockCacheOptions

DefaultDecodedBlockCacheOptions returns the production decoded-cache options.

type DecodedBlockCacheSnapshot added in v0.51.7 

type DecodedBlockCacheSnapshot struct {
	// MaxCost is the configured shared-cache budget.
	MaxCost int64
	// RemainingCost is the current unused budget of the pool.
	RemainingCost int64
	// RetainedCost is the current retained cost estimate.
	RetainedCost int64
	// Hits is the number of shared-cache hits.
	Hits uint64
	// Misses is the number of shared-cache misses.
	Misses uint64
	// Stores is the number of stores into the pool.
	Stores uint64
	// Rejections is the number of stores refused for exceeding a shard budget.
	Rejections uint64
	// Evictions is the number of entries evicted.
	Evictions uint64
	// CostAdded is the sum of costs stored.
	CostAdded int64
	// CostEvicted is the sum of costs evicted.
	CostEvicted int64
}

DecodedBlockCacheSnapshot is a point-in-time decoded-cache metrics snapshot.

type DecodedBlockCacheTransformer added in v0.51.7 

type DecodedBlockCacheTransformer interface {
	// DecodedBlockCacheTransformKey returns the transform's cache identity.
	DecodedBlockCacheTransformKey() string
}

DecodedBlockCacheTransformer identifies a transform boundary for decoded-block caching.

type DecodedBlockCacheable added in v0.51.7 

type DecodedBlockCacheable interface {
	// DecodedBlockCacheTypeKey returns the block type's cache identity.
	DecodedBlockCacheTypeKey() string
}

DecodedBlockCacheable identifies a block type for decoded-block caching.

type DecodedBlockHeapSizer 

type DecodedBlockHeapSizer interface {
	// DecodedHeapSize returns the heap bytes the decoded block retains.
	DecodedHeapSize() int
}

DecodedBlockHeapSizer reports the heap bytes a decoded block retains. The decoded-block cache charges this size against its budget. Implement it for block types whose decoded form is much larger than their encoding, such as pages of many small messages.

type DecodedBlockReadShareable 

type DecodedBlockReadShareable interface {
	// SharesDecodedBlockWithReaders marks the type as shareable with readers.
	SharesDecodedBlockWithReaders()
}

DecodedBlockReadShareable marks a block type that reads and cursor traversal never mutate in place, but whose writers change a loaded instance before handing it back to the cursor. Read-only transactions take the cached instance without a clone; other transactions take a private copy.

type DecodedBlockShareable 

type DecodedBlockShareable interface {
	// SharesDecodedBlock marks the type as shareable.
	SharesDecodedBlock()
}

DecodedBlockShareable marks a block type that reads and cursor traversal never mutate in place. Transactions take the cached instance without a clone. Only transaction writes change a decoded instance (ApplyBlockRef, ApplySubBlock), and the cursor position copies a shared instance first. Writers build replacement blocks instead of changing a loaded one.

type DecodedBlockUnmarshaler 

type DecodedBlockUnmarshaler interface {
	// UnmarshalDecodedBlock unmarshals data into the block.
	UnmarshalDecodedBlock(data []byte) error
}

DecodedBlockUnmarshaler is implemented by a block with a faster decoder for blocks read through a cursor. UnmarshalDecodedBlock must leave the block exactly as UnmarshalVT would, and must not retain data.

type DecodedBlockValidator 

type DecodedBlockValidator interface {
	// ValidateDecodedBlock returns an error if the decoded block is invalid.
	ValidateDecodedBlock() error
}

DecodedBlockValidator checks a block once after a cursor decodes it from stored bytes, before the decoded-block cache shares it.

type DeferFlusher added in v0.52.0 

type DeferFlusher interface {
	// BeginDeferFlush opens a deferred GC-flush scope. Supports nesting.
	BeginDeferFlush()
	// EndDeferFlush closes a scope; the outermost close flushes pending edges.
	EndDeferFlush(ctx context.Context) error
}

DeferFlusher batches deferred GC ref-graph flushes across a write scope.

This is an in-process concern owned by GCStoreOps, distinct from the durability surface (StoreOps.Sync). While a scope is open, accumulated ref-graph edge operations are not applied; the outermost EndDeferFlush applies them in one batch. In-process store wrappers forward the scope to their inner store so a GCStoreOps nested behind wrappers still batches. It is never carried over the block RPC wire.

type GraphCopyOptions 

type GraphCopyOptions struct {
	// Reads is the number of source reads kept in flight. Zero selects
	// DefaultGraphCopyReads.
	Reads int
	// Known reports which refs already have a complete copy. The copy skips
	// their subtrees. It receives each block's unvisited children as a batch.
	Known func(ctx context.Context, refs []*BlockRef) ([]bool, error)
	// Complete runs after a block and every block under it are written, in
	// post-order.
	Complete func(ctx context.Context, ref *BlockRef) error
	// Visited observes each block after its read.
	Visited func(ref *BlockRef, data []byte)
	// Forget drops each finished block from the copy's memory, so the copy
	// holds only the blocks between the root and its frontier. A block reached
	// again through another parent is checked with Known again and copied
	// again when Known does not report it.
	Forget bool
	// Missing handles a block src cannot serve now, because it does not hold
	// it or no source served it (IsNotAvailable). Returning nil leaves a hole:
	// the copy continues, and writes the hole's ancestors without completing
	// them. Without Missing, the copy returns the read error.
	Missing func(ctx context.Context, ref *BlockRef) error
}

GraphCopyOptions configures CopyGraph.

type NamedRoot 

type NamedRoot struct {
	// Name is the root name, unique within the store's bucket.
	Name string
	// Ref is the block the name holds, or empty to release the name.
	Ref *BlockRef
}

NamedRoot is a durable named root and the block it holds. An empty Ref releases the name.

type NamedSubBlock 

type NamedSubBlock interface {
	// SubBlock indicates this is a sub-block.
	SubBlock
	// GetName returns the name of the block.
	GetName() string
}

NamedSubBlock is a sub-block with a name attached.

type NopStoreOps 

type NopStoreOps struct{}

NopStoreOps implements StoreOps defaults for tests.

Production wrappers must not embed this type. Test mocks may embed it and override only the methods exercised by the test.

func (NopStoreOps) BeginReadOperation added in v0.51.7 

func (n NopStoreOps) BeginReadOperation(context.Context) (StoreOps, func(), error)

BeginReadOperation returns the no-op store for a read scope.

func (NopStoreOps) GetBlock 

func (NopStoreOps) GetBlock(context.Context, *BlockRef) ([]byte, bool, error)

GetBlock returns a missing block.

func (NopStoreOps) GetBlockExists 

func (NopStoreOps) GetBlockExists(context.Context, *BlockRef) (bool, error)

GetBlockExists returns false.

func (NopStoreOps) GetBlockExistsBatch 

func (n NopStoreOps) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch loops calling GetBlockExists per ref.

func (NopStoreOps) GetHashType 

func (NopStoreOps) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store.

func (NopStoreOps) GetStoredBlock 

func (NopStoreOps) GetStoredBlock(context.Context, *BlockRef) (*StoredBlock, error)

GetStoredBlock returns a missing block.

func (NopStoreOps) GetSupportedFeatures 

func (NopStoreOps) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (NopStoreOps) PutBlock 

func (NopStoreOps) PutBlock(context.Context, []byte, *PutOpts) (*BlockRef, bool, error)

PutBlock returns ErrBlockStoreUnavailable.

func (NopStoreOps) PutBlockBatch 

func (n NopStoreOps) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) ([]bool, error)

PutBlockBatch loops calling PutBlock or RmBlock per entry.

func (NopStoreOps) RmBlock 

func (NopStoreOps) RmBlock(context.Context, *BlockRef) error

RmBlock returns nil.

func (NopStoreOps) StatBlock 

func (NopStoreOps) StatBlock(context.Context, *BlockRef) (*BlockStat, error)

StatBlock returns nil.

func (NopStoreOps) Sync added in v0.52.0 

func (NopStoreOps) Sync(context.Context) (bool, error)

Sync reports no durability fence because the no-op store holds no writes.

type OverlayMode 

type OverlayMode int32

OverlayMode controls the mode for the block store overlay. 

const (
	// UPPER_ONLY uses the upper store only for reads and writes.
	// reads go to the upper store only.
	// writes go to the upper store only.
	// removes go to the upper store only.
	// the lower store is not used.
	OverlayMode_UPPER_ONLY OverlayMode = 0
	// LOWER_ONLY uses the lower store only for reads and writes.
	// reads go to the lower store only.
	// writes go to the lower store only.
	// removes go to the lower store only.
	// the upper store is not used.
	OverlayMode_LOWER_ONLY OverlayMode = 1
	// UPPER_CACHE uses the upper store as a cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to both stores.
	// removes go to both stores.
	// reads from lower are written back to upper.
	OverlayMode_UPPER_CACHE OverlayMode = 2
	// LOWER_CACHE uses the lower store as a cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to both stores.
	// removes go to both stores.
	// reads from upper are written back to lower.
	OverlayMode_LOWER_CACHE OverlayMode = 3
	// UPPER_READ_CACHE uses the upper store as a cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to the lower store only.
	// removes go to the lower store only.
	// reads from lower are not written back to upper.
	OverlayMode_UPPER_READ_CACHE OverlayMode = 4
	// LOWER_READ_CACHE uses the lower store as a cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to the upper store only.
	// removes go to both stores.
	// reads from upper are not written back to lower.
	OverlayMode_LOWER_READ_CACHE OverlayMode = 5
	// UPPER_WRITE_CACHE uses the upper store as a write cache for the lower store.
	// reads go to the upper store first, then the lower store.
	// writes go to the upper store only.
	// removes go to both stores.
	// reads from lower are not written back to upper.
	OverlayMode_UPPER_WRITE_CACHE OverlayMode = 6
	// LOWER_WRITE_CACHE uses the lower store as a write cache for the upper store.
	// reads go to the lower store first, then the upper store.
	// writes go to the lower store only.
	// removes go to both stores.
	// reads from upper are not written back to lower.
	OverlayMode_LOWER_WRITE_CACHE OverlayMode = 7
	// UPPER_READBACK_CACHE treats the lower store as read-only and immutable
	// (e.g. remote packfiles) while caching reads back into the upper store.
	// reads go to the upper store first, then the lower store.
	// reads from lower are written back to upper.
	// writes go to the upper store only.
	// removes go to the upper store only (lower lifecycle is external, e.g.
	// packfile garbage collection / repack).
	OverlayMode_UPPER_READBACK_CACHE OverlayMode = 8
)

func (OverlayMode) Enum 

func (x OverlayMode) Enum() *OverlayMode

func (OverlayMode) MarshalJSON 

func (x OverlayMode) MarshalJSON() ([]byte, error)

MarshalJSON marshals the OverlayMode to JSON.

func (OverlayMode) MarshalProtoJSON 

func (x OverlayMode) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the OverlayMode to JSON.

func (OverlayMode) MarshalProtoText 

func (x OverlayMode) MarshalProtoText() string

func (OverlayMode) MarshalText 

func (x OverlayMode) MarshalText() ([]byte, error)

MarshalText marshals the OverlayMode to text.

func (OverlayMode) String 

func (x OverlayMode) String() string

func (*OverlayMode) UnmarshalJSON 

func (x *OverlayMode) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the OverlayMode from JSON.

func (*OverlayMode) UnmarshalProtoJSON 

func (x *OverlayMode) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the OverlayMode from JSON.

func (*OverlayMode) UnmarshalText 

func (x *OverlayMode) UnmarshalText(b []byte) error

UnmarshalText unmarshals the OverlayMode from text.

type PeerWait 

type PeerWait struct {
	// contains filtered or unexported fields
}

PeerWait records the first block a read reported unavailable instead of waiting for a peer to connect and serve it.

func GetPeerWait 

func GetPeerWait(ctx context.Context) *PeerWait

GetPeerWait returns the record WithoutPeerWait attached to ctx, or nil when reads under ctx may wait for a peer.

func WithoutPeerWait 

func WithoutPeerWait(ctx context.Context) (context.Context, *PeerWait)

WithoutPeerWait returns a context whose block reads report ErrUnavailable instead of waiting for a peer to connect, and the record of the first block such a read skipped.

func (*PeerWait) GetRef 

func (w *PeerWait) GetRef() *BlockRef

GetRef returns the first skipped block, or nil when no read skipped one.

func (*PeerWait) Skip 

func (w *PeerWait) Skip(ref *BlockRef)

Skip records ref as skipped unless an earlier read already recorded a block.

type PendingBatch 

type PendingBatch struct {
	// Entries are the borrowed batch entries.
	Entries []*PutBatchEntry
	// contains filtered or unexported fields
}

PendingBatch is an immutable borrow of buffered entries. Complete must be called exactly once after the publisher is finished, even on admission failure. Entries remain readable through the BufferedStore until successful completion. The zero PendingBatch borrows nothing.

func (*PendingBatch) Complete 

func (b *PendingBatch) Complete(err error)

Complete resolves the borrow exactly once with the publication result.

type Ptr 

type Ptr[T any] interface {
	*T
	Block
}

Ptr constrains PT to a pointer to T that is a Block, so generic readers can construct a zero block as PT(new(T)) without a constructor argument.

type PublicationReceipt 

type PublicationReceipt struct {
	// contains filtered or unexported fields
}

PublicationReceipt distinguishes admission from completion. Err is meaningful only after Done closes. Wait cancellation means the outcome is still unknown; the receipt remains usable for obtaining the eventual durable result.

func NewPublicationReceipt 

func NewPublicationReceipt() *PublicationReceipt

NewPublicationReceipt constructs an unresolved completion owned by a writer.

func NewRetainedPublicationReceipt 

func NewRetainedPublicationReceipt(release func()) *PublicationReceipt

NewRetainedPublicationReceipt retains a prepared root until Release. Its owner supplies an idempotent cleanup that is valid after either outcome.

func (*PublicationReceipt) Done 

func (r *PublicationReceipt) Done() <-chan struct{}

Done returns a channel that closes when the owning publisher resolves the receipt.

func (*PublicationReceipt) Err 

func (r *PublicationReceipt) Err() error

Err returns the durable result. It is meaningful only after Done closes.

func (*PublicationReceipt) Release 

func (r *PublicationReceipt) Release()

Release waits for resolution and releases publication ownership once. A consumer must first install its own reader pin if it will keep using the root.

func (*PublicationReceipt) Resolve 

func (r *PublicationReceipt) Resolve(err error)

Resolve completes a receipt exactly once. Only its owning publisher calls it.

func (*PublicationReceipt) Wait 

func (r *PublicationReceipt) Wait(ctx context.Context) error

Wait returns the durable result, blocking until resolution when unresolved.

type PutBatchEntry 

type PutBatchEntry struct {
	// Ref is the expected content-addressed block reference.
	Ref *BlockRef
	// Data is the encoded block payload.
	Data []byte
	// Refs are outgoing block references recorded with this write.
	Refs []*BlockRef
	// RefsKnown declares Refs complete, as PutOpts.RefsKnown does.
	RefsKnown bool
	// Tombstone marks the block ref as deleted.
	Tombstone bool
}

PutBatchEntry describes one durable block write in a lower-layer batch.

type PutOpts 

type PutOpts struct {

	// HashType is the hash type to use.
	// If unset (0 value) will use default for the store.
	HashType hash.HashType `protobuf:"varint,1,opt,name=hash_type,json=hashType,proto3" json:"hashType,omitempty"`
	// ForceBlockRef forces the block ref to equal the given ref.
	// Can be unset to indicate none (allow any).
	//
	// If the generated BlockRef does not match, the storage operation is aborted
	// and returns block.ErrBlockRefMismatch.
	ForceBlockRef *BlockRef `protobuf:"bytes,2,opt,name=force_block_ref,json=forceBlockRef,proto3" json:"forceBlockRef,omitempty"`
	// Refs are outgoing block references recorded with this write.
	// GC-aware stores consume these refs as part of the put. Stores that do not
	// participate in GC ignore this field.
	Refs []*BlockRef `protobuf:"bytes,3,rep,name=refs,proto3" json:"refs,omitempty"`
	// Sync requests a durability fence before PutBlock returns. When false, the
	// store may enqueue the write and make it immediately readable through its
	// read-through path without waiting for durability. When true, PutBlock
	// enqueues the write, then calls the store Sync barrier so this write and all
	// prior buffered writes for the store are durable before return.
	Sync bool `protobuf:"varint,4,opt,name=sync,proto3" json:"sync,omitempty"`
	// CacheFill marks a copy of a block read from another store. A GC-aware
	// store records its outgoing refs but gives it no owner: a new block is
	// staged as a garbage candidate, and an existing block keeps its edges. The
	// copy then lives only while a retained root reaches it.
	CacheFill bool `protobuf:"varint,5,opt,name=cache_fill,json=cacheFill,proto3" json:"cacheFill,omitempty"`
	// RefsKnown declares refs complete: the writer decoded the block and refs
	// lists every block it references, so empty refs mean a leaf. A GC-aware
	// store then records the child list of a block it already held without one.
	// Unset, refs may be partial, and an existing block keeps its list.
	RefsKnown bool `protobuf:"varint,6,opt,name=refs_known,json=refsKnown,proto3" json:"refsKnown,omitempty"`
	// contains filtered or unexported fields
}

PutOpts are options that can be passed to PutBlock.

func PutOptsWithoutSync added in v0.52.0 

func PutOptsWithoutSync(opts *PutOpts) (*PutOpts, bool)

PutOptsWithoutSync returns opts with Sync cleared, plus whether Sync was set. Wrappers call inner stores with Sync cleared, then apply their own durability boundary after wrapper-local bookkeeping.

func (*PutOpts) CloneMessageVT 

func (m *PutOpts) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*PutOpts) CloneVT 

func (m *PutOpts) CloneVT() *PutOpts

func (*PutOpts) EqualMessageVT 

func (this *PutOpts) EqualMessageVT(thatMsg any) bool

func (*PutOpts) EqualVT 

func (this *PutOpts) EqualVT(that *PutOpts) bool

func (*PutOpts) GetCacheFill 

func (x *PutOpts) GetCacheFill() bool

func (*PutOpts) GetForceBlockRef 

func (x *PutOpts) GetForceBlockRef() *BlockRef

func (*PutOpts) GetHashType 

func (x *PutOpts) GetHashType() hash.HashType

func (*PutOpts) GetRefs 

func (x *PutOpts) GetRefs() []*BlockRef

func (*PutOpts) GetRefsKnown 

func (x *PutOpts) GetRefsKnown() bool

func (*PutOpts) GetSync added in v0.52.0 

func (x *PutOpts) GetSync() bool

func (*PutOpts) MarshalB58 

func (o *PutOpts) MarshalB58() string

MarshalB58 marshals the put opts to a base58 string form.

func (*PutOpts) MarshalJSON 

func (x *PutOpts) MarshalJSON() ([]byte, error)

MarshalJSON marshals the PutOpts to JSON.

func (*PutOpts) MarshalProtoJSON 

func (x *PutOpts) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the PutOpts message to JSON.

func (*PutOpts) MarshalProtoText 

func (x *PutOpts) MarshalProtoText() string

func (*PutOpts) MarshalString 

func (o *PutOpts) MarshalString() string

MarshalString marshals the put opts to b58 string.

func (*PutOpts) MarshalToSizedBufferVT 

func (m *PutOpts) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*PutOpts) MarshalToVT 

func (m *PutOpts) MarshalToVT(dAtA []byte) (int, error)

func (*PutOpts) MarshalVT 

func (m *PutOpts) MarshalVT() (dAtA []byte, err error)

func (*PutOpts) ProtoMessage 

func (*PutOpts) ProtoMessage()

func (*PutOpts) Reset 

func (x *PutOpts) Reset()

func (*PutOpts) SelectHashType 

func (o *PutOpts) SelectHashType(defHashType hash.HashType) hash.HashType

SelectHashType selects the hash type to use for the operation. The given hash type should be the default value to use.

func (*PutOpts) SizeVT 

func (m *PutOpts) SizeVT() (n int)

func (*PutOpts) String 

func (x *PutOpts) String() string

func (*PutOpts) UnmarshalB58 

func (o *PutOpts) UnmarshalB58(ref string) error

UnmarshalB58 unmarshals the put opts from base58 string form.

func (*PutOpts) UnmarshalJSON 

func (x *PutOpts) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the PutOpts from JSON.

func (*PutOpts) UnmarshalProtoJSON 

func (x *PutOpts) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the PutOpts message from JSON.

func (*PutOpts) UnmarshalVT 

func (m *PutOpts) UnmarshalVT(dAtA []byte) error

func (*PutOpts) Validate 

func (o *PutOpts) Validate() error

Validate validates the put opts.

type ReadCounter added in v0.51.7 

type ReadCounter struct {
	// contains filtered or unexported fields
}

ReadCounter records aggregate block reads for one logical operation.

func WithReadCounter added in v0.51.7 

func WithReadCounter(ctx context.Context) (context.Context, *ReadCounter)

WithReadCounter returns a context that records aggregate block reads.

func (*ReadCounter) Snapshot added in v0.51.7 

func (c *ReadCounter) Snapshot() ReadCounterSnapshot

Snapshot returns the current counter values.

type ReadCounterSnapshot added in v0.51.7 

type ReadCounterSnapshot struct {
	// BlockReadCount is the number of block reads that returned without store error.
	BlockReadCount uint64
	// BlockReadBytes is the number of bytes returned by found block reads.
	BlockReadBytes uint64
	// BlockReadMissCount is the number of block reads that returned not found without store error.
	BlockReadMissCount uint64
	// ResourceGetBlockCount is the number of Resource SDK GetBlock calls.
	ResourceGetBlockCount uint64
	// ResourceGetBlockRefCount is the number of non-empty refs passed to Resource SDK GetBlock.
	ResourceGetBlockRefCount uint64
	// ResourceGetBlockBytes is the number of bytes returned by found Resource SDK GetBlock calls.
	ResourceGetBlockBytes uint64
	// ResourceGetBlockMissCount is the number of Resource SDK GetBlock calls that missed.
	ResourceGetBlockMissCount uint64
	// DecodedBlockUnmarshalCount is the number of decoded block unmarshals.
	DecodedBlockUnmarshalCount uint64
	// DecodedBlockUnmarshalBytes is the number of bytes passed to decoded block unmarshalling.
	DecodedBlockUnmarshalBytes uint64
	// DecodedBlockCacheAttemptCount is the number of decoded-block cache lookup attempts.
	DecodedBlockCacheAttemptCount uint64
	// DecodedBlockCacheHitCount is the number of decoded-block cache hits.
	DecodedBlockCacheHitCount uint64
	// DecodedBlockCacheMissCount is the number of decoded-block cache misses.
	DecodedBlockCacheMissCount uint64
	// DecodedBlockCloneCount is the number of clone-on-hit decoded block returns.
	DecodedBlockCloneCount uint64
	// DecodedBlockUncloneableCount is the number of cache candidates that could not be cloned.
	DecodedBlockUncloneableCount uint64
	// DecodedBlockUncacheableCount is the number of decoded-block cache bypasses for missing exact keys.
	DecodedBlockUncacheableCount uint64
	// DecodedBlockStoreAttemptCount is the number of decoded-block cache store submissions.
	DecodedBlockStoreAttemptCount uint64
	// DecodedBlockStoreAcceptedCount is the number of store submissions accepted by the cache buffer.
	DecodedBlockStoreAcceptedCount uint64
	// DecodedBlockStoreRejectedCount is the number of store submissions rejected or dropped immediately.
	DecodedBlockStoreRejectedCount uint64
	// DecodedBlockStoreCost is the cost submitted for decoded-block cache retention.
	DecodedBlockStoreCost uint64
}

ReadCounterSnapshot is a point-in-time copy of block read counters.

type ReadOnlyStore 

type ReadOnlyStore struct {
	StoreOps
}

ReadOnlyStore reads through a store and rejects every write with tx.ErrNotWrite.

OpenStage opens a writable stage on the inner store: a staging scope is the one way to write through read-only storage, and it owns what it writes.

func NewReadOnlyStore 

func NewReadOnlyStore(store StoreOps) *ReadOnlyStore

NewReadOnlyStore wraps store so that it rejects writes.

func (*ReadOnlyStore) BeginReadOperation 

func (s *ReadOnlyStore) BeginReadOperation(ctx context.Context) (StoreOps, func(), error)

BeginReadOperation opens a read scope that also rejects writes.

func (*ReadOnlyStore) HoldCompleteRoots 

func (s *ReadOnlyStore) HoldCompleteRoots(ctx context.Context, refs []*BlockRef) ([]bool, error)

HoldCompleteRoots forwards stage ownership of already complete roots.

func (*ReadOnlyStore) MarkRootsComplete 

func (s *ReadOnlyStore) MarkRootsComplete(context.Context, []*BlockRef) error

MarkRootsComplete rejects the completion proof.

func (*ReadOnlyStore) OpenStage 

func (s *ReadOnlyStore) OpenStage(ctx context.Context) (StoreOps, func(), error)

OpenStage opens a writable stage on the inner store.

func (*ReadOnlyStore) PinRoot 

func (s *ReadOnlyStore) PinRoot(ctx context.Context, ref *BlockRef) (func(), error)

PinRoot forwards a reader pin to the inner store.

func (*ReadOnlyStore) PutBlock 

func (s *ReadOnlyStore) PutBlock(context.Context, []byte, *PutOpts) (*BlockRef, bool, error)

PutBlock rejects the write.

func (*ReadOnlyStore) PutBlockBatch 

func (s *ReadOnlyStore) PutBlockBatch(context.Context, []*PutBatchEntry) ([]bool, error)

PutBlockBatch rejects the writes.

func (*ReadOnlyStore) ReleaseRoots 

func (s *ReadOnlyStore) ReleaseRoots(context.Context, []*BlockRef) error

ReleaseRoots rejects the release of staging ownership.

func (*ReadOnlyStore) RmBlock 

func (s *ReadOnlyStore) RmBlock(context.Context, *BlockRef) error

RmBlock rejects the delete.

func (*ReadOnlyStore) RootsComplete 

func (s *ReadOnlyStore) RootsComplete(ctx context.Context, refs []*BlockRef) ([]bool, error)

RootsComplete checks the inner store's completion proofs.

func (*ReadOnlyStore) SetRetainedRoots 

func (s *ReadOnlyStore) SetRetainedRoots(context.Context, []*PutBatchEntry, []NamedRoot) error

SetRetainedRoots rejects the change of a named root.

func (*ReadOnlyStore) SupportsRootRetention 

func (s *ReadOnlyStore) SupportsRootRetention() bool

SupportsRootRetention reports the inner store's root ownership.

func (*ReadOnlyStore) Sync 

func (s *ReadOnlyStore) Sync(context.Context) (bool, error)

Sync has nothing to fence: the store accepted no writes.

type RootRetainer 

type RootRetainer interface {
	// SupportsRootRetention reports whether the store owns roots. The other
	// methods require it.
	SupportsRootRetention() bool
	// SetRetainedRoots writes the entries and moves every named root as one
	// change. A root may hold a block the entries write.
	SetRetainedRoots(context.Context, []*PutBatchEntry, []NamedRoot) error
	// PinRoot protects a root until the returned release is called.
	PinRoot(context.Context, *BlockRef) (func(), error)
	// OpenStage returns a store whose writes a new stage owns until release.
	OpenStage(context.Context) (StoreOps, func(), error)
	// ReleaseRoots drops the staging ownership of roots held by the owner of
	// this store's writes.
	ReleaseRoots(context.Context, []*BlockRef) error
	// MarkRootsComplete records that the graphs under the roots are stored.
	MarkRootsComplete(context.Context, []*BlockRef) error
	// RootsComplete reports for each root whether the graph under it is
	// recorded stored, reading all of them in one transaction.
	RootsComplete(context.Context, []*BlockRef) ([]bool, error)
}

RootRetainer owns durable named roots, temporary reader pins, stages, and the staging ownership of roots written outside them. Replacing a named root releases its predecessor; immutable descendants remain protected by their own edges, other named roots, and reader pins. Unsupported stores keep their existing retention policy.

type StoreFeature 

type StoreFeature int32

StoreFeature is a bitmask of native block store capabilities. Each non-zero enum value is a single bit and values are combined with bitwise OR. The values are intentionally not sequential. 

const (
	// STORE_FEATURE_UNKNOWN indicates no native capabilities are reported.
	StoreFeature_STORE_FEATURE_UNKNOWN StoreFeature = 0
	// STORE_FEATURE_NATIVE_BATCH_PUT means PutBlockBatch is implemented natively.
	StoreFeature_STORE_FEATURE_NATIVE_BATCH_PUT StoreFeature = 1
	// STORE_FEATURE_NATIVE_BATCH_EXISTS means GetBlockExistsBatch is implemented natively.
	StoreFeature_STORE_FEATURE_NATIVE_BATCH_EXISTS StoreFeature = 2
	// STORE_FEATURE_SELF_BUFFERED means the store has its own read-through
	// pending buffer, so the world block engine must not wrap it in a
	// BufferedStore. Volume.Sync remains the durability barrier for
	// volume-backed stores.
	StoreFeature_STORE_FEATURE_SELF_BUFFERED StoreFeature = 32
)

func (StoreFeature) Enum 

func (x StoreFeature) Enum() *StoreFeature

func (StoreFeature) Has 

func (s StoreFeature) Has(feat StoreFeature) bool

Has reports whether this feature bitset contains every requested feature.

func (StoreFeature) MarshalJSON 

func (x StoreFeature) MarshalJSON() ([]byte, error)

MarshalJSON marshals the StoreFeature to JSON.

func (StoreFeature) MarshalProtoJSON 

func (x StoreFeature) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the StoreFeature to JSON.

func (StoreFeature) MarshalProtoText 

func (x StoreFeature) MarshalProtoText() string

func (StoreFeature) MarshalText 

func (x StoreFeature) MarshalText() ([]byte, error)

MarshalText marshals the StoreFeature to text.

func (StoreFeature) String 

func (x StoreFeature) String() string

func (*StoreFeature) UnmarshalJSON 

func (x *StoreFeature) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the StoreFeature from JSON.

func (*StoreFeature) UnmarshalProtoJSON 

func (x *StoreFeature) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the StoreFeature from JSON.

func (*StoreFeature) UnmarshalText 

func (x *StoreFeature) UnmarshalText(b []byte) error

UnmarshalText unmarshals the StoreFeature from text.

type StoreHolder 

type StoreHolder interface {
	// GetHeldStore returns the held store, or nil before it is built.
	GetHeldStore() StoreOps
}

StoreHolder is implemented by a controller that holds a block store it built itself, outside the bus.

type StoreOps 

type StoreOps interface {
	// GetHashType returns the preferred hash type for the store.
	// This should return as fast as possible (called frequently).
	// If 0 is returned, uses a default defined by Hydra.
	GetHashType() hash.HashType
	// GetSupportedFeatures returns the native feature bitmask for the store.
	// This should return as fast as possible (called frequently) and remain
	// static for the lifetime of the store.
	GetSupportedFeatures() StoreFeature
	// BeginReadOperation opens a read scope for a bounded logical operation.
	// The returned store must be released after use. Implementations without
	// native read-operation state may return themselves with a no-op release.
	BeginReadOperation(ctx context.Context) (StoreOps, func(), error)
	// PutBlock puts a block into the store. By default this may enqueue a
	// buffered, read-through write without waiting for durability. Set
	// PutOpts.Sync to fence this write and all prior buffered writes before return.
	// The ref should not be modified after return.
	// The second return value can optionally indicate if the block already existed.
	// If the hash type is unset, use the type from GetHashType().
	PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)
	// PutBlockBatch writes a batch of block operations.
	// Implementations without native batching fall back internally.
	// Returns one result per entry, in order: whether the block was already
	// stored before the write, like the second return value of PutBlock.
	// A tombstone entry reports false.
	PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) ([]bool, error)
	// GetBlock gets a block with the given reference.
	// The ref should not be modified or retained by GetBlock.
	// Returns data, found, error.
	// Returns nil, false, nil if not found.
	// Note: the block may not be in the specified bucket.
	GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)
	// GetStoredBlock gets a block and its outgoing block references in one
	// read. RefsKnown is unset when the store holds the bytes without their
	// references; with RefsKnown set, empty Refs means the block is a leaf.
	// Returns nil, nil if not found.
	GetStoredBlock(ctx context.Context, ref *BlockRef) (*StoredBlock, error)
	// GetBlockExists checks if a block exists with a cid reference.
	// The ref should not be modified or retained by GetBlock.
	// Note: the block may not be in the specified bucket.
	GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)
	// GetBlockExistsBatch checks whether each block reference exists.
	// Implementations without native batching fall back internally.
	GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)
	// RmBlock deletes a block from the bucket.
	// Does not return an error if the block was not present.
	// In some cases, will return before confirming delete.
	RmBlock(ctx context.Context, ref *BlockRef) error
	// StatBlock returns metadata about a block without reading its data.
	// Returns nil, nil if the block does not exist.
	StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)
	// Sync is the sole durability barrier: it drains any buffered writes and
	// blocks until every write issued before the call is durable, like POSIX
	// sync(2). The boolean reports whether a fence was applied: true when the
	// barrier completed or the store is always-durable, false when the store
	// provides no durability fence. A nil error with false means "no fence
	// available", not a failure; a non-nil error means a real sync failure.
	Sync(ctx context.Context) (bool, error)
}

StoreOps can read/write blocks.

func NewStoreRW 

func NewStoreRW(readHandle, writeHandle StoreOps) StoreOps

NewStoreRW constructs a new Store handle using a read handle and an optional write handle. If the write handle is not nil, the write (put and delete) calls will go to it. Otherwise, all calls are sent to the read handle.

func OpenStage 

func OpenStage(ctx context.Context, store StoreOps) (StoreOps, func(), error)

OpenStage returns a store whose writes the stage owns, and the stage's release. The stage holds each block until a parent written outside the stage references it; after the release, the sweep collects every block no other owner holds. Hold the stage until the transaction that references the stage's roots returns. A store without root ownership returns itself and a no-op release.

type StoreOverlay 

type StoreOverlay struct {
	// contains filtered or unexported fields
}

StoreOverlay layers an upper block store over a lower store.

ctx is used for writeback requests

func NewOverlay 

func NewOverlay(
	ctx context.Context,
	le *logrus.Entry,
	lower,
	upper StoreOps,
	mode OverlayMode,
	writebackTimeout time.Duration,
	writebackPutOpts *PutOpts,
) *StoreOverlay

NewOverlay constructs a new overlay store.

ctx is used for writeback requests

func (*StoreOverlay) BeginDeferFlush 

func (o *StoreOverlay) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to upper and lower stores.

func (*StoreOverlay) BeginReadOperation added in v0.51.7 

func (o *StoreOverlay) BeginReadOperation(ctx context.Context) (StoreOps, func(), error)

BeginReadOperation opens read scopes for the overlay stores.

func (*StoreOverlay) EndDeferFlush 

func (o *StoreOverlay) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to upper and lower stores.

func (*StoreOverlay) GetBlock 

func (o *StoreOverlay) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block with the given reference. The ref should not be modified or retained by GetBlock. Returns data, found, error. Returns nil, false, nil if not found. Note: the block may not be in the specified bucket.

func (*StoreOverlay) GetBlockExists 

func (o *StoreOverlay) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreOverlay) GetBlockExistsBatch 

func (o *StoreOverlay) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch checks block existence using the same read policy as GetBlockExists.

func (*StoreOverlay) GetHashType 

func (o *StoreOverlay) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store. This should return as fast as possible (called frequently). If 0 is returned, uses a default defined by Hydra.

func (*StoreOverlay) GetStoreParts 

func (o *StoreOverlay) GetStoreParts() []StorePart

GetStoreParts returns the lower and upper stores.

func (*StoreOverlay) GetStoredBlock 

func (o *StoreOverlay) GetStoredBlock(ctx context.Context, ref *BlockRef) (*StoredBlock, error)

GetStoredBlock gets a block and its references using the read policy of GetBlock. Fills carry the references read from the answering store.

func (*StoreOverlay) GetSupportedFeatures 

func (o *StoreOverlay) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the overlay.

func (*StoreOverlay) PutBlock 

func (o *StoreOverlay) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock puts a block into the store. The ref should not be modified after return. The second return value can optionally indicate if the block already existed. If the hash type is unset, use the type from GetHashType().

func (*StoreOverlay) PutBlockBatch 

func (o *StoreOverlay) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) ([]bool, error)

PutBlockBatch writes a batch of blocks using the same target-store policy as PutBlock. A block written to both stores is reported as existing only when both stores held it.

func (*StoreOverlay) RmBlock 

func (o *StoreOverlay) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block from the bucket. Does not return an error if the block was not present. In some cases, will return before confirming delete.

func (*StoreOverlay) StatBlock 

func (o *StoreOverlay) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data. Returns nil, nil if the block does not exist.

func (*StoreOverlay) Sync added in v0.52.0 

func (o *StoreOverlay) Sync(ctx context.Context) (bool, error)

Sync fences both overlay stores; it reports fenced only when both did.

type StorePart 

type StorePart struct {
	// Label names the child's role in its parent, such as "lower" or "upper".
	Label string
	// Store is the child store.
	Store StoreOps
}

StorePart is one labeled child of a block store built from other stores.

type StoreParts 

type StoreParts interface {
	// GetStoreParts returns the child stores.
	GetStoreParts() []StorePart
}

StoreParts is implemented by a block store built from other stores, so an inspector can open it into its children.

type StoreRW 

type StoreRW struct {
	// contains filtered or unexported fields
}

StoreRW combines a read and write store together.

func (*StoreRW) AtomicPublicationVolumeID 

func (b *StoreRW) AtomicPublicationVolumeID() string

AtomicPublicationVolumeID returns the write domain's shared publication namespace.

func (*StoreRW) BeginDeferFlush 

func (b *StoreRW) BeginDeferFlush()

BeginDeferFlush forwards the GC defer-flush scope to the write handle.

func (*StoreRW) BeginReadOperation added in v0.51.7 

func (b *StoreRW) BeginReadOperation(ctx context.Context) (StoreOps, func(), error)

BeginReadOperation opens a read scope on the read handle.

func (*StoreRW) EndDeferFlush 

func (b *StoreRW) EndDeferFlush(ctx context.Context) error

EndDeferFlush forwards closing the GC defer-flush scope to the write handle.

func (*StoreRW) EnsureDecodedBlockCacheFresh added in v0.51.7 

func (b *StoreRW) EnsureDecodedBlockCacheFresh(ctx context.Context) error

EnsureDecodedBlockCacheFresh forwards decoded-cache freshness to the read handle.

func (*StoreRW) GetBlock 

func (b *StoreRW) GetBlock(ctx context.Context, ref *BlockRef) ([]byte, bool, error)

GetBlock gets a block with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreRW) GetBlockExists 

func (b *StoreRW) GetBlockExists(ctx context.Context, ref *BlockRef) (bool, error)

GetBlockExists checks if a block exists with a cid reference. The ref should not be modified or retained by GetBlock. Note: the block may not be in the specified bucket.

func (*StoreRW) GetBlockExistsBatch 

func (b *StoreRW) GetBlockExistsBatch(ctx context.Context, refs []*BlockRef) ([]bool, error)

GetBlockExistsBatch forwards batched existence probes to the read handle when supported.

func (*StoreRW) GetHashType 

func (b *StoreRW) GetHashType() hash.HashType

GetHashType returns the preferred hash type for the store. This should return as fast as possible (called frequently). If 0 is returned, uses a default defined by Hydra.

func (*StoreRW) GetStoredBlock 

func (b *StoreRW) GetStoredBlock(ctx context.Context, ref *BlockRef) (*StoredBlock, error)

GetStoredBlock gets a block and its references from the read handle.

func (*StoreRW) GetSupportedFeatures 

func (b *StoreRW) GetSupportedFeatures() StoreFeature

GetSupportedFeatures returns the native feature bitmask for the store.

func (*StoreRW) HoldCompleteRoots 

func (b *StoreRW) HoldCompleteRoots(ctx context.Context, refs []*BlockRef) ([]bool, error)

HoldCompleteRoots forwards stage ownership to the write handle, never the read handle.

func (*StoreRW) MarkRootsComplete 

func (b *StoreRW) MarkRootsComplete(ctx context.Context, roots []*BlockRef) error

MarkRootsComplete forwards the durable World proof.

func (*StoreRW) OpenStage 

func (b *StoreRW) OpenStage(ctx context.Context) (StoreOps, func(), error)

OpenStage opens a stage on the write handle and keeps reads on the read handle.

func (*StoreRW) PinRoot 

func (b *StoreRW) PinRoot(ctx context.Context, ref *BlockRef) (func(), error)

PinRoot forwards a reader pin to the underlying store.

func (*StoreRW) PublishAtomic 

func (b *StoreRW) PublishAtomic(ctx context.Context, p *AtomicPublication) error

PublishAtomic forwards the uncancelled durability wait to the write handle.

func (*StoreRW) PutBlock 

func (b *StoreRW) PutBlock(ctx context.Context, data []byte, opts *PutOpts) (*BlockRef, bool, error)

PutBlock puts a block into the store. The ref should not be modified after return.

func (*StoreRW) PutBlockBatch 

func (b *StoreRW) PutBlockBatch(ctx context.Context, entries []*PutBatchEntry) ([]bool, error)

PutBlockBatch forwards to the write handle if it supports batched writes.

func (*StoreRW) ReleaseRoots 

func (b *StoreRW) ReleaseRoots(ctx context.Context, refs []*BlockRef) error

ReleaseRoots forwards staging release to the underlying store.

func (*StoreRW) RmBlock 

func (b *StoreRW) RmBlock(ctx context.Context, ref *BlockRef) error

RmBlock deletes a block from the bucket. Does not return an error if the block was not present. In some cases, will return before confirming delete.

func (*StoreRW) RootsComplete 

func (b *StoreRW) RootsComplete(ctx context.Context, refs []*BlockRef) ([]bool, error)

RootsComplete checks the underlying World proofs.

func (*StoreRW) SetRetainedRoots 

func (b *StoreRW) SetRetainedRoots(ctx context.Context, entries []*PutBatchEntry, roots []NamedRoot) error

SetRetainedRoots forwards durable root ownership to the underlying store.

func (*StoreRW) StatBlock 

func (b *StoreRW) StatBlock(ctx context.Context, ref *BlockRef) (*BlockStat, error)

StatBlock returns metadata about a block without reading its data. Returns nil, nil if the block does not exist.

func (*StoreRW) SubmitAtomic 

func (b *StoreRW) SubmitAtomic(ctx context.Context, p *AtomicPublication) (*PublicationReceipt, error)

SubmitAtomic forwards admission to the write handle's publisher.

func (*StoreRW) SupportsAtomicPublication 

func (b *StoreRW) SupportsAtomicPublication() bool

SupportsAtomicPublication follows the write domain, never a cross-volume lookup.

func (*StoreRW) SupportsRootRetention 

func (b *StoreRW) SupportsRootRetention() bool

SupportsRootRetention reports the underlying ownership capability.

func (*StoreRW) Sync added in v0.52.0 

func (b *StoreRW) Sync(ctx context.Context) (bool, error)

Sync forwards the durability barrier to the write handle.

type StoredBlock 

type StoredBlock struct {
	// Data is the stored block data.
	Data []byte
	// Refs are the block's outgoing refs when RefsKnown is set.
	Refs []*BlockRef
	// RefsKnown is false when the source held the bytes without their refs,
	// so a leaf cannot be told from a block whose edges were lost.
	RefsKnown bool
}

StoredBlock is a block's stored bytes with its outgoing refs.

func DecodeBlockObject 

func DecodeBlockObject(value []byte) (*StoredBlock, error)

DecodeBlockObject decodes an encoded BlockObject. The refs are known: an empty list is a leaf. The caller verifies the bytes against their ref.

func GetBlockWithoutRefs 

func GetBlockWithoutRefs(ctx context.Context, store StoreOps, ref *BlockRef) (*StoredBlock, error)

GetBlockWithoutRefs implements GetStoredBlock for a store that keeps block bytes without their refs. Returns nil when the block is not found.

func ReadStoredBlock 

func ReadStoredBlock(ctx context.Context, store StoreOps, ref *BlockRef, withRefs bool) (*StoredBlock, error)

ReadStoredBlock reads a block from store, with its refs when withRefs is set. Returns nil when the block is not found.

func (*StoredBlock) GetData 

func (b *StoredBlock) GetData() []byte

GetData returns the block data, or nil for a nil block.

func (*StoredBlock) GetRefs 

func (b *StoredBlock) GetRefs() []*BlockRef

GetRefs returns the block's outgoing refs, or nil for a nil block.

func (*StoredBlock) GetRefsKnown 

func (b *StoredBlock) GetRefsKnown() bool

GetRefsKnown reports whether the refs are known, false for a nil block.

func (*StoredBlock) PutOpts 

func (b *StoredBlock) PutOpts(ref *BlockRef) *PutOpts

PutOpts returns options that store the block under ref with its refs. Refs are empty when they are unknown, so the caller decides whether a block without known refs may be written.

type SubBlock 

type SubBlock interface {
	BlockWithAliasIdentity
	// IsNil checks if the object is nil.
	IsNil() bool
}

SubBlock is a object contained inside a Block. May optionally implement Block or other Block interfaces.

type SubBlockCtor 

type SubBlockCtor func(create bool) SubBlock

SubBlockCtor constructs a sub-block. If create == false, returns nil if the field is not set.

func NewSubBlockCtor 

func NewSubBlockCtor[T ComparableSubBlock](r *T, ctor func() T) SubBlockCtor

NewSubBlockCtor constructs a new SubBlock constructor.

Returns nil if r is nil. Usage: block.NewSubBlockCtor(r, func() *ChangeLogLL { return &ChangeLogLL{} })

type Transaction 

type Transaction struct {
	// contains filtered or unexported fields
}

Transaction tracks refs traversed between blocks, batching writes and propagating changes through the merkle graph.

The decoded object form of the block can be stored / attached to a block handle. Changes are written to storage with a topological reference sort.

Empty blocks are not written to storage: they are instead represented with a nil BlockRef. SetBlockRef should handle nil BlockRef objects correctly.

func (*Transaction) DiscardStagedWrites added in v0.57.2 

func (t *Transaction) DiscardStagedWrites()

DiscardStagedWrites drops blocks that have not been published.

func (*Transaction) GetPutOpts 

func (t *Transaction) GetPutOpts() *PutOpts

GetPutOpts returns the transaction's put options.

func (*Transaction) GetStagedStore added in v0.57.2 

func (t *Transaction) GetStagedStore() *BufferedStore

GetStagedStore returns the transaction staging store when one exists.

func (*Transaction) GetStoreOps 

func (t *Transaction) GetStoreOps() StoreOps

GetStoreOps returns the transaction's store operations.

func (*Transaction) GetTransformer 

func (t *Transaction) GetTransformer() Transformer

GetTransformer returns the transaction's block transformer.

func (*Transaction) SetBufferedStoreSettings 

func (t *Transaction) SetBufferedStoreSettings(s *BufferedStoreSettings)

SetBufferedStoreSettings overrides the BufferedStore settings used to wrap the write store inside WriteAtRoot. Pass nil to reset to defaults. This must be called before Write/WriteAtRoot begins committing for the override to take effect on that commit.

func (*Transaction) SetDecodedBlockCache added in v0.51.7 

func (t *Transaction) SetDecodedBlockCache(cache *DecodedBlockCache)

SetDecodedBlockCache sets the lifecycle-owned decoded-block cache borrowed by this transaction.

func (*Transaction) SetReadOnly 

func (t *Transaction) SetReadOnly()

SetReadOnly marks the transaction read-only. Write and WriteAtRoot then fail with tx.ErrNotWrite, and sub-blocks of shared decoded blocks are followed without a private copy. Detached transactions that share its blocks inherit the mark.

func (*Transaction) SetRoot 

func (t *Transaction) SetRoot(cursor *Cursor) error

SetRoot sets the root of the transaction to a different position. Clears all parent blocks from the new root.

func (*Transaction) SetStoreOps 

func (t *Transaction) SetStoreOps(store StoreOps)

SetStoreOps replaces the transaction's store implementation. Used to swap in GCStoreOps after the RefGraph is available.

func (*Transaction) SetWriteBuffer added in v0.57.2 

func (t *Transaction) SetWriteBuffer(buffer *BufferedStore)

SetWriteBuffer borrows a buffer that WriteAtRoot must not drain.

func (*Transaction) StageWrites added in v0.57.2 

func (t *Transaction) StageWrites(ctx context.Context, inner StoreOps) *BufferedStore

StageWrites returns the transaction staging store, creating it on first use. SetStoreOps must not be called after staging begins.

func (*Transaction) Write 

func (t *Transaction) Write(ctx context.Context, clearTree bool) (
	res *BlockRef,
	rcursor *Cursor,
	rerr error,
)

Write writes the dirty blocks to the store, propagating reference changes up the tree. Clears the blocks cache if clearTree is set, otherwise the updated references are written to the cursor tree. The final block in the event list will be the new root. The new root cursor is returned. Blocks that are not referenced by the root directly or indirectly are "cut" and removed.

Note: after Write with clearTree, use the new returned rcursor only.

func (*Transaction) WriteAtRoot 

func (t *Transaction) WriteAtRoot(ctx context.Context, clearTree bool, subRoot *Cursor) (
	res *BlockRef,
	rcursor *Cursor,
	rerr error,
)

WriteAtRoot writes dirty blocks to the store starting from a sub-tree root. If subRoot is nil, writes from the transaction root (same as Write). If subRoot is non-nil, writes only the sub-tree rooted at that cursor. Blocks outside the sub-tree are not touched. After writing, the sub-tree nodes are non-dirty with refs set, and their block data is freed if clearTree is set. The parent transaction's Write() will skip these nodes.

type Transformer 

type Transformer interface {
	// EncodeBlock encodes the block according to the config.
	// May reuse the same byte slice if possible.
	EncodeBlock([]byte) ([]byte, error)
	// DecodeBlock decodes the block according to the config.
	// May reuse the same byte slice if possible.
	DecodeBlock([]byte) ([]byte, error)
}

Transformer encodes and decodes blocks for storage. Its functions must be concurrency-safe.

type WriteCounter 

type WriteCounter struct {
	// contains filtered or unexported fields
}

WriteCounter records aggregate block writes for one logical operation.

func WithWriteCounter 

func WithWriteCounter(ctx context.Context) (context.Context, *WriteCounter)

WithWriteCounter returns a context that records aggregate block writes.

func (*WriteCounter) Snapshot 

func (c *WriteCounter) Snapshot() WriteCounterSnapshot

Snapshot returns the current counter values.

type WriteCounterSnapshot 

type WriteCounterSnapshot struct {
	// BlockWriteCount is the number of encoded blocks submitted for storage.
	BlockWriteCount uint64
	// BlockWriteBytes is the number of encoded bytes submitted for storage.
	BlockWriteBytes uint64
}

WriteCounterSnapshot is a point-in-time copy of a WriteCounter.

Source Files

View all Source files

Directories

Path Synopsis
gc
Package block_gc implements garbage collection for Hydra block stores.
 Package block_gc implements garbage collection for Hydra block stores.
rpc
rpc/client
rpc/server
rpc
client
server
bucket
controller
http
http/lookup
http/server
http/test
inmem
kvfile
kvfile/http
kvtx
overlay
redis
rpc
rpc/lookup
rpc/server
rpc/server/bucket
s3
s3/lookup
test
vlogger
writeback
Package block_store_writeback keeps every block write in a local store and uploads it to a remote store in the background.
 Package block_store_writeback keeps every block write in a local store and uploads it to a remote store in the background.
all
blockenc
chksum
e2e
gzip
lz4
s2
snappy

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.