blob

package
v0.55.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 10, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefChunkingMinSize is the default chunk min size.
	DefChunkingMinSize = 2048 * 125 // 256000 bytes
	// DefChunkingTargetSize is the default chunk target size.
	// Only used by the JC chunker.
	DefChunkingTargetSize = 512000 // 512000 bytes
	// DefChunkingMaxSize is the default chunk max size.
	DefChunkingMaxSize = 4096 * (64 * 3) // 786432 bytes
	// DefRawHighWaterMark is the default high water mark for a raw blob.
	// define this to be the max size of a single chunk
	DefRawHighWaterMark = DefChunkingMaxSize
)

Variables

View Source 
var (
	BlobType_name = map[int32]string{
		0: "BlobType_RAW",
		1: "BlobType_CHUNKED",
	}
	BlobType_value = map[string]int32{
		"BlobType_RAW":     0,
		"BlobType_CHUNKED": 1,
	}
)

Enum value maps for BlobType.

View Source 
var (
	ChunkerType_name = map[int32]string{
		0: "ChunkerType_DEFAULT",
		1: "ChunkerType_RABIN",
		2: "ChunkerType_JC",
	}
	ChunkerType_value = map[string]int32{
		"ChunkerType_DEFAULT": 0,
		"ChunkerType_RABIN":   1,
		"ChunkerType_JC":      2,
	}
)

Enum value maps for ChunkerType.

View Source 
var (
	// ErrRawBlobSizeMismatch is returned if the raw blob size field does not match the data len.
	ErrRawBlobSizeMismatch = errors.New("raw blob size must match data len")
	// ErrEmptyChunk is returned if a empty chunk was found (invalid).
	ErrEmptyChunk = errors.New("empty chunk is invalid")
	// ErrOutOfSequenceChunk is returned if a chunk was out-of-sequence (invalid size or start).
	ErrOutOfSequenceChunk = errors.New("invalid chunk sequence")
	// ErrUnknownBlobType is returned for a blob type that is not recognized.
	ErrUnknownBlobType = errors.New("unknown blob type")
	// ErrUnknownChunkerType is returned for a chunker type that is not recognized.
	ErrUnknownChunkerType = errors.New("unknown chunker type")
)

Functions

func CompareBlobs 

func CompareBlobs(ctx context.Context, bcs1, bcs2 *block.Cursor) (bool, error)

CompareBlobs compares the contents of two blobs for equality.

func FetchToBuffer 

func FetchToBuffer(ctx context.Context, bcs *block.Cursor, buf *bytes.Buffer) error

FetchToBuffer fetches a full blob to a buffer. Note: the block cursor context is also used.

func FetchToBytes 

func FetchToBytes(ctx context.Context, bcs *block.Cursor) ([]byte, error)

FetchToBytes fetches to a bytes slice.

func NewBlobBlock 

func NewBlobBlock() block.Block

NewBlobBlock builds a new blob root block.

func NewBlobSubBlockCtor 

func NewBlobSubBlockCtor(r **Blob) block.SubBlockCtor

NewBlobSubBlockCtor returns the sub-block constructor.

func NewChunkBlock 

func NewChunkBlock() block.Block

NewChunkBlock builds a new repo ref block.

func NewChunkIndexBlock 

func NewChunkIndexBlock() block.Block

NewChunkIndexBlock builds a new repo ref block.

func NewChunkSet 

func NewChunkSet(v *[]*Chunk, bcs *block.Cursor) *sbset.SubBlockSet

NewChunkSet builds a new chunk set container.

bcs should be located at the chunk set sub-block.

func ReadFromChunks 

func ReadFromChunks(
	ctx context.Context,
	chunkSet *sbset.SubBlockSet,
	buf []byte,
	start, chunkIdx int,
) (n int, outChunkIdx int, err error)

ReadFromChunks reads up to len(buf) data from the chunks, starting at byte index start. Attempts to start reading from chunkIdx, but will search for the chunk containing start. If the chunk at chunkIdx does not contain start, will binary-search for the appropriate chunk. The value of outChunkIdx should be saved and passed again when stepping through the chunks sequentially. Returns io.EOF if start is past the last chunk.

func WithMetricsRecorder added in v0.52.0 

func WithMetricsRecorder(ctx context.Context, recorder MetricsRecorder) context.Context

WithMetricsRecorder attaches a blob metrics recorder to ctx.

Types

type Blob 

type Blob struct {

	// BlobType is the blob type.
	BlobType BlobType `protobuf:"varint,1,opt,name=blob_type,json=blobType,proto3" json:"blobType,omitempty"`
	// TotalSize is the total size of the blob.
	TotalSize uint64 `protobuf:"varint,2,opt,name=total_size,json=totalSize,proto3" json:"totalSize,omitempty"`
	// RawData contains in-line data for the raw blob type.
	// index=0 size=total_size if BlobType_RAW
	RawData []byte `protobuf:"bytes,3,opt,name=raw_data,json=rawData,proto3" json:"rawData,omitempty"`
	// ChunkIndex contains the information for CHUNKED type.
	ChunkIndex *ChunkIndex `protobuf:"bytes,4,opt,name=chunk_index,json=chunkIndex,proto3" json:"chunkIndex,omitempty"`
	// contains filtered or unexported fields
}

Blob defines multiple patterns for storing large blobs of data. All behaviors are deterministic with change validation routines. The Blob object contains metadata and might contain links to sub-objects.

func BuildBlob 

func BuildBlob(
	ctx context.Context,
	dataLen int64,
	rdr io.Reader,
	bcs *block.Cursor,
	opts *BuildBlobOpts,
) (*Blob, error)

BuildBlob constructs a blob chunking / sharding it. Blocks will be written to the block transaction. The new root Blob block will become the root of bcs. Constructs a blob with a known size.

func BuildBlobWithBytes 

func BuildBlobWithBytes(ctx context.Context, data []byte, bcs *block.Cursor) (*Blob, error)

BuildBlobWithBytes is a shortcut to build a blob from a byte slice.

func BuildBlobWithReader 

func BuildBlobWithReader(
	ctx context.Context,
	rdr io.Reader,
	bcs *block.Cursor,
	opts *BuildBlobOpts,
) (*Blob, error)

BuildBlobWithReader constructs a blob chunking / sharding it. Blocks will be written to the block transaction. The new root Blob block will become the root of bcs. Constructs a blob with an unknown size. If you know the size ahead of time, use BuildBlob.

func NewRawBlob 

func NewRawBlob(data []byte) *Blob

NewRawBlob constructs a new Raw blob. Data is not copied.

func UnmarshalBlob 

func UnmarshalBlob(ctx context.Context, bcs *block.Cursor) (*Blob, error)

UnmarshalBlob unmarshals the Blob block. Returns nil, nil if empty

func (*Blob) AppendData 

func (b *Blob) AppendData(
	ctx context.Context,
	dataLen int64,
	rdr io.Reader,
	bcs *block.Cursor,
	opts *BuildBlobOpts,
) error

AppendData appends data to an existing blob.

func (*Blob) ApplySubBlock 

func (b *Blob) ApplySubBlock(id uint32, next block.SubBlock) error

ApplySubBlock applies a sub-block change with a field id.

func (*Blob) BlockAliasIdentity added in v0.52.0 

func (b *Blob) BlockAliasIdentity() *block.AliasIdentityToken

BlockAliasIdentity returns the in-memory alias token for Blob.

func (*Blob) CloneMessageVT 

func (m *Blob) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*Blob) CloneVT 

func (m *Blob) CloneVT() *Blob

func (*Blob) ComputeStorageSize 

func (b *Blob) ComputeStorageSize(
	ctx context.Context,
	bcs *block.Cursor,
) (uint64, uint64, error)

ComputeStorageSize computes the total size of all blocks making up the Blob.

note: not accurate until the btx has been committed. returns:

  • storageSize: actual size of blocks on disk
  • totalSize: size of blocks on disk ignoring duplicates (for dedupe comparison)
  • err: any error

func (*Blob) EqualMessageVT 

func (this *Blob) EqualMessageVT(thatMsg any) bool

func (*Blob) EqualVT 

func (this *Blob) EqualVT(that *Blob) bool

func (*Blob) GetBlobType 

func (x *Blob) GetBlobType() BlobType

func (*Blob) GetChunkIndex 

func (x *Blob) GetChunkIndex() *ChunkIndex

func (*Blob) GetRawData 

func (x *Blob) GetRawData() []byte

func (*Blob) GetSubBlockCtor 

func (b *Blob) GetSubBlockCtor(id uint32) block.SubBlockCtor

GetSubBlockCtor returns a function which creates or returns the existing sub-block at reference id. Can return nil to indicate invalid reference id.

func (*Blob) GetSubBlocks 

func (b *Blob) GetSubBlocks() map[uint32]block.SubBlock

GetSubBlocks returns all constructed sub-blocks by ID. May return nil, and values may also be nil.

func (*Blob) GetTotalSize 

func (x *Blob) GetTotalSize() uint64

func (*Blob) IsEmpty 

func (b *Blob) IsEmpty() bool

IsEmpty checks if the blob total size is zero.

func (*Blob) IsNil 

func (b *Blob) IsNil() bool

IsNil checks if the object is nil.

func (*Blob) MarshalBlock 

func (b *Blob) MarshalBlock() ([]byte, error)

MarshalBlock marshals the block to binary. This is the initial step of marshaling, before transformations.

func (*Blob) MarshalJSON 

func (x *Blob) MarshalJSON() ([]byte, error)

MarshalJSON marshals the Blob to JSON.

func (*Blob) MarshalProtoJSON 

func (x *Blob) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the Blob message to JSON.

func (*Blob) MarshalProtoText 

func (x *Blob) MarshalProtoText() string

func (*Blob) MarshalToSizedBufferVT 

func (m *Blob) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*Blob) MarshalToVT 

func (m *Blob) MarshalToVT(dAtA []byte) (int, error)

func (*Blob) MarshalVT 

func (m *Blob) MarshalVT() (dAtA []byte, err error)

func (*Blob) ProtoMessage 

func (*Blob) ProtoMessage()

func (*Blob) Reset 

func (x *Blob) Reset()

func (*Blob) SizeVT 

func (m *Blob) SizeVT() (n int)

func (*Blob) String 

func (x *Blob) String() string

func (*Blob) TransformToChunked 

func (b *Blob) TransformToChunked(ctx context.Context, bcs *block.Cursor, blobOpts *BuildBlobOpts) error

TransformToChunked transforms a raw blob to a chunked blob.

func (*Blob) TransformToRaw 

func (b *Blob) TransformToRaw(ctx context.Context, bcs *block.Cursor, nsize uint64) error

TransformToRaw transforms a chunked blob to a raw blob.

func (*Blob) Truncate 

func (b *Blob) Truncate(ctx context.Context, bcs *block.Cursor, blobOpts *BuildBlobOpts, nsize int64) error

Truncate changes the length of the blob.

func (*Blob) UnmarshalBlock 

func (b *Blob) UnmarshalBlock(data []byte) error

UnmarshalBlock unmarshals the block to the object. This is the final step of decoding, after transformations.

func (*Blob) UnmarshalJSON 

func (x *Blob) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the Blob from JSON.

func (*Blob) UnmarshalProtoJSON 

func (x *Blob) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the Blob message from JSON.

func (*Blob) UnmarshalVT 

func (m *Blob) UnmarshalVT(dAtA []byte) error

func (*Blob) Validate 

func (b *Blob) Validate() error

Validate performs cursory validation of the Blob object.

func (*Blob) ValidateFull 

func (b *Blob) ValidateFull(ctx context.Context, bcs *block.Cursor) error

ValidateFull performs a full fetch and validate on the blob. Depending on the blob implementation this will fetch data. The block cursor should be located at the blob root.

func (*Blob) WriteChunkIndex 

func (b *Blob) WriteChunkIndex(ctx context.Context, bcs *block.Cursor, opts *BuildBlobOpts, rdr io.Reader) error

WriteChunkIndex builds and writes the chunk index to the blob. bcs must be located at the blob.

type BlobType 

type BlobType int32

BlobType defines the types of blobs. 

const (
	// BlobType_RAW indicates the blob contains the data inline.
	// Default value: readers should check the value is actually zero.
	// This value being zero will save some encoding space.
	BlobType_BlobType_RAW BlobType = 0
	// BlobType_CHUNKED indicates the chunked blob format.
	// Stores data in sequential chunks of data, selected in a deterministic way.
	BlobType_BlobType_CHUNKED BlobType = 1
)

func (BlobType) Enum 

func (x BlobType) Enum() *BlobType

func (BlobType) MarshalJSON 

func (x BlobType) MarshalJSON() ([]byte, error)

MarshalJSON marshals the BlobType to JSON.

func (BlobType) MarshalProtoJSON 

func (x BlobType) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the BlobType to JSON.

func (BlobType) MarshalProtoText 

func (x BlobType) MarshalProtoText() string

func (BlobType) MarshalText 

func (x BlobType) MarshalText() ([]byte, error)

MarshalText marshals the BlobType to text.

func (BlobType) String 

func (x BlobType) String() string

func (*BlobType) UnmarshalJSON 

func (x *BlobType) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the BlobType from JSON.

func (*BlobType) UnmarshalProtoJSON 

func (x *BlobType) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the BlobType from JSON.

func (*BlobType) UnmarshalText 

func (x *BlobType) UnmarshalText(b []byte) error

UnmarshalText unmarshals the BlobType from text.

func (BlobType) Validate 

func (b BlobType) Validate() error

Validate validates the blob type from known types.

type BuildBlobOpts 

type BuildBlobOpts struct {

	// RawHighWaterMark is the limit for a raw block size.
	// Defaults to 512KB if unset.
	RawHighWaterMark uint64 `protobuf:"varint,1,opt,name=raw_high_water_mark,json=rawHighWaterMark,proto3" json:"rawHighWaterMark,omitempty"`
	// ChunkerArgs configures the chunker to use.
	ChunkerArgs *ChunkerArgs `protobuf:"bytes,2,opt,name=chunker_args,json=chunkerArgs,proto3" json:"chunkerArgs,omitempty"`
	// contains filtered or unexported fields
}

BuildBlobOpts are options to control the BuildBlob process.

func (*BuildBlobOpts) CloneMessageVT 

func (m *BuildBlobOpts) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*BuildBlobOpts) CloneVT 

func (m *BuildBlobOpts) CloneVT() *BuildBlobOpts

func (*BuildBlobOpts) EqualMessageVT 

func (this *BuildBlobOpts) EqualMessageVT(thatMsg any) bool

func (*BuildBlobOpts) EqualVT 

func (this *BuildBlobOpts) EqualVT(that *BuildBlobOpts) bool

func (*BuildBlobOpts) GetChunkerArgs 

func (x *BuildBlobOpts) GetChunkerArgs() *ChunkerArgs

func (*BuildBlobOpts) GetRawHighWaterMark 

func (x *BuildBlobOpts) GetRawHighWaterMark() uint64

func (*BuildBlobOpts) MarshalJSON 

func (x *BuildBlobOpts) MarshalJSON() ([]byte, error)

MarshalJSON marshals the BuildBlobOpts to JSON.

func (*BuildBlobOpts) MarshalProtoJSON 

func (x *BuildBlobOpts) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the BuildBlobOpts message to JSON.

func (*BuildBlobOpts) MarshalProtoText 

func (x *BuildBlobOpts) MarshalProtoText() string

func (*BuildBlobOpts) MarshalToSizedBufferVT 

func (m *BuildBlobOpts) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*BuildBlobOpts) MarshalToVT 

func (m *BuildBlobOpts) MarshalToVT(dAtA []byte) (int, error)

func (*BuildBlobOpts) MarshalVT 

func (m *BuildBlobOpts) MarshalVT() (dAtA []byte, err error)

func (*BuildBlobOpts) ProtoMessage 

func (*BuildBlobOpts) ProtoMessage()

func (*BuildBlobOpts) Reset 

func (x *BuildBlobOpts) Reset()

func (*BuildBlobOpts) SizeVT 

func (m *BuildBlobOpts) SizeVT() (n int)

func (*BuildBlobOpts) String 

func (x *BuildBlobOpts) String() string

func (*BuildBlobOpts) UnmarshalJSON 

func (x *BuildBlobOpts) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the BuildBlobOpts from JSON.

func (*BuildBlobOpts) UnmarshalProtoJSON 

func (x *BuildBlobOpts) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the BuildBlobOpts message from JSON.

func (*BuildBlobOpts) UnmarshalVT 

func (m *BuildBlobOpts) UnmarshalVT(dAtA []byte) error

type Chunk 

type Chunk struct {

	// DataRef is the reference to the data.
	// If empty, indicates a range of zeros.
	DataRef *block.BlockRef `protobuf:"bytes,1,opt,name=data_ref,json=dataRef,proto3" json:"dataRef,omitempty"`
	// Size is the size of the chunk.
	Size uint64 `protobuf:"varint,2,opt,name=size,proto3" json:"size,omitempty"`
	// Start is the start position of the chunk.
	// Must be equal to the sum of all previous chunks sizes.
	Start uint64 `protobuf:"varint,3,opt,name=start,proto3" json:"start,omitempty"`
	// contains filtered or unexported fields
}

Chunk contains in-line information about a data chunk.

func NewChunk 

func NewChunk(dataRef *block.BlockRef, size, start uint64) *Chunk

NewChunk constructs a new chunk.

func (*Chunk) ApplyBlockRef 

func (r *Chunk) ApplyBlockRef(id uint32, ptr *block.BlockRef) error

ApplyBlockRef applies a ref change with a field id. The reference may be nil if the child block is nil.

func (*Chunk) BlockAliasIdentity added in v0.52.0 

func (r *Chunk) BlockAliasIdentity() *block.AliasIdentityToken

BlockAliasIdentity returns the in-memory alias token for Chunk.

func (*Chunk) CloneMessageVT 

func (m *Chunk) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*Chunk) CloneVT 

func (m *Chunk) CloneVT() *Chunk

func (*Chunk) EqualMessageVT 

func (this *Chunk) EqualMessageVT(thatMsg any) bool

func (*Chunk) EqualVT 

func (this *Chunk) EqualVT(that *Chunk) bool

func (*Chunk) FetchData 

func (r *Chunk) FetchData(ctx context.Context, bcs *block.Cursor, copyBuf bool) ([]byte, error)

FetchData fetches the data reference. bcs should be located at chunk

func (*Chunk) FetchDataNoCache added in v0.51.7 

func (r *Chunk) FetchDataNoCache(ctx context.Context, bcs *block.Cursor, copyBuf bool) ([]byte, error)

FetchDataNoCache fetches the data reference without retaining the chunk bytes in the cursor. Use this for sequential read paths that only need the current chunk as scratch data.

func (*Chunk) FollowDataRef 

func (r *Chunk) FollowDataRef(bcs *block.Cursor) *block.Cursor

FollowDataRef follows the data reference.

func (*Chunk) GetBlockRefCtor 

func (r *Chunk) GetBlockRefCtor(id uint32) block.Ctor

GetBlockRefCtor returns the constructor for the block at the ref id. Return nil to indicate invalid ref ID or unknown.

func (*Chunk) GetBlockRefs 

func (r *Chunk) GetBlockRefs() (map[uint32]*block.BlockRef, error)

GetBlockRefs returns all block references by ID. May return nil, and values may also be nil. Note: this does not include pending references (in a cursor)

func (*Chunk) GetDataRef 

func (x *Chunk) GetDataRef() *block.BlockRef

func (*Chunk) GetSize 

func (x *Chunk) GetSize() uint64

func (*Chunk) GetStart 

func (x *Chunk) GetStart() uint64

func (*Chunk) IsNil 

func (r *Chunk) IsNil() bool

IsNil returns if the object is nil.

func (*Chunk) MarshalBlock 

func (r *Chunk) MarshalBlock() ([]byte, error)

MarshalBlock marshals the block to binary.

func (*Chunk) MarshalJSON 

func (x *Chunk) MarshalJSON() ([]byte, error)

MarshalJSON marshals the Chunk to JSON.

func (*Chunk) MarshalProtoJSON 

func (x *Chunk) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the Chunk message to JSON.

func (*Chunk) MarshalProtoText 

func (x *Chunk) MarshalProtoText() string

func (*Chunk) MarshalToSizedBufferVT 

func (m *Chunk) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*Chunk) MarshalToVT 

func (m *Chunk) MarshalToVT(dAtA []byte) (int, error)

func (*Chunk) MarshalVT 

func (m *Chunk) MarshalVT() (dAtA []byte, err error)

func (*Chunk) ProtoMessage 

func (*Chunk) ProtoMessage()

func (*Chunk) Reset 

func (x *Chunk) Reset()

func (*Chunk) SizeVT 

func (m *Chunk) SizeVT() (n int)

func (*Chunk) String 

func (x *Chunk) String() string

func (*Chunk) UnmarshalBlock 

func (r *Chunk) UnmarshalBlock(data []byte) error

UnmarshalBlock unmarshals the block to the object.

func (*Chunk) UnmarshalJSON 

func (x *Chunk) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the Chunk from JSON.

func (*Chunk) UnmarshalProtoJSON 

func (x *Chunk) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the Chunk message from JSON.

func (*Chunk) UnmarshalVT 

func (m *Chunk) UnmarshalVT(dAtA []byte) error

func (*Chunk) Validate 

func (r *Chunk) Validate() error

Validate checks the reference.

type ChunkIndex 

type ChunkIndex struct {

	// Chunks contains the in-line list of chunks.
	// Sequential.
	Chunks []*Chunk `protobuf:"bytes,1,rep,name=chunks,proto3" json:"chunks,omitempty"`
	// ChunkerArgs are optional arguments for the chunker.
	ChunkerArgs *ChunkerArgs `protobuf:"bytes,2,opt,name=chunker_args,json=chunkerArgs,proto3" json:"chunkerArgs,omitempty"`
	// contains filtered or unexported fields
}

ChunkIndex is the root of the chunked blob type.

func BuildChunkIndex 

func BuildChunkIndex(
	ctx context.Context,
	rdr io.Reader,
	bcs *block.Cursor,
	chunkerArgs *ChunkerArgs,
) (*ChunkIndex, uint64, error)

BuildChunkIndex constructs a chunk index. Blocks will be written to the block transaction. If bcs already contains a ChunkIndex, it will be reused. If poly is zero, the default constant polynomial will be used.

func NewChunkIndex 

func NewChunkIndex(chunks []*Chunk) *ChunkIndex

NewChunkIndex constructs a new chunk index.

func UnmarshalChunkIndex 

func UnmarshalChunkIndex(ctx context.Context, bcs *block.Cursor) (*ChunkIndex, error)

UnmarshalChunkIndex unmarshals a chunk index from a cursor. If empty, returns nil, nil

func (*ChunkIndex) AppendChunk 

func (r *ChunkIndex) AppendChunk(chkSet *sbset.SubBlockSet, idx int, size, start uint64, data []byte)

AppendChunk appends a chunk with the given data.

func (*ChunkIndex) ApplySubBlock 

func (r *ChunkIndex) ApplySubBlock(id uint32, next block.SubBlock) error

ApplySubBlock applies a sub-block change with a field id.

func (*ChunkIndex) BlockAliasIdentity added in v0.52.0 

func (r *ChunkIndex) BlockAliasIdentity() *block.AliasIdentityToken

BlockAliasIdentity returns the in-memory alias token for ChunkIndex.

func (*ChunkIndex) CloneMessageVT 

func (m *ChunkIndex) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*ChunkIndex) CloneVT 

func (m *ChunkIndex) CloneVT() *ChunkIndex

func (*ChunkIndex) EqualMessageVT 

func (this *ChunkIndex) EqualMessageVT(thatMsg any) bool

func (*ChunkIndex) EqualVT 

func (this *ChunkIndex) EqualVT(that *ChunkIndex) bool

func (*ChunkIndex) GetChunkSet 

func (r *ChunkIndex) GetChunkSet(bcs *block.Cursor) *sbset.SubBlockSet

GetChunkSet returns the chunk set sub-block.

func (*ChunkIndex) GetChunkerArgs 

func (x *ChunkIndex) GetChunkerArgs() *ChunkerArgs

func (*ChunkIndex) GetChunks 

func (x *ChunkIndex) GetChunks() []*Chunk

func (*ChunkIndex) GetSubBlockCtor 

func (r *ChunkIndex) GetSubBlockCtor(id uint32) block.SubBlockCtor

GetSubBlockCtor returns a function which creates or returns the existing sub-block at reference id. Can return nil to indicate invalid reference id.

func (*ChunkIndex) GetSubBlocks 

func (r *ChunkIndex) GetSubBlocks() map[uint32]block.SubBlock

GetSubBlocks returns all constructed sub-blocks by ID. May return nil, and values may also be nil.

func (*ChunkIndex) IsNil 

func (r *ChunkIndex) IsNil() bool

IsNil returns if the object is nil.

func (*ChunkIndex) MarshalBlock 

func (r *ChunkIndex) MarshalBlock() ([]byte, error)

MarshalBlock marshals the block to binary.

func (*ChunkIndex) MarshalJSON 

func (x *ChunkIndex) MarshalJSON() ([]byte, error)

MarshalJSON marshals the ChunkIndex to JSON.

func (*ChunkIndex) MarshalProtoJSON 

func (x *ChunkIndex) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the ChunkIndex message to JSON.

func (*ChunkIndex) MarshalProtoText 

func (x *ChunkIndex) MarshalProtoText() string

func (*ChunkIndex) MarshalToSizedBufferVT 

func (m *ChunkIndex) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*ChunkIndex) MarshalToVT 

func (m *ChunkIndex) MarshalToVT(dAtA []byte) (int, error)

func (*ChunkIndex) MarshalVT 

func (m *ChunkIndex) MarshalVT() (dAtA []byte, err error)

func (*ChunkIndex) ProtoMessage 

func (*ChunkIndex) ProtoMessage()

func (*ChunkIndex) Reset 

func (x *ChunkIndex) Reset()

func (*ChunkIndex) SizeVT 

func (m *ChunkIndex) SizeVT() (n int)

func (*ChunkIndex) String 

func (x *ChunkIndex) String() string

func (*ChunkIndex) UnmarshalBlock 

func (r *ChunkIndex) UnmarshalBlock(data []byte) error

UnmarshalBlock unmarshals the block to the object.

func (*ChunkIndex) UnmarshalJSON 

func (x *ChunkIndex) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the ChunkIndex from JSON.

func (*ChunkIndex) UnmarshalProtoJSON 

func (x *ChunkIndex) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the ChunkIndex message from JSON.

func (*ChunkIndex) UnmarshalVT 

func (m *ChunkIndex) UnmarshalVT(dAtA []byte) error

func (*ChunkIndex) Validate 

func (r *ChunkIndex) Validate() error

Validate checks the reference.

type ChunkerArgs 

type ChunkerArgs struct {

	// ChunkerType is the chunking algorithm used.
	// Defaults to ChunkerType_JC if not set.
	ChunkerType ChunkerType `protobuf:"varint,1,opt,name=chunker_type,json=chunkerType,proto3" json:"chunkerType,omitempty"`
	// RabinArgs are arguments for the rabin chunker.
	// ChunkerType_RABIN
	RabinArgs *RabinArgs `protobuf:"bytes,2,opt,name=rabin_args,json=rabinArgs,proto3" json:"rabinArgs,omitempty"`
	// JcArgs are arguments for the jc chunker.
	// ChunkerType_JC
	JcArgs *JcArgs `protobuf:"bytes,3,opt,name=jc_args,json=jcArgs,proto3" json:"jcArgs,omitempty"`
	// contains filtered or unexported fields
}

ChunkerArgs configures the chunking algorithm.

func (*ChunkerArgs) ApplyArgs 

func (c *ChunkerArgs) ApplyArgs(other *ChunkerArgs)

ApplyArgs merges another arguments object into ChunkerArgs.

func (*ChunkerArgs) CloneMessageVT 

func (m *ChunkerArgs) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*ChunkerArgs) CloneVT 

func (m *ChunkerArgs) CloneVT() *ChunkerArgs

func (*ChunkerArgs) EqualMessageVT 

func (this *ChunkerArgs) EqualMessageVT(thatMsg any) bool

func (*ChunkerArgs) EqualVT 

func (this *ChunkerArgs) EqualVT(that *ChunkerArgs) bool

func (*ChunkerArgs) GetChunkerType 

func (x *ChunkerArgs) GetChunkerType() ChunkerType

func (*ChunkerArgs) GetJcArgs 

func (x *ChunkerArgs) GetJcArgs() *JcArgs

func (*ChunkerArgs) GetRabinArgs 

func (x *ChunkerArgs) GetRabinArgs() *RabinArgs

func (*ChunkerArgs) MarshalJSON 

func (x *ChunkerArgs) MarshalJSON() ([]byte, error)

MarshalJSON marshals the ChunkerArgs to JSON.

func (*ChunkerArgs) MarshalProtoJSON 

func (x *ChunkerArgs) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the ChunkerArgs message to JSON.

func (*ChunkerArgs) MarshalProtoText 

func (x *ChunkerArgs) MarshalProtoText() string

func (*ChunkerArgs) MarshalToSizedBufferVT 

func (m *ChunkerArgs) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*ChunkerArgs) MarshalToVT 

func (m *ChunkerArgs) MarshalToVT(dAtA []byte) (int, error)

func (*ChunkerArgs) MarshalVT 

func (m *ChunkerArgs) MarshalVT() (dAtA []byte, err error)

func (*ChunkerArgs) ProtoMessage 

func (*ChunkerArgs) ProtoMessage()

func (*ChunkerArgs) Reset 

func (x *ChunkerArgs) Reset()

func (*ChunkerArgs) SizeVT 

func (m *ChunkerArgs) SizeVT() (n int)

func (*ChunkerArgs) String 

func (x *ChunkerArgs) String() string

func (*ChunkerArgs) UnmarshalJSON 

func (x *ChunkerArgs) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the ChunkerArgs from JSON.

func (*ChunkerArgs) UnmarshalProtoJSON 

func (x *ChunkerArgs) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the ChunkerArgs message from JSON.

func (*ChunkerArgs) UnmarshalVT 

func (m *ChunkerArgs) UnmarshalVT(dAtA []byte) error

type ChunkerType 

type ChunkerType int32

ChunkerType is the set of known chunker types. 

const (
	// ChunkerType_DEFAULT builds/appends using the default chunker (JC).
	ChunkerType_ChunkerType_DEFAULT ChunkerType = 0
	// ChunkerType_RABIN uses rabin fingerprinting to chunk.
	ChunkerType_ChunkerType_RABIN ChunkerType = 1
	// ChunkerType_JC uses JC content defined chunk algorithm.
	ChunkerType_ChunkerType_JC ChunkerType = 2
)

func (ChunkerType) Enum 

func (x ChunkerType) Enum() *ChunkerType

func (ChunkerType) MarshalJSON 

func (x ChunkerType) MarshalJSON() ([]byte, error)

MarshalJSON marshals the ChunkerType to JSON.

func (ChunkerType) MarshalProtoJSON 

func (x ChunkerType) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the ChunkerType to JSON.

func (ChunkerType) MarshalProtoText 

func (x ChunkerType) MarshalProtoText() string

func (ChunkerType) MarshalText 

func (x ChunkerType) MarshalText() ([]byte, error)

MarshalText marshals the ChunkerType to text.

func (ChunkerType) String 

func (x ChunkerType) String() string

func (*ChunkerType) UnmarshalJSON 

func (x *ChunkerType) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the ChunkerType from JSON.

func (*ChunkerType) UnmarshalProtoJSON 

func (x *ChunkerType) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the ChunkerType from JSON.

func (*ChunkerType) UnmarshalText 

func (x *ChunkerType) UnmarshalText(b []byte) error

UnmarshalText unmarshals the ChunkerType from text.

type JcArgs 

type JcArgs struct {

	// Key is the key for the chunker.
	// Optional.
	Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
	// ChunkingMinSize is the minimum size for a chunk.
	// Defaults to 256000 bytes.
	ChunkingMinSize uint64 `protobuf:"varint,2,opt,name=chunking_min_size,json=chunkingMinSize,proto3" json:"chunkingMinSize,omitempty"`
	// ChunkingTargetSize is the target size for a chunk.
	// Defaults to 512000 bytes.
	ChunkingTargetSize uint64 `protobuf:"varint,3,opt,name=chunking_target_size,json=chunkingTargetSize,proto3" json:"chunkingTargetSize,omitempty"`
	// ChunkingMaxSize is the maximum size for a chunk.
	// Defaults to 786432 bytes.
	ChunkingMaxSize uint64 `protobuf:"varint,4,opt,name=chunking_max_size,json=chunkingMaxSize,proto3" json:"chunkingMaxSize,omitempty"`
	// contains filtered or unexported fields
}

JcArgs are arguments for the jc chunker.

ChunkerType_JC

func (*JcArgs) ApplyArgs 

func (c *JcArgs) ApplyArgs(other *JcArgs)

ApplyArgs merges another arguments object into JcArgs.

func (*JcArgs) CloneMessageVT 

func (m *JcArgs) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*JcArgs) CloneVT 

func (m *JcArgs) CloneVT() *JcArgs

func (*JcArgs) EqualMessageVT 

func (this *JcArgs) EqualMessageVT(thatMsg any) bool

func (*JcArgs) EqualVT 

func (this *JcArgs) EqualVT(that *JcArgs) bool

func (*JcArgs) GetChunkingMaxSize 

func (x *JcArgs) GetChunkingMaxSize() uint64

func (*JcArgs) GetChunkingMinSize 

func (x *JcArgs) GetChunkingMinSize() uint64

func (*JcArgs) GetChunkingTargetSize 

func (x *JcArgs) GetChunkingTargetSize() uint64

func (*JcArgs) GetKey 

func (x *JcArgs) GetKey() []byte

func (*JcArgs) MarshalJSON 

func (x *JcArgs) MarshalJSON() ([]byte, error)

MarshalJSON marshals the JcArgs to JSON.

func (*JcArgs) MarshalProtoJSON 

func (x *JcArgs) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the JcArgs message to JSON.

func (*JcArgs) MarshalProtoText 

func (x *JcArgs) MarshalProtoText() string

func (*JcArgs) MarshalToSizedBufferVT 

func (m *JcArgs) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*JcArgs) MarshalToVT 

func (m *JcArgs) MarshalToVT(dAtA []byte) (int, error)

func (*JcArgs) MarshalVT 

func (m *JcArgs) MarshalVT() (dAtA []byte, err error)

func (*JcArgs) ProtoMessage 

func (*JcArgs) ProtoMessage()

func (*JcArgs) Reset 

func (x *JcArgs) Reset()

func (*JcArgs) SizeVT 

func (m *JcArgs) SizeVT() (n int)

func (*JcArgs) String 

func (x *JcArgs) String() string

func (*JcArgs) UnmarshalJSON 

func (x *JcArgs) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the JcArgs from JSON.

func (*JcArgs) UnmarshalProtoJSON 

func (x *JcArgs) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the JcArgs message from JSON.

func (*JcArgs) UnmarshalVT 

func (m *JcArgs) UnmarshalVT(dAtA []byte) error

type Metric added in v0.52.0 

type Metric struct {
	Stage            string
	InputBytes       int64
	ChunkBytes       int
	RawHighWaterMark uint64
	ChunkIndex       int
	DirectPut        bool
}

Metric records one blob-build memory-relevant decision point.

type MetricsRecorder added in v0.52.0 

type MetricsRecorder interface {
	RecordBlobMetric(Metric)
}

MetricsRecorder consumes blob-build metrics.

type RabinArgs 

type RabinArgs struct {

	// Rabin polynomial.
	// Optional.
	Pol uint64 `protobuf:"varint,1,opt,name=pol,proto3" json:"pol,omitempty"`
	// RandomPol enables randomizing pol instead of using the default.
	// This is not recommended.
	// If pol != 0 this field is ignored.
	RandomPol bool `protobuf:"varint,4,opt,name=random_pol,json=randomPol,proto3" json:"randomPol,omitempty"`
	// ChunkingMinSize is the minimum size for a chunk.
	// Defaults to 256000 bytes.
	ChunkingMinSize uint64 `protobuf:"varint,2,opt,name=chunking_min_size,json=chunkingMinSize,proto3" json:"chunkingMinSize,omitempty"`
	// ChunkingMaxSize is the maxmium size for a chunk.
	// Defaults to 786432 bytes.
	ChunkingMaxSize uint64 `protobuf:"varint,3,opt,name=chunking_max_size,json=chunkingMaxSize,proto3" json:"chunkingMaxSize,omitempty"`
	// contains filtered or unexported fields
}

RabinArgs are arguments for the rabin chunker.

The default polynomial is 0x2df7f4e3b27061

ChunkerType_RABIN

func (*RabinArgs) ApplyArgs 

func (c *RabinArgs) ApplyArgs(other *RabinArgs)

ApplyArgs merges another arguments object into RabinArgs.

func (*RabinArgs) CloneMessageVT 

func (m *RabinArgs) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*RabinArgs) CloneVT 

func (m *RabinArgs) CloneVT() *RabinArgs

func (*RabinArgs) EqualMessageVT 

func (this *RabinArgs) EqualMessageVT(thatMsg any) bool

func (*RabinArgs) EqualVT 

func (this *RabinArgs) EqualVT(that *RabinArgs) bool

func (*RabinArgs) GetChunkingMaxSize 

func (x *RabinArgs) GetChunkingMaxSize() uint64

func (*RabinArgs) GetChunkingMinSize 

func (x *RabinArgs) GetChunkingMinSize() uint64

func (*RabinArgs) GetPol 

func (x *RabinArgs) GetPol() uint64

func (*RabinArgs) GetRandomPol 

func (x *RabinArgs) GetRandomPol() bool

func (*RabinArgs) MarshalJSON 

func (x *RabinArgs) MarshalJSON() ([]byte, error)

MarshalJSON marshals the RabinArgs to JSON.

func (*RabinArgs) MarshalProtoJSON 

func (x *RabinArgs) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the RabinArgs message to JSON.

func (*RabinArgs) MarshalProtoText 

func (x *RabinArgs) MarshalProtoText() string

func (*RabinArgs) MarshalToSizedBufferVT 

func (m *RabinArgs) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*RabinArgs) MarshalToVT 

func (m *RabinArgs) MarshalToVT(dAtA []byte) (int, error)

func (*RabinArgs) MarshalVT 

func (m *RabinArgs) MarshalVT() (dAtA []byte, err error)

func (*RabinArgs) ProtoMessage 

func (*RabinArgs) ProtoMessage()

func (*RabinArgs) Reset 

func (x *RabinArgs) Reset()

func (*RabinArgs) SizeVT 

func (m *RabinArgs) SizeVT() (n int)

func (*RabinArgs) String 

func (x *RabinArgs) String() string

func (*RabinArgs) UnmarshalJSON 

func (x *RabinArgs) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the RabinArgs from JSON.

func (*RabinArgs) UnmarshalProtoJSON 

func (x *RabinArgs) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the RabinArgs message from JSON.

func (*RabinArgs) UnmarshalVT 

func (m *RabinArgs) UnmarshalVT(dAtA []byte) error

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads from a blob.

func NewRawReader 

func NewRawReader(ctx context.Context, blob *Blob) *Reader

NewRawReader reads blobs of type raw only.

func NewReader 

func NewReader(
	ctx context.Context,
	bcs *block.Cursor,
) (*Reader, error)

NewReader constructs a new reader. bcs is located at the root of the blob. bcs can have an empty block if needed.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the reader, canceling the context.

func (*Reader) Read 

func (r *Reader) Read(p []byte) (n int, err error)

Read implements the reader interface. Read and Seek are not concurrent safe.

func (*Reader) Seek 

func (r *Reader) Seek(offset int64, whence int) (int64, error)

Seek implements the seeking interface. Seek sets the offset for the next Read or Write to offset, interpreted according to whence: SeekStart means relative to the start of the file, SeekCurrent means relative to the current offset, and SeekEnd means relative to the end. Seek returns the new offset relative to the start of the file and an error, if any.

Seeking past the end of the blob does NOT immediately trigger EOF.

Seeking to an offset before the start of the file is an error. Seeking to any positive offset is legal, but the behavior of subsequent I/O operations on the underlying object is implementation-dependent. Read and Seek are not concurrent safe.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.