dag

package
v2.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2025 License: MIT Imports: 19 Imported by: 11

Documentation

Index

Constants

View Source 
const DefaultBatchSize = 4 * 1024 * 1024 // 4MB
View Source 
const DefaultChunkSize = 2048 * 1024 // 2MB

Variables

View Source 
var BatchSize = DefaultBatchSize
View Source 
var ChunkSize = DefaultChunkSize

Functions

func CalculateTotalContentSize 

func CalculateTotalContentSize(dag *Dag) int64

func CalculateTotalDagSize 

func CalculateTotalDagSize(dag *Dag) (int64, error)

func DisableBatching 

func DisableBatching()

func DisableChunking 

func DisableChunking()

func SetBatchSize 

func SetBatchSize(size int)

func SetChunkSize 

func SetChunkSize(size int)

func SetDefaultBatchSize 

func SetDefaultBatchSize()

func SetDefaultChunkSize 

func SetDefaultChunkSize()

func SortMapByKeys 

func SortMapByKeys(inputMap map[string]string) map[string]string

Types

type BatchedTransmissionPacket 

type BatchedTransmissionPacket struct {
	Leaves        []*DagLeaf
	Relationships map[string]string // childHash -> parentHash
	PacketIndex   int
	TotalPackets  int
}

func BatchedTransmissionPacketFromCBOR 

func BatchedTransmissionPacketFromCBOR(data []byte) (*BatchedTransmissionPacket, error)

BatchedTransmissionPacketFromCBOR deserializes a BatchedTransmissionPacket from CBOR format

func BatchedTransmissionPacketFromJSON 

func BatchedTransmissionPacketFromJSON(data []byte) (*BatchedTransmissionPacket, error)

BatchedTransmissionPacketFromJSON deserializes a BatchedTransmissionPacket from JSON format

func BatchedTransmissionPacketFromSerializable 

func BatchedTransmissionPacketFromSerializable(s *SerializableBatchedTransmissionPacket) *BatchedTransmissionPacket

BatchedTransmissionPacketFromSerializable reconstructs a BatchedTransmissionPacket from its serializable form

func (*BatchedTransmissionPacket) GetRootLeaf 

func (packet *BatchedTransmissionPacket) GetRootLeaf() *DagLeaf

GetRootLeaf returns the root leaf from the batch. The root leaf is the one with an empty parent in the Relationships map.

func (*BatchedTransmissionPacket) ToCBOR 

func (packet *BatchedTransmissionPacket) ToCBOR() ([]byte, error)

ToCBOR serializes a BatchedTransmissionPacket to CBOR format

func (*BatchedTransmissionPacket) ToJSON 

func (packet *BatchedTransmissionPacket) ToJSON() ([]byte, error)

ToJSON serializes a BatchedTransmissionPacket to JSON format

func (*BatchedTransmissionPacket) ToSerializable 

func (packet *BatchedTransmissionPacket) ToSerializable() *SerializableBatchedTransmissionPacket

ToSerializable converts a BatchedTransmissionPacket to its serializable form

type ClassicTreeBranch 

type ClassicTreeBranch struct {
	Leaf  string
	Proof *merkletree.Proof
}

type Dag 

type Dag struct {
	Root   string
	Leafs  map[string]*DagLeaf
	Labels map[string]string // label -> leaf hash (excludes root which is always "0")
}

func CreateDag 

func CreateDag(path string, timestampRoot bool) (*Dag, error)

func CreateDagAdvanced 

func CreateDagAdvanced(path string, additionalData map[string]string) (*Dag, error)

func CreateDagCustom 

func CreateDagCustom(path string, rootAdditionalData map[string]string, processor LeafProcessor) (*Dag, error)

func CreateDagWithConfig 

func CreateDagWithConfig(path string, config *DagBuilderConfig) (*Dag, error)

func FromCBOR 

func FromCBOR(data []byte) (*Dag, error)

func FromJSON 

func FromJSON(data []byte) (*Dag, error)

func FromSerializable 

func FromSerializable(s *SerializableDag) *Dag

func ReadDag 

func ReadDag(path string) (*Dag, error)

func (*Dag) ApplyAndVerifyBatchedTransmissionPacket 

func (d *Dag) ApplyAndVerifyBatchedTransmissionPacket(packet *BatchedTransmissionPacket) error

func (*Dag) ApplyAndVerifyTransmissionPacket 

func (d *Dag) ApplyAndVerifyTransmissionPacket(packet *TransmissionPacket) error

func (*Dag) ApplyBatchedTransmissionPacket 

func (d *Dag) ApplyBatchedTransmissionPacket(packet *BatchedTransmissionPacket)

func (*Dag) ApplyTransmissionPacket 

func (d *Dag) ApplyTransmissionPacket(packet *TransmissionPacket)

func (*Dag) CalculateLabels 

func (d *Dag) CalculateLabels() error

CalculateLabels populates the Labels map with deterministic label assignments. Each leaf hash (excluding the root) is assigned a numeric label as a string. The root is always label "0" and is not included in the map. This function is deterministic - calling it multiple times on the same DAG will always produce the same label assignments based on DAG traversal order.

func (*Dag) ClearLabels 

func (d *Dag) ClearLabels()

ClearLabels removes all label assignments from the DAG.

func (*Dag) CreateDirectory 

func (dag *Dag) CreateDirectory(path string) error

func (*Dag) GetBatchedLeafSequence 

func (d *Dag) GetBatchedLeafSequence() []*BatchedTransmissionPacket

GetBatchedLeafSequence groups leaves into batches up to BatchSize for efficient transmission

func (*Dag) GetContentFromLeaf 

func (dag *Dag) GetContentFromLeaf(leaf *DagLeaf) ([]byte, error)

func (*Dag) GetHashesByLabelRange 

func (d *Dag) GetHashesByLabelRange(startLabel, endLabel string) ([]string, error)

GetHashesByLabelRange returns an array of leaf hashes for the specified label range (inclusive). startLabel and endLabel are string representations of numeric labels. For example, GetHashesByLabelRange("20", "48") returns hashes for labels 20, 21, 22, ..., 48. Returns an error if labels are invalid, out of range, or if labels haven't been calculated.

func (*Dag) GetLabel 

func (d *Dag) GetLabel(hash string) (string, error)

GetLabel returns the label for a given leaf hash. Returns "0" if the hash is the root, or the numeric label as a string for other leaves. Returns an error if labels haven't been calculated or if the hash is not found.

func (*Dag) GetLeafSequence 

func (d *Dag) GetLeafSequence() []*TransmissionPacket

GetLeafSequence returns leaves in transmission order (BFS) with parent refs and proofs

func (*Dag) GetPartial 

func (d *Dag) GetPartial(leafHashes []string, pruneLinks bool) (*Dag, error)

GetPartial creates a partial DAG with specified leaves and their verification paths pruneLinks: true = remove unreferenced links (verification), false = keep all links (reconstruction)

func (*Dag) IsPartial 

func (d *Dag) IsPartial() bool

IsPartial returns true if this DAG is a partial DAG (has fewer leaves than the total count)

func (*Dag) IterateDag 

func (d *Dag) IterateDag(processLeaf func(leaf *DagLeaf, parent *DagLeaf) error) error

func (*Dag) RemoveAllContent 

func (d *Dag) RemoveAllContent()

func (*Dag) ToCBOR 

func (dag *Dag) ToCBOR() ([]byte, error)

func (*Dag) ToJSON 

func (dag *Dag) ToJSON() ([]byte, error)

func (*Dag) ToSerializable 

func (dag *Dag) ToSerializable() *SerializableDag

func (*Dag) Verify 

func (d *Dag) Verify() error

Verify checks the integrity of the DAG, automatically choosing between full and partial verification

func (*Dag) VerifyBatchedTransmissionPacket 

func (d *Dag) VerifyBatchedTransmissionPacket(packet *BatchedTransmissionPacket) error

func (*Dag) VerifyTransmissionPacket 

func (d *Dag) VerifyTransmissionPacket(packet *TransmissionPacket) error

type DagBranch 

type DagBranch struct {
	Leaf         *DagLeaf
	Path         []*DagLeaf
	MerkleProofs map[string]*ClassicTreeBranch
}

type DagBuilder 

type DagBuilder struct {
	Leafs map[string]*DagLeaf
	// contains filtered or unexported fields
}

func CreateDagBuilder 

func CreateDagBuilder() *DagBuilder

func (*DagBuilder) AddLeaf 

func (b *DagBuilder) AddLeaf(leaf *DagLeaf, parentLeaf *DagLeaf) error

func (*DagBuilder) AddLeafSafe 

func (b *DagBuilder) AddLeafSafe(leaf *DagLeaf, parentLeaf *DagLeaf) error

AddLeafSafe is a thread-safe version of AddLeaf for parallel processing

func (*DagBuilder) BuildDag 

func (b *DagBuilder) BuildDag(root string) *Dag

type DagBuilderConfig 

type DagBuilderConfig struct {
	// EnableParallel enables parallel processing of files and directories
	// Default: false (sequential processing for backward compatibility)
	EnableParallel bool

	// MaxWorkers controls the maximum number of concurrent goroutines when parallel processing
	// 0 = use runtime.NumCPU() (auto-detect based on available cores)
	// -1 = unlimited workers (not recommended, may overwhelm system)
	// >0 = use exactly this many workers
	MaxWorkers     int  // Parallel only, 0=auto-detect
	TimestampRoot  bool // Add timestamp to root
	AdditionalData map[string]string
	Processor      LeafProcessor
}

DagBuilderConfig controls DAG building behavior

func DefaultConfig 

func DefaultConfig() *DagBuilderConfig

func ParallelConfig 

func ParallelConfig() *DagBuilderConfig

func ParallelConfigWithWorkers 

func ParallelConfigWithWorkers(workers int) *DagBuilderConfig

type DagLeaf 

type DagLeaf struct {
	Hash              string                          `json:"hash"`
	ItemName          string                          `json:"item_name"`
	Type              LeafType                        `json:"type"`
	ContentHash       []byte                          `json:"content_hash,omitempty"`
	Content           []byte                          `json:"content,omitempty"`
	ClassicMerkleRoot []byte                          `json:"classic_merkle_root,omitempty"`
	CurrentLinkCount  int                             `json:"current_link_count"`
	LeafCount         int                             `json:"leaf_count,omitempty"`
	ContentSize       int64                           `json:"content_size,omitempty"`
	DagSize           int64                           `json:"dag_size,omitempty"`
	Links             []string                        `json:"links,omitempty"`
	ParentHash        string                          `json:"parent_hash,omitempty"`
	AdditionalData    map[string]string               `json:"additional_data,omitempty"`
	MerkleTree        *merkletree.MerkleTree          `json:"-"`
	LeafMap           map[string]merkletree.DataBlock `json:"-"`
	Proofs            map[string]*ClassicTreeBranch   `json:"proofs,omitempty"`
}
func (leaf *DagLeaf) AddLink(hash string)

func (*DagLeaf) Clone 

func (leaf *DagLeaf) Clone() *DagLeaf

func (*DagLeaf) CreateDirectoryLeaf 

func (leaf *DagLeaf) CreateDirectoryLeaf(path string, dag *Dag) error

func (*DagLeaf) EstimateSize 

func (leaf *DagLeaf) EstimateSize() int

EstimateSize returns approximate serialized size without full CBOR encoding

func (*DagLeaf) GetBranch 

func (leaf *DagLeaf) GetBranch(key string) (*ClassicTreeBranch, error)

func (*DagLeaf) GetIndexForKey 

func (leaf *DagLeaf) GetIndexForKey(key string) (int, bool)
func (leaf *DagLeaf) HasLink(hash string) bool

func (*DagLeaf) ToSerializable 

func (leaf *DagLeaf) ToSerializable() *SerializableDagLeaf

ToSerializable converts a DagLeaf to its serializable form

func (*DagLeaf) VerifyBranch 

func (leaf *DagLeaf) VerifyBranch(branch *ClassicTreeBranch) error

func (*DagLeaf) VerifyChildrenAgainstMerkleRoot 

func (leaf *DagLeaf) VerifyChildrenAgainstMerkleRoot(dag *Dag) error

VerifyChildrenAgainstMerkleRoot checks ClassicMerkleRoot against actual children

func (*DagLeaf) VerifyLeaf 

func (leaf *DagLeaf) VerifyLeaf() error

func (*DagLeaf) VerifyRootLeaf 

func (leaf *DagLeaf) VerifyRootLeaf(dag *Dag) error

type DagLeafBuilder 

type DagLeafBuilder struct {
	ItemName string
	Label    int64
	LeafType LeafType
	Data     []byte
	Links    []string
}

func CreateDagLeafBuilder 

func CreateDagLeafBuilder(name string) *DagLeafBuilder
func (b *DagLeafBuilder) AddLink(hash string)

func (*DagLeafBuilder) BuildLeaf 

func (b *DagLeafBuilder) BuildLeaf(additionalData map[string]string) (*DagLeaf, error)

func (*DagLeafBuilder) BuildRootLeaf 

func (b *DagLeafBuilder) BuildRootLeaf(dag *DagBuilder, additionalData map[string]string) (*DagLeaf, error)

func (*DagLeafBuilder) SetData 

func (b *DagLeafBuilder) SetData(data []byte)

func (*DagLeafBuilder) SetType 

func (b *DagLeafBuilder) SetType(leafType LeafType)

type KeyValue 

type KeyValue struct {
	Key   string
	Value string
}

func SortMapForVerification 

func SortMapForVerification(inputMap map[string]string) []KeyValue

type LeafProcessor 

type LeafProcessor func(path string, relPath string, entry fs.DirEntry, isRoot bool, leafType LeafType) map[string]string

LeafProcessor generates custom metadata for a leaf during DAG creation

type LeafType 

type LeafType string
const (
	FileLeafType      LeafType = "file"
	ChunkLeafType     LeafType = "chunk"
	DirectoryLeafType LeafType = "directory"
)

type SerializableBatchedTransmissionPacket 

type SerializableBatchedTransmissionPacket struct {
	Leaves        []*SerializableDagLeaf
	Relationships map[string]string
}

type SerializableDag 

type SerializableDag struct {
	Root  string
	Leafs map[string]*SerializableDagLeaf
}

type SerializableDagLeaf 

type SerializableDagLeaf struct {
	Hash              string
	ItemName          string
	Type              LeafType
	ContentHash       []byte
	Content           []byte
	ClassicMerkleRoot []byte
	CurrentLinkCount  int
	LeafCount         int
	ContentSize       int64
	DagSize           int64
	Links             []string
	AdditionalData    map[string]string
	StoredProofs      map[string]*ClassicTreeBranch `json:"stored_proofs,omitempty" cbor:"stored_proofs,omitempty"`
}

type SerializableTransmissionPacket 

type SerializableTransmissionPacket struct {
	Leaf       *SerializableDagLeaf
	ParentHash string
	Proofs     map[string]*ClassicTreeBranch `json:"proofs,omitempty" cbor:"proofs,omitempty"`
}

type TransmissionPacket 

type TransmissionPacket struct {
	Leaf       *DagLeaf
	ParentHash string
	Proofs     map[string]*ClassicTreeBranch
}

func TransmissionPacketFromCBOR 

func TransmissionPacketFromCBOR(data []byte) (*TransmissionPacket, error)

TransmissionPacketFromCBOR deserializes a TransmissionPacket from CBOR format

func TransmissionPacketFromJSON 

func TransmissionPacketFromJSON(data []byte) (*TransmissionPacket, error)

TransmissionPacketFromJSON deserializes a TransmissionPacket from JSON format

func TransmissionPacketFromSerializable 

func TransmissionPacketFromSerializable(s *SerializableTransmissionPacket) *TransmissionPacket

TransmissionPacketFromSerializable reconstructs a TransmissionPacket from its serializable form

func (*TransmissionPacket) ToCBOR 

func (packet *TransmissionPacket) ToCBOR() ([]byte, error)

ToCBOR serializes a TransmissionPacket to CBOR format

func (*TransmissionPacket) ToJSON 

func (packet *TransmissionPacket) ToJSON() ([]byte, error)

ToJSON serializes a TransmissionPacket to JSON format

func (*TransmissionPacket) ToSerializable 

func (packet *TransmissionPacket) ToSerializable() *SerializableTransmissionPacket

ToSerializable converts a TransmissionPacket to its serializable form

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.