storage

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 24, 2019 License: GPL-3.0 Imports: 28 Imported by: 0

Documentation

Overview

Copyright 2016 The go-ethereum Authors This file is part of the go-ethereum library.

The go-ethereum library is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

The go-ethereum library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.

Index

Constants

View Source 
const (
	ErrInit = iota
	ErrNotFound
	ErrUnauthorized
	ErrInvalidValue
	ErrDataOverflow
	ErrNothingToReturn
	ErrInvalidSignature
	ErrNotSynced
)
View Source 
const (
	BMTHash     = "BMT"
	SHA3Hash    = "SHA3" // http://golang.org/pkg/hash/#Hash
	DefaultHash = BMTHash
)
View Source 
const AddressLength = chunk.AddressLength

AddressLength is the same as chunk.AddressLength for backward compatibility.

View Source 
const (
	ChunkProcessors = 8
)
View Source 
const MaxPO = chunk.MaxPO

MaxPO is the same as chunk.MaxPO for backward compatibility.

Variables

View Source 
var (
	ErrChunkNotFound = chunk.ErrChunkNotFound
	ErrChunkInvalid  = chunk.ErrChunkNotFound
)

Errors are the same as the ones in chunk package for backward compatibility.

View Source 
var NewChunk = chunk.NewChunk

NewChunk is the same as chunk.NewChunk for backward compatibility.

View Source 
var Proximity = chunk.Proximity

Proximity is the same as chunk.Proximity for backward compatibility.

View Source 
var ZeroAddr = chunk.ZeroAddr

ZeroAddr is the same as chunk.ZeroAddr for backward compatibility.

Functions

func NewHasherStore added in v1.8.12 

func NewHasherStore(store ChunkStore, hashFunc SwarmHasher, toEncrypt bool, tag *chunk.Tag) *hasherStore

NewHasherStore creates a hasherStore object, which implements Putter and Getter interfaces. With the HasherStore you can put and get chunk data (which is just []byte) into a ChunkStore and the hasherStore will take core of encryption/decryption of data if necessary

Types

type Address added in v1.8.12 

type Address = chunk.Address

Address is an alias for chunk.Address for backward compatibility.

func PyramidAppend added in v1.8.12 

func PyramidAppend(ctx context.Context, addr Address, reader io.Reader, putter Putter, getter Getter, tag *chunk.Tag) (Address, func(context.Context) error, error)

func PyramidSplit added in v1.8.12 

func PyramidSplit(ctx context.Context, reader io.Reader, putter Putter, getter Getter, tag *chunk.Tag) (Address, func(context.Context) error, error)

When splitting, data is given as a SectionReader, and the key is a hashSize long byte slice (Address), the root hash of the entire content will fill this once processing finishes. New chunks to store are store using the putter which the caller provides.

func TreeSplit added in v1.8.12 

func TreeSplit(ctx context.Context, data io.Reader, size int64, putter Putter) (k Address, wait func(context.Context) error, err error)

When splitting, data is given as a SectionReader, and the key is a hashSize long byte slice (Key), the root hash of the entire content will fill this once processing finishes. New chunks to store are store using the putter which the caller provides.

type AddressCollection added in v1.8.12 

type AddressCollection []Address

func NewAddressCollection added in v1.8.12 

func NewAddressCollection(l int) AddressCollection

func (AddressCollection) Len added in v1.8.12 

func (c AddressCollection) Len() int

func (AddressCollection) Less added in v1.8.12 

func (c AddressCollection) Less(i, j int) bool

func (AddressCollection) Swap added in v1.8.12 

func (c AddressCollection) Swap(i, j int)

type Chunk 

type Chunk = chunk.Chunk

Chunk is an alias for chunk.Chunk for backward compatibility.

func GenerateRandomChunk added in v1.8.12 

func GenerateRandomChunk(dataSize int64) Chunk

func GenerateRandomChunks added in v1.8.12 

func GenerateRandomChunks(dataSize int64, count int) (chunks []Chunk)

type ChunkData added in v1.8.12 

type ChunkData []byte

func (ChunkData) Size added in v1.8.12 

func (c ChunkData) Size() uint64

NOTE: this returns invalid data if chunk is encrypted

type ChunkStore 

type ChunkStore = chunk.Store

type ChunkValidator added in v1.8.12 

type ChunkValidator = chunk.Validator

type ChunkerParams 

type ChunkerParams struct {
	// contains filtered or unexported fields
}

type ContentAddressValidator added in v1.8.12 

type ContentAddressValidator struct {
	Hasher SwarmHasher
}

Provides method for validation of content address in chunks Holds the corresponding hasher to create the address

func NewContentAddressValidator added in v1.8.12 

func NewContentAddressValidator(hasher SwarmHasher) *ContentAddressValidator

Constructor

func (*ContentAddressValidator) Validate added in v1.8.12 

func (v *ContentAddressValidator) Validate(ch Chunk) bool

Validate that the given key is a valid content address for the given data

type FakeChunkStore added in v1.8.20 

type FakeChunkStore struct {
}

FakeChunkStore doesn't store anything, just implements the ChunkStore interface It can be used to inject into a hasherStore if you don't want to actually store data just do the hashing

func (*FakeChunkStore) Close added in v1.8.20 

func (f *FakeChunkStore) Close() error

Close doesn't store anything it is just here to implement ChunkStore

func (*FakeChunkStore) Get added in v1.8.20 

func (f *FakeChunkStore) Get(_ context.Context, _ chunk.ModeGet, ref Address) (Chunk, error)

Get doesn't store anything it is just here to implement ChunkStore

func (*FakeChunkStore) Has added in v1.8.23 

func (f *FakeChunkStore) Has(_ context.Context, ref Address) (bool, error)

Has doesn't do anything it is just here to implement ChunkStore

func (*FakeChunkStore) LastPullSubscriptionBinID added in v1.9.0 

func (f *FakeChunkStore) LastPullSubscriptionBinID(bin uint8) (id uint64, err error)

func (*FakeChunkStore) Put added in v1.8.20 

func (f *FakeChunkStore) Put(_ context.Context, _ chunk.ModePut, ch Chunk) (bool, error)

Put doesn't store anything it is just here to implement ChunkStore

func (*FakeChunkStore) Set added in v1.9.0 

func (f *FakeChunkStore) Set(ctx context.Context, mode chunk.ModeSet, addr chunk.Address) (err error)

func (*FakeChunkStore) SubscribePull added in v1.9.0 

func (f *FakeChunkStore) SubscribePull(ctx context.Context, bin uint8, since, until uint64) (c <-chan chunk.Descriptor, stop func())

type FileStore added in v1.8.12 

type FileStore struct {
	ChunkStore
	// contains filtered or unexported fields
}

func NewFileStore added in v1.8.12 

func NewFileStore(store ChunkStore, params *FileStoreParams, tags *chunk.Tags) *FileStore

func NewLocalFileStore added in v1.8.12 

func NewLocalFileStore(datadir string, basekey []byte, tags *chunk.Tags) (*FileStore, error)

for testing locally

func (*FileStore) GetAllReferences added in v1.8.23 

func (f *FileStore) GetAllReferences(ctx context.Context, data io.Reader, toEncrypt bool) (addrs AddressCollection, err error)

GetAllReferences is a public API. This endpoint returns all chunk hashes (only) for a given file

func (*FileStore) HashSize added in v1.8.12 

func (f *FileStore) HashSize() int

func (*FileStore) Retrieve added in v1.8.12 

func (f *FileStore) Retrieve(ctx context.Context, addr Address) (reader *LazyChunkReader, isEncrypted bool)

Retrieve is a public API. Main entry point for document retrieval directly. Used by the FS-aware API and httpaccess Chunk retrieval blocks on netStore requests with a timeout so reader will report error if retrieval of chunks within requested range time out. It returns a reader with the chunk data and whether the content was encrypted

func (*FileStore) Store added in v1.8.12 

func (f *FileStore) Store(ctx context.Context, data io.Reader, size int64, toEncrypt bool) (addr Address, wait func(context.Context) error, err error)

Store is a public API. Main entry point for document storage directly. Used by the FS-aware API and httpaccess

type FileStoreParams added in v1.8.12 

type FileStoreParams struct {
	Hash string
}

func NewFileStoreParams added in v1.8.12 

func NewFileStoreParams() *FileStoreParams

type Getter added in v1.8.12 

type Getter interface {
	Get(context.Context, Reference) (ChunkData, error)
}

Getter is an interface to retrieve a chunk's data by its reference

type HashWithLength 

type HashWithLength struct {
	hash.Hash
}

func (*HashWithLength) ResetWithLength 

func (h *HashWithLength) ResetWithLength(length []byte)

type JoinerParams added in v1.8.12 

type JoinerParams struct {
	ChunkerParams
	// contains filtered or unexported fields
}

type LazyChunkReader 

type LazyChunkReader struct {
	// contains filtered or unexported fields
}

LazyChunkReader implements LazySectionReader

func TreeJoin added in v1.8.12 

func TreeJoin(ctx context.Context, addr Address, getter Getter, depth int) *LazyChunkReader

Join reconstructs original content based on a root key. When joining, the caller gets returned a Lazy SectionReader, which is seekable and implements on-demand fetching of chunks as and where it is read. New chunks to retrieve are coming from the getter, which the caller provides. If an error is encountered during joining, it appears as a reader error. The SectionReader. As a result, partial reads from a document are possible even if other parts are corrupt or lost. The chunks are not meant to be validated by the chunker when joining. This is because it is left to the DPA to decide which sources are trusted.

func (*LazyChunkReader) Context added in v1.8.13 

func (r *LazyChunkReader) Context() context.Context

func (*LazyChunkReader) Read 

func (r *LazyChunkReader) Read(b []byte) (read int, err error)

Read keeps a cursor so cannot be called simulateously, see ReadAt

func (*LazyChunkReader) ReadAt 

func (r *LazyChunkReader) ReadAt(b []byte, off int64) (read int, err error)

read at can be called numerous times concurrent reads are allowed Size() needs to be called synchronously on the LazyChunkReader first

func (*LazyChunkReader) Seek 

func (r *LazyChunkReader) Seek(offset int64, whence int) (int64, error)

func (*LazyChunkReader) Size 

func (r *LazyChunkReader) Size(ctx context.Context, quitC chan bool) (n int64, err error)

Size is meant to be called on the LazySectionReader

type LazySectionReader 

type LazySectionReader interface {
	Context() context.Context
	Size(context.Context, chan bool) (int64, error)
	io.Seeker
	io.Reader
	io.ReaderAt
}

Size, Seek, Read, ReadAt

type LazyTestSectionReader 

type LazyTestSectionReader struct {
	*io.SectionReader
}

func (*LazyTestSectionReader) Context added in v1.8.13 

func (r *LazyTestSectionReader) Context() context.Context

func (*LazyTestSectionReader) Size 

func (r *LazyTestSectionReader) Size(context.Context, chan bool) (int64, error)

type NetFetcher added in v1.8.20 

type NetFetcher interface {
	Request(hopCount uint8)
	Offer(source *enode.ID)
}

type NetStore 

type NetStore struct {
	chunk.Store

	NewNetFetcherFunc NewNetFetcherFunc
	// contains filtered or unexported fields
}

NetStore is an extension of local storage it implements the ChunkStore interface on request it initiates remote cloud retrieval using a fetcher fetchers are unique to a chunk and are stored in fetchers LRU memory cache fetchFuncFactory is a factory object to create a fetch function for a specific chunk address

func NewNetStore 

func NewNetStore(store chunk.Store, nnf NewNetFetcherFunc) (*NetStore, error)

NewNetStore creates a new NetStore object using the given local store. newFetchFunc is a constructor function that can create a fetch function for a specific chunk address.

func (*NetStore) Close 

func (n *NetStore) Close() (err error)

Close chunk store

func (*NetStore) FetchFunc added in v1.8.20 

func (n *NetStore) FetchFunc(ctx context.Context, ref Address) func(context.Context) error

FetchFunc returns nil if the store contains the given address. Otherwise it returns a wait function, which returns after the chunk is available or the context is done

func (*NetStore) Get 

func (n *NetStore) Get(rctx context.Context, mode chunk.ModeGet, ref Address) (Chunk, error)

Get retrieves the chunk from the NetStore DPA synchronously. It calls NetStore.get, and if the chunk is not in local Storage it calls fetch with the request, which blocks until the chunk arrived or context is done

func (*NetStore) Put 

func (n *NetStore) Put(ctx context.Context, mode chunk.ModePut, ch Chunk) (bool, error)

Put stores a chunk in localstore, and delivers to all requestor peers using the fetcher stored in the fetchers cache

func (*NetStore) RequestsCacheLen added in v1.8.20 

func (n *NetStore) RequestsCacheLen() int

RequestsCacheLen returns the current number of outgoing requests stored in the cache

type NewNetFetcherFunc added in v1.8.20 

type NewNetFetcherFunc func(ctx context.Context, addr Address, peers *sync.Map) NetFetcher

type Putter added in v1.8.12 

type Putter interface {
	Put(context.Context, ChunkData) (Reference, error)
	// RefSize returns the length of the Reference created by this Putter
	RefSize() int64
	// Close is to indicate that no more chunk data will be Put on this Putter
	Close()
	// Wait returns if all data has been store and the Close() was called.
	Wait(context.Context) error
}

Putter is responsible to store data and create a reference for it

type PyramidChunker 

type PyramidChunker struct {
	// contains filtered or unexported fields
}

func NewPyramidSplitter added in v1.8.12 

func NewPyramidSplitter(params *PyramidSplitterParams, tag *chunk.Tag) (pc *PyramidChunker)

func (*PyramidChunker) Append 

func (pc *PyramidChunker) Append(ctx context.Context) (k Address, wait func(context.Context) error, err error)

func (*PyramidChunker) Join 

func (pc *PyramidChunker) Join(addr Address, getter Getter, depth int) LazySectionReader

func (*PyramidChunker) Split 

func (pc *PyramidChunker) Split(ctx context.Context) (k Address, wait func(context.Context) error, err error)

type PyramidSplitterParams added in v1.8.12 

type PyramidSplitterParams struct {
	SplitterParams
	// contains filtered or unexported fields
}

func NewPyramidSplitterParams added in v1.8.12 

func NewPyramidSplitterParams(addr Address, reader io.Reader, putter Putter, getter Getter, chunkSize int64) *PyramidSplitterParams

type Reference added in v1.8.12 

type Reference []byte

type SplitterParams added in v1.8.12 

type SplitterParams struct {
	ChunkerParams
	// contains filtered or unexported fields
}

type StoreParams 

type StoreParams struct {
	Hash          SwarmHasher `toml:"-"`
	DbCapacity    uint64
	CacheCapacity uint
	BaseKey       []byte
}

func NewDefaultStoreParams 

func NewDefaultStoreParams() *StoreParams

func NewStoreParams added in v1.8.12 

func NewStoreParams(ldbCap uint64, cacheCap uint, hash SwarmHasher, basekey []byte) *StoreParams

type SwarmHash 

type SwarmHash interface {
	hash.Hash
	ResetWithLength([]byte)
}

type SwarmHasher 

type SwarmHasher func() SwarmHash

func MakeHashFunc 

func MakeHashFunc(hash string) SwarmHasher

type TreeChunker 

type TreeChunker struct {
	// contains filtered or unexported fields
}

func NewTreeJoiner added in v1.8.12 

func NewTreeJoiner(params *JoinerParams) *TreeChunker

func NewTreeSplitter added in v1.8.12 

func NewTreeSplitter(params *TreeSplitterParams) *TreeChunker

func (*TreeChunker) Join 

func (tc *TreeChunker) Join(ctx context.Context) *LazyChunkReader

func (*TreeChunker) Split 

func (tc *TreeChunker) Split(ctx context.Context) (k Address, wait func(context.Context) error, err error)

type TreeEntry 

type TreeEntry struct {
	// contains filtered or unexported fields
}

Entry to create a tree node

func NewTreeEntry 

func NewTreeEntry(pyramid *PyramidChunker) *TreeEntry

type TreeSplitterParams added in v1.8.12 

type TreeSplitterParams struct {
	SplitterParams
	// contains filtered or unexported fields
}

Source Files

View all Source files

Directories

Path Synopsis
Package feeds defines Swarm Feeds.
 Package feeds defines Swarm Feeds.
lookup
Package lookup defines feed lookup algorithms and provides tools to place updates so they can be found
 Package lookup defines feed lookup algorithms and provides tools to place updates so they can be found
Package localstore provides disk storage layer for Swarm Chunk persistence.
 Package localstore provides disk storage layer for Swarm Chunk persistence.
Package mock defines types that are used by different implementations of mock storages.
 Package mock defines types that are used by different implementations of mock storages.
db
Package db implements a mock store that keeps all chunk data in LevelDB database.
 Package db implements a mock store that keeps all chunk data in LevelDB database.
explorer
mem
Package mem implements a mock store that keeps all chunk data in memory.
 Package mem implements a mock store that keeps all chunk data in memory.
rpc
Package rpc implements an RPC client that connect to a centralized mock store.
 Package rpc implements an RPC client that connect to a centralized mock store.
test
Package test provides functions that are used for testing GlobalStorer implementations.
 Package test provides functions that are used for testing GlobalStorer implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.