autoretrieve

package
v0.1.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2022 License: Apache-2.0, MIT Imports: 42 Imported by: 0

Documentation

Index

Constants

View Source 
const QueryNewCIDs string = "" /* 156-byte string literal not displayed */

Variables

This section is empty.

Functions

func ValidatePeerInfo 

func ValidatePeerInfo(pubKeyStr string, addresses []string) (*peer.AddrInfo, error)

Types

type Autoretrieve 

type Autoretrieve struct {
	gorm.Model

	Handle            string `gorm:"unique"`
	Token             string `gorm:"unique"`
	LastConnection    time.Time
	LastAdvertisement time.Time
	PubKey            string `gorm:"unique"`
	Addresses         string
}

type AutoretrieveEngine 

type AutoretrieveEngine struct {
	TickInterval time.Duration
	// contains filtered or unexported fields
}

Engine is an implementation of the core reference provider interface

func New 

func New(o ...Option) (*AutoretrieveEngine, error)

New creates a new index provider Engine as the default implementation of provider.Interface. It provides the ability to advertise the availability of a list of multihashes associated to a context ID as a chain of linked advertisements as defined by the indexer node protocol implemented by "storetheindex". Engine internally uses "go-legs", a protocol for propagating and synchronizing changes an IPLD DAG, to publish advertisements. See:

Published advertisements are signed using the given private key. The retAddrs corresponds to the endpoints at which the data block associated to the advertised multihashes can be retrieved. Note that if no retAddrs is specified the listen addresses of the given libp2p host are used.

The engine also provides the ability to generate advertisements via Engine.NotifyPut and Engine.NotifyRemove as long as a provider.MultihashLister is registered. See: provider.MultihashLister, Engine.RegisterMultihashLister.

The engine must be started via Engine.Start before use and discarded via Engine.Shutdown when no longer needed. See: Engine.Start, Engine.Shutdown.

func NewAutoretrieveEngine 

func NewAutoretrieveEngine(ctx context.Context, cfg *config.Estuary, db *gorm.DB, libp2pHost host.Host, ds datastore.Batching) (*AutoretrieveEngine, error)

newIndexProvider creates a new index-provider engine to send announcements to storetheindex this needs to keep running continuously because storetheindex will come to fetch advertisements "when it feels like it"

func (*AutoretrieveEngine) GetAdv 

func (e *AutoretrieveEngine) GetAdv(_ context.Context, adCid cid.Cid) (*schema.Advertisement, error)

GetAdv gets the advertisement associated to the given cid c. The context is not used.

func (*AutoretrieveEngine) GetLatestAdv 

func (e *AutoretrieveEngine) GetLatestAdv(ctx context.Context) (cid.Cid, *schema.Advertisement, error)

GetLatestAdv gets the latest advertisement by the provider. If there are not previously published advertisements, then cid.Undef is returned as the advertisement CID.

func (*AutoretrieveEngine) NotifyPut 

func (e *AutoretrieveEngine) NotifyPut(ctx context.Context, contextID []byte, providerID string, addresses []string, md metadata.Metadata) (cid.Cid, error)

NotifyPut publishes an advertisement that signals the list of multihashes associated to the given contextID is available by this provider with the given metadata. A provider.MultihashLister is required, and is used to look up the list of multihashes associated to a context ID.

Note that prior to calling this function a provider.MultihashLister must be registered.

See: AutoretrieveEngine.RegisterMultihashLister, AutoretrieveEngine.Publish.

func (*AutoretrieveEngine) NotifyRemove 

func (e *AutoretrieveEngine) NotifyRemove(ctx context.Context, contextID []byte) (cid.Cid, error)

NotifyRemove publishes an advertisement that signals the list of multihashes associated to the given contextID is no longer available by this provider.

Note that prior to calling this function a provider.MultihashLister must be registered.

See: AutoretrieveEngine.RegisterMultihashLister, AutoretrieveEngine.Publish.

func (*AutoretrieveEngine) Publish 

func (e *AutoretrieveEngine) Publish(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)

Publish stores the given advertisement locally via AutoretrieveEngine.PublishLocal first, then publishes a message onto the gossipsub to signal the change in the latest advertisement by the provider to indexer nodes.

The publication mechanism uses legs.Publisher internally. See: https://github.com/filecoin-project/go-legs

func (*AutoretrieveEngine) PublishLocal 

func (e *AutoretrieveEngine) PublishLocal(ctx context.Context, adv schema.Advertisement) (cid.Cid, error)

PublishLocal stores the advertisement in the local link system and marks it locally as the latest advertisement.

The context is used for storing internal mapping information onto the datastore.

See: AutoretrieveEngine.Publish.

func (*AutoretrieveEngine) RegisterMultihashLister 

func (e *AutoretrieveEngine) RegisterMultihashLister(mhl provider.MultihashLister)

RegisterMultihashLister registers a provider.MultihashLister that is used to look up the list of multihashes associated to a context ID. At least one such registration must be registered before calls to AutoretrieveEngine.NotifyPut and AutoretrieveEngine.NotifyRemove.

Note that successive calls to this function will replace the previous registration. Only a single registration is supported.

See: provider.Interface

func (*AutoretrieveEngine) Run 

func (arEng *AutoretrieveEngine) Run()

func (*AutoretrieveEngine) Shutdown 

func (e *AutoretrieveEngine) Shutdown() error

Shutdown shuts down the engine and discards all resources opened by the engine. The engine is no longer usable after the call to this function.

func (*AutoretrieveEngine) Start 

func (e *AutoretrieveEngine) Start(ctx context.Context) error

Start starts the engine by instantiating the internal storage and joins the configured gossipsub topic used for publishing advertisements.

The context is used to instantiate the internal LRU cache storage.

See: AutoretrieveEngine.Shutdown, chunker.NewCachedEntriesChunker, dtsync.NewPublisherFromExisting.

type AutoretrieveInitResponse 

type AutoretrieveInitResponse struct {
	Handle            string         `json:"handle"`
	Token             string         `json:"token"`
	LastConnection    time.Time      `json:"lastConnection"`
	AddrInfo          *peer.AddrInfo `json:"addrInfo"`
	AdvertiseInterval string         `json:AdvertiseInterval`
}

type AutoretrieveListResponse 

type AutoretrieveListResponse struct {
	Handle            string         `json:"handle"`
	LastConnection    time.Time      `json:"lastConnection"`
	LastAdvertisement time.Time      `json:"lastAdvertisement"`
	AddrInfo          *peer.AddrInfo `json:"addrInfo"`
}

type EstuaryMhIterator 

type EstuaryMhIterator struct {
	Cids []cid.Cid
	// contains filtered or unexported fields
}

EstuaryMhIterator contains objects to query the database incrementally, to avoid having all CIDs in memory at once

func (*EstuaryMhIterator) Next 

func (m *EstuaryMhIterator) Next() (multihash.Multihash, error)

type HeartbeatAutoretrieveResponse 

type HeartbeatAutoretrieveResponse struct {
	Handle            string         `json:"handle"`
	LastConnection    time.Time      `json:"lastConnection"`
	LastAdvertisement time.Time      `json:"lastAdvertisement"`
	AddrInfo          *peer.AddrInfo `json:"addrInfo"`
	AdvertiseInterval string         `json:AdvertiseInterval`
}

type Option 

type Option func(*options) error

Option sets a configuration parameter for the provider engine.

func WithDataTransfer 

func WithDataTransfer(dt datatransfer.Manager) Option

WithDataTransfer sets the instance of datatransfer.Manager to use. If unspecified a new instance is created automatically.

Note that this option only takes effect if the PublisherKind is set to DataTransferPublisher. See: WithPublisherKind.

func WithDatastore 

func WithDatastore(ds datastore.Batching) Option

WithDatastore sets the datastore that is used by the engine to store advertisements. If unspecified, an ephemeral in-memory datastore is used. See: datastore.NewMapDatastore.

func WithDirectAnnounce 

func WithDirectAnnounce(announceURLs ...string) Option

WithDirectAnnounce sets indexer URLs to send direct HTTP announcements to.

func WithEntriesCacheCapacity 

func WithEntriesCacheCapacity(s int) Option

WithEntriesCacheCapacity sets the maximum number of advertisement entries chains to cache. If unset, the default capacity of 1024 is used.

The cache is evicted using LRU policy. Note that the capacity dictates the number of complete chains that are cached, not individual entry chunks. This means, the maximum storage used by the cache is a factor of capacity, chunk size and the length of multihashes in each chunk.

As an example, for 128-bit long multihashes the cache with default capacity of 1024, and default chunk size of 16384 can grow up to 256MiB when full.

See: WithEntriesChunkSize, chunker.CachedEntriesChunker.

func WithEntriesChunkSize 

func WithEntriesChunkSize(s int) Option

WithEntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. If unset, the default size of 16384 is used.

See: WithEntriesCacheCapacity, chunker.CachedEntriesChunker

func WithExtraGossipData 

func WithExtraGossipData(extraData []byte) Option

WithExtraGossipData supplies extra data to include in the pubsub announcement. Note that this option only takes effect if the PublisherKind is set to DataTransferPublisher. See: WithPublisherKind.

func WithHost 

func WithHost(h host.Host) Option

WithHost specifies the host to which the provider engine belongs. If unspecified, a host is created automatically. See: libp2p.New.

func WithHttpPublisherListenAddr 

func WithHttpPublisherListenAddr(addr string) Option

WithHttpPublisherListenAddr sets the net listen address for the HTTP publisher. If unset, the default net listen address of '0.0.0.0:3104' is used.

Note that this option only takes effect if the PublisherKind is set to HttpPublisher. See: WithPublisherKind.

func WithProvider 

func WithProvider(provider peer.AddrInfo) Option

WithProvider sets the peer and addresses for the provider to put in indexing advertisements. This value overrides `WithRetrievalAddrs`

func WithPublisherKind 

func WithPublisherKind(k PublisherKind) Option

WithPublisherKind sets the kind of publisher used to announce new advertisements. If unset, advertisements are only stored locally and no announcements are made. See: PublisherKind.

func WithPurgeCacheOnStart 

func WithPurgeCacheOnStart(p bool) Option

WithPurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine starts. If unset, cache is rehydrated from previously cached entries stored in datastore if present. See: WithDatastore.

func WithRetrievalAddrs 

func WithRetrievalAddrs(addr ...multiaddr.Multiaddr) Option

WithRetrievalAddrs sets the addresses that specify where to get the content corresponding to an indexing advertisement. If unspecified, the libp2p host listen addresses are used. See: WithHost.

func WithTopic 

func WithTopic(t *pubsub.Topic) Option

WithTopic sets the pubsub topic on which new advertisements are announced. To use the default pubsub configuration with a specific topic name, use WithTopicName. If both options are specified, WithTopic takes presence.

Note that this option only takes effect if the PublisherKind is set to DataTransferPublisher. See: WithPublisherKind.

func WithTopicName 

func WithTopicName(t string) Option

WithTopicName sets the topic name on which pubsub announcements are published. To override the default pubsub configuration, use WithTopic.

Note that this option only takes effect if the PublisherKind is set to DataTransferPublisher. See: WithPublisherKind.

type PublisherKind 

type PublisherKind string

PublisherKind represents the kind of publisher to use in order to announce a new advertisement to the network. See: WithPublisherKind, NoPublisher, DataTransferPublisher, HttpPublisher. 

const (
	// NoPublisher indicates that no announcements are made to the network and all advertisements
	// are only stored locally.
	NoPublisher PublisherKind = ""

	// DataTransferPublisher makes announcements over a gossipsub topic and exposes a
	// datatransfer/graphsync server that allows peers in the network to sync advertisements.
	DataTransferPublisher PublisherKind = "dtsync"

	// HttpPublisher exposes a HTTP server that announces published advertisements and allows peers
	// in the network to sync them over raw HTTP transport.
	HttpPublisher PublisherKind = "http"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.