crawler

package

v0.0.0-...-06f47f4 Latest Latest Go to latest Published: Sep 2, 2021 License: MIT Imports: 19 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/wadeAlexC/ipbw

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
Variables
type DHT
- func NewDHT(duration uint, network string) (*DHT, error)
- func (dht *DHT) Start()
type DHTStats
- func NewDHTStats() *DHTStats
type NetworkName
- func FromString(network string) (NetworkName, error)

Constants ¶

View Source

const MAX_ATTEMPTS = 3

MAX_ATTEMPTS is the maximum number of attempts we make to connect to a peer before moving them out of the backlog and into "unreachable."

Attempts are not made sequentially - after a connection attempt fails, we push the peer to the end of the backlog so we can prioritize connecting to peers that have been in the backlog longer.

View Source

const MAX_WORKERS = 1500

MAX_WORKERS is the maximum number of workers we allow

View Source

const MIN_CONNECTED = 500

MIN_CONNECTED is the minimum number of connections we can have before trying to connect to peers in the backlog

View Source

const PORT = 1337

View Source

const USER_AGENT = "IPBW"

Variables ¶

View Source

var FilMainnetBootstrapPeers = []string{
	"/dns4/bootstrap-0.mainnet.filops.net/tcp/1347/p2p/12D3KooWCVe8MmsEMes2FzgTpt9fXtmCY7wrq91GRiaC8PHSCCBj",
	"/dns4/bootstrap-1.mainnet.filops.net/tcp/1347/p2p/12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc",
	"/dns4/bootstrap-2.mainnet.filops.net/tcp/1347/p2p/12D3KooWEWVwHGn2yR36gKLozmb4YjDJGerotAPGxmdWZx2nxMC4",
	"/dns4/bootstrap-3.mainnet.filops.net/tcp/1347/p2p/12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R",
	"/dns4/bootstrap-4.mainnet.filops.net/tcp/1347/p2p/12D3KooWL6PsFNPhYftrJzGgF5U18hFoaVhfGk7xwzD8yVrHJ3Uc",
	"/dns4/bootstrap-5.mainnet.filops.net/tcp/1347/p2p/12D3KooWLFynvDQiUpXoHroV1YxKHhPJgysQGH2k3ZGwtWzR4dFH",
	"/dns4/bootstrap-6.mainnet.filops.net/tcp/1347/p2p/12D3KooWP5MwCiqdMETF9ub1P3MbCvQCcfconnYHbWg6sUJcDRQQ",
	"/dns4/bootstrap-7.mainnet.filops.net/tcp/1347/p2p/12D3KooWRs3aY1p3juFjPy8gPN95PEQChm2QKGUCAdcDCC4EBMKf",
	"/dns4/bootstrap-8.mainnet.filops.net/tcp/1347/p2p/12D3KooWScFR7385LTyR4zU1bYdzSiiAb5rnNABfVahPvVSzyTkR",
	"/dns4/lotus-bootstrap.ipfsforce.com/tcp/41778/p2p/12D3KooWGhufNmZHF3sv48aQeS13ng5XVJZ9E6qy2Ms4VzqeUsHk",
	"/dns4/bootstrap-0.starpool.in/tcp/12757/p2p/12D3KooWGHpBMeZbestVEWkfdnC9u7p6uFHXL1n7m1ZBqsEmiUzz",
	"/dns4/bootstrap-1.starpool.in/tcp/12757/p2p/12D3KooWQZrGH1PxSNZPum99M1zNvjNFM33d1AAu5DcvdHptuU7u",
	"/dns4/node.glif.io/tcp/1235/p2p/12D3KooWBF8cpp65hp2u9LK5mh19x67ftAam84z9LsfaquTDSBpt",
	"/dns4/bootstrap-0.ipfsmain.cn/tcp/34721/p2p/12D3KooWQnwEGNqcM2nAcPtRR9rAX8Hrg4k9kJLCHoTR5chJfz6d",
	"/dns4/bootstrap-1.ipfsmain.cn/tcp/34723/p2p/12D3KooWMKxMkD5DMpSWsW7dBddKxKT7L2GgbNuckz9otxvkvByP",
}

From filecoin-project/lotus/build/bootstrap/mainnet.pi

Functions ¶

This section is empty.

Types ¶

type DHT ¶

type DHT struct {
	*events.Emitter
	// contains filtered or unexported fields
}

func NewDHT ¶

func NewDHT(duration uint, network string) (*DHT, error)

NewDHT creates a new DHT, ready for crawling

duration specifies the number of minutes to crawl for, or "0" for unlimited.

network specifies the network to crawl on, either "filecoin" or "ipfs"

func (*DHT) Start ¶

func (dht *DHT) Start()

Start does three important things: 1. Sets up listeners to manage workload during the crawl 2. Connects to bootstrap peers and begins crawling 3. Prints stats at regular intervals

type DHTStats ¶

type DHTStats struct {
	// contains filtered or unexported fields
}

func NewDHTStats ¶

func NewDHTStats() *DHTStats

type NetworkName ¶

type NetworkName string

const (
	IPFS     NetworkName = "ipfs"
	Filecoin NetworkName = "filecoin"
)

func FromString ¶

func FromString(network string) (NetworkName, error)

func (NetworkName) GetBootstrapPeers ¶

func (nn NetworkName) GetBootstrapPeers() []peer.AddrInfo

func (NetworkName) GetDHTProtocol ¶

func (nn NetworkName) GetDHTProtocol() protocol.ID

func (NetworkName) IsSupported ¶

func (nn NetworkName) IsSupported() bool

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
types

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL