spider

package
v0.52.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 19, 2026 License: Unlicense Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// DirectorySpiderDefaultInterval is how often the directory spider runs
	DirectorySpiderDefaultInterval = 24 * time.Hour
	// DirectorySpiderDefaultMaxHops is the maximum hop distance for relay discovery
	DirectorySpiderDefaultMaxHops = 3
	// DirectorySpiderRelayTimeout is the timeout for connecting to and querying a relay
	DirectorySpiderRelayTimeout = 30 * time.Second
	// DirectorySpiderQueryTimeout is the timeout for waiting for EOSE on a query
	DirectorySpiderQueryTimeout = 60 * time.Second
	// DirectorySpiderRelayDelay is the delay between processing relays (rate limiting)
	DirectorySpiderRelayDelay = 500 * time.Millisecond
	// DirectorySpiderMaxEventsPerQuery is the limit for each query
	DirectorySpiderMaxEventsPerQuery = 5000
)
View Source
const (
	// BatchSize is the number of pubkeys per subscription batch
	BatchSize = 20
	// CatchupWindow is the extra time added to disconnection periods for catch-up
	CatchupWindow = 30 * time.Minute
	// ReconnectDelay is the initial delay between reconnection attempts
	ReconnectDelay = 10 * time.Second
	// MaxReconnectDelay is the maximum delay before switching to blackout
	MaxReconnectDelay = 1 * time.Hour
	// BlackoutPeriod is the duration to blacklist a relay after max backoff is reached
	BlackoutPeriod = 24 * time.Hour
	// BatchCreationDelay is the delay between creating each batch subscription
	BatchCreationDelay = 500 * time.Millisecond
	// RateLimitBackoffDuration is how long to wait when we get a rate limit error
	RateLimitBackoffDuration = 1 * time.Minute
	// RateLimitBackoffMultiplier is the factor by which we increase backoff on repeated rate limits
	RateLimitBackoffMultiplier = 2
	// MaxRateLimitBackoff is the maximum backoff duration for rate limiting
	MaxRateLimitBackoff = 30 * time.Minute
	// MainLoopInterval is how often the spider checks for updates
	MainLoopInterval = 5 * time.Minute
	// EventHandlerBufferSize is the buffer size for event channels
	EventHandlerBufferSize = 100
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BatchSubscription added in v0.17.14

type BatchSubscription struct {
	// contains filtered or unexported fields
}

BatchSubscription represents a subscription for a batch of pubkeys

type DirectorySpider added in v0.31.0

type DirectorySpider struct {
	// contains filtered or unexported fields
}

DirectorySpider manages periodic relay discovery and metadata synchronization. It discovers relays by crawling kind 10002 (relay list) events, expanding outward in hops from seed pubkeys (whitelisted users), then fetches essential metadata events (kinds 0, 3, 10000, 10002) from all discovered relays.

func NewDirectorySpider added in v0.31.0

func NewDirectorySpider(
	ctx context.Context,
	db *database.D,
	pub publisher.I,
	interval time.Duration,
	maxHops int,
) (ds *DirectorySpider, err error)

NewDirectorySpider creates a new DirectorySpider instance.

func (*DirectorySpider) LastRun added in v0.31.0

func (ds *DirectorySpider) LastRun() time.Time

LastRun returns the time of the last completed run.

func (*DirectorySpider) SetSeedCallback added in v0.31.0

func (ds *DirectorySpider) SetSeedCallback(getSeedPubkeys func() [][]byte)

SetSeedCallback sets the callback function for getting seed pubkeys (whitelisted users).

func (*DirectorySpider) Start added in v0.31.0

func (ds *DirectorySpider) Start() (err error)

Start begins the directory spider operation.

func (*DirectorySpider) Stop added in v0.31.0

func (ds *DirectorySpider) Stop()

Stop stops the directory spider operation.

func (*DirectorySpider) TriggerNow added in v0.31.0

func (ds *DirectorySpider) TriggerNow()

TriggerNow forces an immediate run of the directory spider.

type DisconnectionPeriod added in v0.17.14

type DisconnectionPeriod struct {
	Start time.Time
	End   time.Time
}

DisconnectionPeriod tracks when a subscription was disconnected

type RelayConnection added in v0.17.14

type RelayConnection struct {
	// contains filtered or unexported fields
}

RelayConnection manages a single relay connection and its subscriptions

type Spider

type Spider struct {
	// contains filtered or unexported fields
}

Spider manages connections to admin relays and syncs events for followed pubkeys

func New

func New(ctx context.Context, db *database.D, pub publisher.I, mode string) (s *Spider, err error)

New creates a new Spider instance

func (*Spider) NotifyFollowListUpdate added in v0.29.0

func (s *Spider) NotifyFollowListUpdate()

NotifyFollowListUpdate signals the spider that the follow list has been updated

func (*Spider) SetCallbacks added in v0.17.14

func (s *Spider) SetCallbacks(getAdminRelays func() []string, getFollowList func() [][]byte)

SetCallbacks sets the callback functions for getting updated admin relays and follow lists

func (*Spider) Start

func (s *Spider) Start() (err error)

Start begins the spider operation

func (*Spider) Stop

func (s *Spider) Stop()

Stop stops the spider operation

Source Files

  • directory.go
  • spider.go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL