Documentation
¶
Index ¶
- type BasicScheduler
- type BasicSpiderScheduler
- func (b *BasicSpiderScheduler) After(delay time.Duration) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) Delay() time.Duration
- func (b *BasicSpiderScheduler) Duplicate(numGoRoutines int64) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) Every(d time.Duration) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) EveryFunc(fn spider.EveryFunc) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) EveryRandom(d, min, max time.Duration) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) From(from time.Time) spider.SpiderScheduler
- func (b *BasicSpiderScheduler) NextSpin() (time.Duration, bool)
- func (b *BasicSpiderScheduler) NextSpinChan() (<-chan struct{}, <-chan struct{})
- func (b *BasicSpiderScheduler) NumGoroutine() int64
- func (b *BasicSpiderScheduler) To(to time.Time) spider.SpiderScheduler
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BasicScheduler ¶
BasicScheduler is a basic Scheduler that implements the Scheduler interface.
func NewBasicScheduler ¶
func NewBasicScheduler() *BasicScheduler
NewBasicScheduler returns a new BasicScheduler.
func (*BasicScheduler) Handle ¶
func (bs *BasicScheduler) Handle(s spider.Spider) spider.SpiderScheduler
Handle a spider.Spider and returns a new BasicSpiderScheduler associated with this spider.
func (*BasicScheduler) Start ¶
func (bs *BasicScheduler) Start() error
Start starts the Scheduler. It will dispatch each spiders in their own watchers for each duplicated (goroutines).
type BasicSpiderScheduler ¶
type BasicSpiderScheduler struct {
// contains filtered or unexported fields
}
BasicSpiderScheduler represents a schedule configuration for a spider.
func NewBasicSpiderScheduler ¶
func NewBasicSpiderScheduler() *BasicSpiderScheduler
NewBasicSpiderScheduler returns a new BasicSpiderScheduler
func (*BasicSpiderScheduler) After ¶
func (b *BasicSpiderScheduler) After(delay time.Duration) spider.SpiderScheduler
After defines a delay between each goroutines
func (*BasicSpiderScheduler) Delay ¶
func (b *BasicSpiderScheduler) Delay() time.Duration
Delay returns the delay between each goroutines
func (*BasicSpiderScheduler) Duplicate ¶
func (b *BasicSpiderScheduler) Duplicate(numGoRoutines int64) spider.SpiderScheduler
Duplicate sets in how many goroutines the current spider will be duplicated.
func (*BasicSpiderScheduler) Every ¶
func (b *BasicSpiderScheduler) Every(d time.Duration) spider.SpiderScheduler
Every sets a fixed duration representing when to spin.
func (*BasicSpiderScheduler) EveryFunc ¶
func (b *BasicSpiderScheduler) EveryFunc(fn spider.EveryFunc) spider.SpiderScheduler
Every sets a function that returns a duration representing when to spin.
func (*BasicSpiderScheduler) EveryRandom ¶
func (b *BasicSpiderScheduler) EveryRandom(d, min, max time.Duration) spider.SpiderScheduler
Every sets a randomized duration representing when to spin. The second and third argument represents min and max duration for randomness. For exemple:
EveryRandom(2*time.Minute, 15*time.Second, 45*time.Second)
Will represent: 2 minutes + a random duration between 15 seconds and 45 seconds
func (*BasicSpiderScheduler) From ¶
func (b *BasicSpiderScheduler) From(from time.Time) spider.SpiderScheduler
From defines when the scheduled spider has to start
func (*BasicSpiderScheduler) NextSpin ¶
func (b *BasicSpiderScheduler) NextSpin() (time.Duration, bool)
NextSpin returns when a spider has to spin a web based on the provided schedule. The second argument is a boolean that indicates if it there is no more spins.
func (*BasicSpiderScheduler) NextSpinChan ¶
func (b *BasicSpiderScheduler) NextSpinChan() (<-chan struct{}, <-chan struct{})
NextSpinChan returns two channels the first one is a spin channel representing when a spider have to spin a web and the second is a done channel representing when there is webs to be spun.
func (*BasicSpiderScheduler) NumGoroutine ¶
func (b *BasicSpiderScheduler) NumGoroutine() int64
NumGoroutine returns the number of goroutines.
func (*BasicSpiderScheduler) To ¶
func (b *BasicSpiderScheduler) To(to time.Time) spider.SpiderScheduler
To defines when the scheduled spider has to stop
Source Files