Documentation
¶
Overview ¶
Package limits provides request filters and throttles.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DefaultThrottle ¶
type DefaultThrottle struct {
// contains filtered or unexported fields
}
DefaultThrottle will throttle all domains
func NewDefaultThrottle ¶
func NewDefaultThrottle(delay time.Duration) *DefaultThrottle
NewDefaultThrottle will throttle all domains
func (*DefaultThrottle) Applies ¶
func (t *DefaultThrottle) Applies(_ *http.Request) bool
Applies returns true if the path matches the Throttle domain regex
func (*DefaultThrottle) SetWaitTime ¶
func (t *DefaultThrottle) SetWaitTime(waitTime time.Duration)
func (*DefaultThrottle) Wait ¶
func (t *DefaultThrottle) Wait(_ *http.Request)
Wait for the throttle
type DomainThrottle ¶
type DomainThrottle struct {
DefaultThrottle
// contains filtered or unexported fields
}
DomainThrottle will throttle a specific domain.
func NewDomainThrottle ¶
func NewDomainThrottle(domain string, delay time.Duration) *DomainThrottle
NewDomainThrottle instantiates a new domain throttle.
func (*DomainThrottle) Applies ¶
func (t *DomainThrottle) Applies(req *http.Request) bool
Applies returns true if the path matches the Throttle domain.
func (*DomainThrottle) SetWaitTime ¶
func (t *DomainThrottle) SetWaitTime(waitTime time.Duration)
type ForbiddenDomain ¶
ForbiddenDomain indicates a request's URL points to a domain not in the spider's allowed domains.
func (ForbiddenDomain) Error ¶
func (e ForbiddenDomain) Error() string
type MaxDepthFilter ¶
type MaxDepthFilter struct {
MaxDepth int
}
MaxDepthFilter will filter a request if it's depth is larger than the maximum.
func NewMaxDepthFilter ¶
func NewMaxDepthFilter(maxDepth int) *MaxDepthFilter
NewMaxDepthFilter instantiates a new max depth filter.
func (*MaxDepthFilter) FilterRequest ¶
func (m *MaxDepthFilter) FilterRequest(req *request.Request) error
FilterRequest returns an
type MaxDepthReached ¶
MaxDepthReached indicates a request's depth went beyond the maximum and was filtered.
func (*MaxDepthReached) Error ¶
func (m *MaxDepthReached) Error() string
type RequestFilter ¶
type RequestFilter interface {
// FilterRequest filters a request before it is enqueued
FilterRequest(req *request.Request) error
}
RequestFilter is used to filter request before they are enqueued.
type Throttle ¶
type Throttle interface {
// Wait for the throttle.
Wait(*http.Request)
// Applies returns true if the throttle applies to a request.
Applies(*http.Request) bool
// SetWaitTime add a wait time and return the total wait time.
SetWaitTime(time.Duration)
}
Throttle is used to limit the rate of requests.
type ThrottleCollection ¶
type ThrottleCollection struct {
// contains filtered or unexported fields
}
ThrottleCollection combines a default and domain specific throttles.
func NewThrottleCollection ¶
func NewThrottleCollection(defaultThrottle *DefaultThrottle, domainThrottles ...*DomainThrottle) ThrottleCollection
func (*ThrottleCollection) Applies ¶
func (t *ThrottleCollection) Applies(_ *http.Request) bool
Applies returns true if the path matches the Throttle domain regex
func (*ThrottleCollection) SetDomainThrottle ¶
func (t *ThrottleCollection) SetDomainThrottle(throttle *DomainThrottle)
SetDomainThrottle sets a domain throttle. Will overwrite existing domain throttle if it already exists.
func (*ThrottleCollection) SetWaitTime ¶
func (t *ThrottleCollection) SetWaitTime(waitTime time.Duration)
SetWaitTime make all throttles block for a duration.
func (*ThrottleCollection) Wait ¶
func (t *ThrottleCollection) Wait(req *http.Request)
Wait blocks until the most approprate timer has ticked over.
Source Files
Directories