README
¶
Website Poller
A simple and lightweight Go package that helps you make recurrent requests to a website of your page.
Overview
The package will poll a website every X seconds or at a random time at each call with a provided list of Headers and a User Agent of your choice. Alternatively, you can provide a list of user agents that can be rotated or chosen randomly each time. A Handler Function of your choice will be executed whenever the requests completes - whether it failed or not.
Features
- Load options from file or define them on your file
- Poll at a fixed time
- Poll at a random time based on a range of seconds to mimick user behavior,
i.e. between
[30 - 50]seconds- Example: first poll after 32 seconds
- second poll after 45 seconds
- third poll after 30 seconds
- fourth poll after 37 seconds
- and so on...
- Provide custom *Headers
- Provide a User Agents list with the ability to either:
- Rotate them at each request
- Pick a random one each time
- Provide no user agents list and let the package choose a random one each time
Limitations and warnings
Remember that if you poll too aggressively you could be probably banned by the website, put behind captchas or exceed quotas to the API service. This package will not prevent you from being banned nor will solve captchas for you. Remember to be polite and respect the rules defined by the website you intend to poll.
The package does not support sending a body with each request yet.
Features that will be introduced on future
- Headers generator to generate headers for every request
- Body generator to generate a different body for every request
- Custom http client
- Custom http request
Install
go get github.com/SunSince90/website-poller
How to use
First of all, import it in your go file:
import (
poller "github.com/SunSince90/website-poller"
)
Then, define a Handler Function that will be called when each request completes.
func handleResponse(id string, resp *http.Response, err error) {
if err != nil {
// handle the error here
}
// Do your stuff here...
}
Define the website to poll:
// Poll a website every 30 seconds
page := &poller.Page {
ID: "github-sunsince90",
URL: "https://api.github.com/users/sunsince90",
}
Finally, start polling:
p := poller.New(page)
p.SetHandlerFunc(handleResponse)
ctx, canc := context.WithCancel(context.Background())
p.Start(ctx)
Examples
The above program will block the main thread, follow the examples contained
in the examples folder to learn more:
- Log: a simple logger
- File: load the pages to poll from a file
- Custom: a more advanced poller with polling options
- Concurrent: how to load multiple pollers and correctly wait for them to finish
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnrecognizedHTTPMethod means that the http method is not recognized ErrUnrecognizedHTTPMethod = errors.New("unrecognized http method") // ErrURLNoScheme means that the url has no http:// or https:// ErrURLNoScheme = errors.New("url does not have a scheme") // ErrInvalidFrequency means that the polling frequency is invalid, i.e. // if it could not be parsed by time.ParseDuration ErrInvalidFrequency = errors.New("invalid polling frequency") // ErrUnsupportedFrequency means that the polling frequency is lower // than five second ErrUnsupportedFrequency = errors.New("frequency cannot be lower than five second") // ErrInvalidRandRange is thrown when the range could not be parsed by // time.ParseDuration or is greater or equal than frequency ErrInvalidRandRange = errors.New("invalid range") )
Functions ¶
This section is empty.
Types ¶
type HandlerFunc ¶
HandlerFunc represents a function that will handle the response returned by the polling.
type Page ¶
type Page struct {
// ID is a short name that will be used by the logs to recognize when
// operations are performed on this page.
ID *string `yaml:"id,omitempty"`
// URL to poll
URL string `yaml:"url"`
// Method of the request, e.g.: GET
Method *string `yaml:"method,omitempty"`
// Headers to send with the request
Headers map[string]string `yaml:"headers,omitempty"`
// UserAgentOptions contains options about the
// user agent
*UserAgentOptions `yaml:"userAgentOptions,omitempty"`
// PollOptions contains options about polling
*PollOptions `yaml:"pollOptions,omitempty"`
// FollowRedirect specifies whether to follow redirects or not.
// Default is false
FollowRedirect bool `yaml:"followRedirect,omitempty"`
}
Page to poll
type PollOptions ¶
type PollOptions struct {
// Frequency of polling in seconds
Frequency int `yaml:"frequency"`
// Randomize specifies whether the next poll should be at a random time or
// at a fixed time
RandomFrequency bool `yaml:"randomFrequency"`
// OffsetRange specifies the range for choosing the next random time.
// For example, if Frequency is 30 and OffsetRange is 10, then each next
// poll will be performed at a random time in the [20, 40] seconds range,
// i.e. 27 seconds.
OffsetRange *int `yaml:"offsetRange,omitempty"`
}
PollOptions contains options about polling
type Poller ¶
type Poller interface {
// Start polling
Start(ctx context.Context, now bool)
// SetHandlerFunc sets the function that will be called when a poll has
// finished
SetHandlerFunc(HandlerFunc)
// GetID returns the ID of this poller. If the `Page` struct provided
// to `New` contained a non-empty `ID`, then this returns the same ID as
// the one contained in there, otherwise it returns a randomly generated
// one.
GetID() string
}
Poller is in charge of polling a website and providing results to a function that will handle the result
type UserAgentOptions ¶
type UserAgentOptions struct {
// UserAgents is a list of user agents that should be used for this page
UserAgents []string `yaml:"userAgents,omitempty"`
// RandomUA specifies whether the user agent for the next request should
// be chosen randomly or should be rotated. This has no effect if
// UserAgents is empty or only has one element. Leave this false if you
// have few user agents.
RandomUA bool `yaml:"randomUA"`
}
UserAgentOptions contains options about the user agent
Source Files