Documentation
¶
Index ¶
- Variables
- func FetchGoogleCrawlerIPs(log *slog.Logger, httpClient *http.Client, urls []string) ([]string, error)
- func FetchGooglebotIPs(log *slog.Logger, httpClient *http.Client, url string) ([]string, error)
- func GetDefaultTmpl() string
- func GetPojJS() string
- func IsIpExcluded(clientIP string, exemptIps []*net.IPNet) bool
- func IsIpGoodBot(clientIP string, goodBots []string) bool
- func ParseCIDR(cidr string) (*net.IPNet, error)
- func ReduceCIDRs(cidrs []string, log *slog.Logger) []string
- func RefreshGoogleCrawlerIPs(log *slog.Logger, httpClient *http.Client, target *GooglebotIPs, urls []string) (int, error)
- type GooglebotIPs
Constants ¶
This section is empty.
Variables ¶
var GoogleCrawlerIPRangeURLs = []string{
"https://developers.google.com/static/search/apis/ipranges/googlebot.json",
"https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
"https://developers.google.com/static/crawling/ipranges/special-crawlers.json",
"https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json",
}
Functions ¶
func FetchGoogleCrawlerIPs ¶ added in v1.12.1
func FetchGoogleCrawlerIPs(log *slog.Logger, httpClient *http.Client, urls []string) ([]string, error)
FetchGoogleCrawlerIPs fetches crawler IP ranges from multiple Google-managed endpoints, then returns a canonical, unique list where broader prefixes replace narrower prefixes.
func FetchGooglebotIPs ¶ added in v1.12.0
FetchGooglebotIPs fetches the list of Googlebot IPs from Google's official endpoint, parses the JSON response, and returns a slice of CIDR strings.
func GetDefaultTmpl ¶
func GetDefaultTmpl() string
need to hardcode a default template given yaegi's constraints on finding files on disk provided by this plugin
func GetPojJS ¶ added in v1.11.0
func GetPojJS() string
GetPojJS returns the proof-of-javascript JavaScript implementation
func IsIpGoodBot ¶
func ReduceCIDRs ¶ added in v1.12.1
ReduceCIDRs canonicalizes CIDRs, removes exact duplicates, and removes narrower ranges when they are fully covered by broader ranges.
func RefreshGoogleCrawlerIPs ¶ added in v1.12.1
func RefreshGoogleCrawlerIPs(log *slog.Logger, httpClient *http.Client, target *GooglebotIPs, urls []string) (int, error)
RefreshGoogleCrawlerIPs fetches crawler IPs from all configured URLs and updates the provided GooglebotIPs set. Returns the number of CIDRs loaded.
Types ¶
type GooglebotIPs ¶ added in v1.12.0
type GooglebotIPs struct {
// contains filtered or unexported fields
}
GooglebotIPs holds the list of Googlebot IP ranges, providing a thread-safe way to check if an IP is a Googlebot.
func NewGooglebotIPs ¶ added in v1.12.0
func NewGooglebotIPs() *GooglebotIPs
NewGooglebotIPs creates and initializes a new GooglebotIPs object.
Source Files