robots

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 26, 2025 License: MIT Imports: 3 Imported by: 0

Documentation

Overview

internal/robots/matcher.go

internal/robots/parser.go

internal/robots/rules.go

Package robots implements a minimal robots.txt parser and matcher. It supports User-agent, Allow and Disallow directives, and computes access rules based on the most specific matching path.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Group 

type Group struct {
	Agents []string
	Rules  []Rule
}

Group represents a User-agent group with associated rules.

type Robots 

type Robots struct {
	Groups []Group
}

Robots is an in-memory representation of a robots.txt file.

func Parse 

func Parse(data []byte) *Robots

Parse constructs a Robots structure from the given robots.txt bytes. The parser is intentionally simple but sufficient for Aether's goal of legal, respectful access.

func (*Robots) Allowed 

func (r *Robots) Allowed(userAgent, path string) bool

Allowed reports whether the given userAgent is allowed to access the specified path according to the robots.txt rules.

It implements the "longest match" rule: the rule with the longest matching path prefix wins. If no rules match, access is allowed.

type Rule 

type Rule struct {
	Allow bool
	Path  string
}

Rule represents a single Allow or Disallow directive.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.