robots

package

v1.0.1 Latest Latest Go to latest Published: Mar 23, 2026 License: MIT Imports: 5 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type Checker struct {
	// contains filtered or unexported fields
}

Checker evaluates robots.txt rules for AI user-agents.

func New(f *fetcher.Fetcher) *Checker

New creates a Checker.

func (c *Checker) IsAllowed(ctx context.Context, rawURL string) bool

IsAllowed checks whether the given URL is allowed by robots.txt. Returns true if allowed (or if robots.txt is absent/unparseable).