Help us shape the future of Go! Take the 2026 Go Developer Survey

agents

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 28, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package agents is the single source of truth for known AI crawlers and assistants: who they are, what they are for, and (in later phases) how to verify them. The registry is embedded JSON so the frontend generator and the Go side read the same file.

Index

Constants

View Source 
const (
	SourceMiddleware = "middleware" // Next.js middleware / robots.txt route ping
	SourceBackend    = "backend"    // captured by the Go API itself
	SourceLog        = "log"        // access-log ingest
)

Capture sources, as stored in hatamoto_agent_hits.source.

View Source 
const (
	VerificationPending      = "pending"
	VerificationVerified     = "verified"
	VerificationSpoofed      = "spoofed"
	VerificationUnverifiable = "unverifiable"
)

Verification states, as stored in hatamoto_agent_hits.verification.

View Source 
const (
	RobotsAllow = "allow"
	RobotsDeny  = "deny"
)
View Source 
const (

	// PendingBatch is the maximum number of (agent, ip) pairs per run.
	PendingBatch = 500
)

Variables

This section is empty.

Functions

func DenyTokens 

func DenyTokens(reg *Registry, denied []Purpose) []string

DenyTokens returns the distinct robots.txt user-agent tokens of every registry agent whose purpose is denied, sorted. Agents without a robots_token are skipped. Token-only entries (Google-Extended) count: they are exactly what robots.txt is for.

func ParsePrefixes 

func ParsePrefixes(body []byte) ([]netip.Prefix, error)

ParsePrefixes reads an operator range document. Accepted shapes:

  • googlebot style: {"prefixes":[{"ipv4Prefix":"..."}|{"ipv6Prefix":"..."}]}
  • {"ips":[ "1.2.3.0/24" | {"ip_address":"..."} ... ]} (Ahrefs-like)
  • a plain JSON array of strings

Bare addresses become /32 or /128. Unparseable entries are skipped; a document with no usable prefix is an error so the caller keeps old data.

func PublicPath 

func PublicPath(p string) (string, bool)

PublicPath maps a backend request path to the public path a crawler asked for, and reports whether it is a crawler-facing file worth recording (sitemaps and llms.txt). Everything else under /api/v1/public (notably /resolve and /page/*, which Next.js calls server-side with its own UA) is not.

Types

type AIReferrer 

type AIReferrer struct {
	Source string   `json:"source"`
	Hosts  []string `json:"hosts"`
	UTM    []string `json:"utm"`
}

AIReferrer describes an AI assistant whose outbound links carry a referrer host or utm_source value.

type AdminHandlers 

type AdminHandlers struct {
	// contains filtered or unexported fields
}

AdminHandlers serves /api/v1/agent-analytics/*. The server mounts it in the authenticated admin group. No response type here carries an IP field; the spoof log exposes a masked network (ip_masked) only.

func NewAdminHandlers 

func NewAdminHandlers(store AdminStore, reg *Registry, stats func() WriterStats) *AdminHandlers

NewAdminHandlers creates the admin handlers. stats may be nil (dropped_hits then reports 0).

func (*AdminHandlers) HandleAgentDetail 

func (h *AdminHandlers) HandleAgentDetail(w http.ResponseWriter, r *http.Request)

HandleAgentDetail serves GET /agents/{id}?from&to. 404 for an id that is neither in the registry nor in the rollup.

func (*AdminHandlers) HandleAgents 

func (h *AdminHandlers) HandleAgents(w http.ResponseWriter, r *http.Request)

HandleAgents serves GET /agents?from&to: every agent seen in the range.

func (*AdminHandlers) HandleEntityActivity 

func (h *AdminHandlers) HandleEntityActivity(w http.ResponseWriter, r *http.Request)

HandleEntityActivity serves GET /entities/{id}: the editor's AI Agents card. Every AI agent in the registry appears, with last_fetch_at null for "never". first_crawled_after_publish_seconds is null when there is no crawl yet, no publish time, the crawl predates the (re)publish, or the entity was published before tracking began (its first crawl may long predate the tracker). first_crawl_at is likewise null for entities published before tracking began: the recorded row is only the first crawl since deploy, not the real first crawl, so the card falls back to the newest per-agent fetch.

func (*AdminHandlers) HandleErrors 

func (h *AdminHandlers) HandleErrors(w http.ResponseWriter, r *http.Request)

HandleErrors serves GET /errors?from&to&page&per_page: agent fetches that got 4xx/5xx, grouped by path and status.

func (*AdminHandlers) HandleGetRobotsPolicy 

func (h *AdminHandlers) HandleGetRobotsPolicy(w http.ResponseWriter, r *http.Request)

HandleGetRobotsPolicy serves GET /robots-policy: one row per purpose with the stored action ("allow" when nothing is stored), the robots.txt tokens it controls and the agents known to ignore robots.txt.

func (*AdminHandlers) HandleOverview 

func (h *AdminHandlers) HandleOverview(w http.ResponseWriter, r *http.Request)

HandleOverview serves GET /overview?from&to&group=operator|purpose. Counts cover AI purposes only (training, search_index, user_fetch, other_ai).

func (*AdminHandlers) HandlePages 

func (h *AdminHandlers) HandlePages(w http.ResponseWriter, r *http.Request)

HandlePages serves GET /pages?from&to&page&per_page: top fetched pages in the range and published entities never fetched by any AI agent (all time). page/per_page apply to both lists.

func (*AdminHandlers) HandlePutRobotsPolicy 

func (h *AdminHandlers) HandlePutRobotsPolicy(w http.ResponseWriter, r *http.Request)

HandlePutRobotsPolicy serves PUT /robots-policy with {"purposes":[{"purpose","action"}]}. It replaces the policy: purposes with action "deny" are stored, everything else (including purposes omitted from the body) is deleted. 400 on an unknown purpose, an action other than allow/deny, or a repeated purpose.

func (*AdminHandlers) HandleSpoofed 

func (h *AdminHandlers) HandleSpoofed(w http.ResponseWriter, r *http.Request)

HandleSpoofed serves GET /spoofed?from&to&page&per_page: the spoof log with the network (/24 or /48) of each offending address, never the address.

type AdminStore 

type AdminStore interface {
	StatsStore
	EntityStore
	RobotsStore
}

AdminStore is everything the admin handlers read from the database.

func NewAdminStore 

func NewAdminStore(db *database.DB) AdminStore

NewAdminStore returns the Postgres AdminStore.

type Agent 

type Agent struct {
	ID            string   `json:"id"`
	Name          string   `json:"name"`
	Operator      string   `json:"operator"`
	Purpose       Purpose  `json:"purpose"`
	UA            []string `json:"ua"`
	RobotsToken   string   `json:"robots_token"`
	IPRanges      []string `json:"ip_ranges"`
	RDNS          []string `json:"rdns"`
	IgnoresRobots bool     `json:"ignores_robots"`
}

Agent is one registry entry. Entries with an empty UA list are token-only (robots.txt tokens such as Google-Extended that never appear as a UA) and are never matched.

type AgentCount 

type AgentCount struct {
	AgentID string `json:"agent_id"`
	Count   int    `json:"count"`
}

AgentCount is a number of first crawls credited to one agent.

type AgentDetailData 

type AgentDetailData struct {
	Totals     AgentTotals
	Series     []DayCount
	TopPaths   []PathCount
	ByStatus   []StatusCount
	LastSeenAt *time.Time
	// PagesRead is the distinct paths the agent was served in the range (TopPaths is capped at 10).
	PagesRead int
	Found     bool
}

AgentDetailData is the data behind GET /agents/{id}.

type AgentTotals 

type AgentTotals struct {
	AgentID      string `json:"agent_id"`
	Operator     string `json:"operator"`
	Purpose      string `json:"purpose"`
	Hits         int64  `json:"hits"`
	Verified     int64  `json:"verified_hits"`
	Spoofed      int64  `json:"spoofed_hits"`
	Unverifiable int64  `json:"unverifiable_hits"`
	LastSeenDay  string `json:"last_seen_day"`
}

AgentTotals aggregates one agent over a range.

type DayCount 

type DayCount struct {
	Day  string `json:"day"`
	Hits int64  `json:"hits"`
}

DayCount is hits for one day.

type EntityActivityData 

type EntityActivityData struct {
	Exists       bool
	FirstAt      *time.Time // from hatamoto_agent_first_crawl
	PublishedAt  *time.Time
	TrackingFrom *time.Time // when first-crawl tracking began (hatamoto_agent_tracking)
	UserFetch7d  int64
	UserFetch30d int64
	Last         map[string]LastFetch // by agent id
}

EntityActivityData is the raw data behind GET /entities/{id}.

type EntityStore 

type EntityStore interface {
	EntityActivity(ctx context.Context, entityID uuid.UUID, agentIDs []string, today time.Time) (EntityActivityData, error)
}

EntityStore reads per-entity AI activity.

type ErrorRow 

type ErrorRow struct {
	Path     string   `json:"path"`
	Status   int      `json:"status"`
	Hits     int64    `json:"hits"`
	AgentIDs []string `json:"agent_ids"`
	LastDay  string   `json:"last_seen_day"`
}

ErrorRow is one (path, status>=400) group.

type FirstCrawl 

type FirstCrawl struct {
	EntityID    uuid.UUID
	AgentID     string
	Operator    string
	Purpose     string
	At          time.Time
	PublishedAt *time.Time
}

FirstCrawl is the first served AI fetch of an entity.

type FirstCrawlAgg 

type FirstCrawlAgg struct {
	SameDay, D1to3, D4to7, D8Plus, NotRecorded int
	ByAgent                                    []AgentCount // most first, then by id
}

FirstCrawlAgg summarises every first crawl in a range (not one page of the list): how long after publishing the page was first fetched, and by whom. NotRecorded holds crawls with no publish time, or one after the crawl.

type FirstCrawlRepository 

type FirstCrawlRepository struct {
	// contains filtered or unexported fields
}

FirstCrawlRepository is the Postgres FirstCrawlStore.

func NewFirstCrawlRepository 

func NewFirstCrawlRepository(db *database.DB) *FirstCrawlRepository

NewFirstCrawlRepository creates the Postgres-backed store.

func (*FirstCrawlRepository) RecordFirstCrawl 

func (r *FirstCrawlRepository) RecordFirstCrawl(ctx context.Context, entityID uuid.UUID, agentID string, at time.Time) (*time.Time, bool, error)

type FirstCrawlRow 

type FirstCrawlRow struct {
	EntityID    uuid.UUID  `json:"entity_id"`
	Name        string     `json:"name"`
	Slug        string     `json:"slug"`
	AgentID     string     `json:"agent_id"`
	FirstAt     time.Time  `json:"first_crawled_at"`
	PublishedAt *time.Time `json:"published_at"`
}

FirstCrawlRow is the first time an AI agent fetched one published page.

type FirstCrawlStore 

type FirstCrawlStore interface {
	// RecordFirstCrawl inserts the row unless the entity already has one (or
	// no longer exists). inserted reports whether this call created it;
	// publishedAt is the entity's publish time at that moment.
	RecordFirstCrawl(ctx context.Context, entityID uuid.UUID, agentID string, at time.Time) (publishedAt *time.Time, inserted bool, err error)
}

FirstCrawlStore persists the first crawl per entity.

type FirstCrawlTracker 

type FirstCrawlTracker struct {
	// contains filtered or unexported fields
}

FirstCrawlTracker watches stored hits and reports the first served AI fetch per entity. It is installed as the Writer observer, which must not block, so OnHits only enqueues; a goroutine does the database work. A full queue drops the candidate: a later hit for the same entity retries.

Accepted risk: hits are stored as pending, so a spoofed crawler can claim the "first crawl". The row is informational (latency card + hook).

func NewFirstCrawlTracker 

func NewFirstCrawlTracker(store FirstCrawlStore, reg *Registry, onFirst func(FirstCrawl), logger *slog.Logger) *FirstCrawlTracker

NewFirstCrawlTracker starts the tracker. onFirst may be nil; it runs on the tracker goroutine and must not block.

func (*FirstCrawlTracker) Close 

func (t *FirstCrawlTracker) Close()

Close drains what is queued and stops the goroutine. Idempotent.

func (*FirstCrawlTracker) OnHits 

func (t *FirstCrawlTracker) OnHits(hits []Hit)

OnHits is the Writer observer. Only AI-purpose, status 200 hits with an entity qualify. Never blocks.

type HTTPDoer 

type HTTPDoer interface {
	Do(*http.Request) (*http.Response, error)
}

HTTPDoer is the subset of *http.Client the refresher needs.

type HealthDay 

type HealthDay struct {
	Day                    string
	Hits                   int64
	S2xx, S3xx, S4xx, S5xx int64
	// AI4xx and AI5xx are the 4xx and 5xx responses to AI purposes only.
	AI4xx, AI5xx                  int64
	Verified, Spoofed, Unverified int64
}

HealthDay is one day of crawler health. Hits and the status classes count every agent (site-wide, like Errors); the verification split counts AI purposes only (like Overview). Status 0 (unknown) is in Hits but in no class.

type Hit 

type Hit struct {
	At           time.Time
	AgentID      string
	Operator     string
	Purpose      Purpose
	Verification string
	IP           *netip.Addr
	Method       string
	Path         string
	EntityID     *uuid.UUID
	Status       int // 0 = unknown
	Bytes        int64
	DurationMS   int
	Source       string
}

Hit is a classified hit ready for storage.

type LastFetch 

type LastFetch struct {
	At     time.Time
	Status *int
}

LastFetch is the most recent fetch of an entity by one agent.

type LogHandlers 

type LogHandlers struct {
	// contains filtered or unexported fields
}

LogHandlers serves the access-log ingest endpoint (Cloudflare Logpush / Caddy JSON, NDJSON). It is only mounted when log ingest is enabled.

func NewLogHandlers 

func NewLogHandlers(writer Recorder, secret string) *LogHandlers

NewLogHandlers creates the log ingest handlers. secret guards the endpoint (header X-Agent-Log-Secret) and is independent from the middleware secret.

func (*LogHandlers) HandleIngest 

func (h *LogHandlers) HandleIngest(w http.ResponseWriter, r *http.Request)

HandleIngest reads an NDJSON body (optionally gzip), keeps the lines that decision 19 assigns to the log source, and records them. Limits are all-or-nothing: a body over the byte, line-length or line-count cap is rejected with 413 before anything is recorded, so a sender notices instead of silently losing a tail. If the writer refuses every kept hit the answer is 503 + Retry-After (the sender retries the batch); if it refuses only some, the 202 body reports them as "dropped".

type NeverFetchedRow 

type NeverFetchedRow struct {
	EntityID    uuid.UUID  `json:"entity_id"`
	Name        string     `json:"name"`
	Slug        string     `json:"slug"`
	SchemaSlug  *string    `json:"schema_slug"`
	PublishedAt *time.Time `json:"published_at"`
}

NeverFetchedRow is a published entity no AI agent has ever fetched.

type OverviewTotals 

type OverviewTotals struct {
	AIFetches    int64
	Verified     int64
	Spoofed      int64
	Unverifiable int64
	AIVisits     int64 // hatamoto_analytics_daily ai_referral rows
	Pageviews    int64 // hatamoto_analytics_daily pageview rows
}

OverviewTotals are the headline numbers. Verified/Spoofed/Unverifiable only count hits the verifier has already decided (pending hits are in AIFetches but in none of the three).

type Page 

type Page struct{ Limit, Offset int }

Page is a limit/offset window.

type PageRow 

type PageRow struct {
	Path       string     `json:"path"`
	EntityID   *uuid.UUID `json:"entity_id"`
	EntityName *string    `json:"entity_name"`
	Hits       int64      `json:"hits"`
	Agents     int64      `json:"agents"`
	LastDay    string     `json:"last_fetch_day"`
	SchemaSlug *string    `json:"schema_slug"`
}

PageRow is one fetched path.

type PairVerifier 

type PairVerifier interface {
	Verify(ctx context.Context, a *Agent, ip netip.Addr) State
}

PairVerifier decides one (agent, ip) pair. *Verifier satisfies it.

type PathCount 

type PathCount struct {
	Path string `json:"path"`
	Hits int64  `json:"hits"`
}

PathCount is hits for one path.

type PendingPair 

type PendingPair struct {
	AgentID  string
	IP       netip.Addr
	OldestAt time.Time
}

PendingPair is one (agent, ip) group of pending hits. IP is the zero Addr when the hits carry no address.

type Purpose 

type Purpose string

Purpose classifies why an agent fetches pages. 

const (
	PurposeTraining    Purpose = "training"
	PurposeSearchIndex Purpose = "search_index"
	PurposeUserFetch   Purpose = "user_fetch"
	PurposeOtherAI     Purpose = "other_ai"
	PurposeSearch      Purpose = "search"
	PurposeSocial      Purpose = "social"
	PurposeSEO         Purpose = "seo"
	PurposeArchive     Purpose = "archive"
)

func (Purpose) IsAI 

func (p Purpose) IsAI() bool

IsAI reports whether the purpose is one of the AI groups.

type Range 

type Range struct{ From, To time.Time }

Range is an inclusive UTC date range.

type RangePersister 

type RangePersister interface {
	// LoadRanges returns every persisted list keyed by registry range key.
	LoadRanges(ctx context.Context) (map[string][]netip.Prefix, error)
	SaveRanges(ctx context.Context, key string, prefixes []netip.Prefix, sourceURLs []string, fetchedAt time.Time) error
}

RangePersister stores fetched range lists so they survive a restart.

type RangeRefresher 

type RangeRefresher interface {
	Refresh(ctx context.Context) RefreshReport
}

RangeRefresher downloads operator ranges. *Refresher satisfies it.

type RangeRepository 

type RangeRepository struct {
	// contains filtered or unexported fields
}

RangeRepository is the Postgres RangePersister.

func NewRangeRepository 

func NewRangeRepository(db *database.DB) *RangeRepository

func (*RangeRepository) LoadRanges 

func (r *RangeRepository) LoadRanges(ctx context.Context) (map[string][]netip.Prefix, error)

func (*RangeRepository) SaveRanges 

func (r *RangeRepository) SaveRanges(ctx context.Context, key string, prefixes []netip.Prefix, sourceURLs []string, fetchedAt time.Time) error

type RangeSource 

type RangeSource struct {
	URLs   []string `json:"urls"`
	Format string   `json:"format"`
}

RangeSource is a published IP range list for one or more agents.

type RangeStore 

type RangeStore struct {
	// contains filtered or unexported fields
}

RangeStore holds the operators' published IP ranges in memory. A key that was never loaded is distinct from one that is loaded and does not match: the verifier must not call a hit spoofed because a download has not succeeded yet.

func NewRangeStore 

func NewRangeStore() *RangeStore

func (*RangeStore) Contains 

func (s *RangeStore) Contains(keys []string, ip netip.Addr) (found, loaded bool)

Contains reports whether ip is inside any of the keys' prefixes (found), and whether at least one of the keys has been loaded (loaded). ip is unmapped.

func (*RangeStore) Has 

func (s *RangeStore) Has(key string) bool

Has reports whether key has ever been loaded.

func (*RangeStore) Load 

func (s *RangeStore) Load(ctx context.Context, p RangePersister) error

Load fills the store from persisted lists. Existing keys are overwritten.

func (*RangeStore) Set 

func (s *RangeStore) Set(key string, prefixes []netip.Prefix)

Set replaces one key's prefixes.

func (*RangeStore) Size 

func (s *RangeStore) Size() int

Size is the number of range keys loaded.

type RawHit 

type RawHit struct {
	At     time.Time
	UA, IP string
	Method string
	Path   string
	Status int
	// RedirectHint is a 3xx fallback used only when the resolver returns 404.
	RedirectHint int
	Bytes        int
	DurationMS   int
	Source       string
}

RawHit is one observed request, before classification. Status 0 means "unknown": the writer derives it (resolver for middleware hits, 200 for backend hits).

func ParseLogLine 

func ParseLogLine(line []byte) (RawHit, bool)

ParseLogLine parses one access-log line (Cloudflare Logpush or Caddy JSON, auto-detected) into a RawHit. It reports false for lines that are not parseable or that decision 19 (as amended in the P5 review) assigns to another capture source: only /media/* lines (any status) are kept, because pages, sitemaps, llms.txt and robots.txt, 5xx included, are already recorded by the middleware, the backend and the robots route. Query strings are stripped.

type Recorder 

type Recorder interface {
	// Record reports false only when the hit was refused (queue full or
	// writer closed). Hits that are intentionally ignored (unknown agent)
	// report true: there is nothing to retry.
	Record(RawHit) bool
}

Recorder is the write side of the Writer, so handlers can be tested without one.

type RefreshFailure 

type RefreshFailure struct {
	Key string
	Err error
}

RefreshFailure is one range key that could not be refreshed.

type RefreshReport 

type RefreshReport struct {
	Refreshed []string
	Failed    []RefreshFailure
}

RefreshReport summarises one Refresh run. A failure never aborts the other keys and never discards the key's previously loaded prefixes.

type Refresher 

type Refresher struct {
	// contains filtered or unexported fields
}

Refresher downloads the operators' published IP ranges into a RangeStore. It runs from a background job only, never at request time.

func NewRefresher 

func NewRefresher(reg *Registry, client HTTPDoer, store *RangeStore, persist RangePersister) *Refresher

NewRefresher returns a refresher. client nil means an http.Client with a 20s timeout; persist nil means in-memory only.

func (*Refresher) Refresh 

func (r *Refresher) Refresh(ctx context.Context) RefreshReport

Refresh fetches every range source. For a key, all URLs must succeed and the union must be non-empty; otherwise the key keeps its previous prefixes and is reported as failed. Cancelling ctx stops between keys.

type Registry 

type Registry struct {
	Version        int                    `json:"version"`
	GenericUAHints []string               `json:"generic_ua_hints"`
	IPRangeSources map[string]RangeSource `json:"ip_range_sources"`
	AIReferrers    []AIReferrer           `json:"ai_referrers"`
	Agents         []Agent                `json:"agents"`
	// contains filtered or unexported fields
}

Registry is the parsed registry.json.

func Default 

func Default() *Registry

Default returns the embedded registry, parsed once. It panics if the embedded file is invalid (a build-time bug caught by the tests).

func Parse 

func Parse(data []byte) (*Registry, error)

Parse decodes and validates registry JSON.

func (*Registry) ByID 

func (r *Registry) ByID(id string) *Agent

ByID returns the agent with the given id, or nil.

func (*Registry) ClassifyReferral 

func (r *Registry) ClassifyReferral(referrer, utmSource string) (string, bool)

ClassifyReferral maps a Referer URL and/or utm_source value to the AI assistant that sent the visitor, using the registry's ai_referrers list. utm_source wins over the referrer. A referrer matches when its host equals a listed host or is a subdomain of it (so "gemini.google.com" never matches "www.google.com", and "evilchatgpt.com" never matches "chatgpt.com"). Pure: no I/O, safe for concurrent use.

func (*Registry) LooksLikeBot 

func (r *Registry) LooksLikeBot(ua string) bool

LooksLikeBot is the broad "worth a closer look" test: a known agent or any generic hint substring.

func (*Registry) Match 

func (r *Registry) Match(ua string) *Agent

Match returns the first agent (file order) with a UA pattern contained in ua, or nil. Token-only entries are never matched.

func (*Registry) UAHintRegexp 

func (r *Registry) UAHintRegexp() *regexp.Regexp

UAHintRegexp is the case-insensitive union of every UA pattern and generic hint. The frontend BOT_UA_HINT is generated from the same data.

type Repository 

type Repository struct {
	// contains filtered or unexported fields
}

Repository is the Postgres Store.

func NewRepository 

func NewRepository(db *database.DB) *Repository

NewRepository creates the Postgres-backed store.

func (*Repository) InsertHits 

func (r *Repository) InsertHits(ctx context.Context, hits []Hit) error

InsertHits writes the raw rows and folds them into the daily rollup in one transaction. The rollup never receives an IP.

type ResolveFunc 

type ResolveFunc func(ctx context.Context, path string) (entityID *uuid.UUID, status int)

ResolveFunc maps a public path to the entity it serves and the HTTP status a visitor would get (301/302 redirect, 200 served, 404 miss). It runs in the writer goroutine, never on the request path.

func NewCachedResolver 

func NewCachedResolver(fn ResolveFunc, ttl time.Duration, max int) ResolveFunc

NewCachedResolver memoises fn per path for ttl, holding at most max entries (the cache is cleared when full: simple and bounded).

type Resolver 

type Resolver interface {
	LookupAddr(ctx context.Context, addr string) ([]string, error)
	LookupIPAddr(ctx context.Context, host string) ([]net.IPAddr, error)
}

Resolver is the DNS subset the verifier needs (*net.Resolver satisfies it).

type RobotsRepository 

type RobotsRepository struct {
	// contains filtered or unexported fields
}

RobotsRepository is the Postgres RobotsStore.

func NewRobotsRepository 

func NewRobotsRepository(db *database.DB) *RobotsRepository

NewRobotsRepository creates the Postgres-backed store.

func (*RobotsRepository) DeniedPurposes 

func (r *RobotsRepository) DeniedPurposes(ctx context.Context) ([]Purpose, error)

func (*RobotsRepository) SetDenied 

func (r *RobotsRepository) SetDenied(ctx context.Context, denied []Purpose) error

type RobotsStore 

type RobotsStore interface {
	DeniedPurposes(ctx context.Context) ([]Purpose, error)
	// SetDenied replaces the whole policy: exactly these purposes are denied,
	// every other row is deleted.
	SetDenied(ctx context.Context, denied []Purpose) error
}

RobotsStore persists the denied purposes.

type SchemaCount 

type SchemaCount struct {
	SchemaSlug *string `json:"schema_slug"`
	Count      int     `json:"count"`
}

SchemaCount is a number of pages of one page type (schema).

type SeriesPoint 

type SeriesPoint struct {
	Day  string `json:"day"`
	Key  string `json:"key"`
	Hits int64  `json:"hits"`
}

SeriesPoint is hits for one day and one group key (operator or purpose).

type SpoofedRow 

type SpoofedRow struct {
	OccurredAt time.Time `json:"occurred_at"`
	AgentID    string    `json:"agent_id"`
	Operator   string    `json:"operator"`
	Path       string    `json:"path"`
	Status     *int      `json:"status"`
	IPMasked   *string   `json:"ip_masked"`
}

SpoofedRow is one spoofed hit. IPMasked is the /24 or /48 network, never the stored address.

type State 

type State string

State is the outcome of verifying one (agent, IP) pair. Its values are the hatamoto_agent_hits.verification column values. 

const (
	StateVerified     State = VerificationVerified
	StateSpoofed      State = VerificationSpoofed
	StateUnverifiable State = VerificationUnverifiable
	// StatePending means "could not tell right now" (transient DNS failure):
	// retry later. It is never cached.
	StatePending State = VerificationPending
)

type StatsRepository 

type StatsRepository struct {
	// contains filtered or unexported fields
}

StatsRepository is the Postgres StatsStore.

func NewStatsRepository 

func NewStatsRepository(db *database.DB) *StatsRepository

NewStatsRepository creates the Postgres-backed stats store.

func (*StatsRepository) AgentDetail 

func (r *StatsRepository) AgentDetail(ctx context.Context, rg Range, agentID string) (AgentDetailData, error)

func (*StatsRepository) Agents 

func (r *StatsRepository) Agents(ctx context.Context, rg Range) ([]AgentTotals, error)

Agents lists every agent (any purpose) seen in the range.

func (*StatsRepository) EntityActivity 

func (r *StatsRepository) EntityActivity(ctx context.Context, entityID uuid.UUID, agentIDs []string, today time.Time) (EntityActivityData, error)

EntityActivity combines three sources, each used where it is the best one:

  • counts (7d/30d user fetches): the daily rollup, complete by construction (the writer upserts it for every hit), so no raw-hit double counting;
  • last fetch per agent: raw hits (second precision, ~90 days); an agent whose hits aged out falls back to the newest rollup day at 00:00 UTC;
  • first crawl: its own table, which outlives hit retention.

func (*StatsRepository) Errors 

func (r *StatsRepository) Errors(ctx context.Context, rg Range, p Page) ([]ErrorRow, int, error)

Errors groups failed (status >= 400) agent fetches by path and status. Read from the rollup, so it covers the whole range regardless of hit retention.

func (*StatsRepository) FetchesBySchema 

func (r *StatsRepository) FetchesBySchema(ctx context.Context, rg Range) ([]SeriesPoint, error)

FetchesBySchema returns daily served AI fetches grouped by page type (schema slug). Fetches with no entity are keyed "".

func (*StatsRepository) FirstCrawlAggregates 

func (r *StatsRepository) FirstCrawlAggregates(ctx context.Context, rg Range) (FirstCrawlAgg, error)

FirstCrawlAggregates counts the first crawls that fell inside rg, by time since publishing and by agent, over the same rows as FirstCrawls.

func (*StatsRepository) FirstCrawls 

func (r *StatsRepository) FirstCrawls(ctx context.Context, rg Range, p Page) ([]FirstCrawlRow, int, error)

FirstCrawls lists the pages whose first AI crawl fell inside rg, newest first. A page published before first-crawl tracking began shows the first crawl since tracking began, not necessarily its real first crawl.

func (*StatsRepository) HealthSeries 

func (r *StatsRepository) HealthSeries(ctx context.Context, rg Range) ([]HealthDay, error)

HealthSeries returns one row per day with data in the range, oldest first.

func (*StatsRepository) NeverFetched 

func (r *StatsRepository) NeverFetched(ctx context.Context, p Page) ([]NeverFetchedRow, int, error)

NeverFetched lists published entities with no served AI fetch in the whole rollup (all time, not just the selected range: "never" means never).

func (*StatsRepository) NeverFetchedBySchema 

func (r *StatsRepository) NeverFetchedBySchema(ctx context.Context) ([]SchemaCount, error)

NeverFetchedBySchema counts every never-fetched page per page type, not just the page of NeverFetched. Largest first.

func (*StatsRepository) Overview 

func (r *StatsRepository) Overview(ctx context.Context, rg Range) (OverviewTotals, error)

func (*StatsRepository) PublishedPages 

func (r *StatsRepository) PublishedPages(ctx context.Context) (int, error)

PublishedPages counts the pages coverage is measured against: the same filter as NeverFetched, so fetched plus never-fetched always adds up.

func (*StatsRepository) Series 

func (r *StatsRepository) Series(ctx context.Context, rg Range, groupBy string) ([]SeriesPoint, error)

Series returns daily hits grouped by "operator" or "purpose" (caller whitelists groupBy; it is interpolated as a column name).

func (*StatsRepository) Spoofed 

func (r *StatsRepository) Spoofed(ctx context.Context, rg Range, p Page) ([]SpoofedRow, int, error)

Spoofed lists spoofed hits (raw hits table: ~90 days) newest first.

func (*StatsRepository) TopAgents 

func (r *StatsRepository) TopAgents(ctx context.Context, rg Range, limit int) ([]AgentTotals, error)

TopAgents returns the AI agents with the most hits.

func (*StatsRepository) TopPages 

func (r *StatsRepository) TopPages(ctx context.Context, rg Range, p Page) ([]PageRow, int, error)

type StatsStore 

type StatsStore interface {
	Overview(ctx context.Context, rg Range) (OverviewTotals, error)
	Series(ctx context.Context, rg Range, groupBy string) ([]SeriesPoint, error)
	TopAgents(ctx context.Context, rg Range, limit int) ([]AgentTotals, error)
	TopPages(ctx context.Context, rg Range, p Page) ([]PageRow, int, error)
	NeverFetched(ctx context.Context, p Page) ([]NeverFetchedRow, int, error)
	Agents(ctx context.Context, rg Range) ([]AgentTotals, error)
	AgentDetail(ctx context.Context, rg Range, agentID string) (AgentDetailData, error)
	Errors(ctx context.Context, rg Range, p Page) ([]ErrorRow, int, error)
	Spoofed(ctx context.Context, rg Range, p Page) ([]SpoofedRow, int, error)
}

StatsStore is the read interface the admin handlers depend on.

type StatusCount 

type StatusCount struct {
	Status int   `json:"status"`
	Hits   int64 `json:"hits"`
}

StatusCount is hits for one status code.

type Store 

type Store interface {
	InsertHits(ctx context.Context, hits []Hit) error
}

Store persists classified hits (hits table + daily rollup).

type Tailer 

type Tailer struct {
	// contains filtered or unexported fields
}

Tailer follows a Caddy JSON access log and records the lines ParseLogLine keeps. It is strictly read-only on the file and never touches Caddy config. There is no offset persistence: a restart resumes at the end of the file, so lines written while the backend was down are not ingested.

func StartTailer 

func StartTailer(ctx context.Context, path string, rec Recorder, logger *slog.Logger) *Tailer

StartTailer starts following path until ctx is cancelled. The file does not need to exist yet (it is retried every poll). Call Wait after cancelling ctx and before closing the writer.

func (*Tailer) Wait 

func (t *Tailer) Wait()

Wait blocks until the tail goroutine has exited.

type Verifier 

type Verifier struct {
	// contains filtered or unexported fields
}

Verifier checks an (agent, IP) pair against the operator's published method: IP ranges first, forward-confirmed reverse DNS as the fallback.

func NewVerifier 

func NewVerifier(ranges *RangeStore, resolver Resolver) *Verifier

NewVerifier returns a verifier. resolver nil means net.DefaultResolver.

func (*Verifier) Verify 

func (v *Verifier) Verify(ctx context.Context, a *Agent, ip netip.Addr) State

Verify decides the state of a hit claiming to be agent a from ip.

  • no usable IP, or an agent with neither ranges nor rDNS: unverifiable
  • ranges loaded and ip inside: verified
  • ranges only, own keys not loaded: pending (never accuse; never terminal)
  • ranges loaded and ip outside: spoofed, even when the agent also has rDNS
  • rDNS (alone, or when the ranges are not loaded): PTR must end in an agent suffix and resolve forward to ip; mismatch/NXDOMAIN is spoofed; a transient DNS error is pending.

type VerifyRepo 

type VerifyRepo interface {
	PendingPairs(ctx context.Context, limit int) ([]PendingPair, error)
	// ApplyVerification flips the pair's pending hits (older than before, when
	// non-zero) to state and moves the rollup counters. Returns hits flipped.
	ApplyVerification(ctx context.Context, p PendingPair, state State, before time.Time) (int64, error)
	// SweepStalePending flips pending hits older than the verify window to
	// unverifiable and moves the rollup. Returns hits flipped.
	SweepStalePending(ctx context.Context) (int64, error)
}

VerifyRepo is the storage the verify worker needs.

type VerifyRepository 

type VerifyRepository struct {
	// contains filtered or unexported fields
}

VerifyRepository is the Postgres VerifyRepo.

func NewVerifyRepository 

func NewVerifyRepository(db *database.DB) *VerifyRepository

func (*VerifyRepository) ApplyVerification 

func (r *VerifyRepository) ApplyVerification(ctx context.Context, p PendingPair, state State, before time.Time) (int64, error)

func (*VerifyRepository) PendingPairs 

func (r *VerifyRepository) PendingPairs(ctx context.Context, limit int) ([]PendingPair, error)

func (*VerifyRepository) SweepStalePending 

func (r *VerifyRepository) SweepStalePending(ctx context.Context) (int64, error)

type VerifyWorker 

type VerifyWorker struct {
	// contains filtered or unexported fields
}

VerifyWorker flips pending hits to verified/spoofed/unverifiable and keeps the operator IP ranges fresh. Network failures are logged and never touch stored hits.

func NewVerifyWorker 

func NewVerifyWorker(repo VerifyRepo, reg *Registry, verifier PairVerifier, refresher RangeRefresher,
	store *RangeStore, persist RangePersister, logger *slog.Logger) *VerifyWorker

NewVerifyWorker wires the worker. store/persist may be nil (no persisted ranges, no readiness guard), as in tests with a fake verifier.

func (*VerifyWorker) NoRangeRefresh 

func (w *VerifyWorker) NoRangeRefresh()

NoRangeRefresh tells the worker no range refresh will ever run (the switch HATAMOTO_AGENT_RANGES=off, or the job disabled), so waiting for one would leave every hit pending. Verification then runs on stored ranges and forward-confirmed reverse DNS only.

func (*VerifyWorker) RunRefresh 

func (w *VerifyWorker) RunRefresh(ctx context.Context) error

RunRefresh downloads the operator IP ranges. It always returns nil: a network failure keeps the old ranges, is logged, and must never fail the job or touch stored hits.

func (*VerifyWorker) RunVerify 

func (w *VerifyWorker) RunVerify(ctx context.Context) error

RunVerify processes one batch of pending (agent, ip) pairs.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer persists agent hits off the request path. Record never blocks: a full queue drops the hit and counts it. The data channel is never closed, so a Record racing Close cannot panic; at worst one hit is lost.

func NewWriter 

func NewWriter(store Store, reg *Registry, resolve ResolveFunc, logger *slog.Logger) *Writer

NewWriter starts the background flush goroutine. resolve may be nil (page hits then keep status 0 and no entity).

func (*Writer) Close 

func (w *Writer) Close()

Close stops accepting hits, flushes what is buffered (bounded) and waits for the goroutine. Idempotent.

func (*Writer) Record 

func (w *Writer) Record(h RawHit) bool

Record classifies and enqueues a hit. Unknown user agents are ignored. Never blocks, never panics, safe after Close. It returns false when the hit was refused (writer closed or queue full), true otherwise.

func (*Writer) SetObserver 

func (w *Writer) SetObserver(fn func([]Hit))

SetObserver installs a callback invoked from the flush goroutine with each successfully stored batch (nil removes it). It must not block.

func (*Writer) Stats 

func (w *Writer) Stats() WriterStats

Stats returns the current counters.

type WriterStats 

type WriterStats struct {
	Queued  int
	Dropped uint64
	Written uint64
	Failed  uint64
}

WriterStats are process-lifetime counters.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.