tracking

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: AGPL-3.0 Imports: 36 Imported by: 0

Documentation

Overview

Package tracking serves the open pixel and the click redirect that go in every tracked email, and records opens and clicks honestly: human opens separated from machine opens (Apple Mail Privacy Protection, image proxy prefetches, security scanners) and bot clicks kept out of the counts.

GET /t/o/{token}   1×1 transparent GIF, never cached; records the open
GET /t/c/{token}   302 to the email's link; records the click

Tokens are signed (internal/linktoken) and name the message, the contact and, for clicks, the link's position in the campaign's links: the redirect only ever goes to a link of that message (no open redirect). Writes are group-committed in the background, so the pixel and the redirect stay fast under load. The unsubscribe page and RFC 8058 one-click POST live with the public pages.

Index

Constants

View Source
const (
	// ClassHuman: a person opened or clicked.
	ClassHuman = "human"
	// ClassPrivacy: a privacy proxy (Apple Mail Privacy Protection) fetched
	// the images on delivery. Whether a person read the email can't be
	// measured: these opens are neither a person's nor a scanner's.
	ClassPrivacy = "privacy"
	// ClassMachine: a scanner or a proxy prefetching on arrival opened it.
	ClassMachine = "machine"
	// ClassBot: a click no person made (scanners following every link).
	ClassBot = "bot"
)

Classes of opens and clicks.

View Source
const (

	// HoneypotLink is the position the hidden link's clicks are stored with.
	HoneypotLink = -1
	// NoLink is the position of a provider-reported click on a link that
	// isn't one of the email's (review AN-4).
	NoLink = -2
)

Clicks settle before they count (review AN-2). A scanner that follows links clicks every link of an email within seconds of its arrival, and follows the hidden link no person can see; a click alone can look like a person's. So a click that looks like a person's is recorded as pending, and settles after settleAfter: the clicks around it decide. Settled as a person's, it counts once (the human open, the unique click on its link, the automation's "clicked", Earn's credit, the transactional webhook); settled as a bot's, it is filed with the bot clicks the report shows apart. Nothing has to be taken back.

Variables

This section is empty.

Functions

func ClientName

func ClientName(ua string) string

ClientName turns a user agent into "Chrome on macOS", "Outlook on Windows", "Apple Mail on iOS".

func DiagnosticIn

func DiagnosticIn(l i18n.Lang, s string) string

DiagnosticIn writes a stored verdict ("Apple Mail Privacy Protection prefetch", "Chrome on macOS") in a reader's language. Text tracking didn't write, such as a provider's bounce message, is returned as it is.

func LinkPosition

func LinkPosition(links map[int]string, clicked string) int

LinkPosition finds which of an email's links (position → the link as written, possibly personalized with Liquid) a provider-reported click was on: the same address, else the one whose personalized parts match; NoLink when none does. Two links to the same address are the first's.

func RegisterNetworks

func RegisterNetworks(h huma.API, s *Service)

RegisterNetworks adds the instance admin's operations on the bot filter's lists: the networks and the user agents (agents.go).

Types

type ClickTarget

type ClickTarget struct {
	WorkspaceID uuid.UUID
	MessageID   uuid.UUID
	ContactID   uuid.UUID
	URL         string
	At          time.Time
}

ClickTarget is a click about to be redirected, as Decorate sees it.

type ClickedLink struct {
	WorkspaceID  uuid.UUID
	MessageID    uuid.UUID
	ContactID    *uuid.UUID
	CampaignID   *uuid.UUID
	AutomationID *uuid.UUID
	Target       string
	At           time.Time
}

ClickedLink is a click about to be redirected: the message's email and the destination.

type HumanClick

type HumanClick struct {
	WorkspaceID uuid.UUID
	MessageID   uuid.UUID
	CampaignID  *uuid.UUID
	ContactID   *uuid.UUID
	Stream      string
	// URL is the link as the email has it (it may hold Liquid).
	URL string
	At  time.Time
}

HumanClick is a person's click, as Clicked hears of it.

type LinkDecorator

type LinkDecorator interface {
	Decorate(ctx context.Context, c ClickedLink) string
}

LinkDecorator may rewrite a click's destination before the redirect. It runs on every click, so it must answer fast (cache what it reads).

type Network

type Network struct {
	// Class is "privacy" (a privacy proxy) or "scanner".
	Class string
	Name  string
}

Network is a network of machines that open and click email.

type PlacementLinks interface {
	TrackedLink(ctx context.Context, copyID uuid.UUID, position int) (string, bool)
}

PlacementLinks resolves a placement test copy's tracked links.

type RetryArgs

type RetryArgs struct{}

RetryArgs applies the opens and clicks kept for a retry (periodic).

func (RetryArgs) InsertOpts

func (RetryArgs) InsertOpts() river.InsertOpts

InsertOpts implements river.JobArgsWithInsertOpts.

func (RetryArgs) Kind

func (RetryArgs) Kind() string

Kind implements river.JobArgs.

type RetryResult

type RetryResult struct {
	Recorded int
	Failed   int
}

RetryResult says what a retry run did.

type RetryWorker

type RetryWorker struct {
	river.WorkerDefaults[RetryArgs]
	S *Service
}

RetryWorker runs RetryArgs.

func (*RetryWorker) Work

func (w *RetryWorker) Work(ctx context.Context, _ *river.Job[RetryArgs]) error

Work implements river.Worker.

type Service

type Service struct {
	Pool        *pgxpool.Pool
	Q           *store.Queries
	Links       *linktoken.Signer
	Log         *slog.Logger
	Automations *automations.Service
	// Transactional resolves the tracked links of transactional messages
	// and fires their message.opened and message.clicked webhook events.
	Transactional TransactionalTracking
	// Placement resolves the links of placement test copies, whose opens
	// and clicks are never counted (internal/placement).
	Placement PlacementLinks
	// Decorators may add parameters to a click's destination: links to a
	// connected store get UTM parameters and the click token
	// (internal/commerce).
	Decorators []LinkDecorator
	// MessageVars are an automation email's own link variables, from what
	// its trigger passed ("checkout_url": the store's recovery link).
	MessageVars func(ctx context.Context, msgCtx map[string]string) map[string]string
	Now         func() time.Time
	// Clicked, when set, hears of each person's click (bot clicks never
	// reach it) in the transaction that records it: Earn counts clicks on
	// upgrade links per campaign.
	Clicked func(ctx context.Context, q *store.Queries, c HumanClick) error
	// Decorate, when set, may add to a click's destination before the
	// redirect (Earn's click token on the publisher's own domains); "" keeps
	// the destination as it is. It runs after Decorators, and adds nothing
	// to a link that already carries rc_click.
	Decorate func(ctx context.Context, c ClickTarget) string
	// Recipients add the per-reader variables of blocks (Grow's
	// referral.* links) to personalized links; nil adds none.
	Recipients func() []liquid.RecipientData

	// Audit records the instance admin's changes to the machine networks.
	Audit *audit.Logger
	// contains filtered or unexported fields
}

Service records opens and clicks.

func (*Service) AddNetwork

AddNetwork adds a network to the list (audited).

func (*Service) AddUserAgent

AddUserAgent adds a user agent to the list (audited). A machine's pattern found in the user agent of a browser or mail client people read email with is refused: it would take them for machines.

func (*Service) Begin

func (s *Service) Begin(ledger *counters.Ledger) ingest.EngagementBatch

Begin implements ingest.Engagement: the webhook batch's opens and clicks move its own ledger, so its counters are locked once, in order.

func (*Service) Networks

func (s *Service) Networks(ctx context.Context) (TrackingNetworkList, error)

Networks lists the instance's networks.

func (*Service) RecordEngagement

func (s *Service) RecordEngagement(ctx context.Context, evs []sending.Engagement) error

RecordEngagement implements sending.Recorder: it classifies and records a batch of opens and clicks in one transaction (retried on deadlocks and serialization failures).

func (*Service) RemoveNetwork

func (s *Service) RemoveNetwork(ctx context.Context, cidr string) error

RemoveNetwork takes a network an admin added off the list (audited).

func (*Service) RemoveUserAgent

func (s *Service) RemoveUserAgent(ctx context.Context, pattern string) error

RemoveUserAgent takes a user agent an admin added off the list (audited).

func (*Service) RetryParked

func (s *Service) RetryParked(ctx context.Context) (RetryResult, error)

RetryParked applies the kept opens and clicks that are due, a batch at a time; a batch that fails again is retried event by event so one bad event doesn't hold the others back.

func (*Service) Routes

func (s *Service) Routes(r chi.Router)

Routes mounts the tracking endpoints.

func (*Service) SettleEngagement

func (s *Service) SettleEngagement(ctx context.Context, now time.Time) (*time.Time, error)

SettleEngagement settles the clicks that are due at now and returns when the next one is due (nil when none waits). The demo transport's pump and the tests call it with their own clock; the tracking.settle job and the writer call it with the real one.

func (*Service) UserAgents

func (s *Service) UserAgents(ctx context.Context) (TrackingUserAgentList, error)

UserAgents lists the instance's user agents.

type SettleArgs

type SettleArgs struct{}

SettleArgs settles the clicks that are due (periodic).

func (SettleArgs) InsertOpts

func (SettleArgs) InsertOpts() river.InsertOpts

InsertOpts implements river.JobArgsWithInsertOpts.

func (SettleArgs) Kind

func (SettleArgs) Kind() string

Kind implements river.JobArgs.

type SettleWorker

type SettleWorker struct {
	river.WorkerDefaults[SettleArgs]
	S *Service
}

SettleWorker runs SettleArgs.

func (*SettleWorker) Work

func (w *SettleWorker) Work(ctx context.Context, _ *river.Job[SettleArgs]) error

Work implements river.Worker.

type Signal

type Signal struct {
	UserAgent string
	IP        netip.Addr
	// Method is the HTTP method; scanners often probe with HEAD.
	Method string
	// SinceSend is how long after the message was handed to the provider
	// the request came (negative when unknown).
	SinceSend time.Duration
	// SinceDelivery is how long after the message was delivered, when
	// Delivered says that is known. Scanners act when mail arrives, which
	// can be minutes after it was sent, so the windows below are measured
	// from delivery when it is known (review AN-2).
	SinceDelivery time.Duration
	Delivered     bool
	// Net is the machine network the address belongs to, from the
	// instance's list (tracking_networks); nil checks the built-in ranges.
	Net *Network
	// Agents is the instance's list of machines' user agents
	// (tracking_user_agents); nil is the built-in list.
	Agents *useragent.List
	// Reported: the sending provider reported the open or click (the
	// workspace keeps the provider's tracking, review AN-4). A provider
	// that doesn't pass the user agent on says nothing about the request,
	// so a missing one isn't a machine's sign.
	Reported bool
}

Signal is what a tracking request tells us about who made it.

type TrackingNetwork

type TrackingNetwork struct {
	CIDR string `json:"cidr" example:"203.0.113.0/24"`
	Name string `json:"name"`
	// Class: privacy (a privacy proxy fetches images on delivery; its opens
	// are privacy-protected) or scanner (a mail-security scanner; its opens
	// and clicks on arrival are a machine's).
	Class string `json:"class" enum:"privacy,scanner"`
	// Source: builtin (comes with ReputeCast) or admin (added here).
	Source        string    `json:"source" enum:"builtin,admin"`
	Note          string    `json:"note"`
	CreatedAt     time.Time `json:"createdAt"`
	CreatedByName string    `json:"createdByName,omitempty"`
}

TrackingNetwork is one network of privacy proxies or mail scanners.

type TrackingNetworkInput

type TrackingNetworkInput struct {
	CIDR  string `json:"cidr" minLength:"3" maxLength:"64" doc:"An address or a network: 203.0.113.0/24, 2001:db8::/48"`
	Name  string `json:"name" minLength:"1" maxLength:"80" doc:"Who runs it: Mimecast, our mail gateway"`
	Class string `json:"class" enum:"privacy,scanner"`
	Note  string `json:"note,omitempty" maxLength:"300"`
}

TrackingNetworkInput adds a network.

type TrackingNetworkList

type TrackingNetworkList struct {
	Items []TrackingNetwork `json:"items"`
}

TrackingNetworkList lists the networks.

type TrackingUserAgent

type TrackingUserAgent struct {
	// Pattern is found in a machine's user agent (any text), or is a word
	// of a person's device (letters and digits).
	Pattern string `json:"pattern" example:"acmegate-linkcheck"`
	// Kind: scanner (a mail-security scanner or link checker), script,
	// headless (a headless browser), preview (a link preview), or person (a
	// person's device a rule would take for a machine).
	Kind string `json:"kind" enum:"scanner,script,headless,preview,person"`
	// Name is the product or who runs it; a script, headless browser or
	// link preview may have none.
	Name string `json:"name"`
	// Source: builtin (comes with ReputeCast) or admin (added here).
	Source        string    `json:"source" enum:"builtin,admin"`
	Note          string    `json:"note"`
	CreatedAt     time.Time `json:"createdAt"`
	CreatedByName string    `json:"createdByName,omitempty"`
}

TrackingUserAgent is one entry of the user agent list.

type TrackingUserAgentInput

type TrackingUserAgentInput struct {
	Pattern string `` /* 137-byte string literal not displayed */
	// Kind is scanner for a machine (a scanner, link checker or script),
	// person for a person's device or app a rule takes for a machine.
	Kind string `json:"kind" enum:"scanner,person"`
	Name string `json:"name" minLength:"1" maxLength:"80" doc:"Who or what it is: our mail gateway, lobby kiosks"`
	Note string `json:"note,omitempty" maxLength:"300"`
}

TrackingUserAgentInput adds a user agent.

type TrackingUserAgentList

type TrackingUserAgentList struct {
	Items []TrackingUserAgent `json:"items"`
}

TrackingUserAgentList lists the user agents.

type TransactionalTracking

type TransactionalTracking interface {
	TrackedLink(ctx context.Context, q *store.Queries, messageID uuid.UUID, position int) (string, bool)
	Engagement(ctx context.Context, tx pgx.Tx, messageID uuid.UUID, click bool, at time.Time) error
}

TransactionalTracking is what the tracking endpoints need from transactional email: each message keeps its own tracked links (their targets differ per recipient), and engagement fires webhooks.

type Verdict

type Verdict struct {
	// Machine is true for opens and clicks no person made: privacy proxies
	// prefetching images, security scanners following links.
	Machine bool
	// Class is ClassHuman, ClassPrivacy or ClassMachine for an open,
	// ClassHuman or ClassBot for a click.
	Class string
	// Reason explains a machine verdict in the live feed's words
	// ("Apple Mail Privacy Protection prefetch").
	Reason string
	// Client is the mail client or browser, for human events
	// ("Chrome on macOS").
	Client string
}

Verdict classifies an open or a click.

func ClassifyClick

func ClassifyClick(s Signal, burst bool) Verdict

ClassifyClick decides whether a click was a person. Bot clicks still get redirected; they are logged but never counted as clicks.

Bot clicks: HEAD requests, empty or scanner user agents, scanner networks within two minutes of sending, clicks within ten seconds of sending, and bursts (burst is true when the same message already clicked several different links within the last few seconds, as scanners that follow every link do).

func ClassifyHoneypot

func ClassifyHoneypot() Verdict

ClassifyHoneypot is a click on the hidden link: no person can see it.

func ClassifyOpen

func ClassifyOpen(s Signal) Verdict

ClassifyOpen decides whether an open pixel fetch was a person.

Machine opens, in order:

  1. HEAD requests and empty user agents.
  2. Apple Mail Privacy Protection: requests from Apple's network (17.0.0.0/8) or with the bare "Mozilla/5.0" user agent its proxies send. They fetch every image on delivery whether or not anyone reads.
  3. Gmail's GoogleImageProxy and Yahoo's YahooMailProxy within 30 s of sending (caching on arrival); later fetches are people opening in Gmail or Yahoo Mail and count as human.
  4. Security scanners and HTTP libraries (user agent), and scanner networks within two minutes of sending.
  5. Anything within two seconds of sending.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL