Documentation
¶
Overview ¶
Package tracking serves the open pixel and the click redirect that go in every tracked email, and records opens and clicks honestly: human opens separated from machine opens (Apple Mail Privacy Protection, image proxy prefetches, security scanners) and bot clicks kept out of the counts.
GET /t/o/{token} 1×1 transparent GIF, never cached; records the open
GET /t/c/{token} 302 to the email's link; records the click
Tokens are signed (internal/linktoken) and name the message, the contact and, for clicks, the link's position in the campaign's links: the redirect only ever goes to a link of that message (no open redirect). Writes are group-committed in the background, so the pixel and the redirect stay fast under load. The unsubscribe page and RFC 8058 one-click POST live with the public pages.
Index ¶
- Constants
- func ClientName(ua string) string
- func DiagnosticIn(l i18n.Lang, s string) string
- func LinkPosition(links map[int]string, clicked string) int
- func RegisterNetworks(h huma.API, s *Service)
- type ClickTarget
- type ClickedLink
- type HumanClick
- type LinkDecorator
- type Network
- type PlacementLinks
- type RetryArgs
- type RetryResult
- type RetryWorker
- type Service
- func (s *Service) AddNetwork(ctx context.Context, in TrackingNetworkInput) (TrackingNetwork, error)
- func (s *Service) AddUserAgent(ctx context.Context, in TrackingUserAgentInput) (TrackingUserAgent, error)
- func (s *Service) Begin(ledger *counters.Ledger) ingest.EngagementBatch
- func (s *Service) Networks(ctx context.Context) (TrackingNetworkList, error)
- func (s *Service) RecordEngagement(ctx context.Context, evs []sending.Engagement) error
- func (s *Service) RemoveNetwork(ctx context.Context, cidr string) error
- func (s *Service) RemoveUserAgent(ctx context.Context, pattern string) error
- func (s *Service) RetryParked(ctx context.Context) (RetryResult, error)
- func (s *Service) Routes(r chi.Router)
- func (s *Service) SettleEngagement(ctx context.Context, now time.Time) (*time.Time, error)
- func (s *Service) UserAgents(ctx context.Context) (TrackingUserAgentList, error)
- type SettleArgs
- type SettleWorker
- type Signal
- type TrackingNetwork
- type TrackingNetworkInput
- type TrackingNetworkList
- type TrackingUserAgent
- type TrackingUserAgentInput
- type TrackingUserAgentList
- type TransactionalTracking
- type Verdict
Constants ¶
const ( // ClassHuman: a person opened or clicked. ClassHuman = "human" // ClassPrivacy: a privacy proxy (Apple Mail Privacy Protection) fetched // the images on delivery. Whether a person read the email can't be // measured: these opens are neither a person's nor a scanner's. ClassPrivacy = "privacy" // ClassMachine: a scanner or a proxy prefetching on arrival opened it. ClassMachine = "machine" // ClassBot: a click no person made (scanners following every link). ClassBot = "bot" )
Classes of opens and clicks.
const ( // HoneypotLink is the position the hidden link's clicks are stored with. HoneypotLink = -1 // NoLink is the position of a provider-reported click on a link that // isn't one of the email's (review AN-4). NoLink = -2 )
Clicks settle before they count (review AN-2). A scanner that follows links clicks every link of an email within seconds of its arrival, and follows the hidden link no person can see; a click alone can look like a person's. So a click that looks like a person's is recorded as pending, and settles after settleAfter: the clicks around it decide. Settled as a person's, it counts once (the human open, the unique click on its link, the automation's "clicked", Earn's credit, the transactional webhook); settled as a bot's, it is filed with the bot clicks the report shows apart. Nothing has to be taken back.
Variables ¶
This section is empty.
Functions ¶
func ClientName ¶
ClientName turns a user agent into "Chrome on macOS", "Outlook on Windows", "Apple Mail on iOS".
func DiagnosticIn ¶
DiagnosticIn writes a stored verdict ("Apple Mail Privacy Protection prefetch", "Chrome on macOS") in a reader's language. Text tracking didn't write, such as a provider's bounce message, is returned as it is.
func LinkPosition ¶
LinkPosition finds which of an email's links (position → the link as written, possibly personalized with Liquid) a provider-reported click was on: the same address, else the one whose personalized parts match; NoLink when none does. Two links to the same address are the first's.
func RegisterNetworks ¶
RegisterNetworks adds the instance admin's operations on the bot filter's lists: the networks and the user agents (agents.go).
Types ¶
type ClickTarget ¶
type ClickTarget struct {
WorkspaceID uuid.UUID
MessageID uuid.UUID
ContactID uuid.UUID
URL string
At time.Time
}
ClickTarget is a click about to be redirected, as Decorate sees it.
type ClickedLink ¶
type ClickedLink struct {
WorkspaceID uuid.UUID
MessageID uuid.UUID
ContactID *uuid.UUID
CampaignID *uuid.UUID
AutomationID *uuid.UUID
Target string
At time.Time
}
ClickedLink is a click about to be redirected: the message's email and the destination.
type HumanClick ¶
type HumanClick struct {
WorkspaceID uuid.UUID
MessageID uuid.UUID
CampaignID *uuid.UUID
ContactID *uuid.UUID
Stream string
// URL is the link as the email has it (it may hold Liquid).
URL string
At time.Time
}
HumanClick is a person's click, as Clicked hears of it.
type LinkDecorator ¶
type LinkDecorator interface {
Decorate(ctx context.Context, c ClickedLink) string
}
LinkDecorator may rewrite a click's destination before the redirect. It runs on every click, so it must answer fast (cache what it reads).
type Network ¶
type Network struct {
// Class is "privacy" (a privacy proxy) or "scanner".
Class string
Name string
}
Network is a network of machines that open and click email.
type PlacementLinks ¶
type PlacementLinks interface {
TrackedLink(ctx context.Context, copyID uuid.UUID, position int) (string, bool)
}
PlacementLinks resolves a placement test copy's tracked links.
type RetryArgs ¶
type RetryArgs struct{}
RetryArgs applies the opens and clicks kept for a retry (periodic).
func (RetryArgs) InsertOpts ¶
func (RetryArgs) InsertOpts() river.InsertOpts
InsertOpts implements river.JobArgsWithInsertOpts.
type RetryResult ¶
RetryResult says what a retry run did.
type RetryWorker ¶
type RetryWorker struct {
river.WorkerDefaults[RetryArgs]
S *Service
}
RetryWorker runs RetryArgs.
type Service ¶
type Service struct {
Pool *pgxpool.Pool
Q *store.Queries
Links *linktoken.Signer
Log *slog.Logger
Automations *automations.Service
// Transactional resolves the tracked links of transactional messages
// and fires their message.opened and message.clicked webhook events.
Transactional TransactionalTracking
// Placement resolves the links of placement test copies, whose opens
// and clicks are never counted (internal/placement).
Placement PlacementLinks
// Decorators may add parameters to a click's destination: links to a
// connected store get UTM parameters and the click token
// (internal/commerce).
Decorators []LinkDecorator
// MessageVars are an automation email's own link variables, from what
// its trigger passed ("checkout_url": the store's recovery link).
MessageVars func(ctx context.Context, msgCtx map[string]string) map[string]string
Now func() time.Time
// Clicked, when set, hears of each person's click (bot clicks never
// reach it) in the transaction that records it: Earn counts clicks on
// upgrade links per campaign.
Clicked func(ctx context.Context, q *store.Queries, c HumanClick) error
// Decorate, when set, may add to a click's destination before the
// redirect (Earn's click token on the publisher's own domains); "" keeps
// the destination as it is. It runs after Decorators, and adds nothing
// to a link that already carries rc_click.
Decorate func(ctx context.Context, c ClickTarget) string
// Recipients add the per-reader variables of blocks (Grow's
// referral.* links) to personalized links; nil adds none.
Recipients func() []liquid.RecipientData
// Audit records the instance admin's changes to the machine networks.
Audit *audit.Logger
// contains filtered or unexported fields
}
Service records opens and clicks.
func (*Service) AddNetwork ¶
func (s *Service) AddNetwork(ctx context.Context, in TrackingNetworkInput) (TrackingNetwork, error)
AddNetwork adds a network to the list (audited).
func (*Service) AddUserAgent ¶
func (s *Service) AddUserAgent(ctx context.Context, in TrackingUserAgentInput) (TrackingUserAgent, error)
AddUserAgent adds a user agent to the list (audited). A machine's pattern found in the user agent of a browser or mail client people read email with is refused: it would take them for machines.
func (*Service) Begin ¶
func (s *Service) Begin(ledger *counters.Ledger) ingest.EngagementBatch
Begin implements ingest.Engagement: the webhook batch's opens and clicks move its own ledger, so its counters are locked once, in order.
func (*Service) Networks ¶
func (s *Service) Networks(ctx context.Context) (TrackingNetworkList, error)
Networks lists the instance's networks.
func (*Service) RecordEngagement ¶
RecordEngagement implements sending.Recorder: it classifies and records a batch of opens and clicks in one transaction (retried on deadlocks and serialization failures).
func (*Service) RemoveNetwork ¶
RemoveNetwork takes a network an admin added off the list (audited).
func (*Service) RemoveUserAgent ¶
RemoveUserAgent takes a user agent an admin added off the list (audited).
func (*Service) RetryParked ¶
func (s *Service) RetryParked(ctx context.Context) (RetryResult, error)
RetryParked applies the kept opens and clicks that are due, a batch at a time; a batch that fails again is retried event by event so one bad event doesn't hold the others back.
func (*Service) SettleEngagement ¶
SettleEngagement settles the clicks that are due at now and returns when the next one is due (nil when none waits). The demo transport's pump and the tests call it with their own clock; the tracking.settle job and the writer call it with the real one.
func (*Service) UserAgents ¶
func (s *Service) UserAgents(ctx context.Context) (TrackingUserAgentList, error)
UserAgents lists the instance's user agents.
type SettleArgs ¶
type SettleArgs struct{}
SettleArgs settles the clicks that are due (periodic).
func (SettleArgs) InsertOpts ¶
func (SettleArgs) InsertOpts() river.InsertOpts
InsertOpts implements river.JobArgsWithInsertOpts.
type SettleWorker ¶
type SettleWorker struct {
river.WorkerDefaults[SettleArgs]
S *Service
}
SettleWorker runs SettleArgs.
func (*SettleWorker) Work ¶
func (w *SettleWorker) Work(ctx context.Context, _ *river.Job[SettleArgs]) error
Work implements river.Worker.
type Signal ¶
type Signal struct {
UserAgent string
IP netip.Addr
// Method is the HTTP method; scanners often probe with HEAD.
Method string
// SinceSend is how long after the message was handed to the provider
// the request came (negative when unknown).
SinceSend time.Duration
// SinceDelivery is how long after the message was delivered, when
// Delivered says that is known. Scanners act when mail arrives, which
// can be minutes after it was sent, so the windows below are measured
// from delivery when it is known (review AN-2).
SinceDelivery time.Duration
Delivered bool
// Net is the machine network the address belongs to, from the
// instance's list (tracking_networks); nil checks the built-in ranges.
Net *Network
// Agents is the instance's list of machines' user agents
// (tracking_user_agents); nil is the built-in list.
Agents *useragent.List
// Reported: the sending provider reported the open or click (the
// workspace keeps the provider's tracking, review AN-4). A provider
// that doesn't pass the user agent on says nothing about the request,
// so a missing one isn't a machine's sign.
Reported bool
}
Signal is what a tracking request tells us about who made it.
type TrackingNetwork ¶
type TrackingNetwork struct {
CIDR string `json:"cidr" example:"203.0.113.0/24"`
Name string `json:"name"`
// Class: privacy (a privacy proxy fetches images on delivery; its opens
// are privacy-protected) or scanner (a mail-security scanner; its opens
// and clicks on arrival are a machine's).
Class string `json:"class" enum:"privacy,scanner"`
// Source: builtin (comes with ReputeCast) or admin (added here).
Source string `json:"source" enum:"builtin,admin"`
Note string `json:"note"`
CreatedAt time.Time `json:"createdAt"`
CreatedByName string `json:"createdByName,omitempty"`
}
TrackingNetwork is one network of privacy proxies or mail scanners.
type TrackingNetworkInput ¶
type TrackingNetworkInput struct {
CIDR string `json:"cidr" minLength:"3" maxLength:"64" doc:"An address or a network: 203.0.113.0/24, 2001:db8::/48"`
Name string `json:"name" minLength:"1" maxLength:"80" doc:"Who runs it: Mimecast, our mail gateway"`
Class string `json:"class" enum:"privacy,scanner"`
Note string `json:"note,omitempty" maxLength:"300"`
}
TrackingNetworkInput adds a network.
type TrackingNetworkList ¶
type TrackingNetworkList struct {
Items []TrackingNetwork `json:"items"`
}
TrackingNetworkList lists the networks.
type TrackingUserAgent ¶
type TrackingUserAgent struct {
// Pattern is found in a machine's user agent (any text), or is a word
// of a person's device (letters and digits).
Pattern string `json:"pattern" example:"acmegate-linkcheck"`
// Kind: scanner (a mail-security scanner or link checker), script,
// headless (a headless browser), preview (a link preview), or person (a
// person's device a rule would take for a machine).
Kind string `json:"kind" enum:"scanner,script,headless,preview,person"`
// Name is the product or who runs it; a script, headless browser or
// link preview may have none.
Name string `json:"name"`
// Source: builtin (comes with ReputeCast) or admin (added here).
Source string `json:"source" enum:"builtin,admin"`
Note string `json:"note"`
CreatedAt time.Time `json:"createdAt"`
CreatedByName string `json:"createdByName,omitempty"`
}
TrackingUserAgent is one entry of the user agent list.
type TrackingUserAgentInput ¶
type TrackingUserAgentInput struct {
Pattern string `` /* 137-byte string literal not displayed */
// Kind is scanner for a machine (a scanner, link checker or script),
// person for a person's device or app a rule takes for a machine.
Kind string `json:"kind" enum:"scanner,person"`
Name string `json:"name" minLength:"1" maxLength:"80" doc:"Who or what it is: our mail gateway, lobby kiosks"`
Note string `json:"note,omitempty" maxLength:"300"`
}
TrackingUserAgentInput adds a user agent.
type TrackingUserAgentList ¶
type TrackingUserAgentList struct {
Items []TrackingUserAgent `json:"items"`
}
TrackingUserAgentList lists the user agents.
type TransactionalTracking ¶
type TransactionalTracking interface {
TrackedLink(ctx context.Context, q *store.Queries, messageID uuid.UUID, position int) (string, bool)
Engagement(ctx context.Context, tx pgx.Tx, messageID uuid.UUID, click bool, at time.Time) error
}
TransactionalTracking is what the tracking endpoints need from transactional email: each message keeps its own tracked links (their targets differ per recipient), and engagement fires webhooks.
type Verdict ¶
type Verdict struct {
// Machine is true for opens and clicks no person made: privacy proxies
// prefetching images, security scanners following links.
Machine bool
// Class is ClassHuman, ClassPrivacy or ClassMachine for an open,
// ClassHuman or ClassBot for a click.
Class string
// Reason explains a machine verdict in the live feed's words
// ("Apple Mail Privacy Protection prefetch").
Reason string
// Client is the mail client or browser, for human events
// ("Chrome on macOS").
Client string
}
Verdict classifies an open or a click.
func ClassifyClick ¶
ClassifyClick decides whether a click was a person. Bot clicks still get redirected; they are logged but never counted as clicks.
Bot clicks: HEAD requests, empty or scanner user agents, scanner networks within two minutes of sending, clicks within ten seconds of sending, and bursts (burst is true when the same message already clicked several different links within the last few seconds, as scanners that follow every link do).
func ClassifyHoneypot ¶
func ClassifyHoneypot() Verdict
ClassifyHoneypot is a click on the hidden link: no person can see it.
func ClassifyOpen ¶
ClassifyOpen decides whether an open pixel fetch was a person.
Machine opens, in order:
- HEAD requests and empty user agents.
- Apple Mail Privacy Protection: requests from Apple's network (17.0.0.0/8) or with the bare "Mozilla/5.0" user agent its proxies send. They fetch every image on delivery whether or not anyone reads.
- Gmail's GoogleImageProxy and Yahoo's YahooMailProxy within 30 s of sending (caching on arrival); later fetches are people opening in Gmail or Yahoo Mail and count as human.
- Security scanners and HTTP libraries (user agent), and scanner networks within two minutes of sending.
- Anything within two seconds of sending.