contentcheck

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: AGPL-3.0 Imports: 27 Imported by: 0

Documentation

Overview

Package contentcheck finds deliverability problems in an email's content: size against Gmail's 102 KB clipping limit, images without alt text, the image-to-text ratio, links on URI blocklists, URL shorteners and broken links, a missing unsubscribe link or postal address, Liquid errors, spam heuristics in the subject, and the plain-text part.

The composer runs the fast checks inline as people edit; Send Readiness runs the full set (including link probing) before a send.

Index

Constants

View Source
const (
	TroubleTimeout = "timeout" // no answer in time
	TroubleRefused = "refused" // the list refused the query (public resolver, rate limit)
	TroubleOther   = "other"   // no usable answer
)

Kinds of lookup trouble (BlocklistReport.Unknown).

View Source
const (
	StageLinks       = "links"        // requesting each link
	StageLinkDomains = "link_domains" // looking link domains up on URI blocklists
)

Stages a check reports progress on (WithProgress).

View Source
const (
	Block = "block"
	Warn  = "warn"
	Info  = "info"
	Pass  = "pass"
)

Severities, in the order checks are listed.

View Source
const (
	IssueFooter         = "compliance.footer"
	IssueLiquidSyntax   = "personalization.syntax"
	IssueSubjectMissing = "subject.missing"
	IssueLinkBlocklist  = "links.blocklist"
)

The checks whose issues block a send.

View Source
const ClipBytes = 102 * 1024

ClipBytes is the size at which Gmail clips a message (102 KB).

View Source
const ImageShareLimit = 0.6

ImageShareLimit is the share of the email's area images may cover before the content checks warn (one threshold for the composer and Send Readiness): image-heavy mail scores higher with spam filters.

View Source
const MaxProbedLinks = 100

MaxProbedLinks is how many links one run requests: a run is a background job (Send Readiness), so a long digest's links are all requested.

View Source
const NearClipBytes = 85 * 1024

NearClipBytes is where the size warning starts.

Variables

View Source
var ErrBlockedAddress = errors.New("the link points at an internal address")

ErrBlockedAddress is returned when a link resolves to an internal address.

View Source
var URIListNames = []string{"Spamhaus DBL", "SURBL"}

URIListNames are the URI blocklists link domains are looked up on.

Functions

func Blocking

func Blocking(id string) bool

Blocking reports whether a check's issues block a send. Such an issue can't be dismissed: it is fixed or the email isn't sent.

func BodyHeuristics

func BodyHeuristics(text, language string) []string

BodyHeuristics returns the spam heuristics an email's text trips: spam phrases repeated through the body (three or more urgency, money or click-bait phrases), and repeated exclamation marks. A word or two is normal; filters weigh the pile-up.

func Minify

func Minify(src string) string

Minify shrinks email HTML without changing how it renders: whitespace between tags and inside style attributes, and comments other than Outlook's conditional comments.

func PlainText

func PlainText(src string) string

PlainText turns email HTML into the text/plain part: one block per line, headings in capitals, buttons and standalone links as "Label: URL", the footer after a "--" separator, hidden preview text and comments dropped.

func PublicOnly

func PublicOnly(_, address string, _ syscall.RawConn) error

PublicOnly is a net.Dialer Control function that refuses non-public IPs (netguard's policy, private networks never allowed).

func RegisterBlocking

func RegisterBlocking(ids ...string)

RegisterBlocking names checks another package adds whose issues block a send (a Sponsor block waiting for its creative), so they can't be dismissed either.

func RegistrableDomain

func RegistrableDomain(host string) string

RegistrableDomain returns the domain someone registers ("acme.io" for "news.acme.io").

func SaveAltLabel

func SaveAltLabel(l i18n.Lang) string

SaveAltLabel is the label of the fix that saves an image's alt text ("Save"), in a reader's language, for other checks that offer the same fix (a transactional template's accessibility rows).

func SectionContainer

func SectionContainer(doc *html.Node) *html.Node

SectionContainer returns the element whose children are the email's top-level sections in MJML output (body > div > div…), or nil.

func SubjectHeuristicPhrase

func SubjectHeuristicPhrase(subject, language string) string

SubjectHeuristicPhrase returns the words of a subject that trip a phrase heuristic ("you won't believe"), "" when only its form does (capitals, punctuation).

func SubjectHeuristics

func SubjectHeuristics(subject string) []string

SubjectHeuristics returns the spam heuristics a subject trips: "all caps", "repeated punctuation", "urgency", "money words".

func SubjectHeuristicsIn

func SubjectHeuristicsIn(subject, language string) []string

SubjectHeuristicsIn returns the heuristics a subject in a language trips: the English phrase lists (urgency, money words) apply to English and to emails without a language; other languages get the language-neutral ones (capitals, punctuation).

func SuggestSubject

func SuggestSubject(subject string) string

SuggestSubject rewrites a shouty subject so that it trips none of the heuristics: sentence case, repeated punctuation collapsed and urgency softened ("FLASH SALE!!! 48 HOURS ONLY" → "48-hour flash sale"). It returns "" when no such rewrite exists.

func TextIn

func TextIn(l i18n.Lang, t i18n.Text) string

TextIn writes a message this package kept in a reader's language. Its "section" argument is a section label (mjml.SectionLabel), named in that language too, and "lists" a list of names, joined the way the language joins them. Messages of other packages are written as i18n.Text.In writes them. It returns "" for a message no catalog has.

func URILists

func URILists() []dnscheck.Blocklist

URILists are the URI blocklists link domains are looked up on, from the built-in lists: Spamhaus DBL and SURBL.

func WithProgress

func WithProgress(ctx context.Context, p Progress) context.Context

WithProgress has the checks run with ctx report their progress (Send Readiness shows "5 of 8 domains").

Types

type Blocklist

type Blocklist interface {
	// Listings returns, for each listed domain, the lists it is on. Clean
	// and unknown domains are absent.
	Listings(ctx context.Context, domains []string) (map[string][]string, error)
}

Blocklist answers which URI blocklists list a domain.

type BlocklistChecker

type BlocklistChecker interface {
	Blocklist
	Check(ctx context.Context, domains []string) (BlocklistReport, error)
}

BlocklistChecker is a Blocklist that also says which domains it couldn't check: a failed lookup is unknown, never clean.

type BlocklistReport

type BlocklistReport struct {
	// Listed maps a listed domain to the lists it is on.
	Listed map[string][]string
	// Unknown maps a domain that couldn't be checked to why (a Trouble*
	// kind): DNS didn't answer, or the list refused the resolver.
	Unknown map[string]string
	// Lists names the lists the domains were looked up on.
	Lists []string
	// Domains is how many domains were checked (cached or live).
	Domains int
}

BlocklistReport is what a live lookup of link domains found.

type Checker

type Checker struct {
	Blocklist Blocklist
	Prober    Prober
	Now       func() time.Time
	// Extra checks run after the built-in ones.
	Extra []Extra
}

func (*Checker) Check

func (c *Checker) Check(ctx context.Context, in Input) (ContentCheckReport, error)

Check runs every check on the content.

type ContentCheckFix

type ContentCheckFix struct {
	Kind  string `json:"kind" enum:"replace_link,edit_link,insert_footer,set_alt,minify,remove_section,use_subject,open"`
	Label string `json:"label"`
	Value string `json:"value,omitempty"`
	Path  []int  `json:"path,omitempty"`
	// LabelText is Label as a message, to write it in a reader's language
	// (ContentCheckReport.In).
	LabelText i18n.Text `json:"-"`
}

ContentCheckFix is a one-click fix offered with an issue. The composer applies it to the canvas (Path points at the component) or to the campaign settings; an open fix opens the page in Value (a Sponsor block's booking).

type ContentCheckImage

type ContentCheckImage struct {
	URL string `json:"url"`
	// Host is where the image loads from; empty for the media library's
	// images (relative until the send makes them absolute) and for images
	// whose address is personalized.
	Host string `json:"host,omitempty"`
	// Insecure is true for an image loaded over plain HTTP.
	Insecure bool  `json:"insecure,omitempty"`
	Path     []int `json:"path"`
	// Probed is true when the image was requested (Send Readiness); Status,
	// Broken and Bytes (the size the server reported, 0 when it didn't)
	// describe the response.
	Probed bool  `json:"probed,omitempty"`
	Status int   `json:"status,omitempty"`
	Broken bool  `json:"broken,omitempty"`
	Bytes  int64 `json:"bytes,omitempty"`
}

ContentCheckImage is one image of the email.

type ContentCheckIssue

type ContentCheckIssue struct {
	ID       string `json:"id" example:"links.blocklist"`
	Severity string `json:"severity" enum:"block,warn,info,pass"`
	Group    string `json:"group" enum:"Content,Links,Compliance,Subject,Personalization"`
	Title    string `json:"title"`
	Detail   string `json:"detail,omitempty"`
	// Tag is the short label shown on the canvas next to the element.
	Tag string `json:"tag,omitempty"`
	// Path locates the element from mj-body (element indices, comments
	// excluded); empty when the issue concerns the whole email.
	Path []int `json:"path"`
	// Section is the top-level section index, -1 for the whole email.
	Section int               `json:"section"`
	Fixes   []ContentCheckFix `json:"fixes"`
	// URL is the link a link issue is about (to find it inside text).
	URL string `json:"url,omitempty"`
	// TitleText, DetailText (one message a sentence) and TagText are the
	// issue's words as messages, to write them in a reader's language
	// (ContentCheckReport.In, TextIn). Title, Detail and Tag are English.
	// Issues another domain adds without them keep their English.
	TitleText  i18n.Text   `json:"-"`
	DetailText []i18n.Text `json:"-"`
	TagText    i18n.Text   `json:"-"`
}

ContentCheckIssue is one finding.

func (ContentCheckIssue) In

In writes one issue's words in a reader's language.

type ContentCheckLink struct {
	URL  string `json:"url"`
	Host string `json:"host"`
	Text string `json:"text"`
	Kind string `json:"kind" enum:"button,link,image,nav,social"`
	Path []int  `json:"path"`
	// Probed is true when the link was requested; Status, Broken, Millis
	// and Final (the address after redirects) describe the response.
	Probed bool   `json:"probed"`
	Status int    `json:"status,omitempty"`
	Broken bool   `json:"broken,omitempty"`
	Millis int    `json:"millis,omitempty"`
	Final  string `json:"final,omitempty"`
	// FinalClean is true when Final, the address the link really leads to,
	// was looked up on the URI blocklists and found on none: only then is a
	// listed or shortened link offered "Replace with destination URL".
	FinalClean bool `json:"finalClean,omitempty"`
	// BotBlocked is true when the site refused the automated request.
	BotBlocked bool `json:"botBlocked,omitempty"`
}

ContentCheckLink is one link of the email.

type ContentCheckReport

type ContentCheckReport struct {
	Verdict          string `json:"verdict" enum:"ready,warnings,blocked"`
	Blocks           int    `json:"blocks"`
	Warnings         int    `json:"warnings"`
	SizeBytes        int    `json:"sizeBytes"`
	ClipBytes        int    `json:"clipBytes"`
	Images           int    `json:"images"`
	ImagesMissingAlt int    `json:"imagesMissingAlt"`
	Links            int    `json:"links"`
	LinksBlocklisted int    `json:"linksBlocklisted"`
	LinksBroken      int    `json:"linksBroken"`
	// SubjectHeuristics, PreheaderHeuristics and BodyHeuristics name the
	// spam heuristics the subject, the preheader and the body's text trip.
	SubjectHeuristics   []string              `json:"subjectHeuristics"`
	PreheaderHeuristics []string              `json:"preheaderHeuristics"`
	BodyHeuristics      []string              `json:"bodyHeuristics"`
	Issues              []ContentCheckIssue   `json:"issues"`
	Sections            []ContentCheckSection `json:"sections"`
	PlainText           string                `json:"plainText"`
	CheckedAt           time.Time             `json:"checkedAt"`
	// ImageShare is the estimated share of the email's area covered by
	// images (0 to 1; 0 without images). Words counts the plain text.
	ImageShare float64 `json:"imageShare"`
	Words      int     `json:"words"`
	// ImagesInsecure counts images loaded over plain HTTP; ImageList is
	// every image with, when images were requested, the response.
	ImagesInsecure int                 `json:"imagesInsecure"`
	ImageList      []ContentCheckImage `json:"imageList"`
	// LinkList is every link with its text and, when links were probed, the
	// response (Send Readiness shows the slowest and the link domains).
	LinkList []ContentCheckLink `json:"linkList"`
	// LinkDomains counts the link domains looked up on the URI blocklists
	// (URILists names them); LinkDomainsUnknown are those whose lookup
	// failed: unknown, not clean.
	LinkDomains        int                         `json:"linkDomains"`
	URILists           []string                    `json:"uriLists"`
	LinkDomainsUnknown []ContentCheckUnknownDomain `json:"linkDomainsUnknown"`
}

ContentCheckReport is the outcome of a check run.

func (*ContentCheckReport) AddIssues

func (r *ContentCheckReport) AddIssues(issues ...ContentCheckIssue)

AddIssues adds findings of other domains (the Sponsor block's) to a report: they count in its verdict and in their sections.

func (ContentCheckReport) In

In writes a report's words in a reader's language: its issues' titles, details, canvas tags and fix labels. Check writes English, and so does In for English readers; issues another domain added without their messages keep their English.

type ContentCheckSection

type ContentCheckSection struct {
	Index         int    `json:"index"`
	Label         string `json:"label"`
	Images        int    `json:"images"`
	ImagesWithAlt int    `json:"imagesWithAlt"`
	Links         int    `json:"links"`
	LinksOK       int    `json:"linksOk"`
	Issues        int    `json:"issues"`
}

ContentCheckSection is what the inspector shows for the selected section.

type ContentCheckUnknownDomain

type ContentCheckUnknownDomain struct {
	Domain string `json:"domain"`
	// Trouble is why: timeout, refused (the list refused the resolver) or other.
	Trouble string `json:"trouble" enum:"timeout,refused,other"`
}

ContentCheckUnknownDomain is a link domain that couldn't be looked up.

type Extra

type Extra func(ctx context.Context, in Input) []ContentCheckIssue

Extra is a check another domain adds (X20: a store block naming a product the store removed). It returns its issues; Section and Path say where they are.

type HTTPProber

type HTTPProber struct {
	Client *http.Client
	// MaxLinks caps how many links one run requests.
	MaxLinks int
	// Concurrency is how many requests run at once.
	Concurrency int
	// contains filtered or unexported fields
}

HTTPProber requests links with HEAD (GET when HEAD isn't allowed), with short timeouts and an SSRF guard: it never connects to loopback, private, link-local or otherwise internal addresses, also after redirects.

func NewHTTPProber

func NewHTTPProber() *HTTPProber

NewHTTPProber returns a prober with the SSRF guard on its dialer.

func (*HTTPProber) Probe

func (p *HTTPProber) Probe(ctx context.Context, urls []string) map[string]ProbeResult

Probe implements Prober. Results are cached for ten minutes.

type Input

type Input struct {
	Subject   string
	Preheader string
	MJML      string
	HTML      string
	// Address is the postal address from the workspace's branding, if set.
	Address string
	// Fields are the personalization tokens that exist (liquid.Catalog keys).
	Fields []string
	// ProbeLinks requests each link (Send Readiness); the composer skips it.
	ProbeLinks bool
	// Language is the language version's code ("de"); the subject's
	// English phrase heuristics only apply to English (or no language).
	Language string
	// WorkspaceID is whose content it is (Extra checks read its data).
	WorkspaceID uuid.UUID
}

Input is the content to check.

type LiveBlocklist

type LiveBlocklist struct {
	Q   store.Querier
	DNS dnscheck.Resolver
	// Lists default to URILists.
	Lists []dnscheck.Blocklist
	// MaxAge is how long an answer is reused (default an hour).
	MaxAge time.Duration
	// Budget bounds the live lookups of one check (default 30 s): the
	// composer, which checks as people type, uses a short one.
	Budget time.Duration
	Now    func() time.Time
}

LiveBlocklist looks link domains up on Spamhaus DBL and SURBL over DNS and keeps the answers in link_domain_listings, shared by every workspace: an answer younger than MaxAge is reused, an older one is looked up again. A lookup that fails is unknown and isn't kept. The composer's inline checks and Send Readiness both use it.

func (*LiveBlocklist) Check

func (b *LiveBlocklist) Check(ctx context.Context, domains []string) (BlocklistReport, error)

Check implements BlocklistChecker.

func (*LiveBlocklist) Listings

func (b *LiveBlocklist) Listings(ctx context.Context, domains []string) (map[string][]string, error)

Listings implements Blocklist.

type ProbeResult

type ProbeResult struct {
	Status int  `json:"status"`
	Broken bool `json:"broken"`
	// BotBlocked is true when the site refused an automated request (999,
	// 429): whether the link works isn't known.
	BotBlocked bool   `json:"botBlocked,omitempty"`
	Error      string `json:"error,omitempty"`
	// Bytes is the size the server reported (Content-Length), 0 when it
	// didn't.
	Bytes int64 `json:"bytes,omitempty"`
	// Millis is how long the response took; Final is the address after
	// redirects (empty when the link didn't redirect).
	Millis int    `json:"millis,omitempty"`
	Final  string `json:"final,omitempty"`
}

ProbeResult is the outcome of requesting one link.

type Prober

type Prober interface {
	Probe(ctx context.Context, urls []string) map[string]ProbeResult
}

Prober requests links to find broken ones.

type Progress

type Progress func(stage string, done, total int)

Progress is told how far a stage of a check got: done of total.

func ProgressFrom

func ProgressFrom(ctx context.Context) Progress

ProgressFrom is the progress observer of ctx, or one that ignores it.

type StaticBlocklist

type StaticBlocklist map[string][]string

StaticBlocklist is a fixed map, for tests and installs without lookups.

func (StaticBlocklist) Listings

func (b StaticBlocklist) Listings(_ context.Context, domains []string) (map[string][]string, error)

Listings implements Blocklist.

type StoreBlocklist

type StoreBlocklist struct {
	Q store.Querier
}

StoreBlocklist reads URI blocklist results cached in link_domain_listings. Deliverability lookups (DNS, HostRepute) write that table; content checks only read it, so they stay fast.

func (StoreBlocklist) Listings

func (b StoreBlocklist) Listings(ctx context.Context, domains []string) (map[string][]string, error)

Listings implements Blocklist.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL