Documentation
¶
Overview ¶
Package quality holds the code quality checks that run before any quality question (spec §7 stage 3): properties of the URL, the HTTP status, the title and the text length that code decides exactly and for free. The gate (ingest) and classify (backfill) both use it.
Index ¶
Constants ¶
const ( NotAnArticle = "not_an_article" Thin = "thin" ErrorPage = "error_page" )
Flag codes (the `quality` / `triage_reason` values they map to).
const HostRepeatThreshold = 3
HostRepeatThreshold is the number of same-host sources a line must appear on to count as site chrome.
const MinWords = 300
MinWords is the body length, in words after removing navigation-like lines, under which a document is flagged thin.
Variables ¶
This section is empty.
Functions ¶
func ContentWords ¶
ContentWords counts the words of body that are not navigation: lines made only of links, only table separators, or repeated on at least HostRepeatThreshold same-host sources (hostLines) do not count.
func HostLineCounts ¶
HostLineCounts counts, per source host, on how many sources of that host each normalized non-empty line appears (a line repeated inside one source counts once). Documents without a source host are ignored.
func NavigationLine ¶
NavigationLine reports whether a line is navigation-like: only links (plus bullets and punctuation), only a `|`-table separator, or repeated on at least HostRepeatThreshold same-host sources.
func NormalizeLine ¶
NormalizeLine is the key HostLines uses: trimmed, lowercased, whitespace collapsed.
Types ¶
type Flag ¶
type Flag struct {
// Code is NotAnArticle, Thin or ErrorPage.
Code string `json:"code"`
// Detail says which rule fired, for run summaries and review evidence.
Detail string `json:"detail"`
}
Flag is one code quality finding.
type Input ¶
type Input struct {
Title string
Body string
// SourceURL is the stored source URL, used as the final URL when FinalURL
// is empty.
SourceURL string
// FinalURL is the URL after redirects, as reported by the fetcher.
FinalURL string
// RequestedURL is the URL the fetch was asked for.
RequestedURL string
// StatusCode is the HTTP status reported by the fetcher (0 = unknown).
StatusCode int
// SiteName is the site name reported by the fetcher; when empty it is
// derived from the host (first label without `www.`).
SiteName string
// HostLines maps a normalized line (NormalizeLine) to the number of
// same-host sources it appears on (see HostLineCounts); nil disables the
// repeated-line rule.
HostLines map[string]int
// SkipThin disables the thin rule (for inputs whose length is not a
// capture property, such as transcripts or local notes).
SkipThin bool
}
Input is everything the checks read. Empty fields disable the checks that need them (a document without a URL is never not_an_article by URL).
Source Files