cleaners

func CleanAuthor ¶

func CleanAuthor(author string) string

CleanAuthor takes an author string (like 'By David Smith ') and cleans it to just the name(s): 'David Smith'.

This is a faithful 1:1 port of the JavaScript cleanAuthor function: - Removes "By", "Posted by", "Written by" prefixes (case insensitive) - Handles optional colons after prefixes - Normalizes all whitespace to single spaces - Trims leading and trailing whitespace

JavaScript equivalent:

export default function cleanAuthor(author) {
  return normalizeSpaces(author.replace(CLEAN_AUTHOR_RE, '$2').trim());
}

func CleanDatePublished ¶

func CleanDatePublished(dateString, timezone, format string) *string

CleanDatePublished takes a date published string and returns a clean ISO date string. Returns nil if the date cannot be parsed or is invalid.

This is a faithful 1:1 port of the JavaScript cleanDatePublished function: - Handles millisecond/second timestamps - Supports relative time expressions ("5 minutes ago") - Handles "now" time indicators - Supports timezone and format parameters - Cleans date strings by removing "published:" prefixes - Returns ISO 8601 formatted string or nil for invalid dates

JavaScript equivalent:

export default function cleanDatePublished(dateString, { timezone, format } = {}) {
  // Timestamp handling, date cleaning, and parsing logic
  return date.isValid() ? date.toISOString() : null;
}

func CleanDek ¶

func CleanDek(dek string, doc *goquery.Document, excerpt string) *string

CleanDek takes a dek HTML fragment and returns the cleaned version of it. Returns nil if the dek wasn't good enough (too short, too long, has URLs, matches excerpt).

This is a faithful 1:1 port of the JavaScript cleanDek function: - Validates length between 5 and 1000 characters - Checks that dek isn't the same as excerpt (first 10 words) - Strips HTML tags using stripTags function - Rejects deks containing plain text URLs (http/https) - Normalizes whitespace using normalizeSpaces

JavaScript equivalent:

export default function cleanDek(dek, { $, excerpt }) {
  if (dek.length > 1000 || dek.length < 5) return null;
  if (excerpt && excerptContent(excerpt, 10) === excerptContent(dek, 10)) return null;
  const dekText = stripTags(dek, $);
  if (TEXT_LINK_RE.test(dekText)) return null;
  return normalizeSpaces(dekText.trim());
}

func CleanDomainFromTitle ¶

func CleanDomainFromTitle(splitTitle []string, urlStr string) string

CleanDomainFromTitle removes domain name matches from title segments Search the ends of the title, looking for bits that fuzzy match the URL too closely. If one is found, discard it and return the rest.

func CleanLeadImageURL ¶

func CleanLeadImageURL(imageURL, baseURL string) string

CleanLeadImageURL ensures image URLs are properly formatted and absolute

func CleanLeadImageURLString ¶

func CleanLeadImageURLString(leadImageURL string) string

CleanLeadImageURLString provides a string-returning version for backward compatibility Returns empty string if URL is invalid, cleaned URL if valid

func CleanLeadImageURLValidated ¶

func CleanLeadImageURLValidated(leadImageURL string) *string

CleanLeadImageURLValidated validates and cleans a lead image URL Returns nil if the URL is invalid, cleaned URL string if valid Matches JavaScript behavior: trim whitespace and validate as web URI

func CleanTitle ¶

func CleanTitle(title string, url string, doc *goquery.Document) string

CleanTitle cleans and normalizes title text by removing site names, HTML tags, and extra whitespace This is a faithful port of the JavaScript cleanTitle function

func CleanTitleSimple ¶

func CleanTitleSimple(title, targetURL string) string

CleanTitleSimple provides a simple title cleaner that doesn't need a document

func ExtractBreadcrumbTitle ¶

func ExtractBreadcrumbTitle(splitTitle []string, text string) string

ExtractBreadcrumbTitle extracts the most relevant title from breadcrumb-style titles This must be a very breadcrumbed title, like: The Best Gadgets on Earth : Bits : Blogs : NYTimes.com NYTimes - Blogs - Bits - The Best Gadgets on Earth

func ExtractCleanNode ¶

func ExtractCleanNode(article *goquery.Selection, doc *goquery.Document, opts ContentCleanOptions) *goquery.Selection

ExtractCleanNode cleans article content, returning a new, cleaned node Direct port of JavaScript extractCleanNode function with identical cleaning pipeline:

1. rewriteTopLevel - Convert HTML/BODY tags to DIV to avoid complications 2. cleanImages - Remove small/spacer images (if defaultCleaner enabled) 3. makeLinksAbsolute - Convert relative URLs to absolute URLs 4. markToKeep - Mark video iframes and important elements for preservation 5. stripJunkTags - Remove script, style, title and other junk tags 6. cleanHOnes - Remove or convert H1 tags based on count 7. cleanHeaders - Clean headers that match article title 8. cleanTags - Remove low-quality tags with high link density (if defaultCleaner enabled) 9. removeEmpty - Remove empty paragraph and other empty elements 10. cleanAttributes - Remove unnecessary attributes

This function matches the JavaScript implementation exactly, including: - Same cleaning order and logic - Same conditional cleaning based on options - Same default behaviors for aggressive vs conservative cleaning

func LevenshteinRatio ¶

func LevenshteinRatio(s1, s2 string) float64

LevenshteinRatio calculates the Levenshtein similarity ratio between two strings This is compatible with the JavaScript wuzzy.levenshtein function

func RegisterCleaner ¶

func RegisterCleaner(fieldType string, cleaner FieldCleaner)

RegisterCleaner registers a new cleaner for a field type

func ResolveSplitTitle ¶

func ResolveSplitTitle(title, url string) string

ResolveSplitTitle resolves whether any of the segments should be removed from a title with separators Given a title with separators in it (colons, dashes, etc), resolve whether any of the segments should be removed.

func SplitTitleWithSeparators ¶

func SplitTitleWithSeparators(title string) []string

SplitTitleWithSeparators splits title while preserving separators This mimics JavaScript's split behavior with capturing groups