text

func ArticleBaseURL ¶

func ArticleBaseURL(urlStr string, parsedURL *url.URL) string

ArticleBaseURL takes a URL and returns the article base of said URL. That is, no pagination data exists in it. Useful for comparing to other links that might have pagination data within them.

This is a faithful port of the JavaScript articleBaseUrl function.

Parameters:

• urlStr: The URL string to process
• parsedURL: Optional pre-parsed URL (can be nil)

Returns:

• string: The base URL with pagination data removed

JavaScript equivalent:

export default function articleBaseUrl(url, parsed) {
  const parsedUrl = parsed || URL.parse(url);
  const { protocol, host, path } = parsedUrl;
  ...
}

func ExcerptContent ¶

func ExcerptContent(content string, words ...int) string

func ExtractFromURL ¶

func ExtractFromURL(url string, regexList []*regexp.Regexp) (string, bool)

ExtractFromURL searches for patterns in a URL and returns the first capture group from the first matching regex. Given a URL and a list of regular expressions, this function tests each regex against the URL and returns the first capture group (group 1) from the first matching pattern. This is primarily used for extracting date information from URLs in date published extraction.

Parameters:

• url: The URL string to search within
• regexList: A slice of compiled regular expressions to test against the URL

Returns:

• string: The first capture group from the first matching regex, or empty string if no match
• bool: true if a match was found, false otherwise

The function expects each regex to have at least one capture group, and will return the content of the first capture group from the first regex that matches the URL.

func FormatDateForJSON ¶

func FormatDateForJSON(t *time.Time) string

FormatDateForJSON formats date for JSON output (compatible with JS version)

func GetEncoding ¶

func GetEncoding(str string) string

GetEncoding extracts and validates character encoding from a string. This function is a faithful port of the JavaScript getEncoding function. It checks a string for encoding using ENCODING_RE pattern and validates the charset exists before returning it, otherwise returns DEFAULT_ENCODING.

func IsGoodSegment ¶

func IsGoodSegment(segment string, index int, firstSegmentHasLetters bool) bool

IsGoodSegment determines if a URL segment should be kept or removed. This is a faithful port of the JavaScript isGoodSegment function.

JavaScript logic: - If segment is purely a number and it's in first/second position with < 3 chars: keep it - If segment is "index" in first position: remove it - If segment is < 3 chars in first/second position and first segment has no letters: remove it - Otherwise: keep it

func IsValidDate ¶

func IsValidDate(t *time.Time) bool

IsValidDate checks if a parsed date is reasonable

func NormalizeSpaces ¶

func NormalizeSpaces(text string) string

NormalizeSpaces normalizes consecutive whitespace characters to single spaces while preserving spacing within pre, code, and textarea HTML tags.

This function provides 100% compatibility with the JavaScript normalizeSpaces function: - Replaces 2+ consecutive whitespace characters with a single space - Preserves whitespace inside <pre>, <code>, and <textarea> tags - Trims leading and trailing whitespace from the result - Handles all types of whitespace: spaces, tabs, newlines, carriage returns

Example:

NormalizeSpaces("text   with    spaces") // returns "text with spaces"
NormalizeSpaces("<pre>  keep  spaces  </pre>") // returns "<pre>  keep  spaces  </pre>"

func PageNumFromURL ¶

func PageNumFromURL(url string) *int

PageNumFromURL extracts a page number from a URL string. This is a faithful port of the JavaScript pageNumFromUrl function.

The function looks for page number patterns in URLs like:

• page=1, pg=1, p=1
• paging=12, pag=7
• pagination/1, paging/88, pa/83, p/11

Returns:

• *int: the page number if found and < 100
• nil: if no page number found or page number >= 100

JavaScript equivalent:

export default function pageNumFromUrl(url) {
  const matches = url.match(PAGE_IN_HREF_RE);
  if (!matches) return null;
  const pageNum = parseInt(matches[6], 10);
  return pageNum < 100 ? pageNum : null;
}

func ParseDate ¶

func ParseDate(dateStr string) (*time.Time, error)

ParseDate attempts to parse a date string using various methods

func ParseDateFromMeta ¶

func ParseDateFromMeta(content string) (*time.Time, error)

ParseDateFromMeta parses dates from meta tag content

func RemoveAnchor ¶

func RemoveAnchor(url string) string

RemoveAnchor removes the anchor/fragment portion from a URL and trailing slashes This function provides 100% compatibility with the JavaScript removeAnchor function: - Splits URL on '#' and takes the first part (removes fragment) - Removes trailing slashes from the result