textutil

package

v0.0.5 Latest Latest Go to latest Published: Feb 9, 2026 License: MIT Imports: 4 Imported by: 0

Details

Package textutil provides text processing utilities for form classification.

This section is empty.

This section is empty.

func Ngrams(s string, minN, maxN int) []string

Ngrams returns min_n to max_n character-level n-grams of the given string.

func Normalize(text string) string

Normalize lowercases text and normalizes whitespace.

func NormalizeWhitespaces(text string) string

NormalizeWhitespaces replaces newlines and multiple whitespace with a single space.

func NumberPattern(text string, ratio float64) string

NumberPattern replaces digits with X and letters with C if the digit ratio >= threshold. Returns empty string otherwise.

func TokenNgrams(tokens []string, minN, maxN int) []string

TokenNgrams returns n-grams from a list of tokens, joined by space.

func Tokenize(text string) []string

Tokenize extracts word tokens from text (Unicode-aware, matching Python's (?u)\b\w+\b).

This section is empty.