textutil

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: MIT, MIT Imports: 6 Imported by: 0

README

textutil

Stdlib-only text helpers for keyword extraction, theme detection, fuzzy alias matching, fragment location, and title normalization.

import "github.com/dotcommander/reliquary/textutil"

keywords := textutil.ExtractKeywords([]string{
    "Go routines and channels make concurrency practical.",
    "Channels help coordinate goroutines in Go.",
}, textutil.KeywordOptions{Limit: 3, MinCount: 1})
// → []string{"channels", "concurrency", "coordinate"}

textutil.TitleWords("hello_world-foo")
// → "Hello World Foo"

match := textutil.AliasQueryScore("father of Jordan Morgan", "Jordan Morgan", nil, nil)
// → textutil.AliasMatch{Score: 1, Reason: "fuzzy_token"}

Install

go get github.com/dotcommander/reliquary/textutil

What it does

textutil provides three categories of helpers:

  • Keyword extraction — ExtractKeywords tokenizes a slice of texts, strips punctuation and stop words, and returns the top tokens by frequency.
  • Theme detection — DetectTheme scores text against named keyword lists and returns the best-matching theme name.
  • Fuzzy matching — AliasQueryScore scores person-name style query/canonical/alias matches and reports why the best match won.
  • Fragment location — FragmentRange returns byte offsets for exact or whitespace-normalized fragment matches.
  • Title normalization — TitleWords converts slug-style identifiers ("hello_world-foo") into human-readable labels ("Hello World Foo").

Supporting primitives: MostFrequentValue, NormalizeKeywordToken, IsStopWord, DefaultStopWords, DefaultStopWordsCopy, StringSimilarity, PhraseTerms, TextTerms, LongTermNearMatch.

DefaultStopWords is a compatibility snapshot for callers that use the old map-shaped API. DefaultStopWordsCopy() returns a fresh copy. Add domain-specific stop words with KeywordOptions.StopWords so concurrent callers do not share mutable global state.

See API reference for every function, option, and example.

Documentation

Overview

Package textutil provides lightweight, stdlib-only text helpers for keyword extraction, theme detection, fuzzy alias matching, fragment location, and title normalization.

Index

Examples

Constants

View Source 
const (
	PersonAliasMinScore      = 0.80
	PersonFuzzyTokenMinScore = 0.88
	LongTermMinLen           = 7
	LongTermShortMaxDistance = 1
	LongTermLongMaxDistance  = 2
	LongTermLongLen          = 10
	PhraseContainmentScore   = 0.96
)

Variables

View Source 
var DefaultStopWords = DefaultStopWordsCopy()

DefaultStopWords preserves the historical map-shaped API for callers that index, range, or pass the default stop-word set.

Package behavior reads an internal immutable default set, so mutating this exported compatibility snapshot never changes IsStopWord or ExtractKeywords. Use DefaultStopWordsCopy, KeywordOptions.StopWords, or KeywordOptions.Include for per-call filtering.

Functions

func DefaultStopWordsCopy 

func DefaultStopWordsCopy() map[string]struct{}

DefaultStopWordsCopy returns a copy of the package default stop-word set.

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	words := textutil.DefaultStopWordsCopy()
	words["channels"] = struct{}{}

	_, inCopy := words["channels"]
	fmt.Println(inCopy)
	fmt.Println(textutil.IsStopWord("channels"))
}

Output:
true
false

func DetectTheme 

func DetectTheme(content string, themeKeywords map[string][]string, fallback string) string
Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	themes := map[string][]string{
		"engineering": {"go", "chan", "channel"},
		"docs":        {"readme", "guide"},
	}

	// Match: "Go" scores for "engineering"; "channel" no longer matches "channels".
	fmt.Println(textutil.DetectTheme("This project uses Go channels heavily.", themes, "fallback"))

	// Fallback: no theme keyword appears in the content.
	fmt.Println(textutil.DetectTheme("No related language appears", themes, "fallback"))
}

Output:
engineering
fallback

func ExtractKeywords 

func ExtractKeywords(texts []string, options KeywordOptions) []string
Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	keywords := textutil.ExtractKeywords(
		[]string{
			"Go routines and channels make concurrency practical.",
			"Channels help coordinate goroutines in Go.",
		},
		textutil.KeywordOptions{Limit: 3, MinCount: 1},
	)
	fmt.Println(keywords)
}

Output:
[channels concurrency coordinate]
Example (PerCallStopWords) 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	keywords := textutil.ExtractKeywords(
		[]string{
			"Channels help coordinate channels in Go.",
			"Goroutines coordinate work through channels.",
		},
		textutil.KeywordOptions{
			Limit:     3,
			MinCount:  1,
			StopWords: map[string]struct{}{"channels": {}},
		},
	)
	fmt.Println(keywords)
}

Output:
[coordinate goroutines help]

func FragmentRange 

func FragmentRange(content, fragment string, cursor int, ord Order) (int, int, bool)

FragmentRange locates fragment in content and returns byte offsets. It first tries an exact match at the cursor, then falls back per ord: ExactFirst prefers any exact match over a normalized one; NormalizedEarly prefers a normalized match at/after the cursor over an exact match earlier in the content. Used by chunkers that join lines or collapse paragraph boundaries.

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	content := "hello\n   world"
	start, end, found := textutil.FragmentRange(content, "hello world", 0, textutil.NormalizedEarly)
	fmt.Println(found)
	fmt.Println(content[start:end])
}

Output:
true
hello
   world

func IsStopWord 

func IsStopWord(word string) bool
Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	fmt.Println(textutil.IsStopWord("the"))
	fmt.Println(textutil.IsStopWord("channels"))
}

Output:
true
false

func LongTermNearMatch 

func LongTermNearMatch(term string, textTerms []string) bool

func MostFrequentValue 

func MostFrequentValue(values []string, minCount int, fallback string) string

MostFrequentValue returns the most frequently occurring trimmed, non-empty value in values, provided it appears at least minCount times; otherwise it returns fallback. When multiple values share the top frequency, the alphabetically smallest value wins (deterministic tie-break via topCountedValues).

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	// Hit: "docs" appears twice, meeting minCount 2.
	fmt.Println(textutil.MostFrequentValue([]string{"docs", "api", "docs"}, 2, "fallback"))

	// Fallback: no value reaches minCount 2.
	fmt.Println(textutil.MostFrequentValue([]string{"docs", "api"}, 2, "fallback"))
}

Output:
docs
fallback

func NormalizeKeywordToken 

func NormalizeKeywordToken(token string) string
Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	fmt.Println(textutil.NormalizeKeywordToken("(channels)"))
	fmt.Println(textutil.NormalizeKeywordToken("go."))
}

Output:
channels
go

func PhraseTerms 

func PhraseTerms(s string, stop StopTermFunc) []string

func StringSimilarity 

func StringSimilarity(a, b string) float64

StringSimilarity returns the case-insensitive Jaro-Winkler similarity of a and b, a number in [0,1].

func TextTerms 

func TextTerms(text string) []string

func TitleWords 

func TitleWords(value string) string

TitleWords normalizes separators and title-cases words for display.

Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	fmt.Println(textutil.TitleWords("hello_world-foo"))
}

Output:
Hello World Foo

Types

type AliasMatch 

type AliasMatch struct {
	Score  float64
	Reason MatchReason
}

func AliasQueryScore 

func AliasQueryScore(query, canonical string, aliases []string, stop StopTermFunc) AliasMatch
Example 
package main

import (
	"fmt"

	"github.com/dotcommander/reliquary/textutil"
)

func main() {
	match := textutil.AliasQueryScore(
		"parent of Alex Quinn Examplf",
		"Alex Quinn Example",
		nil,
		nil,
	)
	fmt.Printf("%.2f %s\n", match.Score, match.Reason)
}

Output:
1.00 fuzzy_token

type KeywordOptions 

type KeywordOptions struct {
	Limit     int
	MinLength int
	MinCount  int
	StopWords map[string]struct{}
	Include   func(string) bool
}

type MatchReason 

type MatchReason string
const (
	ReasonNone          MatchReason = ""
	ReasonExactPhrase   MatchReason = "exact_phrase"
	ReasonAliasPhrase   MatchReason = "alias_phrase"
	ReasonTokenCoverage MatchReason = "token_coverage"
	ReasonFuzzyToken    MatchReason = "fuzzy_token"
)

type Order 

type Order int

Order controls how FragmentRange trades off exact and whitespace-normalized matches once the exact match at the cursor has missed. 

const (
	// ExactFirst tries an exact match anywhere (from the cursor, then from the
	// start of content) before any normalized match. This is the order the
	// chunking pipeline relies on.
	ExactFirst Order = iota
	// NormalizedEarly tries a normalized match at or after the cursor before
	// falling back to an exact match from the start of content.
	NormalizedEarly
)

type StopTermFunc 

type StopTermFunc func(string) bool

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.