Documentation
¶
Index ¶
- Constants
- Variables
- func CosineSimilarity(a, b []float64) (float64, error)
- func Extract(text string, opts ExtractOptions) string
- func Register(registrar interface{ ... })
- func Sentences(text string) []string
- func ToFloat64Slice(obj object.Object) ([]float64, error)
- func VectorFromText(text string, dims int) []float64
- type ExtractOptions
- type ScoredVector
Constants ¶
const ( LibraryName = "scriptling.similarity" LibraryDesc = "String matching and similarity utilities including fuzzy search, MinHash and extractive text shortening" )
const DefaultExtractRatio = 0.3
DefaultExtractRatio is the fraction of sentences kept when Extract is given no bounds.
Variables ¶
var ( ErrDimensionMismatch = errors.New("vectors must have the same length") ErrEmptyVector = errors.New("vectors must not be empty") )
Functions ¶
func CosineSimilarity ¶ added in v0.17.0
CosineSimilarity computes the cosine of the angle between two vectors. Returns 0.0 if either vector has zero magnitude.
func Extract ¶ added in v0.28.0
func Extract(text string, opts ExtractOptions) string
Extract returns the most informative sentences of text, in their original order, within the given bounds. Text that already fits is returned unchanged.
Sentences are scored with TextRank: each sentence is a node, edges are weighted by the cosine similarity of the sentences' hashed word vectors, and the stationary distribution ranks them, so the sentences most representative of the whole text score highest. The similarity matrix is built in parallel across CPUs.
Repetitive text (many near-identical lines once digits are normalised, as in logs) is handled differently: the first and last few lines are kept along with the first occurrence of each distinct line, in order, because there the unusual lines are the informative ones.
func Sentences ¶ added in v0.28.0
Sentences splits text into sentences. A sentence ends at ". ", "! " or "? " (the terminator may be followed by closing quotes or brackets) when the next character starts a new sentence, and at every line break, so transcripts, chat logs and pasted output split one line per sentence. The returned sentences are trimmed and never empty.
func ToFloat64Slice ¶ added in v0.17.0
ToFloat64Slice converts a scriptling Object (FloatArray or List of numbers) to a []float64. Shared by ai and similarity libraries.
func VectorFromText ¶ added in v0.17.0
VectorFromText produces a fixed-dimensional vector from text using the feature-hashing trick (the "hashing trick"). Each token is mapped to a dimension via FNV-1a and contributes +1 or −1 based on a sign hash. The resulting vector is L2-normalised so it can be compared directly with CosineSimilarity. This is a fast, deterministic, CPU-only method that captures lexical overlap — similar texts produce similar vectors.
Types ¶
type ExtractOptions ¶ added in v0.28.0
type ExtractOptions struct {
MaxChars int // keep sentences until adding the next would exceed this many characters
MaxSentences int // keep at most this many sentences
Ratio float64 // keep this fraction of the sentences (0 < Ratio <= 1)
}
ExtractOptions bounds the output of Extract. Zero means no bound on that axis. When every bound is zero, Ratio defaults to DefaultExtractRatio.
type ScoredVector ¶ added in v0.17.0
ScoredVector is a ranked result from MostSimilar.
func MostSimilar ¶ added in v0.17.0
func MostSimilar(query []float64, vectors [][]float64, topK int) []ScoredVector
MostSimilar ranks vectors by cosine similarity to query, returning the top-k results sorted by descending score. If topK <= 0 all results are returned.
Source Files