embed

type Doc ¶

type Doc = textproc.Doc

Doc is a weighted bag of terms, produced by the textproc package.

type Embedder ¶

type Embedder interface {
	Name() string
	Embed(ctx context.Context, docs []Doc) (*Space, error)
}

Embedder builds a Space from the corpus. Docs are index-aligned with the repositories being categorized.

type LSA ¶

type LSA struct {
	// Base produces the term matrix that gets factored.
	Base *TFIDF
	// Dims is the number of latent dimensions to keep.
	Dims int
	// Oversample widens the random sketch. A sketch of exactly Dims columns
	// captures the leading subspace poorly; a handful of extra columns costs
	// almost nothing and markedly improves the result.
	Oversample int
	// Power is the number of power iterations. Each one pushes the sketch
	// further towards the dominant singular directions, which matters because
	// a term-document matrix has a slowly decaying spectrum.
	Power int
	Seed  int64
}

LSA is latent semantic analysis: a truncated singular value decomposition of the TF-IDF matrix.

It exists to fix what TF-IDF alone cannot. Term matching only groups repositories that share vocabulary, so a "container runtime" and an "OCI sandbox" stay apart however obviously they belong together. Factoring the matrix replaces terms with a few hundred latent directions built out of terms that co-occur, and the two land in the same place without sharing a single word.

The decomposition is randomized, following Halko, Martinsson and Tropp: a small random sketch of the matrix captures its dominant subspace, which is far cheaper than a full SVD and accurate enough at this rank. It stays entirely offline: nothing about it leaves the machine.

func (l *LSA) Embed(ctx context.Context, docs []Doc) (*Space, error)

func (l *LSA) Name() string

type Space ¶

type Space struct {
	Dim     int
	Rows    []Vector
	Terms   []string
	Backend string
}

Space is a corpus embedded into a shared vector space. Terms maps a dimension back to the term that produced it, when the backend has one; it is nil for opaque neural embeddings.

type TFIDF ¶

type TFIDF struct {
	// MinDF drops terms that appear in fewer than this many repositories; they
	// carry no grouping signal and only add noise.
	MinDF int
	// MaxDFRatio drops terms that appear in more than this fraction of the
	// corpus, which is how "kubernetes" stops dominating a Kubernetes-heavy
	// star list.
	MaxDFRatio float64
	// MaxVocab caps the vocabulary, keeping the highest-IDF-weighted terms.
	MaxVocab int

	// BM25 switches term weighting from classic TF-IDF to Okapi BM25: term
	// frequency saturates, so the twentieth occurrence of a word adds almost
	// nothing over the fifth, and document length is corrected for by a pivot
	// rather than merely rescaled. It is the textbook choice for a corpus of
	// uneven document lengths, which is what READMEs create here.
	BM25 bool
	// K1 sets how quickly term frequency saturates; B how much length counts.
	K1 float64
	B  float64
}

TFIDF builds the weighted term matrix: sublinear term frequency times smoothed inverse document frequency, restricted to a vocabulary of the most discriminative terms.

It is not offered as a backend of its own. Grouping repositories directly on these vectors only finds those that share wording, which is the limitation LSA exists to remove; this is the matrix LSA factors.

func (t *TFIDF) Embed(_ context.Context, docs []Doc) (*Space, error)

func (t *TFIDF) Name() string

type Vector ¶

type Vector struct {
	Idx []int32
	Val []float32
}

Vector is a sparse L2-normalized vector. Idx is sorted ascending and the same length as Val. TF-IDF produces genuinely sparse rows; dense backends simply fill every index.

func Normalize(idx []int32, val []float32) Vector

func (v Vector) Dot(centroid []float64) float64

func (v Vector) DotSparse(o Vector) float64