sparse

func EnsureModelDownloaded ¶

func EnsureModelDownloaded(ctx context.Context) (string, error)

EnsureModelDownloaded pulls the model files into the local cache if missing. We only need the tokenizers for sparse vector generation.

func GenerateSparseVector ¶

func GenerateSparseVector(ctx context.Context, text string) (*schema.SparseVector, error)

GenerateSparseVector builds a normalized BOW sparse vector from text. Special tokens (PAD, CLS, SEP) are filtered to reduce noise.

Returns error if:

• Text cannot be tokenized
• No valid tokens remain after filtering
• Normalization fails (due to zero norm)

func GetTokenizer ¶ added in v0.21.0

func GetTokenizer(ctx context.Context) (*tokenizer.Tokenizer, error)

GetTokenizer returns the shared tokenizer instance, initializing it if needed.