Documentation
¶
Index ¶
- func CosineSimilarity(a, b []float32) float64
- func DefaultDimForModel(modelDir string) int
- func DeserializeFloat32(buf []byte) []float32
- func GetEmbeddingDim() int
- func SerializeFloat32(v []float32) []byte
- func SetEmbeddingDim(dim int)
- type BPETokenizer
- func (t *BPETokenizer) DecodeTokenIDs(ids []int) string
- func (t *BPETokenizer) EOSID() int
- func (t *BPETokenizer) Encode(text string) []int
- func (t *BPETokenizer) EncodeWithSpecial(text string) (inputIDs, attentionMask []int64)
- func (t *BPETokenizer) PadBatch(inputIDs, attentionMasks [][]int64) (paddedIDs, paddedMasks, positionIDs [][]int64)
- func (t *BPETokenizer) SortedVocab(limit int) []string
- func (t *BPETokenizer) TokenizeToStrings(text string) []string
- func (t *BPETokenizer) VocabSize() int
- type Embedder
- type HashEmbedder
- type LlamaCppEmbedder
- type ONNXEmbedderStub
- type OllamaEmbedder
- type OpenAIEmbedder
- type TFIDFEmbedder
- type WordPieceTokenizer
- func (t *WordPieceTokenizer) CLSID() int
- func (t *WordPieceTokenizer) Encode(text string) []int
- func (t *WordPieceTokenizer) EncodeWithSpecial(text string) (inputIDs, attentionMask []int64)
- func (t *WordPieceTokenizer) PadID() int
- func (t *WordPieceTokenizer) SEPID() int
- func (t *WordPieceTokenizer) VocabSize() int
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CosineSimilarity ¶
CosineSimilarity computes cosine similarity between two vectors
func DefaultDimForModel ¶
DefaultDimForModel reads the model's config.json and returns the appropriate default embedding dimension. Returns 768 for NomicBERT (CodeRankEmbed), 256 for Qwen2 (Jina), and 384 (TF-IDF fallback) if detection fails.
func DeserializeFloat32 ¶
DeserializeFloat32 converts a little-endian byte slice back to float32 slice
func GetEmbeddingDim ¶
func GetEmbeddingDim() int
GetEmbeddingDim returns the current embedding dimension.
func SerializeFloat32 ¶
SerializeFloat32 converts a float32 slice to a little-endian byte slice Compatible with sqlite-vec's expected BLOB format
func SetEmbeddingDim ¶
func SetEmbeddingDim(dim int)
SetEmbeddingDim sets the embedding dimension (call before concurrent access).
Types ¶
type BPETokenizer ¶
type BPETokenizer struct {
// contains filtered or unexported fields
}
BPETokenizer implements byte-level BPE tokenization compatible with HuggingFace tokenizer.json (Qwen2/GPT-style). Pure Go, no CGO deps.
func NewBPETokenizer ¶
func NewBPETokenizer(modelDir string) (*BPETokenizer, error)
NewBPETokenizer loads a tokenizer from a HuggingFace model directory. Expects tokenizer.json to be present in the directory.
func (*BPETokenizer) DecodeTokenIDs ¶
func (t *BPETokenizer) DecodeTokenIDs(ids []int) string
DecodeTokenIDs converts token IDs back to a string (best-effort). M49: The reverse vocab map is built lazily on first call and cached.
func (*BPETokenizer) EOSID ¶
func (t *BPETokenizer) EOSID() int
EOSID returns the end-of-sequence token ID
func (*BPETokenizer) Encode ¶
func (t *BPETokenizer) Encode(text string) []int
Encode tokenizes text and returns token IDs. Does NOT add special tokens (caller should add EOS if needed).
func (*BPETokenizer) EncodeWithSpecial ¶
func (t *BPETokenizer) EncodeWithSpecial(text string) (inputIDs, attentionMask []int64)
EncodeWithSpecial tokenizes text and wraps with model-appropriate special tokens. For this causal embedding model, no BOS is used. EOS is always appended because the Qwen2 model produces embeddings at the EOS position during last-token pooling.
func (*BPETokenizer) PadBatch ¶
func (t *BPETokenizer) PadBatch(inputIDs, attentionMasks [][]int64) ( paddedIDs, paddedMasks, positionIDs [][]int64, )
PadBatch pads a batch of token sequences to the same length. Returns padded input_ids, attention_mask, and position_ids (all int64).
func (*BPETokenizer) SortedVocab ¶
func (t *BPETokenizer) SortedVocab(limit int) []string
SortedVocab returns vocab entries sorted by ID (for debugging).
func (*BPETokenizer) TokenizeToStrings ¶
func (t *BPETokenizer) TokenizeToStrings(text string) []string
TokenizeToStrings is like Encode but returns the token strings instead of IDs. Useful for debugging and testing.
func (*BPETokenizer) VocabSize ¶
func (t *BPETokenizer) VocabSize() int
VocabSize returns the tokenizer vocabulary size
type Embedder ¶
type Embedder interface {
Embed(text string) ([]float32, error)
EmbedBatch(texts []string) ([][]float32, error)
Dim() int
Close() error
}
Embedder is the interface for generating vector embeddings
func NewEmbedder ¶
func NewEmbedder() Embedder
NewEmbedder returns the best available embedder implementation. Currently returns a TFIDFEmbedder which provides real semantic locality (similar code identifiers produce similar vectors) using TF-IDF weighted word and character n-gram features projected to 384 dimensions. Falls back to HashEmbedder only if explicitly requested via NewHashEmbedder.
type HashEmbedder ¶
type HashEmbedder struct{}
HashEmbedder generates deterministic pseudo-embeddings using SHA-256 hashing. This is a last-resort fallback that maintains the correct vector dimensionality but provides NO real semantic similarity. Use TFIDFEmbedder instead.
func NewHashEmbedder ¶
func NewHashEmbedder() *HashEmbedder
NewHashEmbedder creates a new hash-based embedder
func (*HashEmbedder) Close ¶
func (e *HashEmbedder) Close() error
Close is a no-op for the hash embedder
func (*HashEmbedder) Embed ¶
func (e *HashEmbedder) Embed(text string) ([]float32, error)
Embed generates a vector from text using deterministic hashing
func (*HashEmbedder) EmbedBatch ¶
func (e *HashEmbedder) EmbedBatch(texts []string) ([][]float32, error)
EmbedBatch generates embeddings for multiple texts. Note: This iterates sequentially rather than using true batch inference. HashEmbedder is a pure-Go CPU implementation (SHA-256 hashing) with no external runtime that could benefit from batched execution. Each Embed() call is independent with negligible overhead, so sequential iteration is appropriate.
type LlamaCppEmbedder ¶
type LlamaCppEmbedder struct {
// contains filtered or unexported fields
}
LlamaCppEmbedder generates embeddings via the llama.cpp server HTTP API. Implements the Embedder interface using the /embedding endpoint.
func NewLlamaCppEmbedder ¶
func NewLlamaCppEmbedder(endpoint string, dim int) (*LlamaCppEmbedder, error)
NewLlamaCppEmbedder creates an embedder that calls the llama.cpp server API. It verifies connectivity by pinging the /health endpoint.
func (*LlamaCppEmbedder) Close ¶
func (e *LlamaCppEmbedder) Close() error
Close is a no-op for the llama.cpp HTTP embedder.
func (*LlamaCppEmbedder) Dim ¶
func (e *LlamaCppEmbedder) Dim() int
Dim returns the embedding dimension.
func (*LlamaCppEmbedder) Embed ¶
func (e *LlamaCppEmbedder) Embed(text string) (_ []float32, err error)
Embed generates an embedding vector for a single text.
func (*LlamaCppEmbedder) EmbedBatch ¶
func (e *LlamaCppEmbedder) EmbedBatch(texts []string) (_ [][]float32, err error)
EmbedBatch generates embeddings for multiple texts. Attempts batch via /embedding first; falls back to sequential Embed() calls if the server doesn't support batch format or returns an unexpected response.
type ONNXEmbedderStub ¶
type ONNXEmbedderStub struct{}
ONNXEmbedderStub is a placeholder type for non-ONNX builds.
func NewONNXEmbedder ¶
func NewONNXEmbedder(modelDir string, dim int, libPath string) (*ONNXEmbedderStub, error)
NewONNXEmbedder is a stub for builds without the onnx tag. Build with -tags "onnx" to enable ONNX model support.
func (*ONNXEmbedderStub) Close ¶
func (e *ONNXEmbedderStub) Close() error
func (*ONNXEmbedderStub) Dim ¶
func (e *ONNXEmbedderStub) Dim() int
func (*ONNXEmbedderStub) EmbedBatch ¶
func (e *ONNXEmbedderStub) EmbedBatch(texts []string) ([][]float32, error)
type OllamaEmbedder ¶
type OllamaEmbedder struct {
// contains filtered or unexported fields
}
OllamaEmbedder generates embeddings via the Ollama HTTP API. Implements the Embedder interface using the /api/embed endpoint.
func NewOllamaEmbedder ¶
func NewOllamaEmbedder(endpoint, model string, dim int) (*OllamaEmbedder, error)
NewOllamaEmbedder creates an embedder that calls the Ollama API. It verifies connectivity by pinging the /api/tags endpoint.
func (*OllamaEmbedder) Close ¶
func (e *OllamaEmbedder) Close() error
Close is a no-op for the Ollama embedder.
func (*OllamaEmbedder) Dim ¶
func (e *OllamaEmbedder) Dim() int
Dim returns the embedding dimension.
func (*OllamaEmbedder) Embed ¶
func (e *OllamaEmbedder) Embed(text string) (_ []float32, err error)
Embed generates an embedding vector for a single text.
func (*OllamaEmbedder) EmbedBatch ¶
func (e *OllamaEmbedder) EmbedBatch(texts []string) ([][]float32, error)
EmbedBatch generates embeddings for multiple texts sequentially. Ollama's /api/embed endpoint does not support native batching.
type OpenAIEmbedder ¶
type OpenAIEmbedder struct {
// contains filtered or unexported fields
}
OpenAIEmbedder generates embeddings via any OpenAI-compatible /v1/embeddings API (LM Studio, vLLM, text-embeddings-inference, etc.).
func NewOpenAIEmbedder ¶
func NewOpenAIEmbedder(endpoint, model string, dim int) (*OpenAIEmbedder, error)
NewOpenAIEmbedder creates an embedder that calls an OpenAI-compatible API. It verifies connectivity by hitting GET /v1/models.
func (*OpenAIEmbedder) Close ¶
func (e *OpenAIEmbedder) Close() error
func (*OpenAIEmbedder) Dim ¶
func (e *OpenAIEmbedder) Dim() int
func (*OpenAIEmbedder) EmbedBatch ¶
func (e *OpenAIEmbedder) EmbedBatch(texts []string) (_ [][]float32, err error)
EmbedBatch generates embeddings for multiple texts using native batch support. The OpenAI /v1/embeddings API accepts an array of strings as input.
type TFIDFEmbedder ¶
type TFIDFEmbedder struct {
// contains filtered or unexported fields
}
TFIDFEmbedder generates semantically meaningful embeddings by:
- Tokenizing input into words and subword pieces (camelCase split, underscore split)
- Generating character trigrams for subword coverage
- Using TF-IDF-like weighting: rare/long tokens get more weight
- Projecting token hashes into a fixed dim-dimensional space using multiple hash functions
- L2 normalizing the result
This gives real semantic locality: "ReadFile" and "ReadFileContents" will produce similar vectors because they share tokens and trigrams.
func NewTFIDFEmbedder ¶
func NewTFIDFEmbedder(dim int) (*TFIDFEmbedder, error)
NewTFIDFEmbedder creates a new TF-IDF based embedder with the given dimension. Returns an error if dim is not positive, since zero or negative dimensions would cause a division-by-zero panic in projectToken.
func (*TFIDFEmbedder) Close ¶
func (e *TFIDFEmbedder) Close() error
Close is a no-op for the TF-IDF embedder
func (*TFIDFEmbedder) Embed ¶
func (e *TFIDFEmbedder) Embed(text string) ([]float32, error)
Embed generates an embedding vector from text using TF-IDF n-gram features. The dimension is determined by the embedder's own dim field, not the global embeddingDim.
func (*TFIDFEmbedder) EmbedBatch ¶
func (e *TFIDFEmbedder) EmbedBatch(texts []string) ([][]float32, error)
EmbedBatch generates embeddings for multiple texts. Note: This iterates sequentially rather than using true batch inference. TFIDFEmbedder is a pure-Go CPU implementation with no external accelerator or runtime that could benefit from batched execution. Each Embed() call is independent and CPU-bound, so there is no amortizable per-call overhead (no session setup, no GPU kernel launch, etc.). Parallelizing with goroutines was considered but adds complexity without meaningful speedup for the typical batch sizes used in this codebase (chunked documents, ≤100 items).
type WordPieceTokenizer ¶
type WordPieceTokenizer struct {
// contains filtered or unexported fields
}
WordPieceTokenizer implements WordPiece tokenization for BERT-style models. Loads from HuggingFace tokenizer.json format.
func NewWordPieceTokenizer ¶
func NewWordPieceTokenizer(modelDir string) (*WordPieceTokenizer, error)
NewWordPieceTokenizer loads a WordPiece tokenizer from a HuggingFace model directory.
func (*WordPieceTokenizer) CLSID ¶
func (t *WordPieceTokenizer) CLSID() int
CLSID returns the [CLS] token ID.
func (*WordPieceTokenizer) Encode ¶
func (t *WordPieceTokenizer) Encode(text string) []int
Encode tokenizes text into token IDs without special tokens.
func (*WordPieceTokenizer) EncodeWithSpecial ¶
func (t *WordPieceTokenizer) EncodeWithSpecial(text string) (inputIDs, attentionMask []int64)
EncodeWithSpecial tokenizes text and wraps with [CLS] ... [SEP].
func (*WordPieceTokenizer) PadID ¶
func (t *WordPieceTokenizer) PadID() int
PadID returns the [PAD] token ID.
func (*WordPieceTokenizer) SEPID ¶
func (t *WordPieceTokenizer) SEPID() int
SEPID returns the [SEP] token ID.
func (*WordPieceTokenizer) VocabSize ¶
func (t *WordPieceTokenizer) VocabSize() int
VocabSize returns the vocabulary size.
Source Files
Directories