Documentation
¶
Overview ¶
Package localllm provides CGO bindings to llama.cpp for local GGUF model inference.
This package enables NornicDB to run embedding models directly without external services like Ollama. It uses llama.cpp compiled as a static library with GPU acceleration (Metal on macOS, CUDA on Linux) and CPU fallback.
Metal Optimizations (Apple Silicon):
- Configurable flash attention
- Full model GPU offload by default
- Unified memory utilization
- SIMD-optimized CPU fallback
Features:
- GPU-first with automatic CPU fallback
- Memory-mapped model loading for low memory footprint
- Thread-safe embedding generation
- Batch embedding support
Example:
opts := localllm.DefaultOptions("/models/bge-m3.gguf")
model, err := localllm.LoadModel(opts)
if err != nil {
log.Fatal(err)
}
defer model.Close()
embedding, err := model.Embed(ctx, "hello world")
// embedding is a normalized []float32
Index ¶
- type ContextFeatures
- type GenerateParams
- type GenerationModel
- func (g *GenerationModel) Close() error
- func (g *GenerationModel) Generate(ctx context.Context, prompt string, params GenerateParams) (string, error)
- func (g *GenerationModel) GenerateStream(ctx context.Context, prompt string, params GenerateParams, ...) error
- func (g *GenerationModel) ModelPath() string
- type GenerationOptions
- type Model
- func (m *Model) ChunkText(text string, maxTokens, overlap int) ([]string, error)
- func (m *Model) Close() error
- func (m *Model) CountTokens(text string) (int, error)
- func (m *Model) Dimensions() int
- func (m *Model) Embed(ctx context.Context, text string) ([]float32, error)
- func (m *Model) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
- func (m *Model) EmbedRaw(ctx context.Context, text string) ([]float32, error)
- func (m *Model) MaxTokens() int
- func (m *Model) ModelDescription() string
- type Options
- type RerankerModel
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ContextFeatures ¶ added in v1.1.3
type ContextFeatures struct {
// CtxType selects the context type. 0=default, 1=MTP.
// Only set to 1 if the model has MTP layers.
CtxType int
// PoolingType controls embedding pooling. 1=mean (default for embeddings),
// 2=cls, 3=last, 4=rank. Use -1 to leave unspecified.
PoolingType int
// AttentionType controls attention masking. 0=causal (LLM default),
// 1=non-causal (BERT-style, default for embeddings).
AttentionType int
// FlashAttn controls flash attention. -1=auto, 0=disabled (default), 1=enabled.
FlashAttn int
}
ContextFeatures configures llama.cpp context parameters that vary by model.
These are passthrough settings from environment variables so that different models (embedding, MTP, generation) can declare the features they need.
Environment variables (per domain):
Embedding: NORNICDB_EMBEDDING_CTX_TYPE, _POOLING_TYPE, _ATTENTION_TYPE, _FLASH_ATTN Rerank: NORNICDB_RERANK_CTX_TYPE, _POOLING_TYPE, _ATTENTION_TYPE, _FLASH_ATTN Heimdall: NORNICDB_HEIMDALL_CTX_TYPE, _POOLING_TYPE, _ATTENTION_TYPE, _FLASH_ATTN
func DefaultContextFeatures ¶ added in v1.1.3
func DefaultContextFeatures() ContextFeatures
DefaultContextFeatures returns defaults optimized for embedding models.
type GenerateParams ¶
type GenerateParams struct {
MaxTokens int
Temperature float32
TopP float32
TopK int
StopTokens []string
}
GenerateParams configures text generation.
func DefaultGenerateParams ¶
func DefaultGenerateParams() GenerateParams
DefaultGenerateParams returns sensible defaults for structured output.
type GenerationModel ¶
type GenerationModel struct {
// contains filtered or unexported fields
}
GenerationModel wraps a GGUF model for text generation.
func LoadGenerationModel ¶
func LoadGenerationModel(opts GenerationOptions) (*GenerationModel, error)
LoadGenerationModel loads a GGUF model for text generation.
func (*GenerationModel) Close ¶
func (g *GenerationModel) Close() error
Close releases all resources.
func (*GenerationModel) Generate ¶
func (g *GenerationModel) Generate(ctx context.Context, prompt string, params GenerateParams) (string, error)
Generate produces a complete response for the prompt.
func (*GenerationModel) GenerateStream ¶
func (g *GenerationModel) GenerateStream(ctx context.Context, prompt string, params GenerateParams, callback func(token string) error) error
GenerateStream produces tokens via callback for streaming. Uses safe decode functions with signal handling and detailed error reporting.
func (*GenerationModel) ModelPath ¶
func (g *GenerationModel) ModelPath() string
ModelPath returns the loaded model path.
type GenerationOptions ¶
type GenerationOptions struct {
ModelPath string
ContextSize int // Max context window (default: 2048)
BatchSize int // Processing batch size (default: 512)
Threads int // CPU threads (default: NumCPU/2)
GPULayers int // GPU offload (-1=auto, 0=CPU)
Features ContextFeatures
}
GenerationOptions configures generation model loading.
func DefaultGenerationOptions ¶
func DefaultGenerationOptions(modelPath string) GenerationOptions
DefaultGenerationOptions returns sensible defaults for text generation.
type Model ¶
type Model struct {
// contains filtered or unexported fields
}
Model wraps a GGUF model for embedding generation.
Thread-safe: The Embed and EmbedBatch methods can be called concurrently, but operations are serialized internally via mutex to prevent race conditions with the underlying C context.
func LoadModel ¶
LoadModel loads a GGUF model for embedding generation.
The model is memory-mapped for low memory footprint. GPU layers are automatically offloaded based on Options.GPULayers:
- -1: Auto-detect GPU and offload all layers (recommended)
- 0: CPU only (no GPU offload)
- N: Offload N layers to GPU
Metal Optimization (Apple Silicon):
When running on Apple Silicon with Metal support compiled in:
- All model layers are offloaded to GPU by default
- Flash attention defaults to disabled for embedding stability
- Unified memory is utilized efficiently
- Typical speedup: 5-10x over CPU-only
Example:
opts := localllm.DefaultOptions("/models/bge-m3.gguf")
model, err := localllm.LoadModel(opts)
if err != nil {
log.Fatalf("Failed to load model: %v", err)
}
defer model.Close()
fmt.Printf("Model loaded: %d dimensions\n", model.Dimensions())
func (*Model) ChunkText ¶
ChunkText deterministically splits text using the model tokenizer so every returned chunk fits within the provided token cap.
func (*Model) Close ¶
Close releases all resources associated with the model.
After Close is called, the Model must not be used. This properly releases GPU memory on Metal/CUDA.
func (*Model) CountTokens ¶
CountTokens returns the exact tokenizer count for text using the model's vocab.
func (*Model) Dimensions ¶
Dimensions returns the embedding vector size.
This is determined by the model architecture:
- BGE-M3: 1024 dimensions
- E5-large: 1024 dimensions
- Jina-v2-base-code: 768 dimensions
func (*Model) Embed ¶
Embed generates a normalized embedding vector for the given text.
The returned vector is L2-normalized (unit length), suitable for cosine similarity calculations.
Concurrency:
Operations are serialized via mutex because llama.cpp contexts are NOT thread-safe. The C.embed call holds the lock for the duration of GPU/CPU inference (~5-50ms depending on text length and hardware).
For higher throughput under concurrent load, create multiple Model instances (each with its own GPU context). The GPU can process multiple contexts efficiently via kernel scheduling.
GPU Acceleration:
On Apple Silicon with Metal, the embedding is computed on the GPU:
- Tokenization (CPU)
- Model inference (GPU)
- Pooling (GPU)
- Normalization (CPU)
Example:
vec, err := model.Embed(ctx, "graph database")
if err != nil {
return err
}
fmt.Printf("Embedding: %d dimensions\n", len(vec))
func (*Model) EmbedBatch ¶
EmbedBatch generates normalized embeddings for multiple texts.
Each text is processed sequentially through the GPU. For maximum throughput with many texts, consider parallel processing with multiple Model instances.
Note: True batch processing (multiple texts in single GPU kernel) would require llama.cpp changes. Current implementation is efficient for moderate batch sizes due to GPU kernel reuse.
Example:
texts := []string{"hello", "world", "test"}
vecs, err := model.EmbedBatch(ctx, texts)
if err != nil {
return err
}
for i, vec := range vecs {
fmt.Printf("Text %d: %d dims\n", i, len(vec))
}
func (*Model) MaxTokens ¶
MaxTokens returns the effective tokenizer/input limit for this model context.
func (*Model) ModelDescription ¶
ModelDescription returns a human-readable description of the loaded model.
type Options ¶
type Options struct {
ModelPath string
ContextSize int
BatchSize int
Threads int
GPULayers int
Features ContextFeatures
}
Options configures model loading and inference.
Fields:
- ModelPath: Path to .gguf model file
- ContextSize: Max context size for tokenization (default: auto from model cap, up to 8192 for embedding)
- BatchSize: Batch size for processing (default: match effective context size)
- Threads: CPU threads for inference (default: NumCPU/2, min 4)
- GPULayers: GPU layer offload (-1=auto/all, 0=CPU only, N=N layers)
- Features: llama.cpp context features configurable per-model via env
func DefaultOptions ¶
DefaultOptions returns options optimized for embedding generation.
GPU is enabled by default (-1 = auto-detect and use all layers). Set GPULayers to 0 to force CPU-only mode.
For Apple Silicon, this enables full Metal GPU acceleration with:
- Full model offload
- Unified memory optimization
Example:
opts := localllm.DefaultOptions("/models/bge-m3.gguf")
opts.GPULayers = 0 // Force CPU mode
model, err := localllm.LoadModel(opts)
func DefaultRerankerOptions ¶ added in v1.2.3
DefaultRerankerOptions returns options for classifier-head GGUF rerankers. Rank pooling attaches the model's classification head and returns its logits.
type RerankerModel ¶
type RerankerModel struct {
// contains filtered or unexported fields
}
RerankerModel wraps a GGUF model for reranking (query, document) pairs. It encodes the query and document as a model-specific classification pair.
func LoadRerankerModel ¶
func LoadRerankerModel(opts Options) (*RerankerModel, error)
LoadRerankerModel loads a GGUF reranker model (e.g. BGE-Reranker-v2-m3). Uses the same loader as embedding models; the model file must be a reranker/classification GGUF that outputs a single score per (query, doc) pair.
func (*RerankerModel) Close ¶
func (r *RerankerModel) Close() error
Close releases the underlying model.
Source Files