Documentation
¶
Overview ¶
Package embedding provides interfaces and implementations for text embedding generation. It supports both local and remote embedding models with batch processing and caching capabilities.
Package embedding provides interfaces and implementations for text embedding generation. It supports both local and remote embedding models with batch processing and caching capabilities.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BatchOptions ¶
type BatchOptions struct {
MaxBatchSize int // Maximum batch size for embedding generation
MaxConcurrent int // Maximum number of concurrent batches
}
BatchOptions contains configuration options for batch processing.
type BatchProcessor ¶
type BatchProcessor struct {
// contains filtered or unexported fields
}
BatchProcessor handles batch processing of embeddings with optimization features including caching, concurrent processing, and progress tracking.
func NewBatchProcessor ¶
func NewBatchProcessor(provider Provider, options BatchOptions) *BatchProcessor
NewBatchProcessor creates a new batch processor with the given provider and options.
Parameters: - provider: The embedding provider to use for generating embeddings - options: Configuration options for batch processing
Returns: - *BatchProcessor: A new batch processor instance
func (*BatchProcessor) ProcessWithProgress ¶
func (bp *BatchProcessor) ProcessWithProgress( ctx context.Context, texts []string, callback ProgressCallback, ) ([][]float32, error)
ProcessWithProgress processes embeddings with progress tracking
type ProgressCallback ¶
ProgressCallback is a callback function for tracking batch processing progress.
Parameters: - current: Number of texts processed so far - total: Total number of texts to process - error: Error if any occurred during processing
Returns: - bool: True to continue processing, false to cancel the operation
type Provider ¶
type Provider interface {
// Embed generates embeddings for the given texts.
//
// Parameters:
// - ctx: Context for cancellation and timeout
// - texts: Slice of texts to embed
//
// Returns:
// - [][]float32: Slice of embeddings, one for each text
// - error: Error if embedding generation fails
Embed(ctx context.Context, texts []string) ([][]float32, error)
// Dimension returns the dimension of the embeddings generated by this provider.
//
// Returns:
// - int: Embedding dimension
Dimension() int
}
Provider defines the interface for embedding providers.
This interface is implemented by all embedding model providers and allows the application to generate embeddings for text.
Example implementation:
type LocalProvider struct {
model *LocalModel
dimension int
}
func (p *LocalProvider) Embed(ctx context.Context, texts []string) ([][]float32, error) {
// Generate embeddings using local model
}
func (p *LocalProvider) Dimension() int {
return p.dimension
}
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package downloader provides functionality for downloading embedding models from remote sources.
|
Package downloader provides functionality for downloading embedding models from remote sources. |
|
Package local provides implementations for local embedding model providers.
|
Package local provides implementations for local embedding model providers. |
|
Package models provides pre-configured providers for popular embedding models.
|
Package models provides pre-configured providers for popular embedding models. |