embedder

type Base64 ¶

type Base64 string

Base64 is base64 encoded embedding string.

func (s Base64) Decode() (*Embedding, error)

type Chunk ¶

type Chunk struct {
	// Text contains the actual content of the chunk
	Text string
	// TokenSize represents the number of tokens in this chunk
	TokenSize int
	// StartSentence is the index of the first sentence in this chunk
	StartSentence int
	// EndSentence is the index of the last sentence in this chunk (exclusive)
	EndSentence int
}

Chunk represents a piece of text with associated metadata for tracking its position and size within the original document.

type Chunker ¶

type Chunker interface {
	// Chunk splits the input text into a slice of Chunks according to the
	// implementation's strategy.
	Chunk(text string) []Chunk
}

Chunker defines the interface for text chunking implementations. Different implementations can provide various strategies for splitting text while maintaining context and semantic meaning.

type DefaultTokenCounter ¶

type DefaultTokenCounter struct{}

DefaultTokenCounter provides a simple word-based token counting implementation. It splits text on whitespace to approximate token counts. This is suitable for basic use cases but may not accurately reflect subword tokenization used by language models.

func (dtc *DefaultTokenCounter) Count(text string) int

type EmbeddedChunk ¶

type EmbeddedChunk struct {
	// Embedding of their vector representations
	// Multiple embeddings can exist for different models or purposes
	Embedding
	// Chunk is the original chunk content that was embedded
	Chunk *Chunk `json:"text"`
}

EmbeddedChunk represents a chunk of text along with its vector embeddings and associated metadata. This is the core data structure for storing and retrieving embedded content.

func EmbedChunks(ctx context.Context, embedder Embedder, chunks []Chunk, usage *components.LLMUsage) ([]EmbeddedChunk, error)

type Embedder ¶

type Embedder interface {
	Provider() Provider
	Model() string
	Embed(context.Context, string, *Embedding, *components.LLMUsage) error
	BatchEmbed(ctx context.Context, parts []string, usage *components.LLMUsage) ([]Embedding, error)
	DotProduct(context.Context, *Embedding, *Embedding) (float64, error)
}

type Embedding ¶

type Embedding struct {
	Object    string            `json:"object"`
	Embedding []float64         `json:"embedding"`
	Index     int               `json:"index"`
	Meta      map[string]string `json:"meta,omitempty"`
}

Embedding is a special format of data representation that can be easily utilized by machine learning models and algorithms. The embedding is an information dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating point numbers, such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. For example, if two texts are similar, then their vector representations should also be similar.

func (e *Embedding) DotProduct(other *Embedding) (float64, error)

func (e Embedding) UUID() string

type Option ¶

type Option func(*Options)

Option is a function type for configuring the EmbedderConfig. It follows the functional options pattern for clean and flexible configuration.

func WithModel(model string) Option

func WithProvider(provider Provider) Option

type Options ¶

type Options struct {
	// contains filtered or unexported fields
}

Options holds the configuration for creating an Embedder instance. It supports multiple embedding providers and their specific options.

func (i Options) Model() string

func (i Options) Provider() Provider

type Provider ¶

type Provider = string

const (
	ProviderOpenAI      Provider = "OpenAI"
	ProviderVoyageAI    Provider = "VoyageAI"
	ProviderCohere      Provider = "Cohere"
	ProviderGemini      Provider = "Gemini"
	ProviderHuggingFace Provider = "HuggingFace"
)

type TextChunker ¶

type TextChunker struct {
	// ChunkSize is the target size of each chunk in tokens
	ChunkSize int
	// ChunkOverlap is the number of tokens that should overlap between adjacent chunks
	ChunkOverlap int
	// TokenCounter is used to count tokens in text segments
	TokenCounter TokenCounter
	// SentenceSplitter is a function that splits text into sentences
	SentenceSplitter func(string) []string
}

TextChunker provides an advanced implementation of the Chunker interface with support for overlapping chunks and custom tokenization.

func NewTextChunker(options ...TextChunkerOption) (*TextChunker, error)

func (tc *TextChunker) Chunk(text string) []Chunk

type TextChunkerOption ¶

type TextChunkerOption func(*TextChunker)

TextChunkerOption is a function type for configuring TextChunker instances. This follows the functional options pattern for clean and flexible configuration.

type TikTokenCounter ¶

type TikTokenCounter struct {
	// contains filtered or unexported fields
}

TikTokenCounter provides accurate token counting using the tiktoken library, which implements the tokenization schemes used by OpenAI models.

func NewTikTokenCounter(encoding string) (*TikTokenCounter, error)

func (ttc *TikTokenCounter) Count(text string) int

type TokenCounter ¶

type TokenCounter interface {
	// Count returns the number of tokens in the given text according to the
	// implementation's tokenization strategy.
	Count(text string) int
}

TokenCounter defines the interface for counting tokens in a string. This abstraction allows for different tokenization strategies (e.g., words, subwords).