rag

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Chunk

type Chunk struct {
	Key      string
	Text     string
	Position int
}

type Chunker

type Chunker interface {
	Split(ctx context.Context, text string) ([]Chunk, error)
}

Chunker splits a text into indexable chunks. The context allows implementations that call remote services (e.g. SemanticChunker).

func DefaultChunker

func DefaultChunker() Chunker

DefaultChunker returns a ParagraphChunker with sensible defaults.

type Embedder

type Embedder interface {
	EmbedTexts(ctx context.Context, texts []string) ([][]float32, error)
}

type IngestRequest

type IngestRequest struct {
	CorpusID string
	// FileID, when non-empty, produces a deterministic artifact key ("rag/{CorpusID}/{FileID}").
	// This makes Upsert idempotent: re-ingesting the same file replaces its vectors in-place
	// rather than creating orphaned duplicates alongside the old ones.
	FileID   string
	Filename string
	Text     string
}

type IngestResult

type IngestResult struct {
	Artifact storage.ArtifactRef
	Chunks   int
}

type ParagraphChunker

type ParagraphChunker struct {
	MaxChunkChars int
}

ParagraphChunker splits on blank lines and hard-caps each paragraph at MaxChunkChars characters. It does not call any remote service.

func (ParagraphChunker) Split

func (c ParagraphChunker) Split(_ context.Context, text string) ([]Chunk, error)

type Reranker

type Reranker interface {
	Rerank(ctx context.Context, query string, docs []string, topN int) (indices []int, scores []float32, err error)
	IsConfigured() bool
}

Reranker reorders a candidate set of documents by semantic relevance. Implementations are optional — when nil the RAG service returns vector results in score order without a second-pass rerank.

Rerank returns (indices, scores) where indices[i] is the position of the i-th most relevant document in the original docs slice. Pass topN=0 to return all results.

type SearchRequest

type SearchRequest struct {
	CorpusID string
	Query    string
	TopK     int
	// HybridWeight blends vector similarity with BM25 keyword search.
	// 0 (default) = pure vector; 1 = pure BM25; intermediate = linear blend.
	HybridWeight float32
	// Filter restricts results by metadata key-value predicates.
	// Passed through to vector.Query.Filter unchanged.
	Filter map[string]any
}

type SearchResponse

type SearchResponse struct {
	CorpusID string         `json:"corpus_id"`
	Results  []SearchResult `json:"results"`
}

type SearchResult

type SearchResult struct {
	Key      string            `json:"key"`
	Text     string            `json:"text"`
	Score    float32           `json:"score"`
	Metadata map[string]string `json:"metadata,omitempty"`
}

type SemanticChunker

type SemanticChunker struct {

	// Threshold below which two consecutive sentences are considered
	// semantically distinct and a new chunk is started. Default: 0.3.
	Threshold float32
	// MaxChunkChars is the hard character cap per chunk. Default: 2000.
	MaxChunkChars int
	// MinChunkChars is the minimum size a chunk must reach before it can be
	// split off. Chunks below this are merged with the next. Default: 100.
	MinChunkChars int
	// contains filtered or unexported fields
}

SemanticChunker groups sentences into chunks based on cosine similarity between consecutive sentence embeddings. A new chunk is started whenever similarity falls below Threshold. Inspired by the Max-Min algorithm (Springer 2025) as implemented in the mcp-local-rag reference.

Falls back to ParagraphChunker behaviour when the embedder returns an error, so ingest never fails due to a temporary embedding outage.

func NewSemanticChunker

func NewSemanticChunker(embedder Embedder, threshold float32) *SemanticChunker

NewSemanticChunker creates a SemanticChunker with sensible defaults. threshold ≤ 0 → use default 0.3.

func (*SemanticChunker) Split

func (c *SemanticChunker) Split(ctx context.Context, text string) ([]Chunk, error)

Split embeds each sentence and groups them into chunks by similarity. On embedder error it falls back to ParagraphChunker.

type Service

type Service struct {
	// contains filtered or unexported fields
}

func NewService

func NewService(artifacts storage.ArtifactStore, vectors vector.Store, embedder Embedder, chunker Chunker) *Service

func (*Service) DeleteFileChunks

func (s *Service) DeleteFileChunks(ctx context.Context, namespace, artifactKey string, fromChunk, toChunk int) error

DeleteFileChunks removes stale chunk records that exceed the new chunk count. Called after re-ingesting a file that now has fewer chunks than its previous run, to avoid leaving orphaned vectors from the old (longer) version.

func (*Service) DeleteNamespace

func (s *Service) DeleteNamespace(ctx context.Context, namespace string) error

DeleteNamespace removes all vector records for the given corpus namespace.

func (*Service) Ingest

func (s *Service) Ingest(ctx context.Context, request IngestRequest) (IngestResult, error)

func (*Service) Search

func (s *Service) Search(ctx context.Context, request SearchRequest) (SearchResponse, error)

func (*Service) SetReranker

func (s *Service) SetReranker(r Reranker)

SetReranker installs a second-pass reranker applied after vector retrieval. Pass nil to disable reranking.

type VectorStore

type VectorStore = vector.Store

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL