rag

package
v0.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateCollectionSnapshot 

func CreateCollectionSnapshot(ctx context.Context, httpAddr, collection string) (string, error)

CreateCollectionSnapshot triggers a full snapshot of the given collection via the Qdrant HTTP API and returns the snapshot name. The addr should be "host:httpPort" (typically port 6333).

func QdrantServerVersion 

func QdrantServerVersion(ctx context.Context, httpAddr string) (string, error)

QdrantServerVersion queries the Qdrant HTTP root endpoint and returns the reported server version string (e.g. "1.17.1"). The addr should be "host:httpPort" (typically port 6333).

func WordCount 

func WordCount(s string) int

WordCount returns the approximate token count for a string.

Types

type Chunk 

type Chunk struct {
	Index    int
	Text     string
	DocID    string
	Metadata map[string]any
}

Chunk represents a segment of a document.

func ChunkText 

func ChunkText(docID, text string, cfg ChunkConfig, metadata map[string]any) []Chunk

ChunkText splits a document into overlapping chunks. Uses word-count as a token approximation (1 token ≈ 0.75 words, so we use words * 0.75).

type ChunkConfig 

type ChunkConfig struct {
	ChunkSize    int // approximate max tokens per chunk (default 512)
	ChunkOverlap int // tokens of overlap between consecutive chunks (default 64)
}

ChunkConfig controls how documents are split into chunks.

func DefaultChunkConfig 

func DefaultChunkConfig() ChunkConfig

DefaultChunkConfig returns sensible defaults.

type CollectionStats 

type CollectionStats struct {
	PointCount    uint64
	DocumentCount int
}

CollectionStats returns the number of points and unique documents in the collection, along with the on-disk storage size in bytes.

type Document 

type Document struct {
	ID       string
	Content  string
	Metadata map[string]any
}

Document is a single document to ingest.

type EmbeddingClient 

type EmbeddingClient struct {
	// contains filtered or unexported fields
}

EmbeddingClient calls an OpenAI-compatible /v1/embeddings endpoint.

func NewEmbeddingClient 

func NewEmbeddingClient(endpoint, apiKeyFile, model, docPrompt, queryPrompt string) *EmbeddingClient

NewEmbeddingClient creates an embedding client that reads the API key from a file. docPrompt is prepended to document texts; queryPrompt is prepended to query texts. Leave either empty to send raw text (suitable for OpenAI-hosted models).

func NewEmbeddingClientWithKey 

func NewEmbeddingClientWithKey(endpoint, apiKey, model, docPrompt, queryPrompt string) *EmbeddingClient

NewEmbeddingClientWithKey creates an embedding client with an explicit API key. docPrompt is prepended to document texts; queryPrompt is prepended to query texts. Leave either empty to send raw text (suitable for OpenAI-hosted models).

func (*EmbeddingClient) Embed 

func (e *EmbeddingClient) Embed(ctx context.Context, texts []string, forQuery bool) ([][]float32, error)

Embed converts texts into embedding vectors. Handles batching automatically. When forQuery is true, queryPrompt is prepended to each text (used for _rag_search); when false, docPrompt is prepended (used for _rag_ingest). When the relevant prompt is empty, texts are sent as-is — this preserves backward compatibility for models that do not need a task prefix (e.g. OpenAI's text-embedding-3-*).

Texts are grouped into batches bounded by maxBatchCharacterLimit and maxBatchChunks, and batches are dispatched concurrently (up to the configured concurrency) so CPU-bound local backends stay saturated. The returned vectors always align 1:1 with the input texts, in input order.

func (*EmbeddingClient) ProbeDimension 

func (e *EmbeddingClient) ProbeDimension(ctx context.Context) (int, error)

ProbeDimension embeds a single placeholder string and returns the output vector length. Call once on first use to discover the model's actual embedding dimension rather than requiring users to specify it manually. Uses docPrompt for consistency; the dimension is prompt-independent.

type IngestResult 

type IngestResult struct {
	DocumentCount int
	ChunkCount    int
}

IngestResult contains counts from an ingestion operation.

func IngestDocuments 

func IngestDocuments(
	ctx context.Context,
	docs []Document,
	qdrantAddr string,
	collection string,
	dimensions uint64,
	embedder *EmbeddingClient,
	chunkCfg ChunkConfig,
) (*IngestResult, error)

IngestDocuments chunks, embeds, and upserts documents into Qdrant. This is the shared pipeline used by both the API handler and the controller. Documents are processed one-by-one to avoid timeouts with local embedding models.

type Point 

type Point struct {
	ID      string
	Vector  []float32
	Payload map[string]any
}

Point represents a vector with its metadata for upsert operations.

type QdrantClient 

type QdrantClient struct {
	// contains filtered or unexported fields
}

QdrantClient wraps the official Qdrant Go gRPC client for vector operations.

func NewQdrantClient 

func NewQdrantClient(addr string) (*QdrantClient, error)

NewQdrantClient connects to a Qdrant instance at the given address. The address should be in "host:port" format (gRPC port, typically 6334).

func (*QdrantClient) Close 

func (q *QdrantClient) Close() error

Close releases the underlying gRPC connection.

func (*QdrantClient) CollectionStats 

func (q *QdrantClient) CollectionStats(ctx context.Context, collection string) (*CollectionStats, error)

CollectionStats returns point count, unique document count, and storage size.

func (*QdrantClient) EnsureCollection 

func (q *QdrantClient) EnsureCollection(ctx context.Context, name string, dimensions uint64) error

EnsureCollection creates a collection if it doesn't already exist.

func (*QdrantClient) Search 

func (q *QdrantClient) Search(ctx context.Context, collection string, vector []float32, topK int) ([]SearchResult, error)

Search performs a nearest neighbor search and returns the top-K results.

func (*QdrantClient) Upsert 

func (q *QdrantClient) Upsert(ctx context.Context, collection string, points []Point) error

Upsert inserts or updates points in the given collection.

type SearchResult 

type SearchResult struct {
	Score   float32
	Payload map[string]any
}

SearchResult represents a single search hit with its similarity score.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.