README
¶
ragkit: A Go package for document indexing and retrieval in RAG systems
ragkit is a Go package designed to simplify the implementation of Retrieval-Augmented Generation (RAG) systems. It includes the definition and implementation of a vectorizer interface that performs document indexing and retrieval, providing tools for vectorization and semantic search capabilities.
Installation
go get github.com/suapapa/go_ragkit
Quick Start
import (
"context"
"fmt"
ragkit "github.com/suapapa/go_ragkit"
)
func main() {
// Create documents from text
docs := ragkit.MakeDocsFromTexts(
[]string{
"고길동의 집에는 둘리, 도우너, 또치, 희동이, 철수, 영희가 살고 있다.",
"희동이는 고길동의 조카이다.",
},
nil,
)
// Initialize vectorizer (example with Weaviate)
vectorizer, err := ragkit.NewWeaviateVectorizer(...)
if err != nil {
panic(err)
}
// Index documents
ctx := context.Background()
for _, doc := range docs {
_, err = vectorizer.Index(ctx, doc)
if err != nil {
panic(err)
}
}
// Perform semantic search
query := "희동이와 고길동의 관계?"
results, err := vectorizer.RetrieveText(ctx, query, 1)
if err != nil {
panic(err)
}
fmt.Println(results[0].Text) // 희동이는 고길동의 조카이다.
}
Reference
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Documentation
¶
Overview ¶
Package ragkit provides utility functions for RAG (Retrieval-Augmented Generation) applications.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GenerateID ¶
GenerateID creates a UUID-like string from the input text using SHA-256 hash. The generated ID follows the format: 8-4-4-4-12 hexadecimal characters.
Types ¶
type Document ¶
type Document struct {
ID string // Unique ID
Text string // Original text
Metadata map[string]any // Optional: Additional metadata
Vector []float32 // Optional: Embedding vector (generated by Embeder if not provided)
}
Document is a type that represents a document
type Embedder ¶
type Embedder interface {
// Embed: Convert texts to embedding vectors
Embed(ctx context.Context, texts ...string) ([][]float32, error)
}
Embedder is a type that can embed texts into vectors
type Indexer ¶
type Indexer interface {
// Index: Index multiple documents
// Returns: IDs of indexed documents
Index(ctx context.Context, docs ...Document) ([]string, error)
// Delete: Delete a document by ID
Delete(ctx context.Context, id string) error
// Exists: Check if a document with the given ID exists
Exists(ctx context.Context, id string) (bool, error)
}
Indexer is a type that can index documents into a vector database
type RetrievedDoc ¶
type RetrievedDoc struct {
// ID string // Document ID
// Score float32 // Similarity score
Vector []float32 // Retrieved vector
Text string // Retrieved text
Metadata map[string]any // Optional metadata
}
RetrievedDoc is a type that represents a retrieved document from vector database
type Retriever ¶
type Retriever interface {
// Retrieve: Return top-K documents based on query vector
Retrieve(ctx context.Context, query []float32, topK int) ([]RetrievedDoc, error)
// RetrieveText: Return top-K documents based on text query
RetrieveText(ctx context.Context, text string, topK int) ([]RetrievedDoc, error)
}
Retriever is a type that can retrieve documents from a vector database
type Vectorizer ¶
Vectorizer is a combination of Indexer and Retriever
Source Files
Directories