Documentation
¶
Index ¶
- Constants
- func ArtifactKey(corpusID, fileID string) string
- func ChunkEnrichmentCacheKey(chunkText, enricherKey string) string
- func DocumentChunkCacheKey(doc Document, chunkerKey string) string
- func DocumentReaderChunkOptionsForProfile(profile ChunkProfile, opts publicreader.ChunkOptions) publicreader.ChunkOptions
- type ArtifactChunkCache
- type ArtifactEnrichmentCache
- type CachedDocumentChunker
- type CachedEnricher
- type Chunk
- type ChunkCache
- type ChunkCacheKeyProvider
- type ChunkProfile
- type ChunkProfileName
- type Chunker
- type Document
- type DocumentChunker
- type Embedder
- type Enricher
- type EnricherCacheKeyProvider
- type EnrichmentCache
- type HeadingChunker
- type HybridDocumentChunker
- type IngestRequest
- type IngestResult
- type MemoryChunkCache
- type MemoryEnrichmentCache
- type ParagraphChunker
- type QAChunker
- type Reranker
- type SearchRequest
- type SearchResponse
- type SearchResult
- type SemanticChunker
- type Service
- type TableChunker
- type VectorStore
Constants ¶
const ( ChunkProfileSmall = internalrag.ChunkProfileSmall ChunkProfileMedium = internalrag.ChunkProfileMedium ChunkProfileLarge = internalrag.ChunkProfileLarge ChunkProfileStructured = internalrag.ChunkProfileStructured ChunkProfileTable = internalrag.ChunkProfileTable ChunkProfileQA = internalrag.ChunkProfileQA ChunkProfileCustom = internalrag.ChunkProfileCustom )
Variables ¶
This section is empty.
Functions ¶
func ArtifactKey ¶
ArtifactKey builds the deterministic artifact key for a file within a corpus (format: "rag/{corpusID}/{fileID}"), matching what Ingest uses internally when IngestRequest.FileID is set. Useful for a caller building their own delete-by-file tooling on top of Service.DeleteFileChunks.
func ChunkEnrichmentCacheKey ¶
ChunkEnrichmentCacheKey builds the deterministic cache key for a chunk's enrichment result, matching what CachedEnricher uses internally - useful for callers that want to pre-warm or invalidate the cache directly.
func DocumentChunkCacheKey ¶
DocumentChunkCacheKey returns the cache key of a document for a given chunker: a SHA-256 of the file name, the chunker key and the bytes of the document (or its text when it has no bytes).
func DocumentReaderChunkOptionsForProfile ¶
func DocumentReaderChunkOptionsForProfile(profile ChunkProfile, opts publicreader.ChunkOptions) publicreader.ChunkOptions
DocumentReaderChunkOptionsForProfile maps a RAG chunk profile to hybrid document-reader chunk options while preserving caller-provided structural options.
Types ¶
type ArtifactChunkCache ¶
type ArtifactChunkCache = internalrag.ArtifactChunkCache
ArtifactChunkCache persists cached chunks in the configured artifact store.
func NewArtifactChunkCache ¶
func NewArtifactChunkCache(store publicstorage.ArtifactStore) *ArtifactChunkCache
NewArtifactChunkCache creates a chunk cache that keeps the chunks as artifacts in store.
type ArtifactEnrichmentCache ¶
type ArtifactEnrichmentCache = internalrag.ArtifactEnrichmentCache
ArtifactEnrichmentCache persists cached enrichment results in the configured artifact store.
func NewArtifactEnrichmentCache ¶
func NewArtifactEnrichmentCache(store publicstorage.ArtifactStore) *ArtifactEnrichmentCache
NewArtifactEnrichmentCache creates a cache that keeps the generated questions of each chunk as artifacts in store.
type CachedDocumentChunker ¶
type CachedDocumentChunker = internalrag.CachedDocumentChunker
CachedDocumentChunker wraps a DocumentChunker with a deterministic cache.
The cache key includes document bytes/text, filename, the cache schema version, and the wrapped chunker's option fingerprint when available.
func NewCachedDocumentChunker ¶
func NewCachedDocumentChunker(chunker DocumentChunker, cache ChunkCache) *CachedDocumentChunker
NewCachedDocumentChunker wraps chunker so that the chunks of a document that was already split are read from cache instead of being computed again.
type CachedEnricher ¶
type CachedEnricher = internalrag.CachedEnricher
CachedEnricher wraps an Enricher with a deterministic cache, keyed by chunk text plus the wrapped enricher's option fingerprint when available.
func NewCachedEnricher ¶
func NewCachedEnricher(enricher Enricher, cache EnrichmentCache) *CachedEnricher
NewCachedEnricher wraps enricher with a deterministic cache (see internal/rag.CachedEnricher's doc comment) so a re-ingest of unchanged chunks doesn't re-bill the LLM for enrichment.
type Chunk ¶
type Chunk = internalrag.Chunk
Chunk is a piece of a document as it was split for indexing: its key, its text, its position in the document and its metadata.
type ChunkCache ¶
type ChunkCache = internalrag.ChunkCache
ChunkCache stores document-aware chunking outputs under a deterministic key.
type ChunkCacheKeyProvider ¶
type ChunkCacheKeyProvider = internalrag.ChunkCacheKeyProvider
ChunkCacheKeyProvider lets chunkers include their own options in cache keys.
type ChunkProfile ¶
type ChunkProfile = internalrag.ChunkProfile
ChunkProfile describes the chunking policy to use for document ingestion. Token limits are intentionally conservative: large-context LLMs can read more, but retrieval quality usually degrades when indexed chunks become too broad or noisy.
func DefaultChunkProfile ¶
func DefaultChunkProfile() ChunkProfile
DefaultChunkProfile returns the default enterprise RAG profile.
func NewCustomChunkProfile ¶
func NewCustomChunkProfile(maxTokens, overlapTokens int) (ChunkProfile, error)
NewCustomChunkProfile validates and returns a custom chunk profile.
func RecommendedChunkProfile ¶
func RecommendedChunkProfile(name ChunkProfileName) (ChunkProfile, bool)
RecommendedChunkProfile returns one of Seshat's built-in chunk profiles.
type ChunkProfileName ¶
type ChunkProfileName = internalrag.ChunkProfileName
ChunkProfileName identifies a recommended chunking profile for RAG ingestion. The profile controls indexing granularity; answer-time context expansion should still be used when a large LLM needs neighboring chunks.
type Chunker ¶
type Chunker = internalrag.Chunker
Chunker splits a text into indexable chunks. The context allows implementations that call remote services (e.g. SemanticChunker).
func DefaultChunker ¶
func DefaultChunker() Chunker
DefaultChunker returns a ParagraphChunker with sensible defaults - what NewService uses internally when chunker is nil.
type Document ¶
type Document = internalrag.Document
Document is the original source material plus its text representation. Plain text chunkers can ignore Data. Document-aware chunkers can prefer Data to preserve layout, headings, page numbers, tables, and captions.
type DocumentChunker ¶
type DocumentChunker = internalrag.DocumentChunker
DocumentChunker is implemented by chunkers that can split the original document, not just extracted plain text.
type Embedder ¶
type Embedder = internalrag.Embedder
Embedder turns texts into embedding vectors, one vector per text and in the same order.
type Enricher ¶
type Enricher = internalrag.Enricher
Enricher generates synthetic questions per chunk - questions that chunk's content would answer - so they can be indexed alongside it. This is a known retrieval-recall technique: chunk text is usually declarative, but users phrase queries as questions, so embedding the chunk together with questions it answers closes that vocabulary gap. Optional: nil (the default) disables enrichment entirely - see Service.SetEnricher. Costs one LLM call per chunk, so it is opt-in, never a forced default.
type EnricherCacheKeyProvider ¶
type EnricherCacheKeyProvider = internalrag.EnricherCacheKeyProvider
EnricherCacheKeyProvider lets an Enricher include its own configuration (model, questions-per-chunk, prompt version) in cache keys, so changing that configuration doesn't silently reuse stale enrichment results - the same purpose ChunkCacheKeyProvider serves for chunkers.
type EnrichmentCache ¶
type EnrichmentCache = internalrag.EnrichmentCache
EnrichmentCache stores per-chunk enrichment results (synthetic questions) under a deterministic key - the same content-hash-keying idea as ChunkCache, so a re-ingest of unchanged chunks doesn't re-bill the LLM.
type HeadingChunker ¶
type HeadingChunker = internalrag.HeadingChunker
HeadingChunker cuts a markdown document (what the document readers write) along its own structure: headings, paragraphs, lists, tables and fenced code. A block is never cut in the middle of what it is, small sections are joined to their neighbours, and each chunk carries the path of headings it sits under in its text and in Metadata["heading_path"] (and the pages it covers in Metadata["page_numbers"] when the text has page markers). It is the chunker the hosts use for documents they have read natively.
func NewHeadingChunker ¶
func NewHeadingChunker(profile ChunkProfile) *HeadingChunker
NewHeadingChunker creates a structure-aware chunker for markdown, cutting chunks of at most profile.MaxTokens tokens with profile.OverlapTokens of overlap where a paragraph has to be cut. Use RecommendedChunkProfile(ChunkProfileStructured) for the profile hosts use.
type HybridDocumentChunker ¶
type HybridDocumentChunker = internalrag.HybridDocumentChunker
HybridDocumentChunker chunks rich documents with a document-aware hybrid chunker. The backend can be local, docling-serve, seshat-intelligence, or any other document reader implementing documentreader.HybridChunker. It falls back to a plain text chunker unless FailOnError is set.
func NewHybridDocumentChunker ¶
func NewHybridDocumentChunker(client publicreader.HybridChunker, opts publicreader.ChunkOptions) *HybridDocumentChunker
NewHybridDocumentChunker creates a document-aware chunker backed by the given reader.
func NewHybridDocumentChunkerForProfile ¶
func NewHybridDocumentChunkerForProfile(client publicreader.HybridChunker, profile ChunkProfile, opts publicreader.ChunkOptions) *HybridDocumentChunker
NewHybridDocumentChunkerForProfile creates a document-aware chunker using one of Seshat's recommended chunking profiles. When the backend is unavailable or fails, the fallback depends on the profile: "structured" falls back to HeadingChunker (heading/numbering-hierarchy aware), "table" to TableChunker (keeps table rows intact), "qa" to QAChunker (one chunk per detected Q/A pair) - every other profile keeps the plain fallback unchanged.
type IngestRequest ¶
type IngestRequest = internalrag.IngestRequest
IngestRequest is a document to index in a corpus: its text and, when available, its original bytes (for chunkers that read the structure of the document), its name, and a file id that makes ingesting the same file again replace its previous vectors instead of duplicating them.
type IngestResult ¶
type IngestResult = internalrag.IngestResult
IngestResult reports an ingestion: the artifact that holds the document and the number of chunks that were indexed.
type MemoryChunkCache ¶
type MemoryChunkCache = internalrag.MemoryChunkCache
MemoryChunkCache is useful for tests and short-lived local runs.
func NewMemoryChunkCache ¶
func NewMemoryChunkCache() *MemoryChunkCache
NewMemoryChunkCache creates a chunk cache held in memory, safe for concurrent use.
type MemoryEnrichmentCache ¶
type MemoryEnrichmentCache = internalrag.MemoryEnrichmentCache
MemoryEnrichmentCache is useful for tests and short-lived local runs.
func NewMemoryEnrichmentCache ¶
func NewMemoryEnrichmentCache() *MemoryEnrichmentCache
NewMemoryEnrichmentCache creates a cache of the generated questions of each chunk, held in memory and safe for concurrent use.
type ParagraphChunker ¶
type ParagraphChunker = internalrag.ParagraphChunker
ParagraphChunker splits on blank lines with a hard character cap per chunk. The default Chunker (see DefaultChunker) when none is given.
type QAChunker ¶
type QAChunker = internalrag.QAChunker
QAChunker splits a question/answer-formatted document (FAQs, support docs) into one chunk per Q/A pair, detected either from explicit "Q:"/"A:" labels or from Markdown headings ending in "?". Falls back to HeadingChunker behavior for a document with no QA structure.
func NewQAChunker ¶
func NewQAChunker(profile ChunkProfile) *QAChunker
NewQAChunker creates a QA-pair-aware chunker using one of Seshat's recommended chunk profiles - see QAChunker's own doc comment.
type Reranker ¶
type Reranker = internalrag.Reranker
Reranker reorders a candidate set of documents by semantic relevance. Implementations are optional — when nil the RAG service returns vector results in score order without a second-pass rerank.
Rerank returns (indices, scores) where indices[i] is the position of the i-th most relevant document in the original docs slice. Pass topN=0 to return all results.
type SearchRequest ¶
type SearchRequest = internalrag.SearchRequest
SearchRequest is a query against a corpus: how many results to return (TopK), how keyword search is blended with vector similarity (HybridWeight), how the score of a reranker is blended with the retrieval score (RerankWeight), and an optional filter on the metadata.
type SearchResponse ¶
type SearchResponse = internalrag.SearchResponse
SearchResponse holds the results of a search, ordered by relevance, with the corpus they come from.
type SearchResult ¶
type SearchResult = internalrag.SearchResult
SearchResult is a chunk found by a search: its key, text, score and metadata.
type SemanticChunker ¶
type SemanticChunker = internalrag.SemanticChunker
SemanticChunker groups sentences into chunks by embedding-similarity instead of blind paragraph splitting - meaningfully better retrieval quality, at the cost of one extra embedding call per sentence during ingest. Falls back to ParagraphChunker behavior if the embedder errors.
func NewSemanticChunker ¶
func NewSemanticChunker(embedder Embedder, threshold float32) *SemanticChunker
NewSemanticChunker creates a SemanticChunker. threshold <= 0 uses the default (0.3) - see SemanticChunker's doc for the tradeoff it makes.
type Service ¶
type Service = internalrag.Service
Service ingests documents into a corpus and searches it: it splits the text into chunks, embeds them and keeps them in a vector store, and keeps the original document in an artifact store when one is given. A reranker can be added to reorder the results of a search.
func NewService ¶
func NewService(artifacts publicstorage.ArtifactStore, vectors publicvector.Store, embedder Embedder, chunker Chunker) *Service
NewService creates a Service. artifacts may be nil, in which case the original documents are not stored, and a nil chunker means the default chunker.
type TableChunker ¶
type TableChunker = internalrag.TableChunker
TableChunker keeps GFM/Markdown table rows intact as their own chunk(s), splitting an oversized table by data row while repeating the header row on every piece, instead of a generic splitter cutting through the middle of a table. Falls back to HeadingChunker behavior for a document with no table.
func NewTableChunker ¶
func NewTableChunker(profile ChunkProfile) *TableChunker
NewTableChunker creates a table-aware chunker using one of Seshat's recommended chunk profiles - see TableChunker's own doc comment.
type VectorStore ¶
type VectorStore = publicvector.Store
VectorStore abstracts the vector storage/search engine used by RAG. Concrete implementations target SQLite, pgvector, Qdrant, Chroma, or any future provider — the rag.Service and knowledge.Service call only this interface.
Source Files
Directories