Documentation
¶
Overview ¶
Package pipeline provides a composable pipeline engine for orchestrating multi-step RAG operations (load, chunk, embed, store, retrieve, assemble).
Index ¶
- Constants
- type AssembleStep
- type ChunkStep
- type EmbedStep
- type FuncStep
- type LoadStep
- type Pipeline
- type RetrieveStep
- type Step
- type StepContext
- func (sc *StepContext) Get(key string) (any, bool)
- func (sc *StepContext) Keys() []string
- func (sc *StepContext) MustGet(key string) any
- func (sc *StepContext) Set(key string, value any)
- func (sc *StepContext) SetCacheHit(_ string, data any)
- func (sc *StepContext) SetStepName(name string)
- func (sc *StepContext) Snapshot() map[string]any
- func (sc *StepContext) StepName() string
- type StoreStep
Constants ¶
const ( KeyContent = "content" // string: loaded text content KeyChunks = "chunks" // []chunker.ChunkResult KeyEmbeddings = "embeddings" // []embedder.EmbedResult KeyEntries = "entries" // []vectorstore.Entry KeyQuery = "query" // string: retrieval query KeyResults = "results" // []retriever.Result KeyAssembled = "assembled" // *assembler.AssembleResult KeyCollectionID = "collection_id" // string KeyDocumentID = "document_id" // string KeyTenantID = "tenant_id" // string )
Context keys for passing data between steps.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AssembleStep ¶
type AssembleStep struct {
// contains filtered or unexported fields
}
AssembleStep builds context from retrieved results. Input: KeyResults ([]retriever.Result) Output: KeyAssembled (*assembler.AssembleResult)
func NewAssembleStep ¶
func NewAssembleStep(a *assembler.Assembler) *AssembleStep
NewAssembleStep creates an assemble step.
func (*AssembleStep) Run ¶
func (s *AssembleStep) Run(ctx context.Context, sc *StepContext) error
Run assembles context from retrieval results.
type ChunkStep ¶
type ChunkStep struct {
// contains filtered or unexported fields
}
ChunkStep splits text into chunks. Input: KeyContent (string) Output: KeyChunks ([]chunker.ChunkResult)
func NewChunkStep ¶
NewChunkStep creates a chunk step.
type EmbedStep ¶
type EmbedStep struct {
// contains filtered or unexported fields
}
EmbedStep generates embeddings for chunks. Input: KeyChunks ([]chunker.ChunkResult), KeyTenantID (string, required) Output: KeyEmbeddings ([]embedder.EmbedResult), KeyEntries ([]vectorstore.Entry)
func NewEmbedStep ¶
NewEmbedStep creates an embed step.
type FuncStep ¶
type FuncStep struct {
// contains filtered or unexported fields
}
FuncStep wraps a function as a pipeline Step.
func NewFuncStep ¶
NewFuncStep creates a step from a function.
type LoadStep ¶
type LoadStep struct {
// contains filtered or unexported fields
}
LoadStep extracts text content using a Loader. Input: KeyContent (io.Reader via raw content string) Output: KeyContent (extracted text)
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline is an ordered sequence of steps executed in series.
type RetrieveStep ¶
type RetrieveStep struct {
// contains filtered or unexported fields
}
RetrieveStep retrieves relevant chunks for a query. Input: KeyQuery (string) Output: KeyResults ([]retriever.Result)
func NewRetrieveStep ¶
func NewRetrieveStep(r retriever.Retriever, opts *retriever.Options) *RetrieveStep
NewRetrieveStep creates a retrieve step.
func (*RetrieveStep) Run ¶
func (s *RetrieveStep) Run(ctx context.Context, sc *StepContext) error
Run retrieves relevant chunks.
type Step ¶
type Step interface {
// Name returns a human-readable name for this step.
Name() string
// Run executes the step, reading from and writing to the step context.
Run(ctx context.Context, sc *StepContext) error
}
Step is a single unit of work in a pipeline.
type StepContext ¶
type StepContext struct {
// Values is a generic key-value store for passing data between steps.
Values map[string]any
// contains filtered or unexported fields
}
StepContext carries accumulated state through pipeline steps.
func NewStepContext ¶
func NewStepContext() *StepContext
NewStepContext creates an empty step context.
func (*StepContext) Get ¶
func (sc *StepContext) Get(key string) (any, bool)
Get retrieves a value from the step context.
func (*StepContext) Keys ¶
func (sc *StepContext) Keys() []string
Keys returns all keys stored in the context.
func (*StepContext) MustGet ¶
func (sc *StepContext) MustGet(key string) any
MustGet retrieves a value or panics if not found.
func (*StepContext) Set ¶
func (sc *StepContext) Set(key string, value any)
Set stores a value in the step context.
func (*StepContext) SetCacheHit ¶
func (sc *StepContext) SetCacheHit(_ string, data any)
SetCacheHit restores cached values into the step context.
func (*StepContext) SetStepName ¶
func (sc *StepContext) SetStepName(name string)
SetStepName sets the current step name (called by the pipeline runner).
func (*StepContext) Snapshot ¶
func (sc *StepContext) Snapshot() map[string]any
Snapshot returns a shallow copy of the current values.
func (*StepContext) StepName ¶
func (sc *StepContext) StepName() string
StepName returns the name of the currently executing step.
type StoreStep ¶
type StoreStep struct {
// contains filtered or unexported fields
}
StoreStep persists vector entries to the vector store. Input: KeyEntries ([]vectorstore.Entry)
func NewStoreStep ¶
func NewStoreStep(vs vectorstore.VectorStore) *StoreStep
NewStoreStep creates a store step.
Source Files