Documentation
¶
Index ¶
Constants ¶
const ( ChunkerNone = "none" ChunkerSentence = "sentence" ChunkerParagraph = "paragraph" ChunkerChunk = "chunk" ChunkerTag = "tag" )
Chunking strategies
Variables ¶
This section is empty.
Functions ¶
func NewChunker ¶
func NewChunker(config ChunkConfig) iosystem.Processor
NewChunker creates a processor that splits documents into chunks. Returns multiple documents from a single input, one per chunk.
func NewIdentity ¶
NewIdentity creates a processor that passes documents through unchanged.
Types ¶
type Agent ¶
type Agent struct {
// contains filtered or unexported fields
}
Agent wraps a blueprint to use as a pipeline processor. It processes documents through the agent's Prompt() method and returns the agent's response as a new document.
The processor:
- Reads document content as input
- Passes it to agent.Prompt()
- Returns agent response as new document
- Preserves document path with optional suffix
- Supports JSON format output from agents
Example:
agent := getAgentFromBlueprint()
proc := NewAgentProcessor(agent, AgentConfig{
Suffix: ".processed",
})
docs, err := proc.Process(ctx, inputDoc)
func NewAgent ¶
func NewAgent(w Worker, config *AgentConfig) *Agent
NewAgent creates a processor that wraps a blueprint Agent.
func (*Agent) Close ¶
Close releases resources. For AgentProcessor, this is a no-op as the agent lifecycle is managed externally.
func (*Agent) Process ¶
Process transforms a document by passing its content through the agent.
Input document content is read and passed to agent.Prompt() as:
- string if content is valid UTF-8
- []byte otherwise
- map[string]any if document has metadata that should be templated
The agent's response becomes the content of the output document. Output format depends on agent's configuration (text or JSON).
type AgentConfig ¶
type AgentConfig struct {
Suffix string // Suffix to add to output document path (default: empty)
Options []chatter.Opt // Chatter options to pass to agent.Prompt() (temperature, max_tokens, etc.)
}
AgentConfig configures the agent processor.
type ChunkConfig ¶
type ChunkConfig struct {
Strategy string // "none", "sentence", "paragraph", "chunk"
ChunkSize int // Size for chunk strategy (default: 1024)
DelimiterChars string // Delimiter characters (defaults vary by strategy)
Buffer int
}
ChunkConfig configures the chunker processor.
type Chunker ¶
type Chunker struct {
// contains filtered or unexported fields
}
Chunker splits documents into chunks based on a strategy. Integrates with the existing github.com/fogfish/scanner library.
type Identity ¶
type Identity struct{}
Identity is a pass-through processor that returns documents unchanged. Useful for testing and as a base for more complex processors.
type Jsonify ¶
type Jsonify struct {
// contains filtered or unexported fields
}
Jsonify is a processor that formats JSON documents with color and indentation. It only processes documents with content-type "application/json" in metadata. Other documents are passed through unchanged.
The processor:
- Checks document metadata for "content-type"
- Formats JSON with color syntax highlighting
- Pretty-prints with configurable indentation
- Passes through non-JSON documents unchanged
Example:
proc := NewJSONFormatter(JSONFormatConfig{
Indent: 2,
Color: true,
})
docs, err := proc.Process(ctx, inputDoc)
func NewJsonify ¶
func NewJsonify(config JsonifyConfig) *Jsonify
NewJsonify creates a processor that formats JSON documents.
type JsonifyConfig ¶
type JsonifyConfig struct {
Indent int // Number of spaces for indentation (default: 2)
Color bool // Enable color syntax highlighting (default: true)
}
JsonifyConfig configures the JSON formatter processor.
type Prompter ¶
type Prompter struct {
// contains filtered or unexported fields
}
Wrap LLM as processor
func NewPrompter ¶
NewPrompter creates a processor that wraps a blueprint Prompter.
func (*Prompter) Close ¶
Close releases resources. For AgentProcessor, this is a no-op as the agent lifecycle is managed externally.
func (*Prompter) Process ¶
func (p *Prompter) Process(ctx context.Context, doc *iosystem.Document) ([]*iosystem.Document, error)
Process transforms a document by passing its content through the agent.
Input document content is read and passed to agent.Prompt() as:
- string if content is valid UTF-8
- []byte otherwise
- map[string]any if document has metadata that should be templated
The agent's response becomes the content of the output document. Output format depends on agent's configuration (text or JSON).
Source Files