Documentation
¶
Index ¶
- type Builder
- func (b *Builder) Build() (*ConduitWithReporter, error)
- func (b *Builder) Cache(path string) *Builder
- func (b *Builder) Concurrency(n int) *Builder
- func (b *Builder) ErrorMode(mode conduit.ErrorMode) *Builder
- func (b *Builder) ListCollector(enable bool) *Builder
- func (b *Builder) Metrics(fn conduit.MetricsFunc) *Builder
- func (b *Builder) Progress(fn conduit.ProgressFunc) *Builder
- func (b *Builder) Reporter(r *progress.Reporter) *Builder
- func (b *Builder) Runtime() *Builder
- func (b *Builder) Splitter(conf processor.ChunkConfig) *Builder
- func (b *Builder) TextCollector(enable bool) *Builder
- func (b *Builder) Workflow(file string, llm chatter.Chatter, sink iosystem.Sink) *Builder
- type ConduitWithReporter
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder creates a configured conduit with processors using builder pattern.
Example:
conduit, err := worker.New().
Blueprint(bp).
Job("process").
Concurrency(4).
Build()
func (*Builder) Build ¶
func (b *Builder) Build() (*ConduitWithReporter, error)
Build creates the configured conduit with blueprint processor. Returns a wrapped conduit that includes progress reporter in context.
func (*Builder) Concurrency ¶
Concurrency sets the number of parallel processing workers. Default is 1 (sequential processing).
func (*Builder) ErrorMode ¶
ErrorMode sets how errors are handled during processing. Options: conduit.FailFast (default) or conduit.SkipError.
func (*Builder) ListCollector ¶ added in v0.2.0
ArrayMode enables batch processing by collecting all documents into array. When enabled, adds ArrayCollector processor as first stage after Runtime. ArrayCollector buffers all documents until EOF, then emits them as array.
Use cases:
- Process JSON array files
- Batch process split documents
- Enable foreach with selector: document
Memory warning: All documents buffered in memory until EOF. Not suitable for very large document streams.
func (*Builder) Metrics ¶
func (b *Builder) Metrics(fn conduit.MetricsFunc) *Builder
Metrics sets a callback for periodic metrics updates.
func (*Builder) Progress ¶
func (b *Builder) Progress(fn conduit.ProgressFunc) *Builder
Progress sets a callback for progress updates after each document.
func (*Builder) TextCollector ¶ added in v0.2.0
type ConduitWithReporter ¶
ConduitWithReporter wraps a conduit and injects progress reporter into context
Source Files