stream

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 17, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package stream moves records from a Source through a Cascade into a Sink. Unclassified records go to an optional review sink. A nil review sink drops them.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine drives records through the cascade.

func New

func New(options Options) (*Engine, error)

New validates options and returns a ready Engine.

func (*Engine) Run

func (e *Engine) Run(ctx context.Context) (Stats, error)

Run drains the source through the cascade. It returns when the source is drained, the context is cancelled, or a source, stage, or sink fails. Stats reflect everything processed up to the point of return.

type Options

type Options struct {
	// Source yields records and is required.
	Source source.Source
	// Cascade classifies records and is required.
	Cascade *classmesh.Cascade
	// Sink receives classified records and is required.
	Sink sink.Sink
	// Review receives unclassified records with a zero Classification.
	// A nil Review drops those records.
	Review sink.Sink
	// Logger defaults to slog.Default.
	Logger *slog.Logger
	// Workers controls concurrent classification. Zero or one uses the serial loop.
	// Larger values require concurrent-safe stages and preserve serial outcomes.
	Workers int
}

Options configures an Engine.

type Stats

type Stats struct {
	// Processed counts records read from the source.
	Processed int
	// Classified counts records some stage decided.
	Classified int
	// Reviewed counts records no stage decided.
	Reviewed int
	// ByStage counts classifications per stage name.
	ByStage map[string]int
}

Stats summarizes one Run.

Directories

Path Synopsis
Package sink defines where classified records go: stdout, files, review queues, downstream pipelines.
Package sink defines where classified records go: stdout, files, review queues, downstream pipelines.
jsonl
Package jsonl implements sink.Sink as JSON Lines: one JSON object per classified record, the pipe-friendly lingua franca of log tooling.
Package jsonl implements sink.Sink as JSON Lines: one JSON object per classified record, the pipe-friendly lingua franca of log tooling.
Package source defines where Records come from.
Package source defines where Records come from.
jsonl
Package jsonl implements source.Source over JSON Lines: one JSON object per line.
Package jsonl implements source.Source over JSON Lines: one JSON object per line.
text
Package text implements source.Source over line-oriented text: files and stdin.
Package text implements source.Source over line-oriented text: files and stdin.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL