Documentation
¶
Index ¶
- Constants
- func Collect[T any](_ context.Context, stream Stream[T]) ([]T, error)
- func Into[A, B, R any](ctx context.Context, src Source[A], cmd Command[A, B], sink Sink[B, R]) (R, error)
- func NewLineScanner(r io.Reader) *bufio.Scanner
- func Output(stdout io.Writer, data []byte) error
- func OutputString(stdout io.Writer, data string) error
- func OutputStringWithTerminator(stdout io.Writer, terminator byte, data string) error
- func OutputWithTerminator(stdout io.Writer, terminator byte, data []byte) error
- func Pump(ctx context.Context, cmd Command[string, string], stdin io.Reader, ...) (int, error)
- func PumpBytes(ctx context.Context, cmd Command[[]byte, []byte], stdin io.Reader, ...) (int, error)
- func Run(source any, sink any, cmds ...any) (any, error)
- func RunContext(ctx context.Context, source any, sink any, cmds ...any) (any, error)
- type Command
- type Error
- type File
- type FluentPipeline
- func (p *FluentPipeline) Collect() (any, error)
- func (p *FluentPipeline) ForEach(fn any) error
- func (p *FluentPipeline) Sink(sink any) (any, error)
- func (p *FluentPipeline) SinkContext(ctx context.Context, sink any) (any, error)
- func (p *FluentPipeline) To(cmd any) *FluentPipeline
- func (p *FluentPipeline) ToContext(ctx context.Context, cmd any) *FluentPipeline
- type FuncCommand
- type Parameters
- type Pipeline
- type Sink
- type Source
- type Stream
- func From[A, B any](ctx context.Context, src Source[A], cmd Command[A, B]) Stream[B]
- func Generate[T any](ctx context.Context, producer producerFunc[T]) Stream[T]
- func GenerateFrom[In, Out any](ctx context.Context, upstream Stream[In], producer producerFunc[Out]) Stream[Out]
- func StreamOf[T any](items ...T) Stream[T]
- func Wrap[T any](ch <-chan rill.Try[T]) Stream[T]
- func WrapFrom[In, Out any](ch <-chan rill.Try[Out], upstream Stream[In]) Stream[Out]
- type Switch
Examples ¶
Constants ¶
const MaxLineSize = 1 << 30
MaxLineSize bounds how large a single input line may grow (1 GiB). bufio.Scanner's default 64KB token limit would abort pipelines on long lines (minified JSON, JSONL, long log lines), where shell tools like cat, grep, and sed handle arbitrary line lengths. The buffer grows on demand, so the cap costs no memory for typical input.
const (
NewLine = byte('\n')
)
Variables ¶
This section is empty.
Functions ¶
func Collect ¶ added in v0.0.7
Collect gathers all stream items into a slice. On the first error it stops the upstream and drains, so no producer goroutine leaks.
func Into ¶ added in v0.0.7
func Into[A, B, R any](ctx context.Context, src Source[A], cmd Command[A, B], sink Sink[B, R]) (R, error)
Into wires a source through a command into a sink, returning the sink's result. If the sink returns early (an error, or an early-exit sink), the stream is stopped AND drained so no upstream producer goroutine leaks — including pure rill transform stages with no cancellation hook.
func NewLineScanner ¶ added in v0.0.7
NewLineScanner returns a line scanner that handles lines up to MaxLineSize, matching shell line-tool semantics. Command authors reading lines from an io.Reader directly should use this instead of bufio.NewScanner to avoid the default 64KB line-length limit.
func OutputString ¶ added in v0.0.7
OutputString writes a string line followed by a newline to stdout. Uses io.WriteString to avoid a string→[]byte copy when the writer implements io.StringWriter (e.g., os.File, bytes.Buffer).
func OutputStringWithTerminator ¶ added in v0.0.7
OutputStringWithTerminator writes a string followed by a terminator byte to stdout.
func OutputWithTerminator ¶ added in v0.0.7
func Pump ¶ added in v0.0.7
func Pump(ctx context.Context, cmd Command[string, string], stdin io.Reader, stdout io.Writer) (int, error)
Pump runs a string command as a Unix-style filter: it pumps lines from stdin through cmd and out to stdout, returning the number of lines written. It is the bridge from a composed gloo command to a `func main` — the framework analogue of wiring a tool into a shell pipe. Nothing is shelled out; the work stays in-process.
func main() {
upper := patterns.Map(strings.ToUpper) // string→string
if _, err := gloo.Pump(context.Background(), upper, os.Stdin, os.Stdout); err != nil {
log.Fatal(err)
}
}
Example ¶
ExamplePump wires a composed command to stdin/stdout as a Unix filter — the bridge from a gloo pipeline to a func main. Nothing is shelled out.
package main
import (
"context"
"os"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
// grep error | head -2, reading "stdin" and writing "stdout".
cmd := gloo.Compose(
patterns.Filter(func(line string) (bool, error) {
return strings.Contains(line, "error"), nil
}),
).To(patterns.Head[string](2))
stdin := strings.NewReader("error: a\ninfo: b\nerror: c\nerror: d\n")
_, _ = gloo.Pump(context.Background(), cmd, stdin, os.Stdout)
}
Output: error: a error: c
func PumpBytes ¶ added in v0.0.7
func PumpBytes(ctx context.Context, cmd Command[[]byte, []byte], stdin io.Reader, stdout io.Writer) (int, error)
PumpBytes is Pump for byte commands: it pumps lines from stdin through a Command[[]byte, []byte] and out to stdout.
func Run ¶
Run executes a pipeline in one call: source → commands → sink. Uses context.Background().
gloo.Run(source, sink, cmd1, cmd2, cmd3)
Example ¶
package main
import (
"os"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
src := gloo.SliceSource([]string{"hello", "world"})
upper := patterns.Map(func(line string) (string, error) {
return strings.ToUpper(line), nil
})
_, _ = gloo.Run(src, gloo.WriteTo(os.Stdout), upper)
}
Output: HELLO WORLD
Types ¶
type Command ¶
Command transforms an input stream to an output stream.
func Pipe ¶ added in v0.0.4
Pipe composes two commands into a new command. It connects the output of cmd1 to the input of cmd2. The result is immutable and safe to reuse across pipelines.
Example ¶
package main
import (
"context"
"fmt"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
ctx := context.Background()
src := gloo.SliceSource([]string{"apple red", "banana yellow", "cherry red"})
grep := patterns.Filter(func(line string) (bool, error) {
return strings.Contains(line, "red"), nil
})
upper := patterns.Map(func(line string) (string, error) {
return strings.ToUpper(line), nil
})
composed := gloo.Pipe(grep, upper)
stream := gloo.From(ctx, src, composed)
results, _ := stream.Collect()
for _, r := range results {
fmt.Println(r)
}
}
Output: APPLE RED CHERRY RED
type Error ¶ added in v0.0.7
type Error string
Error is the framework's sentinel-error type. Declare every error the package can emit as a const of this type, so each path is matchable with errors.Is instead of by string comparison.
const ( ErrNotSource Error = "gloo: argument must implement Source (Stream(context.Context) Stream[Out])" ErrNotCommand Error = "gloo: stage must implement Command (Execute(context.Context, Stream[In]) Stream[Out])" ErrNotSink Error = "gloo: sink must implement Sink (Consume(context.Context, Stream[In]) (Res, error))" ErrStageTypeMismatch Error = "gloo: pipeline stage type mismatch" ErrSinkTypeMismatch Error = "gloo: sink type mismatch" ErrNotForEachFunc Error = "gloo: ForEach argument must be func(T) error" ErrPipelineConsumed Error = "gloo: pipeline already consumed by a terminal operation" )
Validation errors surfaced by the fluent builder. The reflection-based Chain API cannot enforce stage types at compile time (Go has no generic methods yet), so a mismatch is reported here as a value you can match with errors.Is — never a panic. The first error a builder hits is sticky: later .To() calls are no-ops and the terminal (.Sink/.Collect/.ForEach) returns it.
const ErrFileNotFound Error = "file not found"
const ErrStopReading Error = "gloo: downstream stopped reading"
ErrStopReading is the cancellation cause used when a downstream consumer stops reading — the SIGPIPE analogue. A producer cancelled with this cause closes its stream SILENTLY: no error item is emitted, because downstream completion is graceful, not a failure (a shell without pipefail does not fail a pipeline when `head` exits early).
func (Error) With ¶ added in v0.0.7
With wraps a cause and appends contextual args. A non-nil cause is joined with %w so errors.Is still matches both the sentinel and the cause. The args are rendered space-separated, so callers pass clean key/value pairs — .With(err, "file", name) — without baking separators into the key. (fmt.Sprint would concatenate adjacent strings with no space, collapsing "file"+name into one token.)
type FluentPipeline ¶ added in v0.0.7
type FluentPipeline struct {
// contains filtered or unexported fields
}
FluentPipeline is a pipeline builder created by Chain or ChainContext. Each .To() records a command and validates its type against the running element type via reflection — without wiring any stream. Terminal operations (.Sink(), .Collect(), .ForEach()) build the stream inside their own scope and consume it.
Pointer receiver required: each .To() advances the builder's reflected element type. Single-owner — do not share across goroutines. Once a terminal operation is called the pipeline is consumed and cannot be reused. Any validation error is sticky and surfaced by the terminal.
func Chain ¶ added in v0.0.7
func Chain(source any) *FluentPipeline
Chain starts a fluent pipeline from a source using context.Background(). Use .To() to chain commands and .Sink()/.Collect()/.ForEach() to consume.
gloo.Chain(source).To(grepCmd).To(sortCmd).Sink(sink)
For compile-time-checked composition (no reflection, no runtime type errors), prefer Pipe and Compose; Chain trades that safety for a fluent, type-changing syntax until Go ships generic methods.
Example ¶
package main
import (
"fmt"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
src := gloo.SliceSource([]string{"apple red", "banana yellow", "cherry red"})
grep := patterns.Filter(func(line string) (bool, error) {
return strings.Contains(line, "red"), nil
})
upper := patterns.Map(func(line string) (string, error) {
return strings.ToUpper(line), nil
})
result, _ := gloo.Chain(src).
To(grep).
To(upper).
Collect()
for _, item := range result.([]string) {
fmt.Println(item)
}
}
Output: APPLE RED CHERRY RED
Example (ForEach) ¶
package main
import (
"fmt"
gloo "github.com/gloo-foo/framework"
)
func main() {
src := gloo.SliceSource([]string{"one", "two", "three"})
_ = gloo.Chain(src).ForEach(func(line string) error {
fmt.Println(line)
return nil
})
}
Output: one two three
Example (Sink) ¶
package main
import (
"fmt"
"os"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
src := gloo.SliceSource([]string{"hello", "world"})
upper := patterns.Map(func(line string) (string, error) {
return strings.ToUpper(line), nil
})
count, _ := gloo.Chain(src).To(upper).Sink(gloo.WriteTo(os.Stdout))
fmt.Printf("%d lines written\n", count)
}
Output: HELLO WORLD 2 lines written
func ChainContext ¶ added in v0.0.7
func ChainContext(ctx context.Context, source any) *FluentPipeline
ChainContext starts a fluent pipeline from a source with an explicit context. The context is inherited by .To() stages. Use .ToContext() to override the context for individual stages.
Wiring is LAZY: the source and commands are validated for type compatibility at build time (by inspecting method signatures, without invoking them) but the stream is not created until a terminal operation runs. A chain that is built and never consumed therefore starts no goroutines and leaks nothing. A validation failure is recorded and returned by the terminal, never panicked.
func (*FluentPipeline) Collect ¶ added in v0.0.7
func (p *FluentPipeline) Collect() (any, error)
Collect is a terminal operation that collects all stream items into a slice. Returns ([]T, error) as (any, error) where T is the final element type.
func (*FluentPipeline) ForEach ¶ added in v0.0.7
func (p *FluentPipeline) ForEach(fn any) error
ForEach is a terminal operation that calls fn for each item in the stream. fn must be func(T) error where T matches the final element type.
func (*FluentPipeline) Sink ¶ added in v0.0.7
func (p *FluentPipeline) Sink(sink any) (any, error)
Sink is a terminal operation that consumes the stream via a Sink, using the chain's context. Returns (Res, error) as (any, error).
func (*FluentPipeline) SinkContext ¶ added in v0.0.7
SinkContext is a terminal operation that consumes the stream via a Sink with an explicit context.
func (*FluentPipeline) To ¶ added in v0.0.7
func (p *FluentPipeline) To(cmd any) *FluentPipeline
To records a Command into the chain, using the chain's context. The command must implement Command[In, Out] where In matches the current element type.
func (*FluentPipeline) ToContext ¶ added in v0.0.7
func (p *FluentPipeline) ToContext(ctx context.Context, cmd any) *FluentPipeline
ToContext records a Command into the chain with an explicit context, overriding the chain's context for this stage only. A type or shape mismatch is recorded as the pipeline's sticky error; it is not raised until a terminal.
type FuncCommand ¶ added in v0.0.7
FuncCommand adapts a function to the Command interface. Value type — no pointer receiver, safe to copy and reuse.
type Parameters ¶ added in v0.0.7
Parameters holds parsed command parameters. It is an immutable value.
func NewParameters ¶ added in v0.0.7
func NewParameters[P any, F any](parameters ...any) Parameters[P, F]
NewParameters classifies a heterogeneous argument list into typed positional values, Switch[F] flags, and an ambiguous bin. Files named by File positionals are opened lazily by Reader/ReadersFrom, not here.
func (Parameters[P, F]) Reader ¶ added in v0.0.7
func (p Parameters[P, F]) Reader(stdin io.Reader) (io.ReadCloser, error)
Reader returns a combined reader over all positional arguments, falling back to stdin when there are none, using the OS filesystem for File positionals.
The returned io.ReadCloser owns the file handles it opened: callers MUST Close it to release them. Closing also closes any positional io.ReadCloser arguments; the fallback stdin reader is wrapped so Close is a no-op.
func (Parameters[P, F]) ReaderFrom ¶ added in v0.0.7
func (p Parameters[P, F]) ReaderFrom(fs afero.Fs, stdin io.Reader) (io.ReadCloser, error)
ReaderFrom is Reader with an explicit filesystem for opening File positionals.
func (Parameters[P, F]) ReadersFrom ¶ added in v0.0.7
func (p Parameters[P, F]) ReadersFrom(fs afero.Fs) ([]io.ReadCloser, error)
ReadersFrom opens file handles from positional args using the given filesystem. On error, all previously opened handles are closed.
type Pipeline ¶ added in v0.0.4
type Pipeline[T any] struct { // contains filtered or unexported fields }
Pipeline is an immutable same-type command chain. Implements Command[T, T]. Value receiver on all methods — safe to copy, assign, and reuse.
func Compose ¶ added in v0.0.7
Compose starts a same-type command chain from an initial Command[T, T]. Each .To() returns a NEW Pipeline (immutable). The resulting Pipeline implements Command[T, T] and is safe to reuse across multiple pipelines.
pipeline := gloo.Compose(grepCmd).To(sortCmd).To(uniqCmd) // pipeline implements Command[T, T]
Example ¶
package main
import (
"fmt"
"strings"
gloo "github.com/gloo-foo/framework"
"github.com/gloo-foo/framework/patterns"
)
func main() {
src := gloo.SliceSource([]string{"hello", "world"})
addBang := patterns.Map(func(line string) (string, error) {
return line + "!", nil
})
upper := patterns.Map(func(line string) (string, error) {
return strings.ToUpper(line), nil
})
composed := gloo.Compose(addBang).To(upper)
result, _ := gloo.Chain(src).To(composed).Collect()
for _, item := range result.([]string) {
fmt.Println(item)
}
}
Output: HELLO! WORLD!
type Sink ¶ added in v0.0.7
Sink consumes a stream and produces a typed result.
func ByteWriteTo ¶ added in v0.0.7
ByteWriteTo creates a Sink that writes each []byte item as a line to the given writer.
type Source ¶ added in v0.0.7
Source produces a stream from external data.
func ByteFileSource ¶ added in v0.0.7
ByteFileSource creates a stream of []byte lines from files on the given filesystem. Each line is an independent copy, safe to retain after the next scan.
func ByteReaderSource ¶ added in v0.0.7
ByteReaderSource creates a stream of []byte lines from readers. Each line is an independent copy, safe to retain after the next scan.
func FileSource ¶ added in v0.0.7
FileSource creates a stream of string lines from files on the given filesystem.
Example ¶
package main
import (
"context"
"fmt"
gloo "github.com/gloo-foo/framework"
"github.com/spf13/afero"
)
func main() {
fs := afero.NewMemMapFs()
_ = afero.WriteFile(fs, "test.txt", []byte("line one\nline two\nline three"), 0o644)
ctx := context.Background()
src := gloo.FileSource(fs, []gloo.File{"test.txt"})
results, _ := src.Stream(ctx).Collect()
for _, r := range results {
fmt.Println(r)
}
}
Output: line one line two line three
func ReaderSource ¶ added in v0.0.7
ReaderSource creates a stream of string lines from readers.
func SliceSource ¶ added in v0.0.7
SliceSource creates a Source from an in-memory slice. This is the equivalent of shell here-strings or echo piping.
Example ¶
package main
import (
"context"
"fmt"
gloo "github.com/gloo-foo/framework"
)
func main() {
ctx := context.Background()
src := gloo.SliceSource([]string{"alpha", "bravo", "charlie"})
results, _ := src.Stream(ctx).Collect()
for _, r := range results {
fmt.Println(r)
}
}
Output: alpha bravo charlie
type Stream ¶ added in v0.0.7
type Stream[T any] struct { // contains filtered or unexported fields }
Stream is the framework's pipe: a channel of Try containers (each holding a value or an error) carrying a teardown handle.
Teardown is the SIGPIPE analogue. In a shell, when a downstream stage exits the pipe closes and the next upstream write dies of SIGPIPE, terminating the producer. Stream reproduces that — but makes the safe path the ONLY path. A consumer has exactly three operations, all leak-free: range Chan() to completion, Collect() every item, or Discard() to abandon early. There is deliberately no bare "stop": stopping the upstream without also draining would strand a producer blocked on a send nobody reads (a goroutine leak), so the framework fuses the two into Discard and never exposes the unsafe half. Teardown propagates UPSTREAM only, so `infinite | Take(3) | sort` still sorts the three items.
Command authors using a pattern from patterns/ never touch the channel or teardown. Authors writing an exotic FuncCommand build output with GenerateFrom/WrapFrom, passing the INPUT stream so teardown chains to it — they hold no loose stop handle either.
func From ¶ added in v0.0.7
From wires a source into a command, returning the output stream. The caller owns the returned stream: consume it fully (e.g. via Collect, which drains), or call Discard to abandon it — Discard both stops the upstream and drains, which Stop alone does not (pure rill transform stages must be drained to unblock).
func Generate ¶ added in v0.0.7
Generate runs a cancellation-aware ORIGIN producer (a Source, with no upstream) and returns its Stream. It is the primitive every origin producer is built on. For a producer derived from an input stream, use GenerateFrom so teardown chains upstream.
Teardown semantics, derived from context.Cause of the producer scope:
- Downstream stop (cause == ErrStopReading): close silently.
- External cancellation (any other cause, e.g. ^C / deadline): emit the cause once — unless the producer already emitted an error — then close.
- Natural completion: close.
The returned Stream's Discard cancels this producer scope with ErrStopReading.
func GenerateFrom ¶ added in v0.0.7
func GenerateFrom[In, Out any](ctx context.Context, upstream Stream[In], producer producerFunc[Out]) Stream[Out]
GenerateFrom is Generate for a producer that derives its output from an upstream stream. A downstream Discard cancels this producer AND tears the upstream down, so one Discard collapses the whole chain (upstream only). Authors pass the INPUT stream rather than a loose stop handle, so the safe teardown is the only thing in reach.
func StreamOf ¶ added in v0.0.7
StreamOf builds a finished Stream from in-memory values. It is the canonical way to feed synthetic input to a command in a test — the framework analogue of a shell here-string — and backs SliceSource.
func Wrap ¶ added in v0.0.7
Wrap adapts a standalone channel (an ORIGIN, with no upstream to tear down) into a Stream. StreamOf and a FuncCommand that synthesizes a channel from nothing use it. A transform that wraps a rill stage over an input stream must use WrapFrom so teardown chains upstream.
func WrapFrom ¶ added in v0.0.7
WrapFrom adapts the output channel of a pure rill transform into a Stream, chaining teardown to the upstream stream so a downstream Discard propagates. A pure transform has no producer scope of its own — it terminates when its input closes — so forwarding teardown is all that is required. The forward is guarded so it stays idempotent however many times it fires.
out := rill.OrderedMap(in.Chan(), 1, fn) return gloo.WrapFrom(out, in)
func (Stream[T]) Chan ¶ added in v0.0.7
Chan returns the underlying channel for direct rill interop. Pattern plumbing hides this; only FuncCommand authors handling cases no pattern covers use it. A consumer that ranges Chan() must read it to completion or hand the stream to Discard — never abandon it half-read.
func (Stream[T]) Collect ¶ added in v0.0.7
Collect drains the stream into a slice. On the first error it abandons the rest via Discard (stop + drain) so no producer goroutine leaks, and returns the error. It is the canonical terminal for tests and simple consumers.
func (Stream[T]) Discard ¶ added in v0.0.7
func (s Stream[T]) Discard()
Discard abandons a stream you will not finish consuming. It signals every upstream producer to stop and drains the channel, releasing each one — including pure rill transform stages that have no cancellation hook and would otherwise block on a send nobody reads. It is idempotent and is the ONLY way to walk away from a partially-consumed stream; the framework exposes no stop-without-drain precisely because that combination leaks.