pipeline-segmentation

Overview ¶

Package pipeline_segmentation demonstrates PipePort[T] as a computation pipeline stage boundary — the primary use case. Three named stages (raw → valid → enriched) are connected; side observers tap into any stage without touching the pipeline logic.

Architecture:

SourcePort(sensors) ─→ PipePort("raw") ─Chain(validate)─→ PipePort("valid") ─ChainStream(3 Map steps)─→ PipePort("enriched")
                             ↑ Tap(log)                        ↑ Tap(alerting) ↑ Tap(history)

PipePort is the segmentation glue BETWEEN computation stages. Both stage transitions below use the SAME primitive family:

• ports.Chain(ctx, from, fn, to) — fn is a single value-mapping function. Used for raw→valid (one validation step).
• ports.ChainStream(ctx, from, transform, to) — transform is any gstream.Stream[In] -> gstream.Stream[Out] function, so it can chain as many Map/Filter/etc. steps as a stage needs. Chain is IMPLEMENTED IN TERMS OF ChainStream (wrapping a single gstream.Map) — ChainStream is the general primitive, Chain its single-step convenience. Used for valid→calibrated (three sequential steps: calibrate → classify → annotate).

Neither transition needs a hand-written wrapper function: a multi-step sub-pipeline is just as first-class a ChainStream call as a single-step one is a Chain call. IO/adapter wiring (InputPort/OutputPort with transport adapters) is a supported convenience, not the primary use.

The entire topology — every PipePort, every Chain/ChainStream call, every side observer, every Connect — is declared once in BuildPipeline and never touched again. main is a caller: it applies the already-declared pipeline to one context, then feeds data in and reads results out. This mirrors the "declare once, apply anywhere" shape used throughout go-codex for REST routes, event channels, and forge functions — a pipeline topology is exactly as declarative as any other boundary.

main also demonstrates pipeline spec generation — fully DERIVED, not hand-typed: ports.PipelineSpec reads pipe names, buffer sizes, bound adapter identities, and Chain/ChainStream edges (including the real Go function name behind each edge, captured via reflection) directly from Raw/Valid/Calibrated. Only title, version, and pipe order stay caller-supplied. Rendered via the existing render/stream machinery (see examples/stream-pipeline) — proof that Chain/ChainStream introduced no gap in spec generation, and that the spec cannot silently drift out of sync with the actual wiring the way a hand-typed description could.

Build: go build . Run: ./pipeline-segmentation