flow

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package flow implements stream-based message routing through a block graph, mirroring Astarte Flow's core concepts: messages belong to streams (identified by key), streams are processed in order within a lane, and different streams may interleave across lanes.

Index

Constants

View Source
const DefaultLaneCapacity = 256

DefaultLaneCapacity is the default per-lane channel capacity.

View Source
const DefaultLaneCount = 16

DefaultLaneCount is the default number of processing lanes.

View Source
const WireSchema = "astarte_flow/message/v0.1"

WireSchema is the JSON schema identifier used on the wire.

Variables

View Source
var (
	// ErrFlowExists is returned by StartFlow when a flow with the same
	// instance key is already registered.
	ErrFlowExists = errors.New("flow: already running")
	// ErrFlowNotFound is returned when a flow ID does not match any
	// registered flow.
	ErrFlowNotFound = errors.New("flow: not found")
)
View Source
var ErrUnknownBlockType = errors.New("flow: unknown block type")

ErrUnknownBlockType is returned when Instantiate sees a block_type with no registered constructor.

View Source
var ErrVirtualDeviceRegistered = errors.New("flow: virtual device already registered")

ErrVirtualDeviceRegistered reports that a first-seen virtual device could not be auto-registered because its id is already registered and confirmed: upstream answers 422 already_registered and drops the message, and astrate mirrors that (the message is dropped, never an error).

Functions

func ExpandComposites

func ExpandComposites(def []byte, resolve func(name string) (*UserBlock, error)) ([]byte, error)

ExpandComposites inlines every user-defined block found in def (issue #85): a node whose block_type resolves through resolve is replaced by its stored sub-chain, spliced in place — incoming edges attach to the sub-chain's first node and outgoing edges leave from its last node (stable topological order). Nested composites are expanded recursively and their block names nest as "outer.inner.block". resolve returns (nil, nil) for built-in types, which pass through unchanged. Definitions are handled as raw node/connection slices; Pipeline.Validate is never run on inner bodies.

func InstanceID

func InstanceID(realm, flowName string) string

InstanceID builds the Manager map key for a named flow instance (realm + "/" + flowName). Different flows may share one pipeline name.

func PipelineID

func PipelineID(realm, name string) string

PipelineID is a legacy alias for InstanceID. Prefer InstanceID; the map key is the flow instance name, not the pipeline name.

func SubstituteConfig

func SubstituteConfig(definition []byte, config map[string]any) ([]byte, error)

SubstituteConfig walks a pipeline definition JSON and replaces ${config.key} placeholders inside string values only. Missing keys or non-stringable config values fail loudly. Non-string JSON leaves are left unchanged.

Types

type Block

type Block interface {
	// Process handles one message. A source receives msg == nil and may return
	// zero or more messages. A transform receives exactly one non-nil message
	// and may return zero or more. A sink returns nil.
	Process(msg *Message) ([]*Message, error)
	// Name returns a human-readable label for metrics and logging.
	Name() string
}

Block is a computation unit in a flow graph. Implementations must be safe for concurrent use by a single lane goroutine (one goroutine calls Process sequentially per message); external concurrency safety is the Router's job.

Three roles exist:

  • Source: produces messages from external events (see the Source interface).
  • Transform: consumes one message and emits zero or more transformed messages.
  • Sink: consumes messages for external output (return value is ignored).

func NewSinkBlock

func NewSinkBlock(name string, fn SinkFunc) Block

NewSinkBlock wraps fn as a Block with the given name.

func NewSourceBlock

func NewSourceBlock(name string, fn SourceFunc) Block

NewSourceBlock wraps fn as a Source Block with the given name.

func NewTransformBlock

func NewTransformBlock(name string, fn TransformFunc) Block

NewTransformBlock wraps fn as a Block with the given name.

type BlockGraph

type BlockGraph struct {
	// contains filtered or unexported fields
}

BlockGraph is a linear chain of blocks: source → transform₁ → … → sink. The graph is immutable after construction; calling Run feeds one message through every non-Source stage. Source blocks are driven by the flow source pump (see Manager.StartFlow), not by Run.

func NewBlockGraph

func NewBlockGraph(blocks ...Block) (*BlockGraph, error)

NewBlockGraph validates that the graph has at least one block and that the last block is a sink (it may return nil messages). The first block is typically the source. Returns an error if the chain is empty or nil blocks are present.

func (*BlockGraph) Blocks

func (g *BlockGraph) Blocks() []Block

Blocks returns the graph's blocks in order. The slice must not be mutated.

func (*BlockGraph) Run

func (g *BlockGraph) Run(msg *Message) ([]*Message, error)

Run feeds one message through every non-Source stage. Source stages are skipped: they are polled by the source pump and their outputs are submitted into the router, which calls Run. Returns the messages produced by the final stage — typically nil for a sink.

func (*BlockGraph) Sources

func (g *BlockGraph) Sources() []Source

Sources returns every block that implements Source, in graph order.

type Config

type Config struct {
	// PipelineID is the Manager map key for this instance (realm/flowName).
	// Historical field name; value is InstanceID, not the pipeline recipe name.
	PipelineID string
	// Blocks is the ordered list of blocks forming the processing graph.
	Blocks []Block
	// RouterCfg is applied to the underlying Router.
	RouterCfg RouterConfig
	// Registerer receives Prometheus collectors; nil leaves them
	// unregistered.
	Registerer prometheus.Registerer
}

Config holds the parameters needed to instantiate a running flow.

type Connection

type Connection struct {
	// From is the name of the source block.
	From string `json:"from"`
	// FromPort is the output port on the source block (empty means default).
	FromPort string `json:"from_port,omitempty"`
	// To is the name of the target block.
	To string `json:"to"`
	// ToPort is the input port on the target block (empty means default).
	ToPort string `json:"to_port,omitempty"`
}

Connection describes a typed edge between two blocks in a pipeline.

type Constructor

type Constructor func(name string, config map[string]any, deps Deps) (Block, error)

Constructor builds one Block from a pipeline node. name is the pipeline node name and should be returned by Block.Name() for metrics and logging.

type DataType

type DataType uint8

DataType enumerates the wire-level value types Astarte Flow supports.

const (
	TypeInteger DataType = iota
	TypeReal
	TypeBoolean
	TypeDatetime
	TypeBinary
	TypeString
	// TypeMap indicates the message carries a map payload. Per-field types
	// and subtypes are in FieldTypes and FieldSubtypes.
	TypeMap
)

DataType values, in wire-encoding order. TypeMap is the only aggregate; its per-field types live in FieldTypes and FieldSubtypes.

type Deps

type Deps struct {
	// Bus is the live event bus (required by astarte_source).
	Bus *stream.Bus
	// Realm is the tenant the pipeline runs for; sources may default to it
	// when their config omits realm.
	Realm string
	// FlowName is the durable/named flow instance name (optional). Used by
	// container labels and similar operator metadata.
	FlowName string
	// NotifyFatal, when set, is called by blocks that die at runtime (e.g. a
	// container block whose container exits unexpectedly) so the service layer
	// can fail the flow and schedule a restart. Must be safe to call from any
	// goroutine; blocks must not call it after their Stopper.Stop() ran.
	NotifyFatal func(block string, cause error)
	// Ingest, when set, lets a block land data as if a registered device had
	// produced it (virtual_device_pool, issue #84): full validation plus a
	// datastream/property row, no MQTT delivery. Required by
	// virtual_device_pool; every other block ignores it.
	Ingest IngestFunc
	// Register, when set, lets a block register a first-seen virtual device
	// through the pairing door (virtual_device_pool auto_register, #84).
	// Required by virtual_device_pool with auto_register; every other block
	// ignores it.
	Register RegisterFunc
}

Deps holds process-level dependencies block constructors may need. Zero fields are valid when a pipeline uses only blocks that do not need them.

type Flow

type Flow struct {
	// contains filtered or unexported fields
}

Flow is a running instance of a pipeline. It owns a Router that processes messages through a BlockGraph, a source pump that feeds Source blocks into the router, and exposes status information.

func (*Flow) CreatedAt

func (f *Flow) CreatedAt() time.Time

CreatedAt returns when the flow was created.

func (*Flow) ID

func (f *Flow) ID() string

ID returns the flow's unique identifier.

func (*Flow) PipelineID

func (f *Flow) PipelineID() string

PipelineID returns the pipeline this flow was created from.

func (*Flow) Router

func (f *Flow) Router() *Router

Router returns the underlying router for message submission.

func (*Flow) StartedAt

func (f *Flow) StartedAt() time.Time

StartedAt returns when the flow started processing. Zero value if not yet started.

func (*Flow) Status

func (f *Flow) Status() Status

Status returns the current lifecycle status.

func (*Flow) StoppedAt

func (f *Flow) StoppedAt() time.Time

StoppedAt returns when the flow was stopped. Zero value if still running.

type IngestFunc

type IngestFunc func(ctx context.Context, realm, deviceID, ifaceName, path string, payload json.RawMessage, ts *time.Time) error

IngestFunc lands data as if a registered device had produced it: full validation plus storage, no MQTT delivery (virtual_device_pool, #84).

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages the lifecycle of running flows. It is safe for concurrent use.

func NewManager

func NewManager() *Manager

NewManager returns an empty flow manager.

func (*Manager) GetFlowStatus

func (m *Manager) GetFlowStatus(pipelineID string) (Status, error)

GetFlowStatus returns the current status of the flow identified by pipelineID.

func (*Manager) ListFlows

func (m *Manager) ListFlows() []*Flow

ListFlows returns a snapshot of all registered flows.

func (*Manager) Shutdown

func (m *Manager) Shutdown(ctx context.Context) error

Shutdown drains all running flows. It is intended for orderly process exit; each flow gets the same context deadline.

func (*Manager) StartFlow

func (m *Manager) StartFlow(ctx context.Context, cfg Config) (*Flow, error)

StartFlow instantiates a pipeline into a running Flow. The block graph is built before the flow is registered so construction failures leave no map entry (durable layer records status=failed separately). On success the status transitions to running.

func (*Manager) StopFlow

func (m *Manager) StopFlow(ctx context.Context, pipelineID string) error

StopFlow gracefully shuts down the flow identified by pipelineID. It stops the source pump, drains in-flight messages, calls Stop on every Stopper block, and transitions the status to stopped.

func (*Manager) UnregisterFlow

func (m *Manager) UnregisterFlow(instanceID string)

UnregisterFlow removes an instance key from the manager map. Call after StopFlow when deleting a durable row so the name can be reused.

type Message

type Message struct {
	// Key identifies the stream this message belongs to. It must be non-empty.
	Key string
	// Metadata is an optional string→string map carried alongside the payload.
	Metadata map[string]string
	// Type is the base data type of the payload.
	Type DataType
	// Subtype is an optional MIME hint (meaningful when Type is TypeBinary).
	Subtype string
	// Timestamp is the event-time in microseconds since Unix epoch.
	Timestamp int64
	// Data is the payload; its concrete type must match Type (int64 for
	// TypeInteger, float64 for TypeReal, bool for TypeBoolean, time.Time for
	// TypeDatetime, []byte for TypeBinary, string for TypeString, map[string]any
	// for TypeMap).
	Data any
	// FieldTypes holds per-field types when Type is TypeMap.
	FieldTypes map[string]DataType
	// FieldSubtypes holds per-field subtypes when Type is TypeMap.
	FieldSubtypes map[string]string
}

Message is one unit of data flowing through a block graph. Every message carries a Key that identifies its stream; messages with the same key are processed in submission order by the same lane (consistent hashing).

func (*Message) MarshalJSON

func (m *Message) MarshalJSON() ([]byte, error)

MarshalJSON serialises the Message to the upstream JSON wire format.

func (*Message) UnmarshalJSON

func (m *Message) UnmarshalJSON(b []byte) error

UnmarshalJSON deserialises a Message from the upstream JSON wire format.

type OverflowPolicy

type OverflowPolicy uint8

OverflowPolicy controls what happens when a lane's channel is full.

const (
	// OverflowBlock blocks the caller until space is available (QoS ≥ 1).
	OverflowBlock OverflowPolicy = iota
	// OverflowDrop discards the message without blocking (QoS 0).
	OverflowDrop
)

type Pipeline

type Pipeline struct {
	// ID is a unique identifier for this pipeline.
	ID string `json:"id"`
	// Name is a human-readable label.
	Name string `json:"name"`
	// Blocks is the set of nodes in the graph.
	Blocks []PipelineNode `json:"blocks"`
	// Connections is the set of edges linking block output ports to input ports.
	Connections []Connection `json:"connections"`
}

Pipeline is an acyclic graph (DAG) of named blocks with typed connections. It is a serialisable description; calling Manager.StartFlow instantiates it into a running Flow.

func ParseDefinition

func ParseDefinition(id, name string, definition []byte) (*Pipeline, error)

ParseDefinition unmarshals a stored pipeline definition (blocks + connections JSON) into a Pipeline, setting ID and Name from the caller (the store keeps those outside the definition blob).

func (*Pipeline) MarshalJSON

func (p *Pipeline) MarshalJSON() ([]byte, error)

MarshalJSON serialises the Pipeline. It runs validation before encoding.

func (*Pipeline) Validate

func (p *Pipeline) Validate() error

Validate checks that the pipeline is a valid acyclic directed graph: all block names are unique, all connections reference existing blocks, the graph contains no cycles, and there is at least one source and one sink.

type PipelineNode

type PipelineNode struct {
	// Name is a unique identifier for this node within the pipeline.
	Name string `json:"name"`
	// BlockType identifies which block implementation to use.
	BlockType string `json:"block_type"`
	// Config holds block-specific parameters.
	Config map[string]any `json:"config,omitempty"`
}

PipelineNode describes one block within a pipeline.

type RegisterFunc

type RegisterFunc func(ctx context.Context, realm, deviceID string) error

RegisterFunc registers a first-seen virtual device through the pairing door (#84 auto_register). Implementations return ErrVirtualDeviceRegistered when the id is already taken; any other error is infrastructure.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry maps pipeline block_type strings to constructors.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty block-type registry.

func (*Registry) Has

func (r *Registry) Has(blockType string) bool

Has reports whether blockType has a registered constructor.

func (*Registry) Instantiate

func (r *Registry) Instantiate(p *Pipeline, deps Deps) ([]Block, error)

Instantiate turns a validated Pipeline description into an ordered block list suitable for Manager.StartFlow. Blocks are returned in topological order (sources first, sinks last). Linear graphs are the supported production shape; DAGs are flattened in topo order (fan-in/fan-out is not yet modelled in BlockGraph).

func (*Registry) Register

func (r *Registry) Register(blockType string, ctor Constructor)

Register associates blockType with ctor. Later Register calls for the same type replace the previous constructor.

func (*Registry) Types

func (r *Registry) Types() []string

Types returns the registered block_type names in sorted order.

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router is the stream-based message router. It accepts FlowMessages, hashes their Key to a lane, and processes them through the block graph. Messages with the same Key are always processed in submission order; different Keys may interleave across lanes.

func NewRouter

func NewRouter(graph *BlockGraph, cfg RouterConfig, reg prometheus.Registerer) *Router

NewRouter builds a router that feeds every submitted message through graph.

func (*Router) Drain

func (r *Router) Drain(ctx context.Context) error

Drain stops accepting new messages, lets lanes finish, and waits for all lane goroutines to exit.

func (*Router) Run

func (r *Router) Run(ctx context.Context)

Run starts the lane goroutines. Call it after NewRouter. ctx bounds background work; cancel to initiate drain.

func (*Router) Submit

func (r *Router) Submit(msg *Message, qos byte)

Submit routes msg to the lane determined by FNV-1a(msg.Key). Behaviour depends on qos and the configured overflow policies.

The read lock is held across the send, not just across the closed check. Dropping it in between left a window where Drain could retire the lanes between the check and the send; with OverflowBlock the sender can sit in that window for as long as the lane is full. Sends are what the lock is for, so it covers them.

type RouterConfig

type RouterConfig struct {
	// Lanes is the number of processing lanes (default DefaultLaneCount).
	Lanes int
	// LaneCapacity is the per-lane channel capacity (default
	// DefaultLaneCapacity).
	LaneCapacity int
	// QoS0Overflow is the policy when a QoS 0 message targets a full lane.
	QoS0Overflow OverflowPolicy
	// QoS1Overflow is the policy when a QoS ≥ 1 message targets a full lane.
	QoS1Overflow OverflowPolicy
	// Registerer receives Prometheus collectors; nil leaves them
	// unregistered.
	Registerer prometheus.Registerer
	// Logger receives router logs (default slog.Default()).
	Logger *slog.Logger
}

RouterConfig configures a Router.

type SinkFunc

type SinkFunc func(msg *Message) error

SinkFunc consumes a message for external output. Return value is ignored by the pipeline.

type Source

type Source interface {
	Block
	// Emit returns newly available messages. Implementations may block until
	// at least one message is ready or ctx is cancelled.
	Emit(ctx context.Context) ([]*Message, error)
}

Source is a Block that produces messages from an external system without an input message. The flow source pump calls Emit on every Source in the graph and submits the results into the router; BlockGraph.Run skips Source stages so a submitted message is not re-consumed by the producer.

type SourceFunc

type SourceFunc func() ([]*Message, error)

SourceFunc is a function that produces messages from external events. It receives nil and returns zero or more messages.

type Status

type Status uint8

Status enumerates the lifecycle states of a flow.

const (
	// FlowStatusCreating indicates the flow is being initialised.
	FlowStatusCreating Status = iota
	// FlowStatusRunning indicates the flow is accepting and processing messages.
	FlowStatusRunning
	// FlowStatusStopped indicates the flow has been gracefully shut down.
	FlowStatusStopped
	// FlowStatusFailed indicates the flow failed during initialisation.
	FlowStatusFailed
)

func (Status) String

func (s Status) String() string

String returns a human-readable label for s.

type Stopper

type Stopper interface {
	Stop()
}

Stopper is optionally implemented by Blocks that own resources (bus subscriptions, goroutines, file handles). Manager.StopFlow calls Stop on every Stopper after the source pump exits and the router drains.

type TransformFunc

type TransformFunc func(msg *Message) ([]*Message, error)

TransformFunc consumes one message and returns zero or more transformed messages.

type UserBlock

type UserBlock struct {
	Name         string
	BlockType    string // producer | consumer | producer_consumer; stored, not enforced here
	Source       []byte // a pipeline definition body: {"blocks":[…],"connections":[…]}
	ConfigSchema json.RawMessage
}

UserBlock is one realm-level composite definition (#85).

Directories

Path Synopsis
Package blocks registers the built-in Flow block catalog used to instantiate stored pipelines (milestone v2.0: block factory).
Package blocks registers the built-in Flow block catalog used to instantiate stored pipelines (milestone v2.0: block factory).
astartesource
Package astartesource implements the AstarteSource Flow block (issue #27, astarte_flow parity): a Source that subscribes to Astrate's existing live event bus (internal/engine/stream) and converts device events into FlowMessages, connecting device ingestion to operator-defined pipelines.
Package astartesource implements the AstarteSource Flow block (issue #27, astarte_flow parity): a Source that subscribes to Astrate's existing live event bus (internal/engine/stream) and converts device events into FlowMessages, connecting device ingestion to operator-defined pipelines.
container
Package container implements the Flow "container" block (Design B / #43 PoC).
Package container implements the Flow "container" block (Design B / #43 PoC).
virtualdevicepool
Package virtualdevicepool implements the virtual_device_pool block (issue #84): it publishes pipeline messages as registered virtual devices through the engine ingest path — storage rows without MQTT.
Package virtualdevicepool implements the virtual_device_pool block (issue #84): it publishes pipeline messages as registered virtual devices through the engine ingest path — storage rows without MQTT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL