Documentation
¶
Overview ¶
Package flow implements stream-based message routing through a block graph, mirroring Astarte Flow's core concepts: messages belong to streams (identified by key), streams are processed in order within a lane, and different streams may interleave across lanes.
Index ¶
- Constants
- Variables
- func ExpandComposites(def []byte, resolve func(name string) (*UserBlock, error)) ([]byte, error)
- func InstanceID(realm, flowName string) string
- func PipelineID(realm, name string) string
- func SubstituteConfig(definition []byte, config map[string]any) ([]byte, error)
- type Block
- type BlockGraph
- type Config
- type Connection
- type Constructor
- type DataType
- type Deps
- type Flow
- type IngestFunc
- type Manager
- func (m *Manager) GetFlowStatus(pipelineID string) (Status, error)
- func (m *Manager) ListFlows() []*Flow
- func (m *Manager) Shutdown(ctx context.Context) error
- func (m *Manager) StartFlow(ctx context.Context, cfg Config) (*Flow, error)
- func (m *Manager) StopFlow(ctx context.Context, pipelineID string) error
- func (m *Manager) UnregisterFlow(instanceID string)
- type Message
- type OverflowPolicy
- type Pipeline
- type PipelineNode
- type RegisterFunc
- type Registry
- type Router
- type RouterConfig
- type SinkFunc
- type Source
- type SourceFunc
- type Status
- type Stopper
- type TransformFunc
- type UserBlock
Constants ¶
const DefaultLaneCapacity = 256
DefaultLaneCapacity is the default per-lane channel capacity.
const DefaultLaneCount = 16
DefaultLaneCount is the default number of processing lanes.
const WireSchema = "astarte_flow/message/v0.1"
WireSchema is the JSON schema identifier used on the wire.
Variables ¶
var ( // ErrFlowExists is returned by StartFlow when a flow with the same // instance key is already registered. ErrFlowExists = errors.New("flow: already running") // ErrFlowNotFound is returned when a flow ID does not match any // registered flow. ErrFlowNotFound = errors.New("flow: not found") )
var ErrUnknownBlockType = errors.New("flow: unknown block type")
ErrUnknownBlockType is returned when Instantiate sees a block_type with no registered constructor.
var ErrVirtualDeviceRegistered = errors.New("flow: virtual device already registered")
ErrVirtualDeviceRegistered reports that a first-seen virtual device could not be auto-registered because its id is already registered and confirmed: upstream answers 422 already_registered and drops the message, and astrate mirrors that (the message is dropped, never an error).
Functions ¶
func ExpandComposites ¶
ExpandComposites inlines every user-defined block found in def (issue #85): a node whose block_type resolves through resolve is replaced by its stored sub-chain, spliced in place — incoming edges attach to the sub-chain's first node and outgoing edges leave from its last node (stable topological order). Nested composites are expanded recursively and their block names nest as "outer.inner.block". resolve returns (nil, nil) for built-in types, which pass through unchanged. Definitions are handled as raw node/connection slices; Pipeline.Validate is never run on inner bodies.
func InstanceID ¶
InstanceID builds the Manager map key for a named flow instance (realm + "/" + flowName). Different flows may share one pipeline name.
func PipelineID ¶
PipelineID is a legacy alias for InstanceID. Prefer InstanceID; the map key is the flow instance name, not the pipeline name.
func SubstituteConfig ¶
SubstituteConfig walks a pipeline definition JSON and replaces ${config.key} placeholders inside string values only. Missing keys or non-stringable config values fail loudly. Non-string JSON leaves are left unchanged.
Types ¶
type Block ¶
type Block interface {
// Process handles one message. A source receives msg == nil and may return
// zero or more messages. A transform receives exactly one non-nil message
// and may return zero or more. A sink returns nil.
Process(msg *Message) ([]*Message, error)
// Name returns a human-readable label for metrics and logging.
Name() string
}
Block is a computation unit in a flow graph. Implementations must be safe for concurrent use by a single lane goroutine (one goroutine calls Process sequentially per message); external concurrency safety is the Router's job.
Three roles exist:
- Source: produces messages from external events (see the Source interface).
- Transform: consumes one message and emits zero or more transformed messages.
- Sink: consumes messages for external output (return value is ignored).
func NewSinkBlock ¶
NewSinkBlock wraps fn as a Block with the given name.
func NewSourceBlock ¶
func NewSourceBlock(name string, fn SourceFunc) Block
NewSourceBlock wraps fn as a Source Block with the given name.
func NewTransformBlock ¶
func NewTransformBlock(name string, fn TransformFunc) Block
NewTransformBlock wraps fn as a Block with the given name.
type BlockGraph ¶
type BlockGraph struct {
// contains filtered or unexported fields
}
BlockGraph is a linear chain of blocks: source → transform₁ → … → sink. The graph is immutable after construction; calling Run feeds one message through every non-Source stage. Source blocks are driven by the flow source pump (see Manager.StartFlow), not by Run.
func NewBlockGraph ¶
func NewBlockGraph(blocks ...Block) (*BlockGraph, error)
NewBlockGraph validates that the graph has at least one block and that the last block is a sink (it may return nil messages). The first block is typically the source. Returns an error if the chain is empty or nil blocks are present.
func (*BlockGraph) Blocks ¶
func (g *BlockGraph) Blocks() []Block
Blocks returns the graph's blocks in order. The slice must not be mutated.
func (*BlockGraph) Run ¶
func (g *BlockGraph) Run(msg *Message) ([]*Message, error)
Run feeds one message through every non-Source stage. Source stages are skipped: they are polled by the source pump and their outputs are submitted into the router, which calls Run. Returns the messages produced by the final stage — typically nil for a sink.
func (*BlockGraph) Sources ¶
func (g *BlockGraph) Sources() []Source
Sources returns every block that implements Source, in graph order.
type Config ¶
type Config struct {
// PipelineID is the Manager map key for this instance (realm/flowName).
// Historical field name; value is InstanceID, not the pipeline recipe name.
PipelineID string
// Blocks is the ordered list of blocks forming the processing graph.
Blocks []Block
// RouterCfg is applied to the underlying Router.
RouterCfg RouterConfig
// Registerer receives Prometheus collectors; nil leaves them
// unregistered.
Registerer prometheus.Registerer
}
Config holds the parameters needed to instantiate a running flow.
type Connection ¶
type Connection struct {
// From is the name of the source block.
From string `json:"from"`
// FromPort is the output port on the source block (empty means default).
FromPort string `json:"from_port,omitempty"`
// To is the name of the target block.
To string `json:"to"`
// ToPort is the input port on the target block (empty means default).
ToPort string `json:"to_port,omitempty"`
}
Connection describes a typed edge between two blocks in a pipeline.
type Constructor ¶
Constructor builds one Block from a pipeline node. name is the pipeline node name and should be returned by Block.Name() for metrics and logging.
type DataType ¶
type DataType uint8
DataType enumerates the wire-level value types Astarte Flow supports.
const ( TypeInteger DataType = iota TypeReal TypeBoolean TypeDatetime TypeBinary TypeString // TypeMap indicates the message carries a map payload. Per-field types // and subtypes are in FieldTypes and FieldSubtypes. TypeMap )
DataType values, in wire-encoding order. TypeMap is the only aggregate; its per-field types live in FieldTypes and FieldSubtypes.
type Deps ¶
type Deps struct {
// Bus is the live event bus (required by astarte_source).
Bus *stream.Bus
// Realm is the tenant the pipeline runs for; sources may default to it
// when their config omits realm.
Realm string
// FlowName is the durable/named flow instance name (optional). Used by
// container labels and similar operator metadata.
FlowName string
// NotifyFatal, when set, is called by blocks that die at runtime (e.g. a
// container block whose container exits unexpectedly) so the service layer
// can fail the flow and schedule a restart. Must be safe to call from any
// goroutine; blocks must not call it after their Stopper.Stop() ran.
NotifyFatal func(block string, cause error)
// Ingest, when set, lets a block land data as if a registered device had
// produced it (virtual_device_pool, issue #84): full validation plus a
// datastream/property row, no MQTT delivery. Required by
// virtual_device_pool; every other block ignores it.
Ingest IngestFunc
// Register, when set, lets a block register a first-seen virtual device
// through the pairing door (virtual_device_pool auto_register, #84).
// Required by virtual_device_pool with auto_register; every other block
// ignores it.
Register RegisterFunc
}
Deps holds process-level dependencies block constructors may need. Zero fields are valid when a pipeline uses only blocks that do not need them.
type Flow ¶
type Flow struct {
// contains filtered or unexported fields
}
Flow is a running instance of a pipeline. It owns a Router that processes messages through a BlockGraph, a source pump that feeds Source blocks into the router, and exposes status information.
func (*Flow) PipelineID ¶
PipelineID returns the pipeline this flow was created from.
type IngestFunc ¶
type IngestFunc func(ctx context.Context, realm, deviceID, ifaceName, path string, payload json.RawMessage, ts *time.Time) error
IngestFunc lands data as if a registered device had produced it: full validation plus storage, no MQTT delivery (virtual_device_pool, #84).
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager manages the lifecycle of running flows. It is safe for concurrent use.
func (*Manager) GetFlowStatus ¶
GetFlowStatus returns the current status of the flow identified by pipelineID.
func (*Manager) Shutdown ¶
Shutdown drains all running flows. It is intended for orderly process exit; each flow gets the same context deadline.
func (*Manager) StartFlow ¶
StartFlow instantiates a pipeline into a running Flow. The block graph is built before the flow is registered so construction failures leave no map entry (durable layer records status=failed separately). On success the status transitions to running.
func (*Manager) StopFlow ¶
StopFlow gracefully shuts down the flow identified by pipelineID. It stops the source pump, drains in-flight messages, calls Stop on every Stopper block, and transitions the status to stopped.
func (*Manager) UnregisterFlow ¶
UnregisterFlow removes an instance key from the manager map. Call after StopFlow when deleting a durable row so the name can be reused.
type Message ¶
type Message struct {
// Key identifies the stream this message belongs to. It must be non-empty.
Key string
// Metadata is an optional string→string map carried alongside the payload.
Metadata map[string]string
// Type is the base data type of the payload.
Type DataType
// Subtype is an optional MIME hint (meaningful when Type is TypeBinary).
Subtype string
// Timestamp is the event-time in microseconds since Unix epoch.
Timestamp int64
// Data is the payload; its concrete type must match Type (int64 for
// TypeInteger, float64 for TypeReal, bool for TypeBoolean, time.Time for
// TypeDatetime, []byte for TypeBinary, string for TypeString, map[string]any
// for TypeMap).
Data any
// FieldTypes holds per-field types when Type is TypeMap.
FieldTypes map[string]DataType
// FieldSubtypes holds per-field subtypes when Type is TypeMap.
FieldSubtypes map[string]string
}
Message is one unit of data flowing through a block graph. Every message carries a Key that identifies its stream; messages with the same key are processed in submission order by the same lane (consistent hashing).
func (*Message) MarshalJSON ¶
MarshalJSON serialises the Message to the upstream JSON wire format.
func (*Message) UnmarshalJSON ¶
UnmarshalJSON deserialises a Message from the upstream JSON wire format.
type OverflowPolicy ¶
type OverflowPolicy uint8
OverflowPolicy controls what happens when a lane's channel is full.
const ( // OverflowBlock blocks the caller until space is available (QoS ≥ 1). OverflowBlock OverflowPolicy = iota // OverflowDrop discards the message without blocking (QoS 0). OverflowDrop )
type Pipeline ¶
type Pipeline struct {
// ID is a unique identifier for this pipeline.
ID string `json:"id"`
// Name is a human-readable label.
Name string `json:"name"`
// Blocks is the set of nodes in the graph.
Blocks []PipelineNode `json:"blocks"`
// Connections is the set of edges linking block output ports to input ports.
Connections []Connection `json:"connections"`
}
Pipeline is an acyclic graph (DAG) of named blocks with typed connections. It is a serialisable description; calling Manager.StartFlow instantiates it into a running Flow.
func ParseDefinition ¶
ParseDefinition unmarshals a stored pipeline definition (blocks + connections JSON) into a Pipeline, setting ID and Name from the caller (the store keeps those outside the definition blob).
func (*Pipeline) MarshalJSON ¶
MarshalJSON serialises the Pipeline. It runs validation before encoding.
type PipelineNode ¶
type PipelineNode struct {
// Name is a unique identifier for this node within the pipeline.
Name string `json:"name"`
// BlockType identifies which block implementation to use.
BlockType string `json:"block_type"`
// Config holds block-specific parameters.
Config map[string]any `json:"config,omitempty"`
}
PipelineNode describes one block within a pipeline.
type RegisterFunc ¶
RegisterFunc registers a first-seen virtual device through the pairing door (#84 auto_register). Implementations return ErrVirtualDeviceRegistered when the id is already taken; any other error is infrastructure.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry maps pipeline block_type strings to constructors.
func (*Registry) Instantiate ¶
Instantiate turns a validated Pipeline description into an ordered block list suitable for Manager.StartFlow. Blocks are returned in topological order (sources first, sinks last). Linear graphs are the supported production shape; DAGs are flattened in topo order (fan-in/fan-out is not yet modelled in BlockGraph).
func (*Registry) Register ¶
func (r *Registry) Register(blockType string, ctor Constructor)
Register associates blockType with ctor. Later Register calls for the same type replace the previous constructor.
type Router ¶
type Router struct {
// contains filtered or unexported fields
}
Router is the stream-based message router. It accepts FlowMessages, hashes their Key to a lane, and processes them through the block graph. Messages with the same Key are always processed in submission order; different Keys may interleave across lanes.
func NewRouter ¶
func NewRouter(graph *BlockGraph, cfg RouterConfig, reg prometheus.Registerer) *Router
NewRouter builds a router that feeds every submitted message through graph.
func (*Router) Drain ¶
Drain stops accepting new messages, lets lanes finish, and waits for all lane goroutines to exit.
func (*Router) Run ¶
Run starts the lane goroutines. Call it after NewRouter. ctx bounds background work; cancel to initiate drain.
func (*Router) Submit ¶
Submit routes msg to the lane determined by FNV-1a(msg.Key). Behaviour depends on qos and the configured overflow policies.
The read lock is held across the send, not just across the closed check. Dropping it in between left a window where Drain could retire the lanes between the check and the send; with OverflowBlock the sender can sit in that window for as long as the lane is full. Sends are what the lock is for, so it covers them.
type RouterConfig ¶
type RouterConfig struct {
// Lanes is the number of processing lanes (default DefaultLaneCount).
Lanes int
// LaneCapacity is the per-lane channel capacity (default
// DefaultLaneCapacity).
LaneCapacity int
// QoS0Overflow is the policy when a QoS 0 message targets a full lane.
QoS0Overflow OverflowPolicy
// QoS1Overflow is the policy when a QoS ≥ 1 message targets a full lane.
QoS1Overflow OverflowPolicy
// Registerer receives Prometheus collectors; nil leaves them
// unregistered.
Registerer prometheus.Registerer
// Logger receives router logs (default slog.Default()).
Logger *slog.Logger
}
RouterConfig configures a Router.
type SinkFunc ¶
SinkFunc consumes a message for external output. Return value is ignored by the pipeline.
type Source ¶
type Source interface {
Block
// Emit returns newly available messages. Implementations may block until
// at least one message is ready or ctx is cancelled.
Emit(ctx context.Context) ([]*Message, error)
}
Source is a Block that produces messages from an external system without an input message. The flow source pump calls Emit on every Source in the graph and submits the results into the router; BlockGraph.Run skips Source stages so a submitted message is not re-consumed by the producer.
type SourceFunc ¶
SourceFunc is a function that produces messages from external events. It receives nil and returns zero or more messages.
type Status ¶
type Status uint8
Status enumerates the lifecycle states of a flow.
const ( // FlowStatusCreating indicates the flow is being initialised. FlowStatusCreating Status = iota // FlowStatusRunning indicates the flow is accepting and processing messages. FlowStatusRunning // FlowStatusStopped indicates the flow has been gracefully shut down. FlowStatusStopped // FlowStatusFailed indicates the flow failed during initialisation. FlowStatusFailed )
type Stopper ¶
type Stopper interface {
Stop()
}
Stopper is optionally implemented by Blocks that own resources (bus subscriptions, goroutines, file handles). Manager.StopFlow calls Stop on every Stopper after the source pump exits and the router drains.
type TransformFunc ¶
TransformFunc consumes one message and returns zero or more transformed messages.
type UserBlock ¶
type UserBlock struct {
Name string
BlockType string // producer | consumer | producer_consumer; stored, not enforced here
Source []byte // a pipeline definition body: {"blocks":[…],"connections":[…]}
ConfigSchema json.RawMessage
}
UserBlock is one realm-level composite definition (#85).
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package blocks registers the built-in Flow block catalog used to instantiate stored pipelines (milestone v2.0: block factory).
|
Package blocks registers the built-in Flow block catalog used to instantiate stored pipelines (milestone v2.0: block factory). |
|
astartesource
Package astartesource implements the AstarteSource Flow block (issue #27, astarte_flow parity): a Source that subscribes to Astrate's existing live event bus (internal/engine/stream) and converts device events into FlowMessages, connecting device ingestion to operator-defined pipelines.
|
Package astartesource implements the AstarteSource Flow block (issue #27, astarte_flow parity): a Source that subscribes to Astrate's existing live event bus (internal/engine/stream) and converts device events into FlowMessages, connecting device ingestion to operator-defined pipelines. |
|
container
Package container implements the Flow "container" block (Design B / #43 PoC).
|
Package container implements the Flow "container" block (Design B / #43 PoC). |
|
virtualdevicepool
Package virtualdevicepool implements the virtual_device_pool block (issue #84): it publishes pipeline messages as registered virtual devices through the engine ingest path — storage rows without MQTT.
|
Package virtualdevicepool implements the virtual_device_pool block (issue #84): it publishes pipeline messages as registered virtual devices through the engine ingest path — storage rows without MQTT. |