engine

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package engine is the heart of Astrate (docs/ROADMAP.md §7): it consumes every accepted device PUBLISH from the broker intake, validates it against the realm's compiled interface schemas (docs/DESIGN.md §2.6), and persists it through per-shard micro-batches with strict per-device ordering and ack-after-commit semantics (docs/DESIGN.md §1.4, §5.3).

M6a (this milestone slice) implements the pipeline and the data path: compiled-interface and device caches, the sharded router, topic classification, the §2.6 validation pipeline, and batched persistence. M6b adds the control-channel handlers, server-owned data publishing, triggers, and the live stream — they attach to the seams the Engine struct already exposes (handler fields and the afterCommit hook).

Index

Constants

View Source
const (
	// DefaultShards is the default shard count.
	DefaultShards = 16
	// DefaultShardQueue is the default per-shard channel capacity.
	DefaultShardQueue = 4096
	// DefaultBatchMaxRows flushes a shard batch when it reaches this size.
	DefaultBatchMaxRows = 64
	// DefaultBatchMaxWait flushes a non-empty shard batch after this delay.
	DefaultBatchMaxWait = 50 * time.Millisecond
)

Configuration defaults (docs/DESIGN.md §1.4).

Variables

View Source
var (
	// ErrRealmUnknown: the realm is not in the schema snapshot.
	ErrRealmUnknown = errors.New("engine: realm unknown")
	// ErrInterfaceNotFound: the realm has no installed interface by that name.
	ErrInterfaceNotFound = errors.New("engine: interface not installed")
	// ErrNotServerOwned: the interface is device-owned (docs/DESIGN.md §2.6
	// step 3: AppEngine publishes only on ownership: server).
	ErrNotServerOwned = errors.New("engine: interface is not server-owned")
	// ErrNotDeviceOwned: the interface is server-owned (the mirrored gate of
	// PublishDeviceValue, issue #84: device-owned ingest refuses them).
	ErrNotDeviceOwned = errors.New("engine: interface is not device-owned")
	// ErrPathNotFound: the path resolves no endpoint mapping.
	ErrPathNotFound = errors.New("engine: path matches no endpoint")
	// ErrNotAProperty: a property operation on a datastream interface.
	ErrNotAProperty = errors.New("engine: interface is not a properties interface")
	// ErrUnsetNotAllowed: unset on a mapping without allow_unset.
	ErrUnsetNotAllowed = errors.New("engine: mapping does not allow unset")
)

Sentinel errors of the server-owned publish path (docs/ROADMAP.md §7.2 file 6.9). The AppEngine layer (M7) maps them onto upstream-shaped HTTP statuses; payload validation failures surface as *payload.RejectError.

Functions

This section is empty.

Types

type BrokerPort

type BrokerPort interface {
	// Publish sends a server-side message (docs/ROADMAP.md §6 file 5.7):
	// retain for properties, per-message expiry for datastreams.
	Publish(topic string, payload []byte, qos byte, retain bool, expiry time.Duration) error
	// RefreshIntrospection reloads a connected device's introspection-derived
	// ACL state after the engine persists a new introspection
	// (docs/ROADMAP.md §7.2 file 6.7).
	RefreshIntrospection(ctx context.Context, realm string, id deviceid.ID) error
}

BrokerPort is the engine's broker-side port (hexagonal-lite, docs/DESIGN.md §1.3): server→device publishing and the ACL-relevant introspection refresh. Defined on the consumer side so tests substitute fakes; AdaptBroker wraps the real broker.

func AdaptBroker

func AdaptBroker(b *broker.Broker) BrokerPort

AdaptBroker wraps the embedded broker as the engine's BrokerPort.

type Config

type Config struct {
	// Shards is the number of ordered pipeline shards (default
	// DefaultShards). Messages of one device always land on the same shard.
	Shards int
	// ShardQueue is the per-shard channel capacity (default
	// DefaultShardQueue). A full shard blocks QoS >= 1 submits (deferred-ack
	// backpressure) and drops QoS 0 messages with a metric (§1.4).
	ShardQueue int
	// BatchMaxRows is the micro-batch row cap (default DefaultBatchMaxRows).
	BatchMaxRows int
	// BatchMaxWait is the micro-batch time cap (default DefaultBatchMaxWait).
	BatchMaxWait time.Duration
	// MaxPayloadBytes caps accepted data payload size for both formats
	// (default payload.DefaultMaxSize).
	MaxPayloadBytes int
	// Forwarder receives trigger actions that are not HTTP webhooks
	// (upstream's AMQP actions, docs/DESIGN.md §1.1). nil is the default and
	// means those actions are logged and counted "skipped", which is the
	// designed behaviour rather than a gap.
	Forwarder triggers.Forwarder
	// Registerer receives the engine's Prometheus collectors; nil leaves
	// them unregistered (the collectors still work, which tests rely on).
	Registerer prometheus.Registerer
	// Logger receives engine logs (default slog.Default()).
	Logger *slog.Logger
}

Config carries the engine's operational knobs (TOML wiring lands in M8).

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the sharded ingestion pipeline (docs/ROADMAP.md §7.1 file 6.2). It implements broker.Intake and broker.LifecycleSink. M6a builds the data path; the M6b handlers (introspection, control, triggers, stream) attach to the seam fields below without touching the pipeline.

func New

func New(st Store, bp BrokerPort, cfg Config) (*Engine, error)

New assembles the full engine (docs/ROADMAP.md §7.2 file 6.14): the M6a pipeline plus the M6b control-channel handlers, trigger evaluation, and the live fan-out bus. bp may be nil at construction time — the broker needs the engine as its intake, so M8 wires them in two steps — but must be attached (AttachBroker) before Start.

func (*Engine) AttachBroker

func (e *Engine) AttachBroker(bp BrokerPort)

AttachBroker binds the broker port; it must run before Start (M8 wiring: engine → broker.New(intake=engine) → AttachBroker → Start).

func (*Engine) Bus

func (e *Engine) Bus() *stream.Bus

Bus exposes the live fan-out bus (the M7b stream socket subscribes through it).

func (*Engine) Drain

func (e *Engine) Drain(ctx context.Context) error

Drain gracefully stops the engine (docs/DESIGN.md §5.3): intake is refused, shards drain and flush their final batches, asynchronous control sends finish, the trigger executor drains its queue, and the live bus closes — all bounded by ctx. Call broker.Close first so no submitter blocks indefinitely.

func (*Engine) HandleDeviceDeletionFinished added in v0.2.0

func (e *Engine) HandleDeviceDeletionFinished(realmName string, hwID string, at time.Time)

HandleDeviceDeletionFinished is called by the realm service immediately after a synchronous device delete completes (success or failure after the delete was attempted).

func (*Engine) HandleDeviceDeletionStarted added in v0.2.0

func (e *Engine) HandleDeviceDeletionStarted(realmName string, hwID string, at time.Time)

HandleDeviceDeletionStarted is called by the realm service immediately before a synchronous device delete. It fires device_deletion_started triggers and publishes on the live bus (issue #21; upstream wraps async deletion with the same pair of events).

func (*Engine) HandleDeviceRegistered added in v0.2.0

func (e *Engine) HandleDeviceRegistered(realmName string, hwID string, at time.Time)

HandleDeviceRegistered is called by the pairing service after a successful device registration. It fires device_registered triggers and publishes the event on the live bus — the same path device_connected already uses.

func (*Engine) OnLifecycleEvent

func (e *Engine) OnLifecycleEvent(ev broker.LifecycleEvent)

OnLifecycleEvent implements broker.LifecycleSink: device disconnects evict the per-device cache entry (docs/ROADMAP.md §7.1 file 6.1); the M6b seam then observes the event (device connect/disconnect triggers). It never blocks (broker.LifecycleSink contract).

func (*Engine) PublishDeviceValue added in v0.2.0

func (e *Engine) PublishDeviceValue(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string, value json.RawMessage, ts *time.Time) error

PublishDeviceValue validates and persists one device-owned value without delivering it (issue #84): virtual-device ingest lands storage rows exactly like a real device's data would, but produces no MQTT traffic — the device speaks to the engine directly, so there is no upstream hop to relay onto the broker. Validation and persistence mirror PublishServerValue with the ownership gate flipped to device.

func (*Engine) PublishServerValue

func (e *Engine) PublishServerValue(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string, value json.RawMessage, ts *time.Time) error

PublishServerValue validates, persists, and delivers one server-owned value (docs/ROADMAP.md §7.2 file 6.9): AppEngine PUT/POST bodies land here. value is the raw JSON value (the unwrapped "data" field); it is validated against the mapping exactly like an inbound JSON-profile payload (docs/DESIGN.md §2.6 steps 4–5). ts is the optional explicit timestamp; reception time applies otherwise.

Persistence happens before delivery: a property upsert or a datastream insert, then a broker publish on the device's data topic with the mapping's QoS, retain for properties, per-message expiry for datastreams, and the wire format chosen by the device's payload_format_hint (docs/DESIGN.md §3.4, §3.5.4).

func (*Engine) RefreshInterfaces

func (e *Engine) RefreshInterfaces(ctx context.Context, realmID int16) error

RefreshInterfaces is the in-process cache-invalidation callback (docs/DESIGN.md §2.6): realm management (M7) calls it after interface CRUD, complementing the LISTEN/NOTIFY path.

func (*Engine) RefreshTriggers

func (e *Engine) RefreshTriggers(ctx context.Context, realmID int16) error

RefreshTriggers rebuilds a realm's compiled snapshot — triggers ride in the same snapshot as interfaces, so this is the in-process invalidation callback for M7 trigger CRUD (docs/DESIGN.md §2.6).

func (*Engine) Start

func (e *Engine) Start(ctx context.Context) error

Start loads the schema snapshot, subscribes to interface-change notifications, and launches the shard goroutines. ctx bounds the engine's background work and must outlive Drain.

func (*Engine) Submit

func (e *Engine) Submit(m broker.InboundMessage)

Submit implements broker.Intake (docs/DESIGN.md §1.4): route to the device's shard; block when the shard is full and the message is QoS >= 1 (the broker withholds the device's PUBACK, which is the backpressure); drop QoS 0 messages on a full shard with a metric.

func (*Engine) UnsetServerProperty

func (e *Engine) UnsetServerProperty(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string) error

UnsetServerProperty deletes a server-owned property (AppEngine DELETE, docs/ROADMAP.md §8.2 file 7.7): the row is removed, an empty retained payload clears the topic and signals the unset to a connected device, and the `consumer/properties` purge message follows so offline-window state converges (docs/DESIGN.md §3.4). Unsetting an absent property is a no-op, not an error.

type OpKind

type OpKind uint8

OpKind discriminates PersistOp values.

const (
	// OpIndividual is one individual-datastream insert.
	OpIndividual OpKind = iota + 1
	// OpObject is one object-datastream insert.
	OpObject
	// OpPropertySet is a property upsert.
	OpPropertySet
	// OpPropertyUnset is a property delete (empty payload, allow_unset).
	OpPropertyUnset
)

OpKind values.

func (OpKind) String

func (k OpKind) String() string

String returns the stable snake_case metrics label.

type PersistOp

type PersistOp struct {
	// Kind discriminates the persistence action.
	Kind OpKind
	// Realm and RealmID identify the tenant.
	Realm   string
	RealmID int16
	// DeviceID is the publishing device.
	DeviceID deviceid.ID
	// Interface is the compiled interface the message validated against.
	Interface *interfaceschema.CompiledInterface
	// Mapping is the matched endpoint mapping; nil for object aggregation.
	Mapping *interfaceschema.CompiledMapping
	// Path is the concrete data path ("" only for flat object aggregation).
	Path string
	// Value is the decoded payload value (payload.Value closed set;
	// map[string]payload.Value for objects; nil for property unset).
	Value payload.Value
	// Format is the wire format the payload arrived in.
	Format payload.Format
	// TS is the effective sample timestamp: the explicit `t` when the
	// mapping declares explicit_timestamp, broker reception time otherwise.
	TS time.Time
	// ReceptionTS is the broker reception timestamp.
	ReceptionTS time.Time

	// Previous-value snapshot for the change-derived trigger conditions,
	// captured at accept time by trackPrevious when a trigger is actually
	// watching this endpoint: Prev holds the canonical §2.3 JSON of the
	// previous value, PrevExists reports whether there was one, Tracked
	// marks that a snapshot was taken at all (untracked ops skip change
	// events but still fire incoming_data).
	Tracked    bool
	PrevExists bool
	Prev       []byte
	// contains filtered or unexported fields
}

PersistOp is one validated operation emitted by the §2.6 pipeline (docs/ROADMAP.md §7.1 file 6.4), carried through the shard micro-batch and handed to the afterCommit observers (triggers and live fan-out, M6b).

type Store

type Store interface {
	// ListRealms feeds the realm name/ID resolution in the schema cache.
	ListRealms(ctx context.Context) ([]store.Realm, error)
	// LoadRealmInterfaces returns every installed interface of a realm for
	// schema-snapshot rebuilds.
	LoadRealmInterfaces(ctx context.Context, realmID int16) ([]*store.StoredInterface, error)
	// Listen subscribes to a NOTIFY channel (cache invalidation,
	// docs/DESIGN.md §2.6).
	Listen(ctx context.Context, channel string) (<-chan store.Notification, error)
	// GetDevice loads a device's introspection and payload-format hint.
	GetDevice(ctx context.Context, realmID int16, id deviceid.ID) (*store.Device, error)
	// SetPayloadFormatHint persists the sticky payload-format flip
	// (docs/DESIGN.md §3.5.4).
	SetPayloadFormatHint(ctx context.Context, realmID int16, id deviceid.ID, hint string) error
	// AppendDatastreams commits one micro-batch in a single transaction.
	AppendDatastreams(ctx context.Context, batch store.DatastreamBatch) error
	// UpsertProperty applies a property set (last-value-wins).
	UpsertProperty(ctx context.Context, p store.Property) error
	// UnsetProperty applies a property unset (row delete).
	UnsetProperty(ctx context.Context, realmID int16, deviceID deviceid.ID, interfaceID int64, path string) (bool, error)
	// GetProperty reads one stored property — the previous-value lookup for
	// property change triggers.
	GetProperty(ctx context.Context, realmID int16, deviceID deviceid.ID, interfaceID int64, path string) (*store.Property, error)
	// LatestIndividual reads the newest sample of one datastream series —
	// the previous-value lookup for datastream change triggers.
	LatestIndividual(ctx context.Context, realmID int16, deviceID deviceid.ID, interfaceID int64, path string) (*store.IndividualRow, error)
	// UpdateIntrospection replaces a device's introspection, returning the
	// (name, major) pairs that were dropped (docs/ROADMAP.md §7.2 file 6.7).
	UpdateIntrospection(ctx context.Context, realmID int16, id deviceid.ID, intro map[string]store.InterfaceVersion) (map[string]store.InterfaceVersion, error)
	// PurgeDeviceOwnedExcept implements the `producer/properties` resync
	// (docs/DESIGN.md §3.3): device-owned properties not in keep are deleted.
	PurgeDeviceOwnedExcept(ctx context.Context, realmID int16, deviceID deviceid.ID, keep []store.PropertyRef) (int64, error)
	// ListServerOwnedProperties returns the device's server-owned properties
	// (emptyCache resend + `consumer/properties` payload, docs/DESIGN.md §3.4).
	ListServerOwnedProperties(ctx context.Context, realmID int16, deviceID deviceid.ID) ([]store.Property, error)
	// ListTriggers returns a realm's installed triggers for the compiled
	// trigger cache (docs/ROADMAP.md §7.2 file 6.10).
	ListTriggers(ctx context.Context, realmID int16) ([]store.Trigger, error)
	// ListTriggerPolicies returns every delivery policy of a realm so the
	// snapshot can attach compiled policies to triggers at load time.
	ListTriggerPolicies(ctx context.Context, realmID int16) ([]store.TriggerPolicy, error)
}

Store is the engine's persistence port (hexagonal-lite, docs/DESIGN.md §1.3): the subset of *store.Store the M6a pipeline needs. The interface is defined here, on the consumer side, so tests substitute fakes and M8 wiring is a plain assignment.

Directories

Path Synopsis
Package forward provides Forwarder implementations that deliver non-HTTP trigger actions to downstream systems.
Package forward provides Forwarder implementations that deliver non-HTTP trigger actions to downstream systems.
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters.
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters.
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1).
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL