Documentation
¶
Overview ¶
Package engine is the heart of Astrate (docs/ROADMAP.md §7): it consumes every accepted device PUBLISH from the broker intake, validates it against the realm's compiled interface schemas (docs/DESIGN.md §2.6), and persists it through per-shard micro-batches with strict per-device ordering and ack-after-commit semantics (docs/DESIGN.md §1.4, §5.3).
M6a (this milestone slice) implements the pipeline and the data path: compiled-interface and device caches, the sharded router, topic classification, the §2.6 validation pipeline, and batched persistence. M6b adds the control-channel handlers, server-owned data publishing, triggers, and the live stream — they attach to the seams the Engine struct already exposes (handler fields and the afterCommit hook).
Index ¶
- Constants
- Variables
- type BrokerPort
- type Config
- type Engine
- func (e *Engine) AttachBroker(bp BrokerPort)
- func (e *Engine) Bus() *stream.Bus
- func (e *Engine) Drain(ctx context.Context) error
- func (e *Engine) OnLifecycleEvent(ev broker.LifecycleEvent)
- func (e *Engine) PublishServerValue(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string, ...) error
- func (e *Engine) RefreshInterfaces(ctx context.Context, realmID int16) error
- func (e *Engine) RefreshTriggers(ctx context.Context, realmID int16) error
- func (e *Engine) Start(ctx context.Context) error
- func (e *Engine) Submit(m broker.InboundMessage)
- func (e *Engine) UnsetServerProperty(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string) error
- type OpKind
- type PersistOp
- type Store
Constants ¶
const ( // DefaultShards is the default shard count. DefaultShards = 16 // DefaultShardQueue is the default per-shard channel capacity. DefaultShardQueue = 4096 // DefaultBatchMaxRows flushes a shard batch when it reaches this size. DefaultBatchMaxRows = 64 // DefaultBatchMaxWait flushes a non-empty shard batch after this delay. DefaultBatchMaxWait = 50 * time.Millisecond )
Configuration defaults (docs/DESIGN.md §1.4).
Variables ¶
var ( // ErrRealmUnknown: the realm is not in the schema snapshot. ErrRealmUnknown = errors.New("engine: realm unknown") // ErrInterfaceNotFound: the realm has no installed interface by that name. ErrInterfaceNotFound = errors.New("engine: interface not installed") // ErrNotServerOwned: the interface is device-owned (docs/DESIGN.md §2.6 // step 3: AppEngine publishes only on ownership: server). ErrNotServerOwned = errors.New("engine: interface is not server-owned") // ErrPathNotFound: the path resolves no endpoint mapping. ErrPathNotFound = errors.New("engine: path matches no endpoint") // ErrNotAProperty: a property operation on a datastream interface. ErrNotAProperty = errors.New("engine: interface is not a properties interface") // ErrUnsetNotAllowed: unset on a mapping without allow_unset. ErrUnsetNotAllowed = errors.New("engine: mapping does not allow unset") )
Sentinel errors of the server-owned publish path (docs/ROADMAP.md §7.2 file 6.9). The AppEngine layer (M7) maps them onto upstream-shaped HTTP statuses; payload validation failures surface as *payload.RejectError.
Functions ¶
This section is empty.
Types ¶
type BrokerPort ¶
type BrokerPort interface {
// Publish sends a server-side message (docs/ROADMAP.md §6 file 5.7):
// retain for properties, per-message expiry for datastreams.
Publish(topic string, payload []byte, qos byte, retain bool, expiry time.Duration) error
// RefreshIntrospection reloads a connected device's introspection-derived
// ACL state after the engine persists a new introspection
// (docs/ROADMAP.md §7.2 file 6.7).
RefreshIntrospection(ctx context.Context, realm string, id deviceid.ID) error
}
BrokerPort is the engine's broker-side port (hexagonal-lite, docs/DESIGN.md §1.3): server→device publishing and the ACL-relevant introspection refresh. Defined on the consumer side so tests substitute fakes; AdaptBroker wraps the real broker.
func AdaptBroker ¶
func AdaptBroker(b *broker.Broker) BrokerPort
AdaptBroker wraps the embedded broker as the engine's BrokerPort.
type Config ¶
type Config struct {
// Shards is the number of ordered pipeline shards (default
// DefaultShards). Messages of one device always land on the same shard.
Shards int
// ShardQueue is the per-shard channel capacity (default
// DefaultShardQueue). A full shard blocks QoS >= 1 submits (deferred-ack
// backpressure) and drops QoS 0 messages with a metric (§1.4).
ShardQueue int
// BatchMaxRows is the micro-batch row cap (default DefaultBatchMaxRows).
BatchMaxRows int
// BatchMaxWait is the micro-batch time cap (default DefaultBatchMaxWait).
BatchMaxWait time.Duration
// MaxPayloadBytes caps accepted data payload size for both formats
// (default payload.DefaultMaxSize).
MaxPayloadBytes int
// Registerer receives the engine's Prometheus collectors; nil leaves
// them unregistered (the collectors still work, which tests rely on).
Registerer prometheus.Registerer
// Logger receives engine logs (default slog.Default()).
Logger *slog.Logger
}
Config carries the engine's operational knobs (TOML wiring lands in M8).
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is the sharded ingestion pipeline (docs/ROADMAP.md §7.1 file 6.2). It implements broker.Intake and broker.LifecycleSink. M6a builds the data path; the M6b handlers (introspection, control, triggers, stream) attach to the seam fields below without touching the pipeline.
func New ¶
func New(st Store, bp BrokerPort, cfg Config) (*Engine, error)
New assembles the full engine (docs/ROADMAP.md §7.2 file 6.14): the M6a pipeline plus the M6b control-channel handlers, trigger evaluation, and the live fan-out bus. bp may be nil at construction time — the broker needs the engine as its intake, so M8 wires them in two steps — but must be attached (AttachBroker) before Start.
func (*Engine) AttachBroker ¶
func (e *Engine) AttachBroker(bp BrokerPort)
AttachBroker binds the broker port; it must run before Start (M8 wiring: engine → broker.New(intake=engine) → AttachBroker → Start).
func (*Engine) Bus ¶
Bus exposes the live fan-out bus (the M7b stream socket subscribes through it).
func (*Engine) Drain ¶
Drain gracefully stops the engine (docs/DESIGN.md §5.3): intake is refused, shards drain and flush their final batches, asynchronous control sends finish, the trigger executor drains its queue, and the live bus closes — all bounded by ctx. Call broker.Close first so no submitter blocks indefinitely.
func (*Engine) OnLifecycleEvent ¶
func (e *Engine) OnLifecycleEvent(ev broker.LifecycleEvent)
OnLifecycleEvent implements broker.LifecycleSink: device disconnects evict the per-device cache entry (docs/ROADMAP.md §7.1 file 6.1); the M6b seam then observes the event (device connect/disconnect triggers). It never blocks (broker.LifecycleSink contract).
func (*Engine) PublishServerValue ¶
func (e *Engine) PublishServerValue(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string, value json.RawMessage, ts *time.Time) error
PublishServerValue validates, persists, and delivers one server-owned value (docs/ROADMAP.md §7.2 file 6.9): AppEngine PUT/POST bodies land here. value is the raw JSON value (the unwrapped "data" field); it is validated against the mapping exactly like an inbound JSON-profile payload (docs/DESIGN.md §2.6 steps 4–5). ts is the optional explicit timestamp; reception time applies otherwise.
Persistence happens before delivery: a property upsert or a datastream insert, then a broker publish on the device's data topic with the mapping's QoS, retain for properties, per-message expiry for datastreams, and the wire format chosen by the device's payload_format_hint (docs/DESIGN.md §3.4, §3.5.4).
func (*Engine) RefreshInterfaces ¶
RefreshInterfaces is the in-process cache-invalidation callback (docs/DESIGN.md §2.6): realm management (M7) calls it after interface CRUD, complementing the LISTEN/NOTIFY path.
func (*Engine) RefreshTriggers ¶
RefreshTriggers rebuilds a realm's compiled snapshot — triggers ride in the same snapshot as interfaces, so this is the in-process invalidation callback for M7 trigger CRUD (docs/DESIGN.md §2.6).
func (*Engine) Start ¶
Start loads the schema snapshot, subscribes to interface-change notifications, and launches the shard goroutines. ctx bounds the engine's background work and must outlive Drain.
func (*Engine) Submit ¶
func (e *Engine) Submit(m broker.InboundMessage)
Submit implements broker.Intake (docs/DESIGN.md §1.4): route to the device's shard; block when the shard is full and the message is QoS >= 1 (the broker withholds the device's PUBACK, which is the backpressure); drop QoS 0 messages on a full shard with a metric.
func (*Engine) UnsetServerProperty ¶
func (e *Engine) UnsetServerProperty(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string) error
UnsetServerProperty deletes a server-owned property (AppEngine DELETE, docs/ROADMAP.md §8.2 file 7.7): the row is removed, an empty retained payload clears the topic and signals the unset to a connected device, and the `consumer/properties` purge message follows so offline-window state converges (docs/DESIGN.md §3.4). Unsetting an absent property is a no-op, not an error.
type OpKind ¶
type OpKind uint8
OpKind discriminates PersistOp values.
const ( // OpIndividual is one individual-datastream insert. OpIndividual OpKind = iota + 1 // OpObject is one object-datastream insert. OpObject // OpPropertySet is a property upsert. OpPropertySet // OpPropertyUnset is a property delete (empty payload, allow_unset). OpPropertyUnset )
OpKind values.
type PersistOp ¶
type PersistOp struct {
// Kind discriminates the persistence action.
Kind OpKind
// Realm and RealmID identify the tenant.
Realm string
RealmID int16
// DeviceID is the publishing device.
DeviceID deviceid.ID
// Interface is the compiled interface the message validated against.
Interface *interfaceschema.CompiledInterface
// Mapping is the matched endpoint mapping; nil for object aggregation.
Mapping *interfaceschema.CompiledMapping
// Path is the concrete data path ("" only for flat object aggregation).
Path string
// Value is the decoded payload value (payload.Value closed set;
// map[string]payload.Value for objects; nil for property unset).
Value payload.Value
// Format is the wire format the payload arrived in.
Format payload.Format
// TS is the effective sample timestamp: the explicit `t` when the
// mapping declares explicit_timestamp, broker reception time otherwise.
TS time.Time
// ReceptionTS is the broker reception timestamp.
ReceptionTS time.Time
// contains filtered or unexported fields
}
PersistOp is one validated operation emitted by the §2.6 pipeline (docs/ROADMAP.md §7.1 file 6.4), carried through the shard micro-batch and handed to the afterCommit observers (triggers and live fan-out, M6b).
type Store ¶
type Store interface {
// ListRealms feeds the realm name/ID resolution in the schema cache.
ListRealms(ctx context.Context) ([]store.Realm, error)
// LoadRealmInterfaces returns every installed interface of a realm for
// schema-snapshot rebuilds.
LoadRealmInterfaces(ctx context.Context, realmID int16) ([]*store.StoredInterface, error)
// Listen subscribes to a NOTIFY channel (cache invalidation,
// docs/DESIGN.md §2.6).
Listen(ctx context.Context, channel string) (<-chan store.Notification, error)
// GetDevice loads a device's introspection and payload-format hint.
GetDevice(ctx context.Context, realmID int16, id deviceid.ID) (*store.Device, error)
// SetPayloadFormatHint persists the sticky payload-format flip
// (docs/DESIGN.md §3.5.4).
SetPayloadFormatHint(ctx context.Context, realmID int16, id deviceid.ID, hint string) error
// AppendDatastreams commits one micro-batch in a single transaction.
AppendDatastreams(ctx context.Context, batch store.DatastreamBatch) error
// UpsertProperty applies a property set (last-value-wins).
UpsertProperty(ctx context.Context, p store.Property) error
// UnsetProperty applies a property unset (row delete).
UnsetProperty(ctx context.Context, realmID int16, deviceID deviceid.ID, interfaceID int64, path string) (bool, error)
// UpdateIntrospection replaces a device's introspection, returning the
// (name, major) pairs that were dropped (docs/ROADMAP.md §7.2 file 6.7).
UpdateIntrospection(ctx context.Context, realmID int16, id deviceid.ID, intro map[string]store.InterfaceVersion) (map[string]store.InterfaceVersion, error)
// PurgeDeviceOwnedExcept implements the `producer/properties` resync
// (docs/DESIGN.md §3.3): device-owned properties not in keep are deleted.
PurgeDeviceOwnedExcept(ctx context.Context, realmID int16, deviceID deviceid.ID, keep []store.PropertyRef) (int64, error)
// ListServerOwnedProperties returns the device's server-owned properties
// (emptyCache resend + `consumer/properties` payload, docs/DESIGN.md §3.4).
ListServerOwnedProperties(ctx context.Context, realmID int16, deviceID deviceid.ID) ([]store.Property, error)
// ListTriggers returns a realm's installed triggers for the compiled
// trigger cache (docs/ROADMAP.md §7.2 file 6.10).
ListTriggers(ctx context.Context, realmID int16) ([]store.Trigger, error)
}
Store is the engine's persistence port (hexagonal-lite, docs/DESIGN.md §1.3): the subset of *store.Store the M6a pipeline needs. The interface is defined here, on the consumer side, so tests substitute fakes and M8 wiring is a plain assignment.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters.
|
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters. |
|
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1).
|
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1). |