engine

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package engine is the heart of Astrate (docs/ROADMAP.md §7): it consumes every accepted device PUBLISH from the broker intake, validates it against the realm's compiled interface schemas (docs/DESIGN.md §2.6), and persists it through per-shard micro-batches with strict per-device ordering and ack-after-commit semantics (docs/DESIGN.md §1.4, §5.3).

M6a (this milestone slice) implements the pipeline and the data path: compiled-interface and device caches, the sharded router, topic classification, the §2.6 validation pipeline, and batched persistence. M6b adds the control-channel handlers, server-owned data publishing, triggers, and the live stream — they attach to the seams the Engine struct already exposes (handler fields and the afterCommit hook).

Index

Constants

View Source
const (
	// DefaultShards is the default shard count.
	DefaultShards = 16
	// DefaultShardQueue is the default per-shard channel capacity.
	DefaultShardQueue = 4096
	// DefaultBatchMaxRows flushes a shard batch when it reaches this size.
	DefaultBatchMaxRows = 64
	// DefaultBatchMaxWait flushes a non-empty shard batch after this delay.
	DefaultBatchMaxWait = 50 * time.Millisecond
)

Configuration defaults (docs/DESIGN.md §1.4).

Variables

View Source
var (
	// ErrRealmUnknown: the realm is not in the schema snapshot.
	ErrRealmUnknown = errors.New("engine: realm unknown")
	// ErrInterfaceNotFound: the realm has no installed interface by that name.
	ErrInterfaceNotFound = errors.New("engine: interface not installed")
	// ErrNotServerOwned: the interface is device-owned (docs/DESIGN.md §2.6
	// step 3: AppEngine publishes only on ownership: server).
	ErrNotServerOwned = errors.New("engine: interface is not server-owned")
	// ErrPathNotFound: the path resolves no endpoint mapping.
	ErrPathNotFound = errors.New("engine: path matches no endpoint")
	// ErrNotAProperty: a property operation on a datastream interface.
	ErrNotAProperty = errors.New("engine: interface is not a properties interface")
	// ErrUnsetNotAllowed: unset on a mapping without allow_unset.
	ErrUnsetNotAllowed = errors.New("engine: mapping does not allow unset")
)

Sentinel errors of the server-owned publish path (docs/ROADMAP.md §7.2 file 6.9). The AppEngine layer (M7) maps them onto upstream-shaped HTTP statuses; payload validation failures surface as *payload.RejectError.

Functions

This section is empty.

Types

type BrokerPort

type BrokerPort interface {
	// Publish sends a server-side message (docs/ROADMAP.md §6 file 5.7):
	// retain for properties, per-message expiry for datastreams.
	Publish(topic string, payload []byte, qos byte, retain bool, expiry time.Duration) error
	// RefreshIntrospection reloads a connected device's introspection-derived
	// ACL state after the engine persists a new introspection
	// (docs/ROADMAP.md §7.2 file 6.7).
	RefreshIntrospection(ctx context.Context, realm string, id deviceid.ID) error
}

BrokerPort is the engine's broker-side port (hexagonal-lite, docs/DESIGN.md §1.3): server→device publishing and the ACL-relevant introspection refresh. Defined on the consumer side so tests substitute fakes; AdaptBroker wraps the real broker.

func AdaptBroker

func AdaptBroker(b *broker.Broker) BrokerPort

AdaptBroker wraps the embedded broker as the engine's BrokerPort.

type Config

type Config struct {
	// Shards is the number of ordered pipeline shards (default
	// DefaultShards). Messages of one device always land on the same shard.
	Shards int
	// ShardQueue is the per-shard channel capacity (default
	// DefaultShardQueue). A full shard blocks QoS >= 1 submits (deferred-ack
	// backpressure) and drops QoS 0 messages with a metric (§1.4).
	ShardQueue int
	// BatchMaxRows is the micro-batch row cap (default DefaultBatchMaxRows).
	BatchMaxRows int
	// BatchMaxWait is the micro-batch time cap (default DefaultBatchMaxWait).
	BatchMaxWait time.Duration
	// MaxPayloadBytes caps accepted data payload size for both formats
	// (default payload.DefaultMaxSize).
	MaxPayloadBytes int
	// Registerer receives the engine's Prometheus collectors; nil leaves
	// them unregistered (the collectors still work, which tests rely on).
	Registerer prometheus.Registerer
	// Logger receives engine logs (default slog.Default()).
	Logger *slog.Logger
}

Config carries the engine's operational knobs (TOML wiring lands in M8).

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is the sharded ingestion pipeline (docs/ROADMAP.md §7.1 file 6.2). It implements broker.Intake and broker.LifecycleSink. M6a builds the data path; the M6b handlers (introspection, control, triggers, stream) attach to the seam fields below without touching the pipeline.

func New

func New(st Store, bp BrokerPort, cfg Config) (*Engine, error)

New assembles the full engine (docs/ROADMAP.md §7.2 file 6.14): the M6a pipeline plus the M6b control-channel handlers, trigger evaluation, and the live fan-out bus. bp may be nil at construction time — the broker needs the engine as its intake, so M8 wires them in two steps — but must be attached (AttachBroker) before Start.

func (*Engine) AttachBroker

func (e *Engine) AttachBroker(bp BrokerPort)

AttachBroker binds the broker port; it must run before Start (M8 wiring: engine → broker.New(intake=engine) → AttachBroker → Start).

func (*Engine) Bus

func (e *Engine) Bus() *stream.Bus

Bus exposes the live fan-out bus (the M7b stream socket subscribes through it).

func (*Engine) Drain

func (e *Engine) Drain(ctx context.Context) error

Drain gracefully stops the engine (docs/DESIGN.md §5.3): intake is refused, shards drain and flush their final batches, asynchronous control sends finish, the trigger executor drains its queue, and the live bus closes — all bounded by ctx. Call broker.Close first so no submitter blocks indefinitely.

func (*Engine) OnLifecycleEvent

func (e *Engine) OnLifecycleEvent(ev broker.LifecycleEvent)

OnLifecycleEvent implements broker.LifecycleSink: device disconnects evict the per-device cache entry (docs/ROADMAP.md §7.1 file 6.1); the M6b seam then observes the event (device connect/disconnect triggers). It never blocks (broker.LifecycleSink contract).

func (*Engine) PublishServerValue

func (e *Engine) PublishServerValue(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string, value json.RawMessage, ts *time.Time) error

PublishServerValue validates, persists, and delivers one server-owned value (docs/ROADMAP.md §7.2 file 6.9): AppEngine PUT/POST bodies land here. value is the raw JSON value (the unwrapped "data" field); it is validated against the mapping exactly like an inbound JSON-profile payload (docs/DESIGN.md §2.6 steps 4–5). ts is the optional explicit timestamp; reception time applies otherwise.

Persistence happens before delivery: a property upsert or a datastream insert, then a broker publish on the device's data topic with the mapping's QoS, retain for properties, per-message expiry for datastreams, and the wire format chosen by the device's payload_format_hint (docs/DESIGN.md §3.4, §3.5.4).

func (*Engine) RefreshInterfaces

func (e *Engine) RefreshInterfaces(ctx context.Context, realmID int16) error

RefreshInterfaces is the in-process cache-invalidation callback (docs/DESIGN.md §2.6): realm management (M7) calls it after interface CRUD, complementing the LISTEN/NOTIFY path.

func (*Engine) RefreshTriggers

func (e *Engine) RefreshTriggers(ctx context.Context, realmID int16) error

RefreshTriggers rebuilds a realm's compiled snapshot — triggers ride in the same snapshot as interfaces, so this is the in-process invalidation callback for M7 trigger CRUD (docs/DESIGN.md §2.6).

func (*Engine) Start

func (e *Engine) Start(ctx context.Context) error

Start loads the schema snapshot, subscribes to interface-change notifications, and launches the shard goroutines. ctx bounds the engine's background work and must outlive Drain.

func (*Engine) Submit

func (e *Engine) Submit(m broker.InboundMessage)

Submit implements broker.Intake (docs/DESIGN.md §1.4): route to the device's shard; block when the shard is full and the message is QoS >= 1 (the broker withholds the device's PUBACK, which is the backpressure); drop QoS 0 messages on a full shard with a metric.

func (*Engine) UnsetServerProperty

func (e *Engine) UnsetServerProperty(ctx context.Context, realm string, id deviceid.ID, ifaceName, path string) error

UnsetServerProperty deletes a server-owned property (AppEngine DELETE, docs/ROADMAP.md §8.2 file 7.7): the row is removed, an empty retained payload clears the topic and signals the unset to a connected device, and the `consumer/properties` purge message follows so offline-window state converges (docs/DESIGN.md §3.4). Unsetting an absent property is a no-op, not an error.

type OpKind

type OpKind uint8

OpKind discriminates PersistOp values.

const (
	// OpIndividual is one individual-datastream insert.
	OpIndividual OpKind = iota + 1
	// OpObject is one object-datastream insert.
	OpObject
	// OpPropertySet is a property upsert.
	OpPropertySet
	// OpPropertyUnset is a property delete (empty payload, allow_unset).
	OpPropertyUnset
)

OpKind values.

func (OpKind) String

func (k OpKind) String() string

String returns the stable snake_case metrics label.

type PersistOp

type PersistOp struct {
	// Kind discriminates the persistence action.
	Kind OpKind
	// Realm and RealmID identify the tenant.
	Realm   string
	RealmID int16
	// DeviceID is the publishing device.
	DeviceID deviceid.ID
	// Interface is the compiled interface the message validated against.
	Interface *interfaceschema.CompiledInterface
	// Mapping is the matched endpoint mapping; nil for object aggregation.
	Mapping *interfaceschema.CompiledMapping
	// Path is the concrete data path ("" only for flat object aggregation).
	Path string
	// Value is the decoded payload value (payload.Value closed set;
	// map[string]payload.Value for objects; nil for property unset).
	Value payload.Value
	// Format is the wire format the payload arrived in.
	Format payload.Format
	// TS is the effective sample timestamp: the explicit `t` when the
	// mapping declares explicit_timestamp, broker reception time otherwise.
	TS time.Time
	// ReceptionTS is the broker reception timestamp.
	ReceptionTS time.Time
	// contains filtered or unexported fields
}

PersistOp is one validated operation emitted by the §2.6 pipeline (docs/ROADMAP.md §7.1 file 6.4), carried through the shard micro-batch and handed to the afterCommit observers (triggers and live fan-out, M6b).

type Store

type Store interface {
	// ListRealms feeds the realm name/ID resolution in the schema cache.
	ListRealms(ctx context.Context) ([]store.Realm, error)
	// LoadRealmInterfaces returns every installed interface of a realm for
	// schema-snapshot rebuilds.
	LoadRealmInterfaces(ctx context.Context, realmID int16) ([]*store.StoredInterface, error)
	// Listen subscribes to a NOTIFY channel (cache invalidation,
	// docs/DESIGN.md §2.6).
	Listen(ctx context.Context, channel string) (<-chan store.Notification, error)
	// GetDevice loads a device's introspection and payload-format hint.
	GetDevice(ctx context.Context, realmID int16, id deviceid.ID) (*store.Device, error)
	// SetPayloadFormatHint persists the sticky payload-format flip
	// (docs/DESIGN.md §3.5.4).
	SetPayloadFormatHint(ctx context.Context, realmID int16, id deviceid.ID, hint string) error
	// AppendDatastreams commits one micro-batch in a single transaction.
	AppendDatastreams(ctx context.Context, batch store.DatastreamBatch) error
	// UpsertProperty applies a property set (last-value-wins).
	UpsertProperty(ctx context.Context, p store.Property) error
	// UnsetProperty applies a property unset (row delete).
	UnsetProperty(ctx context.Context, realmID int16, deviceID deviceid.ID, interfaceID int64, path string) (bool, error)
	// UpdateIntrospection replaces a device's introspection, returning the
	// (name, major) pairs that were dropped (docs/ROADMAP.md §7.2 file 6.7).
	UpdateIntrospection(ctx context.Context, realmID int16, id deviceid.ID, intro map[string]store.InterfaceVersion) (map[string]store.InterfaceVersion, error)
	// PurgeDeviceOwnedExcept implements the `producer/properties` resync
	// (docs/DESIGN.md §3.3): device-owned properties not in keep are deleted.
	PurgeDeviceOwnedExcept(ctx context.Context, realmID int16, deviceID deviceid.ID, keep []store.PropertyRef) (int64, error)
	// ListServerOwnedProperties returns the device's server-owned properties
	// (emptyCache resend + `consumer/properties` payload, docs/DESIGN.md §3.4).
	ListServerOwnedProperties(ctx context.Context, realmID int16, deviceID deviceid.ID) ([]store.Property, error)
	// ListTriggers returns a realm's installed triggers for the compiled
	// trigger cache (docs/ROADMAP.md §7.2 file 6.10).
	ListTriggers(ctx context.Context, realmID int16) ([]store.Trigger, error)
}

Store is the engine's persistence port (hexagonal-lite, docs/DESIGN.md §1.3): the subset of *store.Store the M6a pipeline needs. The interface is defined here, on the consumer side, so tests substitute fakes and M8 wiring is a plain assignment.

Directories

Path Synopsis
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters.
Package stream is the in-process live fan-out bus (docs/ROADMAP.md §7.2 file 6.13, docs/DESIGN.md §1.1): the engine publishes every committed data operation and device lifecycle event, and consumers — the M7b WebSocket/SSE endpoint, tests — subscribe per realm with optional filters.
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1).
Package triggers compiles stored Astarte trigger definitions into fast matchers, renders the upstream-parity SimpleEvent JSON payloads, and executes HTTP webhook actions with retry (docs/ROADMAP.md §7.2 files 6.10–6.12, docs/DESIGN.md §1.1).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL