Documentation
¶
Overview ¶
Package message provides the core message infrastructure for SemEngine. It defines interfaces and types for creating, validating, and processing messages that flow through the semantic event mesh.
Architecture ¶
The package follows a clean, domain-agnostic design with three core concepts:
1. Messages - Containers that combine typed payloads with metadata 2. Payloads - Domain-specific data that may implement behavioral interfaces 3. Metadata - Information about message lifecycle and origin
Message Structure ¶
Every message consists of:
- A unique ID for tracking and deduplication
- A structured Type (domain, category, version)
- A Payload containing the actual data
- Metadata about creation time, source, etc.
- A content-based hash for integrity
Behavioral Interfaces ¶
The message package uses runtime capability discovery through optional interfaces. Payloads implement only the interfaces relevant to their domain, and services discover these capabilities dynamically through type assertions.
## Entity Interface
Storable: Declares an entity, its facts, and where its full data is stored
- EntityID() string - Returns federated entity identifier
- Triples() []Triple - Returns semantic facts about the entity
- StorageRef() *StorageReference - Returns where the full data is stored, or nil
- Use when: Payload represents an entity whose facts and stored data a consumer needs
- ContentStorable and BinaryStorable extend it for text and binary content
## Rule Interfaces
RuleReadable: Declares the fields rule conditions and `$message.*` substitutions may read from this payload
- RuleFields() map[string]any - Returns the payload's declared projection
- Use when: A rule should be able to condition or substitute on this payload. WITHOUT it the rule engine cannot read the payload at all: every `$message.*` condition against it is false forever, and the engine reports the rule/payload-type pairing once rather than failing silently.
- Expose structural facts (ids, roles, outcomes, states, counts, routing); withhold LLM-authored and user content and open caller-populated maps. There is no reflective default, by design — a field reaches a rule only because its author exposed it.
- Example: Loop outcome events, tool-call identity, approval decisions. GenericJSONPayload implements it by returning its whole Data map.
## Runtime Discovery Pattern
Services discover capabilities at runtime through type assertions:
// Check for entity data
if entity, ok := msg.Payload().(Storable); ok {
entityID := entity.EntityID()
triples := entity.Triples()
ref := entity.StorageRef() // nil when the data is not stored externally
// Record the facts, fetch the stored data through ref when needed, etc.
}
// Check for rule-readable fields
if readable, ok := msg.Payload().(RuleReadable); ok {
fields := readable.RuleFields()
// Evaluate conditions, resolve $message.* substitutions, etc.
}
This pattern enables services to process any message type without prior knowledge of the specific payload structure, discovering capabilities dynamically.
Type System Hierarchy ¶
Two related but distinct identifiers appear in this package. Both are defined in pkg/types and produce dotted notation through a Key() method, for NATS routing, storage keys, and entity identification. Type is an alias for types.Type, kept so payload code can say message.Type. Entity identifiers are types.EntityID and are not aliased here.
## 1. Type (Message Schema) - "domain.category.version"
Purpose: Identifies message schemas for routing, validation, and evolution. Format: Type{Domain: "sensors", Category: "gps", Version: "v1"} -> "sensors.gps.v1"
Use When:
- Defining message schemas in payload implementations (Schema() method)
- Configuring component input/output types
- Routing messages through NATS subjects
- Versioning payload schemas for evolution
Example:
type TemperaturePayload struct { /* fields */ }
func (t *TemperaturePayload) Schema() Type {
return Type{Domain: "sensors", Category: "temperature", Version: "v1"}
}
// Used for: NATS subject "sensors.temperature.v1"
## 2. EntityID (Federated Identity) - "org.platform.system.domain.type.instance"
Purpose: Provides globally unique entity identifiers across federated platforms. Format: EntityID with 6 parts -> "c360.platform1.mav1.robotics.drone.1"
Use When:
- Multi-platform deployments need unique entity identity
- Merging data from multiple sources with overlapping IDs
- Federating entities across organizational boundaries
- Building distributed knowledge graphs
Example:
entityID := types.EntityID{
Org: "c360", // Organization namespace (platform.org)
Platform: "platform1", // Minting deployment authority (platform.id)
System: "mav1", // Source that produced the entity
Domain: "robotics", // Delegated taxonomy
Type: "drone", // Entity type within the domain
Instance: "1", // Leaf identifier
}
// Key(): "c360.platform1.mav1.robotics.drone.1"
// Used for: Federated entity resolution and deduplication
## Choosing the Right Type
Ask yourself:
- Defining a message schema? → Use Type (Schema() method)
- Providing entity identity? → Use types.EntityID (Storable.EntityID() method)
Most payloads only need Type (for Schema()). Only implement Storable if your payload represents an entity whose facts and stored data a consumer needs.
Usage Example ¶
// Define a payload type
type TemperaturePayload struct {
SensorID string `json:"sensor_id"`
Temperature float64 `json:"temperature"`
Unit string `json:"unit"`
Timestamp time.Time `json:"timestamp"`
}
// Implement required Payload interface
func (t *TemperaturePayload) Schema() Type {
return Type{
Domain: "sensors",
Category: "temperature",
Version: "v1",
}
}
func (t *TemperaturePayload) Validate() error {
if t.SensorID == "" {
return errors.New("sensor ID required")
}
return nil
}
func (t *TemperaturePayload) MarshalJSON() ([]byte, error) {
// Use alias to avoid infinite recursion
type Alias TemperaturePayload
return json.Marshal((*Alias)(t))
}
func (t *TemperaturePayload) UnmarshalJSON(data []byte) error {
// Use alias to avoid infinite recursion
type Alias TemperaturePayload
return json.Unmarshal(data, (*Alias)(t))
}
// Implement an optional behavioral interface (RuleReadable)
func (t *TemperaturePayload) RuleFields() map[string]any {
return map[string]any{"sensor_id": t.SensorID, "temperature": t.Temperature}
}
// Register the type so a Decoder can rebuild it from the wire
reg := payloadregistry.New()
if err := reg.Register(&payloadregistry.Registration{
Domain: "sensors",
Category: "temperature",
Version: "v1",
Description: "Temperature reading",
Factory: func() any { return &TemperaturePayload{} },
}); err != nil {
return err
}
// Create and use a message
payload := &TemperaturePayload{
SensorID: "temp-001",
Temperature: 22.5,
Unit: "celsius",
Timestamp: time.Now(),
}
msg := NewBaseMessage(
payload.Schema(),
payload,
"temperature-monitor",
)
// After NewDecoder(reg).Decode, a service reads the payload by its
// concrete type, or by an interface it implements
if reading, ok := msg.Payload().(*TemperaturePayload); ok {
fmt.Printf("%s: %.1f %s\n", reading.SensorID, reading.Temperature, reading.Unit)
}
if readable, ok := msg.Payload().(RuleReadable); ok {
fields := readable.RuleFields()
fmt.Printf("rule fields: %v\n", fields)
}
Message Lifecycle ¶
Messages follow a predictable lifecycle from creation through processing:
## 1. Creation
Messages are created using NewBaseMessage with optional configuration:
// Simple message with current timestamp
msg := NewBaseMessage(payload.Schema(), payload, "my-service")
// Historical data with specific timestamp
msg := NewBaseMessage(payload.Schema(), payload, "my-service",
WithTime(historicalTime))
Messages are immutable after creation - all fields are set during construction and cannot be modified. This ensures message integrity throughout processing.
## 2. Validation
Validation happens at two levels:
Structural Validation (Required):
- Message type must be valid (domain, category, version all present)
- Payload must not be nil
- Metadata must not be nil
Payload Validation (Domain-specific):
- Implemented by Payload.Validate() method
- Checks domain-specific business rules
- May be structural (required fields) or semantic (value ranges)
Example validation:
if err := msg.Validate(); err != nil {
// Handle validation error
// Check error type: Fatal, Transient, or Invalid
}
## 3. Serialization
Messages are serialized to JSON for transmission over NATS:
// Serialize message
data, err := json.Marshal(msg)
// Deserialize message: the Decoder resolves the payload type against
// the registry it was built with
reg := payloadregistry.New()
if err := RegisterPayloads(reg); err != nil {
return err
}
decoded, err := NewDecoder(reg).Decode(data)
if err != nil {
return err
}
Wire format preserves:
- Message ID for deduplication and tracking
- Type information for routing and schema validation
- Payload data using Payload.MarshalJSON()
- Metadata timestamps (millisecond precision) and source
Note: there is no global registry. Deserialization goes through a Decoder built with a payloadregistry.Registry that holds the message's payload type; decoding a zero-value BaseMessage with json.Unmarshal always fails, because it has no registry. For generic JSON processing, register the well-known type "core.json.v1" (GenericJSONPayload) with RegisterPayloads.
GenericJSONPayload is the fallback for JSON whose shape is not known when the code is written: outside input, or a transform the user configures. Code that builds a shape it knows registers a payload type instead. Its Data holds JSON-shaped values only (map[string]any, []any, string, Go numbers, json.Number, bool and nil); encoding refuses any other value with an error naming its type and path, and any string or key that is not valid UTF-8.
## 4. Transmission
Messages are published to NATS subjects derived from their type:
subject := msg.Type().Key() // e.g., "sensors.temperature.v1" nc.Publish(subject, data)
This enables:
- Type-based routing using NATS wildcards
- Version-specific processing
- Domain isolation and security policies
## 5. Processing
Services receive and process messages based on their subscriptions:
// Subscribe to specific type
nc.Subscribe("sensors.temperature.v1", handler)
// Subscribe to all versions
nc.Subscribe("sensors.temperature.*", handler)
// Subscribe to entire domain
nc.Subscribe("sensors.>", handler)
Services discover payload capabilities at runtime:
// decoder is built once at startup: NewDecoder(reg)
func handler(m *nats.Msg) {
msg, err := decoder.Decode(m.Data)
if err != nil {
return // malformed, or a payload type reg does not hold
}
// Discover capabilities
if entity, ok := msg.Payload().(Storable); ok {
// Process entity data: entity.EntityID(), entity.Triples()
}
}
Best Practices ¶
## Payload Implementation
1. Implement Required Methods
- Schema() Type - Return structured type information
- Validate() error - Perform domain-specific validation
- MarshalJSON() ([]byte, error) - Use alias pattern to avoid recursion
- UnmarshalJSON([]byte) error - Use alias pattern to avoid recursion
2. Implement Optional Interfaces Thoughtfully
- Only implement behavioral interfaces that make semantic sense
- Consider whether your payload truly represents an entity before implementing Storable
3. Validation Philosophy
- Structural validation: Check required fields and basic types
- Semantic validation: Check business rules and value ranges (optional)
- Fail fast: Return first validation error encountered
- Provide clear error messages with context
4. Use Type Constants
Define message types as package constants
Don't inline type construction at call sites
Example:
// Good: Type as constant var TemperatureType = message.Type{ Domain: "sensors", Category: "temperature", Version: "v1", }
// Bad: Inline type construction msg := NewBaseMessage( message.Type{Domain: "sensors", Category: "temperature", Version: "v1"}, payload, source)
## Message Creation
1. Use Functional Options for Configuration
- Start with simple NewBaseMessage(type, payload, source)
- Add options only when needed: WithTime(), WithMeta()
- Don't create custom constructors - use options instead
2. Set Source Meaningfully
- Use service name or component identifier
- Be consistent across your application
- Example: "temperature-monitor", "gps-processor", "user-service"
3. Timestamp Considerations
- Default timestamp (time.Now()) is correct for most cases
- Use WithTime() only for historical data or testing
## Message Processing
1. Type Assertion Pattern
Use comma-ok idiom for capability discovery
Don't assume interfaces are implemented
Example:
if entity, ok := msg.Payload().(Storable); ok { // Safe to use entity's methods } else { // Payload doesn't implement Storable, skip or handle differently }
2. Error Handling
- Always check validation errors before processing
- Use errors.As() to detect error types (Fatal, Transient, Invalid)
- Don't rely on error message strings for logic
3. Immutability
- Don't modify message fields after creation
- Create new messages for transformations
- Payload data can be read but not written
## Testing
1. Use Real Test Data
- Create actual payload instances, don't use mocks
- Test with both valid and invalid data
- Test serialization round-trips
2. Test Behavioral Interfaces
- Verify interface implementations return correct values
- Test type assertions work as expected
- Ensure optional interfaces are truly optional
3. Test Validation
- Test both valid and invalid payloads
- Verify error messages are clear
- Test edge cases and boundary conditions
Design Philosophy ¶
The messages package embodies several key design principles:
1. Separation of Concerns: Messages know nothing about routing, storage, or entity extraction. They are pure data containers.
2. Discoverability: Behavioral interfaces allow services to discover and utilize message capabilities at runtime without tight coupling.
3. Domain Agnosticism: The core package contains no domain-specific code. All domain logic lives in domain packages that use these interfaces.
4. Extensibility: New behavioral interfaces can be added without breaking existing code. Payloads can implement any combination of interfaces.
5. Type Safety: Structured Type provides compile-time safety while maintaining flexibility.
Package message - the RuleReadable behavioral interface.
A sibling of the optional behavior in behaviors.go (IndexingProfiler), discovered the same way and optional in the same sense.
Package message provides core message infrastructure for SemEngine. See doc.go for complete package documentation.
Index ¶
- Constants
- func AppendIdentityKey(t Triple) string
- func IsValidEntityID(s string) bool
- func ParseEntityID(value string) (types.EntityID, error)
- func RegisterPayloads(reg *payloadregistry.Registry) error
- func SameAppendTuple(left, right Triple) bool
- type BaseMessage
- type BinaryContent
- type BinaryStorable
- type ContentStorable
- type Decoder
- type DefaultMeta
- type GenericJSONPayload
- type IndexingProfiler
- type Message
- type Meta
- type Option
- type Payload
- type RuleReadable
- type Storable
- type StorageReference
- type Triple
- type Type
Constants ¶
const ( // ContentRoleBody is the primary text content (full document text). // Embedding workers prioritize this role for text extraction. ContentRoleBody = "body" // ContentRoleAbstract is a brief summary or description. // Used when body is not available or for additional context. ContentRoleAbstract = "abstract" // ContentRoleTitle is the document or entity title. // Typically included in embeddings for context. ContentRoleTitle = "title" // ContentRoleMedia is the primary media content (video, image, audio). // Used for binary content that may be processed by multimodal embedders. ContentRoleMedia = "media" // ContentRoleThumbnail is a preview image for media content. ContentRoleThumbnail = "thumbnail" )
ContentRole constants define standard semantic roles for ContentFields. Using these constants ensures consistency across implementations.
const EntityReferenceDatatype = "@id"
EntityReferenceDatatype marks a triple object as an explicit entity reference. Complete EntityState validation requires marked objects to be canonical entity-ID strings; unmarked canonical-ID-shaped strings retain their existing structural relationship behavior.
Variables ¶
This section is empty.
Functions ¶
func AppendIdentityKey ¶
AppendIdentityKey returns the canonical six-field add-lane identity key for a triple. Two triples denote the same assertion for append purposes if and only if their keys are equal; SameAppendTuple is defined in exactly those terms so a key-based set (O(n+k)) and a pairwise comparison can never disagree.
The key is LENGTH-PREFIXED, not delimiter-joined. Predicates, sources, and contexts are arbitrary operator- and producer-supplied strings; joining them on a dot or a pipe would make ("a.b", "c") and ("a", "b.c") the same key — the raw-key-collision class that causes silent data loss elsewhere in the system. Length prefixes make the encoding injective for every possible field content, including embedded NUL bytes and embedded digits.
func IsValidEntityID ¶
IsValidEntityID checks if a string conforms to the canonical 6-part EntityID format. Valid format: organization.platform.system.domain.type.instance (e.g., "c360.platform1.mav1.robotics.drone.0") Each part must contain only alphanumeric characters, hyphens, or underscores. This prevents false positives from strings that happen to have 5 dots (e.g., JSON with file extensions like ".go", ".md") and ensures the ID is safe as a NATS KV key.
func ParseEntityID ¶
ParseEntityID delegates canonical parsing to pkg/types.
func RegisterPayloads ¶
func RegisterPayloads(reg *payloadregistry.Registry) error
RegisterPayloads registers the GenericJSON payload type (core.json.v1) with the supplied registry. Nothing registers it implicitly: a process that decodes core.json.v1 calls this on the registry its Decoder uses.
func SameAppendTuple ¶
SameAppendTuple reports whether two triples are the same assertion on the add lane. It is exactly key equality — see AppendIdentityKey.
Types ¶
type BaseMessage ¶
type BaseMessage struct {
// contains filtered or unexported fields
}
BaseMessage provides the standard implementation of the Message interface. It combines a typed payload with metadata to create a complete message ready for transmission through the semantic event mesh.
BaseMessage is immutable after creation - all fields are set during construction and cannot be modified. This ensures message integrity throughout the processing pipeline.
Construction using Functional Options:
NewBaseMessage uses the functional options pattern for clean, composable configuration:
// Simple message (most common) msg := NewBaseMessage(msgType, payload, "my-service") // With specific timestamp (testing/historical data) msg := NewBaseMessage(msgType, payload, "my-service", WithTime(pastTime))
func NewBaseMessage ¶
func NewBaseMessage(msgType Type, payload Payload, source string, opts ...Option) *BaseMessage
NewBaseMessage creates a new BaseMessage with optional configuration.
Parameters:
- msgType: Structured type information (domain, category, version)
- payload: The message payload implementing the Payload interface
- source: Identifier of the service or component creating this message
- opts: Optional configuration functions
Examples:
// Simple message with current timestamp msg := NewBaseMessage(msgType, payload, "my-service") // Message with specific timestamp (for historical data) msg := NewBaseMessage(msgType, payload, "my-service", WithTime(pastTime))
func (*BaseMessage) ID ¶
func (m *BaseMessage) ID() string
ID returns the unique message identifier.
func (*BaseMessage) MarshalJSON ¶
func (m *BaseMessage) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler for BaseMessage. This allows BaseMessage to be serialized to JSON even though its fields are private.
Validation is performed before serialization to ensure invalid messages cannot be serialized and published to the message bus. A meta source or a type component that is not valid UTF-8 is refused with an invalid-data error rather than written with its invalid bytes replaced.
func (*BaseMessage) Payload ¶
func (m *BaseMessage) Payload() Payload
Payload returns the message payload.
func (*BaseMessage) Type ¶
func (m *BaseMessage) Type() Type
Type returns the structured message type.
func (*BaseMessage) UnmarshalJSON ¶
func (m *BaseMessage) UnmarshalJSON(data []byte) error
UnmarshalJSON implements json.Unmarshaler for BaseMessage.
Resolves the payload type discriminator against m.registry, which must be set by the Decoder before json.Unmarshal is invoked. Callers, tests included, MUST go through message.NewDecoder(reg).Decode(data). Zero-value BaseMessage is a hard error — there is no global fallback and no test-mode side door. Mirrors stdlib's bufio.Scanner state-set-by-constructor pattern.
func (*BaseMessage) Validate ¶
func (m *BaseMessage) Validate() error
Validate performs comprehensive message validation.
type BinaryContent ¶
type BinaryContent struct {
ContentType string // MIME type: "image/jpeg", "video/mp4", "application/pdf"
Data []byte // The actual binary data
}
BinaryContent describes a binary field for storage. Used by BinaryStorable implementations to provide binary data along with its content type for proper handling.
type BinaryStorable ¶
type BinaryStorable interface {
ContentStorable
// BinaryFields returns binary content to store in ObjectStore.
//
// Each field is stored as a separate binary blob with its own key.
// The returned map keys are field names that can be referenced in
// ContentFields for semantic role mapping.
//
// Example return value:
// {"video": {ContentType: "video/mp4", Data: videoBytes},
// "thumbnail": {ContentType: "image/jpeg", Data: thumbBytes}}
BinaryFields() map[string]BinaryContent
}
BinaryStorable extends ContentStorable for payloads with binary content.
This interface enables storage of images, videos, PDFs, and other binary files alongside text content. Binary data is stored directly in ObjectStore (no base64 encoding) with JSON metadata referencing the binary keys.
Example implementation:
type VideoDocument struct {
Title string
Description string
VideoData []byte
Thumbnail []byte
}
func (v *VideoDocument) RawContent() map[string]string {
return map[string]string{
"title": v.Title,
"description": v.Description,
}
}
func (v *VideoDocument) BinaryFields() map[string]BinaryContent {
return map[string]BinaryContent{
"video": {ContentType: "video/mp4", Data: v.VideoData},
"thumbnail": {ContentType: "image/jpeg", Data: v.Thumbnail},
}
}
type ContentStorable ¶
type ContentStorable interface {
Storable // EntityID() + Triples() + StorageRef()
// ContentFields returns semantic role → field name mapping.
//
// This tells consumers how to find content in the stored data without
// hardcoding field names. Keys are semantic roles understood by consumers
// (like embedding workers), values are field names in RawContent().
//
// Standard semantic roles:
// - "body": Primary text content (full document text)
// - "abstract": Brief summary or description
// - "title": Document title
//
// Example return value:
// {"body": "content", "abstract": "description", "title": "title"}
//
// The map should only include roles that have non-empty content.
ContentFields() map[string]string
// RawContent returns the content to store in ObjectStore.
//
// Field names in this map should match values in ContentFields().
// objectstore.Store.StoreContent stores the body field's text as raw
// bytes and refuses any other non-empty field (see the type's doc), so a
// consumer that opens the StorageRef reads the body and nothing else.
//
// Example return value:
// {"body": "Full document text..."}
RawContent() map[string]string
}
ContentStorable extends Storable for payloads with large content fields.
This interface enables the "process → store → graph" pattern where:
- Processor creates semantic understanding (triples with metadata only)
- ObjectStore stores raw content and returns StorageReference
- GraphProcessor receives ContentStorable with both semantics and content reference
- EmbeddingWorker uses ContentFields to find text for embedding
The key insight is that ContentStorable separates metadata (in triples) from content (in ObjectStore), while providing a semantic map of content structure via ContentFields. This avoids bloating triples with large text and enables efficient embedding extraction without hardcoded field name coupling.
What the store keeps: objectstore.Store.StoreContent stores only the text of the "body" role, as raw bytes, and the StorageReference it returns opens to exactly that text. A payload with any other non-empty content (text under a field another role names, an undeclared field, or a non-empty binary field of a BinaryStorable) is refused with an invalid-input error before anything is written. Text that is not the body, such as a title or an abstract, belongs in the graph as triples, which the producer emits from Triples().
Example implementation (a document whose only stored content is its body; the title and description reach the graph as triples):
type Document struct {
Title string
Description string
Body string
storageRef *StorageReference
}
func (d *Document) EntityID() string { return d.entityID }
func (d *Document) Triples() []Triple { return metadataTriples } // NO body
func (d *Document) StorageRef() *StorageReference { return d.storageRef }
func (d *Document) ContentFields() map[string]string {
return map[string]string{
"body": "body", // semantic role → field name
}
}
func (d *Document) RawContent() map[string]string {
return map[string]string{"body": d.Body}
}
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder unmarshal JSON wire-format messages into a *BaseMessage, resolving payload type discriminators against the supplied registry. It is the only public path to a registry-aware BaseMessage. Mirrors stdlib json.Decoder / gob.Decoder.
dec := message.NewDecoder(deps.PayloadRegistry) msg, err := dec.Decode(data)
In hot paths (per-message handlers), construct the Decoder once at component init/start time and store it on the component to avoid per-message allocation:
c.decoder = message.NewDecoder(c.deps.PayloadRegistry) // ...later, per message: msg, err := c.decoder.Decode(data)
func NewDecoder ¶
func NewDecoder(reg *payloadregistry.Registry) *Decoder
NewDecoder constructs a Decoder bound to the supplied registry. reg must be non-nil; a Decoder built with nil fails fast on every Decode call so misconfiguration surfaces at the first message rather than silently using a stale registry.
type DefaultMeta ¶
type DefaultMeta struct {
// contains filtered or unexported fields
}
DefaultMeta provides the standard implementation of the Meta interface. It tracks when an event occurred, when it was received by the system, and where it originated from.
func NewDefaultMeta ¶
func NewDefaultMeta(createdAt time.Time, source string) *DefaultMeta
NewDefaultMeta creates a new DefaultMeta instance with the given creation time and source. The received time is automatically set to the current time.
func NewDefaultMetaWithReceivedAt ¶
func NewDefaultMetaWithReceivedAt(createdAt, receivedAt time.Time, source string) *DefaultMeta
NewDefaultMetaWithReceivedAt creates a new DefaultMeta instance with explicit creation and received times. This is useful for testing or when importing historical data.
func (*DefaultMeta) CreatedAt ¶
func (m *DefaultMeta) CreatedAt() time.Time
CreatedAt returns when the original event occurred.
func (*DefaultMeta) ReceivedAt ¶
func (m *DefaultMeta) ReceivedAt() time.Time
ReceivedAt returns when the system received the message.
func (*DefaultMeta) Source ¶
func (m *DefaultMeta) Source() string
Source returns the origin of the message.
type GenericJSONPayload ¶
type GenericJSONPayload struct {
// Data contains the JSON payload as a map of JSON-shaped values (see
// MarshalJSON for the values it accepts).
Data map[string]any `json:"data"`
}
GenericJSONPayload is the core.json.v1 payload: the fallback for JSON whose shape is not known when the code is written, such as outside input or the output of a transform the user configures. It is never the primary shape for data: code that builds a shape it knows registers a payload type for it instead. Decoding core.json.v1 needs this type registered explicitly, with RegisterPayloads, in the registry the Decoder uses.
Numbers depend on where the payload came from. A decoded payload holds every number as a json.Number carrying its exact literal (see UnmarshalJSON); an in-process payload holds whatever Go numbers its caller supplied, so a reader that accepts both handles json.Number and the Go numeric kinds.
Example usage:
payload := &GenericJSONPayload{
Data: map[string]any{
"sensor_id": "temp-001",
"temperature": 23.5,
"unit": "celsius",
},
}
func NewGenericJSON ¶
func NewGenericJSON(data map[string]any) *GenericJSONPayload
NewGenericJSON creates a new GenericJSON payload with the given data.
func (*GenericJSONPayload) MarshalJSON ¶
func (g *GenericJSONPayload) MarshalJSON() ([]byte, error)
MarshalJSON serializes the GenericJSON payload to JSON format. The output format matches the input structure with a "data" wrapper.
Data must be JSON-shaped: map[string]any, []any, string, a Go number (int, int8 to int64, uint, uint8 to uint64, uintptr, float32, float64), json.Number, bool or nil, nested to any depth. Any other value (a struct, time.Time, a pointer, a named type, a typed map or slice such as map[string]string or []string) is refused with an invalid-data error that names its type and its path, such as data.a[2].b; so is a string or map key that is not valid UTF-8, and a map or list that contains itself.
func (*GenericJSONPayload) RuleFields ¶
func (g *GenericJSONPayload) RuleFields() map[string]any
RuleFields implements RuleReadable: the whole data map IS this payload's declared projection.
core.json.v1 exists to carry arbitrary caller-supplied JSON with no schema of its own, so there is no narrower honest answer — the payload's author is the caller, and the caller already chose every key. It is also the behavior every rule written before RuleReadable existed depends on, which is why exposing Data verbatim keeps them all valid.
func (*GenericJSONPayload) Schema ¶
func (g *GenericJSONPayload) Schema() Type
Schema returns the payload type identifier for GenericJSON, which is always core.json.v1.
func (*GenericJSONPayload) UnmarshalJSON ¶
func (g *GenericJSONPayload) UnmarshalJSON(data []byte) error
UnmarshalJSON deserializes JSON data into the GenericJSON payload.
Every number in Data decodes as a json.Number holding its literal, never as a float64, so an accepted integer beyond 2^53 (an int64, a uint64 or a json.Number) keeps its exact value and re-encodes to the same text. A reader converts with json.Number's Int64, Float64 or String.
func (*GenericJSONPayload) Validate ¶
func (g *GenericJSONPayload) Validate() error
Validate refuses a nil Data, and runs the shape check MarshalJSON runs: it refuses, with the same invalid-data error naming the value's type and path, every value MarshalJSON refuses as not JSON-shaped, not valid UTF-8 or containing itself. Validate passing does not promise encoding passes: a NaN or infinite float32 or float64 passes Validate and is refused only by encoding. A json.Number's text must be valid UTF-8, which Validate checks; whether it is a number literal is left to encoding/json, which decides what MarshalJSON accepts. Decoded data never holds a NaN, an infinity or a json.Number that is not a literal.
type IndexingProfiler ¶
type IndexingProfiler interface {
// IndexingProfile returns one of "content" | "control" | "signal" | "trace".
IndexingProfile() string
}
IndexingProfiler lets an entity payload declare its indexing profile — the producer's hint for which semantic substrates (embedding, community clustering, search) should index the entity (ADR-054). It does NOT affect the structural graph, which is always queryable/traversable regardless of profile ("graph visibility is storage; indexing is policy").
Optional and additive: a payload that does not implement it falls through to its consumer's own resolution order. graph-ingest keeps a profile declared on the mutation envelope or through this interface, and otherwise applies the floor registered with the message type. Absence means "let the framework decide", never "no profile".
Valid return values are the four indexing profiles in the vocabulary package: IndexingProfileContent, IndexingProfileControl, IndexingProfileSignal, IndexingProfileTrace. An empty or unrecognized return is treated as absent (the consumer falls through to the floor).
type Message ¶
type Message interface {
// ID returns a unique identifier for this message instance.
// Typically a UUID, this ID is immutable and globally unique.
ID() string
// Type returns structured type information used for routing and processing.
// The type contains domain, category, and version information.
Type() Type
// Payload returns the message payload that may implement behavioral interfaces.
// Payloads expose capabilities like Storable, RuleReadable and IndexingProfiler.
Payload() Payload
// Meta returns metadata about the message lifecycle and origin.
// Includes creation time, receipt time, and source service information.
Meta() Meta
// Validate performs comprehensive validation of the message.
// Checks message type validity, payload presence, and payload-specific validation.
Validate() error
}
Message represents the core message interface for the SemEngine platform. Messages are the fundamental unit of data flow, carrying typed payloads with metadata through the semantic event mesh.
Design principles:
- Infrastructure-agnostic: Messages contain only data, no routing or storage logic
- Behavioral payloads: Payloads expose capabilities through -able interfaces
- Flexible metadata: Meta interface allows different metadata implementations
Example:
msg := NewBaseMessage(
Type{Domain: "sensors", Category: "gps", Version: "v1"},
gpsPayload,
"gps-reader-service"
)
type Meta ¶
type Meta interface {
// CreatedAt returns when the original event or observation occurred.
// For sensor data, this is the measurement time.
// For business events, this is when the event happened.
CreatedAt() time.Time
// ReceivedAt returns when the message entered the processing system.
// This helps track ingestion latency and message age.
// May be the same as CreatedAt for real-time streams.
ReceivedAt() time.Time
// Source returns the identifier of the message originator.
// Examples: "gps-reader-service", "sensor-12345", "order-processor"
// Used for debugging, tracing, and access control.
Source() string
}
Meta provides metadata about a message's lifecycle and origin. This interface enables tracking of when messages were created, when they entered the system, and where they originated.
Using an interface rather than a concrete type allows for:
- Custom metadata implementations for specific domains
- Extended metadata with additional fields when needed
- Easier testing with mock implementations
type Option ¶
type Option func(*BaseMessage)
Option is a functional option for configuring BaseMessage construction.
type Payload ¶
type Payload interface {
// Schema returns the Type that defines this payload's structure.
// This enables type-safe routing and processing throughout the system.
Schema() Type
// Validate checks the payload data for correctness.
// Returns nil if valid, or an error describing the validation failure.
// Should validate:
// - Required fields are present
// - Values are within acceptable ranges
// - Business rules are satisfied
Validate() error
// JSON serialization using standard Go interfaces.
// Payloads must implement json.Marshaler and json.Unmarshaler
// for deterministic serialization. The same payload must always
// produce the same JSON output.
json.Marshaler
json.Unmarshaler
}
Payload represents the data carried by a message. All message payloads must implement this interface to provide schema information, validation, and serialization capabilities.
Payloads may also implement behavioral interfaces (Storable, RuleReadable, IndexingProfiler) to expose additional capabilities that can be discovered and utilized at runtime.
Example implementation:
type GPSPayload struct {
DeviceID string `json:"device_id"`
Latitude float64 `json:"latitude"`
Longitude float64 `json:"longitude"`
Timestamp time.Time `json:"timestamp"`
}
func (p *GPSPayload) Schema() Type {
return Type{Domain: "sensors", Category: "gps", Version: "v1"}
}
func (p *GPSPayload) Validate() error {
if p.DeviceID == "" {
return errors.New("device ID is required")
}
if p.Latitude < -90 || p.Latitude > 90 {
return errors.New("invalid latitude")
}
if p.Longitude < -180 || p.Longitude > 180 {
return errors.New("invalid longitude")
}
return nil
}
func (p *GPSPayload) MarshalJSON() ([]byte, error) {
// Use alias to avoid infinite recursion
type Alias GPSPayload
return json.Marshal((*Alias)(p))
}
func (p *GPSPayload) UnmarshalJSON(data []byte) error {
// Use alias to avoid infinite recursion
type Alias GPSPayload
return json.Unmarshal(data, (*Alias)(p))
}
type RuleReadable ¶
type RuleReadable interface {
// RuleFields returns this payload's declared rule-readable fields.
// A nil or empty map is a valid answer meaning "nothing to match on" —
// distinct from not implementing the interface at all, which means the
// payload is unreadable and the engine reports it.
RuleFields() map[string]any
}
RuleReadable exposes the fields a payload is willing to have rule conditions and `$message.*` substitutions read.
Optional and discovered at runtime by type assertion, exactly like its sibling in behaviors.go. A payload that does not implement it is not rule-readable: the rule engine reads no fields from it, the rule does not fire, and the engine reports the rule/payload-type pairing rather than evaluating false in silence.
The PAYLOAD DECLARES its projection; the engine never derives one. There is deliberately no reflective default — reflection would make every payload rule-readable by accident and expose fields the author never intended a rule to match on, and it is the wrong trade in the evaluation hot path.
What to expose, and what not to:
- EXPOSE structural facts — identifiers, roles, outcomes, states, classifications, counts, routing addresses, timestamps.
- WITHHOLD LLM-authored and user content — prompts, message text, model output, tool-result bodies (ADR-036), and open caller-populated maps whose contents the payload's author does not control.
The reason is not privacy alone. A rule cannot make a quality judgement over unstructured text, and content in a rule payload is how a small model's context window silently truncates. A content-shaped decision belongs to a coordinator that emits a structured result which a later rule matches on deterministically.
Keys are dotted-path addressable: a nested map[string]any value is reachable as `$message.parent.child`.
Three sharp edges to know before you write a rule against a projection:
- ABSENCE IS THE FALSE CASE FOR BOOLEANS. A projection that mirrors `omitempty` omits a false bool entirely, so `stop_loop` is absent rather than present-and-false. Test for absence (or condition on the true case); a condition comparing it to `false` matches nothing, silently. The same applies to any zero-valued `omitempty` field.
- FIELD NAMES ARE NOT GLOBALLY UNIQUE. `$message.type` means the context event kind on ContextEvent, the control-signal verb on UserSignal, and the response kind on UserResponse. A rule subscribed to more than one subject must pin the payload with a second condition, because the engine resolves the path, not the type.
- PROJECTED VALUES FEED SUBJECT TEMPLATES. `$message.*` resolves in action subjects and reasons as well as in conditions, so anything exposed here can end up as a NATS subject token. Dotted values (`model`) and RFC3339 nano timestamps arrive intact — which also means an exposed field with a `.` or a space in it becomes a malformed subject. Prefer exposing tokens a subject can carry.
type Storable ¶
type Storable interface {
// EntityID returns deterministic 6-part ID: org.platform.system.domain.type.instance
EntityID() string
// Triples returns all facts about this entity
Triples() []Triple
// StorageRef returns reference to where full data is stored.
// May return nil if data is not stored externally.
StorageRef() *StorageReference
}
Storable is a payload that names its entity and its facts (EntityID and Triples) and carries a reference to where its full data is stored. Components that implement Storable can provide both semantic information and a reference to their full data.
This interface enables the lightweight message pattern where:
- Domain processors create messages with semantic data
- A storage component stores the full message and adds a StorageReference
- A graph consumer receives the Storable with both semantics and reference
- Consumers can access full data via StorageReference when needed
EntityID and Triples are also the methods of graph.Graphable, so a Storable satisfies that interface too.
Example implementation:
type StoredEntity struct {
entityID string
triples []Triple
storage *StorageReference
}
func (s *StoredEntity) EntityID() string { return s.entityID }
func (s *StoredEntity) Triples() []Triple { return s.triples }
func (s *StoredEntity) StorageRef() *StorageReference { return s.storage }
type StorageReference ¶
type StorageReference struct {
// StorageInstance identifies which storage component holds the data.
// This enables federation across multiple storage instances.
// Examples: "message-store", "cache-1", "objectstore-primary"
StorageInstance string `json:"storage_instance"`
// Key is the storage-specific key to retrieve the data.
// Format depends on the storage backend but typically includes
// time-based partitioning for efficient retrieval.
// Examples: "2025/01/13/14/msg_abc123", "robotics/drone/1/latest"
Key string `json:"key"`
// ContentType specifies the MIME type of the stored content.
// This helps consumers understand how to process the data.
// Examples: "application/json", "application/protobuf", "application/avro"
ContentType string `json:"content_type"`
// Size is an optional hint about the stored data size in bytes.
// This helps consumers decide whether to fetch the full data.
// A value of 0 indicates the size is unknown.
Size int64 `json:"size,omitempty"`
}
StorageReference points to where the full message data is stored. This enables lightweight message passing where components can reference stored data without transmitting the full payload.
The StorageReference pattern supports the "store once, reference everywhere" architecture, reducing data duplication and enabling efficient processing of large messages.
Example usage:
- A storage component stores the full message and returns a StorageReference
- A graph consumer receives a Storable with a reference to the full data
- Components can fetch full data only when needed
type Triple ¶
type Triple struct {
// Subject identifies the entity this triple describes.
// Must use EntityID.Key() format for consistency with federated entity management.
// Examples: "telemetry.robotics.drone.1", "gcs-alpha.robotics.drone.001"
Subject string `json:"subject"`
// Predicate identifies the semantic property using three-level dotted notation.
// Format: domain.category.property (e.g., "robotics.battery.level")
// This maintains consistency with the unified dotted notation from Alpha Week 1.
// Predicates should be defined in the vocabulary package for consistency.
Predicate string `json:"predicate"`
// Object contains the property value or entity reference.
// For literals: primitive types (float64, bool, string, int)
// For entity references: entity ID strings using EntityID.Key() format
// Complex objects should be flattened into multiple triples.
Object any `json:"object"`
// Source identifies where this assertion came from.
// Examples: "mavlink_heartbeat", "gps_fix", "operator_input", "ai_inference"
// Enables traceability and conflict resolution during entity merging.
Source string `json:"source"`
// Timestamp indicates when this assertion was made.
// Should typically be the message timestamp or processing time.
// Enables temporal queries and helps with entity state evolution.
Timestamp time.Time `json:"timestamp"`
// Confidence indicates the reliability of this assertion (0.0 to 1.0).
// Higher values indicate greater certainty about the triple.
//
// Typical confidence levels:
// - 1.0: Direct telemetry or explicit data
// - 0.9: High-confidence sensor readings
// - 0.7: Calculated or derived values
// - 0.5: Inferred relationships
// - 0.0: Uncertain or placeholder data
Confidence float64 `json:"confidence"`
// Context provides correlation ID for message batches or request tracking.
// This enables grouping related triples from the same processing batch
// or correlating triples across distributed systems.
// Examples: message ID, batch ID, correlation ID, request ID
Context string `json:"context,omitempty"`
// Datatype provides optional RDF datatype hint for the Object value.
// This helps with type interpretation and validation in downstream systems.
// Examples: "xsd:float", "xsd:dateTime", "xsd:boolean", "rdf:JSON"
// If omitted, the type is inferred from the Go type of Object.
//
// Only the "xsd:" and "rdf:" prefixes are expanded to full IRIs on export;
// an absolute IRI is passed through as written. Any OTHER prefix reaches
// the output as a relative reference, which is not valid RDF — so use one
// of those two prefixes, or write the IRI out in full. A prefix such as
// "geo:point" looks like a natural example and is exactly that trap.
Datatype string `json:"datatype,omitempty"`
// ExpiresAt indicates when this triple should be considered expired.
// When nil, the triple never expires and remains valid indefinitely.
// When set, the triple is considered expired after this timestamp.
// This enables TTL-based triple expiration for temporal data management.
ExpiresAt *time.Time `json:"expires_at,omitempty"`
}
Triple represents a semantic statement about an entity following the Subject-Predicate-Object pattern. This enables RDF-like knowledge graphs while maintaining simplicity for Go developers.
Triple design follows these principles:
- Subject: Always an entity ID using EntityID.Key() format (e.g., "telemetry.robotics.drone.1")
- Predicate: Semantic property using three-level dotted notation (e.g., "robotics.battery.level")
- Object: Typed value (literals) or entity references (other entity IDs)
- Metadata: Source, timestamp, and confidence for provenance tracking
Example triples from a drone heartbeat message:
- ("telemetry.robotics.drone.1", "robotics.battery.level", 85.5)
- ("telemetry.robotics.drone.1", "robotics.flight.armed", true)
- ("telemetry.robotics.drone.1", "robotics.component.has", "telemetry.robotics.battery.1")
This structure enables:
- NATS wildcard queries: "robotics.battery.*" finds all battery predicates
- Entity relationship modeling: Objects can reference other entities
- Temporal tracking: Each triple has timestamp and confidence
- Provenance: Source field tracks where each assertion came from
- Federation: Works with federated entity IDs from multiple sources
func DedupeAppendTriples ¶
DedupeAppendTriples returns the subset of incoming triples that are not already present in stored and not repeated earlier within incoming itself, preserving first-input order, plus the number suppressed.
Both the stored set and the within-request collapse are seeded into one hash set, so the cost is O(len(stored) + len(incoming)) rather than the O(n*k) a pairwise scan would cost. That matters precisely where duplicate suppression fires hardest: a hierarchy container entity accumulates one containment edge per child, so its stored triple count grows with the population it contains.
stored may be nil, which reduces this to a within-request collapse.
func (Triple) IsRelationship ¶
IsRelationship checks if this triple represents a relationship between entities rather than a property with a literal value. Returns true if Object is a valid canonical EntityID.
type Type ¶
Type provides structured type information for messages. It enables type-safe routing and processing by clearly identifying the domain, category, and version of each message. It is an alias: the definition lives in pkg/types, so payload code can say message.Type without importing that package.