message

package
v0.1.0-alpha.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 15 Imported by: 0

README

Message Package

Polymorphic message handling for SemEngine data flows with behavior-based processing and type-safe payload management.

Overview

The message package provides the foundation for type-safe, extensible message processing in SemEngine. Messages flow through components as polymorphic envelopes (BaseMessage) containing typed payloads that can be type-asserted to behavioral interfaces for specialized processing.

Key design principles:

  • Polymorphic by default: A Decoder deserializes messages to the concrete payload types its registry holds
  • Behavior-based processing: Type-assert to capabilities (RuleReadable, Storable, etc.) not concrete types
  • Registry-driven: A payload registry, created by the application and passed to message.NewDecoder, adds payload types without changing this package
  • Type-safe: Compile-time type checking with runtime polymorphism
  • Immutable: Messages are read-only after creation (safe for concurrent access)

Installation

import "github.com/c360studio/semengine/message"

Core Concepts

Message Structure

Every SemEngine message consists of:

  1. Envelope (BaseMessage): Polymorphic wrapper with type metadata
  2. Payload: Concrete data implementing the Payload interface
  3. Behavioral Interfaces: Optional capabilities for specialized processing
┌─ BaseMessage ────────────────────────────┐
│ Type: "core.json.v1"                     │
│ PayloadData: <raw JSON>                  │
│ ├─ payloadInstance ─────────────────┐    │
│ │  GenericJSONPayload               │    │
│ │  ├─ Data: map[string]any          │    │
│ │  ├─ Implements: Payload           │    │
│ │  └─ Optionally: RuleReadable,     │    │
│ │                 Storable, etc.    │    │
│ └───────────────────────────────────┘    │
└──────────────────────────────────────────┘
Type System

Payloads are identified by "domain.category.version" strings:

  • core.json.v1 - Generic JSON (built-in type, registered by RegisterPayloads)
  • robotics.mavlink.v1 - MAVLink messages (domain-specific)
  • iot.sensor.v2 - IoT sensor data (domain-specific)

In Go, a payload's Schema() returns message.Type, an alias of types.Type from pkg/types, so payload code does not import that package. Entity identifiers are types.EntityID; message.ParseEntityID returns one.

Components declare accepted types in their configuration, and the flow engine validates compatibility before deployment.

Behavioral Interfaces

Optional capabilities that payloads can implement:

Interface Purpose Methods Use Case
IndexingProfiler Indexing profile hint IndexingProfile() string Embedding and clustering eligibility
RuleReadable Fields rules may read RuleFields() map[string]any Rule conditions and substitutions
Storable Entity with a storage reference EntityID(), Triples(), StorageRef() Graph storage of large payloads

Processors type-assert to the specific behaviors they need, making components reusable across different payload types.

Quick Start

Creating Messages
// Create a GenericJSON payload
payload := message.NewGenericJSON(map[string]any{
    "sensor_id": "temp-001",
    "temperature": 23.5,
    "unit": "celsius",
})

// Wrap in BaseMessage: type, payload, source
msg := message.NewBaseMessage(payload.Schema(), payload, "temperature-monitor")

// Serialize to JSON
data, err := json.Marshal(msg)
// Result: {"id":"…","type":{"domain":"core","category":"json","version":"v1"},
//          "payload":{"data":{"sensor_id":"temp-001"…}},
//          "meta":{"created_at":…,"received_at":…,"source":"temperature-monitor"}}
Deserializing Messages
// A Decoder resolves the "type" field against the registry it was built
// with; there is no global registry. Build the registry once at startup.
reg := payloadregistry.New()
if err := message.RegisterPayloads(reg); err != nil { // core.json.v1
    return err
}
decoder := message.NewDecoder(reg)

msg, err := decoder.Decode(data)
if err != nil {
    return err // malformed JSON, unregistered type, or a payload that does not fit it
}

// json.Unmarshal into a zero-value message.BaseMessage always fails: it has
// no registry to resolve the payload type.

// Access the payload
payload := msg.Payload()
if genericJSON, ok := payload.(*message.GenericJSONPayload); ok {
    if temperature, ok := number(genericJSON.Data["temperature"]); ok {
        fmt.Printf("Temperature: %.1f°C\n", temperature)
    }
}

// number reads a JSON number from GenericJSON data. A decoded payload holds a json.Number (its exact
// literal, so no integer is rounded); a payload built in process holds the Go number it was given.
// A missing key or a value that is not a number reports false.
func number(v any) (float64, bool) {
    switch n := v.(type) {
    case json.Number:
        f, err := n.Float64()
        return f, err == nil
    case float64:
        return n, true
    case int:
        return float64(n), true
    case int64:
        return float64(n), true
    }
    return 0, false
}
Using Behavioral Interfaces
// Check if payload declares the fields rules may read
if readable, ok := msg.Payload().(message.RuleReadable); ok {
    fields := readable.RuleFields()
    fmt.Printf("Rule fields: %v\n", fields)
}

Architecture

Message Flow
┌─────────────┐     ┌────────────────┐     ┌─────────────┐     ┌─────────────┐
│   Input     │────▶│   Processor    │────▶│  Processor  │────▶│   Output    │
│ (UDP/File)  │     │  (Filter/Map)  │     │(Transform)  │     │(File/WS)    │
└─────────────┘     └────────────────┘     └─────────────┘     └─────────────┘
      │                     │                      │                    │
      ▼                     ▼                      ▼                    ▼
 BaseMessage          BaseMessage            BaseMessage          BaseMessage
  (Payload)            (Payload)              (Payload)            (Payload)

Each component:

  1. Receives BaseMessage from NATS
  2. Type-checks the payload type
  3. Type-asserts to required behavioral interfaces
  4. Processes the payload
  5. Creates new BaseMessage with transformed payload
  6. Publishes to next component
Payload Registry

There is no global registry. The application creates a payloadregistry.Registry (payloadregistry.New()), registers its payload types on it, and builds a message.Decoder from it. The registry maps type identifiers to factory functions:

Registry: {
  "core.json.v1" → func() any { return &GenericJSONPayload{} }
  "robotics.position.v1" → func() any { return &PositionPayload{} }
  "iot.sensor.v2" → func() any { return &SensorPayload{} }
}

When Decoder.Decode reads a message whose type is core.json.v1, it:

  1. Looks up core.json.v1 in the decoder's registry, and fails if it is not registered
  2. Calls the factory to create an empty payload instance
  3. Unmarshals the payload JSON into that instance, and fails if it does not fit
  4. Returns a *BaseMessage holding the typed payload

This enables polymorphic deserialization without reflection or code generation.

Integration Points

NATS: Messages serialize to JSON for NATS pub/sub:

  • Subjects: sensors.temperature, robots.position, etc.
  • Payload: Complete BaseMessage JSON

Component Registry: Components declare type compatibility:

  • input_types: ["core.json.v1", "iot.sensor.v2"]
  • output_types: ["core.json.v1"]

Flow Engine: Validates type compatibility in flow graphs before deployment

Metrics: Message counts tracked by type (via Schema().String())

Usage Patterns

Creating Custom Payload Types
package myapp

import (
    "time"
    "github.com/c360studio/semengine/component"
    "github.com/c360studio/semengine/message"
)

// Define your payload
type RobotPositionPayload struct {
    RobotID   string    `json:"robot_id"`
    Latitude  float64   `json:"latitude"`
    Longitude float64   `json:"longitude"`
    Timestamp time.Time `json:"timestamp"`
}

// Implement Payload interface
// PayloadType() is not part of the Payload interface
// Use Schema().String() to get the type identifier
// Schema() method is implemented above

func (p *RobotPositionPayload) Validate() error {
    // Structural validation: required fields
    if p.RobotID == "" {
        return errors.WrapInvalid(errors.ErrInvalidData, "RobotPositionPayload", "Validate", "robot_id is required")
    }

    // Semantic validation: value ranges
    if p.Latitude < -90 || p.Latitude > 90 {
        return errors.WrapInvalid(errors.ErrInvalidData, "RobotPositionPayload", "Validate",
            fmt.Sprintf("latitude must be between -90 and 90, got: %.6f", p.Latitude))
    }
    if p.Longitude < -180 || p.Longitude > 180 {
        return errors.WrapInvalid(errors.ErrInvalidData, "RobotPositionPayload", "Validate",
            fmt.Sprintf("longitude must be between -180 and 180, got: %.6f", p.Longitude))
    }
    return nil
}

// Implement behavioral interfaces
func (p *RobotPositionPayload) EntityID() string {
    return p.RobotID
}

func (p *RobotPositionPayload) EntityType() string {
    return "robot"
}

// RegisterPayloads registers the type with the application's registry, the
// pattern message.RegisterPayloads follows; the application calls it at startup
// on the registry it passes to message.NewDecoder.
func RegisterPayloads(reg *payloadregistry.Registry) error {
    return reg.Register(&payloadregistry.Registration{
        Domain:      "robotics",
        Category:    "position",
        Version:     "v1",
        Description: "Robot geographic position with timestamp",
        Factory: func() any {
            return &RobotPositionPayload{}
        },
        Example: RobotPositionPayload{
            RobotID:   "robot-001",
            Latitude:  40.7128,
            Longitude: -74.0060,
            Timestamp: time.Now(),
        },
    })
}
GenericJSON as the Fallback

GenericJSONPayload (core.json.v1) is the fallback for JSON whose shape is not known when the code is written: outside input, or a transform the user configures. Code that builds a shape it knows registers a payload type through the registry instead.

Data holds JSON-shaped values only: map[string]any, []any, string, Go numbers, json.Number, bool and nil, nested to any depth. Encoding refuses any other value (a struct, time.Time, a pointer, a typed map or slice such as map[string]string) with an error naming its type and path, such as data.a[2].b. It also refuses a string or key that is not valid UTF-8.

// Wrap a JSON object from outside whose shape the code does not know.
func wrapExternal(body []byte, source string) (*message.BaseMessage, error) {
    var data map[string]any
    d := json.NewDecoder(bytes.NewReader(body))
    d.UseNumber() // keep each number's exact value, as a decoded core.json.v1 payload does
    if err := d.Decode(&data); err != nil {
        return nil, err
    }
    payload := message.NewGenericJSON(data)
    if err := payload.Validate(); err != nil { // a JSON null body gives nil data
        return nil, err
    }
    return message.NewBaseMessage(payload.Schema(), payload, source), nil
}

// Read a field a user-configured rule names, such as metrics.cpu > limit. The message may come
// from wrapExternal in the same process or off the wire; number (above) reads either.
func cpuAbove(msg *message.BaseMessage, limit float64) bool {
    genericJSON, ok := msg.Payload().(*message.GenericJSONPayload)
    if !ok {
        return false
    }
    metrics, ok := genericJSON.Data["metrics"].(map[string]any)
    if !ok {
        return false
    }
    cpu, ok := number(metrics["cpu"])
    return ok && cpu > limit
}

When to use GenericJSON:

  • ✅ JSON from outside whose shape the code does not know
  • ✅ User-configured transforms (filter, map) over such JSON

When NOT to use GenericJSON:

  • ❌ Any shape the code builds from fields it knows: register a payload type
  • ❌ Type-safe domain models
  • ❌ Performance-critical paths (use typed payloads)

Testing

Testing Round-Trip Serialization
func TestMessageRoundTrip(t *testing.T) {
    // Create original message
    original := message.NewGenericJSON(map[string]any{
        "test": "value",
        "number": 42,
    })
    originalMsg := message.NewBaseMessage(original.Schema(), original, "test")

    // Marshal to JSON
    data, err := json.Marshal(originalMsg)
    require.NoError(t, err)

    // Decode back through a registry holding core.json.v1
    reg := payloadregistry.New()
    require.NoError(t, message.RegisterPayloads(reg))
    reconstructed, err := message.NewDecoder(reg).Decode(data)
    require.NoError(t, err)

    // Verify type and payload
    assert.Equal(t, "core.json.v1", reconstructed.Type().Key())
    payload, ok := reconstructed.Payload().(*message.GenericJSONPayload)
    require.True(t, ok, "payload is %T", reconstructed.Payload())
    assert.Equal(t, "value", payload.Data["test"])
    assert.Equal(t, json.Number("42"), payload.Data["number"])
}

Implementation Notes

Thread Safety

Messages are immutable after creation:

  • BaseMessage fields set once during construction or unmarshaling
  • Payload instances should be immutable or use copy-on-write
  • Safe for concurrent reads from multiple goroutines
  • NOT safe for concurrent modification

The payload registry is thread-safe:

  • payloadregistry.Registry uses internal synchronization
  • Safe to register types concurrently (at startup, before decoding)
  • Safe to create payloads, and to share one Decoder, across goroutines
Performance

Polymorphic deserialization overhead:

  • Registry lookup: O(1) map access (~10ns)
  • Type assertion: Zero-cost Go interface check
  • JSON unmarshaling: Standard library performance

For high-throughput scenarios (>10K msg/sec):

  • Use typed payloads (avoid map[string]any)
  • Pool BaseMessage instances if needed
  • Consider binary serialization (protobuf, msgpack)

Benchmarks (typical hardware):

BenchmarkBaseMessage_Marshal       1000000   1200 ns/op
BenchmarkBaseMessage_Unmarshal     800000    1500 ns/op
BenchmarkTypedPayload_Marshal      2000000    600 ns/op
BenchmarkTypedPayload_Unmarshal    1500000    900 ns/op
Error Handling

Uses the pkg/errs error classification:

import "github.com/c360studio/semengine/pkg/errs"

// Invalid input (malformed JSON, unknown type)
errors.WrapInvalid(err, "BaseMessage", "UnmarshalJSON", "validate type format")

// Programming errors (factory returns wrong type)
errors.WrapFatal(err, "BaseMessage", "UnmarshalJSON", "factory contract violation")

// General errors (JSON marshaling)
errors.Wrap(err, "BaseMessage", "MarshalJSON", "marshal payload")

Error classification enables retry logic and proper error handling in components.

Security Considerations

Input Validation:

  • Payload types validated against "domain.category.version" format
  • Unknown types rejected (must be registered)
  • Factory contract enforced (must return Payload instance)
  • Payload-specific validation via Validate() method

GenericJSON Limitations:

  • No built-in size limits (enforce at transport layer)
  • No depth limits (enforce at transport layer)
  • No schema validation (use typed payloads for strict schemas)

Best Practices:

  • Validate payloads in components before processing
  • Enforce message size limits at NATS/transport level
  • Use typed payloads for production systems
  • Implement Validate() thoroughly for custom payloads

Common Patterns

Enrichment Pattern
// Add metadata to existing message
func enrichMessage(msg *message.BaseMessage) (*message.BaseMessage, error) {
    // Extract original data
    original, ok := msg.Payload().(*message.GenericJSONPayload)
    if !ok {
        return nil, fmt.Errorf("enrich: payload is %T, want core.json.v1", msg.Payload())
    }

    // Create enriched payload
    enriched := message.NewGenericJSON(map[string]any{
        "original": original.Data,
        "enriched_at": time.Now().UTC().Format(time.RFC3339Nano), // time.Time is not JSON-shaped
        "enrichment_version": "v1",
    })

    return message.NewBaseMessage(enriched.Schema(), enriched, msg.Meta().Source()), nil
}
Filtering Pattern
// Filter messages based on behavioral capabilities
func filterStorableOnly(msgs []*message.BaseMessage) []*message.BaseMessage {
    var result []*message.BaseMessage
    for _, msg := range msgs {
        if _, ok := msg.Payload().(message.Storable); ok {
            result = append(result, msg)
        }
    }
    return result
}

Migration from Legacy Systems

If migrating from string-based message types:

// Old: Type in payload
type OldMessage struct {
    Type    string
    Payload json.RawMessage
}

// New: Use BaseMessage
// decoder is built from a registry holding the new payload types
func migrateOldMessage(decoder *message.Decoder, old *OldMessage) (*message.BaseMessage, error) {
    // Map old types to new type identifiers
    typeMap := map[string]message.Type{
        "temperature": {Domain: "iot", Category: "sensor", Version: "v1"},
        "position":    {Domain: "robotics", Category: "position", Version: "v1"},
    }

    newType, ok := typeMap[old.Type]
    if !ok {
        // Fall back to GenericJSON
        var data map[string]any
        if err := json.Unmarshal(old.Payload, &data); err != nil {
            return nil, err
        }
        payload := message.NewGenericJSON(data)
        return message.NewBaseMessage(payload.Schema(), payload, "migration"), nil
    }

    // Reconstruct as BaseMessage: the wire's type is an object, not a dotted string
    envelope, err := json.Marshal(map[string]any{"type": newType, "payload": old.Payload})
    if err != nil {
        return nil, err
    }
    return decoder.Decode(envelope)
}

Troubleshooting

"unregistered payload type" Error

Problem: Decoder.Decode fails with "unregistered payload type: X.Y.Z"

Solution: Register the payload type on the registry the decoder was built from, before decoding:

reg := payloadregistry.New()
if err := reg.Register(&payloadregistry.Registration{
    Domain:   "X",
    Category: "Y",
    Version:  "Z",
    Factory:  func() any { return &MyPayload{} },
}); err != nil {
    return err
}
decoder := message.NewDecoder(reg)
Type Assertion Fails

Problem: Type assertion returns ok=false

Debugging:

// Check actual type
payload := msg.Payload()
fmt.Printf("Payload type: %T\n", payload)
fmt.Printf("Payload type ID: %s\n", payload.Schema().String())

// Check behavioral interface
if _, ok := payload.(message.Storable); ok {
    fmt.Println("Payload IS Storable")
} else {
    fmt.Println("Payload is NOT Storable")
    // Check what interfaces it does implement
}
JSON Deserialization Fails

Problem: Decoder.Decode succeeds but payload fields are zero-valued

Cause: JSON field names don't match struct tags

Solution: Ensure struct tags match JSON field names:

// Wrong
type MyPayload struct {
    Value string // Missing json tag
}

// Correct
type MyPayload struct {
    Value string `json:"value"`
}

Examples

See *_test.go files for comprehensive examples:

  • base_message_test.go - Round-trip serialization, behavioral interfaces
  • generic_json_test.go - GenericJSON usage, nested structures
  • behaviors.go - Behavioral interface definitions with documentation

References

Contributing

When adding new behavioral interfaces:

  1. Keep interfaces small (1-3 methods)
  2. Document use cases clearly
  3. Provide example implementations
  4. Add tests demonstrating usage
  5. Update this README with the new interface

When adding new payload types:

  1. Implement Payload interface
  2. Implement relevant behavioral interfaces
  3. Add a RegisterPayloads(reg *payloadregistry.Registry) error function that registers it
  4. Add comprehensive tests
  5. Document type identifier format

Documentation

Overview

Package message provides the core message infrastructure for SemEngine. It defines interfaces and types for creating, validating, and processing messages that flow through the semantic event mesh.

Architecture

The package follows a clean, domain-agnostic design with three core concepts:

1. Messages - Containers that combine typed payloads with metadata 2. Payloads - Domain-specific data that may implement behavioral interfaces 3. Metadata - Information about message lifecycle and origin

Message Structure

Every message consists of:

  • A unique ID for tracking and deduplication
  • A structured Type (domain, category, version)
  • A Payload containing the actual data
  • Metadata about creation time, source, etc.
  • A content-based hash for integrity

Behavioral Interfaces

The message package uses runtime capability discovery through optional interfaces. Payloads implement only the interfaces relevant to their domain, and services discover these capabilities dynamically through type assertions.

## Entity Interface

Storable: Declares an entity, its facts, and where its full data is stored

  • EntityID() string - Returns federated entity identifier
  • Triples() []Triple - Returns semantic facts about the entity
  • StorageRef() *StorageReference - Returns where the full data is stored, or nil
  • Use when: Payload represents an entity whose facts and stored data a consumer needs
  • ContentStorable and BinaryStorable extend it for text and binary content

## Rule Interfaces

RuleReadable: Declares the fields rule conditions and `$message.*` substitutions may read from this payload

  • RuleFields() map[string]any - Returns the payload's declared projection
  • Use when: A rule should be able to condition or substitute on this payload. WITHOUT it the rule engine cannot read the payload at all: every `$message.*` condition against it is false forever, and the engine reports the rule/payload-type pairing once rather than failing silently.
  • Expose structural facts (ids, roles, outcomes, states, counts, routing); withhold LLM-authored and user content and open caller-populated maps. There is no reflective default, by design — a field reaches a rule only because its author exposed it.
  • Example: Loop outcome events, tool-call identity, approval decisions. GenericJSONPayload implements it by returning its whole Data map.

## Runtime Discovery Pattern

Services discover capabilities at runtime through type assertions:

// Check for entity data
if entity, ok := msg.Payload().(Storable); ok {
    entityID := entity.EntityID()
    triples := entity.Triples()
    ref := entity.StorageRef() // nil when the data is not stored externally
    // Record the facts, fetch the stored data through ref when needed, etc.
}

// Check for rule-readable fields
if readable, ok := msg.Payload().(RuleReadable); ok {
    fields := readable.RuleFields()
    // Evaluate conditions, resolve $message.* substitutions, etc.
}

This pattern enables services to process any message type without prior knowledge of the specific payload structure, discovering capabilities dynamically.

Type System Hierarchy

Two related but distinct identifiers appear in this package. Both are defined in pkg/types and produce dotted notation through a Key() method, for NATS routing, storage keys, and entity identification. Type is an alias for types.Type, kept so payload code can say message.Type. Entity identifiers are types.EntityID and are not aliased here.

## 1. Type (Message Schema) - "domain.category.version"

Purpose: Identifies message schemas for routing, validation, and evolution. Format: Type{Domain: "sensors", Category: "gps", Version: "v1"} -> "sensors.gps.v1"

Use When:

  • Defining message schemas in payload implementations (Schema() method)
  • Configuring component input/output types
  • Routing messages through NATS subjects
  • Versioning payload schemas for evolution

Example:

type TemperaturePayload struct { /* fields */ }
func (t *TemperaturePayload) Schema() Type {
    return Type{Domain: "sensors", Category: "temperature", Version: "v1"}
}
// Used for: NATS subject "sensors.temperature.v1"

## 2. EntityID (Federated Identity) - "org.platform.system.domain.type.instance"

Purpose: Provides globally unique entity identifiers across federated platforms. Format: EntityID with 6 parts -> "c360.platform1.mav1.robotics.drone.1"

Use When:

  • Multi-platform deployments need unique entity identity
  • Merging data from multiple sources with overlapping IDs
  • Federating entities across organizational boundaries
  • Building distributed knowledge graphs

Example:

entityID := types.EntityID{
    Org:      "c360",      // Organization namespace (platform.org)
    Platform: "platform1", // Minting deployment authority (platform.id)
    System:   "mav1",      // Source that produced the entity
    Domain:   "robotics",  // Delegated taxonomy
    Type:     "drone",     // Entity type within the domain
    Instance: "1",         // Leaf identifier
}
// Key(): "c360.platform1.mav1.robotics.drone.1"
// Used for: Federated entity resolution and deduplication

## Choosing the Right Type

Ask yourself:

  1. Defining a message schema? → Use Type (Schema() method)
  2. Providing entity identity? → Use types.EntityID (Storable.EntityID() method)

Most payloads only need Type (for Schema()). Only implement Storable if your payload represents an entity whose facts and stored data a consumer needs.

Usage Example

// Define a payload type
type TemperaturePayload struct {
    SensorID    string    `json:"sensor_id"`
    Temperature float64   `json:"temperature"`
    Unit        string    `json:"unit"`
    Timestamp   time.Time `json:"timestamp"`
}

// Implement required Payload interface
func (t *TemperaturePayload) Schema() Type {
    return Type{
        Domain:   "sensors",
        Category: "temperature",
        Version:  "v1",
    }
}

func (t *TemperaturePayload) Validate() error {
    if t.SensorID == "" {
        return errors.New("sensor ID required")
    }
    return nil
}

func (t *TemperaturePayload) MarshalJSON() ([]byte, error) {
    // Use alias to avoid infinite recursion
    type Alias TemperaturePayload
    return json.Marshal((*Alias)(t))
}

func (t *TemperaturePayload) UnmarshalJSON(data []byte) error {
    // Use alias to avoid infinite recursion
    type Alias TemperaturePayload
    return json.Unmarshal(data, (*Alias)(t))
}

// Implement an optional behavioral interface (RuleReadable)
func (t *TemperaturePayload) RuleFields() map[string]any {
    return map[string]any{"sensor_id": t.SensorID, "temperature": t.Temperature}
}

// Register the type so a Decoder can rebuild it from the wire
reg := payloadregistry.New()
if err := reg.Register(&payloadregistry.Registration{
    Domain:      "sensors",
    Category:    "temperature",
    Version:     "v1",
    Description: "Temperature reading",
    Factory:     func() any { return &TemperaturePayload{} },
}); err != nil {
    return err
}

// Create and use a message
payload := &TemperaturePayload{
    SensorID:    "temp-001",
    Temperature: 22.5,
    Unit:        "celsius",
    Timestamp:   time.Now(),
}

msg := NewBaseMessage(
    payload.Schema(),
    payload,
    "temperature-monitor",
)

// After NewDecoder(reg).Decode, a service reads the payload by its
// concrete type, or by an interface it implements
if reading, ok := msg.Payload().(*TemperaturePayload); ok {
    fmt.Printf("%s: %.1f %s\n", reading.SensorID, reading.Temperature, reading.Unit)
}
if readable, ok := msg.Payload().(RuleReadable); ok {
    fields := readable.RuleFields()
    fmt.Printf("rule fields: %v\n", fields)
}

Message Lifecycle

Messages follow a predictable lifecycle from creation through processing:

## 1. Creation

Messages are created using NewBaseMessage with optional configuration:

// Simple message with current timestamp
msg := NewBaseMessage(payload.Schema(), payload, "my-service")

// Historical data with specific timestamp
msg := NewBaseMessage(payload.Schema(), payload, "my-service",
    WithTime(historicalTime))

Messages are immutable after creation - all fields are set during construction and cannot be modified. This ensures message integrity throughout processing.

## 2. Validation

Validation happens at two levels:

Structural Validation (Required):

  • Message type must be valid (domain, category, version all present)
  • Payload must not be nil
  • Metadata must not be nil

Payload Validation (Domain-specific):

  • Implemented by Payload.Validate() method
  • Checks domain-specific business rules
  • May be structural (required fields) or semantic (value ranges)

Example validation:

if err := msg.Validate(); err != nil {
    // Handle validation error
    // Check error type: Fatal, Transient, or Invalid
}

## 3. Serialization

Messages are serialized to JSON for transmission over NATS:

// Serialize message
data, err := json.Marshal(msg)

// Deserialize message: the Decoder resolves the payload type against
// the registry it was built with
reg := payloadregistry.New()
if err := RegisterPayloads(reg); err != nil {
    return err
}
decoded, err := NewDecoder(reg).Decode(data)
if err != nil {
    return err
}

Wire format preserves:

  • Message ID for deduplication and tracking
  • Type information for routing and schema validation
  • Payload data using Payload.MarshalJSON()
  • Metadata timestamps (millisecond precision) and source

Note: there is no global registry. Deserialization goes through a Decoder built with a payloadregistry.Registry that holds the message's payload type; decoding a zero-value BaseMessage with json.Unmarshal always fails, because it has no registry. For generic JSON processing, register the well-known type "core.json.v1" (GenericJSONPayload) with RegisterPayloads.

GenericJSONPayload is the fallback for JSON whose shape is not known when the code is written: outside input, or a transform the user configures. Code that builds a shape it knows registers a payload type instead. Its Data holds JSON-shaped values only (map[string]any, []any, string, Go numbers, json.Number, bool and nil); encoding refuses any other value with an error naming its type and path, and any string or key that is not valid UTF-8.

## 4. Transmission

Messages are published to NATS subjects derived from their type:

subject := msg.Type().Key()  // e.g., "sensors.temperature.v1"
nc.Publish(subject, data)

This enables:

  • Type-based routing using NATS wildcards
  • Version-specific processing
  • Domain isolation and security policies

## 5. Processing

Services receive and process messages based on their subscriptions:

// Subscribe to specific type
nc.Subscribe("sensors.temperature.v1", handler)

// Subscribe to all versions
nc.Subscribe("sensors.temperature.*", handler)

// Subscribe to entire domain
nc.Subscribe("sensors.>", handler)

Services discover payload capabilities at runtime:

// decoder is built once at startup: NewDecoder(reg)
func handler(m *nats.Msg) {
    msg, err := decoder.Decode(m.Data)
    if err != nil {
        return // malformed, or a payload type reg does not hold
    }

    // Discover capabilities
    if entity, ok := msg.Payload().(Storable); ok {
        // Process entity data: entity.EntityID(), entity.Triples()
    }
}

Best Practices

## Payload Implementation

1. Implement Required Methods

  • Schema() Type - Return structured type information
  • Validate() error - Perform domain-specific validation
  • MarshalJSON() ([]byte, error) - Use alias pattern to avoid recursion
  • UnmarshalJSON([]byte) error - Use alias pattern to avoid recursion

2. Implement Optional Interfaces Thoughtfully

  • Only implement behavioral interfaces that make semantic sense
  • Consider whether your payload truly represents an entity before implementing Storable

3. Validation Philosophy

  • Structural validation: Check required fields and basic types
  • Semantic validation: Check business rules and value ranges (optional)
  • Fail fast: Return first validation error encountered
  • Provide clear error messages with context

4. Use Type Constants

  • Define message types as package constants

  • Don't inline type construction at call sites

  • Example:

    // Good: Type as constant var TemperatureType = message.Type{ Domain: "sensors", Category: "temperature", Version: "v1", }

    // Bad: Inline type construction msg := NewBaseMessage( message.Type{Domain: "sensors", Category: "temperature", Version: "v1"}, payload, source)

## Message Creation

1. Use Functional Options for Configuration

  • Start with simple NewBaseMessage(type, payload, source)
  • Add options only when needed: WithTime(), WithMeta()
  • Don't create custom constructors - use options instead

2. Set Source Meaningfully

  • Use service name or component identifier
  • Be consistent across your application
  • Example: "temperature-monitor", "gps-processor", "user-service"

3. Timestamp Considerations

  • Default timestamp (time.Now()) is correct for most cases
  • Use WithTime() only for historical data or testing

## Message Processing

1. Type Assertion Pattern

  • Use comma-ok idiom for capability discovery

  • Don't assume interfaces are implemented

  • Example:

    if entity, ok := msg.Payload().(Storable); ok { // Safe to use entity's methods } else { // Payload doesn't implement Storable, skip or handle differently }

2. Error Handling

  • Always check validation errors before processing
  • Use errors.As() to detect error types (Fatal, Transient, Invalid)
  • Don't rely on error message strings for logic

3. Immutability

  • Don't modify message fields after creation
  • Create new messages for transformations
  • Payload data can be read but not written

## Testing

1. Use Real Test Data

  • Create actual payload instances, don't use mocks
  • Test with both valid and invalid data
  • Test serialization round-trips

2. Test Behavioral Interfaces

  • Verify interface implementations return correct values
  • Test type assertions work as expected
  • Ensure optional interfaces are truly optional

3. Test Validation

  • Test both valid and invalid payloads
  • Verify error messages are clear
  • Test edge cases and boundary conditions

Design Philosophy

The messages package embodies several key design principles:

1. Separation of Concerns: Messages know nothing about routing, storage, or entity extraction. They are pure data containers.

2. Discoverability: Behavioral interfaces allow services to discover and utilize message capabilities at runtime without tight coupling.

3. Domain Agnosticism: The core package contains no domain-specific code. All domain logic lives in domain packages that use these interfaces.

4. Extensibility: New behavioral interfaces can be added without breaking existing code. Payloads can implement any combination of interfaces.

5. Type Safety: Structured Type provides compile-time safety while maintaining flexibility.

Package message - the RuleReadable behavioral interface.

A sibling of the optional behavior in behaviors.go (IndexingProfiler), discovered the same way and optional in the same sense.

Package message provides core message infrastructure for SemEngine. See doc.go for complete package documentation.

Index

Constants

View Source
const (
	// ContentRoleBody is the primary text content (full document text).
	// Embedding workers prioritize this role for text extraction.
	ContentRoleBody = "body"

	// ContentRoleAbstract is a brief summary or description.
	// Used when body is not available or for additional context.
	ContentRoleAbstract = "abstract"

	// ContentRoleTitle is the document or entity title.
	// Typically included in embeddings for context.
	ContentRoleTitle = "title"

	// ContentRoleMedia is the primary media content (video, image, audio).
	// Used for binary content that may be processed by multimodal embedders.
	ContentRoleMedia = "media"

	// ContentRoleThumbnail is a preview image for media content.
	ContentRoleThumbnail = "thumbnail"
)

ContentRole constants define standard semantic roles for ContentFields. Using these constants ensures consistency across implementations.

View Source
const EntityReferenceDatatype = "@id"

EntityReferenceDatatype marks a triple object as an explicit entity reference. Complete EntityState validation requires marked objects to be canonical entity-ID strings; unmarked canonical-ID-shaped strings retain their existing structural relationship behavior.

Variables

This section is empty.

Functions

func AppendIdentityKey

func AppendIdentityKey(t Triple) string

AppendIdentityKey returns the canonical six-field add-lane identity key for a triple. Two triples denote the same assertion for append purposes if and only if their keys are equal; SameAppendTuple is defined in exactly those terms so a key-based set (O(n+k)) and a pairwise comparison can never disagree.

The key is LENGTH-PREFIXED, not delimiter-joined. Predicates, sources, and contexts are arbitrary operator- and producer-supplied strings; joining them on a dot or a pipe would make ("a.b", "c") and ("a", "b.c") the same key — the raw-key-collision class that causes silent data loss elsewhere in the system. Length prefixes make the encoding injective for every possible field content, including embedded NUL bytes and embedded digits.

func IsValidEntityID

func IsValidEntityID(s string) bool

IsValidEntityID checks if a string conforms to the canonical 6-part EntityID format. Valid format: organization.platform.system.domain.type.instance (e.g., "c360.platform1.mav1.robotics.drone.0") Each part must contain only alphanumeric characters, hyphens, or underscores. This prevents false positives from strings that happen to have 5 dots (e.g., JSON with file extensions like ".go", ".md") and ensures the ID is safe as a NATS KV key.

func ParseEntityID

func ParseEntityID(value string) (types.EntityID, error)

ParseEntityID delegates canonical parsing to pkg/types.

func RegisterPayloads

func RegisterPayloads(reg *payloadregistry.Registry) error

RegisterPayloads registers the GenericJSON payload type (core.json.v1) with the supplied registry. Nothing registers it implicitly: a process that decodes core.json.v1 calls this on the registry its Decoder uses.

func SameAppendTuple

func SameAppendTuple(left, right Triple) bool

SameAppendTuple reports whether two triples are the same assertion on the add lane. It is exactly key equality — see AppendIdentityKey.

Types

type BaseMessage

type BaseMessage struct {
	// contains filtered or unexported fields
}

BaseMessage provides the standard implementation of the Message interface. It combines a typed payload with metadata to create a complete message ready for transmission through the semantic event mesh.

BaseMessage is immutable after creation - all fields are set during construction and cannot be modified. This ensures message integrity throughout the processing pipeline.

Construction using Functional Options:

NewBaseMessage uses the functional options pattern for clean, composable configuration:

// Simple message (most common)
msg := NewBaseMessage(msgType, payload, "my-service")

// With specific timestamp (testing/historical data)
msg := NewBaseMessage(msgType, payload, "my-service", WithTime(pastTime))

func NewBaseMessage

func NewBaseMessage(msgType Type, payload Payload, source string, opts ...Option) *BaseMessage

NewBaseMessage creates a new BaseMessage with optional configuration.

Parameters:

  • msgType: Structured type information (domain, category, version)
  • payload: The message payload implementing the Payload interface
  • source: Identifier of the service or component creating this message
  • opts: Optional configuration functions

Examples:

// Simple message with current timestamp
msg := NewBaseMessage(msgType, payload, "my-service")

// Message with specific timestamp (for historical data)
msg := NewBaseMessage(msgType, payload, "my-service", WithTime(pastTime))

func (*BaseMessage) ID

func (m *BaseMessage) ID() string

ID returns the unique message identifier.

func (*BaseMessage) MarshalJSON

func (m *BaseMessage) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler for BaseMessage. This allows BaseMessage to be serialized to JSON even though its fields are private.

Validation is performed before serialization to ensure invalid messages cannot be serialized and published to the message bus. A meta source or a type component that is not valid UTF-8 is refused with an invalid-data error rather than written with its invalid bytes replaced.

func (*BaseMessage) Meta

func (m *BaseMessage) Meta() Meta

Meta returns the message metadata.

func (*BaseMessage) Payload

func (m *BaseMessage) Payload() Payload

Payload returns the message payload.

func (*BaseMessage) Type

func (m *BaseMessage) Type() Type

Type returns the structured message type.

func (*BaseMessage) UnmarshalJSON

func (m *BaseMessage) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler for BaseMessage.

Resolves the payload type discriminator against m.registry, which must be set by the Decoder before json.Unmarshal is invoked. Callers, tests included, MUST go through message.NewDecoder(reg).Decode(data). Zero-value BaseMessage is a hard error — there is no global fallback and no test-mode side door. Mirrors stdlib's bufio.Scanner state-set-by-constructor pattern.

func (*BaseMessage) Validate

func (m *BaseMessage) Validate() error

Validate performs comprehensive message validation.

type BinaryContent

type BinaryContent struct {
	ContentType string // MIME type: "image/jpeg", "video/mp4", "application/pdf"
	Data        []byte // The actual binary data
}

BinaryContent describes a binary field for storage. Used by BinaryStorable implementations to provide binary data along with its content type for proper handling.

type BinaryStorable

type BinaryStorable interface {
	ContentStorable

	// BinaryFields returns binary content to store in ObjectStore.
	//
	// Each field is stored as a separate binary blob with its own key.
	// The returned map keys are field names that can be referenced in
	// ContentFields for semantic role mapping.
	//
	// Example return value:
	//   {"video": {ContentType: "video/mp4", Data: videoBytes},
	//    "thumbnail": {ContentType: "image/jpeg", Data: thumbBytes}}
	BinaryFields() map[string]BinaryContent
}

BinaryStorable extends ContentStorable for payloads with binary content.

This interface enables storage of images, videos, PDFs, and other binary files alongside text content. Binary data is stored directly in ObjectStore (no base64 encoding) with JSON metadata referencing the binary keys.

Example implementation:

type VideoDocument struct {
    Title       string
    Description string
    VideoData   []byte
    Thumbnail   []byte
}

func (v *VideoDocument) RawContent() map[string]string {
    return map[string]string{
        "title":       v.Title,
        "description": v.Description,
    }
}

func (v *VideoDocument) BinaryFields() map[string]BinaryContent {
    return map[string]BinaryContent{
        "video":     {ContentType: "video/mp4", Data: v.VideoData},
        "thumbnail": {ContentType: "image/jpeg", Data: v.Thumbnail},
    }
}

type ContentStorable

type ContentStorable interface {
	Storable // EntityID() + Triples() + StorageRef()

	// ContentFields returns semantic role → field name mapping.
	//
	// This tells consumers how to find content in the stored data without
	// hardcoding field names. Keys are semantic roles understood by consumers
	// (like embedding workers), values are field names in RawContent().
	//
	// Standard semantic roles:
	//   - "body":     Primary text content (full document text)
	//   - "abstract": Brief summary or description
	//   - "title":    Document title
	//
	// Example return value:
	//   {"body": "content", "abstract": "description", "title": "title"}
	//
	// The map should only include roles that have non-empty content.
	ContentFields() map[string]string

	// RawContent returns the content to store in ObjectStore.
	//
	// Field names in this map should match values in ContentFields().
	// objectstore.Store.StoreContent stores the body field's text as raw
	// bytes and refuses any other non-empty field (see the type's doc), so a
	// consumer that opens the StorageRef reads the body and nothing else.
	//
	// Example return value:
	//   {"body": "Full document text..."}
	RawContent() map[string]string
}

ContentStorable extends Storable for payloads with large content fields.

This interface enables the "process → store → graph" pattern where:

  1. Processor creates semantic understanding (triples with metadata only)
  2. ObjectStore stores raw content and returns StorageReference
  3. GraphProcessor receives ContentStorable with both semantics and content reference
  4. EmbeddingWorker uses ContentFields to find text for embedding

The key insight is that ContentStorable separates metadata (in triples) from content (in ObjectStore), while providing a semantic map of content structure via ContentFields. This avoids bloating triples with large text and enables efficient embedding extraction without hardcoded field name coupling.

What the store keeps: objectstore.Store.StoreContent stores only the text of the "body" role, as raw bytes, and the StorageReference it returns opens to exactly that text. A payload with any other non-empty content (text under a field another role names, an undeclared field, or a non-empty binary field of a BinaryStorable) is refused with an invalid-input error before anything is written. Text that is not the body, such as a title or an abstract, belongs in the graph as triples, which the producer emits from Triples().

Example implementation (a document whose only stored content is its body; the title and description reach the graph as triples):

type Document struct {
    Title       string
    Description string
    Body        string
    storageRef  *StorageReference
}

func (d *Document) EntityID() string { return d.entityID }
func (d *Document) Triples() []Triple { return metadataTriples } // NO body
func (d *Document) StorageRef() *StorageReference { return d.storageRef }

func (d *Document) ContentFields() map[string]string {
    return map[string]string{
        "body": "body", // semantic role → field name
    }
}

func (d *Document) RawContent() map[string]string {
    return map[string]string{"body": d.Body}
}

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder unmarshal JSON wire-format messages into a *BaseMessage, resolving payload type discriminators against the supplied registry. It is the only public path to a registry-aware BaseMessage. Mirrors stdlib json.Decoder / gob.Decoder.

dec := message.NewDecoder(deps.PayloadRegistry)
msg, err := dec.Decode(data)

In hot paths (per-message handlers), construct the Decoder once at component init/start time and store it on the component to avoid per-message allocation:

c.decoder = message.NewDecoder(c.deps.PayloadRegistry)
// ...later, per message:
msg, err := c.decoder.Decode(data)

func NewDecoder

func NewDecoder(reg *payloadregistry.Registry) *Decoder

NewDecoder constructs a Decoder bound to the supplied registry. reg must be non-nil; a Decoder built with nil fails fast on every Decode call so misconfiguration surfaces at the first message rather than silently using a stale registry.

func (*Decoder) Decode

func (d *Decoder) Decode(data []byte) (*BaseMessage, error)

Decode unmarshal data into a fresh *BaseMessage bound to this Decoder's registry.

type DefaultMeta

type DefaultMeta struct {
	// contains filtered or unexported fields
}

DefaultMeta provides the standard implementation of the Meta interface. It tracks when an event occurred, when it was received by the system, and where it originated from.

func NewDefaultMeta

func NewDefaultMeta(createdAt time.Time, source string) *DefaultMeta

NewDefaultMeta creates a new DefaultMeta instance with the given creation time and source. The received time is automatically set to the current time.

func NewDefaultMetaWithReceivedAt

func NewDefaultMetaWithReceivedAt(createdAt, receivedAt time.Time, source string) *DefaultMeta

NewDefaultMetaWithReceivedAt creates a new DefaultMeta instance with explicit creation and received times. This is useful for testing or when importing historical data.

func (*DefaultMeta) CreatedAt

func (m *DefaultMeta) CreatedAt() time.Time

CreatedAt returns when the original event occurred.

func (*DefaultMeta) ReceivedAt

func (m *DefaultMeta) ReceivedAt() time.Time

ReceivedAt returns when the system received the message.

func (*DefaultMeta) Source

func (m *DefaultMeta) Source() string

Source returns the origin of the message.

type GenericJSONPayload

type GenericJSONPayload struct {
	// Data contains the JSON payload as a map of JSON-shaped values (see
	// MarshalJSON for the values it accepts).
	Data map[string]any `json:"data"`
}

GenericJSONPayload is the core.json.v1 payload: the fallback for JSON whose shape is not known when the code is written, such as outside input or the output of a transform the user configures. It is never the primary shape for data: code that builds a shape it knows registers a payload type for it instead. Decoding core.json.v1 needs this type registered explicitly, with RegisterPayloads, in the registry the Decoder uses.

Numbers depend on where the payload came from. A decoded payload holds every number as a json.Number carrying its exact literal (see UnmarshalJSON); an in-process payload holds whatever Go numbers its caller supplied, so a reader that accepts both handles json.Number and the Go numeric kinds.

Example usage:

payload := &GenericJSONPayload{
    Data: map[string]any{
        "sensor_id": "temp-001",
        "temperature": 23.5,
        "unit": "celsius",
    },
}

func NewGenericJSON

func NewGenericJSON(data map[string]any) *GenericJSONPayload

NewGenericJSON creates a new GenericJSON payload with the given data.

func (*GenericJSONPayload) MarshalJSON

func (g *GenericJSONPayload) MarshalJSON() ([]byte, error)

MarshalJSON serializes the GenericJSON payload to JSON format. The output format matches the input structure with a "data" wrapper.

Data must be JSON-shaped: map[string]any, []any, string, a Go number (int, int8 to int64, uint, uint8 to uint64, uintptr, float32, float64), json.Number, bool or nil, nested to any depth. Any other value (a struct, time.Time, a pointer, a named type, a typed map or slice such as map[string]string or []string) is refused with an invalid-data error that names its type and its path, such as data.a[2].b; so is a string or map key that is not valid UTF-8, and a map or list that contains itself.

func (*GenericJSONPayload) RuleFields

func (g *GenericJSONPayload) RuleFields() map[string]any

RuleFields implements RuleReadable: the whole data map IS this payload's declared projection.

core.json.v1 exists to carry arbitrary caller-supplied JSON with no schema of its own, so there is no narrower honest answer — the payload's author is the caller, and the caller already chose every key. It is also the behavior every rule written before RuleReadable existed depends on, which is why exposing Data verbatim keeps them all valid.

func (*GenericJSONPayload) Schema

func (g *GenericJSONPayload) Schema() Type

Schema returns the payload type identifier for GenericJSON, which is always core.json.v1.

func (*GenericJSONPayload) UnmarshalJSON

func (g *GenericJSONPayload) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes JSON data into the GenericJSON payload.

Every number in Data decodes as a json.Number holding its literal, never as a float64, so an accepted integer beyond 2^53 (an int64, a uint64 or a json.Number) keeps its exact value and re-encodes to the same text. A reader converts with json.Number's Int64, Float64 or String.

func (*GenericJSONPayload) Validate

func (g *GenericJSONPayload) Validate() error

Validate refuses a nil Data, and runs the shape check MarshalJSON runs: it refuses, with the same invalid-data error naming the value's type and path, every value MarshalJSON refuses as not JSON-shaped, not valid UTF-8 or containing itself. Validate passing does not promise encoding passes: a NaN or infinite float32 or float64 passes Validate and is refused only by encoding. A json.Number's text must be valid UTF-8, which Validate checks; whether it is a number literal is left to encoding/json, which decides what MarshalJSON accepts. Decoded data never holds a NaN, an infinity or a json.Number that is not a literal.

type IndexingProfiler

type IndexingProfiler interface {
	// IndexingProfile returns one of "content" | "control" | "signal" | "trace".
	IndexingProfile() string
}

IndexingProfiler lets an entity payload declare its indexing profile — the producer's hint for which semantic substrates (embedding, community clustering, search) should index the entity (ADR-054). It does NOT affect the structural graph, which is always queryable/traversable regardless of profile ("graph visibility is storage; indexing is policy").

Optional and additive: a payload that does not implement it falls through to its consumer's own resolution order. graph-ingest keeps a profile declared on the mutation envelope or through this interface, and otherwise applies the floor registered with the message type. Absence means "let the framework decide", never "no profile".

Valid return values are the four indexing profiles in the vocabulary package: IndexingProfileContent, IndexingProfileControl, IndexingProfileSignal, IndexingProfileTrace. An empty or unrecognized return is treated as absent (the consumer falls through to the floor).

type Message

type Message interface {
	// ID returns a unique identifier for this message instance.
	// Typically a UUID, this ID is immutable and globally unique.
	ID() string

	// Type returns structured type information used for routing and processing.
	// The type contains domain, category, and version information.
	Type() Type

	// Payload returns the message payload that may implement behavioral interfaces.
	// Payloads expose capabilities like Storable, RuleReadable and IndexingProfiler.
	Payload() Payload

	// Meta returns metadata about the message lifecycle and origin.
	// Includes creation time, receipt time, and source service information.
	Meta() Meta

	// Validate performs comprehensive validation of the message.
	// Checks message type validity, payload presence, and payload-specific validation.
	Validate() error
}

Message represents the core message interface for the SemEngine platform. Messages are the fundamental unit of data flow, carrying typed payloads with metadata through the semantic event mesh.

Design principles:

  • Infrastructure-agnostic: Messages contain only data, no routing or storage logic
  • Behavioral payloads: Payloads expose capabilities through -able interfaces
  • Flexible metadata: Meta interface allows different metadata implementations

Example:

msg := NewBaseMessage(
    Type{Domain: "sensors", Category: "gps", Version: "v1"},
    gpsPayload,
    "gps-reader-service"
)

type Meta

type Meta interface {
	// CreatedAt returns when the original event or observation occurred.
	// For sensor data, this is the measurement time.
	// For business events, this is when the event happened.
	CreatedAt() time.Time

	// ReceivedAt returns when the message entered the processing system.
	// This helps track ingestion latency and message age.
	// May be the same as CreatedAt for real-time streams.
	ReceivedAt() time.Time

	// Source returns the identifier of the message originator.
	// Examples: "gps-reader-service", "sensor-12345", "order-processor"
	// Used for debugging, tracing, and access control.
	Source() string
}

Meta provides metadata about a message's lifecycle and origin. This interface enables tracking of when messages were created, when they entered the system, and where they originated.

Using an interface rather than a concrete type allows for:

  • Custom metadata implementations for specific domains
  • Extended metadata with additional fields when needed
  • Easier testing with mock implementations

type Option

type Option func(*BaseMessage)

Option is a functional option for configuring BaseMessage construction.

func WithMeta

func WithMeta(meta Meta) Option

WithMeta replaces the default metadata with a custom Meta implementation.

func WithTime

func WithTime(createdAt time.Time) Option

WithTime sets a specific creation timestamp instead of using time.Now(). Useful for historical data import or testing.

type Payload

type Payload interface {
	// Schema returns the Type that defines this payload's structure.
	// This enables type-safe routing and processing throughout the system.
	Schema() Type

	// Validate checks the payload data for correctness.
	// Returns nil if valid, or an error describing the validation failure.
	// Should validate:
	//   - Required fields are present
	//   - Values are within acceptable ranges
	//   - Business rules are satisfied
	Validate() error

	// JSON serialization using standard Go interfaces.
	// Payloads must implement json.Marshaler and json.Unmarshaler
	// for deterministic serialization. The same payload must always
	// produce the same JSON output.
	json.Marshaler
	json.Unmarshaler
}

Payload represents the data carried by a message. All message payloads must implement this interface to provide schema information, validation, and serialization capabilities.

Payloads may also implement behavioral interfaces (Storable, RuleReadable, IndexingProfiler) to expose additional capabilities that can be discovered and utilized at runtime.

Example implementation:

type GPSPayload struct {
    DeviceID  string    `json:"device_id"`
    Latitude  float64   `json:"latitude"`
    Longitude float64   `json:"longitude"`
    Timestamp time.Time `json:"timestamp"`
}

func (p *GPSPayload) Schema() Type {
    return Type{Domain: "sensors", Category: "gps", Version: "v1"}
}

func (p *GPSPayload) Validate() error {
    if p.DeviceID == "" {
        return errors.New("device ID is required")
    }
    if p.Latitude < -90 || p.Latitude > 90 {
        return errors.New("invalid latitude")
    }
    if p.Longitude < -180 || p.Longitude > 180 {
        return errors.New("invalid longitude")
    }
    return nil
}

func (p *GPSPayload) MarshalJSON() ([]byte, error) {
    // Use alias to avoid infinite recursion
    type Alias GPSPayload
    return json.Marshal((*Alias)(p))
}

func (p *GPSPayload) UnmarshalJSON(data []byte) error {
    // Use alias to avoid infinite recursion
    type Alias GPSPayload
    return json.Unmarshal(data, (*Alias)(p))
}

type RuleReadable

type RuleReadable interface {
	// RuleFields returns this payload's declared rule-readable fields.
	// A nil or empty map is a valid answer meaning "nothing to match on" —
	// distinct from not implementing the interface at all, which means the
	// payload is unreadable and the engine reports it.
	RuleFields() map[string]any
}

RuleReadable exposes the fields a payload is willing to have rule conditions and `$message.*` substitutions read.

Optional and discovered at runtime by type assertion, exactly like its sibling in behaviors.go. A payload that does not implement it is not rule-readable: the rule engine reads no fields from it, the rule does not fire, and the engine reports the rule/payload-type pairing rather than evaluating false in silence.

The PAYLOAD DECLARES its projection; the engine never derives one. There is deliberately no reflective default — reflection would make every payload rule-readable by accident and expose fields the author never intended a rule to match on, and it is the wrong trade in the evaluation hot path.

What to expose, and what not to:

  • EXPOSE structural facts — identifiers, roles, outcomes, states, classifications, counts, routing addresses, timestamps.
  • WITHHOLD LLM-authored and user content — prompts, message text, model output, tool-result bodies (ADR-036), and open caller-populated maps whose contents the payload's author does not control.

The reason is not privacy alone. A rule cannot make a quality judgement over unstructured text, and content in a rule payload is how a small model's context window silently truncates. A content-shaped decision belongs to a coordinator that emits a structured result which a later rule matches on deterministically.

Keys are dotted-path addressable: a nested map[string]any value is reachable as `$message.parent.child`.

Three sharp edges to know before you write a rule against a projection:

  • ABSENCE IS THE FALSE CASE FOR BOOLEANS. A projection that mirrors `omitempty` omits a false bool entirely, so `stop_loop` is absent rather than present-and-false. Test for absence (or condition on the true case); a condition comparing it to `false` matches nothing, silently. The same applies to any zero-valued `omitempty` field.
  • FIELD NAMES ARE NOT GLOBALLY UNIQUE. `$message.type` means the context event kind on ContextEvent, the control-signal verb on UserSignal, and the response kind on UserResponse. A rule subscribed to more than one subject must pin the payload with a second condition, because the engine resolves the path, not the type.
  • PROJECTED VALUES FEED SUBJECT TEMPLATES. `$message.*` resolves in action subjects and reasons as well as in conditions, so anything exposed here can end up as a NATS subject token. Dotted values (`model`) and RFC3339 nano timestamps arrive intact — which also means an exposed field with a `.` or a space in it becomes a malformed subject. Prefer exposing tokens a subject can carry.

type Storable

type Storable interface {
	// EntityID returns deterministic 6-part ID: org.platform.system.domain.type.instance
	EntityID() string

	// Triples returns all facts about this entity
	Triples() []Triple

	// StorageRef returns reference to where full data is stored.
	// May return nil if data is not stored externally.
	StorageRef() *StorageReference
}

Storable is a payload that names its entity and its facts (EntityID and Triples) and carries a reference to where its full data is stored. Components that implement Storable can provide both semantic information and a reference to their full data.

This interface enables the lightweight message pattern where:

  1. Domain processors create messages with semantic data
  2. A storage component stores the full message and adds a StorageReference
  3. A graph consumer receives the Storable with both semantics and reference
  4. Consumers can access full data via StorageReference when needed

EntityID and Triples are also the methods of graph.Graphable, so a Storable satisfies that interface too.

Example implementation:

type StoredEntity struct {
    entityID string
    triples  []Triple
    storage  *StorageReference
}

func (s *StoredEntity) EntityID() string { return s.entityID }
func (s *StoredEntity) Triples() []Triple { return s.triples }
func (s *StoredEntity) StorageRef() *StorageReference { return s.storage }

type StorageReference

type StorageReference struct {
	// StorageInstance identifies which storage component holds the data.
	// This enables federation across multiple storage instances.
	// Examples: "message-store", "cache-1", "objectstore-primary"
	StorageInstance string `json:"storage_instance"`

	// Key is the storage-specific key to retrieve the data.
	// Format depends on the storage backend but typically includes
	// time-based partitioning for efficient retrieval.
	// Examples: "2025/01/13/14/msg_abc123", "robotics/drone/1/latest"
	Key string `json:"key"`

	// ContentType specifies the MIME type of the stored content.
	// This helps consumers understand how to process the data.
	// Examples: "application/json", "application/protobuf", "application/avro"
	ContentType string `json:"content_type"`

	// Size is an optional hint about the stored data size in bytes.
	// This helps consumers decide whether to fetch the full data.
	// A value of 0 indicates the size is unknown.
	Size int64 `json:"size,omitempty"`
}

StorageReference points to where the full message data is stored. This enables lightweight message passing where components can reference stored data without transmitting the full payload.

The StorageReference pattern supports the "store once, reference everywhere" architecture, reducing data duplication and enabling efficient processing of large messages.

Example usage:

  • A storage component stores the full message and returns a StorageReference
  • A graph consumer receives a Storable with a reference to the full data
  • Components can fetch full data only when needed

type Triple

type Triple struct {
	// Subject identifies the entity this triple describes.
	// Must use EntityID.Key() format for consistency with federated entity management.
	// Examples: "telemetry.robotics.drone.1", "gcs-alpha.robotics.drone.001"
	Subject string `json:"subject"`

	// Predicate identifies the semantic property using three-level dotted notation.
	// Format: domain.category.property (e.g., "robotics.battery.level")
	// This maintains consistency with the unified dotted notation from Alpha Week 1.
	// Predicates should be defined in the vocabulary package for consistency.
	Predicate string `json:"predicate"`

	// Object contains the property value or entity reference.
	// For literals: primitive types (float64, bool, string, int)
	// For entity references: entity ID strings using EntityID.Key() format
	// Complex objects should be flattened into multiple triples.
	Object any `json:"object"`

	// Source identifies where this assertion came from.
	// Examples: "mavlink_heartbeat", "gps_fix", "operator_input", "ai_inference"
	// Enables traceability and conflict resolution during entity merging.
	Source string `json:"source"`

	// Timestamp indicates when this assertion was made.
	// Should typically be the message timestamp or processing time.
	// Enables temporal queries and helps with entity state evolution.
	Timestamp time.Time `json:"timestamp"`

	// Confidence indicates the reliability of this assertion (0.0 to 1.0).
	// Higher values indicate greater certainty about the triple.
	//
	// Typical confidence levels:
	//   - 1.0: Direct telemetry or explicit data
	//   - 0.9: High-confidence sensor readings
	//   - 0.7: Calculated or derived values
	//   - 0.5: Inferred relationships
	//   - 0.0: Uncertain or placeholder data
	Confidence float64 `json:"confidence"`

	// Context provides correlation ID for message batches or request tracking.
	// This enables grouping related triples from the same processing batch
	// or correlating triples across distributed systems.
	// Examples: message ID, batch ID, correlation ID, request ID
	Context string `json:"context,omitempty"`

	// Datatype provides optional RDF datatype hint for the Object value.
	// This helps with type interpretation and validation in downstream systems.
	// Examples: "xsd:float", "xsd:dateTime", "xsd:boolean", "rdf:JSON"
	// If omitted, the type is inferred from the Go type of Object.
	//
	// Only the "xsd:" and "rdf:" prefixes are expanded to full IRIs on export;
	// an absolute IRI is passed through as written. Any OTHER prefix reaches
	// the output as a relative reference, which is not valid RDF — so use one
	// of those two prefixes, or write the IRI out in full. A prefix such as
	// "geo:point" looks like a natural example and is exactly that trap.
	Datatype string `json:"datatype,omitempty"`

	// ExpiresAt indicates when this triple should be considered expired.
	// When nil, the triple never expires and remains valid indefinitely.
	// When set, the triple is considered expired after this timestamp.
	// This enables TTL-based triple expiration for temporal data management.
	ExpiresAt *time.Time `json:"expires_at,omitempty"`
}

Triple represents a semantic statement about an entity following the Subject-Predicate-Object pattern. This enables RDF-like knowledge graphs while maintaining simplicity for Go developers.

Triple design follows these principles:

  • Subject: Always an entity ID using EntityID.Key() format (e.g., "telemetry.robotics.drone.1")
  • Predicate: Semantic property using three-level dotted notation (e.g., "robotics.battery.level")
  • Object: Typed value (literals) or entity references (other entity IDs)
  • Metadata: Source, timestamp, and confidence for provenance tracking

Example triples from a drone heartbeat message:

  • ("telemetry.robotics.drone.1", "robotics.battery.level", 85.5)
  • ("telemetry.robotics.drone.1", "robotics.flight.armed", true)
  • ("telemetry.robotics.drone.1", "robotics.component.has", "telemetry.robotics.battery.1")

This structure enables:

  • NATS wildcard queries: "robotics.battery.*" finds all battery predicates
  • Entity relationship modeling: Objects can reference other entities
  • Temporal tracking: Each triple has timestamp and confidence
  • Provenance: Source field tracks where each assertion came from
  • Federation: Works with federated entity IDs from multiple sources

func DedupeAppendTriples

func DedupeAppendTriples(stored, incoming []Triple) (survivors []Triple, suppressed int)

DedupeAppendTriples returns the subset of incoming triples that are not already present in stored and not repeated earlier within incoming itself, preserving first-input order, plus the number suppressed.

Both the stored set and the within-request collapse are seeded into one hash set, so the cost is O(len(stored) + len(incoming)) rather than the O(n*k) a pairwise scan would cost. That matters precisely where duplicate suppression fires hardest: a hierarchy container entity accumulates one containment edge per child, so its stored triple count grows with the population it contains.

stored may be nil, which reduces this to a within-request collapse.

func (Triple) IsRelationship

func (t Triple) IsRelationship() bool

IsRelationship checks if this triple represents a relationship between entities rather than a property with a literal value. Returns true if Object is a valid canonical EntityID.

type Type

type Type = types.Type

Type provides structured type information for messages. It enables type-safe routing and processing by clearly identifying the domain, category, and version of each message. It is an alias: the definition lives in pkg/types, so payload code can say message.Type without importing that package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL