config

package
v0.1.0-alpha.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 36 Imported by: 0

README

config

Package config reads a SemEngine process's configuration: one JSON document, laid over built-in defaults, with a few environment variables on top. It also declares the JetStream streams that configuration needs and creates them on a NATS server, and its Manager keeps the configuration in a NATS key-value bucket so it outlives the process.

What it does not do: it does not connect to NATS (the caller hands StreamsManager and Manager a connected client), and it does not write configuration back to a file. A change the Manager sees in its bucket while it runs updates what GetConfig and WatchModelRegistry report; the Manager itself never starts, stops or rewires a component.

Loading a configuration

cfg, err := config.NewLoader().LoadFile("config.json")
if err != nil {
    return err // the file could not be read, or the document was refused
}
if err := cfg.Validate(); err != nil {
    return err // the document loaded but breaks a rule, such as a missing platform.id
}

LoadFromBytes(data) does the same for a document already in memory, for example after the caller has expanded ${VAR} references in it. Both loaders run the same steps, in this order:

  1. Read the document and refuse it if it breaks a size, depth or key rule (below).
  2. Lay it over the defaults. A key the document sets replaces the default; a key it leaves out keeps the default. Objects merge key by key, so {"platform":{"id":"p1"}} keeps the default platform.region. A value written in the document, including false, 0 or an empty list, replaces the default.
  3. Apply the environment overrides.
  4. Check that platform.org and platform.id together leave room for the entity IDs this deployment will create (ADR-104).

Write every key exactly as this README shows it. A key that differs from a key Config declares only in letter case fails the load, naming the right spelling (#325): "nats":{"URLs":[...]} fails with an error ending config key nats.URLs must be written nats.urls. Names you choose (component, service, stream, archival and model registry names) are yours to spell, and a component's or service's config payload belongs to that component or service: loading does not check its keys.

Loading does not call Validate. The caller calls it, as in the example.

Validate also checks the host: that each configured TLS certificate, key and CA file exists, and it prints a warning on standard error when insecure_skip_verify is set. ValidateDocument runs every other check and reads no file and writes nothing, so a configuration reviewed on another machine (as composition.Validate does) gets the same answer. Both check component entries in name order, so the first error for a document is the same on every run.

The defaults are: platform.region gulf_mexico, nats.urls ["nats://localhost:4222"], nats.jetstream.enabled true, and one service, heartbeat (enabled, every 30 seconds).

An example document

This document loads and validates as written:

{
  "version": "1.0.0",
  "platform": {
    "org": "c360",
    "id": "vessel-001",
    "type": "vessel",
    "region": "gulf_mexico",
    "capabilities": ["radar", "ctd"],
    "environment": "prod"
  },
  "nats": {
    "urls": ["nats://nats-1.example.internal:4222", "nats://nats-2.example.internal:4222"],
    "jetstream": {
      "enabled": true,
      "max_memory": 1073741824,
      "max_file_store": 10737418240
    }
  },
  "services": {
    "metrics": {
      "enabled": true,
      "config": {"port": 9090, "path": "/metrics"}
    }
  },
  "components": {
    "udp-mavlink": {
      "type": "input",
      "name": "udp",
      "enabled": true,
      "config": {"port": 14550, "subject": "input.udp.mavlink"}
    }
  },
  "streams": {
    "SENSOR": {
      "subjects": ["sensor.>"],
      "max_age": "7d",
      "max_bytes": 1073741824,
      "discard": "old"
    }
  }
}

The sections:

  • platform: who this deployment is. org and id are required by Validate; they are the first two parts of every entity ID the deployment creates (ADR-102). environment is a label for logs only and does not separate two deployments: give each deployment its own id. platform.instance_id was removed and is refused at load.
  • nats: the server URLs, and whether JetStream is used. max_memory and max_file_store are what the operator expects the server's account limits to be; StreamsManager.VerifyJetStreamLimits logs a warning when the server's real limit is lower. They do not change the server's limits, which are set in the server's own configuration.
  • services and components: one entry per service or component instance, keyed by its name. Each config object is passed to that service or component unread. A component entry needs type (one of input, processor, output, storage, gateway) and name (the factory that builds it).
  • streams: JetStream streams to create, keyed by stream name (see "Streams" below).
What loading refuses

Loading fails, and returns no Config, when:

  • The document is not exactly one JSON object (a document that is null is refused too), is larger than 10 MB, or nests deeper than 100 levels.
  • A value does not fit its field: for example "urls": "nats://a:4222" (a string where a list is expected), a fraction in max_memory, or an unknown key inside a services entry. The error carries the decoder's message.
  • The document sets a NATS setting that nothing in SemEngine applies yet. These keys are refused when present, whatever their value and in any letter case: nats.username, nats.password, nats.token, nats.tls, nats.max_reconnects, nats.reconnect_wait, nats.jetstream.domain, nats.jetstream.retention_policy and nats.jetstream.replication_factor. Each comes back when the code that connects to NATS uses it (#74).
  • SEMENGINE_NATS_USERNAME, SEMENGINE_NATS_PASSWORD or SEMENGINE_NATS_TOKEN is set to a value that is not empty, for the same reason.
  • The document sets platform.instance_id, in any letter case.
  • A key differs from the key Config declares only in letter case, such as "NATS" or "Enabled" inside a services entry (above). When the document also sets one of the refused keys above, the error names that key instead.

Other unknown keys at the top level, or inside platform or nats, are accepted and ignored.

Which files LoadFile reads

The path must end in .json or .json5; either way the content is read as plain JSON. A relative path must stay inside the current working directory once . and .. are resolved, so ../config.json is refused. An absolute path may point anywhere, but is refused if a file or directory name in it contains ... Symbolic links are followed: a link to a regular file is read, wherever the file is. A directory or a device is refused.

Environment overrides

These variables, when set and not empty, replace the document's value:

Variable Replaces
SEMENGINE_PLATFORM_ID platform.id
SEMENGINE_PLATFORM_TYPE platform.type
SEMENGINE_PLATFORM_REGION platform.region
SEMENGINE_NATS_URLS nats.urls, split at commas; spaces are not trimmed, so write a,b, not a, b

An empty variable is the same as an unset one, so an override cannot set a value to empty.

Sharing a configuration between goroutines

NewSafeConfig(cfg) wraps a Config for concurrent use. Get returns a deep copy, so a caller can change it without affecting anyone else. Update validates a new Config and swaps it in. To change part of the configuration, use Mutate, which holds the lock from the read to the swap:

err := safe.Mutate(func(cfg *config.Config) error {
    c := cfg.Components["udp-mavlink"]
    c.Enabled = false
    cfg.Components["udp-mavlink"] = c
    return nil
})

Get, then a change, then Update can lose a change another goroutine made between the two calls. The function passed to Mutate must not call Get, Update or Mutate itself: it would wait forever on the lock it is running under.

The configuration manager

Manager stores a deployment's configuration in a NATS key-value bucket, the configuration bucket, so that it survives a restart and another process (an operator's tool, a UI) can change what the next boot runs. The bucket is named semengine_config_<org>_<stem> from the document's platform.org and the platform.id it declares: config.BucketName("acme", "dep") returns semengine_config_acme_dep.

func run(ctx context.Context, cfg *config.Config, client *natsclient.Client, logger *slog.Logger) (err error) {
    mgr, err := config.NewConfigManager(cfg, client, logger)
    if err != nil {
        return err
    }
    // Installed before Start: a failed Start whose rollback also fails leaves its cleanup for Stop.
    defer func() {
        // ctx may have ended by now; Stop needs a context that has not.
        stopCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
        defer cancel()
        err = errors.Join(err, mgr.Stop(stopCtx))
    }()
    if err := mgr.Start(ctx); err != nil {
        return err
    }

    platformID := mgr.GetConfig().Get().Platform.ID // the effective identifier, such as "dep-800dff"
    logger.Info("configuration ready", "platform_id", platformID)
    <-ctx.Done()
    return nil
}
Start and Stop

Start(ctx) does all of its work under ctx: it acquires the bucket, establishes the platform identity (below), decides whether the file or the bucket wins, and opens the watches. It refuses a nil or ended ctx, and refuses every call after the first with errs.ErrAlreadyStarted, including a call after Stop: build a new Manager to start again. Once Start has returned, ending its ctx stops nothing; the watches run until Stop (ADR-113).

Stop(ctx) closes every channel the Manager handed out, stops every watch and waits for the Manager's goroutines. It returns nil only when all of that is done. If ctx ends first it returns ctx's error, and a later Stop tries again. A ctx that has already ended stops nothing, which is why the example builds a fresh bounded context instead of passing the signal context that is ending the program.

Platform identity

The first Start on an empty bucket mints the deployment's effective platform.id, the declared value plus a six-character hexadecimal suffix such as dep-800dff, and records it in the bucket's platform_identity key. Every later Start, in this process or another with the same document, adopts the recorded value. Start refuses a bucket whose record names another platform.org or another declared platform.id, and a bucket that holds configuration but no record (ADR-104).

Which copy wins
At Start Result
The bucket holds no configuration (first boot) The file is written to the bucket
The file's version is newer than the bucket's The file is written to the bucket
The versions are equal, or the file's is older The bucket wins
Either version is not a semantic version, including a file with no version The bucket wins

Versions are compared with CompareVersions. The bucket's version is stored as a JSON string. A bucket with no version key compares as 0.0.0, and so does a stored value that fails to decode into a string, such as a number (with a warning in the log), so a file at 1.0.0 wins over either. A stored JSON string that is not a semantic version, such as "latest", cannot be compared, and the bucket wins. A stored null decodes as an empty string, which is not a semantic version either, so the bucket wins. When the bucket wins, its services and components entries replace the file's, so a component deleted at run time stays deleted at the next boot; to bring the file's set back, raise the file's version. A stored model_registry replaces the file's; a stored nats is decoded over the file's nats section, so a key it leaves out keeps the file's value. The stored platform key is a copy for readers and is never applied. When the file is written, version is written last, so a write cut short leaves the bucket's old version, or none. The next boot writes the file again only when the file's version is newer than that. A file with no version, or 0.0.0, is not newer than a missing one, so the next boot selects the partly written bucket instead; raise the file's version to write it again.

Names

Each service and component name is stored as the key services.<name> or components.<name>, with every space written as _. The name must then be one key token: no ., * or >. Start refuses such a name, and two names that map to the same key (a b and a_b), before it touches the bucket. A name with a space appears under its key's name from Start on: a file component a b is a_b in GetConfig.

When Start fails

Start returns an error naming the key, and leaves every writer refusing, when:

  • a name or a value cannot be written (found before the bucket is touched, so nothing is written, not even the identity record);
  • the bucket cannot be listed or read;
  • a stored services, components, nats or model_registry entry does not decode;
  • a stored nats entry sets a key loading refuses (see "What loading refuses");
  • writing the file to the bucket fails;
  • any watch does not open.

Before it returns, a failed Start rolls back what it acquired, such as a watch it opened, with a budget of five seconds of its own. When the rollback succeeds the Manager holds nothing. When it fails, its error is joined into Start's error and the rest of the cleanup is kept for Stop, which is why the example above installs its bounded Stop before calling Start.

A change written to the bucket while the Manager runs that does not decode is logged and skipped.

Next-boot writes

PutComponentToKV(ctx, name, cfg) and DeleteComponentFromKV(ctx, name) write or delete components.<name> and update GetConfig before they return; a put whose key the watcher has already applied at a newer revision keeps that newer value, and a delete never erases a later put of its key: no watcher event is applied between the delete reaching the bucket and its removal from GetConfig. They record what the next boot runs: they never start, stop or restart a component in the running process. WatchModelRegistry() returns a channel that receives the model registry each time its key changes, and closes when Stop runs.

Key families

A key family gives another owner its own keys, <prefix>.<name>, in the same bucket:

rules, err := config.NewKeyFamily("rules", func(e config.KeyFamilyEntry) {
    logger.Info("rule changed", "name", e.Name, "deleted", e.Operation == config.KeyFamilyDelete, "initial", e.Initial)
})
if err != nil {
    return err
}
mgr, err := config.NewConfigManager(cfg, client, logger, config.WithKeyFamily(rules))

After Start, the handler receives every entry present when the watch opened (Initial is true), then each put and delete. It runs on the Manager's goroutine and must not block: Stop waits for it. The family's Get, Put, Create, Delete and Names reach only rules.* keys, and return an error until Start has succeeded. The Manager never decodes a family's values. A prefix the Manager uses itself (services, components, platform, nats, model_registry, platform_identity) is refused by NewConfigManager.

Streams

NewStreamsManager(client, logger).EnsureStreams(ctx, cfg) creates the JetStream streams the configuration needs. For a stream that already exists, it compares the stream's settings with the declaration and updates the ones that differ. The streams come from three places: the framework's own streams, the streams section, and the JetStream output ports of the configured components. A stream named in streams wins over one derived from a port.

Every stream must be bounded. Each needs a finite max_age (a Go duration such as 168h, or a number of days such as 7d), a max_bytes above zero, and a discard policy: old drops the oldest messages when the stream is full, and new refuses new ones. No default fills these in. A stream that must keep everything is declared under archival_streams with an owner and a reason; it is the only exception to the finite bounds. To watch an archival stream's size, turn on the storage observability service with "storage-observability": {"enabled": true} under services (see "Enabling it" in service/README.md). Validate checks all of this without a server (ValidateStreamDeclarations), and EnsureStreams checks it again before it creates anything, so a configuration with a missing bound creates no streams at all.

EnsureStreams also refuses to create a stream whose name belongs to a NATS key-value bucket or object store (ErrBackingStreamNotProvisionable). It fails, changing nothing on that stream, when an existing stream's storage type or retention policy differs from the declaration (ErrStreamNonEditableDrift): those two are not updated in place.

Comparing versions

CompareVersions("1.2.0", "1.10.0") compares two semantic version strings, such as Config.Version, and returns -1, 0 or 1.

Documentation

Overview

Package config reads a SemEngine process's configuration and declares the JetStream streams it needs, and its Manager keeps that configuration in a NATS key-value bucket across restarts.

Loading

A Loader reads one JSON document, lays it over built-in defaults, applies the environment overrides, and checks that platform.org and platform.id leave room for the entity IDs the deployment will create (ADR-104). It does not call Validate; the caller does:

cfg, err := config.NewLoader().LoadFile("config.json")
if err != nil {
	return err
}
if err := cfg.Validate(); err != nil {
	return err
}

LoadFromBytes does the same for a document already in memory. A key the document sets replaces the default, including false, 0 or an empty list; a key it leaves out keeps the default; objects merge key by key.

Write every key exactly as the README shows it. A key that differs from a declared key only in letter case fails the load, naming the right spelling (#325). Names a deployment chooses (component, service, stream, archival and model registry names) are not checked, and a component's or service's config payload belongs to its owner, which decodes it.

Loading fails, returning no Config, when the document:

  • is not exactly one JSON object (null is refused too), is larger than 10 MB, or nests deeper than 100 levels;
  • has a value that does not fit its field, such as a string where a list is expected, or an unknown key inside a services entry (the error carries the decoder's message);
  • sets platform.instance_id, which was removed (ADR-102);
  • sets a NATS setting nothing in SemEngine applies yet: nats.username, nats.password, nats.token, nats.tls, nats.max_reconnects, nats.reconnect_wait, nats.jetstream.domain, nats.jetstream.retention_policy or nats.jetstream.replication_factor. These are refused when present, whatever their value and in any letter case, so a setting an operator wrote is never silently ignored (#74);
  • has a key that differs from a declared key only in letter case, such as "NATS" or nats "URLs": the error names the key as written and as declared, for example "config key nats.URLs must be written nats.urls". When the document also sets one of the keys above, that refusal wins.

Other unknown keys are accepted and ignored.

Environment overrides

When set and not empty, SEMENGINE_PLATFORM_ID, SEMENGINE_PLATFORM_TYPE, SEMENGINE_PLATFORM_REGION and SEMENGINE_NATS_URLS replace the document's platform.id, platform.type, platform.region and nats.urls. The URL list is split at commas and spaces are not trimmed, so write a,b, not a, b. Loading fails when SEMENGINE_NATS_USERNAME, SEMENGINE_NATS_PASSWORD or SEMENGINE_NATS_TOKEN is set and not empty, naming the variable: those credentials are not applied yet.

Which files LoadFile reads

The path must end in .json or .json5; the content is read as plain JSON either way. A relative path must stay inside the current working directory once . and .. are resolved. An absolute path is not confined to any directory, but is refused if a name in it contains "..". Symbolic links are followed, so a link to a regular file is read wherever that file is. A directory or device is refused.

Sharing a configuration

SafeConfig wraps a Config for concurrent use. Get returns a deep copy. Mutate holds the write lock from the read to the swap, so use it, not Get followed by Update, to change part of a configuration another goroutine may also change.

The configuration manager

Manager keeps a deployment's configuration in a NATS key-value bucket, the configuration bucket, so it outlives the process and another process can change what the next boot runs. BucketName names the bucket from the document's platform.org and declared platform.id: semengine_config_<org>_<stem>.

Start(ctx) does its startup work under ctx: it acquires the bucket, establishes the deployment's platform identity from the bucket's identity record (ADR-104), decides whether the file or the bucket wins, and opens the watches. It refuses a nil or ended ctx, and refuses every Start after the first with errs.ErrAlreadyStarted. Ending ctx after Start returned stops nothing (ADR-113): the watches run until Stop.

Stop(ctx) closes every channel the Manager handed out, stops every watch and waits for the Manager's goroutines. It returns nil only once all of that is done. If ctx ends first it returns ctx's error, and a later Stop tries again. A ctx that has already ended stops nothing, so a caller on its shutdown path passes a fresh bounded context, not its ended signal context:

stopCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
err := mgr.Stop(stopCtx)

Which copy wins at Start: on a first boot, and when the file's version is newer than the bucket's version key (CompareVersions), Start writes the file to the bucket. Otherwise (equal versions, an older file, or either version not a semantic version, a file with no version among them) the bucket wins: its services and components replace the file's, so one deleted at run time stays deleted. A missing version key, and a stored version that fails to decode into a string, such as a number (logged as a warning), compare as 0.0.0, so a file at 1.0.0 wins; a stored string that is not a semantic version selects the bucket, and so does a stored null, which decodes as an empty string. A stored model_registry replaces the file's; a stored nats is decoded over the file's nats section, so a key it leaves out keeps the file's value. The stored platform key is a copy for readers and is never applied. Start writes the version key last, so a write cut short leaves the old version, or none; the next boot writes again only when the file's version is newer than that. A file with no version, or 0.0.0, is not, and selects the partly written bucket.

Each service and component name must be one NATS KV key token once its spaces are written as '_': no '.', '*' or '>'. A name with a space is kept under its key's name from Start on, so GetConfig reports a file component "a b" as "a_b", and two names that map to one key are refused.

Start fails closed. It returns an error and leaves every writer refusing when a name or value cannot be written, the bucket cannot be listed or read, a stored services, components, nats or model_registry entry does not decode, a stored nats entry sets a key loading refuses, the write to the bucket fails, or a watch does not open. The error names the key. A refused name or value is found before the bucket is touched, so nothing is written, not even the identity record. A failed Start rolls back what it acquired before returning; if that rollback fails, its error is joined into Start's and the rest of the cleanup is kept for Stop, so a caller installs its bounded Stop before calling Start.

PutComponentToKV and DeleteComponentFromKV change the components the next boot runs. They update GetConfig before they return; a put whose key the watcher has already applied at a newer revision keeps that newer value, and no watcher event is applied between a delete reaching the bucket and its removal from GetConfig, so a delete never erases a later put. They never start, stop or restart a running component. WatchModelRegistry delivers the model registry each time its key changes.

A key family (NewKeyFamily, WithKeyFamily) hands another owner the keys "<prefix>.<name>" in the same bucket: its handler receives the entries present when Start opened its watch, then each put and delete, and its Get, Put, Create, Delete and Names reach only those keys. The Manager never decodes a family's values. A prefix the Manager uses itself (services, components, platform, nats, model_registry, platform_identity) is refused.

Streams

StreamsManager.EnsureStreams creates or repairs the JetStream streams a Config needs: the framework's own, those in its streams section, and those its components' JetStream output ports imply. Every stream must declare a finite max_age, a max_bytes above zero and a discard policy, unless it is declared archival. Config.Validate checks those declarations without a server (ValidateStreamDeclarations).

The package README gives an example document and the rules in full.

Index

Constants

View Source
const (
	// StreamDiscardOld evicts the oldest messages once the stream reaches a
	// limit. The write always succeeds; the data loss is silent and at the tail.
	StreamDiscardOld = "old"
	// StreamDiscardNew refuses the write once the stream reaches a limit. Nothing
	// is evicted; producers see the rejection instead. See discardPolicyGuidance.
	StreamDiscardNew = "new"
)

Discard-policy declaration values for StreamConfig.Discard. These are the operator-facing spellings of jetstream.DiscardOld / jetstream.DiscardNew; the mapping happens once, in buildStreamConfig.

Variables

View Source
var (

	// ErrStreamBoundsUndeclared is returned when an ordinary stream carries no
	// explicitly declared finite MaxAge, finite MaxBytes, or discard policy and
	// is not admitted by an archival declaration.
	//
	// It is natsclient's value rather than a second one of the same name. The same
	// requirement is enforced at two seams — this declarative path and
	// natsclient.Client.EnsureStream — and a caller testing for "bounds are not
	// declared" should not have to know which door refused it.
	ErrStreamBoundsUndeclared = natsclient.ErrStreamBoundsUndeclared

	// ErrArchivalStreamInvalid is returned when an archival declaration omits its
	// owner or its reason, so archival cannot become a silent way to opt out of
	// bounds.
	ErrArchivalStreamInvalid = errors.New("archival stream declaration is incomplete")
)

Bounds-contract sentinels. Classifiable so a boot path, an operator tool, or a test can tell an undeclared bound from a malformed archival declaration, which fail validation for different reasons and have different fixes.

View Source
var ErrBackingStreamNotProvisionable = natsclient.ErrBackingStreamNotProvisionable

ErrBackingStreamNotProvisionable is the refusal sentinel, re-exported from natsclient so config callers and tests need not reach across for it. It is the SAME error value, so errors.Is matches a refusal raised at any provisioning seam — this provisioner, Client.EnsureStream, or Client.CreateStream.

View Source
var ErrStreamNonEditableDrift = errors.New("existing stream diverges in a field that cannot be reconciled in place")

ErrStreamNonEditableDrift is returned when an existing ordinary stream's live configuration diverges from its declaration in a field the provisioner will not change on a live stream. Readiness fails rather than accepting the divergence in silence, because a stream that reports one storage tier or retention policy while its declaration states another is a configuration two people can read opposite answers from.

Functions

func BucketName

func BucketName(org, stem string) (string, error)

BucketName is the ONE derivation of a deployment's configuration bucket name: "semengine_config_<org>_<stem>", from the configuration document's platform.org and its DECLARED platform.id — never the minted identifier, which lives inside the bucket and so cannot name it (owner ruling on #1188, 2026-09-01: "Bucket = semengine_config_<org>_<stem>, looked up by the same").

It refuses an empty part, a byte outside the NATS bucket grammar [A-Za-z0-9_-] (nats.go jetstream/kv.go validBucketRe), and a result longer than NATS accepts. Every pair Config.Validate admits passes: a segment is already a subset of that grammar, and a declared pair is at most 163 bytes.

The separator is not injective: `_` is legal inside both parts, so org "a_b" with stem "c" and org "a" with stem "b_c" name the same bucket. That alias is refused at Start by the identity record's org/stem compare, not prevented here (owner ruling on #1188, 2026-09-27, Q2 (a)).

func CompareVersions

func CompareVersions(v1, v2 string) (int, error)

CompareVersions compares two semver version strings Returns:

-1 if v1 < v2
 0 if v1 == v2
 1 if v1 > v2
error if either version is invalid

func DeriveStreamName

func DeriveStreamName(subject string) string

DeriveStreamName extracts stream name from subject convention. Convention: subject "component.action.type" → stream "COMPONENT" Examples:

"objectstore.stored.entity" → "OBJECTSTORE"
"sensor.processed.entity"   → "SENSOR"
"rule.triggered.alert"      → "RULE"

Types

type ArchivalStream

type ArchivalStream struct {
	// Owner is the team or component accountable for this stream's growth.
	// Required — an archival stream's only remaining lever is capacity, and
	// capacity questions need an addressee.
	Owner string `json:"owner"`

	// Reason states why permanence is the contract. Required — without it,
	// "archival" is just an unbounded stream with better vocabulary.
	Reason string `json:"reason"`
}

ArchivalStream declares ONE ordinary stream whose contract is permanence: nothing may ever be evicted from it. It is exempt from the finite-bounds requirement by declaration, and validation reports it as a named PERMANENT exception.

Both fields are required, so archival cannot become an unowned, unexplained way to opt out of bounds.

type ArchivalStreamStatus

type ArchivalStreamStatus struct {
	Stream string // the permanently unbounded ordinary stream
	Owner  string // who is accountable for its growth
	Reason string // why permanence is the contract
}

ArchivalStreamStatus is one permanent exception as validation reports it. It has no expiry: archival is permanent by contract.

func ValidateStreamDeclarations

func ValidateStreamDeclarations(cfg *Config) ([]ArchivalStreamStatus, error)

ValidateStreamDeclarations resolves every ordinary stream this configuration would provision and enforces the bounds contract on it, returning one entry per archival_streams entry, sorted by stream name, nil when there is none.

It is pure and I/O-free, which is what lets Config.Validate run it: without that, `semstreams --validate` would print "✓ Configuration is valid" for a configuration that hard-fails at boot.

type ArchivalStreams

type ArchivalStreams map[string]ArchivalStream

ArchivalStreams maps ordinary stream name to its archival declaration.

type ComponentConfigs

type ComponentConfigs map[string]types.ComponentConfig

ComponentConfigs holds component instance configurations. The map key is the instance name (e.g., "udp-sensor-main"). Components are only created if both: 1. Their factory has been registered via init() 2. They have an entry in this config map with enabled=true

type Config

type Config struct {
	Version       string               `json:"version"` // Semantic version (e.g., "1.0.0") for KV sync control
	Platform      PlatformConfig       `json:"platform"`
	Security      security.Config      `json:"security,omitempty"` // Platform-wide security configuration
	NATS          NATSConfig           `json:"nats"`
	Services      types.ServiceConfigs `json:"services"`                 // Map of service configs
	Components    ComponentConfigs     `json:"components"`               // Map of component instance configs
	Streams       StreamConfigs        `json:"streams,omitempty"`        // Explicit JetStream stream definitions
	ModelRegistry *model.Registry      `json:"model_registry,omitempty"` // Unified model endpoint registry

	// ArchivalStreams declares ordinary streams whose contract is permanence:
	// the one exception to the finite-bounds requirement.
	ArchivalStreams ArchivalStreams `json:"archival_streams,omitempty"`
}

Config represents the complete application configuration Simplified to 6 fields: Version (semver), Platform (identity), Security (TLS), NATS (connection), Services, Components

func (*Config) Clone

func (c *Config) Clone() *Config

Clone creates a deep copy of the configuration

func (*Config) GetOrg

func (c *Config) GetOrg() string

GetOrg returns the organization from platform config

func (*Config) GetPlatform

func (c *Config) GetPlatform() string

GetPlatform returns the minting deployment authority: platform.id, the one identity field (ADR-102, ruled O-2; platform.instance_id was removed).

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration for use on this host: ValidateDocument, then the host checks: each configured server certificate, server key and client CA file exists, and a warning on standard error when client certificate verification is switched off.

func (*Config) ValidateDocument

func (c *Config) ValidateDocument() error

ValidateDocument checks the configuration document alone: it reads no file and writes nothing, so offline review (composition.Validate) judges a deployment's document the same way on any machine. It reads nothing from outside the document. Component entries are visited in instance name order, so the first error for a document is the same on every call. Like Validate, it lowercases platform.org in place.

type JetStreamConfig

type JetStreamConfig struct {
	Enabled bool `json:"enabled"`
	// MaxMemory is the operator's expected server-side
	// max_memory_store limit. VerifyJetStreamLimits Warns at boot if
	// this exceeds the server's actual AccountInfo.Limits.MaxMemory.
	// 0 disables the check (the default for operators who manage
	// JetStream sizing entirely via nats.conf without surfacing intent
	// in the framework config).
	MaxMemory int64 `json:"max_memory,omitempty"`
	// MaxFileStore is the operator's expected server-side
	// max_file_store limit. Same Warn-on-gap semantics as MaxMemory.
	MaxFileStore int64 `json:"max_file_store,omitempty"`
}

JetStreamConfig for JetStream settings. MaxMemory and MaxFileStore are verification hints; JetStream account limits are exclusively server-side configuration via nats.conf — the nats.go SDK exposes AccountInfo as read-only and offers no client-side mutation surface, so the framework cannot push operator intent. What it CAN do is compare configured-vs-server at boot and Warn on gap (see StreamsManager.VerifyJetStreamLimits) so an operator who set max_file_store: 10GB in the framework config but didn't update nats.conf isn't left wondering why stream-create eventually fails with "insufficient storage resources".

type KeyFamily

type KeyFamily struct {
	// contains filtered or unexported fields
}

KeyFamily is the family of keys "<prefix>.<name>" in the configuration bucket, together with the handler that receives the family's entries.

Its reads and writes are scoped to the family's "<prefix>." keys, and NewConfigManager refuses a family whose prefix is one of the Manager's own key prefixes, so a holder cannot reach the identity record or any other key the Manager owns, and never learns the bucket's name. They return the not-acquired error until a Manager that registered the family has started successfully.

func NewKeyFamily

func NewKeyFamily(prefix string, handle func(KeyFamilyEntry)) (*KeyFamily, error)

NewKeyFamily builds a family for the keys "<prefix>.<name>".

prefix and every member name are each one NATS KV literal token (natsclient.ValidateKVLiteralToken): the family's watch is "<prefix>.*", which matches exactly one token after the prefix, so a member name holding a "." would be stored but never delivered.

handle runs on the Manager's watch goroutine, in delivery order, and MUST NOT block: the Manager's Stop waits for that goroutine. A handler that has real work to do should signal a goroutine it owns.

func (*KeyFamily) Create

func (f *KeyFamily) Create(ctx context.Context, name string, value []byte) error

Create writes one member only if it is absent, or returns natsclient.ErrKVKeyExists.

func (*KeyFamily) Delete

func (f *KeyFamily) Delete(ctx context.Context, name string) error

Delete removes one member.

func (*KeyFamily) Get

func (f *KeyFamily) Get(ctx context.Context, name string) ([]byte, error)

Get returns the value of one member, or natsclient.ErrKVKeyNotFound.

func (*KeyFamily) Names

func (f *KeyFamily) Names(ctx context.Context) ([]string, error)

Names lists the family's current member names. A key under "<prefix>." with more than one token after it is not a member, because the family's watch never delivers it, so Names omits it too, and logs a Warn naming it.

func (*KeyFamily) Put

func (f *KeyFamily) Put(ctx context.Context, name string, value []byte) error

Put writes one member unconditionally.

type KeyFamilyEntry

type KeyFamilyEntry struct {
	// Name is the member name: the key with "<prefix>." removed.
	Name string
	// Value is the entry's value. It is empty for a delete.
	Value []byte
	// Operation says whether the entry was put or deleted.
	Operation KeyFamilyOperation
	// Initial is true for an entry that was present when the watch opened.
	Initial bool
}

KeyFamilyEntry is one entry of a key family, delivered to its handler.

type KeyFamilyOperation

type KeyFamilyOperation uint8

KeyFamilyOperation says what happened to one family entry.

const (
	// KeyFamilyPut is a created or updated entry.
	KeyFamilyPut KeyFamilyOperation = iota + 1
	// KeyFamilyDelete is a deleted or purged entry.
	KeyFamilyDelete
)

type Loader

type Loader struct {
	// contains filtered or unexported fields
}

Loader loads one configuration document over the defaults.

func NewLoader

func NewLoader() *Loader

NewLoader creates a new configuration loader.

func (*Loader) LoadFile

func (l *Loader) LoadFile(path string) (*Config, error)

LoadFile loads configuration from a single file. It applies the defaults, the document, the environment overrides and the authority-pair reserve; it does not call Validate, which the caller does.

func (*Loader) LoadFromBytes

func (l *Loader) LoadFromBytes(data []byte) (*Config, error)

LoadFromBytes loads configuration from JSON bytes. This is useful when you need to pre-process the configuration (e.g., environment variable expansion) before loading.

The data is merged with the defaults exactly as LoadFile merges a file.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager provides centralized configuration management with channel-based updates

func NewConfigManager

func NewConfigManager(cfg *Config, natsClient *natsclient.Client, logger *slog.Logger, opts ...ManagerOption) (*Manager, error)

NewConfigManager creates a new configuration manager

func (*Manager) DeleteComponentFromKV

func (cm *Manager) DeleteComponentFromKV(ctx context.Context, name string) error

DeleteComponentFromKV removes a component from durable next-boot desired state. A push only puts keys that exist in memory; it does not delete keys that are absent.

The removal is also applied synchronously to the Manager's current desired-state view. It does not tear down or otherwise change the running ComponentManager; composition changes only on process restart.

The NATS API does not return the Delete's revision, so the bucket delete and the in-memory removal run under revMu, the lock every revisioned apply takes: no watcher event applies between them. A later put of the key therefore applies after the removal, never before it, and the delete's own event is either an idempotent reapply or, after that later put, skipped as older.

func (*Manager) GetConfig

func (cm *Manager) GetConfig() *SafeConfig

GetConfig returns the current configuration

func (*Manager) PutComponentToKV

func (cm *Manager) PutComponentToKV(ctx context.Context, name string, compConfig types.ComponentConfig) error

PutComponentToKV writes a single component's configuration to NATS KV. This is more efficient than a full push when only one component has changed, and avoids race conditions with KV watchers when multiple operations are in flight.

The method performs write KV → apply desired state in memory and record the key's revision. It records a component candidate for the next process start; it does not add or restart a component in the running ComponentManager. The revision returned by KV.Put lets handleUpdate skip the redundant reapply while still notifying desired-state observers. KV-write is first so a failed Put leaves memory untouched.

func (*Manager) Start

func (cm *Manager) Start(ctx context.Context) error

Start acquires the configuration bucket, establishes the platform identity, arbitrates between the file and the bucket, and opens the watches, all under ctx. It refuses a nil or ended ctx, and any Start after the first with errs.ErrAlreadyStarted, before contacting NATS. A Start that fails arms no writer and rolls back what it acquired; a rollback that fails is joined into Start's error and leaves its cleanup for Stop.

Ending ctx after Start returned stops nothing: the watches run under a run context Start creates and Stop ends (ADR-113). Stop is the only stop.

func (*Manager) Stop

func (cm *Manager) Stop(ctx context.Context) error

Stop stops the watches, closes every channel onChange and WatchModelRegistry returned, and waits for every goroutine the Manager started. It returns nil only once all of that is done. If ctx ends first it returns ctx's error, and a later Stop tries again.

A ctx that has already ended stops nothing. A caller on its shutdown path therefore passes a fresh bounded context, for example context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second), not its ended signal context.

func (*Manager) WatchModelRegistry

func (cm *Manager) WatchModelRegistry() <-chan *model.Registry

WatchModelRegistry returns a channel that emits the latest *model.Registry whenever the model_registry KV key changes. The channel is buffered (cap 1); slow consumers see the most recent registry on their next read — intermediate updates coalesce.

Use this for external library consumers that deliberately maintain a live model-registry view. SemEngine components receive the registry selected at boot; later writes are durable desired state for the next process start and do not restart or rewire the running ComponentManager.

See model.Watch for a one-line consumer pattern.

The channel closes when Stop runs, and a reader still holding it does not hold Stop. A channel asked for after Stop is returned closed.

type ManagerOption

type ManagerOption func(*Manager)

ManagerOption configures a Manager at construction.

func WithKeyFamily

func WithKeyFamily(family *KeyFamily) ManagerOption

WithKeyFamily registers a key family whose owner is not the Manager.

The Manager is the configuration bucket's only acquirer, and the bucket's name is derived from the deployment's declared authority pair, which only the Manager holds. A second writer therefore cannot find the bucket on its own. It registers its key family here and reaches the bucket through the family instead. Owner ruling on #1188, 2026-09-27 (Q1 (d)).

type NATSConfig

type NATSConfig struct {
	URLs      []string        `json:"urls,omitempty"`
	JetStream JetStreamConfig `json:"jetstream,omitempty"`
}

NATSConfig defines NATS connection settings. It carries only the settings something reads: boot dials URLs, and VerifyJetStreamLimits reads JetStream. Credentials, TLS and reconnect settings are refused at load until boot applies them (unappliedNATSKeys).

type PlatformConfig

type PlatformConfig = platform.Config

PlatformConfig is an alias for platform.Config.

PlatformConfig was promoted to its own leaf package so message/ (and any future consumer of platform identity) can reference it without importing all of config. Keeping the alias preserves config.PlatformConfig usage at every call site that already had it. New code should prefer platform.Config directly. Removing the alias would be a mechanical rename across ~10 files; defer until there's a reason.

type SafeConfig

type SafeConfig struct {
	// contains filtered or unexported fields
}

SafeConfig provides thread-safe access to configuration

func NewSafeConfig

func NewSafeConfig(cfg *Config) *SafeConfig

NewSafeConfig creates a new thread-safe config wrapper

func (*SafeConfig) Get

func (sc *SafeConfig) Get() *Config

Get returns a deep copy of the current configuration

func (*SafeConfig) Mutate

func (sc *SafeConfig) Mutate(fn func(*Config) error) error

Mutate atomically applies a read-modify-write to the configuration. It holds the write lock across the WHOLE clone → mutate → validate → swap sequence, so two concurrent mutations cannot lose one another's change (gh#515). Without this, a caller doing Get() (a clone) → mutate → Update() races another doing the same: each starts from the same base and the second swap clobbers the first. -race does not flag it — each Get/Update is individually locked; the atomicity violation is at the compound level.

fn receives a private deep copy it may mutate freely; returning a non-nil error aborts the mutation with no change. The result is validated before the swap.

Re-entrancy contract: fn MUST NOT call any SafeConfig method (Get/Update/Mutate) — it operates only on the draft it is handed, or it self-deadlocks on sc.mu. Do post-mutation side effects (KV writes, subscriber notification) AFTER Mutate returns; capture any value they need into a variable inside fn.

Unlike Update, validation runs INSIDE the write lock (it must, to keep the whole RMW atomic). Validate does filesystem I/O (os.Stat) when TLS is configured, so a mutation briefly blocks concurrent Get() readers — acceptable because config mutations are rare and off the hot path.

func (*SafeConfig) Update

func (sc *SafeConfig) Update(cfg *Config) error

Update atomically updates the configuration after validation

type StreamConfig

type StreamConfig struct {
	Subjects  []string `json:"subjects"`            // Subjects captured by this stream
	Storage   string   `json:"storage,omitempty"`   // "file" or "memory" (default: file)
	MaxAge    string   `json:"max_age,omitempty"`   // Required: message TTL (e.g., "168h", "7d")
	MaxBytes  int64    `json:"max_bytes,omitempty"` // Required: max storage in bytes; must be > 0
	Retention string   `json:"retention,omitempty"` // "limits", "interest", "workqueue" (default: limits)
	Replicas  int      `json:"replicas,omitempty"`  // Replication factor (default: 1)

	// Discard is the operator's choice of what happens when the stream reaches
	// MaxBytes or MaxAge: StreamDiscardOld evicts the oldest messages,
	// StreamDiscardNew refuses the write (producers see NATS 503
	// err_code=10077). Required for an ordinary stream — it was hardcoded to
	// DiscardOld before, so the policy was never the operator's choice.
	Discard string `json:"discard,omitempty"`

	// Duplicates is the server-side duplicate-detection window for the
	// Nats-Msg-Id header (e.g. "2m", "30m", "1h"). Producers using
	// Client.PublishToStreamWithMsgID rely on this window to collapse
	// redeliveries of the same logical event (ADR-055 §5, "T1"). Empty
	// leaves the window unset, so the NATS server applies its default of
	// 2 minutes — adequate for steady-state redelivery but shorter than a
	// restart/recovery replay horizon. Streams whose producers need
	// dedup across that horizon set this explicitly. Must be <= MaxAge:
	// the NATS server REJECTS (does not clamp) an explicit window larger
	// than MaxAge, so createStream clamps it down to MaxAge with a warning
	// rather than letting EnsureStreams abort boot.
	Duplicates string `json:"duplicates,omitempty"`
}

StreamConfig defines configuration for a JetStream stream.

MaxAge, MaxBytes, and Discard are REQUIRED for an ordinary stream and are never supplied by a framework default. A bound the operator never chose is indistinguishable in the operator surface from one they did, which is the condition the bounds contract exists to end. The one way out is an explicit declaration of its own: `archival_streams` for a stream whose contract is permanence. See ValidateStreamDeclarations.

type StreamConfigs

type StreamConfigs map[string]StreamConfig

StreamConfigs is a map of stream name to configuration.

type StreamsManager

type StreamsManager struct {
	// contains filtered or unexported fields
}

StreamsManager handles JetStream stream creation and management.

func NewStreamsManager

func NewStreamsManager(natsClient *natsclient.Client, logger *slog.Logger) *StreamsManager

NewStreamsManager creates a new StreamsManager.

func (*StreamsManager) EnsureStreams

func (sm *StreamsManager) EnsureStreams(ctx context.Context, cfg *Config) error

EnsureStreams creates all required JetStream streams based on: 1. System streams (LOGS for out-of-band logging) 2. Explicit streams defined in config.Streams (highest priority) 3. Streams derived from component JetStream output ports

Every resolved ordinary stream must carry an explicitly declared finite MaxAge, a finite MaxBytes, and a discard policy, unless an archival declaration admits it. That check runs BEFORE any stream is created, so a configuration missing a bound fails closed with a complete diagnostic rather than provisioning half its streams and then stopping — a partially-provisioned account is harder to reason about than one that never started.

func (*StreamsManager) VerifyJetStreamLimits

func (sm *StreamsManager) VerifyJetStreamLimits(ctx context.Context, cfg *Config) error

VerifyJetStreamLimits reads the operator's MaxMemory / MaxFileStore hints from cfg.NATS.JetStream and logs a Warn for each value that exceeds the server's actual account limit. JetStream account limits are server-side configuration (nats.conf or jetstream-domain configuration); the nats.go SDK exposes AccountInfo as read-only, so the framework cannot push the operator's intent — but it CAN surface the gap loudly so an operator who set max_file_store: 10GB in the framework config but didn't update nats.conf isn't left wondering why stream-create fails with "insufficient storage resources" at runtime (#101). Zero or unset values skip the check.

Best-effort: a failure to fetch AccountInfo (server down, JetStream disabled, AccountInfo unsupported) logs at Debug and returns nil — the check is diagnostic, not gating. EnsureStreams runs anyway and any actual capacity miss surfaces as a CreateStream error there.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL