objectstore

package
v0.1.0-alpha.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 29 Imported by: 0

README

objectstore

objectstore.Store keeps byte objects in one NATS ObjectStore bucket (NATS JetStream's store for large values, named by a string key). It implements storage.StreamableStore: Put, Get, Open, List and Delete. It also has StoreContent, which stores a payload's body text and returns a message.StorageReference that other code resolves back to this Store.

The package also has Component, which wraps the Store with NATS ports so that other components can hand it messages whose body should be stored. See "The Component" below.

Use

store, err := objectstore.NewStoreWithConfig(ctx, client, objectstore.Config{BucketName: "MESSAGES"})
if err != nil {
    return err
}
err = store.Put(ctx, "events/a", []byte("hello"))
data, err := store.Get(ctx, "events/a")
keys, err := store.List(ctx, "events/")
err = store.Delete(ctx, "events/a")

ExampleNewStoreWithConfig in example_test.go is this code as a compiled Go example. client is a connected *natsclient.Client. NewStoreWithConfigAndMetrics also takes a *metric.MetricsRegistry and records the Store's counters and latencies on it, in names that start semengine_objectstore_; NewStoreWithConfig records nothing.

What each call does

  • NewStoreWithConfig creates the bucket, or opens it when it exists. Config.BucketName defaults to MESSAGES. Config.InstanceName is the name stamped into every reference the Store returns (StorageReference.StorageInstance) and defaults to the bucket name; the code that owns the Store sets it.
  • Put replaces the object at the key. No earlier version is kept. NATS was observed to behave this way with nats.go v1.54.0 and nats-server v2.15.1 (probe P4, PR #432 comment); the library does not promise it.
  • Get and Open of a missing key return an error that matches storage.ErrObjectNotFound and is not transient.
  • Delete of a key with no object returns nil.
  • List(prefix) returns the names that start with the prefix, sorted, and an empty slice when none match. It reads the metadata of every object in the bucket and filters in the process, so its cost grows with the bucket.
  • Every read goes to NATS. The Store keeps no copy of an object, holds no context and starts no goroutine, so it has nothing to close. A nil context is refused by every call as invalid input.

No age or size limit

Graph entities point at stored objects for as long as the entities live, so an object that expires would strand them (ADR-068, ADR-063). The bucket is created with no TTL and no size limit. If an existing bucket's backing stream (OBJ_<bucket>) has either, construction removes it, logs a warning naming the bucket and what it removed, and fails with an error wrapping natsclient.ErrGraphBucketRetention when it cannot. Removing a limit deletes no object.

StoreContent keeps only the body

StoreContent(ctx, cs) writes the text under cs's body role as raw UTF-8 bytes at a new key (content/<yyyy>/<mm>/<dd>/<hh>/<entityID>_<uuid>) and returns a reference with the Store's instance name, the key, text/plain; charset=utf-8 and the byte size. Opening the reference gives the body and nothing else.

It refuses with an invalid-input error, before writing anything, an invalid entity ID and any other non-empty content: text under a field that is not the body's, text under a field that a non-body role also names, or a non-empty binary field of a message.BinaryStorable. The error names each refused field. Put non-body text in the graph as triples, which the producer emits. A payload with no content stores an empty object.

The Component

objectstore.Register(registry) adds the factory under the name objectstore. A running Component subscribes to its input ports; for each message it decodes the payload through the payload registry, and when the payload is a message.ContentStorable it stores the body with StoreContent and publishes a StoredMessage (payload type storage.stored.v1; RegisterPayloads registers it) on the stored output. The StoredMessage carries the entity ID, its triples and the reference, so a reader opens the body through the store registry. Any other message is refused: the Component stores bodies only.

Configuration

Two keys, both optional. Any other key is refused when the Component is built, naming it.

  • bucket_name: the NATS ObjectStore bucket, default MESSAGES.
  • ports: inputs and outputs. With no ports, the Component has the input write (core NATS, subject storage.objectstore.write) and the output stored (core NATS, subject storage.objectstore.stored).
{
  "bucket_name": "MESSAGES",
  "ports": {
    "inputs": [
      {"name": "write", "config": {"kind": "jetstream", "stream_name": "STORAGE", "subjects": ["storage.objectstore.write"]}}
    ],
    "outputs": [
      {"name": "stored", "config": {"kind": "nats", "subject": "storage.objectstore.stored"}}
    ]
  }
}
  • An input is core NATS or JetStream, one subject each. A Component with an input must have the stored output, and stored is the only output name allowed, because a body stored with no reference published reaches nobody. A Component with no inputs only offers its Store and needs no output.
  • A JetStream input's stream must already exist. Start waits for it and fails if it does not appear; it does not create it. The durable consumer is created with a name derived from the subject, so an input that declares consumer_name is refused. The other consumer settings, ack_wait included, reach the consumer (TestJetStreamInputAckWaitReachesConsumer), and delivery starts at the beginning of the stream unless the port says otherwise.
  • The Component registers its Store under the name objectstore through ProvidedStores, and every reference it publishes carries that name. A process runs one such Component; starting a second one fails the boot (TestSecondComponentRefusedAtStart).
What happens to a message
Input Stored and published Refused or failed
core NATS (the default) the message is done logged at error level, counted in ErrorCount, then gone
JetStream acknowledged after the body is stored and its reference published refused (not decodable, not a ContentStorable, or content the Store refuses): terminated, not redelivered; a failed store or publish: NAK, redelivered after 30 seconds (5 seconds if the handler's context ended); each is logged at error level and counted

The core NATS input is at-most-once: NATS keeps nothing to redeliver, so a refused message is lost after the log line. The JetStream input is at-least-once. If publishing the reference fails after the body was stored, the redelivery stores the body again under a new key, so one object can be left that no reference names. A delivery that reaches the consumer's MaxDeliver limit is not delivered again, and the Component reports nothing about it.

Health, Stop and the Start context
  • If a JetStream input's consume loop ends while the Component runs, or its durable consumer cannot be found by name, Health reports the Component not healthy and names the input (TestLostJetStreamInputReportsUnhealthy, TestMemoryInputLostAtBrokerRestartStaysUnhealthy). A check runs every five seconds. The Component does not bind the input again; restarting the process is the recovery. Whether the server deleting a consumer ends the consume loop is not promised (observed once with nats.go v1.54.0); the lookup by name is what catches the other cases.
  • Stop drains every input and waits for its running handlers to return (TestStopJoinsHandlerAfterConnectionLoss). If Stop's context ends first, the inputs stay held and a later Stop finishes the job (TestFailedCleanupRetainsInputsForLaterStop).
  • The input handlers keep running after the context given to Start ends; only Stop ends them (TestStartContextEndKeepsBothLanesWriting).

Not here

These are not in this package: the in-memory data cache and Close of the SemStreams original (dropped by the owner, PR #432 comment), the SemStreams Component's raw lane (storing the wire bytes of any message under a generated key), its events output and its api input, and a way to name several Store instances in one process (ADR-063 left it open; the answer is one name, objectstore, per process).

Documentation

Overview

Package objectstore is a storage.StreamableStore over one NATS ObjectStore bucket.

NewStoreWithConfig creates the bucket, or opens it when it exists. The bucket has no TTL and no size limit: graph entities point at its objects for as long as they live (ADR-068). If an existing bucket's backing stream (OBJ_<bucket>) has either, construction removes it, logs a warning naming the bucket and what was removed, and fails with an error wrapping natsclient.ErrGraphBucketRetention if it cannot.

Put replaces the object at a key; no earlier version is kept. Get and Open of a missing key return an error matching storage.ErrObjectNotFound. Delete of a missing key returns nil. List returns the matching names sorted, reading every object's metadata to do so. The Store keeps no copy of any object and has nothing to close.

StoreContent stores a message.ContentStorable's body as raw bytes and returns a message.StorageReference that opens to that body. It refuses any other content.

Component

Component wraps the Store with NATS ports. Register adds its factory to a component.Registry under the name "objectstore". Each message on an input port is decoded through the payload registry; if its payload is a message.ContentStorable, the body is stored through StoreContent and a StoredMessage (payload type storage.stored.v1, which RegisterPayloads registers) carrying the entity, its triples and the body's reference is published on the "stored" output port. Any other message is refused: the Component stores bodies only.

The configuration has two keys, both optional: bucket_name (default MESSAGES) and ports. An unknown key is refused at construction, naming it. With no ports the Component has one input, "write", a core NATS subscription on storage.objectstore.write, and one output, "stored", on storage.objectstore.stored. A configured output must be named "stored", and a Component with an input must have it. An input may be core NATS or JetStream. A JetStream input must name a stream that exists: Start does not create it, waits for it, and fails if it never appears. A JetStream input that declares consumer_name is refused, because the durable consumer is named from the subject; its ack_wait and the other consumer settings are used.

The default core NATS input keeps nothing: a message that is refused, or whose write fails, is logged at error level, counted in Health's ErrorCount, and gone. A JetStream input is at-least-once: a delivery is acknowledged only after its body is stored and its reference published. A refusal ends the delivery (Term); a failed store or publish is negatively acknowledged (NAK) for redelivery, after 30 seconds, or 5 seconds when the handler's context ended. Every refusal and failure is logged and counted. A redelivery after a failed publish stores the body again under a new key, so it can leave one stored object that no reference names. A delivery that reaches the consumer's MaxDeliver is not delivered again and the Component reports nothing about it.

The Component offers its Store, under the instance name "objectstore", through ProvidedStores. That one name is stamped into every reference. A process runs one such Component: a second is refused when the component manager starts it.

If a JetStream input's consume loop ends while the Component runs, or its durable consumer cannot be found on the server, Health reports the Component not healthy, naming the input. A lost consume loop is not rebound: the recovery is to restart the process. The check runs every five seconds, so the report can take that long. Whether the server's deleting a consumer ends the consume loop is not promised (observed once with nats.go v1.54.0). The lookup by name catches the cases where it does not.

Stop drains every input and waits for its running handlers to return, through natsclient's WaitConsumerHandlers. The handlers keep running after the context given to Start ends; only Stop ends them.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Register

func Register(registry *component.Registry) error

Register registers the objectstore factory with registry.

func RegisterPayloads

func RegisterPayloads(reg *payloadregistry.Registry) error

RegisterPayloads registers StoredMessage as storage.stored.v1 with reg. Importing this package registers nothing: the consumer's composition root calls RegisterPayloads on the registry it gives its decoders, and a test on the registry it builds.

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component stores the bodies of the messages on its inputs in a NATS ObjectStore bucket and publishes a StoredMessage carrying each body's reference on its stored output (ADR-063).

Only a message.ContentStorable payload's body is stored, through the Store's StoreContent; every other message is refused (design D3, owner ruling 1). A refusal or a failed write is logged at error level and counted in Health's ErrorCount. A core-NATS input is at-most-once: such a message is then gone. A JetStream input is at-least-once: a refusal is terminated, a failure NAKed for redelivery, and a delivery is acked only after its body is stored and its reference published (design D4, owner ruling 2).

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports the lifecycle shell's counts.

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the lifecycle shell's verdict over the Component's one stuck state: a JetStream input whose consume loop ended while running, which only a process restart clears, or whose consumer the latest input check could not find (#216's ruling; checkInputs). ErrorCount is reported and never lowers Healthy.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the Component initialized (no I/O). The factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns the input ports, fixed at construction.

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns the output ports, fixed at construction: stored, if configured, and store-provide.

func (*Component) ProvidedStores

func (c *Component) ProvidedStores() map[string]storage.StreamableStore

ProvidedStores offers the running Component's Store under the instance name it stamps into every reference, for the component manager to register (ADR-063). It returns nil before Start and after Stop.

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start opens the Store and binds every input, through the lifecycle shell: a nil or ended context and a second Start are refused before anything is acquired, and a Start that fails is rolled back through release; a rollback that fails leaves release pending for the next Stop. A Component with no NATS client is refused before the shell is consulted.

ctx bounds Start only: opening the bucket, waiting for each JetStream input's stream and binding the inputs run under it, and a ctx that ends first fails Start. The inputs' handlers run under contexts natsclient owns, and the input check under a run context start creates and release ends, so ending ctx after Start returned stops nothing (ADR-113).

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop stops the Component through the shell, which runs release: Stop before Start releases nothing; a Stop whose release fails keeps it pending, and the next Stop runs it again; after a release that succeeded, Stop returns nil and runs nothing.

type Config

type Config struct {
	// BucketName is the NATS ObjectStore bucket. Empty means "MESSAGES".
	BucketName string `json:"bucket_name"`

	// InstanceName is stamped into every StorageReference.StorageInstance the
	// Store returns: the name a storeregistry.Registry resolves back to this
	// Store (ADR-063). Empty means BucketName. Set by the owning code, not by an
	// operator.
	InstanceName string `json:"-"`

	// Logger receives the retention guard's warning when construction strips a
	// TTL or size limit from the bucket. Nil means slog.Default().
	Logger *slog.Logger `json:"-"`
}

Config configures a Store.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a storage.StreamableStore over one NATS ObjectStore bucket. It is safe for concurrent use. It keeps no copy of any object: every read goes to NATS. It holds no context and runs no goroutine, so it has nothing to close.

func NewStoreWithConfig

func NewStoreWithConfig(ctx context.Context, client *natsclient.Client, cfg Config) (*Store, error)

NewStoreWithConfig creates the configured bucket, or opens it when it exists, and removes any TTL or size limit from it (see the package doc). It records no metrics.

Example

ExampleNewStoreWithConfig stores, reads, lists and deletes one object. It has no Output comment because it needs a running NATS server with JetStream, so `go test` compiles it and does not run it.

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"os"

	"github.com/c360studio/semengine/natsclient"
	"github.com/c360studio/semengine/storage"
	"github.com/c360studio/semengine/storage/objectstore"
)

func main() {
	ctx := context.Background()

	// NATS_URL is the URL of a NATS server with JetStream enabled.
	client, err := natsclient.NewClient(os.Getenv("NATS_URL"))
	if err != nil {
		log.Fatal(err)
	}
	if err := client.Connect(ctx); err != nil {
		log.Fatal(err)
	}
	defer client.Close(ctx)

	// The bucket is created if it is missing. The Store has no Close.
	store, err := objectstore.NewStoreWithConfig(ctx, client, objectstore.Config{BucketName: "MESSAGES"})
	if err != nil {
		log.Fatal(err)
	}

	// A second Put to the same key replaces the object.
	if err := store.Put(ctx, "events/a", []byte("hello")); err != nil {
		log.Fatal(err)
	}

	data, err := store.Get(ctx, "events/a")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(data))

	// List returns the matching names sorted.
	keys, err := store.List(ctx, "events/")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(keys)

	// Delete removes the object; deleting a key with no object also returns nil.
	if err := store.Delete(ctx, "events/a"); err != nil {
		log.Fatal(err)
	}

	// A missing object matches storage.ErrObjectNotFound.
	if _, err := store.Get(ctx, "events/a"); errors.Is(err, storage.ErrObjectNotFound) {
		fmt.Println("gone")
	}
}

func NewStoreWithConfigAndMetrics

func NewStoreWithConfigAndMetrics(
	ctx context.Context, client *natsclient.Client, cfg Config, registry *metric.MetricsRegistry,
) (*Store, error)

NewStoreWithConfigAndMetrics is NewStoreWithConfig recording the Store's metrics on registry. A nil registry records nothing.

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, key string) (err error)

Delete removes the object at key. A key with no object returns nil.

func (*Store) Get

func (s *Store) Get(ctx context.Context, key string) (_ []byte, err error)

Get returns the bytes stored at key. A missing object returns an error matching storage.ErrObjectNotFound, which is not transient.

func (*Store) InstanceName

func (s *Store) InstanceName() string

InstanceName returns the value the Store stamps into StorageReference.StorageInstance: Config.InstanceName, or the bucket name.

func (*Store) List

func (s *Store) List(ctx context.Context, prefix string) (_ []string, err error)

List returns the names of the objects that start with prefix, sorted, and an empty slice when none do. It reads every object's metadata from NATS and filters here, so its cost grows with the number of objects in the bucket.

func (*Store) Open

func (s *Store) Open(ctx context.Context, key string) (_ io.ReadCloser, err error)

Open returns a reader that streams the object at key in chunks. The caller closes it. A missing object returns an error matching storage.ErrObjectNotFound, which is not transient.

func (*Store) Put

func (s *Store) Put(ctx context.Context, key string, data []byte) (err error)

Put stores data at key, replacing any object already there. NATS keeps no earlier version.

func (*Store) StoreContent

func (s *Store) StoreContent(
	ctx context.Context, cs message.ContentStorable,
) (_ *message.StorageReference, err error)

StoreContent stores the body of cs, the RawContent field its "body" role names, as raw UTF-8 bytes, and returns a reference to it: the Store's instance name, the object's key, text/plain and the body's size. Opening the reference returns the body; there is no envelope (ADR-063).

It refuses, as invalid input and before writing anything, an invalid entity ID and any other content: a non-empty RawContent value under a field that is not the body's or that a role other than "body" also names, or a non-empty binary field of a message.BinaryStorable. The error names the refused fields and roles. Non-body text belongs in the graph as triples, which the producer emits. A payload with no content stores an empty object.

type StoredMessage

type StoredMessage struct {
	// contains filtered or unexported fields
}

StoredMessage is a message.Storable for content the Component has stored: the entity and triples of the original message, and the reference that opens to its stored body. A reader resolves the reference through the store registry (ADR-063).

Its fields are unexported, so a received StoredMessage cannot be changed; MarshalJSON and UnmarshalJSON spell its wire form.

func NewStoredMessage

func NewStoredMessage(graphable graph.Graphable, storageRef *message.StorageReference, messageType string) *StoredMessage

NewStoredMessage returns a StoredMessage for graphable's entity and triples, stored at storageRef now. messageType is the original message's type.

func (*StoredMessage) EntityID

func (s *StoredMessage) EntityID() string

EntityID implements graph.Graphable.

func (*StoredMessage) MarshalJSON

func (s *StoredMessage) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*StoredMessage) MessageType

func (s *StoredMessage) MessageType() string

MessageType returns the original message's type.

func (*StoredMessage) Schema

func (s *StoredMessage) Schema() message.Type

Schema implements message.Payload: storage.stored.v1.

func (*StoredMessage) StorageRef

func (s *StoredMessage) StorageRef() *message.StorageReference

StorageRef implements message.Storable.

func (*StoredMessage) StoredAt

func (s *StoredMessage) StoredAt() time.Time

StoredAt returns when the message was stored.

func (*StoredMessage) Triples

func (s *StoredMessage) Triples() []message.Triple

Triples implements graph.Graphable.

func (*StoredMessage) UnmarshalJSON

func (s *StoredMessage) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler.

func (*StoredMessage) Validate

func (s *StoredMessage) Validate() error

Validate implements message.Payload. It refuses an entity ID pkg/types refuses, keeping that error's code, and a missing reference.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL