Documentation
¶
Overview ¶
Package objectstore is a storage.StreamableStore over one NATS ObjectStore bucket.
NewStoreWithConfig creates the bucket, or opens it when it exists. The bucket has no TTL and no size limit: graph entities point at its objects for as long as they live (ADR-068). If an existing bucket's backing stream (OBJ_<bucket>) has either, construction removes it, logs a warning naming the bucket and what was removed, and fails with an error wrapping natsclient.ErrGraphBucketRetention if it cannot.
Put replaces the object at a key; no earlier version is kept. Get and Open of a missing key return an error matching storage.ErrObjectNotFound. Delete of a missing key returns nil. List returns the matching names sorted, reading every object's metadata to do so. The Store keeps no copy of any object and has nothing to close.
StoreContent stores a message.ContentStorable's body as raw bytes and returns a message.StorageReference that opens to that body. It refuses any other content.
Component ¶
Component wraps the Store with NATS ports. Register adds its factory to a component.Registry under the name "objectstore". Each message on an input port is decoded through the payload registry; if its payload is a message.ContentStorable, the body is stored through StoreContent and a StoredMessage (payload type storage.stored.v1, which RegisterPayloads registers) carrying the entity, its triples and the body's reference is published on the "stored" output port. Any other message is refused: the Component stores bodies only.
The configuration has two keys, both optional: bucket_name (default MESSAGES) and ports. An unknown key is refused at construction, naming it. With no ports the Component has one input, "write", a core NATS subscription on storage.objectstore.write, and one output, "stored", on storage.objectstore.stored. A configured output must be named "stored", and a Component with an input must have it. An input may be core NATS or JetStream. A JetStream input must name a stream that exists: Start does not create it, waits for it, and fails if it never appears. A JetStream input that declares consumer_name is refused, because the durable consumer is named from the subject; its ack_wait and the other consumer settings are used.
The default core NATS input keeps nothing: a message that is refused, or whose write fails, is logged at error level, counted in Health's ErrorCount, and gone. A JetStream input is at-least-once: a delivery is acknowledged only after its body is stored and its reference published. A refusal ends the delivery (Term); a failed store or publish is negatively acknowledged (NAK) for redelivery, after 30 seconds, or 5 seconds when the handler's context ended. Every refusal and failure is logged and counted. A redelivery after a failed publish stores the body again under a new key, so it can leave one stored object that no reference names. A delivery that reaches the consumer's MaxDeliver is not delivered again and the Component reports nothing about it.
The Component offers its Store, under the instance name "objectstore", through ProvidedStores. That one name is stamped into every reference. A process runs one such Component: a second is refused when the component manager starts it.
If a JetStream input's consume loop ends while the Component runs, or its durable consumer cannot be found on the server, Health reports the Component not healthy, naming the input. A lost consume loop is not rebound: the recovery is to restart the process. The check runs every five seconds, so the report can take that long. Whether the server's deleting a consumer ends the consume loop is not promised (observed once with nats.go v1.54.0). The lookup by name catches the cases where it does not.
Stop drains every input and waits for its running handlers to return, through natsclient's WaitConsumerHandlers. The handlers keep running after the context given to Start ends; only Stop ends them.
Index ¶
- func Register(registry *component.Registry) error
- func RegisterPayloads(reg *payloadregistry.Registry) error
- type Component
- func (c *Component) DataFlow() component.FlowMetrics
- func (c *Component) Health() component.HealthStatus
- func (c *Component) Initialize() error
- func (c *Component) InputPorts() []component.Port
- func (c *Component) OutputPorts() []component.Port
- func (c *Component) ProvidedStores() map[string]storage.StreamableStore
- func (c *Component) Start(ctx context.Context) error
- func (c *Component) Stop(ctx context.Context) error
- type Config
- type Store
- func (s *Store) Delete(ctx context.Context, key string) (err error)
- func (s *Store) Get(ctx context.Context, key string) (_ []byte, err error)
- func (s *Store) InstanceName() string
- func (s *Store) List(ctx context.Context, prefix string) (_ []string, err error)
- func (s *Store) Open(ctx context.Context, key string) (_ io.ReadCloser, err error)
- func (s *Store) Put(ctx context.Context, key string, data []byte) (err error)
- func (s *Store) StoreContent(ctx context.Context, cs message.ContentStorable) (_ *message.StorageReference, err error)
- type StoredMessage
- func (s *StoredMessage) EntityID() string
- func (s *StoredMessage) MarshalJSON() ([]byte, error)
- func (s *StoredMessage) MessageType() string
- func (s *StoredMessage) Schema() message.Type
- func (s *StoredMessage) StorageRef() *message.StorageReference
- func (s *StoredMessage) StoredAt() time.Time
- func (s *StoredMessage) Triples() []message.Triple
- func (s *StoredMessage) UnmarshalJSON(data []byte) error
- func (s *StoredMessage) Validate() error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RegisterPayloads ¶
func RegisterPayloads(reg *payloadregistry.Registry) error
RegisterPayloads registers StoredMessage as storage.stored.v1 with reg. Importing this package registers nothing: the consumer's composition root calls RegisterPayloads on the registry it gives its decoders, and a test on the registry it builds.
Types ¶
type Component ¶
type Component struct {
// contains filtered or unexported fields
}
Component stores the bodies of the messages on its inputs in a NATS ObjectStore bucket and publishes a StoredMessage carrying each body's reference on its stored output (ADR-063).
Only a message.ContentStorable payload's body is stored, through the Store's StoreContent; every other message is refused. A refusal or a failed write is logged at error level and counted in Health's ErrorCount. A core-NATS input is at-most-once: such a message is then gone. A JetStream input is at-least-once: a refusal is terminated, a failure NAKed for redelivery, and a delivery is acked only after its body is stored and its reference published.
func (*Component) DataFlow ¶
func (c *Component) DataFlow() component.FlowMetrics
DataFlow reports the lifecycle shell's counts.
func (*Component) Health ¶
func (c *Component) Health() component.HealthStatus
Health reports the lifecycle shell's verdict over the Component's one stuck state: a JetStream input whose consume loop ended while running, which only a process restart clears, or whose consumer the latest input check could not find (checkInputs). ErrorCount is reported and never lowers Healthy.
func (*Component) Initialize ¶
Initialize marks the Component initialized (no I/O). The factory validated the configuration.
func (*Component) InputPorts ¶
InputPorts returns the input ports, fixed at construction.
func (*Component) OutputPorts ¶
OutputPorts returns the output ports, fixed at construction: stored, if configured, and store-provide.
func (*Component) ProvidedStores ¶
func (c *Component) ProvidedStores() map[string]storage.StreamableStore
ProvidedStores offers the running Component's Store under the instance name it stamps into every reference, for the component manager to register (ADR-063). It returns nil before Start and after Stop.
func (*Component) Start ¶
Start opens the Store and binds every input, through the lifecycle shell: a nil or ended context and a second Start are refused before anything is acquired, and a Start that fails is rolled back through release; a rollback that fails leaves release pending for the next Stop. A Component with no NATS client is refused before the shell is consulted.
ctx bounds Start only: opening the bucket, waiting for each JetStream input's stream and binding the inputs run under it, and a ctx that ends first fails Start. The inputs' handlers run under contexts natsclient owns, and the input check under a run context start creates and release ends, so ending ctx after Start returned stops nothing (ADR-113).
type Config ¶
type Config struct {
// BucketName is the NATS ObjectStore bucket. Empty means "MESSAGES".
BucketName string `json:"bucket_name"`
// InstanceName is stamped into every StorageReference.StorageInstance the
// Store returns: the name a storeregistry.Registry resolves back to this
// Store (ADR-063). Empty means BucketName. Set by the owning code, not by an
// operator.
InstanceName string `json:"-"`
// Logger receives the retention guard's warning when construction strips a
// TTL or size limit from the bucket. Nil means slog.Default().
Logger *slog.Logger `json:"-"`
}
Config configures a Store.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a storage.StreamableStore over one NATS ObjectStore bucket. It is safe for concurrent use. It keeps no copy of any object: every read goes to NATS. It holds no context and runs no goroutine, so it has nothing to close.
func NewStoreWithConfig ¶
NewStoreWithConfig creates the configured bucket, or opens it when it exists, and removes any TTL or size limit from it (see the package doc). It records no metrics.
Example ¶
ExampleNewStoreWithConfig stores, reads, lists and deletes one object. It has no Output comment because it needs a running NATS server with JetStream, so `go test` compiles it and does not run it.
package main
import (
"context"
"errors"
"fmt"
"log"
"os"
"github.com/c360studio/semengine/natsclient"
"github.com/c360studio/semengine/storage"
"github.com/c360studio/semengine/storage/objectstore"
)
func main() {
ctx := context.Background()
// NATS_URL is the URL of a NATS server with JetStream enabled.
client, err := natsclient.NewClient(os.Getenv("NATS_URL"))
if err != nil {
log.Fatal(err)
}
if err := client.Connect(ctx); err != nil {
log.Fatal(err)
}
defer client.Close(ctx)
// The bucket is created if it is missing. The Store has no Close.
store, err := objectstore.NewStoreWithConfig(ctx, client, objectstore.Config{BucketName: "MESSAGES"})
if err != nil {
log.Fatal(err)
}
// A second Put to the same key replaces the object.
if err := store.Put(ctx, "events/a", []byte("hello")); err != nil {
log.Fatal(err)
}
data, err := store.Get(ctx, "events/a")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
// List returns the matching names sorted.
keys, err := store.List(ctx, "events/")
if err != nil {
log.Fatal(err)
}
fmt.Println(keys)
// Delete removes the object; deleting a key with no object also returns nil.
if err := store.Delete(ctx, "events/a"); err != nil {
log.Fatal(err)
}
// A missing object matches storage.ErrObjectNotFound.
if _, err := store.Get(ctx, "events/a"); errors.Is(err, storage.ErrObjectNotFound) {
fmt.Println("gone")
}
}
Output:
func NewStoreWithConfigAndMetrics ¶
func NewStoreWithConfigAndMetrics( ctx context.Context, client *natsclient.Client, cfg Config, registry *metric.MetricsRegistry, ) (*Store, error)
NewStoreWithConfigAndMetrics is NewStoreWithConfig recording the Store's metrics on registry. A nil registry records nothing.
func (*Store) Get ¶
Get returns the bytes stored at key. A missing object returns an error matching storage.ErrObjectNotFound, which is not transient.
func (*Store) InstanceName ¶
InstanceName returns the value the Store stamps into StorageReference.StorageInstance: Config.InstanceName, or the bucket name.
func (*Store) List ¶
List returns the names of the objects that start with prefix, sorted, and an empty slice when none do. It reads every object's metadata from NATS and filters here, so its cost grows with the number of objects in the bucket.
func (*Store) Open ¶
Open returns a reader that streams the object at key in chunks. The caller closes it. A missing object returns an error matching storage.ErrObjectNotFound, which is not transient.
func (*Store) Put ¶
Put stores data at key, replacing any object already there. NATS keeps no earlier version.
func (*Store) StoreContent ¶
func (s *Store) StoreContent( ctx context.Context, cs message.ContentStorable, ) (_ *message.StorageReference, err error)
StoreContent stores the body of cs, the RawContent field its "body" role names, as raw UTF-8 bytes, and returns a reference to it: the Store's instance name, the object's key, text/plain and the body's size. Opening the reference returns the body; there is no envelope (ADR-063).
It refuses, as invalid input and before writing anything, an invalid entity ID and any other content: a non-empty RawContent value under a field that is not the body's or that a role other than "body" also names, or a non-empty binary field of a message.BinaryStorable. The error names the refused fields and roles. Non-body text belongs in the graph as triples, which the producer emits. A payload with no content stores an empty object.
type StoredMessage ¶
type StoredMessage struct {
// contains filtered or unexported fields
}
StoredMessage is a message.Storable for content the Component has stored: the entity and triples of the original message, and the reference that opens to its stored body. A reader resolves the reference through the store registry (ADR-063).
Its fields are unexported, so a received StoredMessage cannot be changed; MarshalJSON and UnmarshalJSON spell its wire form.
func NewStoredMessage ¶
func NewStoredMessage(graphable graph.Graphable, storageRef *message.StorageReference, messageType string) *StoredMessage
NewStoredMessage returns a StoredMessage for graphable's entity and triples, stored at storageRef now. messageType is the original message's type.
func (*StoredMessage) EntityID ¶
func (s *StoredMessage) EntityID() string
EntityID implements graph.Graphable.
func (*StoredMessage) MarshalJSON ¶
func (s *StoredMessage) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler.
func (*StoredMessage) MessageType ¶
func (s *StoredMessage) MessageType() string
MessageType returns the original message's type.
func (*StoredMessage) Schema ¶
func (s *StoredMessage) Schema() message.Type
Schema implements message.Payload: storage.stored.v1.
func (*StoredMessage) StorageRef ¶
func (s *StoredMessage) StorageRef() *message.StorageReference
StorageRef implements message.Storable.
func (*StoredMessage) StoredAt ¶
func (s *StoredMessage) StoredAt() time.Time
StoredAt returns when the message was stored.
func (*StoredMessage) Triples ¶
func (s *StoredMessage) Triples() []message.Triple
Triples implements graph.Graphable.
func (*StoredMessage) UnmarshalJSON ¶
func (s *StoredMessage) UnmarshalJSON(data []byte) error
UnmarshalJSON implements json.Unmarshaler.
func (*StoredMessage) Validate ¶
func (s *StoredMessage) Validate() error
Validate implements message.Payload. It refuses an entity ID pkg/types refuses, keeping that error's code, and a missing reference.