storage

package
v0.1.0-alpha.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Overview

Package storage provides pluggable backend interfaces for storage operations.

Overview

The storage package defines the core Store interface and related abstractions for persisting binary data with hierarchical key-value semantics. It provides a clean, implementation-agnostic API that supports multiple storage backends:

  • NATS JetStream ObjectStore (storage/objectstore; Put replaces, no versions kept)
  • AWS S3 or MinIO (mutable, object storage) - future
  • PostgreSQL (mutable, relational) - future

Core Concepts

Store Interface:

The Store interface uses a simple key-value pattern where:

  • Keys are strings (hierarchical paths supported via "/" separators)
  • Values are binary data ([]byte) - supports any format
  • Operations are context-aware for cancellation and timeouts

This simplicity enables a wide range of use cases: JSON messages, video files, images, or any binary data can be stored with the same interface.

Architecture Decisions

Simple Key-Value Model:

The Store interface intentionally uses a simple key-value model rather than richer abstractions like queries, indexes, or transactions. This decision:

  • Keeps implementations simple and focused
  • Allows diverse backends (object stores, databases, filesystems)
  • Pushes complex logic to higher layers where it belongs
  • Enables easy testing with mock implementations

Alternative considered: Query-based interface with filtering Rejected because: Too complex, limits backend options, semantic queries belong in semantic layers not generic storage.

Hierarchical Keys via "/" Convention:

Keys support hierarchical organization using "/" separators:

  • "video/sensor-123/2024-10-08.mp4"
  • "events/robotics/entity-456/2024-10-08T10:30:00Z"

This convention (not enforced by interface):

  • Works naturally with object stores (S3, NATS ObjectStore)
  • Enables prefix-based listing and filtering
  • Mirrors filesystem-like organization users expect

No Forced Immutability:

The Store interface allows implementations to choose mutability semantics:

  • NATS ObjectStore (objectstore.Store): Put() replaces the object at the key and keeps no earlier version (observed with nats.go v1.54.0, probe P4 on PR #432 comment 6445520357; the library does not promise it)
  • Mutable stores (S3, SQL): Put() overwrites existing values

This flexibility enables diverse backends while documenting behavior per implementation.

Context Everywhere:

All Store operations accept context.Context as the first parameter. This enables:

  • Cancellation of long-running operations
  • Timeout enforcement per operation
  • Request-scoped tracing and logging
  • Graceful shutdown of in-flight requests

Usage Examples

Basic Store Usage:

// Create NATS ObjectStore backend
// The Store holds no copy and no goroutine, so there is nothing to close.
store, err := objectstore.NewStoreWithConfig(ctx, natsClient, objectstore.Config{BucketName: "MESSAGES"})
if err != nil {
    log.Fatal(err)
}

// Store JSON message
msgJSON := []byte(`{"sensor": "temp-123", "value": 22.5}`)
key := "events/sensors/temp-123/2024-10-08T10:30:00Z"
err = store.Put(ctx, key, msgJSON)

// Retrieve message
data, err := store.Get(ctx, key)

// List all sensor events
keys, err := store.List(ctx, "events/sensors/")

// Delete old data (a key with no object returns nil)
err = store.Delete(ctx, key)

Performance Characteristics

The performance of Store operations depends entirely on the backend implementation:

NATS ObjectStore (storage/objectstore):

  • Put: O(1) - direct write to JetStream
  • Get: O(1) - direct read, no cache
  • List: O(n) - reads every object's metadata, filters and sorts client-side
  • Delete: O(1) - removes the object

Future S3/MinIO Backend:

  • Put: O(1) - direct S3 upload
  • Get: O(1) - direct S3 download with caching
  • List: O(n) - S3 ListObjects with pagination
  • Delete: O(1) - S3 DeleteObject

Future PostgreSQL Backend:

  • Put: O(1) - INSERT or UPDATE
  • Get: O(log n) - indexed key lookup
  • List: O(n) - prefix query with index scan
  • Delete: O(log n) - indexed DELETE

Memory:

Store implementations should have bounded memory usage:

  • Get operations: O(message_size) for returned data
  • List operations: O(num_matching_keys) for key list
  • Put operations: O(message_size) during write

Thread Safety

All Store implementations MUST be safe for concurrent use from multiple goroutines. This is a contract requirement of the Store interface.

Example implementations demonstrate thread safety:

  • objectstore.Store: checked under the race detector (TestStoreConcurrentUseUnderRace)

Integration with Components

The storage/objectstore package has the Store and a Component that wraps it. The Component subscribes to NATS inputs, stores the body of each message.ContentStorable it receives through the Store, and publishes a StoredMessage carrying the body's reference on its "stored" output. It offers the Store to the component manager under the instance name "objectstore", which is how a reader resolves a reference (storage/storeregistry). Its configuration, delivery guarantees and health reporting are described in storage/objectstore's package documentation.

Error Handling

Store implementations should return errors classified by the framework's error package:

  • errs.WrapInvalid: Invalid keys, malformed input
  • errs.WrapTransient: Network timeouts, temporary failures
  • errs.WrapFatal: Programming errors, nil pointers

Callers can distinguish error types for appropriate retry/recovery strategies.

Testing

The storage package emphasizes testing with real backends:

  • Use testcontainers for NATS JetStream
  • Avoid mocks - test actual storage behavior
  • Test with race detector enabled
  • Test context cancellation and timeout behavior

Example test pattern:

func TestStore_PutGet(t *testing.T) {
    natsClient := getSharedNATSClient(t) // Real NATS
    store, err := objectstore.NewStoreWithConfig(ctx, natsClient, objectstore.Config{BucketName: "TEST"})
    require.NoError(t, err)

    // Test actual storage operations
    data := []byte("test data")
    err = store.Put(ctx, "test-key", data)
    require.NoError(t, err)

    retrieved, err := store.Get(ctx, "test-key")
    require.NoError(t, err)
    assert.Equal(t, data, retrieved)
}

See Also

  • storage/objectstore: NATS ObjectStore implementation and its Component
  • component: Component interface and lifecycle
  • message: Message types for semantic storage

Package storage provides pluggable backend interfaces for storage operations.

Index

Constants

This section is empty.

Variables

View Source
var ErrObjectNotFound = errors.New("object not found")

ErrObjectNotFound is the backend-agnostic sentinel a Store.Get returns (wrapped with %w) when the key resolved to NO stored object — as opposed to a transient backend fault. Callers match it with errors.Is to distinguish "the object is absent" (e.g. content-addressed evidence that was reclaimed, #600) from "the read faulted", which is a genuine error worth retrying/alerting. A not-found is deliberately NOT classified as transient: retrying will not conjure the object.

Functions

This section is empty.

Types

type Store

type Store interface {
	// Put stores binary data at the specified key.
	// If the key already exists, behavior is implementation-specific:
	//   - NATS ObjectStore (objectstore.Store) replaces the object and keeps no
	//     earlier version. Observed with nats.go v1.54.0 and nats-server v2.15.1
	//     (probe P4, PR #432 comment 6445520357); the library does not promise it.
	//   - Mutable stores (S3, SQL) overwrite the existing value
	//
	// The data parameter accepts any binary format:
	//   - JSON-encoded messages
	//   - Video files (MP4, etc.)
	//   - Images (JPEG, PNG, etc.)
	//   - Any []byte data
	Put(ctx context.Context, key string, data []byte) error

	// Get retrieves binary data for the specified key.
	// Returns an error if the key does not exist.
	//
	// The returned []byte should be interpreted by the caller based on
	// their knowledge of what was stored (JSON, video, etc.).
	Get(ctx context.Context, key string) ([]byte, error)

	// List returns all keys matching the specified prefix.
	// The prefix parameter supports hierarchical key patterns:
	//   - "" (empty) lists all keys
	//   - "video/" lists all keys starting with "video/"
	//   - "video/sensor-123/" lists keys for a specific sensor
	//
	// Keys are returned in lexicographic order.
	// Returns an empty slice if no keys match the prefix.
	List(ctx context.Context, prefix string) ([]string, error)

	// Delete removes the data at the specified key.
	// Returns nil if the key doesn't exist (idempotent operation).
	//
	// objectstore.Store removes the object; it keeps no earlier versions to
	// remove (see Put).
	Delete(ctx context.Context, key string) error
}

Store is the pluggable backend interface for storage operations.

Each storage component instance creates its own Store implementation with its own configuration (bucket name, connection string, etc.). Multiple Store instances can run concurrently, each backing a different storage component.

The Store interface uses a simple key-value pattern where:

  • Keys are strings (hierarchical paths supported via "/" separators)
  • Values are binary data ([]byte) - supports JSON, videos, images, any binary format
  • Operations are context-aware for cancellation and timeouts

Example implementations:

  • objectstore.Store: NATS JetStream ObjectStore backend
  • s3store.Store: AWS S3 or MinIO backend (future)
  • sqlstore.Store: PostgreSQL with (key, data) table (future)

Thread Safety: All Store implementations must be safe for concurrent use from multiple goroutines.

Example Usage:

// Create a NATS ObjectStore backend; the Store has nothing to close
store, err := objectstore.NewStoreWithConfig(ctx, natsClient, objectstore.Config{BucketName: "MESSAGES"})

// Store video file
videoData := []byte{...}
err = store.Put(ctx, "video/sensor-123/2024-10-08.mp4", videoData)

// Retrieve video
data, err := store.Get(ctx, "video/sensor-123/2024-10-08.mp4")

// List all videos for a sensor
keys, err := store.List(ctx, "video/sensor-123/")

type StreamableStore

type StreamableStore interface {
	Store

	// Open returns a streaming reader for the content at key.
	// The caller MUST close the reader when done.
	Open(ctx context.Context, key string) (io.ReadCloser, error)
}

StreamableStore extends Store with streaming read support. Implementations that support chunked/streamed reads (NATS ObjectStore, filesystem) implement this for large content without loading everything into memory. Backends without native streaming wrap Get() bytes in an io.NopCloser(bytes.NewReader(data)).

Directories

Path Synopsis
Package objectstore is a storage.StreamableStore over one NATS ObjectStore bucket.
Package objectstore is a storage.StreamableStore over one NATS ObjectStore bucket.
Package storeregistry provides a concurrency-safe registry mapping a message.StorageReference.StorageInstance to the live store that backs it (ADR-063).
Package storeregistry provides a concurrency-safe registry mapping a message.StorageReference.StorageInstance to the live store that backs it (ADR-063).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL