state

package
v0.0.1-alpha.19 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package state defines the Store interface and provides four implementations:

  • MemoryStore — in-process maps; lost on restart; fastest; best for tests & CI.
  • SQLiteStore — synchronous SQLite writes (persistent mode).
  • WALStore — memory reads + append-log durability with replay on startup.
  • HybridStore — memory reads + async SQLite flush; default for local development.
  • NamespacedStore — dispatcher that routes operations to per-service stores.

All service handlers receive a Store — they never know or care which implementation is backing them. This is Go's equivalent of programming to an interface rather than a concrete type.

TypeScript analogy: 

interface Store {
  get(namespace: string, key: string): Promise<string | null>
  set(namespace: string, key: string, value: string): Promise<void>
  delete(namespace: string, key: string): Promise<void>
  list(namespace: string, prefix: string): Promise<string[]>
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type HybridStore 

type HybridStore struct {
	// contains filtered or unexported fields
}

HybridStore serves all reads from an in-memory map (memory speed) and asynchronously flushes writes to SQLite at a configurable interval.

On startup it opens the SQLite file AND seeds the in-memory layer from it in a background goroutine. The constructor returns immediately. Reads use indexed SQLite fallback until seeding completes, then switch to memory; writes are accepted immediately and take precedence over loaded state. DB() blocks only until the SQLite handle is ready. This keeps full-table restore and the modernc SQLite driver's cold-start cost off the startup path.

Durability trade-off: up to one flushInterval of writes may be lost if the process exits uncleanly (kill -9 / OOM). Close() always performs a final synchronous flush before returning, so clean shutdowns are fully durable.

func NewHybridStore 

func NewHybridStore(dataDir string, flushInterval time.Duration) (*HybridStore, error)

NewHybridStore creates a HybridStore backed by a SQLite file in dataDir. SQLite is opened and existing data loaded in a background goroutine; the constructor returns immediately. Reads fall back to SQLite until the cache is loaded. Writes are accepted right away and take precedence over loaded state.

func NewHybridStoreWithLogger 

func NewHybridStoreWithLogger(dataDir string, flushInterval time.Duration, logger *zap.Logger) (*HybridStore, error)

NewHybridStoreWithLogger creates a HybridStore and emits structured timing diagnostics for startup seeding and flushes when logger is non-nil.

func (*HybridStore) Close 

func (s *HybridStore) Close() error

Close stops the background flush goroutine, performs a final synchronous flush of all pending dirty entries to SQLite, then closes the database.

func (*HybridStore) DB 

func (s *HybridStore) DB() *sql.DB

DB returns the underlying *sql.DB, satisfying SQLiteDBProvider. Blocks until the background SQLite open has completed. Returns nil if the open failed — callers that hit this path should have already seen the load error propagate via a Get/List/Scan call.

func (*HybridStore) Delete 

func (s *HybridStore) Delete(ctx context.Context, namespace, key string) error

Delete removes from memory immediately and marks the entry as a tombstone.

func (*HybridStore) DeletePrefix 

func (s *HybridStore) DeletePrefix(ctx context.Context, namespace, prefix string) error

DeletePrefix removes matching keys from memory immediately and records tombstones so the next flush removes them from SQLite.

func (*HybridStore) Get 

func (s *HybridStore) Get(ctx context.Context, namespace, key string) (string, bool, error)

Get serves from memory after the initial seed completes. During seed it falls back to SQLite so persisted state stays visible without blocking on a full-table cache warmup.

func (*HybridStore) List 

func (s *HybridStore) List(ctx context.Context, namespace, prefix string) ([]string, error)

List serves from memory after the initial seed completes. During seed it uses SQLite plus the dirty overlay to avoid first-request stalls.

func (*HybridStore) Scan 

func (s *HybridStore) Scan(ctx context.Context, namespace, prefix string) ([]KV, error)

Scan serves from memory after the initial seed completes. During seed it uses SQLite plus the dirty overlay to avoid first-request stalls.

func (*HybridStore) Set 

func (s *HybridStore) Set(ctx context.Context, namespace, key, value string) error

Set writes to memory immediately and marks the entry dirty for the next flush.

func (*HybridStore) WaitReady 

func (s *HybridStore) WaitReady(ctx context.Context) error

WaitReady blocks until the background SQLite open and migration has completed, satisfying state.ReadyAwaiter. Returns nil when the store is ready for reads, or ctx.Err() if the context is cancelled first. Callers that need to guarantee persisted data is visible (e.g. startup reload routines) should call this before performing a full Scan.

type KV 

type KV struct {
	Key   string
	Value string
}

KV is a key-value pair returned by Scan.

type MemoryStore 

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is the default Store implementation. All data lives in-process and is lost when the process exits. This is the right default for local development: zero configuration, instant startup, deterministic test state.

Performance characteristics:

  • Get/Set/Delete: O(log n) per namespace, protected by RWMutex
  • List/Scan: O(log n + m) prefix scan via btree (m = matching keys)
  • RWMutex allows many concurrent readers OR one exclusive writer

Memory note: MemoryStore does not enforce size limits. For local dev and CI the footprint is naturally bounded by test duration. Overcast is not designed for production use where unbounded growth would be a concern.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an initialised MemoryStore.

func (*MemoryStore) Close 

func (s *MemoryStore) Close() error

func (*MemoryStore) Delete 

func (s *MemoryStore) Delete(_ context.Context, namespace, key string) error

func (*MemoryStore) DeletePrefix 

func (s *MemoryStore) DeletePrefix(_ context.Context, namespace, prefix string) error

DeletePrefix removes all keys with prefix under a single write lock.

func (*MemoryStore) Get 

func (s *MemoryStore) Get(_ context.Context, namespace, key string) (string, bool, error)

func (*MemoryStore) Len 

func (s *MemoryStore) Len() int

Len returns the number of entries. Used by /_debug/state and tests.

func (*MemoryStore) List 

func (s *MemoryStore) List(_ context.Context, namespace, prefix string) ([]string, error)

func (*MemoryStore) Reset 

func (s *MemoryStore) Reset()

Reset wipes all stored data atomically.

func (*MemoryStore) Scan 

func (s *MemoryStore) Scan(_ context.Context, namespace, prefix string) ([]KV, error)

Scan returns all key-value pairs whose keys start with prefix, under a single RLock. This is the preferred way to read a batch of items — it avoids N individual Get calls each acquiring their own lock.

func (*MemoryStore) Set 

func (s *MemoryStore) Set(_ context.Context, namespace, key, value string) error

type NamespacedStore 

type NamespacedStore struct {
	// contains filtered or unexported fields
}

NamespacedStore routes Store operations to different backends based on the service prefix of the namespace argument (the segment before the first ":"). 

namespace "sqs:queues"   → service prefix "sqs"
namespace "s3:objects"   → service prefix "s3"

Operations whose namespace has no matching override are forwarded to the default store. This allows per-service storage modes with a single Store interface visible to the rest of the codebase.

func NewNamespacedStore 

func NewNamespacedStore(defaultStore Store, routes map[string]Store) *NamespacedStore

NewNamespacedStore creates a NamespacedStore. routes maps service prefixes (e.g. "s3", "sqs") to their dedicated Store. Services absent from routes use defaultStore.

func (*NamespacedStore) Close 

func (s *NamespacedStore) Close() error

Close closes the default store and all per-service stores. Each underlying store is closed exactly once even if it serves multiple namespace prefixes.

func (*NamespacedStore) Delete 

func (s *NamespacedStore) Delete(ctx context.Context, namespace, key string) error

func (*NamespacedStore) DeletePrefix 

func (s *NamespacedStore) DeletePrefix(ctx context.Context, namespace, prefix string) error

func (*NamespacedStore) Get 

func (s *NamespacedStore) Get(ctx context.Context, namespace, key string) (string, bool, error)

func (*NamespacedStore) List 

func (s *NamespacedStore) List(ctx context.Context, namespace, prefix string) ([]string, error)

func (*NamespacedStore) Scan 

func (s *NamespacedStore) Scan(ctx context.Context, namespace, prefix string) ([]KV, error)

func (*NamespacedStore) Set 

func (s *NamespacedStore) Set(ctx context.Context, namespace, key, value string) error

type PrefixDeleter 

type PrefixDeleter interface {
	DeletePrefix(ctx context.Context, namespace, prefix string) error
}

PrefixDeleter is an optional Store extension for deleting a key range without first reading values. Callers should type-assert this for large purges.

type ReadyAwaiter 

type ReadyAwaiter interface {
	// WaitReady blocks until the store's background initialisation is
	// complete or ctx is cancelled. Returns nil when the store is ready.
	WaitReady(ctx context.Context) error
}

ReadyAwaiter is implemented by stores that have an asynchronous initialisation phase (e.g. HybridStore, which opens SQLite in the background). Callers that need to guarantee all persisted data is visible before reading — such as startup reload routines — should type-assert the Store to ReadyAwaiter and wait before scanning.

Stores that are always immediately ready (MemoryStore, SQLiteStore) do not need to implement this interface; callers must treat its absence as "already ready".

type SQLiteDBProvider 

type SQLiteDBProvider interface {
	DB() *sql.DB
}

SQLiteDBProvider is implemented by stores backed by SQLite (SQLiteStore, HybridStore). Services that need dedicated tables (e.g. DynamoDB items) can type-assert a Store to this interface to get direct DB access.

type SQLiteStore 

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore is the persistent Store implementation. State survives process restarts, stored in a single SQLite file under DataDir.

Schema is a single key-value table — deliberately simple. We don't need relational features; we need durable K/V storage with prefix scanning.

The file path is: <DataDir>/overcast.db.

Construction is non-blocking: `sql.Open` is cheap (it does not connect or migrate), and the schema migration runs in a background goroutine. The first DB-touching method (Get/Set/Delete/List/Scan/DB/loadAll) blocks on a ready channel until the migration finishes. This keeps the modernc/sqlite cold-start cost (~200–300ms for the first CREATE TABLE when the driver initialises its parser) off the critical startup path — the same approach used by HybridStore. Quick-start aware.

func NewSQLiteStore 

func NewSQLiteStore(dataDir string) (*SQLiteStore, error)

NewSQLiteStore opens (or creates) the SQLite database at dataDir/overcast.db with PRAGMA synchronous=NORMAL — writes are durable across OS crashes. The data directory is created if it doesn't exist.

Returns immediately; the schema migration runs in a background goroutine. The first call into any DB-touching method blocks until the migration completes (or returns the migration error if it failed).

func NewSQLiteStoreWAL 

func NewSQLiteStoreWAL(dataDir string) (*SQLiteStore, error)

NewSQLiteStoreWAL opens (or creates) the SQLite database at dataDir/overcast.db with PRAGMA synchronous=OFF for maximum write throughput. The OS may buffer writes and a power loss between writes can corrupt data — acceptable for local-only emulation but not for production use.

Returns immediately; the schema migration runs in a background goroutine (see NewSQLiteStore for details).

func (*SQLiteStore) Close 

func (s *SQLiteStore) Close() error

func (*SQLiteStore) DB 

func (s *SQLiteStore) DB() *sql.DB

DB returns the underlying *sql.DB so that service-specific stores can add their own tables to the same database file. Blocks until the background schema migration has finished.

If the migration failed, the returned *sql.DB is still non-nil (sql.Open itself succeeded) but service-specific schema setup against it will likely fail. The migration error is logged eagerly by runMigrate so it is visible in the daemon log even if no kv operation is ever issued. Callers must not close the returned DB — call Close() on the SQLiteStore.

func (*SQLiteStore) Delete 

func (s *SQLiteStore) Delete(ctx context.Context, namespace, key string) error

func (*SQLiteStore) DeletePrefix 

func (s *SQLiteStore) DeletePrefix(ctx context.Context, namespace, prefix string) error

func (*SQLiteStore) Get 

func (s *SQLiteStore) Get(ctx context.Context, namespace, key string) (string, bool, error)

func (*SQLiteStore) List 

func (s *SQLiteStore) List(ctx context.Context, namespace, prefix string) ([]string, error)

func (*SQLiteStore) Scan 

func (s *SQLiteStore) Scan(ctx context.Context, namespace, prefix string) ([]KV, error)

Scan returns all key-value pairs whose keys start with prefix in a single query — prefer this over List+Get when you need both keys and values.

func (*SQLiteStore) Set 

func (s *SQLiteStore) Set(ctx context.Context, namespace, key, value string) error

type Store 

type Store interface {
	// Get retrieves a value by namespace+key.
	// Returns ("", false, nil) if the key does not exist.
	Get(ctx context.Context, namespace, key string) (value string, found bool, err error)

	// Set stores a value. Overwrites any existing value for the same key.
	Set(ctx context.Context, namespace, key, value string) error

	// Delete removes a key. Returns nil (not an error) if the key does not exist.
	Delete(ctx context.Context, namespace, key string) error

	// List returns all keys in namespace whose names start with prefix.
	// Returns an empty slice (not nil) when no keys match.
	List(ctx context.Context, namespace, prefix string) (keys []string, err error)

	// Scan returns all key-value pairs in namespace whose keys start with prefix,
	// in a single atomic read. Prefer Scan over List+Get when you need both keys
	// and values — it avoids N individual Get calls and holds the lock only once.
	// Returns an empty slice (not nil) when no keys match.
	Scan(ctx context.Context, namespace, prefix string) ([]KV, error)

	// Close releases any resources held by the store (file handles, DB connections).
	// Called once on graceful shutdown.
	Close() error
}

Store is the single interface all service state flows through. Implementations must be safe for concurrent use from multiple goroutines.

The namespace parameter segments keys by service (e.g. "s3", "sqs") so that different services can use the same key names without collision.

type Tier 

type Tier int

Tier classifies a namespace for the HybridStore's memory management strategy. 

const (
	// TierHot namespaces are always held in memory and never evicted.
	// These contain resource definitions (queues, topics, tables, etc.) which
	// are small, finite, and needed for instant topology/dashboard renders.
	TierHot Tier = iota

	// TierCached namespaces are served from an LRU-bounded memory cache with
	// SQLite as the overflow tier. When the memory budget is exceeded, least
	// recently accessed entries are evicted from memory but remain on disk.
	TierCached
)

func TierFor 

func TierFor(namespace string) Tier

TierFor returns the tier for a namespace. Unknown namespaces default to TierHot.

type WALOptions 

type WALOptions struct {
	SyncMode WALSyncMode

	// SyncInterval is used only when SyncMode is WALSyncInterval.
	SyncInterval time.Duration

	// MaxLogBytes triggers compaction when the append log reaches this size.
	MaxLogBytes int64
}

WALOptions configures WALStore durability and compaction behavior.

type WALStore 

type WALStore struct {
	// contains filtered or unexported fields
}

WALStore is a memory-first store with an append-only write-ahead log. Reads are served from memory. Every mutation is appended to disk and can be replayed on restart.

func NewWALStore 

func NewWALStore(dataDir string, opts WALOptions) (*WALStore, error)

NewWALStore creates or opens the WAL-backed store rooted at dataDir.

func (*WALStore) Close 

func (s *WALStore) Close() error

func (*WALStore) Delete 

func (s *WALStore) Delete(ctx context.Context, namespace, key string) error

func (*WALStore) DeletePrefix 

func (s *WALStore) DeletePrefix(ctx context.Context, namespace, prefix string) error

func (*WALStore) Get 

func (s *WALStore) Get(ctx context.Context, namespace, key string) (string, bool, error)

func (*WALStore) List 

func (s *WALStore) List(ctx context.Context, namespace, prefix string) ([]string, error)

func (*WALStore) Scan 

func (s *WALStore) Scan(ctx context.Context, namespace, prefix string) ([]KV, error)

func (*WALStore) Set 

func (s *WALStore) Set(ctx context.Context, namespace, key, value string) error

type WALSyncMode 

type WALSyncMode string

WALSyncMode controls how frequently WAL writes are fsync'd to disk. 

const (
	WALSyncAlways   WALSyncMode = "always"
	WALSyncInterval WALSyncMode = "interval"
	WALSyncNever    WALSyncMode = "never"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.