store

package
v1.53.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package store implements per-(org, namespace) SQLite shards with optional consensus replication.

Layout on disk — hanzoai/namespace's, shared with every other store in the estate rather than one of tasks' own:

<rootDir>/orgs/<org>/<namespace>.db
<rootDir>/orgs/<org>/projects/<project>/<namespace>.db
<rootDir>/orgs/_platform/<namespace>.db     the root (unscoped) tenant

A shard is created on first use. Namespaces are not declared up front.

A Principal that narrows to a USER has no place in this layout and is refused — see ErrUserLeg.

Schema (single table; key/value blob):

CREATE TABLE kv(
  key   TEXT PRIMARY KEY,
  value BLOB NOT NULL,
  upd   INTEGER NOT NULL
);

Encryption:

Every shard is encrypted at rest, keyed by hanzoai/cek from the process
master and the namespace that owns it. There is no plaintext posture and
no key material beside the file: a shard is born encrypted or it does not
exist. A deployment with no master mints a random one that dies with the
process, which is the honest shape of a keyless run.

Replication:

A Replicator may be attached via WithReplicator. Every put/del is
wrapped in a replication.Frame and Propose'd before the local apply
commits. On Accept the frame is also dispatched to peers via the
driver-internal Subscribe path.

Index

Constants

View Source
const Depth = 3

Depth is the number of legs in a Principal: org, project, user.

View Source
const MasterKeyLen = cek.KeyLen

MasterKeyLen is the required length of the store's root key.

View Source
const Sentinel = "_"

Sentinel is the path segment standing for an unset leg of a Principal. It is not a legal id, so a Principal with an unset leg can never be confused with one that names a tenant of that name.

Variables

View Source
var CrossNamespaceKinds = map[string]bool{
	"ns": true,
	"nx": true,
}

CrossNamespaceKinds lists the kinds whose List(prefix) operations must enumerate every shard under the org. Anything else routes to a single shard.

View Source
var ErrClosed = errors.New("store: manager closed")

ErrClosed is returned when a method runs after Close.

View Source
var ErrKey = errors.New("store: a key must be valid UTF-8 without NUL, and name its kind (kind/...)")

ErrKey is the refusal of a key the database cannot hold.

View Source
var ErrUserLeg = errors.New("store: the shared namespace layout has no place for a user-scoped shard; scope to the org or its project")

ErrUserLeg reports that a Principal narrows to a USER, which the shared namespace layout has no place for.

hanzoai/namespace names an org, an org's project, and the deployment itself. namespace.Key returns an error for a user namespace on purpose — "this layout has no place for them, and inventing one silently is how a second convention starts". Tasks used to write <root>/<org>/<project>/<user>/, a layout of its own; it now shares the estate's, and the user leg is the one part of its tenancy that does not survive the move.

It is refused at the door rather than only when the file is keyed, so a user-scoped shard fails the same way in dev as in production instead of working plaintext and dying the day a master key is set.

View Source
var IdleEvictAfter = 10 * time.Minute

IdleEvictAfter sets how long an open shard may sit unused before the manager closes it. Mutable for tests.

View Source
var SweepEvery = time.Minute

SweepEvery is how often idle shards are evicted and resident shards sealed. Mutable for tests.

Functions

func CopyFile

func CopyFile(dst, src string) (int64, error)

CopyFile is a helper used by the migration tool. Returns bytes copied.

func HasShards added in v1.53.7

func HasShards(dir string) bool

HasShards reports whether dir holds any shard file: a directory that has never been opened, or only ever by a deployment that has since moved its rows into a shared database and cleared it, has none.

func IsCrossNamespacePrefix

func IsCrossNamespacePrefix(prefix string) bool

IsCrossNamespacePrefix reports whether prefix is a bare kind/ scan that must fan out across shards.

func NsFromKey

func NsFromKey(key string) (kind, ns, rest string, ok bool)

nsFromKey parses the canonical key layout to derive the namespace segment. Returns ("", false) for keys that are themselves the namespace registry (`ns/<name>`); the caller treats those specially.

Layout reference:

ns/<name>
wf/<ns>/<workflowId>/<runId>
wfh/<ns>/<workflowId>/<runId>/<eventId>
sc/<ns>/<scheduleId>
bt/<ns>/<batchId>
dp/<ns>/<deploymentName>
nx/<ns>/<endpointName>
id/<ns>/<email>
sa/<ns>/<attrName>
idem/<ns>/<workflowId>/<requestId>
wfopen/<ns>/<workflowId>/<runId>
wfidx/<ns>
purge/<ns>

func RowSum added in v1.53.7

func RowSum(key string, value []byte) uint64

RowSum is one row's part of a Digest.

func SplitPrefix

func SplitPrefix(prefix string) (kind, ns, suffix string)

SplitPrefix returns (kind, ns, suffix) for a list prefix. ns may be empty when IsCrossNamespacePrefix(prefix) is true.

func ValidName added in v1.52.4

func ValidName(name string) error

ValidName rejects anything that would escape or alias a shard directory or file, or collide with the unset-leg sentinel. Every name that becomes a path segment — a principal's legs and a namespace alike — passes through here, so there is one rule for all of them.

The rule is hanzoai/namespace's own segment rule, asked rather than restated: a name must already BE a legal namespace segment, exactly as written. That settles what a name cannot be by construction instead of by denylist — no separator so no path, no dot so no ".." and no hidden name, no trailing space, no encoded form of any of those — and it holds for the namespace leg too, which cek renders straight into a filename.

"Exactly as written" is the strict part, and it is deliberate. The namespace constructor case-folds, so it would ACCEPT "Acme" and quietly store it as "acme"; a leg that only becomes legal after folding has two spellings, and the second one names the same file under a different key. Legs arrive already folded (see Org), so anything that would change here did not come through the door.

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager owns the on-disk shard layout and the open-shard cache.

It holds no key. The master lives in hanzoai/cek — one key per process, for every database the process opens — because threading it through every caller would not make it less global, only harder to see.

func New

func New(rootDir string, master []byte) (*Manager, error)

New opens the manager rooted at rootDir. The directory is created on demand; errors are returned only for unrecoverable IO problems.

master is the 32-byte root key every shard is derived from, resolved from KMS by whoever runs this. It is installed into hanzoai/cek, which is where the process keeps it.

A nil master does NOT mean plaintext — there is no such posture any more. It means "I have no key to state":

  • Embedded in a process that already installed one (cloud resolves it through credz at boot), that key is used. One deployment, one key — an embedded engine minting a second would write files its host could not read.
  • Otherwise a random master is minted for this process. Nothing it writes outlives the run, by construction, which is the honest shape of a deployment that was given no key and the reason a developer needs no configuration at all.

func (*Manager) Close

func (m *Manager) Close() error

Close flushes and closes every open shard. Safe to call twice.

func (*Manager) Encrypted added in v1.52.4

func (m *Manager) Encrypted() bool

Encrypted reports whether shards are opened at rest encrypted. They always are; it answers whether this process holds a key to open any at all.

func (*Manager) Get

func (m *Manager) Get(ctx context.Context, p Principal, ns string) (*Shard, error)

Get returns the open shard for (principal, ns), creating it on disk if needed. The returned Shard is safe for concurrent use.

func (*Manager) ListPrincipals added in v1.52.4

func (m *Manager) ListPrincipals(ctx context.Context) ([]Principal, error)

ListPrincipals enumerates every tenant that owns at least one shard. Used by the cron sweeper, which must see EVERY tenant's schedules from the root engine.

It walks the namespace layout: orgs/_platform is the root tenant, every other orgs/<slug> is an org, and each of those may hold projects/<slug>. The slug it reads back is ALREADY the folded storage identity, so the Principal it builds renders straight back to the directory it came from (see Org for why folding twice would not).

func (*Manager) ListShards

func (m *Manager) ListShards(ctx context.Context, p Principal) ([]*Shard, error)

ListShards enumerates every namespace shard the principal owns. Used by cross-namespace operations like ListNamespaces().

func (*Manager) OpenShardCount

func (m *Manager) OpenShardCount() int

OpenShardCount reports the number of resident shards (for /v1/tasks/cluster).

func (*Manager) Replicator

func (m *Manager) Replicator() replication.Replicator

Replicator returns the currently-installed driver, or nil.

func (*Manager) RootDir

func (m *Manager) RootDir() string

RootDir returns the on-disk root.

func (*Manager) ShardPath

func (m *Manager) ShardPath(p Principal, ns string) string

ShardPath returns the on-disk file for (principal, ns), or "" for a principal the layout cannot name.

func (*Manager) WithReplicator

func (m *Manager) WithReplicator(r replication.Replicator)

WithReplicator installs r as the consensus driver for every shard opened from now on, and re-installs it on already-open shards.

type Principal added in v1.52.4

type Principal struct {
	Org     string
	Project string
	User    string
}

Principal is the tenant that owns a shard: the org, optionally narrowed to a project and to a user. It is the SAME value in both places tenancy is decided, so the two can never disagree:

  • the directory the shard's file lives in (String), and
  • the key-encryption key that file's DEK is wrapped under (KEK).

Legs narrow left to right and each unset leg is written as Sentinel, so the encoding is fixed-width and injective: acme/_/z (an org's user, no project) and acme/z/_ (an org's project) are distinct paths and derive distinct keys. The zero Principal is the root — the unscoped embedded / dev tenant.

The org is what a tenant IS; project and user narrow it and are each independently optional, because IAM mints identities that carry a user without a project. Nothing may be set without an org.

func Org added in v1.52.4

func Org(org string) Principal

Org returns the principal naming an org.

It is THE DOOR: the raw IAM org a caller holds is folded here, once, through namespace.Sanitize — the one injective slugger — so everything downstream carries the STORAGE identity. That matters because a Principal is both the routing key on a replication frame and the name of the directory its shard lives in: folding again at the open would re-suffix a slug that already carries a disambiguation suffix, and land the tenant in a different, empty database than the one ListPrincipals just read off disk.

A name Sanitize REFUSES — it carries whitespace, a control or a format rune, the class no injective fold survives — is kept RAW rather than folded, so that Valid rejects it. Folding it would produce the empty string, and an empty org leg is the ROOT principal: a name too hostile to store would have become the deployment's own platform tenant. Fail-closed, loudly.

func OrgProject added in v1.52.9

func OrgProject(org, project string) Principal

OrgProject is the door for a principal narrowed to one of an org's projects. Both legs are folded, by the same rule, in the same place.

func ParsePrincipal added in v1.52.4

func ParsePrincipal(s string) (Principal, error)

ParsePrincipal is the inverse of String.

func (Principal) Namespace added in v1.52.9

func (p Principal) Namespace() (namespace.Namespace, error)

Namespace is the entity hanzoai/namespace names this shard's tenant, and therefore both where its file lives and what its key is derived from — one name, two renderings, so the two cannot drift apart:

root (zero)        →  system            →  orgs/_platform/<ns>.db
{Org}              →  org/<org>         →  orgs/<org>/<ns>.db
{Org, Project}     →  org/<org>/<proj>  →  orgs/<org>/projects/<proj>/<ns>.db
anything with User →  ErrUserLeg

The legs are already the folded storage identity (see Org), so this VALIDATES them and does not fold again — namespace.Sanitize is injective but not idempotent, and a second fold would rename a tenant whose first fold carried a disambiguation suffix.

func (Principal) Root added in v1.52.4

func (p Principal) Root() bool

Root reports whether p names no tenant at all.

func (Principal) String added in v1.52.4

func (p Principal) String() string

String is the canonical encoding: the three legs joined by "/", each unset leg written as Sentinel. It is BOTH the shard's directory (relative to the store root) and the routing key carried on a replication frame — one encoding, so a frame always lands in the directory its principal names.

func (Principal) Valid added in v1.52.4

func (p Principal) Valid() error

Valid reports whether p is well formed: every set leg is a usable path segment, and nothing is set without an org to narrow.

type Row added in v1.53.7

type Row struct {
	Key   string
	Value []byte
}

Row is one key and its value.

type Shard

type Shard struct {
	// contains filtered or unexported fields
}

Shard owns one SQLite file. WAL mode + foreign keys + 5s busy timeout. One writer; readers share the cache. Replicator hooks fire from put/del before local commit so the cluster sees the mutation first.

func (*Shard) Checkpoint

func (s *Shard) Checkpoint() error

Checkpoint makes the on-disk file fully self-contained: it truncates the WAL and, for an encrypted shard the pure-Go codec envelope backs, seals the committed pages back into the ciphertext file. The seal is a successful no-op for a shard that persists per commit (the live codec and plaintext paths), so callers never ask which one they hold. Used by the migration tool before a copy and by the manager's sweep, which is what bounds an envelope-backed shard's exposure to an unclean exit.

func (*Shard) Close

func (s *Shard) Close() error

Close flushes WAL and releases the connection. Idempotent.

func (*Shard) Del

func (s *Shard) Del(ctx context.Context, key string) error

Del removes key. No-op if missing.

func (*Shard) Get

func (s *Shard) Get(ctx context.Context, key string) ([]byte, bool, error)

Get reads key.

func (*Shard) List

func (s *Shard) List(ctx context.Context, prefix string, fn func(key string, value []byte) error) error

List walks every kv row whose key starts with prefix in lexicographic order.

func (*Shard) Namespace

func (s *Shard) Namespace() string

Namespace returns the shard's namespace.

func (*Shard) Path

func (s *Shard) Path() string

Path returns the underlying file path.

func (*Shard) Principal added in v1.52.4

func (s *Shard) Principal() Principal

Principal returns the tenant that owns the shard.

func (*Shard) Put

func (s *Shard) Put(ctx context.Context, key string, value []byte) error

Put writes value at key. If a Replicator is installed it runs Propose first; on Accept the local commit happens. On Reject the transaction is dropped and an error is returned.

func (*Shard) Reclaim added in v1.53.6

func (s *Shard) Reclaim(ctx context.Context, rebuild bool) error

Reclaim gives the shard's free pages back to the filesystem. SQLite keeps the pages of deleted rows on a freelist and reuses them for new rows, but never shrinks the file on its own.

A shard in incremental auto-vacuum mode releases up to reclaimPages per call. A shard in the default mode cannot release pages in place; it can only be rebuilt with VACUUM, which costs a copy of everything still in it. So it is rebuilt only when the caller passes rebuild — it has finished deleting for now, which is when the shard is smallest — and at least a quarter of it is free. The rebuild also switches the shard to incremental mode, so it happens once per shard. VACUUM keeps the page size and the reserved bytes per page that the encryption codec writes into.

The smaller file reaches disk at the shard's next seal (Checkpoint or Close). Reclaim is local to this file and never replicated: each replica's pages are its own.

func (*Shard) Scan added in v1.53.6

func (s *Shard) Scan(ctx context.Context, prefix, after string, limit int, fn func(key string, value []byte) error) error

Scan walks, in lexicographic order, at most limit kv rows (all of them when limit <= 0) whose key starts with prefix and sorts after `after`. Passing the last key one page returned as the next page's `after` walks a prefix in pieces; the shard's single connection is free between pages, so a long walk never holds the shard against the engine.

type Shared added in v1.53.7

type Shared struct {
	// contains filtered or unexported fields
}

Shared keeps every tenant's rows in the tables of ONE database that several replicas open at once, where Manager keeps them in a file per (tenant, namespace) that one process owns.

The engine's rows are a keyed document store (see Shard), and Shared is that store's second home: the same keys, the same values, the same prefix scans, in

tasks_kv(kind, principal, key, value, upd)  primary key (kind, principal, key)

A key already carries its namespace as its second segment, so there is no namespace column to keep in step with it. Its first segment is the kind ("wf", "sc", "wfh", ...), made a column so that the one scan which has to cross tenants, every schedule of every org, is a range over one kind and not a walk of every history row ever written.

The database is the handle the host passes in, on either engine. The SQL is the dialect PostgreSQL and SQLite both read: numbered $n placeholders, ON CONFLICT, integers for times. What they do not agree on is the column type of a blob and the collation of text, and those two are read off the handle's driver here.

Beside the rows, Shared keeps two small tables the replicas need to agree on something: tasks_lease, a row per name with a holder and an expiry in the DATABASE'S clock (membership, leadership and the short locks around a run), and tasks_meta (what has been moved in from files). Time is the database's, never a replica's, so a replica with a slow clock can neither keep a lease it lost nor take one that is not yet over.

The handle belongs to the caller: Shared migrates it, uses it and does not close it.

func OpenShared added in v1.53.7

func OpenShared(ctx context.Context, db *sql.DB) (*Shared, error)

OpenShared migrates db and returns the store on it. Two replicas opening one database at once is how a deployment starts, so the migration is serialised on PostgreSQL by a transaction-scoped advisory lock, and each statement is safe to run again.

func (*Shared) Claim added in v1.53.7

func (s *Shared) Claim(ctx context.Context, name, holder string, ttl time.Duration) (bool, error)

Claim takes, or renews, holder's lease on name for ttl, and reports whether holder has it. A lease another holder has is not taken until its expiry has passed on the DATABASE's clock. The ask is a plain SELECT first, because an INSERT ... ON CONFLICT locks the row it conflicts with and burns a transaction id even when its WHERE refuses it, and a replica that does not hold a lease asks for it every few seconds.

func (*Shared) Clear added in v1.53.9

func (s *Shared) Clear(ctx context.Context) error

Clear removes every row. It is for a move that is starting again over the rows its own earlier attempt left; nothing else clears an engine's state.

func (*Shared) Close added in v1.53.7

func (s *Shared) Close() error

Close ends the store. The database handle is the caller's and stays open.

func (*Shared) Count added in v1.53.9

func (s *Shared) Count(ctx context.Context) (int64, error)

Count is how many rows the store holds, of every tenant.

func (*Shared) Del added in v1.53.7

func (s *Shared) Del(ctx context.Context, p Principal, key string) error

Del removes key for p. Missing is not an error.

func (*Shared) Digest added in v1.53.7

func (s *Shared) Digest(ctx context.Context, p Principal) (n int64, sum uint64, err error)

Digest is how many rows p has and a sum over them that does not depend on the order they are read in: the two ends of a move agree on it when they hold the same rows.

func (*Shared) Expire added in v1.53.7

func (s *Shared) Expire(ctx context.Context, name, holder string) error

Expire ends holder's lease on name now, leaving the row: its term is over, so it reads as lapsed to whoever looks next. It is how a member leaves, as opposed to a lock being released, which leaves nothing.

func (*Shared) Forget added in v1.53.7

func (s *Shared) Forget(ctx context.Context, prefix string, age time.Duration) error

Forget drops every lease under prefix whose term ended before the given age, so the names of locks taken once do not accumulate.

func (*Shared) Get added in v1.53.7

func (s *Shared) Get(ctx context.Context, p Principal, key string) ([]byte, bool, error)

Get reads key for p.

func (*Shared) Held added in v1.53.7

func (s *Shared) Held(ctx context.Context, prefix string) ([]string, error)

Held lists the names under prefix whose lease is still in force, and Lapsed the ones whose term is over and that nobody has released: a holder that died.

func (*Shared) Lapsed added in v1.53.7

func (s *Shared) Lapsed(ctx context.Context, prefix string) ([]string, error)

Lapsed: see Held.

func (*Shared) List added in v1.53.7

func (s *Shared) List(ctx context.Context, p Principal, prefix string, fn func(key string, value []byte) error) error

List walks every row of p whose key starts with prefix.

func (*Shared) Meta added in v1.53.7

func (s *Shared) Meta(ctx context.Context, key string) (string, bool, error)

Meta reads a note the store keeps about itself.

func (*Shared) Postgres added in v1.53.7

func (s *Shared) Postgres() bool

Postgres reports whether the database is PostgreSQL.

func (*Shared) Principals added in v1.53.7

func (s *Shared) Principals(ctx context.Context) ([]Principal, error)

Principals lists every tenant that has registered a namespace, sorted.

func (*Shared) Put added in v1.53.7

func (s *Shared) Put(ctx context.Context, p Principal, key string, value []byte) error

Put writes value at key for p.

func (*Shared) PutMany added in v1.53.7

func (s *Shared) PutMany(ctx context.Context, p Principal, rows []Row) error

PutMany writes rows for p in one transaction, a statement to every putChunk rows. A key already there is replaced. It is how rows are moved in from files, where one round trip a row would be hours.

func (*Shared) Release added in v1.53.7

func (s *Shared) Release(ctx context.Context, name, holder string) error

Release gives up holder's lease on name, now and not at its expiry. A lease the holder no longer has (it lapsed and was taken) is left to its new holder.

func (*Shared) Scan added in v1.53.7

func (s *Shared) Scan(ctx context.Context, p Principal, prefix, after string, limit int, fn func(key string, value []byte) error) error

Scan walks, in key order, at most limit of p's rows (all of them when limit <= 0) whose key starts with prefix and sorts after `after`. The rows of the page are read before fn sees the first, so fn may ask the store for more: on SQLite the handle holds one connection, and a callback that queried it while the page was still being read would wait for itself.

func (*Shared) ScanAll added in v1.53.7

func (s *Shared) ScanAll(ctx context.Context, prefix string, fn func(p Principal, key string, value []byte) error) error

ScanAll walks every tenant's rows whose key starts with prefix, the tenant beside each. It is the cron sweeper's view of the world, and it is one range over one kind, so it costs what the kind holds and not what the database does.

func (*Shared) SetMeta added in v1.53.7

func (s *Shared) SetMeta(ctx context.Context, key, value string) error

SetMeta writes a note the store keeps about itself.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL