store

package
v1.52.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package store implements per-(org, namespace) SQLite shards with optional consensus replication.

Layout on disk — hanzoai/namespace's, shared with every other store in the estate rather than one of tasks' own: 

<rootDir>/orgs/<org>/<namespace>.db
<rootDir>/orgs/<org>/projects/<project>/<namespace>.db
<rootDir>/orgs/_platform/<namespace>.db     the root (unscoped) tenant

A shard is created on first use. Namespaces are not declared up front.

A Principal that narrows to a USER has no place in this layout and is refused — see ErrUserLeg.

Schema (single table; key/value blob): 

CREATE TABLE kv(
  key   TEXT PRIMARY KEY,
  value BLOB NOT NULL,
  upd   INTEGER NOT NULL
);

Encryption: 

Every shard is encrypted at rest, keyed by hanzoai/cek from the process
master and the namespace that owns it. There is no plaintext posture and
no key material beside the file: a shard is born encrypted or it does not
exist. A deployment with no master mints a random one that dies with the
process, which is the honest shape of a keyless run.

Replication: 

A Replicator may be attached via WithReplicator. Every put/del is
wrapped in a replication.Frame and Propose'd before the local apply
commits. On Accept the frame is also dispatched to peers via the
driver-internal Subscribe path.

Index

Constants

View Source 
const Depth = 3

Depth is the number of legs in a Principal: org, project, user.

View Source 
const MasterKeyLen = cek.KeyLen

MasterKeyLen is the required length of the store's root key.

View Source 
const Sentinel = "_"

Sentinel is the path segment standing for an unset leg of a Principal. It is not a legal id, so a Principal with an unset leg can never be confused with one that names a tenant of that name.

Variables

View Source 
var CrossNamespaceKinds = map[string]bool{
	"ns": true,
	"nx": true,
}

CrossNamespaceKinds lists the kinds whose List(prefix) operations must enumerate every shard under the org. Anything else routes to a single shard.

View Source 
var ErrClosed = errors.New("store: manager closed")

ErrClosed is returned when a method runs after Close.

View Source 
var ErrUserLeg = errors.New("store: the shared namespace layout has no place for a user-scoped shard; scope to the org or its project")

ErrUserLeg reports that a Principal narrows to a USER, which the shared namespace layout has no place for.

hanzoai/namespace names an org, an org's project, and the deployment itself. namespace.Key returns an error for a user namespace on purpose — "this layout has no place for them, and inventing one silently is how a second convention starts". Tasks used to write <root>/<org>/<project>/<user>/, a layout of its own; it now shares the estate's, and the user leg is the one part of its tenancy that does not survive the move.

It is refused at the door rather than only when the file is keyed, so a user-scoped shard fails the same way in dev as in production instead of working plaintext and dying the day a master key is set.

View Source 
var IdleEvictAfter = 10 * time.Minute

IdleEvictAfter sets how long an open shard may sit unused before the manager closes it. Mutable for tests.

View Source 
var SweepEvery = time.Minute

SweepEvery is how often idle shards are evicted and resident shards sealed. Mutable for tests.

Functions

func CopyFile 

func CopyFile(dst, src string) (int64, error)

CopyFile is a helper used by the migration tool. Returns bytes copied.

func IsCrossNamespacePrefix 

func IsCrossNamespacePrefix(prefix string) bool

IsCrossNamespacePrefix reports whether prefix is a bare kind/ scan that must fan out across shards.

func NsFromKey 

func NsFromKey(key string) (kind, ns, rest string, ok bool)

nsFromKey parses the canonical key layout to derive the namespace segment. Returns ("", false) for keys that are themselves the namespace registry (`ns/<name>`); the caller treats those specially.

Layout reference: 

ns/<name>
wf/<ns>/<workflowId>/<runId>
wfh/<ns>/<workflowId>/<runId>/<eventId>
sc/<ns>/<scheduleId>
bt/<ns>/<batchId>
dp/<ns>/<deploymentName>
nx/<ns>/<endpointName>
id/<ns>/<email>
sa/<ns>/<attrName>
idem/<ns>/<workflowId>/<requestId>

func SplitPrefix 

func SplitPrefix(prefix string) (kind, ns, suffix string)

SplitPrefix returns (kind, ns, suffix) for a list prefix. ns may be empty when IsCrossNamespacePrefix(prefix) is true.

func ValidName added in v1.52.4 

func ValidName(name string) error

ValidName rejects anything that would escape or alias a shard directory or file, or collide with the unset-leg sentinel. Every name that becomes a path segment — a principal's legs and a namespace alike — passes through here, so there is one rule for all of them.

The rule is hanzoai/namespace's own segment rule, asked rather than restated: a name must already BE a legal namespace segment, exactly as written. That settles what a name cannot be by construction instead of by denylist — no separator so no path, no dot so no ".." and no hidden name, no trailing space, no encoded form of any of those — and it holds for the namespace leg too, which cek renders straight into a filename.

"Exactly as written" is the strict part, and it is deliberate. The namespace constructor case-folds, so it would ACCEPT "Acme" and quietly store it as "acme"; a leg that only becomes legal after folding has two spellings, and the second one names the same file under a different key. Legs arrive already folded (see Org), so anything that would change here did not come through the door.

Types

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager owns the on-disk shard layout and the open-shard cache.

It holds no key. The master lives in hanzoai/cek — one key per process, for every database the process opens — because threading it through every caller would not make it less global, only harder to see.

func New 

func New(rootDir string, master []byte) (*Manager, error)

New opens the manager rooted at rootDir. The directory is created on demand; errors are returned only for unrecoverable IO problems.

master is the 32-byte root key every shard is derived from, resolved from KMS by whoever runs this. It is installed into hanzoai/cek, which is where the process keeps it.

A nil master does NOT mean plaintext — there is no such posture any more. It means "I have no key to state":

  • Embedded in a process that already installed one (cloud resolves it through credz at boot), that key is used. One deployment, one key — an embedded engine minting a second would write files its host could not read.
  • Otherwise a random master is minted for this process. Nothing it writes outlives the run, by construction, which is the honest shape of a deployment that was given no key and the reason a developer needs no configuration at all.

func (*Manager) Close 

func (m *Manager) Close() error

Close flushes and closes every open shard. Safe to call twice.

func (*Manager) Encrypted added in v1.52.4 

func (m *Manager) Encrypted() bool

Encrypted reports whether shards are opened at rest encrypted. They always are; it answers whether this process holds a key to open any at all.

func (*Manager) Get 

func (m *Manager) Get(ctx context.Context, p Principal, ns string) (*Shard, error)

Get returns the open shard for (principal, ns), creating it on disk if needed. The returned Shard is safe for concurrent use.

func (*Manager) ListPrincipals added in v1.52.4 

func (m *Manager) ListPrincipals(ctx context.Context) ([]Principal, error)

ListPrincipals enumerates every tenant that owns at least one shard. Used by the cron sweeper, which must see EVERY tenant's schedules from the root engine.

It walks the namespace layout: orgs/_platform is the root tenant, every other orgs/<slug> is an org, and each of those may hold projects/<slug>. The slug it reads back is ALREADY the folded storage identity, so the Principal it builds renders straight back to the directory it came from (see Org for why folding twice would not).

func (*Manager) ListShards 

func (m *Manager) ListShards(ctx context.Context, p Principal) ([]*Shard, error)

ListShards enumerates every namespace shard the principal owns. Used by cross-namespace operations like ListNamespaces().

func (*Manager) OpenShardCount 

func (m *Manager) OpenShardCount() int

OpenShardCount reports the number of resident shards (for /v1/tasks/cluster).

func (*Manager) Replicator 

func (m *Manager) Replicator() replication.Replicator

Replicator returns the currently-installed driver, or nil.

func (*Manager) RootDir 

func (m *Manager) RootDir() string

RootDir returns the on-disk root.

func (*Manager) ShardPath 

func (m *Manager) ShardPath(p Principal, ns string) string

ShardPath returns the on-disk file for (principal, ns), or "" for a principal the layout cannot name.

func (*Manager) WithReplicator 

func (m *Manager) WithReplicator(r replication.Replicator)

WithReplicator installs r as the consensus driver for every shard opened from now on, and re-installs it on already-open shards.

type Principal added in v1.52.4 

type Principal struct {
	Org     string
	Project string
	User    string
}

Principal is the tenant that owns a shard: the org, optionally narrowed to a project and to a user. It is the SAME value in both places tenancy is decided, so the two can never disagree:

  • the directory the shard's file lives in (String), and
  • the key-encryption key that file's DEK is wrapped under (KEK).

Legs narrow left to right and each unset leg is written as Sentinel, so the encoding is fixed-width and injective: acme/_/z (an org's user, no project) and acme/z/_ (an org's project) are distinct paths and derive distinct keys. The zero Principal is the root — the unscoped embedded / dev tenant.

The org is what a tenant IS; project and user narrow it and are each independently optional, because IAM mints identities that carry a user without a project. Nothing may be set without an org.

func Org added in v1.52.4 

func Org(org string) Principal

Org returns the principal naming an org.

It is THE DOOR: the raw IAM org a caller holds is folded here, once, through namespace.Sanitize — the one injective slugger — so everything downstream carries the STORAGE identity. That matters because a Principal is both the routing key on a replication frame and the name of the directory its shard lives in: folding again at the open would re-suffix a slug that already carries a disambiguation suffix, and land the tenant in a different, empty database than the one ListPrincipals just read off disk.

A name Sanitize REFUSES — it carries whitespace, a control or a format rune, the class no injective fold survives — is kept RAW rather than folded, so that Valid rejects it. Folding it would produce the empty string, and an empty org leg is the ROOT principal: a name too hostile to store would have become the deployment's own platform tenant. Fail-closed, loudly.

func OrgProject added in v1.52.9 

func OrgProject(org, project string) Principal

OrgProject is the door for a principal narrowed to one of an org's projects. Both legs are folded, by the same rule, in the same place.

func ParsePrincipal added in v1.52.4 

func ParsePrincipal(s string) (Principal, error)

ParsePrincipal is the inverse of String.

func (Principal) Namespace added in v1.52.9 

func (p Principal) Namespace() (namespace.Namespace, error)

Namespace is the entity hanzoai/namespace names this shard's tenant, and therefore both where its file lives and what its key is derived from — one name, two renderings, so the two cannot drift apart: 

root (zero)        →  system            →  orgs/_platform/<ns>.db
{Org}              →  org/<org>         →  orgs/<org>/<ns>.db
{Org, Project}     →  org/<org>/<proj>  →  orgs/<org>/projects/<proj>/<ns>.db
anything with User →  ErrUserLeg

The legs are already the folded storage identity (see Org), so this VALIDATES them and does not fold again — namespace.Sanitize is injective but not idempotent, and a second fold would rename a tenant whose first fold carried a disambiguation suffix.

func (Principal) Root added in v1.52.4 

func (p Principal) Root() bool

Root reports whether p names no tenant at all.

func (Principal) String added in v1.52.4 

func (p Principal) String() string

String is the canonical encoding: the three legs joined by "/", each unset leg written as Sentinel. It is BOTH the shard's directory (relative to the store root) and the routing key carried on a replication frame — one encoding, so a frame always lands in the directory its principal names.

func (Principal) Valid added in v1.52.4 

func (p Principal) Valid() error

Valid reports whether p is well formed: every set leg is a usable path segment, and nothing is set without an org to narrow.

type Shard 

type Shard struct {
	// contains filtered or unexported fields
}

Shard owns one SQLite file. WAL mode + foreign keys + 5s busy timeout. One writer; readers share the cache. Replicator hooks fire from put/del before local commit so the cluster sees the mutation first.

func (*Shard) Checkpoint 

func (s *Shard) Checkpoint() error

Checkpoint makes the on-disk file fully self-contained: it truncates the WAL and, for an encrypted shard the pure-Go codec envelope backs, seals the committed pages back into the ciphertext file. The seal is a successful no-op for a shard that persists per commit (the live codec and plaintext paths), so callers never ask which one they hold. Used by the migration tool before a copy and by the manager's sweep, which is what bounds an envelope-backed shard's exposure to an unclean exit.

func (*Shard) Close 

func (s *Shard) Close() error

Close flushes WAL and releases the connection. Idempotent.

func (*Shard) Del 

func (s *Shard) Del(ctx context.Context, key string) error

Del removes key. No-op if missing.

func (*Shard) Get 

func (s *Shard) Get(ctx context.Context, key string) ([]byte, bool, error)

Get reads key.

func (*Shard) List 

func (s *Shard) List(ctx context.Context, prefix string, fn func(key string, value []byte) error) error

List walks every kv row whose key starts with prefix in lexicographic order.

func (*Shard) Namespace 

func (s *Shard) Namespace() string

Namespace returns the shard's namespace.

func (*Shard) Path 

func (s *Shard) Path() string

Path returns the underlying file path.

func (*Shard) Principal added in v1.52.4 

func (s *Shard) Principal() Principal

Principal returns the tenant that owns the shard.

func (*Shard) Put 

func (s *Shard) Put(ctx context.Context, key string, value []byte) error

Put writes value at key. If a Replicator is installed it runs Propose first; on Accept the local commit happens. On Reject the transaction is dropped and an error is returned.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.