Documentation
¶
Overview ¶
Package store implements per-(org, namespace) SQLite shards with optional consensus replication.
Layout on disk:
<rootDir>/<org>/<project>/<user>/<namespace>.db
The three directory levels are the Principal that owns the shard, each unset leg written as the sentinel "_" (see principal.go). The root tenant — the unscoped embedded / dev path — is therefore <rootDir>/_/_/_.
A shard is created on first use. Namespaces are not declared up front.
Schema (single table; key/value blob):
CREATE TABLE kv( key TEXT PRIMARY KEY, value BLOB NOT NULL, upd INTEGER NOT NULL );
Encryption:
A non-nil master key opens every shard at rest encrypted, under a DEK of its own wrapped for its Principal (see dek.go). A nil master key opens plaintext — the zero-config dev posture.
Replication:
A Replicator may be attached via WithReplicator. Every put/del is wrapped in a replication.Frame and Propose'd before the local apply commits. On Accept the frame is also dispatched to peers via the driver-internal Subscribe path.
Index ¶
- Constants
- Variables
- func CopyFile(dst, src string) (int64, error)
- func IsCrossNamespacePrefix(prefix string) bool
- func NsFromKey(key string) (kind, ns, rest string, ok bool)
- func SplitPrefix(prefix string) (kind, ns, suffix string)
- func ValidName(name string) error
- type Manager
- func (m *Manager) Close() error
- func (m *Manager) Encrypted() bool
- func (m *Manager) Get(ctx context.Context, p Principal, ns string) (*Shard, error)
- func (m *Manager) ListPrincipals(ctx context.Context) ([]Principal, error)
- func (m *Manager) ListShards(ctx context.Context, p Principal) ([]*Shard, error)
- func (m *Manager) OpenShardCount() int
- func (m *Manager) Replicator() replication.Replicator
- func (m *Manager) RootDir() string
- func (m *Manager) ShardPath(p Principal, ns string) string
- func (m *Manager) WithReplicator(r replication.Replicator)
- type Move
- type Principal
- type Shard
- func (s *Shard) Checkpoint() error
- func (s *Shard) Close() error
- func (s *Shard) Del(ctx context.Context, key string) error
- func (s *Shard) Get(ctx context.Context, key string) ([]byte, bool, error)
- func (s *Shard) List(ctx context.Context, prefix string, fn func(key string, value []byte) error) error
- func (s *Shard) Namespace() string
- func (s *Shard) Path() string
- func (s *Shard) Principal() Principal
- func (s *Shard) Put(ctx context.Context, key string, value []byte) error
Constants ¶
const Depth = 3
Depth is the number of legs in a Principal: org, project, user.
const MasterKeyLen = 32
MasterKeyLen is the required length of the store's root key.
const Sentinel = "_"
Sentinel is the path segment standing for an unset leg of a Principal. It is not a legal id, so a Principal with an unset leg can never be confused with one that names a tenant of that name.
Variables ¶
var CrossNamespaceKinds = map[string]bool{ "ns": true, "nx": true, }
CrossNamespaceKinds lists the kinds whose List(prefix) operations must enumerate every shard under the org. Anything else routes to a single shard.
var ErrClosed = errors.New("store: manager closed")
ErrClosed is returned when a method runs after Close.
var ErrOccupied = errors.New("store: a shard already had a file at its destination")
ErrOccupied reports that at least one shard had a file at its destination. Nothing was overwritten and nothing was deleted; the run carried on past it, so the other tenants still moved.
var IdleEvictAfter = 10 * time.Minute
IdleEvictAfter sets how long an open shard may sit unused before the manager closes it. Mutable for tests.
var SweepEvery = time.Minute
SweepEvery is how often idle shards are evicted and resident shards sealed. Mutable for tests.
Functions ¶
func IsCrossNamespacePrefix ¶
IsCrossNamespacePrefix reports whether prefix is a bare kind/ scan that must fan out across shards.
func NsFromKey ¶
nsFromKey parses the canonical key layout to derive the namespace segment. Returns ("", false) for keys that are themselves the namespace registry (`ns/<name>`); the caller treats those specially.
Layout reference:
ns/<name> wf/<ns>/<workflowId>/<runId> wfh/<ns>/<workflowId>/<runId>/<eventId> sc/<ns>/<scheduleId> bt/<ns>/<batchId> dp/<ns>/<deploymentName> nx/<ns>/<endpointName> id/<ns>/<email> sa/<ns>/<attrName> idem/<ns>/<workflowId>/<requestId>
func SplitPrefix ¶
SplitPrefix returns (kind, ns, suffix) for a list prefix. ns may be empty when IsCrossNamespacePrefix(prefix) is true.
Types ¶
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager owns the on-disk shard layout and the open-shard cache.
func New ¶
New opens the manager rooted at rootDir. The directory is created on demand; errors are returned only for unrecoverable IO problems.
master is the 32-byte root key every shard's DEK is wrapped under. A nil master opens shards plaintext — the zero-config dev posture; any other length is rejected here rather than at the first shard open.
func (*Manager) Encrypted ¶ added in v1.52.4
Encrypted reports whether shards are opened at rest encrypted.
func (*Manager) Get ¶
Get returns the open shard for (principal, ns), creating it on disk if needed. The returned Shard is safe for concurrent use.
func (*Manager) ListPrincipals ¶ added in v1.52.4
ListPrincipals enumerates every tenant that owns at least one shard. Used by the cron sweeper, which must see EVERY tenant's schedules from the root engine.
func (*Manager) ListShards ¶
ListShards enumerates every namespace shard the principal owns. Used by cross-namespace operations like ListNamespaces().
func (*Manager) OpenShardCount ¶
OpenShardCount reports the number of resident shards (for /v1/tasks/cluster).
func (*Manager) Replicator ¶
func (m *Manager) Replicator() replication.Replicator
Replicator returns the currently-installed driver, or nil.
func (*Manager) WithReplicator ¶
func (m *Manager) WithReplicator(r replication.Replicator)
WithReplicator installs r as the consensus driver for every shard opened from now on, and re-installs it on already-open shards.
type Move ¶ added in v1.52.4
type Move struct {
From string `json:"from"`
To string `json:"to"`
// Skipped is set when the destination was already occupied. Both files
// survive untouched and Upgrade returns ErrOccupied, because only an
// operator can say which one is authoritative: the destination can only
// have got there by this binary booting before the move ran, which
// creates an EMPTY shard where the full one belongs — and anything
// started after that boot is in the empty one.
Skipped bool `json:"skipped,omitempty"`
}
Move is one file's relocation.
func Upgrade ¶ added in v1.52.4
Upgrade moves every shard under root into the principal-addressed layout and reports what it moved. Idempotent: a moved shard leaves nothing at its old path, and a directory already in this layout holds no shard file at its top level, so a re-run costs a scan and nothing else.
master must be nil. A store written by the older binary is PLAINTEXT — encryption arrived with this layout — and moving a file does not encrypt it. Boot once with the key unset and the shards read; boot with the key set and each one gets a fresh DEK sidecar and is then opened as SQLCipher ciphertext, which it is not. Move first, encrypt second.
type Principal ¶ added in v1.52.4
Principal is the tenant that owns a shard: the org, optionally narrowed to a project and to a user. It is the SAME value in both places tenancy is decided, so the two can never disagree:
- the directory the shard's file lives in (String), and
- the key-encryption key that file's DEK is wrapped under (KEK).
Legs narrow left to right and each unset leg is written as Sentinel, so the encoding is fixed-width and injective: acme/_/z (an org's user, no project) and acme/z/_ (an org's project) are distinct paths and derive distinct keys. The zero Principal is the root — the unscoped embedded / dev tenant.
The org is what a tenant IS; project and user narrow it and are each independently optional, because IAM mints identities that carry a user without a project. Nothing may be set without an org.
func ParsePrincipal ¶ added in v1.52.4
ParsePrincipal is the inverse of String.
func (Principal) KEK ¶ added in v1.52.4
KEK derives the key-encryption key for p from the master key by walking the principal chain: master → org → project → user, skipping any leg p leaves unset. Each narrowing step derives from the key it narrows, so a user's key is cryptographically contained in its org's — reaching a user's data requires the whole chain, not just the master. Each step carries its own domain-separation tag, so narrowing to a project named "z" and to a user named "z" derive different keys.
The KEK never encrypts a page. It wraps that shard's own DEK (see dek.go), which is what makes master-key rotation O(1) per shard.
func (Principal) String ¶ added in v1.52.4
String is the canonical encoding: the three legs joined by "/", each unset leg written as Sentinel. It is BOTH the shard's directory (relative to the store root) and the routing key carried on a replication frame — one encoding, so a frame always lands in the directory its principal names.
type Shard ¶
type Shard struct {
// contains filtered or unexported fields
}
Shard owns one SQLite file. WAL mode + foreign keys + 5s busy timeout. One writer; readers share the cache. Replicator hooks fire from put/del before local commit so the cluster sees the mutation first.
func (*Shard) Checkpoint ¶
Checkpoint makes the on-disk file fully self-contained: it truncates the WAL and, for an encrypted shard the pure-Go codec envelope backs, seals the committed pages back into the ciphertext file. The seal is a successful no-op for a shard that persists per commit (the live codec and plaintext paths), so callers never ask which one they hold. Used by the migration tool before a copy and by the manager's sweep, which is what bounds an envelope-backed shard's exposure to an unclean exit.
func (*Shard) List ¶
func (s *Shard) List(ctx context.Context, prefix string, fn func(key string, value []byte) error) error
List walks every kv row whose key starts with prefix in lexicographic order.
Source Files