Documentation
¶
Index ¶
- Variables
- func ProbeCAS(ctx context.Context, store replica.ConditionalStore, keyPrefix string) error
- type CASFencer
- type Cipher
- type ConditionalStore
- type DB
- type Durability
- type DurabilityOption
- type Durable
- func (d *Durable) Bind(db *sql.DB)
- func (d *Durable) Close(ctx context.Context) error
- func (d *Durable) Hydrate(ctx context.Context) error
- func (d *Durable) Owned() bool
- func (d *Durable) PendingPromotion() bool
- func (d *Durable) Sync(ctx context.Context) (acked bool, err error)
- func (d *Durable) TryClaim(ctx context.Context) (bool, error)
- type FencedStore
- type Fencer
- type Lease
- type Member
- type Membership
- type Option
- type Replicator
- type Round
- type S3ConditionalStore
- type Source
- type Store
Constants ¶
This section is empty.
Variables ¶
var ( // Owner returns the single writer for orgID (HRW), fail-closed on empty. Owner = ha.Owner // IsOwner reports whether selfID owns the writer for orgID. IsOwner = ha.IsOwner // Replicas returns the owner then ordered failover successors. Replicas = ha.Replicas // DBPath is the canonical object-store location of an org's SQLite DB. DBPath = replica.DBPath // NewFencedStore wraps a ConditionalStore as the round-fenced per-org ship path. NewFencedStore = replica.NewFencedStore // StaticFencer is the single-process Fencer (round always 1) for dev and tests. StaticFencer = ha.StaticFencer // ErrStaleRound is the deposed-writer rejection: a ship below the recorded round. ErrStaleRound = replica.ErrStaleRound )
var ErrNoMembership = errors.New("org: empty membership — no safe writer")
ErrNoMembership is returned by a Fencer when the live replica set is empty: no safe owner can be named, so the caller fails CLOSED (does not write).
var ErrNonAtomicStore = errors.New("org: object store does not enforce conditional-PUT atomically — refusing to fence (would risk split-brain)")
ErrNonAtomicStore reports that the object store did not enforce conditional-PUT atomically — the caller MUST fail safe (run local-only, never fence on it), because fencing on a store that admits two winners for one round is split-brain by construction.
var ErrNotOwner = errors.New("org: not the elected writer for this org")
ErrNotOwner is returned when this replica is not the HRW-elected writer for the org. The caller does not write; it forwards to, or steps aside for, the owner.
Functions ¶
func ProbeCAS ¶ added in v1.801.231
ProbeCAS proves the store enforces compare-and-set atomically, exercising BOTH preconditions the fence relies on, against a fresh throwaway key:
create-race : probeRacers goroutines each PutIfVersion(key, _, "") — If-None-Match
on a non-existent object — so exactly one may CREATE it.
update-race : read the create winner's version, then probeRacers goroutines each
PutIfVersion(key, _, thatVersion) — If-Match — so exactly one may
ADVANCE it.
Exactly-one-winner in BOTH phases ⇒ atomic-confirmed (nil). Any other outcome — more than one winner (non-atomic ⇒ ErrNonAtomicStore), zero winners, or a hard store error (unreachable / refused) — is returned for the caller to fail safe on. The probe writes only under the caller's chosen throwaway prefix (a ".probe/" object per boot, reapable by a bucket lifecycle rule), never into the orgs/ tree.
Types ¶
type CASFencer ¶ added in v1.786.165
type CASFencer struct {
// contains filtered or unexported fields
}
CASFencer implements ha.Fencer by claiming/renewing a per-org writer lease in a linearizable object store. Acquire returns the caller's monotone lease round; FencedStore then stamps every ship with it.
func NewCASFencer ¶ added in v1.786.165
func NewCASFencer(store replica.ConditionalStore, view ownerView) *CASFencer
NewCASFencer builds a CASFencer over an atomic-CAS store and a membership view. The SAME store instance also backs the per-org replica.FencedStore, so the lease and the data ships share one linearizable register.
func (*CASFencer) Acquire ¶ added in v1.786.165
Acquire implements ha.Fencer. It fails CLOSED unless this replica is the elected writer over a non-empty membership, then claims (strictly bumping the round) or renews (keeping the round) the org's lease.
func (*CASFencer) ElectsSelf ¶ added in v1.801.231
ElectsSelf reports whether this replica is the HRW-elected writer for orgID under the CURRENT live membership — the election half of Acquire WITHOUT the lease CAS, so it does no I/O. It is the cheap gate a degraded (read-only) store consults to decide whether a membership change has made it the owner and it should attempt promotion. Fail-closed on an empty set (no safe owner ⇒ not self).
type Cipher ¶ added in v1.786.12
type Cipher struct {
// contains filtered or unexported fields
}
Cipher seals one database's snapshot at rest, so the object in SeaweedFS is ciphertext and a leaked file is useless without the KMS master key.
The key is cek's, and only cek's: cek.DeriveKey(master, ns, subsystem) — the same derivation, from the same master, that keyed the local file this snapshot is a copy of. A file and its snapshot are two renderings of one key, and the estate has one place that turns a master into a key. This package used to hold a second HKDF of its own, which meant the local file and the object it ships to were keyed by two different functions that no test compared; keeping them equal was a promise in a comment rather than a property of the code.
Binding the subsystem is what the org slug alone could not do. Every database an org owned used to seal under a single key, so its settings snapshot and its ledger snapshot were interchangeable to anyone holding either one. cek binds the subsystem into the derivation, so they no longer are.
AES-256-GCM gives confidentiality and integrity. The AAD is the same namespace/subsystem binding the key was derived under, so a blob lifted into another database's slot fails the tag instead of decrypting to garbage.
The GCM nonce is DERIVED from (key, plaintext) — HMAC(key,"nonce"||plaintext), so identical plaintext seals to identical ciphertext (content-addressable, keeps the Replicator's version-skip working) while distinct plaintexts get distinct nonces (GCM safety: the same (key,nonce) never covers two different messages).
func NewCipher ¶ added in v1.786.12
NewCipher builds a Cipher from the KMS master key. master must be exactly 32 bytes — the same master SetMaster installs, so the snapshot and the file it came from derive from one secret.
type ConditionalStore ¶ added in v1.786.165
type ConditionalStore = replica.ConditionalStore
type Durability ¶ added in v1.801.191
type Durability struct {
// contains filtered or unexported fields
}
Durability is the per-deployment, org-agnostic durable-store factory: the shared election+fence over ONE object store, plus the optional at-rest envelope. It holds no per-org state — For() mints a Durable per org DB. nil ⇒ durability disabled (local-only single-node/dev), the caller's default open path unchanged.
func NewDurability ¶ added in v1.801.191
func NewDurability(cond replica.ConditionalStore, view ownerView, cipher *Cipher, opts ...DurabilityOption) *Durability
NewDurability builds the factory over an atomic-CAS object store (the SeaweedFS S3 If-Match ConditionalStore), the live membership view (election input), and an optional per-org envelope Cipher (nil ⇒ the durable object is stored in the clear; pure-Go dev only). The SAME cond backs both the lease and the data ships, so they share one linearizable register.
The cond is an INTERFACE by design (replica.ConditionalStore): a read-through/ write-through cache tier (KV in front of S3) wraps it with a one-line decorator at the buildDurability construction site, with no change here — the fence reads and CASes through whatever store it is handed. The ship mechanism is likewise swappable: the default wholeFile codec can be replaced by a WAL-frame delta codec behind snapshotCodec without touching the fence or round. WithCheckpoint injects the envelope's re-encrypting Checkpoint (crypto-integration seam).
func (*Durability) For ¶ added in v1.801.191
func (dy *Durability) For(ns namespace.Namespace, subsystem, dbKey, dbPath string) *Durable
For mints the Durable binding for one org DB. It takes the NAME — the same (ns, subsystem) the local file was opened under — rather than the parts a name is made of, so the snapshot cannot be keyed for one database and shipped as another: the election key is ns.ID() (the ENTITY, so every one of an org's project-scoped files has one elected writer) and the snapshot key is cek's over (ns, subsystem). dbKey is the durable object location (replica.DBPath). dbPath is the local SQLite file.
type DurabilityOption ¶ added in v1.801.231
type DurabilityOption func(*durabilityOpts)
DurabilityOption configures a Durability.
func WithCheckpoint ¶ added in v1.801.231
WithCheckpoint injects the operation that folds the WAL into the real on-disk file and, on the pure-Go encryption ENVELOPE, re-encrypts that real path — so a fenced ship reads FRESH bytes, never stale ciphertext (which would be a lost acked write on takeover). The composition root wires cek's Checkpoint here on the envelope backend; the default (no option) is a raw TRUNCATE checkpoint, correct for the SQLCipher page-level and plaintext backends that encrypt on write. This composes ship-before-ack with encrypt-on-checkpoint.
type Durable ¶ added in v1.801.191
type Durable struct {
// contains filtered or unexported fields
}
Durable binds one org's local SQLite file to its fenced durable object slot, gated by the org's single-writer lease. It never opens or closes the local handle (the store owns that) — Bind lends it the handle so Sync can checkpoint on the same single connection the store writes through.
func (*Durable) Bind ¶ added in v1.801.191
Bind lends Durable the store's live handle so Sync checkpoints on the SAME single connection the store writes through (serializing snapshot against writes without a second handle). Call after the store opens the local file.
func (*Durable) Close ¶ added in v1.801.191
Close best-effort ships any final state under the caller's (time-bounded) ctx — ship-before-ack already covers every acknowledged write, so this is belt-and-suspenders — and releases. It does NOT close the live *sql.DB; that is the store handle the caller owns.
func (*Durable) Hydrate ¶ added in v1.801.191
Hydrate acquires the writer lease and restores the latest durable snapshot into the local file, BEFORE the store opens its handle. As the elected owner it CarryForward-seals the durable object to the lease round while hydrating (safe takeover: no acknowledged write lost). A non-owner refreshes read-only. Any other condition (store unreachable, empty membership, lost race) degrades to read-only on the local file and is returned for the caller to log — it NEVER makes the store unopenable, so a store is always available for reads; writes fail closed until a later open re-acquires.
RECOVERY from a degraded open (M3): a pod that could not acquire at open stays read-only until this replica becomes the org's elected owner (a membership change), at which point the OrgStore promotes the store IN PLACE — PendingPromotion gates it, TryClaim proves the lease is claimable, then a quiesce-close-reopen swaps in a writer handle and CarryForward-restores the latest snapshot under the FRESH handle (never under the live one — the swap is why the reopen is required). No process restart.
func (*Durable) Owned ¶ added in v1.801.191
Owned reports whether this replica currently holds the writer lease. A store may gate a write on it, but the authoritative gate is Sync's fenced ship.
func (*Durable) PendingPromotion ¶ added in v1.801.231
PendingPromotion reports whether this store opened degraded (does not hold the lease) yet this replica is NOW the org's elected owner — a membership change made it the writer, so the store must be promoted (re-acquire + hydrate + reopen) to serve writes. Cheap and I/O-free: a lock plus one HRW over the live member snapshot, evaluated only when the store is not already owned. It is the per-request gate the OrgStore checks on a cache hit; the actual promotion runs (rarely) only when this returns true.
func (*Durable) Sync ¶ added in v1.801.191
Sync snapshots the local file and ships it to the durable object fenced at the lease round — the ship-before-ack step. acked is true only if the fenced store admitted our round. A ship rejected as ErrStaleRound returns (false, nil): this replica was deposed, so it drops ownership and does NOT acknowledge — the caller retries on the new owner. A non-owner returns (false, ErrNotOwner). Call AFTER the write transaction commits and OUTSIDE any open transaction (Sync takes the sole connection to checkpoint).
func (*Durable) TryClaim ¶ added in v1.801.231
TryClaim probes whether this replica can hold the org's writer lease right now and, if so, claims it — the promotion gate. It performs ONLY the lease CAS (via the fencer), no local-file I/O, so a failed probe (not the elected owner, or the store unreachable) costs nothing and leaves any live handle untouched. On success the lease object names this replica at a fresh round (fencing any prior owner); the caller then quiesces and reopens, whose Hydrate renews THIS same lease and CarryForward-restores the latest snapshot under the fresh handle. Returns (false, nil) when not the elected owner, (false, err) on a store error, (true, nil) when claimed.
type FencedStore ¶ added in v1.786.165
type FencedStore = replica.FencedStore
FencedStore admits a per-org-DB ship only at a round >= the recorded one, fencing a deposed writer. ConditionalStore is its atomic-CAS backing seam.
type Lease ¶ added in v1.786.165
Lease binds an elected owner to the monotone round it writes at (the fencing value). Round is that monotone epoch. Fencer issues a Lease from a linearizable round source (CASFencer here; the Lux BFT round later).
type Membership ¶
type Membership struct {
// contains filtered or unexported fields
}
Membership caches the latest replica snapshot and refreshes it from a Source on an interval. Reads are lock-free via an atomic snapshot pointer, so the per-request AmOwner hot-path never blocks on the refresher.
func NewMembership ¶
func NewMembership(selfID string, src Source, interval time.Duration) *Membership
NewMembership builds a Membership over src. selfID is this replica's stable id (see Member.ID). interval<=0 defaults to 5s.
func (*Membership) AmOwner ¶
func (m *Membership) AmOwner(orgID string) bool
AmOwner reports whether THIS replica owns orgID's writer right now — the per-write hot-path check. Lock-free.
func (*Membership) Members ¶
func (m *Membership) Members() []Member
Members returns the current membership snapshot (never nil; may be empty).
func (*Membership) OwnerOf ¶
func (m *Membership) OwnerOf(orgID string) (Member, bool)
OwnerOf resolves the writer-owner replica for orgID under the live set.
func (*Membership) Start ¶
func (m *Membership) Start(ctx context.Context) error
Start does an initial synchronous refresh (so Members() is populated before the first request) then refreshes on the interval until Stop or ctx cancel. Returns the initial refresh error, if any; callers may serve with a stale or self-only set regardless.
The FIRST refresh is bounded and the loop is NOT, and they cannot share one deadline. Start runs on the boot path, so an unreachable membership source (the k8s API, for the LIVE selector) must not hold the process short of its listener — the caller already treats the error as advisory. But a deadline on the ctx the loop receives would stop membership refresh for the rest of the process's life, which is far worse than a slow start: a drained pod would stay elected owner forever. So the bound wraps only the synchronous call.
type Replicator ¶
type Replicator = replica.Replicator
Replicator binds one per-org SQLite to its object-store slot.
type S3ConditionalStore ¶ added in v1.801.191
S3ConditionalStore is a replica.ConditionalStore over the SeaweedFS S3 gateway's native optimistic-locking extension (SetMatchETag / SetMatchETagExcept). The server — not this process — evaluates the precondition atomically, which is the property the fence's correctness depends on.
func NewS3ConditionalStore ¶ added in v1.801.191
func NewS3ConditionalStore(client *s3.Client, bucket string) *S3ConditionalStore
NewS3ConditionalStore builds a ConditionalStore over an already-constructed S3 client and bucket. The caller owns the client lifecycle (credentials, TLS, endpoint) — this type adds only the conditional-write semantics.
func (*S3ConditionalStore) Get ¶ added in v1.801.191
Get returns the object bytes and its ETag from ONE GET response (obj.Stat reads the header of the same request io.ReadAll drains), so the version is guaranteed consistent with the bytes. Absent key maps to replica.ErrNotFound.
func (*S3ConditionalStore) PutIfVersion ¶ added in v1.801.191
func (s *S3ConditionalStore) PutIfVersion(ctx context.Context, key string, data []byte, expectVersion string) (string, error)
PutIfVersion writes data at key iff the store's current ETag still equals expectVersion (SetMatchETag), or iff the object does not exist when expectVersion is "" (SetMatchETagExcept "*"). A failed precondition is mapped to replica.ErrConflict.
type Source ¶
Source yields the current live replica set. It is the ONE thing in the horizontally-scaled cloud that needs a peer view — everything else is a pure function of (org, members). Any discovery mechanism plugs in as a Source without touching the ownership core: a static list (dev), a K8s Endpoints poll (prod), or a zapd gossip view.
func StaticSource ¶
StaticSource yields a fixed set — single-node / local dev. With no explicit members it reads CLOUD_REPLICAS ("id@addr,id2@addr2", or bare "id" with addr==id).
Source Files