Documentation
¶
Overview ¶
Package writefence is the two-plane epoch write-fence: the primitive that makes a same-epoch double-write to a (plugin, shard) mirror physically impossible, regardless of what the CONTROL plane's writer-lease election believes or how stale a DATA-plane process's local view of that election is.
Where this sits (HIP-0107 / HIP-0116) ¶
The data plane (github.com/hanzoai/vfs/replica, wired in internal/org for per-org SQLite; the same shape governs any (plugin, shard) mirror) ships a writer's WAL/snapshot to an S3/vfs object-store mirror. Today the "am I allowed to push" decision is TWO comment-only, non-atomic things layered on top of an unconditional object overwrite:
- replica.IsOwner(orgID, selfID, members) — a PURE, LOCAL computation over whatever membership view the caller happens to hold. Two processes with different (stale) membership views can both compute themselves as owner.
- the K8s deployment shape (StatefulSet replicas:1 + Recreate — see role.Role's doc comment) — a coarse, non-epoch-aware "there should only be one" guarantee that does not hold under a partition/force-delete: k8s can start a new writer pod while the old one's process is still running and still able to open its RWO mount and still able to reach the object store.
- replica.Store.Put / vfs's backend.Backend.Put — "Overwriting is allowed" (verbatim doc comment upstream). No conditional write, no epoch, no admission check of any kind at the storage layer.
So the actual "canPush" predicate that exists today is: nothing. Any process that believes it is the owner can overwrite the mirror at any time. A deposed/partitioned old writer and a freshly-elected new writer can BOTH push — same-epoch (or no-epoch-at-all) double-write, silently corrupting the mirror / collapsing KMS share-isolation.
This package closes that gap as an orthogonal, storage-agnostic primitive that sits IN FRONT of the existing Push call, without forking github.com/hanzoai/vfs or github.com/hanzoai/vfs/replica and without touching clients/controlplane (owned by a concurrent effort — Stage 1, shadow-only, not yet consulted on the live write path):
- EpochSource is the seam controlplane's granted lease epoch drops into once it graduates from shadow. Nothing here imports controlplane.
- ConditionalStore models the one storage primitive every real object-store CAS boils down to (S3 If-Match / GCS generation-match): "write iff the object is still at the version I last observed."
- Fence.Push performs the STRICT epoch check and the CAS in one call, so there is no window between "epoch admitted" and "data written" for a second writer to land in.
See store.go for the real, already-vendored S3-compatible implementation (minio-go v7's native SetMatchETag/SetMatchETagExcept — no new dependency) and the doc comment there for exactly how this plugs into internal/org's today-vfs-only binding.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrConflict = errors.New("writefence: conditional write lost a concurrent race")
ErrConflict is returned when the store's atomic conditional write rejected our attempt because the object changed between our Get and our PutIfVersion — we lost a race against a concurrent pusher. Fence.Push retries internally (re-reading the new authoritative state and re-running the strict-epoch check), so ErrConflict escaping Push means retries were exhausted, not that a caller should blindly retry with the same epoch.
var ErrNotFound = errors.New("writefence: marker not found")
ErrNotFound is returned by ConditionalStore.Get when key has never been written. Fence treats this as "recorded epoch 0" — any candidateEpoch >= 1 is admissible against an empty mirror.
var ErrStaleEpoch = errors.New("writefence: candidate epoch is not strictly greater than the recorded epoch")
ErrStaleEpoch is returned when candidateEpoch is not STRICTLY greater than the epoch already recorded in the store for this (plugin, shard). This is the invariant that closes the same-epoch double-write: a deposed or partitioned writer retrying at the epoch it was (or believes it was) granted is rejected even though it never raced anyone at the store level — the check runs before any write is attempted.
Functions ¶
This section is empty.
Types ¶
type ConditionalStore ¶
type ConditionalStore interface {
// Get returns the marker bytes and the store's authoritative version
// token (e.g. ETag / generation) for key. Returns ErrNotFound (wrapped)
// when key has never been written.
Get(ctx context.Context, key string) (data []byte, version string, err error)
// PutIfVersion writes data at key IFF the store's current version for
// key still equals expectVersion. Returns the new version on success, or
// a wrapped ErrConflict when the precondition failed.
PutIfVersion(ctx context.Context, key string, data []byte, expectVersion string) (newVersion string, err error)
}
ConditionalStore is the atomic-CAS object-store surface the fence needs. It models the S3 If-Match / GCS generation-match primitive directly: PutIfVersion succeeds only if the store's CURRENT version for key still equals expectVersion at the moment of the write. expectVersion == "" means "key must not exist yet" (a create-only put, i.e. S3 If-None-Match: *).
Implementations MUST perform this check atomically, server-side. A client-side "Get, compare in Go, then Put" is NOT a valid implementation — that non-atomicity is exactly the race this package exists to close. See store.go for a real implementation over minio-go's native SetMatchETag/SetMatchETagExcept, and fake_test.go for the in-memory model used by the concurrency tests.
type EpochSource ¶
EpochSource resolves the CANDIDATE epoch a writer should push at for a given (plugin, shard). It is the seam the control-plane epoch-fenced writer lease (clients/controlplane — Stage 1, shadow-only today) drops into once it graduates from shadow: a live implementation there reads the currently-granted lease epoch for (plugin, shard) from the Quasar PQ-BFT RSM. This package imports nothing from clients/controlplane and has no opinion on how the epoch is decided — only on what happens once a push arrives claiming one.
type Fence ¶
type Fence struct {
Store ConditionalStore
}
Fence enforces strict-epoch, atomically-CAS'd admission for a shard's single mirror object: one shard, one object, one conditional write. The epoch advance and the payload append happen in the SAME PutIfVersion call, so there is no window between "epoch admitted" and "data written" in which a second writer can land a conflicting append.
func (*Fence) Latest ¶
func (f *Fence) Latest(ctx context.Context, plugin, shard string) (epoch uint64, writer string, payload []byte, err error)
Latest returns the currently-recorded (epoch, writer, payload) for (plugin, shard), for readers / diagnostics / tests. ErrNotFound if the shard has never been pushed.
func (*Fence) Push ¶
func (f *Fence) Push(ctx context.Context, plugin, shard, writer string, epoch uint64, payload []byte) error
Push admits and (atomically with admission) writes payload for (plugin, shard) at epoch, attributed to writer. It enforces, in order:
- STRICT monotonicity: epoch must be > the epoch currently recorded in the store for this shard (0 if the shard has never been pushed). A push at an epoch equal to or below the recorded one is rejected with ErrStaleEpoch BEFORE any write is attempted — this is what closes the same-epoch double-write: a deposed/partitioned writer retrying at the epoch it was granted can never pass this check once ANY push at that epoch (its own original one, or a racing peer's) has already landed.
- Atomicity: the header (epoch, writer) advance and the payload append are written in a single PutIfVersion, conditioned on the store's CURRENT version for the shard — read fresh on every attempt, never cached. If a concurrent pusher's write lands first, our PutIfVersion fails with ErrConflict and we retry: re-read (now sees the winner's epoch), re-run the strict check. A same-epoch racer loses this second check (ErrStaleEpoch); a legitimately higher epoch retries the CAS against the new version and, absent further contention, wins.
The store — never an in-memory cache — is the sole arbiter of "what epoch currently owns this shard's mirror."
type MinioConditionalStore ¶
MinioConditionalStore is a real ConditionalStore backed by minio-go's native optimistic-locking extension (PutObjectOptions.SetMatchETag / SetMatchETagExcept — an If-Match / If-None-Match conditional PUT against any S3-compatible endpoint). No new dependency: github.com/minio/minio-go/v7 is already vendored at v7.0.100 (clients/s3 uses the same client against the SeaweedFS S3 gateway) — nothing here bumps go.mod.
WIRING NOTE — this is a real, working atomic-CAS store, but it is NOT (yet) how internal/org's HIP-0107 per-org replication path talks to storage: that path goes through github.com/hanzoai/vfs's content-addressable block layer (see internal/org/vfsstore.go), whose Store/Backend interfaces expose NO conditional-write primitive at all — "Overwriting is allowed" is the verbatim upstream doc comment on backend.Backend.Put. Wiring this fence under that specific path would require hanzoai/vfs (an external module this repo does not own) to grow a conditional-Put capability at the backend layer — out of scope for a non-forking, surgical patch here.
Until that lands upstream, MinioConditionalStore is the fence's real backing store for any (plugin, shard) mirror addressed directly against an S3-compatible endpoint (e.g. the same SeaweedFS gateway clients/s3 already speaks to via minio-go) rather than through vfs's block/backend abstraction. The recommended near-term wiring for a shard that needs the fence TODAY: point its mirror at a dedicated bucket/prefix through this store, independent of the vfs-block path other per-org data still uses.
func NewMinioConditionalStore ¶
func NewMinioConditionalStore(client *minio.Client, bucket string) *MinioConditionalStore
NewMinioConditionalStore builds a ConditionalStore over an already constructed minio client and bucket. The caller owns the client's lifecycle (credentials, TLS, endpoint resolution) — this type adds only the conditional-write semantics on top.
func (*MinioConditionalStore) Get ¶
Get implements ConditionalStore. The ETag and the bytes come from ONE GET response: obj.Stat() reads the header of the same request io.ReadAll drains, so the returned version is guaranteed consistent with the returned data. (A separate StatObject + GetObject would be two round trips whose ETag and body could straddle a concurrent write — not unsound here since a stale version only makes the subsequent PutIfVersion CAS fail and retry, but this closes the window rather than relying on the CAS to absorb it.)
func (*MinioConditionalStore) PutIfVersion ¶
func (s *MinioConditionalStore) PutIfVersion(ctx context.Context, key string, data []byte, expectVersion string) (string, error)
PutIfVersion implements ConditionalStore using MinIO's If-Match (advance) / If-None-Match: * (create-only) conditional PUT. The server — not this process — evaluates the precondition atomically.
type StaticEpochSource ¶
type StaticEpochSource uint64
StaticEpochSource is the trivial EpochSource used before the control plane graduates from shadow: every (plugin, shard) reports a fixed epoch chosen by the caller (e.g. 1, matching today's "there is only one writer, ever" assumption). It exists so callers can adopt the Fence/EpochSource seam now without waiting on controlplane, and swap in the real lease-backed source later with no change to call sites.
Source Files