Documentation
¶
Overview ¶
Package partsync replicates flushed, immutable parts between nodes whose backends are per-node private (shared-nothing cluster mode). Head replication (cluster/replica) protects only the unflushed window; over a shared object store the flushed parts need no replication at all — but with a local-disk backend a peer cannot see them, so a replica instead *mirrors* the owner's backend objects over HTTP: it picks the newest peer copy of the engine's bucket index, copies the part objects it lacks, and installs the index last. The engine then reconciles via its ordinary LoadParts/RefreshReplica path — partsync moves backend objects, never engine state.
Ordering makes a crashed sync harmless: within a part the manifest is copied after the part's other objects, and the bucket index is written after every part, so the local index only ever references fully-copied parts (the same commit-point discipline flush uses). A half-copied part is an unreferenced orphan retried on the next pass.
Objects are content-immutable except the bucket index and the record engines' whole-set stream identity object (streams.bin, plus the metrics series.bin a prefix written before identity became part-scoped still has), so a plain presence diff drives the copy; the mutable objects are re-fetched whenever the index changed. Every fetched object is verified against the sender's checksum. Local objects the peer no longer has are pruned only after being absent for two consecutive passes, giving in-flight readers a full maintenance cycle to drain (quarantine-by-delay rather than immediate delete).
Index ¶
- Constants
- Variables
- func ListHandler(be backend.Backend) http.Handler
- func ObjectHandler(be backend.Backend) http.Handler
- func ValidKey(k string) bool
- type Client
- type KeepFunc
- type Option
- type Stats
- type Syncer
- func (s *Syncer) FetchWant(ctx context.Context, enginePrefix string, peers []string, w bucketindex.Want) (bucketindex.Entry, bool, error)
- func (s *Syncer) FetchWants(ctx context.Context, enginePrefix string, peers []string, ...) []WantResult
- func (s *Syncer) Sync(ctx context.Context, enginePrefix string, peers []string, strict bool, ...) (Stats, error)
- func (s *Syncer) Totals() Totals
- type Totals
- type WantResult
Constants ¶
const ( // ListPath is the HTTP path serving a node's backend key listing under a prefix. ListPath = "/internal/parts/list" // ObjectPath is the HTTP path serving one backend object verbatim. ObjectPath = "/internal/parts/object" // NotifyPath is the HTTP path an owner POSTs to after a flush/merge so a secondary mirrors // immediately instead of waiting for its next maintenance tick. Advisory and best-effort — // the periodic pull remains the anti-entropy source of truth. NotifyPath = "/internal/parts/notify" )
Variables ¶
var ErrNotExist = errors.New("partsync: object does not exist on peer")
ErrNotExist is returned by Client.Fetch for a key the peer does not have.
Functions ¶
func ListHandler ¶
ListHandler serves the backend keys under the "prefix" query parameter, framed as a uvarint count followed by uvarint-length-prefixed keys.
func ObjectHandler ¶
ObjectHandler serves one backend object (the "key" query parameter) verbatim, with its xxh3 checksum in a response header. A missing key is a 404.
func ValidKey ¶
ValidKey reports whether a remotely-supplied key or prefix is safe to hand to a backend: relative, slash-delimited, and free of traversal or NUL. Backends validate again (the file backend keeps every path under its root); this check is defense-in-depth at every network boundary — the serving handlers reject hostile request parameters, and the syncer rejects hostile key names a compromised peer could return.
Types ¶
type Client ¶
type Client struct {
// HTTP is the client used for peer requests; nil uses [http.DefaultClient]. Pass one with
// timeouts in production (the cluster's tuned client).
HTTP *http.Client
}
Client fetches backend listings and objects from a peer's partsync endpoints.
func (*Client) Fetch ¶
Fetch returns one object from the peer, verified against the sender's checksum. A key the peer lacks returns ErrNotExist.
type KeepFunc ¶
KeepFunc decides whether a peer-listed object key should be mirrored into this node. It lets the caller narrow a pull to a subset of a part's objects — erasure coding passes one that keeps only this node's own shard slot (plus every non-shard object), so a replica stores one shard per part instead of the whole k+m set. A nil KeepFunc keeps everything.
type Option ¶ added in v0.45.0
type Option func(*Syncer)
Option configures a Syncer.
func WithRandSeed ¶ added in v0.45.0
WithRandSeed fixes the source a repair shuffles peers with, making peer selection reproducible. For tests; production leaves it unset and gets a per-process random source.
type Stats ¶
type Stats struct {
// Synced is true when a newer peer copy was found and mirrored (Copied may still be zero
// if only the mutable objects changed).
Synced bool
// Copied is the number of objects fetched from the peer.
Copied int
// CopiedBytes is the total size of the fetched objects.
CopiedBytes int64
// Pruned is the number of stale local objects deleted.
Pruned int
// Withheld is the number of local objects a prune declined to delete because nothing
// authorized it: their part is absent from the peer's index and the peer did not say it
// removed one. That is a peer missing data it should hold, so a non-zero value is a repair
// signal, not a tuning knob.
Withheld int
// Retained is the number of parts this node kept in the installed index because the peer's
// index neither names nor accounts for them — no want, no hole, no tombstone, no successor
// containing their blocks. Like Withheld it names a peer missing data, and it is the half that
// covers a peer that made no claim at all.
Retained int
}
Stats reports what one Syncer.Sync pass did.
type Syncer ¶
type Syncer struct {
// contains filtered or unexported fields
}
Syncer mirrors engine prefixes of a per-node private backend from cluster peers. Safe for concurrent use across distinct prefixes; per-prefix passes are expected to be serial (the maintenance loop runs one task per engine).
func (*Syncer) FetchWant ¶ added in v0.45.0
func (s *Syncer) FetchWant( ctx context.Context, enginePrefix string, peers []string, w bucketindex.Want, ) (bucketindex.Entry, bool, error)
FetchWant is Syncer.FetchWants for a single want.
func (*Syncer) FetchWants ¶ added in v0.45.0
func (s *Syncer) FetchWants( ctx context.Context, enginePrefix string, peers []string, wants []bucketindex.Want, ) []WantResult
FetchWants makes local a part discharging each want — the part itself, or the largest part containing every block it covered at a higher level — by pulling its objects from whichever peer holds it, and returns one result per want in the same order.
The whole cycle's wants are answered together because that is what bounds the work: each peer's bucket index is read exactly once for the batch, and a part discharging several wants is copied once. Nothing survives the call — a peer index cached across cycles would let repair act on a stale view of what peers hold.
Peers are consulted in a shuffled order, so equal candidates spread the repair load instead of every recovering node converging on whichever peer sorts first.
A want no peer's index names is then asked of the peers' disks — one listing per peer, again once for the batch — and answered by the exact prefix from any peer holding a complete copy of it. An index is a copy of whichever owner's index last superseded it and is not evidence of what that peer holds.
OK is false with a nil error only when every peer *given to it* answered, both its index and its listing, and none of them named or held a satisfying part. A peer that could not be reached is a transient failure and is reported as an error, because the difference decides whether the owner may ever conclude the data is gone.
That is absence over the peers asked, not over the shard's owners, and the two coincide only when the caller says so: only the caller knows the expected owner set (see bucketindex.WantIncomplete). Acknowledging a loss on this answer alone would fabricate a hole over an owner that is merely restarting.
A peer's own hole never counts as a satisfying part — bucketindex.Index.Satisfying admits only data-bearing entries — so an acknowledged loss cannot propagate from one owner to the next.
The index is deliberately not installed: the caller commits the entries through its own index commit, which is what discharges the wants.
func (*Syncer) Sync ¶
func (s *Syncer) Sync(ctx context.Context, enginePrefix string, peers []string, strict bool, keep KeepFunc) (Stats, error)
Sync mirrors one engine prefix (e.g. "default/metrics") from the newest of peers into the local backend. In strict mode (an owner backfilling before it compacts) the peer copy must be strictly newer than the local one; otherwise (a replica mirroring its owner) any differing peer copy at least as new is installed. Unreachable peers are skipped; having no usable peer index is a no-op, not an error.
type Totals ¶
type Totals struct {
// Passes is every Sync attempt, including no-ops (no usable peer index, nothing newer) —
// the "is the sync loop running?" liveness probe.
Passes int64
// Mirrored is the passes that installed a newer peer copy.
Mirrored int64
// Copied is the objects fetched from peers, CopiedBytes their total size.
Copied int64
CopiedBytes int64
// Pruned is the stale local objects deleted (after the quarantine delay).
Pruned int64
// Withheld is the local objects a prune declined to delete because no peer claimed to have
// removed their part — a peer missing data it should hold. It is a repair signal: steady
// state is zero, and a value that keeps climbing names a damaged owner.
Withheld int64
// Retained is the parts kept in an installed index because the peer's index did not account
// for them. The same repair signal as Withheld, for a peer that omits rather than claims.
Retained int64
// Errors is the passes that failed part-way (retried by the next maintenance tick).
Errors int64
// LastSyncUnixNano is the wall-clock completion time of the most recent mirroring pass
// (zero until one succeeds) — the "is replication current?" staleness probe.
LastSyncUnixNano int64
}
Totals is a Syncer's cumulative activity across every prefix and pass, for the operator stats surface (storage.StoreStats). Counters only — reading it does no I/O.
type WantResult ¶ added in v0.45.0
type WantResult struct {
// Entry is the index entry of the part that came across, valid only when OK.
Entry bucketindex.Entry
// OK reports that a satisfying part is now local.
OK bool
// Err is a transient failure — a peer that could not be asked, or a copy that did not finish.
Err error
}
WantResult is one want's outcome in a repair batch, positionally matched to the want.
OK false with a nil Err is definitive absence over the peers asked; a non-nil Err is a transient failure and says nothing about whether the part exists.
Source Files