Documentation
¶
Overview ¶
Package remoteindex implements snapshot.DedupStore over the blobgw control plane using NATS request-reply and the ctlproto wire contract (ADR-001 §2.2, §2.5). An mlfs node creates one RemoteDedupStore per NATS connection and passes it as the DedupStore when constructing a casstore GlobalIndex.
Transport is abstracted behind Requester so tests can substitute an in-memory fake without a real NATS server. The production path uses NatsRequester, which wraps *nats.Conn.RequestWithContext and maps nats.ErrNoResponders to ErrNoResponders so callers can distinguish "nobody home" from other errors.
Batching: LookupBatch and Record automatically split input slices into sub-batches of at most ctlproto.MaxChunkHashesPerBatch to respect the NATS 1 MB max-payload limit. PurgePacks does the same for pack hashes.
Index ¶
- Variables
- type NatsRequester
- type Option
- type RemoteDedupStore
- func (s *RemoteDedupStore) Associate(ctx context.Context, domain string, assocs []snapshot.PackAssociation) error
- func (s *RemoteDedupStore) Lookup(ctx context.Context, domain, chunkHash string) (snapshot.PackRef, bool, error)
- func (s *RemoteDedupStore) LookupBatch(ctx context.Context, domain string, chunkHashes []string) (map[string]snapshot.PackRef, error)
- func (s *RemoteDedupStore) PurgePacks(ctx context.Context, domain string, packHashes []string) error
- func (s *RemoteDedupStore) Record(ctx context.Context, domain string, locs []snapshot.ChunkLocation) error
- type Requester
Constants ¶
This section is empty.
Variables ¶
var ErrNoResponders = errors.New("remoteindex: no responders on subject")
ErrNoResponders is returned by NatsRequester when NATS reports that no subscriber is listening on the requested subject. This is the sentinel that lets callers distinguish "blobgw is down / not yet subscribed" from a generic network or decode error.
Functions ¶
This section is empty.
Types ¶
type NatsRequester ¶
type NatsRequester struct {
// contains filtered or unexported fields
}
NatsRequester adapts a *nats.Conn to Requester. It wraps RequestWithContext and converts nats.ErrNoResponders to the package-local ErrNoResponders so callers can type-switch on it without importing the NATS package.
func NewNatsRequester ¶
func NewNatsRequester(conn *nats.Conn, timeout time.Duration) *NatsRequester
NewNatsRequester constructs a NatsRequester. timeout is the per-request deadline applied when the supplied context carries no deadline of its own. Pass 0 to use the package default (5 s).
type Option ¶
type Option func(*options)
Option is a functional option for NewRemoteDedupStore.
func WithCodec ¶
WithCodec overrides the codec used to marshal/unmarshal control-plane messages. Defaults to ctlproto.DefaultCodec (currently JSONCodec).
func WithTimeout ¶
WithTimeout sets the per-request timeout applied when the caller's context has no deadline. Defaults to 5 s. Has no effect when using a NatsRequester constructed with an explicit timeout — this option controls a timeout that RemoteDedupStore itself can apply via context wrapping when the per-request context is background/TODO.
type RemoteDedupStore ¶
type RemoteDedupStore struct {
// contains filtered or unexported fields
}
RemoteDedupStore holds the transport and codec needed to speak the ctlproto control-plane protocol.
func NewRemoteDedupStore ¶
func NewRemoteDedupStore(r Requester, opts ...Option) *RemoteDedupStore
NewRemoteDedupStore constructs a RemoteDedupStore. r must not be nil. Optional Option values tune the codec and per-request timeout.
func (*RemoteDedupStore) Associate ¶
func (s *RemoteDedupStore) Associate(ctx context.Context, domain string, assocs []snapshot.PackAssociation) error
Associate implements snapshot.PackAssociator over the control plane: it tells blobgw which owner context references which packs, so reclamation can later be scoped to that owner without walking the store.
Only the association WRITE lives on this client. The rest of the reclamation lifecycle (dropping a context, publishing marks, deciding a pack may die) needs the durable index and blob-deletion authority together and therefore runs blobgw-side — a filesystem node is authoritative for neither.
A server refusal because a pack is being reclaimed is surfaced as snapshot.ErrPackObliterating, so the ChunkedStore write path fails the Put cleanly (before its manifest exists) rather than referencing bytes on their way out. Retrying re-packs the chunks, because the reclaiming pack is by then hidden from dedup lookups.
func (*RemoteDedupStore) Lookup ¶
func (s *RemoteDedupStore) Lookup(ctx context.Context, domain, chunkHash string) (snapshot.PackRef, bool, error)
Lookup returns the stored location of chunkHash in domain, or ok=false if absent. It delegates to LookupBatch with a single-element slice; LookupBatch owns the timeout wrapping so Lookup must not add its own.
func (*RemoteDedupStore) LookupBatch ¶
func (s *RemoteDedupStore) LookupBatch(ctx context.Context, domain string, chunkHashes []string) (map[string]snapshot.PackRef, error)
LookupBatch resolves many chunk hashes in one or more NATS round-trips. The returned map contains only the hashes that were found; missing hashes are absent. Input slices larger than ctlproto.MaxChunkHashesPerBatch are split into sub-batches automatically.
func (*RemoteDedupStore) PurgePacks ¶
func (s *RemoteDedupStore) PurgePacks(ctx context.Context, domain string, packHashes []string) error
PurgePacks removes every index entry in domain that points to any of the given pack hashes. Input is split into sub-batches of ctlproto.MaxPackHashesPerBatch. Idempotent; purging an absent pack is a no-op.
func (*RemoteDedupStore) Record ¶
func (s *RemoteDedupStore) Record(ctx context.Context, domain string, locs []snapshot.ChunkLocation) error
Record idempotently persists chunk locations in domain. Input slices larger than ctlproto.MaxChunkHashesPerBatch are split into sub-batches. Errors from the server or transport are returned; the casstore GlobalIndex treats DedupStore errors as best-effort (pack normally), but we do not swallow them.
type Requester ¶
type Requester interface {
Request(ctx context.Context, subject string, data []byte) ([]byte, error)
}
Requester is the single-method transport interface RemoteDedupStore relies on. It matches the semantics of nats.Conn.RequestWithContext: send data on subject, wait for exactly one reply, return its payload. Implementations must respect ctx cancellation and deadline.
Source Files