Documentation
¶
Overview ¶
Package sourcecache defines the connector-facing surface of source-cache replay (see proto/c1/connector/v2/annotation_source_cache.proto).
NOT YET WIRED. This package describes the intended contract, and the storage and eligibility machinery beneath it is in place, but the syncer does not install a Lookup or consume these annotations yet: SyncOpAttrs carries no source-cache field, and nothing outside this package references Lookup, SetLookup, or NoopLookup. A connector written against the surface below will compile and do nothing until the orchestration lands. The wiring described here is present tense on purpose — it is the contract the orchestration must satisfy, not a description of today's behavior.
A connector that can cheaply revalidate upstream data — HTTP conditional requests (GitHub), delta queries (Microsoft Graph) — opts in by attaching SourceCacheCapability MODE_READ_WRITE to its Validate response. During a sync it looks up the previous validator for a scope via the Lookup the SDK provides on SyncOpAttrs, revalidates upstream, and either emits fresh rows tagged with SourceCacheRecord or asks the SDK to replay the previous rows with SourceCacheReplay.
The connector owns scope computation; the SDK only keys storage by the connector-supplied scope key. The validator (etag, delta token) is opaque to the SDK.
Invariant that keeps replay safe: a connector must only emit SourceCacheReplay for a scope whose validator it received from THIS sync's Lookup. The lookup need not happen in the same call that emits the replay: a planning call may batch-resolve many scopes and pass the verdicts to sibling cursors through EnqueuePageTokens page tokens — that satisfies the invariant, because the validator still originates from the consuming sync. What's forbidden is a validator that outlives a sync (connector-side caches, config, upstream echoes). When source cache is disabled or degraded (no capability, no usable previous sync, unsupported storage engine) the SDK installs NoopLookup, every lookup misses, and a well-behaved connector naturally falls back to full fetch.
Replay equivalence: a cached sync must reproduce what a full resync would produce. Replayed rows are verbatim copies of the previous sync's rows with one deliberate exception — expander-written Sources on direct grants (classified by a self-source entry, mirroring RollbackExpansion) are stripped at copy time so the current sync's expansion recomputes them from true state; re-expansion only adds contributions, so carrying them verbatim would immortalize contributions removed upstream. Connector-set Sources (no self-source) are public connector data and survive replay byte-for-byte.
Index ¶
- func HashScope(canonicalScope string) string
- func ScopeFromContext(ctx context.Context) string
- func ValidateRowKind(rowKind RowKind) error
- func ValidateScopeKey(scopeKey string) error
- func WithScope(ctx context.Context, scopeKey string) context.Context
- type Entry
- type Lookup
- type NoopLookup
- type RowKind
- type SetLookup
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HashScope ¶
HashScope returns the lowercase-hex sha256 of a canonical scope string. Convenience for connectors; any stable identifier is acceptable as a scope key (only non-emptiness and a length cap are enforced).
func ScopeFromContext ¶
ScopeFromContext returns the scope key set by WithScope, or "".
func ValidateRowKind ¶
ValidateRowKind returns an error if rowKind is not one of the known RowKind* constants.
func ValidateScopeKey ¶
ValidateScopeKey returns an error when scopeKey is empty or unreasonably long. Connectors conventionally use HashScope, but any stable identifier is accepted.
Types ¶
type Entry ¶
type Entry struct {
// CacheValidator is opaque to the SDK: an HTTP ETag, delta token, etc.
CacheValidator string
// DiscoveredAt is when the entry was written.
DiscoveredAt time.Time
}
Entry is a previous sync's persisted validator for one scope.
type Lookup ¶
type Lookup interface {
// LookupPreviousSourceCache returns the previous sync's entry for
// (rowKind, scopeKey). found=false means no entry: fetch fresh.
// Implementations must treat internal read errors that leave fresh
// fetch available as misses rather than failing the connector call.
LookupPreviousSourceCache(ctx context.Context, rowKind RowKind, scopeKey string) (entry Entry, found bool, err error)
}
Lookup resolves a scope's previous-sync validator. The SDK provides an implementation on SyncOpAttrs; connectors call it before revalidating upstream.
type NoopLookup ¶
type NoopLookup struct{}
NoopLookup is the Lookup installed when source cache is disabled or degraded. Every lookup misses.
func (NoopLookup) LookupPreviousSourceCache ¶
type RowKind ¶
type RowKind string
RowKind partitions source-cache scopes by the row type they produce. It doubles as the row_kind value stored in SourceCacheEntryRecord.
Source Files