README
¶
manifeststore — PostgreSQL manifest storage
Stores snapshot manifests in PostgreSQL, including version metadata, batched latest-manifest reads, and payload walking for garbage collection.
Go module: github.com/scitrera/memorylayer-storage/manifeststore.
Initial version: 0.7.1, licensed under AGPL-3.0-only.
Pass a database connection to the store. The manifest database used by garbage collection must be the same source of live manifests used by writers.
From the repository root:
cd manifeststore
go test ./...
go build ./...
The root go.work resolves sibling modules for development. Published module
versions use manifeststore/vX.Y.Z tags. See development and releases
and operations.
Documentation
¶
Overview ¶
Package manifeststore provides a PostgreSQL-backed snapshot.SnapshotStore for mlfs slice manifests. It stores the manifest payload bytes alongside the SnapshotMetadata fields in a dedicated table, avoiding per-read S3 round-trips and keeping slice manifest metadata consistent with the rest of the mlfs PG metadata.
The store is stateless over a *sql.DB pool and is safe for concurrent use. Schema migration follows the same pattern as internal/meta: a single idempotent DDL string applied via Migrate, registered separately from the meta engine so that the manifeststore can be wired independently.
Index ¶
- Constants
- func Migrate(ctx context.Context, db *sql.DB) error
- type Store
- func (s *Store) Delete(ctx context.Context, key snapshot.SnapshotKey, version string) error
- func (s *Store) Get(ctx context.Context, key snapshot.SnapshotKey, version string) (io.ReadCloser, snapshot.SnapshotMetadata, error)
- func (s *Store) GetLatest(ctx context.Context, key snapshot.SnapshotKey) (io.ReadCloser, snapshot.SnapshotMetadata, error)
- func (s *Store) GetLatestBatch(ctx context.Context, keys []snapshot.SnapshotKey) (map[snapshot.SnapshotKey]snapshot.BatchLatestEntry, error)
- func (s *Store) GetLatestMetadata(ctx context.Context, key snapshot.SnapshotKey) (snapshot.SnapshotMetadata, error)
- func (s *Store) List(ctx context.Context, key snapshot.SnapshotKey) ([]snapshot.SnapshotMetadata, error)
- func (s *Store) Put(ctx context.Context, key snapshot.SnapshotKey, meta snapshot.SnapshotMetadata, ...) (snapshot.SnapshotMetadata, error)
- func (s *Store) Walk(ctx context.Context, ...) error
- func (s *Store) WalkPayloads(ctx context.Context, ...) error
Constants ¶
const Schema = `` /* 713-byte string literal not displayed */
Schema is the manifeststore DDL, idempotent. The table uses (tenant, workspace, user_field, owner_key, version) as a natural composite key. version is the RFC3339Nano+hex string produced by snapshot.NewVersion, which sorts lexicographically newest-last, so "MAX(version)" gives GetLatest without a separate index scan — ORDER BY version DESC LIMIT 1 is the query shape.
Columns mirror SnapshotMetadata exactly plus the raw payload blob.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a PostgreSQL-backed snapshot.SnapshotStore. It is stateless and concurrency-safe.
func New ¶
New wraps db as a Store. db must already be migrated (call Migrate first). log may be nil; in that case slog.Default() is used.
func (*Store) Delete ¶
Delete removes a specific version. Returns nil if already absent (idempotent).
func (*Store) Get ¶
func (s *Store) Get(ctx context.Context, key snapshot.SnapshotKey, version string) (io.ReadCloser, snapshot.SnapshotMetadata, error)
Get retrieves the payload and metadata for a specific version. Caller must Close the returned reader. Returns ErrNoSnapshot if the version is absent.
func (*Store) GetLatest ¶
func (s *Store) GetLatest(ctx context.Context, key snapshot.SnapshotKey) (io.ReadCloser, snapshot.SnapshotMetadata, error)
GetLatest returns the blob and metadata for the highest-version snapshot of key. Returns ErrNoSnapshot when none exist.
func (*Store) GetLatestBatch ¶
func (s *Store) GetLatestBatch(ctx context.Context, keys []snapshot.SnapshotKey) (map[snapshot.SnapshotKey]snapshot.BatchLatestEntry, error)
GetLatestBatch resolves the latest version of each key in keys, returning each key's manifest payload + metadata. Keys with no manifest are absent from the result (not an error). It runs ONE query per distinct (tenant, workspace, user_field) partition; mlfs slice keys set only OwnerKey, so that is a single query. The shape is DISTINCT ON (owner_key) ... ORDER BY owner_key, version DESC — the slice_manifest_latest index serves it directly — which removes the ~N−1 round trips a per-key GetLatest loop costs a prefetch window (and relieves the metadata-pool pressure that bounds -readahead-concurrency). Implements snapshot.BatchLatestGetter.
func (*Store) GetLatestMetadata ¶
func (s *Store) GetLatestMetadata(ctx context.Context, key snapshot.SnapshotKey) (snapshot.SnapshotMetadata, error)
GetLatestMetadata returns only the metadata for the highest-version snapshot without fetching the payload. Returns ErrNoSnapshot when none exist.
func (*Store) List ¶
func (s *Store) List(ctx context.Context, key snapshot.SnapshotKey) ([]snapshot.SnapshotMetadata, error)
List returns metadata for all versions of key, newest first (descending version order). Returns an empty slice (not ErrNoSnapshot) when none exist.
func (*Store) Put ¶
func (s *Store) Put(ctx context.Context, key snapshot.SnapshotKey, meta snapshot.SnapshotMetadata, r io.Reader) (snapshot.SnapshotMetadata, error)
Put writes a new snapshot version for key. The store generates the version via snapshot.NewVersion, reads the payload fully from r, stores both, then returns the stamped metadata. SizeBytes in the input meta is ignored; the store computes it from the stream.
func (*Store) Walk ¶
func (s *Store) Walk(ctx context.Context, yield func(snapshot.SnapshotKey, string, snapshot.SnapshotMetadata) error) error
Walk yields every (key, version, meta) tuple in the store, in unspecified order. If the yield function returns an error, Walk stops and returns that error. Used by GC to compute the live set across all stored manifests. Payload is NOT fetched by Walk to bound memory on large stores.
Conservative GC behaviour: if a row's tags JSON fails to unmarshal, Walk does NOT skip the row. Instead it yields the manifest with nil Tags and logs a WARN. Skipping a manifest would allow GC to treat the corresponding blobs as unreferenced and delete them — a silent data-loss risk. Callers that need valid tags should handle nil Tags defensively.
func (*Store) WalkPayloads ¶
func (s *Store) WalkPayloads(ctx context.Context, yield func(key snapshot.SnapshotKey, version string, meta snapshot.SnapshotMetadata, payload []byte) error) error
WalkPayloads streams every manifest's payload ALONGSIDE its metadata in one query, so the casstore GC can parse chunk refs inline instead of issuing a Get per manifest (the M-individual-round-trips cost, TECH_DEBT #28). Rows are streamed (the payload is handed to the callback, never accumulated), so memory stays bounded on large stores. Same conservative tag handling as Walk: a corrupt tags blob yields nil Tags + a WARN rather than skipping the row (skipping would let GC delete referenced blobs). Implements snapshot.PayloadWalker.
Source Files
Directories