storeseed

package
v0.2.17 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 27, 2026 License: MPL-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package storeseed implements the declarative store-seed channel: reserved stack subtrees (VECTORS/, KV/) that `txco apply` reconciles into the runtime stores (the vector store, the KV store), so a stack's known data set travels WITH the deploy and the tree is the desired state (sync = re-apply).

This file (paths.go) is the leaf layer: just the reserved-path vocabulary, with no dependency on the stores. It is imported by the CLI (to collect the packs into the bundle), the admin producer (to CAS-back the pack bytes like FILES/), and the control-event applier (to keep pack bytes out of the in-memory runtime DB). The materializer registry that actually reconciles packs into stores lives alongside in this package (storeseed.go).

Index

Constants

View Source
const (
	DirVectors = "VECTORS"
	DirKV      = "KV"

	// Pack store kinds — the Materializer.Kind() each pack dir routes to.
	KindVector = "vector"
	KindKV     = "kv"

	// PackExt is the required extension for a pack file. We pin it so the
	// pack channel stays unambiguous (a stray FILES-style asset under
	// VECTORS/ is a deploy error, not silently reconciled).
	PackExt = ".jsonl"
)

Reserved top-level pack directories (siblings to OPS/ and FILES/). Each holds NDJSON packs: VECTORS/<collection>.jsonl, KV/<namespace>.jsonl.

Variables

This section is empty.

Functions

func IsPackPath

func IsPackPath(p string) bool

IsPackPath reports whether a stack_files path is a store-seed pack — i.e. it lives under one of the reserved pack directories. Used by the producer / applier to give packs the same CAS-backed, out-of-runtime-DB treatment as FILES/ static assets (large pre-computed vectors must never inline into a control event or a data-plane node's in-memory DB).

func KindForPath

func KindForPath(p string) string

KindForPath returns the store kind ("vector" | "kv") a pack path routes to, or "" when the path is not under a reserved pack directory.

func PackName

func PackName(p string) string

PackName returns the collection/namespace name a pack path seeds: "VECTORS/books.jsonl" → "books". It returns "" when the path is not a pack, is nested below the pack dir (no slashes allowed in the name), or lacks the .jsonl extension — the same shape validateStackFilePath enforces at upload.

Types

type Materializer

type Materializer interface {
	// Kind is the pack kind this materializer owns (KindVector | KindKV).
	Kind() string

	// Shared reports whether the backing store is shared across fleet nodes
	// (pgvector, redis) rather than per-node (sqlite-vec, boltdb). A shared
	// store is reconciled exactly ONCE — on the activation origin (the control
	// plane) — so concurrent data-plane appliers don't race the delete-missing
	// pass. A per-node store is reconciled on every node from its CAS-resolved
	// pack. The wiring layer that picks the backend declares this.
	Shared() bool

	// Reconcile makes the store match packs — the full set of this kind's packs
	// in this version. Each pack owns its collection/namespace (managed scope):
	// upsert the pack's items + delete managed items absent from it. A pack with
	// zero items empties (but does not necessarily drop) its target. Whole-
	// collection drop (a pack removed entirely between versions) is the
	// Reconciler's job (P4 prior-version diff), not the Materializer's.
	Reconcile(ctx context.Context, scope Scope, packs []RawPack) error
}

Materializer reconciles the store-seed packs of one Kind into its backing store. The concrete materializers live in subpackages (vecseed, kvseed) so this core package stays free of store dependencies — the CLI and the control-event applier import storeseed only for the path vocabulary.

type RawPack

type RawPack struct {
	Path  string // e.g. "VECTORS/books.jsonl"
	Kind  string // KindVector | KindKV
	Name  string // collection / namespace the pack owns
	Bytes []byte // raw NDJSON
}

RawPack is a store-seed pack as it travels through the deploy: the stack_files path it came from, the store kind + owned name derived from that path, and the resolved NDJSON bytes (one item per line). The control plane reads the bytes inline from stack_files; a data-plane node resolves them from the shared CAS by fingerprint. The materializer framework parses Bytes into typed items per kind.

func NewRawPack

func NewRawPack(path string, b []byte) (RawPack, bool)

NewRawPack builds a RawPack from a pack path + its resolved bytes, deriving Kind and Name from the path. It returns (RawPack{}, false) when the path is not a well-formed pack path (caller should have validated, but this keeps the loader honest).

func (RawPack) Lines

func (p RawPack) Lines() [][]byte

Lines returns the pack's non-empty, whitespace-trimmed NDJSON lines, in file order. Blank lines (and a trailing newline) are skipped so an editor-friendly pack with stray blank lines still parses cleanly.

type Reconciler

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler dispatches a version's packs to the registered materializers. It is built once at boot (server.go) from the live stores and injected into the activation path; reconcile is invoked best-effort AFTER the activation tx commits, so a slow or failing store never stalls or rolls back a deploy.

func NewReconciler

func NewReconciler(ms ...Materializer) *Reconciler

NewReconciler builds a Reconciler from the given materializers. A nil or empty set yields a no-op Reconciler (Reconcile returns nil) — the open-core default when no seedable store is configured. A duplicate Kind is a programming error and panics at boot.

func (*Reconciler) Kinds

func (r *Reconciler) Kinds() []string

Kinds returns the pack kinds this Reconciler can materialise (sorted), for diagnostics.

func (*Reconciler) Reconcile

func (r *Reconciler) Reconcile(ctx context.Context, scope Scope, packs []RawPack, origin bool) error

Reconcile groups packs by kind and dispatches each group to its materializer. origin marks the activation origin (the control plane): shared-store materializers reconcile only when origin is true; per-node materializers always run. Best-effort and exhaustive — every materializer is attempted and the errors are aggregated, so one failing store doesn't skip the others. A pack kind with no registered materializer is reported (a deploy referencing a store this node can't seed should be visible, not silent).

type Scope

type Scope struct {
	Tenant  string
	Stack   string
	Version int64
}

Scope identifies the stack version a reconcile is materialising. Tenant scopes every store call; Stack/Version are carried for logging + future prior-version diffing (whole-collection drop, P4).

Directories

Path Synopsis
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL