storeseed

package
v0.2.24 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: MPL-2.0 Imports: 6 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DirVectors = "VECTORS"
	DirKV      = "KV"
	DirBlobs   = "BLOBS"

	KindVector = "vector"
	KindKV     = "kv"
	KindBlob   = "blob"

	PackExt = ".jsonl"
)

Reserved store-seed trees under OPS/<stack>/. Each maps to a pack kind owned by one Materializer (vecseed, kvseed, blobseed).

Variables

This section is empty.

Functions

func IsBlobPath added in v0.2.24

func IsBlobPath(p string) bool

IsBlobPath reports whether p is a BLOBS/ pack row (one blob per file).

func IsPackPath

func IsPackPath(p string) bool

IsPackPath reports whether p lives in a store-seed tree.

func KindForPath

func KindForPath(p string) string

KindForPath returns the pack kind for p ("" when p is not a pack path). Prefix match, exact-case — the SQL classification downstream is LIKE 'VECTORS/%' etc., and Postgres LIKE is case-sensitive.

func PackName

func PackName(p string) string

PackName returns the name the pack owns, or "" when p is not a well-formed pack path. VECTORS/ and KV/ packs are a single "<name>.jsonl" segment (the collection / namespace is unambiguous; no nesting). A BLOBS/ row is the opposite shape: the tree IS the hierarchy, so the name is everything after "BLOBS/" — "BLOBS/faqs/house-01.doc" owns the blob name "faqs/house-01.doc". Whether that is a VALID blob name is the blob package's call (blob.ValidName), enforced at the write boundary and by the collector.

Types

type Materializer

type Materializer interface {
	// Kind is the pack kind this materializer owns (KindVector | KindKV).
	Kind() string

	// Shared reports whether the backing store is shared across fleet nodes
	// (pgvector, redis) rather than per-node (sqlite-vec, boltdb). A shared
	// store is reconciled exactly ONCE — on the activation origin (the control
	// plane) — so concurrent data-plane appliers don't race the delete-missing
	// pass. A per-node store is reconciled on every node from its CAS-resolved
	// pack. The wiring layer that picks the backend declares this.
	Shared() bool

	// Reconcile makes the store match packs — the full set of this kind's packs
	// in this version. Each pack owns its collection/namespace (managed scope):
	// upsert the pack's items + delete managed items absent from it. A pack with
	// zero items empties (but does not necessarily drop) its target. Whole-
	// collection drop (a pack removed entirely between versions) is the
	// Reconciler's job (P4 prior-version diff), not the Materializer's.
	Reconcile(ctx context.Context, scope Scope, packs []RawPack) error
}

Materializer reconciles the store-seed packs of one Kind into its backing store. The concrete materializers live in subpackages (vecseed, kvseed) so this core package stays free of store dependencies — the CLI and the control-event applier import storeseed only for the path vocabulary.

type RawPack

type RawPack struct {
	Path  string // e.g. "VECTORS/books.jsonl", "BLOBS/faqs/house-01.doc"
	Kind  string // KindVector | KindKV | KindBlob
	Name  string // collection / namespace / blob name the pack owns
	Bytes []byte // raw NDJSON (nil for a blob row)
	Hash  string // sha256 hex of the row's bytes (blob rows: the CAS key)
}

RawPack is a store-seed pack as it travels through the deploy: the stack_files path it came from, the store kind + owned name derived from that path, and the resolved NDJSON bytes (one item per line). The control plane reads the bytes inline from stack_files; a data-plane node resolves them from the shared CAS by fingerprint. The materializer framework parses Bytes into typed items per kind.

A BLOBS/ row is the exception: its bytes are the blob itself and are NEVER loaded — the name index only needs the hash (Hash), which the loader sets from the stack_files row; Bytes stays nil.

func EmptyTree added in v0.2.24

func EmptyTree(kind string) RawPack

EmptyTree is the marker pack a Reconciler passes to a kind whose tree was removed entirely between versions (Path == ""), so the materializer still runs its delete-missing pass. Only the blob kind consumes it today; the single-file kinds keep "pack removed = stop managing".

func NewRawPack

func NewRawPack(path string, b []byte) (RawPack, bool)

NewRawPack builds a RawPack from a pack path + its resolved bytes, deriving Kind and Name from the path. It returns (RawPack{}, false) when the path is not a well-formed pack path (caller should have validated, but this keeps the loader honest).

func (RawPack) Lines

func (p RawPack) Lines() [][]byte

Lines returns the pack's non-empty, whitespace-trimmed NDJSON lines, in file order. Blank lines (and a trailing newline) are skipped so an editor-friendly pack with stray blank lines still parses cleanly.

type Reconciler

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler dispatches a version's packs to the registered materializers. It is built once at boot (server.go) from the live stores and injected into the activation path; reconcile is invoked best-effort AFTER the activation tx commits, so a slow or failing store never stalls or rolls back a deploy.

func NewReconciler

func NewReconciler(ms ...Materializer) *Reconciler

NewReconciler builds a Reconciler from the given materializers. A nil or empty set yields a no-op Reconciler (Reconcile returns nil) — the open-core default when no seedable store is configured. A duplicate Kind is a programming error and panics at boot.

func (*Reconciler) Kinds

func (r *Reconciler) Kinds() []string

Kinds returns the pack kinds this Reconciler can materialise (sorted), for diagnostics.

func (*Reconciler) LocalKinds added in v0.2.24

func (r *Reconciler) LocalKinds() []string

LocalKinds returns the kinds whose materializer reconciles PER-NODE (Shared() == false), sorted. The activation origin uses it to warn when a fleet node seeds into a node-local store — the seed would be invisible to every other node.

func (*Reconciler) Reconcile

func (r *Reconciler) Reconcile(ctx context.Context, scope Scope, packs []RawPack, origin bool) error

Reconcile groups packs by kind and dispatches each group to its materializer. origin marks the activation origin (the control plane): shared-store materializers reconcile only when origin is true; per-node materializers always run. Best-effort and exhaustive — every materializer is attempted and the errors are aggregated, so one failing store doesn't skip the others. A pack kind with no registered materializer is reported (a deploy referencing a store this node can't seed should be visible, not silent).

type Scope

type Scope struct {
	Tenant  string
	Stack   string
	Version int64
	// Force is the `txco data apply --force` signal: the tree WINS over
	// runtime drift — a materializer re-asserts every seeded item and drops
	// the ones the tree no longer ships, even where it would otherwise leave
	// a runtime edit alone (blobseed). The caller also reconciles every pack,
	// not just the changed ones.
	Force bool
}

Scope identifies the stack version a reconcile is materialising. Tenant scopes every store call; Stack/Version are carried for logging + future prior-version diffing (whole-collection drop, P4).

Directories

Path Synopsis
Package blobseed is the BLOBS/ store-seed Materializer: it reconciles a stack's BLOBS/** tree (chassis/storeseed) into the tenant's blob name index (chassis/blob).
Package blobseed is the BLOBS/ store-seed Materializer: it reconciles a stack's BLOBS/** tree (chassis/storeseed) into the tenant's blob name index (chassis/blob).
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL