storeseed

package
v0.2.44 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: MPL-2.0 Imports: 6 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DirVectors   = "VECTORS"
	DirKV        = "KV"
	DirBlobs     = "BLOBS"
	DirCalendars = "CALENDARS"
	DirContacts  = "CONTACTS"
	DirSources   = "SOURCES"

	KindVector   = "vector"
	KindKV       = "kv"
	KindBlob     = "blob"
	KindCalendar = "calendar"
	KindContact  = "contact"
	KindSource   = "source"

	PackExt = ".jsonl"
)

Reserved store-seed trees under OPS/<stack>/. Each maps to a pack kind owned by one Materializer (vecseed, kvseed, blobseed, calseed, conseed).

Variables

This section is empty.

Functions

func CalendarPackName added in v0.2.26

func CalendarPackName(name string) (username, calendar string, ok bool)

CalendarPackName splits a CALENDARS/ pack name into its account and calendar ("<username>/<calendar>" → username, calendar).

func ContactsPackName added in v0.2.26

func ContactsPackName(name string) (username, addressbook string, ok bool)

ContactsPackName splits a CONTACTS/ pack name into its account and address book ("<username>/<addressbook>" → username, addressbook).

func IsBlobPath added in v0.2.24

func IsBlobPath(p string) bool

IsBlobPath reports whether p is a BLOBS/ pack row (one blob per file).

func IsPackPath

func IsPackPath(p string) bool

IsPackPath reports whether p lives in a store-seed tree.

func KindForPath

func KindForPath(p string) string

KindForPath returns the pack kind for p ("" when p is not a pack path). Prefix match, exact-case — the SQL classification downstream is LIKE 'VECTORS/%' etc., and Postgres LIKE is case-sensitive.

func PackName

func PackName(p string) string

PackName returns the name the pack owns, or "" when p is not a well-formed pack path. VECTORS/ and KV/ packs are a single "<name>.jsonl" segment (the collection / namespace is unambiguous; no nesting). A CALENDARS/ pack is exactly two segments, "CALENDARS/<username>/<calendar>.jsonl", and owns that one calendar of that account — the name is "<username>/<calendar>"; a CONTACTS/ pack is the same shape, "CONTACTS/<username>/<addressbook>.jsonl". A BLOBS/ row is the opposite shape: the tree IS the hierarchy, so the name is everything after "BLOBS/" — "BLOBS/faqs/house-01.doc" owns the blob name "faqs/house-01.doc". Whether that is a VALID blob name is the blob package's call (blob.ValidName), enforced at the write boundary and by the collector.

Types

type Materializer

type Materializer interface {
	// Kind is the pack kind this materializer owns (KindVector | KindKV).
	Kind() string

	// Shared reports whether the backing store is shared across fleet nodes
	// (pgvector, redis) rather than per-node (sqlite-vec, boltdb). A shared
	// store is reconciled exactly ONCE — on the activation origin (the control
	// plane) — so concurrent data-plane appliers don't race the delete-missing
	// pass. A per-node store is reconciled on every node from its CAS-resolved
	// pack. The wiring layer that picks the backend declares this.
	Shared() bool

	// Reconcile makes the store match packs — the full set of this kind's packs
	// in this version. Each pack owns its collection/namespace (managed scope):
	// upsert the pack's items + delete managed items absent from it. A pack with
	// zero items empties (but does not necessarily drop) its target. Whole-
	// collection drop (a pack removed entirely between versions) is the
	// Reconciler's job (P4 prior-version diff), not the Materializer's.
	Reconcile(ctx context.Context, scope Scope, packs []RawPack) error
}

Materializer reconciles the store-seed packs of one Kind into its backing store. The concrete materializers live in subpackages (vecseed, kvseed) so this core package stays free of store dependencies — the CLI and the control-event applier import storeseed only for the path vocabulary.

type RawPack

type RawPack struct {
	Path  string // e.g. "VECTORS/books.jsonl", "BLOBS/faqs/house-01.doc"
	Kind  string // KindVector | KindKV | KindBlob
	Name  string // collection / namespace / blob name the pack owns
	Bytes []byte // raw NDJSON (nil for a blob row)
	Hash  string // sha256 hex of the row's bytes (blob rows: the CAS key)
}

RawPack is a store-seed pack as it travels through the deploy: the stack_files path it came from, the store kind + owned name derived from that path, and the resolved NDJSON bytes (one item per line). The control plane reads the bytes inline from stack_files; a data-plane node resolves them from the shared CAS by fingerprint. The materializer framework parses Bytes into typed items per kind.

A BLOBS/ row is the exception: its bytes are the blob itself and are NEVER loaded — the name index only needs the hash (Hash), which the loader sets from the stack_files row; Bytes stays nil.

func EmptyTree added in v0.2.24

func EmptyTree(kind string) RawPack

EmptyTree is the marker pack a Reconciler passes to a kind whose tree was removed entirely between versions (Path == ""), so the materializer still runs its delete-missing pass. Only the blob kind consumes it today; the single-file kinds keep "pack removed = stop managing".

func NewRawPack

func NewRawPack(path string, b []byte) (RawPack, bool)

NewRawPack builds a RawPack from a pack path + its resolved bytes, deriving Kind and Name from the path. It returns (RawPack{}, false) when the path is not a well-formed pack path (caller should have validated, but this keeps the loader honest).

func (RawPack) Lines

func (p RawPack) Lines() [][]byte

Lines returns the pack's non-empty, whitespace-trimmed NDJSON lines, in file order. Blank lines (and a trailing newline) are skipped so an editor-friendly pack with stray blank lines still parses cleanly.

type Reconciler

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler dispatches a version's packs to the registered materializers. It is built once at boot (server.go) from the live stores and injected into the activation path; reconcile is invoked best-effort AFTER the activation tx commits, so a slow or failing store never stalls or rolls back a deploy.

func NewReconciler

func NewReconciler(ms ...Materializer) *Reconciler

NewReconciler builds a Reconciler from the given materializers. A nil or empty set yields a no-op Reconciler (Reconcile returns nil) — the open-core default when no seedable store is configured. A duplicate Kind is a programming error and panics at boot.

func (*Reconciler) Kinds

func (r *Reconciler) Kinds() []string

Kinds returns the pack kinds this Reconciler can materialise (sorted), for diagnostics.

func (*Reconciler) LocalKinds added in v0.2.24

func (r *Reconciler) LocalKinds() []string

LocalKinds returns the kinds whose materializer reconciles PER-NODE (Shared() == false), sorted. The activation origin uses it to warn when a fleet node seeds into a node-local store — the seed would be invisible to every other node.

func (*Reconciler) Reconcile

func (r *Reconciler) Reconcile(ctx context.Context, scope Scope, packs []RawPack, origin bool) error

Reconcile groups packs by kind and dispatches each group to its materializer. origin marks the activation origin (the control plane): shared-store materializers reconcile only when origin is true; per-node materializers always run. Best-effort and exhaustive — every materializer is attempted and the errors are aggregated, so one failing store doesn't skip the others. A pack kind with no registered materializer is reported (a deploy referencing a store this node can't seed should be visible, not silent).

type Scope

type Scope struct {
	Tenant  string
	Stack   string
	Version int64
	// Force is the `txco data apply --force` signal: the tree WINS over
	// runtime drift — a materializer re-asserts every seeded item and drops
	// the ones the tree no longer ships, even where it would otherwise leave
	// a runtime edit alone (blobseed). The caller also reconciles every pack,
	// not just the changed ones.
	Force bool
}

Scope identifies the stack version a reconcile is materialising. Tenant scopes every store call; Stack/Version are carried for logging + future prior-version diffing (whole-collection drop, P4).

Directories

Path Synopsis
Package blobseed is the BLOBS/ store-seed Materializer: it reconciles a stack's BLOBS/** tree (chassis/storeseed) into the tenant's blob name index (chassis/blob).
Package blobseed is the BLOBS/ store-seed Materializer: it reconciles a stack's BLOBS/** tree (chassis/storeseed) into the tenant's blob name index (chassis/blob).
Package calseed is the CALENDARS/ store-seed Materializer: it reconciles CALENDARS/<username>/<calendar>.jsonl packs (chassis/storeseed) into the calendar store (chassis/calendar) the `calendar` personality serves.
Package calseed is the CALENDARS/ store-seed Materializer: it reconciles CALENDARS/<username>/<calendar>.jsonl packs (chassis/storeseed) into the calendar store (chassis/calendar) the `calendar` personality serves.
Package conseed is the CONTACTS/ store-seed Materializer: it reconciles CONTACTS/<username>/<addressbook>.jsonl packs (chassis/storeseed) into the contacts store (chassis/contacts) the `contacts` personality serves.
Package conseed is the CONTACTS/ store-seed Materializer: it reconciles CONTACTS/<username>/<addressbook>.jsonl packs (chassis/storeseed) into the contacts store (chassis/contacts) the `contacts` personality serves.
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package kvseed is the KV store-seed Materializer: it reconciles KV/<namespace>.jsonl packs (chassis/storeseed) into the tenant KV store.
Package srcseed is the SOURCES/ store-seed Materializer: it reconciles SOURCES/<pack>.jsonl packs (chassis/storeseed) into the tenant_sources runtime table (chassis/source).
Package srcseed is the SOURCES/ store-seed Materializer: it reconciles SOURCES/<pack>.jsonl packs (chassis/storeseed) into the tenant_sources runtime table (chassis/source).
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.
Package vecseed is the vector store-seed Materializer: it reconciles VECTORS/<collection>.jsonl packs (chassis/storeseed) into a vector.Store.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL