index

package
v0.18.0-alpha.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 12, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.

Storage layout (under .sdd/index/): 

chromem/        # chromem-go gob persistence
manifest.json   # entry tracking sidecar (see manifest.go)

Machine-global store resolution and the locking seam. The vector index lives outside every working tree in one content-addressed store per (repo-key, embedder-fingerprint) under the cache root — shared by a repo's checkout, its worktrees, and its connected-repo cache, so the same content is embedded once per machine (d-cpt-6cq / d-tac-nhx). All index locking is centralized here: no other package takes index locks.

Index

Constants

View Source 
const (
	// GroupCurrent holds the versions the checkout's search reads. Never
	// droppable.
	GroupCurrent = DerivationCurrent + "-current"
	// GroupBranch holds current-rule versions no entry of the checkout has: other
	// branches' states, edited entries, removed entries.
	GroupBranch = DerivationCurrent + "-branch"
	// GroupPreDerivation is the rule before derivation was recorded, recognised
	// by its eight-character version segment in chunk IDs.
	GroupPreDerivation = "v0"
	// GroupLegacy is the single-version store's unversioned chunk IDs.
	GroupLegacy = "legacy"
	// GroupStale selects every droppable group at once.
	GroupStale = "stale"
)

Version groups as `sdd index gc` reports and drops them. A version's group is its derivation rule, except that the current rule splits by whether the version is current for the collecting checkout.

View Source 
const (
	MetaEntryID              = "entry_id"
	MetaEntryHash            = "entry_hash"
	MetaChunkPath            = "chunk_path"
	MetaDepth                = "depth"
	MetaContentHash          = "content_hash"
	MetaModelFingerprint     = "model_fingerprint"
	MetaIsSummary            = "is_summary"
	MetaIsAttachment         = "is_attachment"
	MetaSourceAttachmentPath = "source_attachment_path"
)

MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.

View Source 
const CollectionName = "sdd-graph"

CollectionName is the chromem-go collection used for SDD chunks.

View Source 
const DerivationCurrent = "v1"

DerivationCurrent names the entry-derivation rule this binary hashes and chunks under (the prefix of chunking.EntryStateHash). Every version written records it, so binaries with different rules sharing one store keep their versions apart instead of collecting each other's (d-tac-c9c). Bump it when a fixed derivation rule changes.

Variables

This section is empty.

Functions

func AttachmentChunkID 

func AttachmentChunkID(entryID, attachmentPath string, n int) string

AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path. Legacy (unversioned) form — see SummaryChunkID.

func AttachmentChunkIDVersioned added in v0.17.0 

func AttachmentChunkIDVersioned(entryID, entryHash, attachmentPath string, n int) string

AttachmentChunkIDVersioned is the version-qualified n-th attachment chunk ID: entryID#v-<hash>#attach-<p6>-N.

func BodyChunkID 

func BodyChunkID(entryID string, n int) string

BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order. Legacy (unversioned) form — see SummaryChunkID.

func BodyChunkIDVersioned added in v0.17.0 

func BodyChunkIDVersioned(entryID, entryHash string, n int) string

BodyChunkIDVersioned is the version-qualified n-th body chunk ID: entryID#v-<hash>#body-N.

func ChunkIDPrefix 

func ChunkIDPrefix(entryID string) string

ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).

func DocumentsSize 

func DocumentsSize(indexDir string, chunkIDs []string) int64

DocumentsSize sums the on-disk size of the given rows. A row without a file counts zero.

func HashContent 

func HashContent(text string) string

HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.

func ManifestFingerprint added in v0.15.0 

func ManifestFingerprint(m *Manifest) string

ManifestFingerprint returns the dominant embedder fingerprint recorded in a manifest — the fingerprint a legacy in-tree index was built under, used to pick its target store during migration. Empty when the manifest holds no fingerprints.

func MigrateDir added in v0.15.0 

func MigrateDir(legacyDir, cacheRoot, repoKey string) (target string, migrated bool, err error)

MigrateDir moves a legacy index directory (an in-tree .sdd/index or a clone-cache .index) into the machine-global store, keyed by the fingerprint its own manifest records. Move-if-absent: when the target store already exists the legacy directory is left untouched and skipped — never clobbered, never merged. Returns the target dir and what happened.

func Orphans 

func Orphans(indexDir string, m *Manifest) ([]string, int64, error)

Orphans lists the row files under the collection directory that no version in m references, with their total size. They are what a delete that failed after the manifest was saved leaves behind; nothing reads them, and `sdd index gc --drop` removes them so that failure converges. chromem's collection metadata file is not a row and is never listed.

func ReadCached added in v0.17.0 

func ReadCached(ctx context.Context, indexDir string, cache *SnapshotCache, fn func(*Index) error) (reloaded bool, err error)

ReadCached runs fn against a store snapshot under the shared lock, reusing cache when the on-disk generation is unchanged and loading a fresh snapshot otherwise. The generation is read AFTER acquiring the lock and the lock is held through fn, so a writer cannot slip between the check and the query and go unseen. It reports whether it loaded a fresh snapshot (for reload counting). cache must be non-nil; the caller guards it against concurrent use.

func ReadManifestLocked 

func ReadManifestLocked(ctx context.Context, indexDir string, fn func(*Manifest) error) error

ReadManifestLocked runs fn against the manifest under the store's shared lock, without decoding the vector rows — what a report over stored versions needs. fn runs inside the lock so file sizes it reads match the manifest.

func ReadStore added in v0.16.2 

func ReadStore(ctx context.Context, indexDir string, fn func(*Index) error) error

ReadStore runs fn against a freshly loaded store while holding the store's shared lock — the read counterpart to WriteStore. A reader loads its snapshot under the lock so it never decodes half-written chromem documents, and it opens fresh per call so it reflects writes committed by the CLI or another process. An empty indexDir runs fn against a fresh in-memory store without locking (tests).

func RepoKey added in v0.15.0 

func RepoKey(repoID, repoRoot string) string

RepoKey returns the identity a repository's index store is keyed by: the declared repo_id when the repo has one, else a hash of the absolute repo root under the "local" namespace. A moved identity-less repo therefore re-embeds — accepted: `sdd init` migration covers the common case, and minting a synthetic ID would invent a second identity concept.

func StoreDir added in v0.15.0 

func StoreDir(cacheRoot, repoKey, fingerprint string) string

StoreDir resolves the machine-global store directory for one (repo-key, embedder-fingerprint) pair: <cacheRoot>/index/<repo-key>/<fp-hash>. The fingerprint is hashed because it is a free-form embedder string, not a path; keying by fingerprint means a changed embedder starts a fresh store instead of drifting inside an existing one.

func StoreSize 

func StoreSize(indexDir string) (int64, error)

StoreSize is the on-disk size of the whole store directory.

func SummaryChunkID 

func SummaryChunkID(entryID string) string

SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.

This is the LEGACY (unversioned) form: a v1 store owns exactly this ID per entry summary, interpreted as the entry's sole pre-existing version. New writes mint version-qualified IDs (SummaryChunkIDVersioned) so two versions of one entry can coexist in the shared store without colliding.

func SummaryChunkIDVersioned added in v0.17.0 

func SummaryChunkIDVersioned(entryID, entryHash string) string

SummaryChunkIDVersioned is the version-qualified summary chunk ID: entryID#v-<hash>#summary. New writes mint versioned IDs so a changed entry adds a version rather than overwriting the old one — two branches holding different versions of one entry each own their own rows in the shared store.

func VersionSegment added in v0.17.0 

func VersionSegment(entryHash string) string

VersionSegment preserves the full hash so different published versions cannot overwrite one another through a truncated chunk identity.

func WriteStore added in v0.16.2 

func WriteStore(ctx context.Context, indexDir string, fn func(*Index) error) error

WriteStore runs fn against a freshly loaded store while holding the store's exclusive lock — the one write boundary for every index mutation (build, lazy-fill, MCP reconcile). The lock is acquired BEFORE the snapshot is loaded and held through fn, so a writer never operates on a snapshot that a concurrent process has moved on from. Manifest reads and saves belonging to the write must happen inside fn so concurrent writers cannot clobber each other's manifest state. An empty indexDir runs fn against a fresh in-memory store without locking (tests). Blocks until the lock is available or ctx ends.

Types

type EntryState 

type EntryState struct {
	Versions []EntryVersion
}

EntryState is one entry's set of stored versions. See Manifest for the on-disk shape (legacy flat object for a single version, {"versions": [...]} for many).

func (EntryState) AllChunkIDs added in v0.17.0 

func (s EntryState) AllChunkIDs() []string

AllChunkIDs returns every chunk ID across all of the entry's versions.

func (EntryState) ChunkIDsOf 

func (s EntryState) ChunkIDsOf(derivation string) []string

ChunkIDsOf returns the chunk IDs of the entry's versions under the given derivation rule — what a force rebuild replaces, leaving other rules' rows in place.

func (EntryState) HasVersion added in v0.17.0 

func (s EntryState) HasVersion(hash, fingerprint string) bool

HasVersion reports whether the entry has a stored version matching both the given state hash and embedder fingerprint — the presence test both the CLI indexer and the application reconcile use to decide whether to (re-)embed.

func (EntryState) MarshalJSON added in v0.17.0 

func (s EntryState) MarshalJSON() ([]byte, error)

MarshalJSON emits the legacy flat shape for a single version and the versions-list shape for many. An entry with no versions should never be persisted (GC drops the map key instead), but it serializes harmlessly as an empty versions list.

func (*EntryState) UnmarshalJSON added in v0.17.0 

func (s *EntryState) UnmarshalJSON(data []byte) error

UnmarshalJSON loads either shape: a {"versions": [...]} object as-is, or a legacy flat {hash, fingerprint, chunk_ids, indexed_at} object as the entry's sole version.

type EntryVersion added in v0.17.0 

type EntryVersion struct {
	// Hash is the entry-state hash (entry content + summary + attachment
	// bytes) this version was built from. Read-time freshness keeps a hit only
	// when its version's Hash equals the current entry's state hash.
	Hash string `json:"hash"`
	// Fingerprint is the embedder fingerprint that produced this version's
	// chunks. Within one store (keyed per fingerprint) this is constant, but
	// it is retained per version for lint drift reporting.
	Fingerprint string `json:"fingerprint"`
	// Derivation is the derivation rule this version was written under
	// (DerivationCurrent at write time). Versions written before the field
	// existed leave it empty and are classified by chunk-ID shape instead; a
	// binary that predates the field drops it on save, and the same
	// classification recovers it. Unlike Fingerprint it never partitions the
	// store: a rule change recuts chunks in the same vector space.
	Derivation string `json:"derivation,omitempty"`
	// ChunkIDs are the IDs this version contributed. Used to resolve a hit's
	// version (legacy rows carry no entry_hash metadata), to size and delete a
	// version's rows in `sdd index gc`, and to replace them in a force rebuild.
	ChunkIDs []string `json:"chunk_ids"`
	// IndexedAt is when this version was last written; `sdd index gc` reports
	// it per group.
	IndexedAt time.Time `json:"indexed_at"`
}

EntryVersion is one stored version of an entry: the material it was built from (Hash), the embedder that produced it (Fingerprint), the derivation rule that cut it (Derivation), the chunk IDs it contributed (ChunkIDs), and when it was indexed (IndexedAt).

type Hit 

type Hit struct {
	EntryID string
	// EntryHash is the version this hit belongs to (from row metadata). Empty
	// for a legacy v1 row — the caller recovers the version through the
	// manifest (Manifest.VersionHashForChunk).
	EntryHash            string
	ChunkID              string
	Score                float32
	Text                 string
	Body                 string
	Breadcrumb           []string
	Depth                int
	IsSummary            bool
	IsAttachment         bool
	SourceAttachmentPath string
	ContentHash          string
	ModelFingerprint     string
}

Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index wraps a chromem-go persistent DB and a single collection.

func Open 

func Open(indexDir string) (*Index, error)

Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).

On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.

The load phase holds the store's shared lock so a reader never decodes half-written documents from a concurrent write session; the lock is released before Open returns — queries run on the in-memory copy. Open is the read-side entry point (the CLI finders); mutation goes through WriteStore, which acquires the exclusive lock before loading its snapshot.

func OpenInMemory 

func OpenInMemory() *Index

OpenInMemory returns a non-persistent index. Used by tests.

func (*Index) Count 

func (i *Index) Count() int

Count returns the total number of chunk rows in the index.

func (*Index) DeleteEntry 

func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error

DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.

func (*Index) Path 

func (i *Index) Path() string

Path returns the on-disk root of the index. Empty for in-memory indexes.

func (*Index) Query 

func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)

Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.

func (*Index) RemoveFiles 

func (i *Index) RemoveFiles(paths []string) error

RemoveFiles deletes row files chromem no longer needs to know about — the orphans Orphans lists — and marks the store changed so readers reload. A file already gone is not an error.

func (*Index) UpsertEntry 

func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error

UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.

Pre-conditions:

  • Each row's Embedding is populated (the index does not embed).
  • row.EntryID == entryID (validated; mismatch is a programmer error).
  • row.ChunkID is unique within rows.

type Manifest 

type Manifest struct {
	// Version is the manifest schema version. Kept at 1: the per-entry format
	// is self-describing (see EntryState), so no file-level bump is needed to
	// distinguish legacy single-version entries from multi-version ones.
	Version int                   `json:"version"`
	Entries map[string]EntryState `json:"entries"`
}

Manifest is the sidecar tracking which entries are indexed and, per entry, the set of stored versions (each with its content hash, embedder fingerprint, chunk IDs, and index time). It lives next to the chromem-go DB at .sdd/index/manifest.json.

Multi-version is the point: the shared machine-global store is written from several checkouts/branches (d-cpt-6cq), and a changed entry (summary regeneration, a mechanical fix) adds a version instead of overwriting the old one, so two branches never flip-flop each other's rows. Versions are dropped only on request — `sdd index gc`, or a `--force` rebuild replacing the entry's versions under the current derivation rule (d-tac-c9c) — never by a read or an ordinary write.

Schema evolution: a v1 manifest recorded exactly one state per entry as a flat {hash, fingerprint, chunk_ids, indexed_at} object. That shape still loads — as an entry's sole version — with NO migration or re-embedding, and an entry still holding a single version serializes back in that same flat shape, so an untouched v1 manifest round-trips byte-for-byte. Only an entry that has accumulated more than one version serializes as {"versions": [...]}. The manifest is a sidecar file, not public API, so this per-entry self-describing format is free to evolve.

func LoadManifest 

func LoadManifest(indexDir string) (*Manifest, error)

LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.

func (*Manifest) AddVersion added in v0.17.0 

func (m *Manifest) AddVersion(entryID string, v EntryVersion)

AddVersion records a version for an entry (monotonic accumulation). A version with the same Hash is replaced in place (idempotent re-embed of the same state); otherwise the version is appended, leaving prior versions intact — this is the no-delete write path. The force/rebuild path uses SetDerivationVersion instead.

func (*Manifest) DropGroups 

func (m *Manifest) DropGroups(selected []string, currentHashes map[string]string) (chunkIDs []string, versions int)

DropGroups removes every version in the selected groups and returns the chunk IDs whose rows the caller must delete, with the count of versions removed. It mutates the manifest in place — an entry left with no version is removed — and performs no I/O: the row delete and the manifest save are the write session's job.

func (*Manifest) EntryIDsSorted 

func (m *Manifest) EntryIDsSorted() []string

EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.

func (*Manifest) MismatchCount 

func (m *Manifest) MismatchCount(current string) int

MismatchCount returns the number of entries that have NO version recorded under the current embedder fingerprint — the entries a build/lazy-fill would re-embed on an embedder change. Empty current returns 0. Used by sdd lint.

func (*Manifest) PendingCount added in v0.9.0 

func (m *Manifest) PendingCount(entryIDs []string, fingerprint string) int

PendingCount returns how many of entryIDs are absent from the manifest or have no version under the current fingerprint — the entries a build or lazy-fill would embed at least once. It is presence- and fingerprint-based (not hash-based): it drives whether to show a transient progress view, and the current hash of every on-disk entry is not cheaply available here.

func (*Manifest) Save 

func (m *Manifest) Save(indexDir string) error

Save writes the manifest atomically (write to temp then rename) so a crash in the middle of indexing doesn't leave a partial JSON file. The atomic rename also gives an unlocked presence read a torn-read-safe file and changes the manifest's on-disk identity on every write (the generation fallback for legacy stores relies on this).

func (*Manifest) SelectGroups 

func (m *Manifest) SelectGroups(names []string, currentHashes map[string]string) (selected, missing []string, err error)

SelectGroups resolves `--drop` names against the groups present: GroupStale expands to every droppable group, a present droppable group selects itself, and a name this store does not hold is returned in missing so the caller can say so — cleanup is best effort across stores, not all-or-nothing. Only GroupCurrent is an error: dropping what the checkout searches is never meant.

func (*Manifest) SetDerivationVersion 

func (m *Manifest) SetDerivationVersion(entryID string, v EntryVersion)

SetDerivationVersion replaces every version of v's derivation rule with v, keeping versions written under other rules. The force rebuild path uses it after deleting the replaced versions' rows (see EntryState.ChunkIDsOf).

func (*Manifest) VersionGroups 

func (m *Manifest) VersionGroups(currentHashes map[string]string) []VersionGroup

VersionGroups groups every stored version for the report. currentHashes maps entry ID to the checkout's current state hash; an entry absent from it has no current version. Groups come current first, then branch, then the other rules by name.

func (*Manifest) VersionHashForChunk added in v0.17.0 

func (m *Manifest) VersionHashForChunk(entryID, chunkID string) string

VersionHashForChunk returns the state hash of the version that owns chunkID, or "" when no version does. It resolves a hit whose row carries no entry_hash metadata (a legacy v1 row): the legacy version's ChunkIDs hold the unversioned chunk ID, so this recovers that version's recorded hash.

type ManifestCache 

type ManifestCache struct {
	// contains filtered or unexported fields
}

ManifestCache reads one immutable publication file per identity change. Callers serialize access. Atomic rename makes file identity the freshness token even when a writer dies before updating the index generation marker.

func (*ManifestCache) Loads 

func (c *ManifestCache) Loads() int

func (*ManifestCache) Read 

func (c *ManifestCache) Read(dir string) (_ *Manifest, err error)

type Row 

type Row struct {
	EntryID string
	// EntryHash is the entry-state hash of the version this row belongs to.
	// Persisted so a read can decide the hit is fresh (its version equals the
	// current entry state) without re-deriving the chunk. Empty only for
	// legacy v1 rows, whose version is recovered from the manifest.
	EntryHash            string
	ChunkID              string
	Text                 string // embedded text (with Entry/Breadcrumb preamble)
	Body                 string // citation snippet source (without preamble)
	Breadcrumb           []string
	Depth                int
	IsSummary            bool
	IsAttachment         bool
	SourceAttachmentPath string
	ContentHash          string
	ModelFingerprint     string
	Embedding            []float32
}

Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.

type SnapshotCache added in v0.17.0 

type SnapshotCache struct {
	// contains filtered or unexported fields
}

SnapshotCache holds a read snapshot and the store generation it was loaded at, so a long-running reader answers from memory when the store is unchanged. A caller shares one per store directory and guards it with its own mutex — SnapshotCache carries no locking of its own.

type VersionGroup 

type VersionGroup struct {
	Name     string
	Versions int
	Entries  int
	Oldest   time.Time
	Newest   time.Time
	ChunkIDs []string
	// Droppable is false only for GroupCurrent.
	Droppable bool
}

VersionGroup is one group of stored versions in the `sdd index gc` report.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.