Documentation
¶
Overview ¶
Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.
Storage layout (under .sdd/index/):
chromem/ # chromem-go gob persistence manifest.json # entry tracking sidecar (see manifest.go)
Machine-global store resolution and the locking seam. The vector index lives outside every working tree in one content-addressed store per (repo-key, embedder-fingerprint) under the cache root — shared by a repo's checkout, its worktrees, and its connected-repo cache, so the same content is embedded once per machine (d-cpt-6cq / d-tac-nhx). All index locking is centralized here: no other package takes index locks.
Index ¶
- Constants
- func AttachmentChunkID(entryID, attachmentPath string, n int) string
- func AttachmentChunkIDVersioned(entryID, entryHash, attachmentPath string, n int) string
- func BodyChunkID(entryID string, n int) string
- func BodyChunkIDVersioned(entryID, entryHash string, n int) string
- func ChunkIDPrefix(entryID string) string
- func HashContent(text string) string
- func ManifestFingerprint(m *Manifest) string
- func MigrateDir(legacyDir, cacheRoot, repoKey string) (target string, migrated bool, err error)
- func ReadCached(ctx context.Context, indexDir string, cache *SnapshotCache, ...) (reloaded bool, err error)
- func ReadStore(ctx context.Context, indexDir string, fn func(*Index) error) error
- func RepoKey(repoID, repoRoot string) string
- func StoreDir(cacheRoot, repoKey, fingerprint string) string
- func SummaryChunkID(entryID string) string
- func SummaryChunkIDVersioned(entryID, entryHash string) string
- func VersionSegment(entryHash string) string
- func WriteStore(ctx context.Context, indexDir string, fn func(*Index) error) error
- type EntryState
- type EntryVersion
- type Hit
- type Index
- func (i *Index) Count() int
- func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error
- func (i *Index) Path() string
- func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)
- func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
- type Manifest
- func (m *Manifest) AddVersion(entryID string, v EntryVersion)
- func (m *Manifest) CollectStaleVersions(currentHashes map[string]string, now time.Time, retention time.Duration) []string
- func (m *Manifest) EntryIDsSorted() []string
- func (m *Manifest) MismatchCount(current string) int
- func (m *Manifest) PendingCount(entryIDs []string, fingerprint string) int
- func (m *Manifest) Save(indexDir string) error
- func (m *Manifest) SetSingleVersion(entryID string, v EntryVersion)
- func (m *Manifest) VersionHashForChunk(entryID, chunkID string) string
- type Row
- type SnapshotCache
Constants ¶
const ( MetaEntryID = "entry_id" MetaEntryHash = "entry_hash" MetaChunkPath = "chunk_path" MetaDepth = "depth" MetaContentHash = "content_hash" MetaModelFingerprint = "model_fingerprint" MetaIsSummary = "is_summary" MetaIsAttachment = "is_attachment" MetaSourceAttachmentPath = "source_attachment_path" )
MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.
const CollectionName = "sdd-graph"
CollectionName is the chromem-go collection used for SDD chunks.
const VersionRetention = 14 * 24 * time.Hour
VersionRetention is how long a stored entry version that is no longer any current version survives before version GC may drop it. The window protects recent branch work: a version indexed within it is kept even when the collecting writer's graph no longer holds it, so switching back to that branch shortly after does not re-embed. 14 days balances bounded store growth against the cost of re-embedding a branch revisited after a break.
It is a fixed constant, not configuration: no existing config surface fits a vector-store retention knob, and inventing one for a self-healing derived-data cleanup is not worth the maintenance surface. The re-embed after collection is bounded and self-healing (vectors are derived data), so the exact value is not load-bearing.
Variables ¶
This section is empty.
Functions ¶
func AttachmentChunkID ¶
AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path. Legacy (unversioned) form — see SummaryChunkID.
func AttachmentChunkIDVersioned ¶ added in v0.17.0
AttachmentChunkIDVersioned is the version-qualified n-th attachment chunk ID: entryID#v-<hash8>#attach-<p6>-N.
func BodyChunkID ¶
BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order. Legacy (unversioned) form — see SummaryChunkID.
func BodyChunkIDVersioned ¶ added in v0.17.0
BodyChunkIDVersioned is the version-qualified n-th body chunk ID: entryID#v-<hash8>#body-N.
func ChunkIDPrefix ¶
ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).
func HashContent ¶
HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.
func ManifestFingerprint ¶ added in v0.15.0
ManifestFingerprint returns the dominant embedder fingerprint recorded in a manifest — the fingerprint a legacy in-tree index was built under, used to pick its target store during migration. Empty when the manifest holds no fingerprints.
func MigrateDir ¶ added in v0.15.0
MigrateDir moves a legacy index directory (an in-tree .sdd/index or a clone-cache .index) into the machine-global store, keyed by the fingerprint its own manifest records. Move-if-absent: when the target store already exists the legacy directory is left untouched and skipped — never clobbered, never merged. Returns the target dir and what happened.
func ReadCached ¶ added in v0.17.0
func ReadCached(ctx context.Context, indexDir string, cache *SnapshotCache, fn func(*Index) error) (reloaded bool, err error)
ReadCached runs fn against a store snapshot under the shared lock, reusing cache when the on-disk generation is unchanged and loading a fresh snapshot otherwise. The generation is read AFTER acquiring the lock and the lock is held through fn, so a writer cannot slip between the check and the query and go unseen. It reports whether it loaded a fresh snapshot (for reload counting). cache must be non-nil; the caller guards it against concurrent use.
func ReadStore ¶ added in v0.16.2
ReadStore runs fn against a freshly loaded store while holding the store's shared lock — the read counterpart to WriteStore. A reader loads its snapshot under the lock so it never decodes half-written chromem documents, and it opens fresh per call so it reflects writes committed by the CLI or another process. An empty indexDir runs fn against a fresh in-memory store without locking (tests).
func RepoKey ¶ added in v0.15.0
RepoKey returns the identity a repository's index store is keyed by: the declared repo_id when the repo has one, else a hash of the absolute repo root under the "local" namespace. A moved identity-less repo therefore re-embeds — accepted: `sdd init` migration covers the common case, and minting a synthetic ID would invent a second identity concept.
func StoreDir ¶ added in v0.15.0
StoreDir resolves the machine-global store directory for one (repo-key, embedder-fingerprint) pair: <cacheRoot>/index/<repo-key>/<fp-hash>. The fingerprint is hashed because it is a free-form embedder string, not a path; keying by fingerprint means a changed embedder starts a fresh store instead of drifting inside an existing one.
func SummaryChunkID ¶
SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.
This is the LEGACY (unversioned) form: a v1 store owns exactly this ID per entry summary, interpreted as the entry's sole pre-existing version. New writes mint version-qualified IDs (SummaryChunkIDVersioned) so two versions of one entry can coexist in the shared store without colliding.
func SummaryChunkIDVersioned ¶ added in v0.17.0
SummaryChunkIDVersioned is the version-qualified summary chunk ID: entryID#v-<hash8>#summary. New writes mint versioned IDs so a changed entry adds a version rather than overwriting the old one — two branches holding different versions of one entry each own their own rows in the shared store.
func VersionSegment ¶ added in v0.17.0
VersionSegment derives the short version tag embedded in a versioned chunk ID from an entry-state hash — the first 8 hex chars, enough to distinguish an entry's stored versions while keeping IDs bounded. It is an identity tag only: the FULL entry hash lives in the row's entry_hash metadata and in the manifest, and that is what read-time freshness compares against. A hash shorter than 8 chars (only in tests) is used whole.
func WriteStore ¶ added in v0.16.2
WriteStore runs fn against a freshly loaded store while holding the store's exclusive lock — the one write boundary for every index mutation (build, lazy-fill, MCP reconcile). The lock is acquired BEFORE the snapshot is loaded and held through fn, so a writer never operates on a snapshot that a concurrent process has moved on from. Manifest reads and saves belonging to the write must happen inside fn so concurrent writers cannot clobber each other's manifest state. An empty indexDir runs fn against a fresh in-memory store without locking (tests). Blocks until the lock is available or ctx ends.
Types ¶
type EntryState ¶
type EntryState struct {
Versions []EntryVersion
}
EntryState is one entry's set of stored versions. See Manifest for the on-disk shape (legacy flat object for a single version, {"versions": [...]} for many).
func (EntryState) AllChunkIDs ¶ added in v0.17.0
func (s EntryState) AllChunkIDs() []string
AllChunkIDs returns every chunk ID across all of the entry's versions — what a force rebuild deletes before writing the single current version.
func (EntryState) HasVersion ¶ added in v0.17.0
func (s EntryState) HasVersion(hash, fingerprint string) bool
HasVersion reports whether the entry has a stored version matching both the given state hash and embedder fingerprint — the presence test both the CLI indexer and the application reconcile use to decide whether to (re-)embed.
func (EntryState) MarshalJSON ¶ added in v0.17.0
func (s EntryState) MarshalJSON() ([]byte, error)
MarshalJSON emits the legacy flat shape for a single version and the versions-list shape for many. An entry with no versions should never be persisted (GC drops the map key instead), but it serializes harmlessly as an empty versions list.
func (*EntryState) UnmarshalJSON ¶ added in v0.17.0
func (s *EntryState) UnmarshalJSON(data []byte) error
UnmarshalJSON loads either shape: a {"versions": [...]} object as-is, or a legacy flat {hash, fingerprint, chunk_ids, indexed_at} object as the entry's sole version.
type EntryVersion ¶ added in v0.17.0
type EntryVersion struct {
// Hash is the entry-state hash (entry content + summary + attachment
// bytes) this version was built from. Read-time freshness keeps a hit only
// when its version's Hash equals the current entry's state hash.
Hash string `json:"hash"`
// Fingerprint is the embedder fingerprint that produced this version's
// chunks. Within one store (keyed per fingerprint) this is constant, but
// it is retained per version for lint drift reporting.
Fingerprint string `json:"fingerprint"`
// ChunkIDs are the IDs this version contributed. Used to resolve a hit's
// version (legacy rows carry no entry_hash metadata) and to delete a
// version's rows during GC or a force rebuild.
ChunkIDs []string `json:"chunk_ids"`
// IndexedAt is when this version was last written. The retention window in
// version GC reads it; nothing else depends on it.
IndexedAt time.Time `json:"indexed_at"`
}
EntryVersion is one stored version of an entry: the material it was built from (Hash), the embedder that produced it (Fingerprint), the chunk IDs it contributed (ChunkIDs), and when it was indexed (IndexedAt, which the retention side of GC reads).
type Hit ¶
type Hit struct {
EntryID string
// EntryHash is the version this hit belongs to (from row metadata). Empty
// for a legacy v1 row — the caller recovers the version through the
// manifest (Manifest.VersionHashForChunk).
EntryHash string
ChunkID string
Score float32
Text string
Body string
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
}
Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index wraps a chromem-go persistent DB and a single collection.
func Open ¶
Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).
On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.
The load phase holds the store's shared lock so a reader never decodes half-written documents from a concurrent write session; the lock is released before Open returns — queries run on the in-memory copy. Open is the read-side entry point (the CLI finders); mutation goes through WriteStore, which acquires the exclusive lock before loading its snapshot.
func OpenInMemory ¶
func OpenInMemory() *Index
OpenInMemory returns a non-persistent index. Used by tests.
func (*Index) DeleteEntry ¶
DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.
func (*Index) Query ¶
Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.
func (*Index) UpsertEntry ¶
func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.
Pre-conditions:
- Each row's Embedding is populated (the index does not embed).
- row.EntryID == entryID (validated; mismatch is a programmer error).
- row.ChunkID is unique within rows.
type Manifest ¶
type Manifest struct {
// Version is the manifest schema version. Kept at 1: the per-entry format
// is self-describing (see EntryState), so no file-level bump is needed to
// distinguish legacy single-version entries from multi-version ones.
Version int `json:"version"`
Entries map[string]EntryState `json:"entries"`
}
Manifest is the sidecar tracking which entries are indexed and, per entry, the set of stored versions (each with its content hash, embedder fingerprint, chunk IDs, and index time). It lives next to the chromem-go DB at .sdd/index/manifest.json.
Multi-version is the point: the shared machine-global store is written from several checkouts/branches (d-cpt-6cq), and a changed entry (summary regeneration, a mechanical fix) adds a version instead of overwriting the old one, so two branches never flip-flop each other's rows. A stale version is dropped only by write-session GC or an explicit `--force` rebuild — never by a read.
Schema evolution: a v1 manifest recorded exactly one state per entry as a flat {hash, fingerprint, chunk_ids, indexed_at} object. That shape still loads — as an entry's sole version — with NO migration or re-embedding, and an entry still holding a single version serializes back in that same flat shape, so an untouched v1 manifest round-trips byte-for-byte. Only an entry that has accumulated more than one version serializes as {"versions": [...]}. The manifest is a sidecar file, not public API, so this per-entry self-describing format is free to evolve.
func LoadManifest ¶
LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.
func (*Manifest) AddVersion ¶ added in v0.17.0
func (m *Manifest) AddVersion(entryID string, v EntryVersion)
AddVersion records a version for an entry (monotonic accumulation). A version with the same Hash is replaced in place (idempotent re-embed of the same state); otherwise the version is appended, leaving prior versions intact — this is the no-delete lazy write path. The force/rebuild path uses SetSingleVersion instead.
func (*Manifest) CollectStaleVersions ¶ added in v0.17.0
func (m *Manifest) CollectStaleVersions(currentHashes map[string]string, now time.Time, retention time.Duration) []string
CollectStaleVersions drops every stored version that is neither a current version (its entry's hash appears in currentHashes) nor indexed within the retention window, and returns the chunk IDs whose rows the caller must delete from the index. It mutates the manifest in place — an entry left with no surviving version is removed entirely — but performs no I/O: the delete and the manifest save are the write session's job (the sole sanctioned delete paths are this GC under the write lock and an explicit force rebuild).
currentHashes maps entry ID to the collecting writer's current state hash. An entry absent from it (removed from the writer's graph, or one whose hash could not be computed) has no current version, so only the retention window protects its versions.
func (*Manifest) EntryIDsSorted ¶
EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.
func (*Manifest) MismatchCount ¶
MismatchCount returns the number of entries that have NO version recorded under the current embedder fingerprint — the entries a build/lazy-fill would re-embed on an embedder change. Empty current returns 0. Used by sdd lint.
func (*Manifest) PendingCount ¶ added in v0.9.0
PendingCount returns how many of entryIDs are absent from the manifest or have no version under the current fingerprint — the entries a build or lazy-fill would embed at least once. It is presence- and fingerprint-based (not hash-based): it drives whether to show a transient progress view, and the current hash of every on-disk entry is not cheaply available here.
func (*Manifest) Save ¶
Save writes the manifest atomically (write to temp then rename) so a crash in the middle of indexing doesn't leave a partial JSON file. The atomic rename also gives an unlocked presence read a torn-read-safe file and changes the manifest's on-disk identity on every write (the generation fallback for legacy stores relies on this).
func (*Manifest) SetSingleVersion ¶ added in v0.17.0
func (m *Manifest) SetSingleVersion(entryID string, v EntryVersion)
SetSingleVersion collapses an entry to exactly the given version, discarding any others. Used by the force rebuild path, whose caller has already deleted the entry's old chunk rows from the index.
func (*Manifest) VersionHashForChunk ¶ added in v0.17.0
VersionHashForChunk returns the state hash of the version that owns chunkID, or "" when no version does. It resolves a hit whose row carries no entry_hash metadata (a legacy v1 row): the legacy version's ChunkIDs hold the unversioned chunk ID, so this recovers that version's recorded hash.
type Row ¶
type Row struct {
EntryID string
// EntryHash is the entry-state hash of the version this row belongs to.
// Persisted so a read can decide the hit is fresh (its version equals the
// current entry state) without re-deriving the chunk. Empty only for
// legacy v1 rows, whose version is recovered from the manifest.
EntryHash string
ChunkID string
Text string // embedded text (with Entry/Breadcrumb preamble)
Body string // citation snippet source (without preamble)
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
Embedding []float32
}
Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.
type SnapshotCache ¶ added in v0.17.0
type SnapshotCache struct {
// contains filtered or unexported fields
}
SnapshotCache holds a read snapshot and the store generation it was loaded at, so a long-running reader answers from memory when the store is unchanged. A caller shares one per store directory and guards it with its own mutex — SnapshotCache carries no locking of its own.
Source Files