Documentation
¶
Overview ¶
Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.
Storage layout (under .sdd/index/):
chromem/ # chromem-go gob persistence manifest.json # entry tracking sidecar (see manifest.go)
Machine-global store resolution and the locking seam. The vector index lives outside every working tree in one content-addressed store per (repo-key, embedder-fingerprint) under the cache root — shared by a repo's checkout, its worktrees, and its connected-repo cache, so the same content is embedded once per machine (d-cpt-6cq / d-tac-nhx). All index locking is centralized here: no other package takes index locks.
Index ¶
- Constants
- func AttachmentChunkID(entryID, attachmentPath string, n int) string
- func BodyChunkID(entryID string, n int) string
- func ChunkIDPrefix(entryID string) string
- func HashContent(text string) string
- func ManifestFingerprint(m *Manifest) string
- func MigrateDir(legacyDir, cacheRoot, repoKey string) (target string, migrated bool, err error)
- func ReadStore(ctx context.Context, indexDir string, fn func(*Index) error) error
- func RepoKey(repoID, repoRoot string) string
- func StoreDir(cacheRoot, repoKey, fingerprint string) string
- func SummaryChunkID(entryID string) string
- func WriteStore(ctx context.Context, indexDir string, fn func(*Index) error) error
- type EntryState
- type Hit
- type Index
- func (i *Index) Count() int
- func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error
- func (i *Index) Path() string
- func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)
- func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
- type Manifest
- type Row
Constants ¶
const ( MetaEntryID = "entry_id" MetaChunkPath = "chunk_path" MetaDepth = "depth" MetaContentHash = "content_hash" MetaModelFingerprint = "model_fingerprint" MetaIsSummary = "is_summary" MetaIsAttachment = "is_attachment" MetaSourceAttachmentPath = "source_attachment_path" )
MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.
const CollectionName = "sdd-graph"
CollectionName is the chromem-go collection used for SDD chunks.
Variables ¶
This section is empty.
Functions ¶
func AttachmentChunkID ¶
AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path.
func BodyChunkID ¶
BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order.
func ChunkIDPrefix ¶
ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).
func HashContent ¶
HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.
func ManifestFingerprint ¶ added in v0.15.0
ManifestFingerprint returns the dominant embedder fingerprint recorded in a manifest — the fingerprint a legacy in-tree index was built under, used to pick its target store during migration. Empty when the manifest holds no fingerprints.
func MigrateDir ¶ added in v0.15.0
MigrateDir moves a legacy index directory (an in-tree .sdd/index or a clone-cache .index) into the machine-global store, keyed by the fingerprint its own manifest records. Move-if-absent: when the target store already exists the legacy directory is left untouched and skipped — never clobbered, never merged. Returns the target dir and what happened.
func ReadStore ¶ added in v0.16.2
ReadStore runs fn against a freshly loaded store while holding the store's shared lock — the read counterpart to WriteStore. A reader loads its snapshot under the lock so it never decodes half-written chromem documents, and it opens fresh per call so it reflects writes committed by the CLI or another process. An empty indexDir runs fn against a fresh in-memory store without locking (tests).
func RepoKey ¶ added in v0.15.0
RepoKey returns the identity a repository's index store is keyed by: the declared repo_id when the repo has one, else a hash of the absolute repo root under the "local" namespace. A moved identity-less repo therefore re-embeds — accepted: `sdd init` migration covers the common case, and minting a synthetic ID would invent a second identity concept.
func StoreDir ¶ added in v0.15.0
StoreDir resolves the machine-global store directory for one (repo-key, embedder-fingerprint) pair: <cacheRoot>/index/<repo-key>/<fp-hash>. The fingerprint is hashed because it is a free-form embedder string, not a path; keying by fingerprint means a changed embedder starts a fresh store instead of drifting inside an existing one.
func SummaryChunkID ¶
SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.
func WriteStore ¶ added in v0.16.2
WriteStore runs fn against a freshly loaded store while holding the store's exclusive lock — the one write boundary for every index mutation (build, lazy-fill, MCP reconcile). The lock is acquired BEFORE the snapshot is loaded and held through fn, so a writer never operates on a snapshot that a concurrent process has moved on from. Manifest reads and saves belonging to the write must happen inside fn so concurrent writers cannot clobber each other's manifest state. An empty indexDir runs fn against a fresh in-memory store without locking (tests). Blocks until the lock is available or ctx ends.
Types ¶
type EntryState ¶
type EntryState struct {
// Hash is the content hash of the source material the index was
// built from (entry body + each attachment). Changes invalidate the
// entry's chunks and trigger re-indexing on lazy-fill.
Hash string `json:"hash"`
// Fingerprint is the embedder fingerprint that produced this entry's
// chunks. Compared against the configured embedder at search-time
// (lazy re-embed) and lint-time (drift count).
Fingerprint string `json:"fingerprint"`
// ChunkIDs are the IDs this entry produced last time it was indexed.
// Used to delete stale rows before re-adding under new content/
// fingerprint.
ChunkIDs []string `json:"chunk_ids"`
// IndexedAt is the time the entry was last indexed. Informational
// only — not used for any lifecycle decision.
IndexedAt time.Time `json:"indexed_at"`
}
EntryState tracks indexing status for a single entry.
type Hit ¶
type Hit struct {
EntryID string
ChunkID string
Score float32
Text string
Body string
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
}
Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index wraps a chromem-go persistent DB and a single collection.
func Open ¶
Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).
On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.
The load phase holds the store's shared lock so a reader never decodes half-written documents from a concurrent write session; the lock is released before Open returns — queries run on the in-memory copy. Open is the read-side entry point (the CLI finders); mutation goes through WriteStore, which acquires the exclusive lock before loading its snapshot.
func OpenInMemory ¶
func OpenInMemory() *Index
OpenInMemory returns a non-persistent index. Used by tests.
func (*Index) DeleteEntry ¶
DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.
func (*Index) Query ¶
Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.
func (*Index) UpsertEntry ¶
func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.
Pre-conditions:
- Each row's Embedding is populated (the index does not embed).
- row.EntryID == entryID (validated; mismatch is a programmer error).
- row.ChunkID is unique within rows.
type Manifest ¶
type Manifest struct {
// Version is bumped when the manifest schema changes. Currently 1.
Version int `json:"version"`
Entries map[string]EntryState `json:"entries"`
}
Manifest is the sidecar tracking which entries are indexed, their content hash (for lazy-fill staleness detection), the fingerprint of the embedder used (for drift counting), and the chunk IDs each entry produced (so re-indexing can delete the old set before adding the new).
Lives next to the chromem-go DB at .sdd/index/manifest.json.
func LoadManifest ¶
LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.
func (*Manifest) EntryIDsSorted ¶
EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.
func (*Manifest) MismatchCount ¶
MismatchCount returns the number of entries whose recorded fingerprint differs from current. Empty fingerprint counts as a mismatch (the entry has never been embedded under the active embedder). Used by sdd lint.
func (*Manifest) PendingCount ¶ added in v0.9.0
PendingCount returns how many of entryIDs are absent from the manifest or recorded under a different embedder fingerprint — the entries a build or lazy-fill would (re-)embed. Entry bodies are immutable, so a stored content hash never goes stale on its own; presence and fingerprint are the reconciliation axes that decide whether there is work worth showing a transient progress view for.
type Row ¶
type Row struct {
EntryID string
ChunkID string
Text string // embedded text (with Entry/Breadcrumb preamble)
Body string // citation snippet source (without preamble)
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
Embedding []float32
}
Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.
Source Files