Documentation
¶
Overview ¶
Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.
Storage layout (under .sdd/index/):
chromem/ # chromem-go gob persistence manifest.json # entry tracking sidecar (see manifest.go)
Machine-global store resolution and the locking seam. The vector index lives outside every working tree in one content-addressed store per (repo-key, embedder-fingerprint) under the cache root — shared by a repo's checkout, its worktrees, and its connected-repo cache, so the same content is embedded once per machine (d-cpt-6cq / d-tac-nhx). All index locking is centralized here: no other package takes index locks.
Index ¶
- Constants
- func AttachmentChunkID(entryID, attachmentPath string, n int) string
- func BodyChunkID(entryID string, n int) string
- func ChunkIDPrefix(entryID string) string
- func HashContent(text string) string
- func ManifestFingerprint(m *Manifest) string
- func MigrateDir(legacyDir, cacheRoot, repoKey string) (target string, migrated bool, err error)
- func RepoKey(repoID, repoRoot string) string
- func StoreDir(cacheRoot, repoKey, fingerprint string) string
- func SummaryChunkID(entryID string) string
- type EntryState
- type Hit
- type Index
- func (i *Index) Count() int
- func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error
- func (i *Index) Path() string
- func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)
- func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
- func (i *Index) WriteSession(ctx context.Context, fn func() error) error
- type Manifest
- type Row
Constants ¶
const ( MetaEntryID = "entry_id" MetaChunkPath = "chunk_path" MetaDepth = "depth" MetaContentHash = "content_hash" MetaModelFingerprint = "model_fingerprint" MetaIsSummary = "is_summary" MetaIsAttachment = "is_attachment" MetaSourceAttachmentPath = "source_attachment_path" )
MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.
const CollectionName = "sdd-graph"
CollectionName is the chromem-go collection used for SDD chunks.
Variables ¶
This section is empty.
Functions ¶
func AttachmentChunkID ¶
AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path.
func BodyChunkID ¶
BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order.
func ChunkIDPrefix ¶
ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).
func HashContent ¶
HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.
func ManifestFingerprint ¶ added in v0.15.0
ManifestFingerprint returns the dominant embedder fingerprint recorded in a manifest — the fingerprint a legacy in-tree index was built under, used to pick its target store during migration. Empty when the manifest holds no fingerprints.
func MigrateDir ¶ added in v0.15.0
MigrateDir moves a legacy index directory (an in-tree .sdd/index or a clone-cache .index) into the machine-global store, keyed by the fingerprint its own manifest records. Move-if-absent: when the target store already exists the legacy directory is left untouched and skipped — never clobbered, never merged. Returns the target dir and what happened.
func RepoKey ¶ added in v0.15.0
RepoKey returns the identity a repository's index store is keyed by: the declared repo_id when the repo has one, else a hash of the absolute repo root under the "local" namespace. A moved identity-less repo therefore re-embeds — accepted: `sdd init` migration covers the common case, and minting a synthetic ID would invent a second identity concept.
func StoreDir ¶ added in v0.15.0
StoreDir resolves the machine-global store directory for one (repo-key, embedder-fingerprint) pair: <cacheRoot>/index/<repo-key>/<fp-hash>. The fingerprint is hashed because it is a free-form embedder string, not a path; keying by fingerprint means a changed embedder starts a fresh store instead of drifting inside an existing one.
func SummaryChunkID ¶
SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.
Types ¶
type EntryState ¶
type EntryState struct {
// Hash is the content hash of the source material the index was
// built from (entry body + each attachment). Changes invalidate the
// entry's chunks and trigger re-indexing on lazy-fill.
Hash string `json:"hash"`
// Fingerprint is the embedder fingerprint that produced this entry's
// chunks. Compared against the configured embedder at search-time
// (lazy re-embed) and lint-time (drift count).
Fingerprint string `json:"fingerprint"`
// ChunkIDs are the IDs this entry produced last time it was indexed.
// Used to delete stale rows before re-adding under new content/
// fingerprint.
ChunkIDs []string `json:"chunk_ids"`
// IndexedAt is the time the entry was last indexed. Informational
// only — not used for any lifecycle decision.
IndexedAt time.Time `json:"indexed_at"`
}
EntryState tracks indexing status for a single entry.
type Hit ¶
type Hit struct {
EntryID string
ChunkID string
Score float32
Text string
Body string
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
}
Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index wraps a chromem-go persistent DB and a single collection.
func Open ¶
Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).
On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.
The load phase holds the store's shared lock so a reader never decodes half-written documents from a concurrent write session; the lock is released before Open returns — queries run on the in-memory copy, and mutation goes through WriteSession's exclusive lock.
func OpenInMemory ¶
func OpenInMemory() *Index
OpenInMemory returns a non-persistent index. Used by tests.
func (*Index) DeleteEntry ¶
DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.
func (*Index) Query ¶
Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.
func (*Index) UpsertEntry ¶
func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.
Pre-conditions:
- Each row's Embedding is populated (the index does not embed).
- row.EntryID == entryID (validated; mismatch is a programmer error).
- row.ChunkID is unique within rows.
func (*Index) WriteSession ¶ added in v0.15.0
WriteSession runs fn while holding the store's exclusive lock — the one write boundary for every index mutation (build, lazy-fill). Manifest reads belonging to the write must happen inside fn so concurrent writers cannot clobber each other's manifest state. In-memory indexes (tests) run fn without locking. Blocks until the lock is available or ctx ends.
type Manifest ¶
type Manifest struct {
// Version is bumped when the manifest schema changes. Currently 1.
Version int `json:"version"`
Entries map[string]EntryState `json:"entries"`
}
Manifest is the sidecar tracking which entries are indexed, their content hash (for lazy-fill staleness detection), the fingerprint of the embedder used (for drift counting), and the chunk IDs each entry produced (so re-indexing can delete the old set before adding the new).
Lives next to the chromem-go DB at .sdd/index/manifest.json.
func LoadManifest ¶
LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.
func (*Manifest) EntryIDsSorted ¶
EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.
func (*Manifest) MismatchCount ¶
MismatchCount returns the number of entries whose recorded fingerprint differs from current. Empty fingerprint counts as a mismatch (the entry has never been embedded under the active embedder). Used by sdd lint.
func (*Manifest) PendingCount ¶ added in v0.9.0
PendingCount returns how many of entryIDs are absent from the manifest or recorded under a different embedder fingerprint — the entries a build or lazy-fill would (re-)embed. Entry bodies are immutable, so a stored content hash never goes stale on its own; presence and fingerprint are the reconciliation axes that decide whether there is work worth showing a transient progress view for.
type Row ¶
type Row struct {
EntryID string
ChunkID string
Text string // embedded text (with Entry/Breadcrumb preamble)
Body string // citation snippet source (without preamble)
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
Embedding []float32
}
Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.
Source Files