index

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.

Storage layout (under .sdd/index/): 

chromem/        # chromem-go gob persistence
manifest.json   # entry tracking sidecar (see manifest.go)

Machine-global store resolution and the locking seam. The vector index lives outside every working tree in one content-addressed store per (repo-key, embedder-fingerprint) under the cache root — shared by a repo's checkout, its worktrees, and its connected-repo cache, so the same content is embedded once per machine (d-cpt-6cq / d-tac-nhx). All index locking is centralized here: no other package takes index locks.

Index

Constants

View Source 
const (
	MetaEntryID              = "entry_id"
	MetaEntryHash            = "entry_hash"
	MetaChunkPath            = "chunk_path"
	MetaDepth                = "depth"
	MetaContentHash          = "content_hash"
	MetaModelFingerprint     = "model_fingerprint"
	MetaIsSummary            = "is_summary"
	MetaIsAttachment         = "is_attachment"
	MetaSourceAttachmentPath = "source_attachment_path"
)

MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.

View Source 
const CollectionName = "sdd-graph"

CollectionName is the chromem-go collection used for SDD chunks.

View Source 
const VersionRetention = 14 * 24 * time.Hour

VersionRetention is how long a stored entry version that is no longer any current version survives before version GC may drop it. The window protects recent branch work: a version indexed within it is kept even when the collecting writer's graph no longer holds it, so switching back to that branch shortly after does not re-embed. 14 days balances bounded store growth against the cost of re-embedding a branch revisited after a break.

It is a fixed constant, not configuration: no existing config surface fits a vector-store retention knob, and inventing one for a self-healing derived-data cleanup is not worth the maintenance surface. The re-embed after collection is bounded and self-healing (vectors are derived data), so the exact value is not load-bearing.

Variables

This section is empty.

Functions

func AttachmentChunkID 

func AttachmentChunkID(entryID, attachmentPath string, n int) string

AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path. Legacy (unversioned) form — see SummaryChunkID.

func AttachmentChunkIDVersioned added in v0.17.0 

func AttachmentChunkIDVersioned(entryID, entryHash, attachmentPath string, n int) string

AttachmentChunkIDVersioned is the version-qualified n-th attachment chunk ID: entryID#v-<hash8>#attach-<p6>-N.

func BodyChunkID 

func BodyChunkID(entryID string, n int) string

BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order. Legacy (unversioned) form — see SummaryChunkID.

func BodyChunkIDVersioned added in v0.17.0 

func BodyChunkIDVersioned(entryID, entryHash string, n int) string

BodyChunkIDVersioned is the version-qualified n-th body chunk ID: entryID#v-<hash8>#body-N.

func ChunkIDPrefix 

func ChunkIDPrefix(entryID string) string

ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).

func HashContent 

func HashContent(text string) string

HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.

func ManifestFingerprint added in v0.15.0 

func ManifestFingerprint(m *Manifest) string

ManifestFingerprint returns the dominant embedder fingerprint recorded in a manifest — the fingerprint a legacy in-tree index was built under, used to pick its target store during migration. Empty when the manifest holds no fingerprints.

func MigrateDir added in v0.15.0 

func MigrateDir(legacyDir, cacheRoot, repoKey string) (target string, migrated bool, err error)

MigrateDir moves a legacy index directory (an in-tree .sdd/index or a clone-cache .index) into the machine-global store, keyed by the fingerprint its own manifest records. Move-if-absent: when the target store already exists the legacy directory is left untouched and skipped — never clobbered, never merged. Returns the target dir and what happened.

func ReadCached added in v0.17.0 

func ReadCached(ctx context.Context, indexDir string, cache *SnapshotCache, fn func(*Index) error) (reloaded bool, err error)

ReadCached runs fn against a store snapshot under the shared lock, reusing cache when the on-disk generation is unchanged and loading a fresh snapshot otherwise. The generation is read AFTER acquiring the lock and the lock is held through fn, so a writer cannot slip between the check and the query and go unseen. It reports whether it loaded a fresh snapshot (for reload counting). cache must be non-nil; the caller guards it against concurrent use.

func ReadStore added in v0.16.2 

func ReadStore(ctx context.Context, indexDir string, fn func(*Index) error) error

ReadStore runs fn against a freshly loaded store while holding the store's shared lock — the read counterpart to WriteStore. A reader loads its snapshot under the lock so it never decodes half-written chromem documents, and it opens fresh per call so it reflects writes committed by the CLI or another process. An empty indexDir runs fn against a fresh in-memory store without locking (tests).

func RepoKey added in v0.15.0 

func RepoKey(repoID, repoRoot string) string

RepoKey returns the identity a repository's index store is keyed by: the declared repo_id when the repo has one, else a hash of the absolute repo root under the "local" namespace. A moved identity-less repo therefore re-embeds — accepted: `sdd init` migration covers the common case, and minting a synthetic ID would invent a second identity concept.

func StoreDir added in v0.15.0 

func StoreDir(cacheRoot, repoKey, fingerprint string) string

StoreDir resolves the machine-global store directory for one (repo-key, embedder-fingerprint) pair: <cacheRoot>/index/<repo-key>/<fp-hash>. The fingerprint is hashed because it is a free-form embedder string, not a path; keying by fingerprint means a changed embedder starts a fresh store instead of drifting inside an existing one.

func SummaryChunkID 

func SummaryChunkID(entryID string) string

SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.

This is the LEGACY (unversioned) form: a v1 store owns exactly this ID per entry summary, interpreted as the entry's sole pre-existing version. New writes mint version-qualified IDs (SummaryChunkIDVersioned) so two versions of one entry can coexist in the shared store without colliding.

func SummaryChunkIDVersioned added in v0.17.0 

func SummaryChunkIDVersioned(entryID, entryHash string) string

SummaryChunkIDVersioned is the version-qualified summary chunk ID: entryID#v-<hash8>#summary. New writes mint versioned IDs so a changed entry adds a version rather than overwriting the old one — two branches holding different versions of one entry each own their own rows in the shared store.

func VersionSegment added in v0.17.0 

func VersionSegment(entryHash string) string

VersionSegment derives the short version tag embedded in a versioned chunk ID from an entry-state hash — the first 8 hex chars, enough to distinguish an entry's stored versions while keeping IDs bounded. It is an identity tag only: the FULL entry hash lives in the row's entry_hash metadata and in the manifest, and that is what read-time freshness compares against. A hash shorter than 8 chars (only in tests) is used whole.

func WriteStore added in v0.16.2 

func WriteStore(ctx context.Context, indexDir string, fn func(*Index) error) error

WriteStore runs fn against a freshly loaded store while holding the store's exclusive lock — the one write boundary for every index mutation (build, lazy-fill, MCP reconcile). The lock is acquired BEFORE the snapshot is loaded and held through fn, so a writer never operates on a snapshot that a concurrent process has moved on from. Manifest reads and saves belonging to the write must happen inside fn so concurrent writers cannot clobber each other's manifest state. An empty indexDir runs fn against a fresh in-memory store without locking (tests). Blocks until the lock is available or ctx ends.

Types

type EntryState 

type EntryState struct {
	Versions []EntryVersion
}

EntryState is one entry's set of stored versions. See Manifest for the on-disk shape (legacy flat object for a single version, {"versions": [...]} for many).

func (EntryState) AllChunkIDs added in v0.17.0 

func (s EntryState) AllChunkIDs() []string

AllChunkIDs returns every chunk ID across all of the entry's versions — what a force rebuild deletes before writing the single current version.

func (EntryState) HasVersion added in v0.17.0 

func (s EntryState) HasVersion(hash, fingerprint string) bool

HasVersion reports whether the entry has a stored version matching both the given state hash and embedder fingerprint — the presence test both the CLI indexer and the application reconcile use to decide whether to (re-)embed.

func (EntryState) MarshalJSON added in v0.17.0 

func (s EntryState) MarshalJSON() ([]byte, error)

MarshalJSON emits the legacy flat shape for a single version and the versions-list shape for many. An entry with no versions should never be persisted (GC drops the map key instead), but it serializes harmlessly as an empty versions list.

func (*EntryState) UnmarshalJSON added in v0.17.0 

func (s *EntryState) UnmarshalJSON(data []byte) error

UnmarshalJSON loads either shape: a {"versions": [...]} object as-is, or a legacy flat {hash, fingerprint, chunk_ids, indexed_at} object as the entry's sole version.

type EntryVersion added in v0.17.0 

type EntryVersion struct {
	// Hash is the entry-state hash (entry content + summary + attachment
	// bytes) this version was built from. Read-time freshness keeps a hit only
	// when its version's Hash equals the current entry's state hash.
	Hash string `json:"hash"`
	// Fingerprint is the embedder fingerprint that produced this version's
	// chunks. Within one store (keyed per fingerprint) this is constant, but
	// it is retained per version for lint drift reporting.
	Fingerprint string `json:"fingerprint"`
	// ChunkIDs are the IDs this version contributed. Used to resolve a hit's
	// version (legacy rows carry no entry_hash metadata) and to delete a
	// version's rows during GC or a force rebuild.
	ChunkIDs []string `json:"chunk_ids"`
	// IndexedAt is when this version was last written. The retention window in
	// version GC reads it; nothing else depends on it.
	IndexedAt time.Time `json:"indexed_at"`
}

EntryVersion is one stored version of an entry: the material it was built from (Hash), the embedder that produced it (Fingerprint), the chunk IDs it contributed (ChunkIDs), and when it was indexed (IndexedAt, which the retention side of GC reads).

type Hit 

type Hit struct {
	EntryID string
	// EntryHash is the version this hit belongs to (from row metadata). Empty
	// for a legacy v1 row — the caller recovers the version through the
	// manifest (Manifest.VersionHashForChunk).
	EntryHash            string
	ChunkID              string
	Score                float32
	Text                 string
	Body                 string
	Breadcrumb           []string
	Depth                int
	IsSummary            bool
	IsAttachment         bool
	SourceAttachmentPath string
	ContentHash          string
	ModelFingerprint     string
}

Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index wraps a chromem-go persistent DB and a single collection.

func Open 

func Open(indexDir string) (*Index, error)

Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).

On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.

The load phase holds the store's shared lock so a reader never decodes half-written documents from a concurrent write session; the lock is released before Open returns — queries run on the in-memory copy. Open is the read-side entry point (the CLI finders); mutation goes through WriteStore, which acquires the exclusive lock before loading its snapshot.

func OpenInMemory 

func OpenInMemory() *Index

OpenInMemory returns a non-persistent index. Used by tests.

func (*Index) Count 

func (i *Index) Count() int

Count returns the total number of chunk rows in the index.

func (*Index) DeleteEntry 

func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error

DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.

func (*Index) Path 

func (i *Index) Path() string

Path returns the on-disk root of the index. Empty for in-memory indexes.

func (*Index) Query 

func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)

Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.

func (*Index) UpsertEntry 

func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error

UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.

Pre-conditions:

  • Each row's Embedding is populated (the index does not embed).
  • row.EntryID == entryID (validated; mismatch is a programmer error).
  • row.ChunkID is unique within rows.

type Manifest 

type Manifest struct {
	// Version is the manifest schema version. Kept at 1: the per-entry format
	// is self-describing (see EntryState), so no file-level bump is needed to
	// distinguish legacy single-version entries from multi-version ones.
	Version int                   `json:"version"`
	Entries map[string]EntryState `json:"entries"`
}

Manifest is the sidecar tracking which entries are indexed and, per entry, the set of stored versions (each with its content hash, embedder fingerprint, chunk IDs, and index time). It lives next to the chromem-go DB at .sdd/index/manifest.json.

Multi-version is the point: the shared machine-global store is written from several checkouts/branches (d-cpt-6cq), and a changed entry (summary regeneration, a mechanical fix) adds a version instead of overwriting the old one, so two branches never flip-flop each other's rows. A stale version is dropped only by write-session GC or an explicit `--force` rebuild — never by a read.

Schema evolution: a v1 manifest recorded exactly one state per entry as a flat {hash, fingerprint, chunk_ids, indexed_at} object. That shape still loads — as an entry's sole version — with NO migration or re-embedding, and an entry still holding a single version serializes back in that same flat shape, so an untouched v1 manifest round-trips byte-for-byte. Only an entry that has accumulated more than one version serializes as {"versions": [...]}. The manifest is a sidecar file, not public API, so this per-entry self-describing format is free to evolve.

func LoadManifest 

func LoadManifest(indexDir string) (*Manifest, error)

LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.

func (*Manifest) AddVersion added in v0.17.0 

func (m *Manifest) AddVersion(entryID string, v EntryVersion)

AddVersion records a version for an entry (monotonic accumulation). A version with the same Hash is replaced in place (idempotent re-embed of the same state); otherwise the version is appended, leaving prior versions intact — this is the no-delete lazy write path. The force/rebuild path uses SetSingleVersion instead.

func (*Manifest) CollectStaleVersions added in v0.17.0 

func (m *Manifest) CollectStaleVersions(currentHashes map[string]string, now time.Time, retention time.Duration) []string

CollectStaleVersions drops every stored version that is neither a current version (its entry's hash appears in currentHashes) nor indexed within the retention window, and returns the chunk IDs whose rows the caller must delete from the index. It mutates the manifest in place — an entry left with no surviving version is removed entirely — but performs no I/O: the delete and the manifest save are the write session's job (the sole sanctioned delete paths are this GC under the write lock and an explicit force rebuild).

currentHashes maps entry ID to the collecting writer's current state hash. An entry absent from it (removed from the writer's graph, or one whose hash could not be computed) has no current version, so only the retention window protects its versions.

func (*Manifest) EntryIDsSorted 

func (m *Manifest) EntryIDsSorted() []string

EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.

func (*Manifest) MismatchCount 

func (m *Manifest) MismatchCount(current string) int

MismatchCount returns the number of entries that have NO version recorded under the current embedder fingerprint — the entries a build/lazy-fill would re-embed on an embedder change. Empty current returns 0. Used by sdd lint.

func (*Manifest) PendingCount added in v0.9.0 

func (m *Manifest) PendingCount(entryIDs []string, fingerprint string) int

PendingCount returns how many of entryIDs are absent from the manifest or have no version under the current fingerprint — the entries a build or lazy-fill would embed at least once. It is presence- and fingerprint-based (not hash-based): it drives whether to show a transient progress view, and the current hash of every on-disk entry is not cheaply available here.

func (*Manifest) Save 

func (m *Manifest) Save(indexDir string) error

Save writes the manifest atomically (write to temp then rename) so a crash in the middle of indexing doesn't leave a partial JSON file. The atomic rename also gives an unlocked presence read a torn-read-safe file and changes the manifest's on-disk identity on every write (the generation fallback for legacy stores relies on this).

func (*Manifest) SetSingleVersion added in v0.17.0 

func (m *Manifest) SetSingleVersion(entryID string, v EntryVersion)

SetSingleVersion collapses an entry to exactly the given version, discarding any others. Used by the force rebuild path, whose caller has already deleted the entry's old chunk rows from the index.

func (*Manifest) VersionHashForChunk added in v0.17.0 

func (m *Manifest) VersionHashForChunk(entryID, chunkID string) string

VersionHashForChunk returns the state hash of the version that owns chunkID, or "" when no version does. It resolves a hit whose row carries no entry_hash metadata (a legacy v1 row): the legacy version's ChunkIDs hold the unversioned chunk ID, so this recovers that version's recorded hash.

type Row 

type Row struct {
	EntryID string
	// EntryHash is the entry-state hash of the version this row belongs to.
	// Persisted so a read can decide the hit is fresh (its version equals the
	// current entry state) without re-deriving the chunk. Empty only for
	// legacy v1 rows, whose version is recovered from the manifest.
	EntryHash            string
	ChunkID              string
	Text                 string // embedded text (with Entry/Breadcrumb preamble)
	Body                 string // citation snippet source (without preamble)
	Breadcrumb           []string
	Depth                int
	IsSummary            bool
	IsAttachment         bool
	SourceAttachmentPath string
	ContentHash          string
	ModelFingerprint     string
	Embedding            []float32
}

Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.

type SnapshotCache added in v0.17.0 

type SnapshotCache struct {
	// contains filtered or unexported fields
}

SnapshotCache holds a read snapshot and the store generation it was loaded at, so a long-running reader answers from memory when the store is unchanged. A caller shares one per store directory and guards it with its own mutex — SnapshotCache carries no locking of its own.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.