Documentation
¶
Overview ¶
Package index wraps chromem-go's persistent vector store with the SDD chunk and citation model. It is the side-effecting backing store for the search index — no domain reasoning happens here. The IndexHandler (internal/handlers) orchestrates: load entry → split → embed → upsert via this package.
Storage layout (under .sdd/index/):
chromem/ # chromem-go gob persistence manifest.json # entry tracking sidecar (see manifest.go)
Index ¶
- Constants
- func AttachmentChunkID(entryID, attachmentPath string, n int) string
- func BodyChunkID(entryID string, n int) string
- func ChunkIDPrefix(entryID string) string
- func HashContent(text string) string
- func SummaryChunkID(entryID string) string
- type EntryState
- type Hit
- type Index
- func (i *Index) Count() int
- func (i *Index) DeleteEntry(ctx context.Context, chunkIDs []string) error
- func (i *Index) Path() string
- func (i *Index) Query(ctx context.Context, embedding []float32, nResults int) ([]Hit, error)
- func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
- type Manifest
- type Row
Constants ¶
const ( MetaEntryID = "entry_id" MetaChunkPath = "chunk_path" MetaDepth = "depth" MetaContentHash = "content_hash" MetaModelFingerprint = "model_fingerprint" MetaIsSummary = "is_summary" MetaIsAttachment = "is_attachment" MetaSourceAttachmentPath = "source_attachment_path" )
MetadataKey constants name the per-row chromem metadata fields. Kept as constants so the indexer, finder, and tests share one source of truth.
const CollectionName = "sdd-graph"
CollectionName is the chromem-go collection used for SDD chunks.
Variables ¶
This section is empty.
Functions ¶
func AttachmentChunkID ¶
AttachmentChunkID is the deterministic chunk ID for the n-th chunk of the named attachment under entryID. attachmentPath is the entry-relative attachment path.
func BodyChunkID ¶
BodyChunkID is the deterministic chunk ID for the n-th body chunk of an entry. n is positional in the chunker's emit order.
func ChunkIDPrefix ¶
ChunkIDPrefix returns the per-entry prefix all this entry's chunk IDs share — used by Index.DeleteEntry's whereDocument filter (chromem-go supports id-prefix matching at delete-time via the metadata).
func HashContent ¶
HashContent returns a stable hex sha-256 of the chunk's embedded text. Used as the per-row content_hash metadata so future builds can detect when a row's source content changed.
func SummaryChunkID ¶
SummaryChunkID is the deterministic chunk ID for an entry's summary chunk. Re-indexing the same entry produces the same ID — the indexer upserts via id rather than delete-and-add for the summary.
Types ¶
type EntryState ¶
type EntryState struct {
// Hash is the content hash of the source material the index was
// built from (entry body + each attachment). Changes invalidate the
// entry's chunks and trigger re-indexing on lazy-fill.
Hash string `json:"hash"`
// Fingerprint is the embedder fingerprint that produced this entry's
// chunks. Compared against the configured embedder at search-time
// (lazy re-embed) and lint-time (drift count).
Fingerprint string `json:"fingerprint"`
// ChunkIDs are the IDs this entry produced last time it was indexed.
// Used to delete stale rows before re-adding under new content/
// fingerprint.
ChunkIDs []string `json:"chunk_ids"`
// IndexedAt is the time the entry was last indexed. Informational
// only — not used for any lifecycle decision.
IndexedAt time.Time `json:"indexed_at"`
}
EntryState tracks indexing status for a single entry.
type Hit ¶
type Hit struct {
EntryID string
ChunkID string
Score float32
Text string
Body string
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
}
Hit is one query result with the metadata needed to render a citation and to decide whether to re-embed (for fingerprint drift).
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index wraps a chromem-go persistent DB and a single collection.
func Open ¶
Open opens or creates the persistent index under indexDir. The chromem store lives in indexDir/chromem; the manifest sidecar at indexDir/manifest.json (managed by callers via LoadManifest/Save).
On a fresh directory, both subdirs are created. The collection is named CollectionName and is auto-created on first open.
func OpenInMemory ¶
func OpenInMemory() *Index
OpenInMemory returns a non-persistent index. Used by tests.
func (*Index) DeleteEntry ¶
DeleteEntry removes all rows whose chunk IDs are listed. Called when an entry is removed from the graph or when reconciling the manifest.
func (*Index) Query ¶
Query returns the top-N matches for the given query embedding. The nResults parameter is clamped to the collection's current count to avoid chromem-go's "n_results larger than collection" behavior.
func (*Index) UpsertEntry ¶
func (i *Index) UpsertEntry(ctx context.Context, entryID string, oldChunkIDs []string, rows []Row) error
UpsertEntry replaces all rows for entryID with the given rows in a single transaction-shaped pass. Old chunk IDs (from the manifest) are deleted first; the new rows are added afterward. Caller is responsible for updating the manifest with the new chunk IDs.
Pre-conditions:
- Each row's Embedding is populated (the index does not embed).
- row.EntryID == entryID (validated; mismatch is a programmer error).
- row.ChunkID is unique within rows.
type Manifest ¶
type Manifest struct {
// Version is bumped when the manifest schema changes. Currently 1.
Version int `json:"version"`
Entries map[string]EntryState `json:"entries"`
}
Manifest is the sidecar tracking which entries are indexed, their content hash (for lazy-fill staleness detection), the fingerprint of the embedder used (for drift counting), and the chunk IDs each entry produced (so re-indexing can delete the old set before adding the new).
Lives next to the chromem-go DB at .sdd/index/manifest.json.
func LoadManifest ¶
LoadManifest reads .sdd/index/manifest.json or returns an empty manifest when the file does not exist.
func (*Manifest) EntryIDsSorted ¶
EntryIDsSorted returns the manifest's entry IDs in lexicographic order. Useful for deterministic iteration in tests and lint output.
func (*Manifest) MismatchCount ¶
MismatchCount returns the number of entries whose recorded fingerprint differs from current. Empty fingerprint counts as a mismatch (the entry has never been embedded under the active embedder). Used by sdd lint.
type Row ¶
type Row struct {
EntryID string
ChunkID string
Text string // embedded text (with Entry/Breadcrumb preamble)
Body string // citation snippet source (without preamble)
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
ContentHash string
ModelFingerprint string
Embedding []float32
}
Row is one chunk-as-it-lives-in-the-index. The Index ingests Rows the IndexHandler has already populated from the splitter + embedder. Embedding must be non-empty; the index does not call out to embedders.
Source Files