Documentation
¶
Overview ¶
Package chunking is the single place a graph entry becomes index chunks. Both the CLI indexer (internal/handlers) and the application vector search derive chunks and entry-state hashes through here, so the shared persistent store holds identical content regardless of which path wrote it — the same chunk IDs, the same embedded text, the same entry hash.
The chunk IDs and content hash come from internal/index; the splitting from internal/textsplitter. This package owns only the composition: how an entry and its Markdown attachments fan out into the summary chunk, body chunks, and per-attachment chunk sets that the index stores.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EntryStateHash ¶
func EntryStateHash(ctx context.Context, entry *model.Entry, attachments AttachmentReader) (string, error)
EntryStateHash combines the entry's body, summary, and each attachment's bytes into a single sha-256 digest. Stored in the manifest so a later build can detect changes without re-chunking and re-embedding. This is the definition CanonicalChunk.EntryHash and the CLI manifest hash share.
func IncludeEntry ¶
IncludeEntry reports whether an entry belongs in a store given the embedded-exclusion rule. The local/base store includes embedded (binary-shipped) entries — finders treat static and on-disk entries identically (d-cpt-dtv). Connected-repo stores exclude them so base facts embed once per machine, not once per connected repo.
Types ¶
type AttachmentReader ¶
type AttachmentReader interface {
ReadAttachment(ctx context.Context, entry *model.Entry, relPath string) ([]byte, error)
}
AttachmentReader supplies the full bytes of one entry attachment named by its graph-relative path. The CLI reads from disk; the application pages the GraphStore. Both must yield identical bytes so derived chunks and the entry hash match across the two write paths that share one persistent store.
type Chunk ¶
type Chunk struct {
ChunkID string
Chunk textsplitter.Chunk
}
Chunk pairs a derived chunk with its deterministic, stable ID.
func DeriveChunks ¶
func DeriveChunks(ctx context.Context, entry *model.Entry, entryHash string, splitter *textsplitter.Splitter, attachments AttachmentReader) ([]Chunk, error)
DeriveChunks produces every chunk an entry contributes to the index: its summary chunk, body chunks, and one chunk-set per Markdown attachment. Non-markdown attachments are skipped silently — chunking arbitrary binary or non-markdown text is out of scope for v1.
entryHash is the entry's state hash (from EntryStateHash); new writes mint version-qualified chunk IDs (entryID#v-<hash8>#…) so a changed entry adds a version to the shared store rather than overwriting the old one. Both write paths (CLI indexer, application vector search) pass the same hash, so the derived IDs match across paths.
type DiskAttachmentReader ¶
type DiskAttachmentReader struct {
GraphDir string
}
DiskAttachmentReader reads attachments from a graph directory on disk. Used by the CLI indexer, whose entry.Attachments are graph-relative paths.
Source Files