Documentation
¶
Overview ¶
Package chunking is the single place a graph entry becomes index chunks. Both the CLI indexer (internal/handlers) and the application vector search derive chunks and entry-state hashes through here, so the shared persistent store holds identical content regardless of which path wrote it — the same chunk IDs, the same embedded text, the same entry hash.
The chunk IDs and content hash come from internal/index; the splitting from internal/textsplitter. This package owns only the composition: how an entry and its Markdown attachments fan out into the summary chunk, body chunks, and per-attachment chunk sets that the index stores.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EntryStateHash ¶
func EntryStateHash(ctx context.Context, entry *model.Entry, attachments AttachmentReader) (string, error)
EntryStateHash combines the entry's body, summary, and each attachment's bytes into a single sha-256 digest. Stored in the manifest so a later build can detect changes without re-chunking and re-embedding. This is the definition CanonicalChunk.EntryHash and the CLI manifest hash share.
func IncludeEntry ¶
IncludeEntry reports whether an entry belongs in a store given the embedded-exclusion rule. The local/base store includes embedded (binary-shipped) entries — finders treat static and on-disk entries identically (d-cpt-dtv). Connected-repo stores exclude them so base facts embed once per machine, not once per connected repo.
Types ¶
type AttachmentReader ¶
type AttachmentReader interface {
ReadAttachment(ctx context.Context, entry *model.Entry, relPath string) ([]byte, error)
}
AttachmentReader supplies the full bytes of one entry attachment named by its graph-relative path. The CLI reads from disk; the application pages the GraphStore. Both must yield identical bytes so derived chunks and the entry hash match across the two write paths that share one persistent store.
type Chunk ¶
type Chunk struct {
ChunkID string
Chunk textsplitter.Chunk
}
Chunk pairs a derived chunk with its deterministic, stable ID.
func DeriveChunks ¶
func DeriveChunks(ctx context.Context, entry *model.Entry, splitter *textsplitter.Splitter, attachments AttachmentReader) ([]Chunk, error)
DeriveChunks produces every chunk an entry contributes to the index: its summary chunk, body chunks, and one chunk-set per Markdown attachment. Non-markdown attachments are skipped silently — chunking arbitrary binary or non-markdown text is out of scope for v1.
type DiskAttachmentReader ¶
type DiskAttachmentReader struct {
GraphDir string
}
DiskAttachmentReader reads attachments from a graph directory on disk. Used by the CLI indexer, whose entry.Attachments are graph-relative paths.
Source Files