Documentation
¶
Overview ¶
Package fs implements knowledge.DocumentRepo, knowledge.ChunkRepo and knowledge.LayerRepo on top of any workspace.Workspace.
Layout under <prefix>/<dataset>/:
<doc> raw source content (unchanged) <doc>.meta.json SourceDocument.Version + Metadata sidecar .chunks.json DerivedChunk[] for the dataset (atomic write) <doc>.abstract L0 layer text (Q7=A: human-readable) <doc>.abstract.vec L0 layer embedding (binary, see vec.go) <doc>.overview L1 layer text <doc>.overview.vec L1 layer embedding <doc>.layers.json per-doc layer DerivedSig sidecar .abstract.md dataset-level L0 text .abstract.md.vec dataset-level L0 embedding .overview.md dataset-level L1 text .overview.md.vec dataset-level L1 embedding .dataset_layers.json dataset-level layer DerivedSig sidecar
Atomic writes go through workspace.Rename(tmp, final); the workspace contract requires Rename to be POSIX-atomic when the medium supports it, so partial writes are never observable.
Deprecated: use github.com/GizClaw/flowcraft/memory/knowledge/backend/fs instead. This package will be removed in v0.5.0.
Index ¶
- Constants
- type FSChunkRepodeprecated
- func (r *FSChunkRepo) DeleteByDataset(ctx context.Context, datasetID string) error
- func (r *FSChunkRepo) DeleteByDoc(ctx context.Context, datasetID, docName string) error
- func (r *FSChunkRepo) GetDocSig(ctx context.Context, datasetID, docName string) (knowledge.DerivedSig, bool, error)
- func (r *FSChunkRepo) Load(ctx context.Context) error
- func (r *FSChunkRepo) Replace(ctx context.Context, datasetID, docName string, ...) error
- func (r *FSChunkRepo) Search(ctx context.Context, q knowledge.ChunkQuery) ([]knowledge.Candidate, error)
- func (r *FSChunkRepo) SearchDocs(ctx context.Context, q knowledge.ChunkQuery) ([]knowledge.Candidate, error)
- type FSDocumentRepo
- func (r *FSDocumentRepo) Delete(ctx context.Context, datasetID, name string) error
- func (r *FSDocumentRepo) Get(ctx context.Context, datasetID, name string) (*knowledge.SourceDocument, error)
- func (r *FSDocumentRepo) List(ctx context.Context, datasetID string) ([]knowledge.SourceDocument, error)
- func (r *FSDocumentRepo) ListDatasets(ctx context.Context) ([]string, error)
- func (r *FSDocumentRepo) Put(ctx context.Context, doc knowledge.SourceDocument) (*knowledge.SourceDocument, error)
- func (r *FSDocumentRepo) WithNow(now func() time.Time) *FSDocumentRepo
- type FSLayerRepodeprecated
- func (r *FSLayerRepo) DeleteByDataset(ctx context.Context, datasetID string) error
- func (r *FSLayerRepo) DeleteByDoc(ctx context.Context, datasetID, docName string) error
- func (r *FSLayerRepo) Get(ctx context.Context, datasetID, docName string, layer knowledge.Layer) (*knowledge.DerivedLayer, error)
- func (r *FSLayerRepo) Put(ctx context.Context, layer knowledge.DerivedLayer) error
- func (r *FSLayerRepo) Search(ctx context.Context, q knowledge.LayerQuery) ([]knowledge.Candidate, error)
Constants ¶
const DefaultPrefix = "knowledge"
DefaultPrefix is the workspace sub-tree owned by the knowledge backend.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FSChunkRepo
deprecated
type FSChunkRepo struct {
// contains filtered or unexported fields
}
FSChunkRepo persists DerivedChunks per dataset. Each dataset has one .chunks.json file and an in-memory inverted index built lazily on first access (or on Load).
Concurrency: every public method is safe for concurrent use; the repo's RWMutex protects the state map and each datasetState.
Deprecated: FSChunkRepo is a unit-test / demo-grade backend. Replace rewrites the entire dataset .chunks.json on every per-doc ingest and rebuilds both chunk-level and doc-level inverted indices from scratch, all under a single write lock — making per-doc ingest O(N) and total ingest O(N^2) in the dataset (see #134). For any non-trivial dataset use factory.NewRetrieval with an in-process sdk/retrieval/memory.Index or a production retrieval.Index from sdkx/retrieval/{sqlite,postgres,workspace}. Slated for removal in v0.5.0; see docs/migrations/v0.5.0.md.
func NewChunkRepo
deprecated
func NewChunkRepo(ws workspace.Workspace, prefix string, tok textsearch.Tokenizer) *FSChunkRepo
NewChunkRepo constructs an FSChunkRepo. Tokenizer is auto-detected from the first content seen when nil; explicit override wins.
Deprecated: see FSChunkRepo. Use factory.NewRetrieval with a sdk/retrieval/memory.Index (or any production retrieval.Index) instead. Slated for removal in v0.5.0.
func (*FSChunkRepo) DeleteByDataset ¶
func (r *FSChunkRepo) DeleteByDataset(ctx context.Context, datasetID string) error
DeleteByDataset removes every chunk for the dataset and the chunks.json file.
func (*FSChunkRepo) DeleteByDoc ¶
func (r *FSChunkRepo) DeleteByDoc(ctx context.Context, datasetID, docName string) error
DeleteByDoc removes every chunk for (datasetID, docName). Missing target documents are idempotent success and do not return NotFound.
Same locking discipline as Replace: read-mutate-persist-install runs under the write lock so concurrent Delete + Replace cannot lose docs by racing on a stale state snapshot.
func (*FSChunkRepo) GetDocSig ¶ added in v0.4.0
func (r *FSChunkRepo) GetDocSig(ctx context.Context, datasetID, docName string) (knowledge.DerivedSig, bool, error)
GetDocSig satisfies knowledge.ChunkSigReader for the filesystem backend. Missing datasets or documents return (DerivedSig{}, false, nil).
func (*FSChunkRepo) Load ¶
func (r *FSChunkRepo) Load(ctx context.Context) error
Load rehydrates state for every dataset under the prefix. Idempotent; safe to call on startup. Failures for individual datasets are collected and returned as a joined error.
func (*FSChunkRepo) Replace ¶
func (r *FSChunkRepo) Replace(ctx context.Context, datasetID, docName string, chunks []knowledge.DerivedChunk) error
Replace atomically swaps every chunk for (datasetID, docName). The in-memory index is rebuilt from the merged chunk set and the dataset chunks.json is written atomically.
The full read-merge-persist-install sequence runs under the write lock. The pre-fix layout dropped the lock between read-merge and install so persist (slow IO) could overlap other Replace calls; that races against any concurrent Replace because both goroutines see the SAME pre-mutation state, build conflicting merged slices, and whichever installs last silently drops the other's chunks. Observed on BEIR scifact: ingest_concurrency=8 lost ~60% of docs vs ingest=1, dragging chunk-level BM25 nDCG@10 from 0.180 to 0.071 (see eval/leaderboard.md follow-up). Correctness wins over the persist-out-of-lock micro-optimisation; callers that need higher throughput should batch upstream.
func (*FSChunkRepo) Search ¶
func (r *FSChunkRepo) Search(ctx context.Context, q knowledge.ChunkQuery) ([]knowledge.Candidate, error)
Search runs the requested mode against the in-memory index. Vector recall returns nothing when q.Vector is empty; callers are expected to feed an embedded query vector for ModeVector / ModeHybrid.
The function snapshots the per-dataset state under RLock, releases the lock, then scores outside the lock so long-running CPU work does not stall writers.
func (*FSChunkRepo) SearchDocs ¶ added in v0.3.10
func (r *FSChunkRepo) SearchDocs(ctx context.Context, q knowledge.ChunkQuery) ([]knowledge.Candidate, error)
SearchDocs runs a doc-level BM25 query against the dataset.
This is the doc-granularity counterpart of Search: instead of returning one Candidate per matching chunk (which forces callers like eval/beir to collapse chunks→docID themselves), SearchDocs scores every doc as a whole and returns at most one Candidate per matching docName. ChunkIndex is set to -1 and Layer is left empty in the resulting Hit, signalling "doc-level, no specific chunk".
Vector / Hybrid modes are not supported at the doc level yet (vectors are chunk-embedded by design); ModeVector/ModeHybrid currently fall back to BM25-only doc scoring with a soft error log. Tracked in #126.
type FSDocumentRepo ¶
type FSDocumentRepo struct {
// contains filtered or unexported fields
}
FSDocumentRepo persists SourceDocuments under <prefix>/<dataset>/.
Concurrency: Put against the same (datasetID, name) is serialised by a per-document mutex so the read-modify-write Version increment is deterministic. Reads do not acquire the per-document mutex.
func NewDocumentRepo ¶
func NewDocumentRepo(ws workspace.Workspace, prefix string) *FSDocumentRepo
NewDocumentRepo wires an FSDocumentRepo to ws under the given prefix ("knowledge" if empty).
func (*FSDocumentRepo) Delete ¶
func (r *FSDocumentRepo) Delete(ctx context.Context, datasetID, name string) error
Delete removes the raw document, its meta sidecar and its layer sidecars. Idempotent: missing files are ignored.
func (*FSDocumentRepo) Get ¶
func (r *FSDocumentRepo) Get(ctx context.Context, datasetID, name string) (*knowledge.SourceDocument, error)
Get returns the SourceDocument with Content losslessly preserved.
Returns errdefs.NotFound when the document does not exist.
func (*FSDocumentRepo) List ¶
func (r *FSDocumentRepo) List(ctx context.Context, datasetID string) ([]knowledge.SourceDocument, error)
List returns SourceDocuments for the dataset, sorted by Name. Content is loaded for each document; callers that only need names should use the lighter ListDatasets+manual List pattern in higher tiers.
func (*FSDocumentRepo) ListDatasets ¶
func (r *FSDocumentRepo) ListDatasets(ctx context.Context) ([]string, error)
ListDatasets enumerates dataset IDs by listing the prefix directory.
Workspace.List in some implementations (notably MemWorkspace) emits the queried directory itself as an entry; we filter that case out so the dataset list never contains the prefix name itself.
func (*FSDocumentRepo) Put ¶
func (r *FSDocumentRepo) Put(ctx context.Context, doc knowledge.SourceDocument) (*knowledge.SourceDocument, error)
Put atomically increments SourceDocument.Version, writes the raw content, updates the .meta.json sidecar, and returns the authoritative stored SourceDocument. Doc.Version / UpdatedAt supplied by the caller are ignored; the repo is the source of truth for both fields.
func (*FSDocumentRepo) WithNow ¶
func (r *FSDocumentRepo) WithNow(now func() time.Time) *FSDocumentRepo
WithNow overrides the time source (test injection point).
type FSLayerRepo
deprecated
type FSLayerRepo struct {
// contains filtered or unexported fields
}
FSLayerRepo persists DerivedLayers as human-readable sidecars (Q7=A).
Layout per document:
<doc>.abstract L0 layer text <doc>.abstract.vec L0 layer embedding (binary, see vec.go) <doc>.overview L1 layer text <doc>.overview.vec L1 layer embedding <doc>.layers.json DerivedSig metadata for both layers
Layout per dataset:
.abstract.md dataset L0 .abstract.md.vec dataset L0 embedding .overview.md dataset L1 .overview.md.vec dataset L1 embedding .dataset_layers.json DerivedSig metadata for both
Vectors are stored as separate sidecars so the human-readable text stays diffable in source control while embeddings remain binary. A missing or malformed .vec is a recoverable miss (vector lane skips the entry); the BM25 lane keeps working unchanged.
Detail (L2) layers are not persisted by this repo: detail content already lives in chunks via FSChunkRepo. Put for LayerDetail returns a validation error.
Deprecated: FSLayerRepo is a unit-test / demo-grade backend. Like FSChunkRepo it scales poorly on multi-doc datasets and is paired exclusively with FSChunkRepo through factory.NewLocal — once that pairing is retired in v0.5.0 there is no remaining caller. For production wiring use factory.NewRetrieval, which stores layers inside the retrieval.Index alongside chunks (see #134 and docs/migrations/v0.5.0.md).
func NewLayerRepo
deprecated
func NewLayerRepo(ws workspace.Workspace, prefix string, tok textsearch.Tokenizer) *FSLayerRepo
NewLayerRepo wires an FSLayerRepo to ws under prefix.
Deprecated: see FSLayerRepo. Use factory.NewRetrieval instead; slated for removal in v0.5.0.
func (*FSLayerRepo) DeleteByDataset ¶
func (r *FSLayerRepo) DeleteByDataset(ctx context.Context, datasetID string) error
DeleteByDataset removes the dataset-level layer files, their vec sidecars and the sig sidecar.
Per-document layers must be cleaned via DeleteByDoc; this method does NOT enumerate documents to keep its cost predictable.
func (*FSLayerRepo) DeleteByDoc ¶
func (r *FSLayerRepo) DeleteByDoc(ctx context.Context, datasetID, docName string) error
DeleteByDoc removes per-document layer files, their vec sidecars and the sig sidecar. Missing files are ignored so the call is idempotent.
func (*FSLayerRepo) Get ¶
func (r *FSLayerRepo) Get(ctx context.Context, datasetID, docName string, layer knowledge.Layer) (*knowledge.DerivedLayer, error)
Get returns the layer for (datasetID, docName, layer); docName == "" reads the dataset-level layer. Returns (nil, nil) when missing.
func (*FSLayerRepo) Put ¶
func (r *FSLayerRepo) Put(ctx context.Context, layer knowledge.DerivedLayer) error
Put writes a layer text, refreshes its DerivedSig sidecar and persists the embedding vector when present.
- layer.DocName == "" denotes a dataset-level layer.
- layer.Layer must be LayerAbstract or LayerOverview.
- layer.Vector, when non-empty, is encoded via encodeVec and stored in a co-located ".vec" sidecar; when empty, any pre-existing sidecar is removed so the vector and text never drift apart.
func (*FSLayerRepo) Search ¶
func (r *FSLayerRepo) Search(ctx context.Context, q knowledge.LayerQuery) ([]knowledge.Candidate, error)
Search performs a layer-tier scan with separate BM25 and vector lanes. The selected mode picks which lane runs:
- ModeBM25: keyword scan only (vectors ignored).
- ModeVector: cosine scan only (text ignored); requires q.Vector and a layer that has a stored vector sidecar.
- ModeHybrid (and the legacy default): both lanes run, results are merged keyed by (dataset, docName), and the higher score wins so downstream rankers see one Candidate per layer entity.
Hits carry Layer == q.Layer (contract guarantee #3: queries never cross layers). When neither lane has anything to score, Search returns (nil, nil) so SearchEngine can keep fanning out without a hard failure.
Source Files