Documentation
¶
Overview ¶
Package ingest implements the V0 ingestion pipeline: discover files, hash them, skip unchanged ones, parse, chunk, and write documents/chunks/links and an event in a single transaction per document. Parse and chunk run in a bounded worker pool; all writes are funneled through one writer because the SQLite handle is single-connection (one writer).
Index ¶
- func File(ctx context.Context, db *sql.DB, logger *slog.Logger, ...) (docID string, chunks int, err error)
- func ForgetPath(ctx context.Context, db *sql.DB, logger *slog.Logger, abs, uri string) (bool, error)
- func Match(relPath string, include, exclude, securityExclude []string) bool
- func RenderOKF(in CaptureInput) (filename string, content []byte)
- func WriteCapture(dir, filename string, content []byte) (string, error)
- func WriteFileAtomic(absPath string, content []byte) (string, error)
- type CaptureInput
- type MoveEntry
- type MoveResult
- type OkfyOptions
- type OkfyResult
- type Option
- type Options
- type Pipeline
- type Rules
- type Summary
- type WatchConfig
- type Watcher
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func File ¶
func File(ctx context.Context, db *sql.DB, logger *slog.Logger, absPath, uri, collection string, cfg chunk.Config) (docID string, chunks int, err error)
File performs a one-shot ingest of a single explicit file, reusing the existing parse→chunk→write path via Pipeline.IngestPath. It does not scan a directory and changes no pipeline behavior. uri is the stable, project-root- relative identifier stored as documents.uri (the captured file's path relative to the project root, e.g. ".mnemos/capture/idea-20260625-ab12cd34.md"), which is what mnemos.search and mnemos.read cite.
One-shot ingest is the default: remember both writes and ingests, and the call is intentionally retained. Setting defer_to_watcher=true makes remember write-only, leaving capture_dir ingestion to a running watcher (the watcher then owns capture). The two modes stay safe together because Phase 1 hash-skip turns the watcher's re-sighting of an already-ingested file into a no-op, so there is no double index either way.
func ForgetPath ¶
func ForgetPath(ctx context.Context, db *sql.DB, logger *slog.Logger, abs, uri string) (bool, error)
ForgetPath deletes a file from the OKF tree: it removes the document from the index and the backing file from disk, returning whether a file was actually removed. It is idempotent — forgetting a path whose file is already gone is not an error and reports deleted=false. Both the CLI `forget` command and the mnemos.forget MCP tool route through it, so the two surfaces cannot drift in behavior or crash semantics.
The index is cleared BEFORE the file is removed. A DB failure before any disk mutation leaves the system coherent: the file and its index entry both still exist, so a re-run completes cleanly. The reverse order risks a ghost document — indexed but missing on disk — that survives until the next reconcile. The opposite residue, an orphaned file present on disk but not indexed, is benign: the next ingest or watcher reconcile re-indexes or re-removes it automatically.
func Match ¶
Match reports whether the file at root-relative path relPath is selected by the include/exclude glob sets, using the exact same predicate the directory scanner applies. relPath is matched after slash-normalization, so callers may pass an OS-native relative path. The watcher uses this to decide, per live event, whether a file should be ingested — sharing one predicate guarantees the watcher and the batch pipeline never disagree about what is indexable.
func RenderOKF ¶
func RenderOKF(in CaptureInput) (filename string, content []byte)
RenderOKF serializes a capture into an OKF markdown document: a YAML frontmatter block (type, tags, timestamp in RFC3339, collection) followed by a blank line and the body. The returned filename is collision-safe and readable — "<type>-<YYYYMMDD>-<shorthash>.md" — where the short hash is derived from the body so identical notes captured the same day reuse a filename (idempotent re-ingest) while distinct notes diverge.
func WriteCapture ¶
WriteCapture writes content to dir/filename atomically. It is the auto-named capture wrapper: it joins filename under dir and delegates the atomic write to WriteFileAtomic. It returns the absolute path of the written file.
func WriteFileAtomic ¶
WriteFileAtomic writes content to absPath atomically: it creates the parent directory if missing, writes to a temp file in the same directory, then renames it over the destination. The same-directory temp file guarantees the rename is atomic (no cross-device copy). It returns absPath on success.
Types ¶
type CaptureInput ¶
type CaptureInput struct {
Type string
Body string
Tags []string
Collection string
// Timestamp is the capture time. The program may use time.Now(); only the
// eval/workflow scripts are forbidden wall-clock access, not the server.
Timestamp time.Time
}
CaptureInput is the content an agent hands to mnemos.remember, normalized for serialization. Type is the OKF document type ("idea" or "document"), Body the note text, Tags the optional frontmatter tags, and Collection the logical space the captured note belongs to.
type MoveEntry ¶
MoveEntry records one file relocated by MovePath: its old and new uris and the document id it was re-indexed under (the id derives from collection + uri, so it changes on every move).
type MoveResult ¶
MoveResult is the outcome of a MovePath call. IsDir reports whether the source was a directory (a subtree move). DanglingLinks is the total number of inbound markdown links left pointing at the old uris — not rewritten in V0.
func MovePath ¶
func MovePath(ctx context.Context, db *sql.DB, logger *slog.Logger, absFrom, absTo, oldURI, newURI string, cfg chunk.Config) (MoveResult, error)
MovePath moves a file or directory within the OKF tree: it renames the path on disk and re-indexes every affected indexed document under its new uri, preserving each document's collection. For a directory it relocates the whole subtree and re-indexes each document that was indexed under the old prefix (files present on disk but never indexed simply move with the subtree). Both the CLI `mv` command and the mnemos.move MCP tool route through it, so the two surfaces cannot drift.
Inbound markdown links to moved uris are not rewritten (a V0 limitation); their total is returned in MoveResult.DanglingLinks and logged as a warning.
type OkfyOptions ¶
type OkfyOptions struct {
// TreeRoot is the writable OKF tree root used to confine Source and Out.
TreeRoot string
// Exclude is the [security].exclude glob set applied to Source and Out.
Exclude []string
// Source is the tree-root-relative (or absolute, within the tree) path of the
// plain file to convert. It must resolve to a .txt or .md file.
Source string
// Out is the output path within the tree. When empty it defaults to Source
// with a .md extension. It must resolve to a .md file and must not equal Source.
Out string
// Collection is the collection the OKF document is indexed under. Empty means
// "default".
Collection string
// Type is the OKF note type recorded in the frontmatter (e.g. "document").
Type string
// Tags are the optional frontmatter tags.
Tags []string
// Force overwrites the output file when it already exists.
Force bool
// Chunking is the token budget applied to the ingest.
Chunking chunk.Config
// Scanner screens the source body for secrets before anything is written.
Scanner security.SecretScanner
}
OkfyOptions configures Okfy: the shared core that turns a plain .txt or .md file into an OKF document and indexes it, leaving the source intact. It is called by both the `okfy` CLI command and the mnemos.okfy MCP tool so the two surfaces share one implementation.
type OkfyResult ¶
type OkfyResult struct {
URI string
SourceURI string
Collection string
DocumentID string
Chunks int
}
OkfyResult reports the outcome of Okfy: the output document's tree-root- relative URI (an immediate citation), the source URI, the resolved collection, the document id, and the number of chunks indexed.
func Okfy ¶
func Okfy(ctx context.Context, db *sql.DB, logger *slog.Logger, opts OkfyOptions) (OkfyResult, error)
Okfy validates Source within the tree, derives and validates the output path, secret-scans the source body, renders an OKF document, writes it atomically, indexes it, and keeps the source. The source file is never overwritten: an output equal to the source is rejected, and an existing output requires Force.
type Option ¶
type Option func(*Pipeline)
Option customizes a Pipeline at construction.
func WithMaxFileBytes ¶
WithMaxFileBytes sets the per-file size cap. A file larger than n is skipped (with a warning) instead of being read into memory. n <= 0 disables the cap.
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline ingests files into the store. It holds the dependencies the run needs; construct one per command invocation.
func New ¶
New builds a Pipeline over db using the default offline token estimator and the default file-size cap. Pass WithMaxFileBytes to override the cap.
func (*Pipeline) IngestPath ¶
func (p *Pipeline) IngestPath(ctx context.Context, absPath, uri, collection string, cfg chunk.Config) (docID string, chunks int, err error)
IngestPath ingests a single explicit file using the same prepare→write path the directory pipeline uses, then returns the document id and the number of chunks written. absPath is the file to read; uri is the stable, project-root- relative identifier stored as documents.uri (capture passes the path relative to the project root so the captured note is a stable citation). A file whose content hash is unchanged is a no-op: it returns the existing document id with zero chunks. This is the one-shot ingest seam used by mnemos.remember; it adds no scan and no pipeline behavior of its own beyond reusing prepare/write.
func (*Pipeline) Run ¶
Run executes the pipeline and returns a Summary. Discovery and hashing run on the calling goroutine; parse+chunk run in a bounded worker pool; every result flows through one writer goroutine that owns all DB writes and all tallying, so the counters are race-free by construction.
type WatchConfig ¶
type WatchConfig struct {
// Include/Exclude/SecurityExclude are the same glob sets the scanner uses;
// the watcher applies them through the shared Match predicate so a live event
// and a batch scan agree on what is indexable.
Include []string
Exclude []string
SecurityExclude []string
// Chunking is the token budget for reindexing a changed file.
Chunking chunk.Config
// StorageDir is the directory holding the SQLite database (e.g. ".mnemos").
// Events under it are ignored so the watcher does not churn on WAL/SHM writes
// it triggers itself.
StorageDir string
// MaxFileBytes caps the size of a single file read into memory; a larger file
// is skipped with a warning. A value <= 0 disables the cap, matching the
// [indexing].max_file_bytes config contract.
MaxFileBytes int64
}
WatchConfig carries the selection and chunking settings the watcher reuses from the batch pipeline, plus the storage directory to ignore so the watcher never reacts to its own database writes.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher incrementally keeps a collection in sync with a directory tree. It performs a startup reconcile (full hash-skip scan plus removal of vanished documents) and then watches for live changes, reindexing modified files and deleting removed ones. It reuses the Phase 1 pipeline for all ingestion, so it adds no parsing or chunking logic of its own.
func NewWatcher ¶
func NewWatcher(db *sql.DB, logger *slog.Logger, root, collection string, cfg WatchConfig) (*Watcher, error)
NewWatcher builds a Watcher over db. root is the directory to watch (relative or absolute; it is resolved to an absolute path), collection the logical space reindexed files belong to, and cfg the shared selection/chunking settings.
func (*Watcher) Ready ¶
func (w *Watcher) Ready() <-chan struct{}
Ready returns a channel that is closed once Run has completed its startup reconcile and registered the live filesystem watch. Until then, file changes may not yet be observed; after it is closed, they are. Callers that mutate the watched tree and expect the watcher to react (notably tests, but also any orchestrator that wants to know the watcher is live) can block on it. If Run returns before going live (a reconcile or watch-setup error), the channel is never closed; select on it together with the Run error or a timeout.
Source Files