cortex

package
v0.10.0-beta.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 24, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	FederationModeSync      = "sync"      // bidirectional: pull from peers + serve events
	FederationModePublish   = "publish"   // outbound only: serve events, never pull
	FederationModeSubscribe = "subscribe" // inbound only: pull from peers, refuse to serve
)

Federation mode constants.

View Source 
const (
	PeerModeSync   = "sync"   // actively pull from this peer
	PeerModePaused = "paused" // configured but skipped by the syncer
)

Peer mode constants.

View Source 
const AccessKeyEnvVar = "NOEMA_MCP_KEY"

AccessKeyEnvVar is the environment variable that overrides cortex.md's access.shared_key_file. When set to a non-empty, non- whitespace value it is used as the MCP shared key, and any configured file path is recorded so the caller can warn about the override.

View Source 
const ManifestVersion = 2

ManifestVersion is the current cortex.md schema version. Cortexes written at this version carry an `id` field; cortexes at any earlier version must be migrated via `noema migrate cortex-id` before federation will accept them.

View Source 
const MaxSearchQueryLen = 1000

MaxSearchQueryLen caps the length of FTS5 search queries to prevent denial of service via expensive wildcard or deeply nested expressions.

Variables

View Source 
var ErrDistillSourceMissing = errors.New("source trace not found")

ErrDistillSourceMissing is returned when one of the declared source trace IDs does not exist in the cortex. Keeps the derived_from lineage honest — a consolidation event that points at a ghost trace breaks retrospective source-recall queries downstream.

View Source 
var ErrDistillSourcesInsufficient = errors.New("distilled trace requires >= 2 source IDs")

ErrDistillSourcesInsufficient is returned when a caller tries to record a consolidation result with fewer than two source traces. Below that threshold the operation is a rename, not a consolidation — the event log records the category via the action constant, so this constraint protects the distinction.

View Source 
var ErrSourceLocked = errors.New("trace is source-locked")

ErrSourceLocked is returned when a mutation is attempted on a source-locked trace from a foreign origin.

View Source 
var ErrTierMismatch = errors.New("tier mismatch")

ErrTierMismatch is returned when a caller asserts an expected tier on AdminPurge and the actual row tier disagrees. The safety rail prevents fat-finger destruction of long-term rows while the caller thought they were purging something short-term.

Functions

func KeyFingerprint added in v0.3.0 

func KeyFingerprint(key string) string

KeyFingerprint returns a non-secret SHA-256 fingerprint of an MCP shared key, formatted SSH-style as SHA256:<pair>:<pair>:... Safe to log, display in federation_status, and read aloud over an out-of-band channel when confirming a pairing.

func SanitizeFTS5Query added in v0.7.0 

func SanitizeFTS5Query(q string) string

SanitizeFTS5Query quotes each whitespace-delimited token that contains any FTS5 structural character (., -, :, /, (, ), etc.) so the parser treats it as a literal phrase. Tokens that are already quoted or use explicit FTS5 operators (AND, OR, NOT, prefix*) are passed through unchanged to preserve power-user syntax. A bareword token is anything composed solely of Unicode letters, digits, and underscore, with an optional single trailing '*' for prefix search.

func WriteManifest added in v0.3.0 

func WriteManifest(dir string, m Manifest) error

WriteManifest writes the manifest back to cortex.md in the given directory. The output is always framed as markdown frontmatter: a `---` fence, the YAML, a closing `---` fence, then m.Body if non-empty. Legacy bare-YAML manifests are silently upgraded to framed form on the first write.

Types

type AccessConfig added in v0.3.0 

type AccessConfig struct {
	// SharedKeyFile is a path to a sidecar file whose first non-empty
	// line is the shared bearer token. Relative paths are resolved
	// against the cortex directory. The manifest itself never holds the
	// secret — only a pointer to where it lives.
	SharedKeyFile string `yaml:"shared_key_file,omitempty"`
}

AccessConfig holds MCP endpoint authentication settings for cortex.md. When SharedKeyFile is set, the HTTP MCP endpoint runs in shared-key mode and every incoming request must carry a matching Authorization bearer header. See docs/design/mcp-auth-plan.md for the full design.

type AccessKey added in v0.3.0 

type AccessKey struct {
	// Value is the raw bearer token. It must never be logged, written to
	// the event log, serialised to MCP responses, or echoed in error
	// messages. Only the Fingerprint is safe to surface.
	Value string

	// Source is "env" (NOEMA_MCP_KEY), "file" (read from disk), or ""
	// (open mode).
	Source string

	// Path is the absolute file path the key was read from when
	// Source == "file", or the configured-but-overridden file path when
	// Source == "env" and the manifest also declared a shared_key_file.
	// Empty when neither applies.
	Path string

	// Fingerprint is a non-secret SHA-256 digest of Value, formatted
	// SSH-style (e.g. SHA256:a3:f1:...:c2). Safe to log and display.
	Fingerprint string
}

AccessKey is the resolved shared key for the HTTP MCP endpoint, along with metadata about where it came from and a non-secret fingerprint. The zero value represents open mode (no authentication).

func LoadAccessKey added in v0.3.0 

func LoadAccessKey(cortexDir string, cfg *AccessConfig) (AccessKey, error)

LoadAccessKey resolves the active MCP shared key for a cortex.

Resolution order (highest priority first):

  1. NOEMA_MCP_KEY — if set to a non-empty value, wins. The configured file path is still recorded in AccessKey.Path so the caller can log that the env var overrode it.
  2. cfg.SharedKeyFile — read relative to cortexDir unless already absolute. The file is validated for permissions, size, and format (see loadKeyFile).
  3. Open mode — returns the zero AccessKey with a nil error.

Errors are returned when the env var is set to only whitespace, when a configured file cannot be read, when permissions are looser than 0600, when the file exceeds 4 KiB, when it is empty or whitespace only, or when it contains two or more non-empty lines.

func (AccessKey) EnvOverride added in v0.3.0 

func (k AccessKey) EnvOverride() bool

EnvOverride reports whether NOEMA_MCP_KEY took precedence over a configured access.shared_key_file. Callers use this to emit the one-line warning on startup.

func (AccessKey) Keyed added in v0.3.0 

func (k AccessKey) Keyed() bool

Keyed reports whether shared-key mode is active.

type BackfillResult added in v0.3.0 

type BackfillResult struct {
	BackfilledIDs []string // active traces that received a synthetic create event
	SkippedIDs    []string // traces with no create event but currently archived/trashed
}

BackfillResult summarises a `noema events backfill` operation. The slices hold trace IDs (not row counts) so the caller can render them line-by-line for the operator's audit trail.

type ConsolidationConfig 

type ConsolidationConfig struct {
	// Enabled is the master opt-in. The feature ships off by default so
	// existing cortexes are unaffected until users explicitly turn it on.
	Enabled bool `yaml:"enabled,omitempty"`

	// Cron is the nightly trigger time in "HH:MM" local-clock format
	// (e.g. "03:00"). Empty means no cron trigger.
	Cron string `yaml:"cron,omitempty"`

	// IdleMinutes fires a pass after N minutes of no trace mutations.
	// Zero disables the idle trigger. Cooldown equal to IdleMinutes is
	// enforced so a quiet cortex doesn't consolidate on every tick.
	IdleMinutes int `yaml:"idle_minutes,omitempty"`

	// ThresholdShort fires a pass when the short-term tier count
	// exceeds this many active traces. Zero disables the trigger.
	// Hysteresis: once tripped, re-arms only when the count drops
	// back below 0.8 * ThresholdShort so a cortex hovering near the
	// threshold doesn't thrash.
	ThresholdShort int `yaml:"threshold_short,omitempty"`

	// WindowHours bounds the candidate pool for a pass to traces
	// created within the last N hours. Zero defaults to 24.
	WindowHours int `yaml:"window_hours,omitempty"`

	// LLMEnabled opts into the LLM-driven distillation path used by
	// `noema consolidate`. When false (default), the in-process agent
	// runs pure-heuristic 1:1 promotion only. When true, operators are
	// expected to run `noema consolidate` periodically (or wire the
	// subcommand up to cron / launchd) so clusters get distilled into
	// mid-tier memories instead of promoted one-to-one.
	LLMEnabled bool `yaml:"llm_enabled,omitempty"`

	// ModelTier is the prompt-style profile the consolidation pipeline
	// uses when calling the LLM: "small" (7B-13B, multi-step template),
	// "large" (30B-70B, same plus confidence step), or "frontier"
	// (single-shot JSON). See docs/plans/consolidation-plan.md §6 in
	// the Noema-design repo for the full profile matrix.
	ModelTier string `yaml:"model_tier,omitempty"`

	// LocalLLMEndpoint is the OpenAI-compatible base URL to post
	// chat-completion requests to. Covers Ollama (/v1), LMStudio,
	// llama.cpp server, vLLM, and OpenAI itself. Empty disables the
	// LLM path even when LLMEnabled is true; `noema consolidate` exits
	// with a clear error rather than trying to guess a default.
	LocalLLMEndpoint string `yaml:"local_llm_endpoint,omitempty"`

	// ModelName is the model identifier passed in the `model` field of
	// the chat-completion request body (e.g. "llama3.1:70b",
	// "claude-opus-4-7", "gpt-4o"). Must match what the endpoint
	// recognizes — no translation layer here.
	ModelName string `yaml:"model_name,omitempty"`

	// APIKeyEnv names an environment variable whose value is attached
	// as a Bearer token to outgoing requests. Empty means no auth
	// header — correct for local runners like Ollama that don't care.
	// The key itself never lives in cortex.md; this is a pointer to
	// where the operator keeps it, matching the access.shared_key_file
	// pattern for the MCP server.
	APIKeyEnv string `yaml:"api_key_env,omitempty"`

	// Graduation controls the mid→long promotion heuristic (Phase 15).
	// Leaving the block unset uses the defaults. Setting
	// Graduation.Enabled=false on an existing cortex keeps mid as the
	// terminal tier for automatic promotion — useful for operators who
	// want to curate the long tier by hand via `noema memory promote`.
	Graduation *GraduationConfig `yaml:"graduation,omitempty"`
}

ConsolidationConfig controls the background memory-consolidation agent. All three triggers (cron, idle, threshold) are composable — set any combination, the agent fires on whichever triggers first. Leaving all three empty disables the agent even when Enabled is true.

See docs/plans/consolidation-plan.md §4 in the Noema-design repo for the cadence design. Model-tier and local-LLM fields are deferred to later phases; this config block only covers scheduling for now.

func (*ConsolidationConfig) EffectiveModelTier 

func (cc *ConsolidationConfig) EffectiveModelTier() string

EffectiveModelTier returns the configured model-tier profile or "large" as the default. Large is the conservative middle ground — it works well on local 30B-70B models and stays safe on frontier models that would also accept tighter prompts.

func (*ConsolidationConfig) EffectiveWindowHours 

func (cc *ConsolidationConfig) EffectiveWindowHours() time.Duration

EffectiveWindowHours returns the window duration with the 24h default applied when unset.

type Cortex 

type Cortex struct {
	ID   string // ULID, stable across renames; the federation identity key
	Name string // human-readable display label
	Dir  string
	DB   *db.DB
	// contains filtered or unexported fields
}

func Open 

func Open(name, dir string) (*Cortex, error)

Open opens an existing Cortex by directory path. It ensures required subdirectories exist and auto-purges expired trash.

func (*Cortex) Add 

func (c *Cortex) Add(t *trace.Trace) error

Add writes a new Trace to disk and inserts it into the DB.

func (*Cortex) AdminPurge 

func (c *Cortex) AdminPurge(id, reason, expectedTier string, hard bool, actor ReadActor) error

AdminPurge is the sanctioned ceremonious-delete path for any trace, including long-term ones the immutability trigger would otherwise block. It is deliberately verbose at the call site (expected tier, reason, and the CLI flag --confirm to reach it) so accidental invocation is hard.

Behaviour:

  • expectedTier must equal the trace's actual tier, else ErrTierMismatch. Prevents the classic accident of purging a long-term trace while thinking it's short-term.
  • When hard is false (default), the row is tombstoned: body wiped to "[purged: <reason>]", purge_reason/purged_at stamped, file deleted from disk, tags cleared, FTS index updated. The DB row stays so lineage references continue to resolve and federation peers can apply the same tombstone on replay.
  • When hard is true, the row and all lineage references to it are removed outright. Reserved for GDPR-style mandates where even the ID must not persist.

Emits ActionPurgeLongTerm for tier='long' soft-purges, ActionPurgeHard for any --hard, and ActionPurge for short/mid soft-purges. The event data payload carries the original content_hash as durable proof of what was destroyed, plus the reason and actor identity.

For long-term rows, the immutability trigger is suspended for the duration of the transaction via DROP+re-CREATE using the trigger's own SQL as recorded in sqlite_master. If the tx rolls back for any reason, SQLite restores the trigger automatically along with the other DDL changes, so a mid-purge failure cannot leave the database in a state where long-term immutability is silently broken.

func (*Cortex) Append added in v0.6.0 

func (c *Cortex) Append(id, content string) error

Append adds content to the end of an existing trace's body. It reads the current file, appends the new content (with a newline separator if the existing body doesn't already end with one), recomputes the content hash, and emits a standard "update" event. Designed for fire-and-forget logging where agents append to a running trace without consuming its full body.

func (*Cortex) ApplyExternalPurge added in v0.9.0 

func (c *Cortex) ApplyExternalPurge(id string) error

ApplyExternalPurge handles the case where a trace file vanishes from trash/traces/ — the user emptied Noema's trash via the filesystem. Deletes the DB row and emits ActionPurge. This is the only path outside the age-based Purge() that permanently removes a trace through the event log, so federation peers see it too.

func (*Cortex) Archive 

func (c *Cortex) Archive(id string) error

func (*Cortex) ArchiveDir 

func (c *Cortex) ArchiveDir() string

func (*Cortex) BackfillCreateEvents added in v0.3.0 

func (c *Cortex) BackfillCreateEvents(dryRun bool) (BackfillResult, error)

BackfillCreateEvents emits synthetic `create` events for any active trace that lacks one in the event log. This folds traces that pre-date the event log — or that landed via `noema sync`, which intentionally emits no events because it is reconciliation, not a semantic mutation — back into the federated history so peers can replay them.

Each backfilled event uses a fresh ULID, the local cortex_id and origin, the current wall-clock time as the event timestamp, and a JSON snapshot of the trace's current frontmatter + body. The trace's own `created` field (in the markdown frontmatter and the DB row) is left untouched, so the audit trail still surfaces "this happened on <real date>" — the event timestamp only records when the backfill ran. Using wall-clock time keeps per-cortex ULID monotonicity and avoids the event log lying about when the event was actually appended.

Archived and trashed traces are skipped: emitting only a `create` event for them would leave federation diverged (peers would materialise the trace as active and never see the archive/trash). Recover or unarchive the trace first if it needs to federate.

If dryRun is true, no events are written and the vector clock is not touched, but the returned result still lists every trace that would have been backfilled or skipped — so operators can preview before committing.

The iteration is idempotent: traces that already have a create event in the log (whether locally emitted or replayed from a peer) are not in the candidate set, so running this twice is a no-op on the second pass.

func (*Cortex) CheckSourceLock added in v0.5.0 

func (c *Cortex) CheckSourceLock(id string) error

CheckSourceLock returns ErrSourceLocked if the trace is source-locked by a foreign origin. The check is skipped when forceSourceLock is set.

func (*Cortex) Close 

func (c *Cortex) Close() error

func (*Cortex) CreateDistilledTrace 

func (c *Cortex) CreateDistilledTrace(spec DistilledTraceSpec) (string, error)

CreateDistilledTrace materialises the result of an LLM-driven consolidation pass: a new mid-tier trace whose derived_from lineage points at the short-term sources that fed the distillation. Emits ActionConsolidate alongside the standard ActionCreate so the event log carries both "a new trace exists" and "this is why it exists / which model produced it / how confident" as separate, replayable records.

Source handling (v1 — "net-add with source promotion"): every short-tier source is promoted to mid once the distillation lands. The distilled trace is the retrievable summary; sources remain individually addressable at mid so agents can pull the full detail via derived_from when the distillation's compression is lossy. This also prevents the next consolidation pass from re-clustering the same sources into a duplicate distillation — the candidate query filters to tier='short', so promoted sources drop out of the candidate pool.

FUTURE (v2+): "source archival" — instead of promoting sources to mid, move them to archived once the distillation is trusted (confidence threshold + retention window). That keeps mid-tier curated and closer to the biological metaphor where the detailed memory fades as the distillation takes its place. Not v1 because the grace-period machinery adds a new daemon and needs a confidence-calibration pass first. The current design's derived_from lineage already supports the retrieval path that option 2 would need, so the switch is local to this function + a new archival sweep — no schema changes.

func (*Cortex) Demote 

func (c *Cortex) Demote(id, newTier string) error

Demote steps a trace back a tier: mid -> short. Long demotion is reserved for the admin-purge ceremony (Phase 6) which suspends the immutability trigger explicitly; Demote refuses it here to keep the "long is terminal in routine operation" invariant intact. Emits ActionDemote with {from, to}.

func (*Cortex) DerivedBy added in v0.3.0 

func (c *Cortex) DerivedBy(id string) ([]string, error)

DerivedBy returns all trace IDs that list the given trace as a source.

func (*Cortex) DivergenceCount added in v0.3.0 

func (c *Cortex) DivergenceCount() (int, error)

DivergenceCount returns the number of unresolved divergence traces.

func (*Cortex) EmitCoordinationEvent 

func (c *Cortex) EmitCoordinationEvent(action event.Action, windowID string, data any) error

emitEvent appends an event to the log inside the given transaction, incrementing the local vector clock. All reads/writes go through the tx to avoid SQLite lock contention. Vector clocks are keyed on the cortex ID (a stable ULID), not the cortex name — see docs/design/cortex-uuid-plan.md. EmitCoordinationEvent emits an event that isn't tied to a specific trace mutation — used by the consolidation election protocol for Claim / Success / Fail events (see consolidation-plan.md §14). The windowID serves as trace_id in the events table (the column is NOT NULL but doesn't enforce a foreign key on traces — coordination events use a synthetic ID scoped to the election window).

Data is JSON-marshaled by the caller's choice of struct; pass nil to emit an event with an empty payload.

func (*Cortex) Events added in v0.3.0 

func (c *Cortex) Events(traceID string) ([]event.Event, error)

Events returns the event log for a specific trace, ordered chronologically.

func (*Cortex) EventsSince added in v0.3.0 

func (c *Cortex) EventsSince(afterID string, limit int) ([]event.Event, error)

EventsSince returns events after the given ULID cursor, up to limit.

func (*Cortex) Get 

func (c *Cortex) Get(id string) (*Row, error)

func (*Cortex) GetAs 

func (c *Cortex) GetAs(id string, actor ReadActor) (*Row, error)

GetAs is the actor-aware counterpart to Get. It returns the same Row and bumps read_count + last_read_at when the caller is ActorAgent. Other actors (ActorHuman, ActorSystem) short-circuit to plain Get behavior.

The bump writes to trace_usage keyed on (trace_id, local cortex ID) — CRDT-style per-peer counters. Federated peers receive each other's counters via sync_read_signal and the heuristic queries the aggregate. Long-tier traces skip the bump because the immutability trigger blocks updates; revisit if long-tier usage signal ever proves useful.

func (*Cortex) GetClock added in v0.3.0 

func (c *Cortex) GetClock() (federation.VClock, error)

GetClock returns the current vector clock.

func (*Cortex) GraduationCandidates 

func (c *Cortex) GraduationCandidates(minAge time.Duration) ([]PromotionCandidate, error)

GraduationCandidates returns every active tier='mid' trace older than minAge. The mirror of PromotionCandidates — that one narrows to the rolling short-term pool for short→mid evaluation; graduation evaluates mid→long on traces that have had time to prove durability, so the inequality flips (`created_at <= cutoff`) and the lower bound is open. Archived/trashed/purged rows are excluded for the same reason as PromotionCandidates.

func (*Cortex) IngestExternalDelete added in v0.9.0 

func (c *Cortex) IngestExternalDelete(id string) error

IngestExternalDelete handles the case where a trace file vanishes from traces/ or archive/traces/ entirely (user deleted via Finder or rm). Restores the last known body from the local event log into trash/traces/<id>.md so the delete is recoverable, then stamps trashed_at and emits ActionTrash. When no event snapshot exists (pre-event-log traces), skips the restore and emits the event anyway with an empty body so federation peers still see the state transition.

Source-locked foreign traces are refused with ErrSourceLocked — the watcher must skip rather than poison downstream peers with a tamper.

func (*Cortex) LastMutationTime 

func (c *Cortex) LastMutationTime() (time.Time, error)

LastMutationTime returns the timestamp of the most recent event in the local log. Used by the consolidation agent's idle trigger to decide whether the cortex has been quiet long enough to consolidate. Returns zero time (not an error) on an empty log so a cortex with no history yet reads as "idle since beginning of time".

func (*Cortex) List 

func (c *Cortex) List(opts ListOptions) ([]Row, error)

func (*Cortex) LocalUsageSince 

func (c *Cortex) LocalUsageSince(since string, limit int) ([]federation.TraceUsage, error)

LocalUsageSince returns trace_usage rows owned by the local peer (peer_cortex_id = c.ID) with updated_at > since, ordered by updated_at so callers can advance their cursor to the last row's UpdatedAt on each batch. A peer only publishes its own deltas; remote rows we've synced in are never re-broadcast (prevents amplification loops and keeps bandwidth linear).

A zero-value since returns everything this peer owns.

func (*Cortex) MarkArchivedNoMove added in v0.9.0 

func (c *Cortex) MarkArchivedNoMove(id string) error

MarkArchivedNoMove updates the DB to mark a trace as archived and emits an ActionArchive event — without moving any files. The filesystem watcher uses this when an external tool (e.g. Finder drag) has already moved the file from traces/ to archive/traces/. Returns nil if the trace is already archived (idempotent). Errors if the trace is in the trash.

func (*Cortex) MarkRecoveredNoMove added in v0.9.0 

func (c *Cortex) MarkRecoveredNoMove(id string) error

MarkRecoveredNoMove clears trashed_at and emits ActionRecover without moving the file. Used when the watcher sees a trashed trace's file reappear in traces/ or archive/traces/.

func (*Cortex) MarkTrashedNoMove added in v0.9.0 

func (c *Cortex) MarkTrashedNoMove(id string) error

MarkTrashedNoMove stamps trashed_at and emits ActionTrash without moving the file. Used when the watcher sees a file appear in trash/traces/ via an external move. Clears archived_at to match existing Trash semantics. Idempotent if already trashed.

func (*Cortex) MarkUnarchivedNoMove added in v0.9.0 

func (c *Cortex) MarkUnarchivedNoMove(id string) error

MarkUnarchivedNoMove clears archived_at and emits ActionUnarchive without moving the file. Used by the watcher when a file reappears in traces/ after being in archive/traces/. Idempotent if already unarchived.

func (*Cortex) MergeClock added in v0.3.0 

func (c *Cortex) MergeClock(remote federation.VClock) error

MergeClock merges a remote vector clock into the local clock. The merge is capped at federation.MaxVClockEntries to prevent a malicious peer from inflating the clock with synthetic cortex IDs.

func (*Cortex) MergeRemoteUsage 

func (c *Cortex) MergeRemoteUsage(rows []federation.TraceUsage) error

MergeRemoteUsage upserts a batch of rows from a remote peer into trace_usage using CRDT MAX-merge semantics. Safe against out-of-order arrivals and duplicate deliveries — an older row re-arriving after a newer one leaves the stored values untouched. A peer must never call this with rows whose peer_cortex_id matches its own (that would let a remote overwrite the local peer's authoritative counters); the caller is responsible for that invariant, but as a guard we skip any such rows instead of applying them.

Rows for trace_ids that don't exist locally yet are inserted anyway — the corresponding create event will arrive separately via sync_events and the FK enforces consistency eventually. If the create never shows up (peer disagrees about trace existence) the orphan is harmless; the aggregate query joins traces as LEFT so it's simply ignored.

func (*Cortex) Promote 

func (c *Cortex) Promote(id, newTier string) error

Promote advances a trace to the next memory tier: short -> mid, or mid -> long. Cross-skips (short -> long) and reverse transitions are refused; callers use Demote for mid -> short, and long is terminal from routine operation (the admin-purge path in Phase 6 handles any legitimate long-term exit). Emits ActionPromote with {from, to} so federation peers replicate the same transition.

Only the DB tier column is updated; the on-disk frontmatter keeps whatever tier value was last written to it and re-syncs to DB state on the next legitimate Cortex.Update (see Phase 1's file-drift rail). Body bytes are untouched, so content_hash stays valid and the filesystem watcher's loopback detection skips the rewrite.

func (*Cortex) PromotionCandidates 

func (c *Cortex) PromotionCandidates(tier string, window time.Duration) ([]PromotionCandidate, error)

PromotionCandidates returns every active trace in the given tier whose created_at falls within the rolling window. The caller does the scoring; this method is only responsible for narrowing to the pool worth considering. Archived, trashed, and purged rows are excluded — only memory currently in use is a candidate for promotion.

derived_from_count joins the lineage view added in migration 008 so the scorer can weight "others reference this" alongside reads and modifies without an extra round-trip per candidate.

func (*Cortex) Purge added in v0.2.0 

func (c *Cortex) Purge(days int) error

Purge permanently deletes traces that have been in the trash for more than days days. A days value of 0 is treated as 30.

func (*Cortex) RebuildFTSIfStale added in v0.9.2 

func (c *Cortex) RebuildFTSIfStale() error

RebuildFTSIfStale repopulates traces_fts when it contains fewer rows than the traces table (including trashed, archived, and active rows — everything that should be searchable). The mismatch is expected exactly once per cortex, immediately after migration 008 runs and replaces the virtual table. On fresh cortexes and subsequent opens the counts match and this is a no-op.

The rebuild walks every row in the traces table, reads the trace's body from disk (since body is not in SQL), and re-inserts all four FTS columns. Rows whose files are missing on disk are inserted with empty body so at least their title and tags remain searchable — operators can fix those with `noema sync` later.

Runs outside any caller's transaction because it iterates many traces and couldn't share a short-lived transaction with them cleanly. Each row is reinserted in its own tx so a mid-rebuild failure leaves the index partially populated rather than empty.

func (*Cortex) Recover added in v0.2.0 

func (c *Cortex) Recover(id string) error

Recover moves a trace out of the trash and back to the active traces directory.

func (*Cortex) Remove 

func (c *Cortex) Remove(id string) error

Remove permanently deletes a trace from disk and the database. Use Trash for recoverable deletion.

func (*Cortex) ReplayEvent added in v0.3.0 

func (c *Cortex) ReplayEvent(e event.Event) error

ReplayEvent materializes a remote event locally without emitting a new event. The remote event is stored in the local log with its original ID and origin.

func (*Cortex) ResolveDivergence added in v0.3.0 

func (c *Cortex) ResolveDivergence(divergenceID, acceptOrigin, customBody string) error

ResolveDivergence resolves a divergence trace by either picking one of the versions stored in the divergence body (by origin name) or applying a caller-supplied custom merge. Exactly one of acceptOrigin or customBody must be non-empty. The divergence trace is trashed once the original is updated.

func (*Cortex) Search 

func (c *Cortex) Search(query string, opts ListOptions) ([]Row, error)

func (*Cortex) SetForceSourceLock added in v0.5.0 

func (c *Cortex) SetForceSourceLock(v bool)

SetForceSourceLock enables or disables the source-lock override. When enabled, mutations on source-locked traces succeed with a warning instead of being refused. Intended for CLI --force flags only.

func (*Cortex) ShortTierCount 

func (c *Cortex) ShortTierCount() (int, error)

ShortTierCount returns the number of active (not archived, trashed, or purged) short-term traces. Used by the consolidation agent's threshold trigger.

func (*Cortex) Sync added in v0.2.2 

func (c *Cortex) Sync() (SyncResult, error)

Sync reconciles the database with the current state of the markdown files on disk. It walks traces/, archive/traces/, and trash/traces/, upserts every file it finds, and reports orphaned DB rows (not deleted — just reported).

func (*Cortex) SyncWithOptions added in v0.3.0 

func (c *Cortex) SyncWithOptions(opts SyncOptions) (SyncResult, error)

SyncWithOptions is Sync with explicit options. See SyncOptions.

func (*Cortex) TierStats 

func (c *Cortex) TierStats() (TierStats, error)

TierStats counts active traces by tier and purged tombstones. Archived and trashed traces are excluded so the numbers reflect "memory currently in use" — the signal users reason about when deciding whether to tune consolidation thresholds.

func (*Cortex) TierVotes 

func (c *Cortex) TierVotes(id string) (int, error)

Vote records a tier-preference signal for a trace. Positive delta nudges the consolidation scorer toward promotion; negative delta nudges toward demotion or keeping the trace low-tier. delta must be +1 or -1 — anything else is rejected to keep the event log unambiguous.

ActorSystem callers are rejected because voting is by definition an explicit-intent signal and the system has no intent. Agents vote on a user's behalf (e.g. "user said this really matters"); humans vote from the TUI.

Vote works on all tiers, including long-term: the refined immutability trigger (migration 009) lets tier_votes change on tier='long' rows while still blocking content and identity fields. Votes on long-term are a meaningful signal that a base-truth memory is still being referenced. TierVotes returns the current tier_votes count for a trace. Surfaced to UIs (TUI detail pane, future CLI `noema memory votes`) so users voting on a trace can see their vote's effect. A missing trace returns sql.ErrNoRows.

func (*Cortex) TraceFile 

func (c *Cortex) TraceFile(id string, archived bool) string

TraceFile returns the absolute path to a trace's markdown file.

func (*Cortex) TracesDir 

func (c *Cortex) TracesDir() string

func (*Cortex) Trash added in v0.2.0 

func (c *Cortex) Trash(id string) error

Trash moves a trace to the trash directory for deferred deletion.

func (*Cortex) TrashDir added in v0.2.0 

func (c *Cortex) TrashDir() string

func (*Cortex) TrashFile added in v0.2.0 

func (c *Cortex) TrashFile(id string) string

TrashFile returns the path for a trace in the trash.

func (*Cortex) Unarchive 

func (c *Cortex) Unarchive(id string) error

func (*Cortex) Update 

func (c *Cortex) Update(id string) error

Update rewrites an existing trace's DB row and FTS entry from its (potentially edited) markdown file on disk.

func (*Cortex) UpdateAs 

func (c *Cortex) UpdateAs(id string, actor ReadActor) error

UpdateAs is the actor-aware counterpart to Update. It runs the regular update (which internally handles source-lock checks, event emission, FTS refresh, etc.) and bumps modify_count when the caller is ActorAgent. A failed Update short-circuits before the bump.

The bump writes to trace_usage keyed on (trace_id, local cortex ID). Long-term traces can never reach the bump step: the immutability trigger aborts the inner Update transaction first.

func (*Cortex) Vote 

func (c *Cortex) Vote(id string, delta int, actor ReadActor) error

type DistilledTraceSpec 

type DistilledTraceSpec struct {
	Title              string
	Body               string
	Tags               []string
	Author             string
	SourceIDs          []string
	ModelName          string
	ModelTierProfile   string
	CohesionConfidence float64
}

DistilledTraceSpec is the payload record_consolidation_result (Phase 9 MCP tool) hands to Cortex. The fields mirror the event data schema in docs/plans/consolidation-plan.md §9 so event replay on a federation peer can reconstruct the same distillation.

type FederationConfig added in v0.3.0 

type FederationConfig struct {
	Mode     string      `yaml:"mode,omitempty"` // sync | publish | subscribe
	Peers    []PeerEntry `yaml:"peers,omitempty"`
	Interval string      `yaml:"interval,omitempty"` // e.g. "30s", "1m"
}

FederationConfig holds peer declarations for cortex.md.

func (*FederationConfig) EffectiveMode added in v0.5.0 

func (fc *FederationConfig) EffectiveMode() string

EffectiveMode returns the configured federation mode, defaulting to "sync".

type GraduationConfig 

type GraduationConfig struct {
	// Enabled defaults to true when the parent consolidation block is
	// enabled. Set to false to pause automatic mid→long graduation
	// while leaving the short→mid promoter and LLM distillation
	// pipeline running.
	Enabled *bool `yaml:"enabled,omitempty"`

	// MinAgeDays is the minimum age (in days) before a mid-tier trace
	// can graduate. Zero defaults to 14 — two weeks of stability
	// signals the trace hasn't been a transient idea.
	MinAgeDays int `yaml:"min_age_days,omitempty"`

	// MinReadCount is the minimum read_count required for graduation.
	// Zero defaults to 3 — at least a few deliberate reads to prove
	// the trace carries ongoing value.
	MinReadCount int `yaml:"min_read_count,omitempty"`

	// RequireUnmodified defaults to true. When true, a trace graduates
	// only if modify_count == 0 since creation; any edit resets the
	// stability clock. Flip to false in cortexes where edits are
	// routine and don't indicate churn.
	RequireUnmodified *bool `yaml:"require_unmodified,omitempty"`
}

GraduationConfig controls the mid→long promotion heuristic. A trace graduates when every criterion is simultaneously true — a simple AND-gate rather than a blended score because the long tier is meant to be slow-moving and auditable. Bumping any threshold makes graduation stricter; lowering any makes it looser.

func (*GraduationConfig) EffectiveEnabled 

func (gc *GraduationConfig) EffectiveEnabled() bool

EffectiveEnabled returns true when graduation should run. Defaults to true on a nil config so a cortex that enables consolidation without an explicit graduation block gets the tier model completed.

func (*GraduationConfig) EffectiveMinAge 

func (gc *GraduationConfig) EffectiveMinAge() time.Duration

EffectiveMinAge returns the configured minimum age, defaulting to 14 days. The return type is time.Duration for direct use in candidate queries; the YAML knob is days for human readability.

func (*GraduationConfig) EffectiveMinReadCount 

func (gc *GraduationConfig) EffectiveMinReadCount() int

EffectiveMinReadCount returns the configured floor, defaulting to 3.

func (*GraduationConfig) EffectiveRequireUnmodified 

func (gc *GraduationConfig) EffectiveRequireUnmodified() bool

EffectiveRequireUnmodified returns the stability-gate flag, defaulting to true on a nil config.

type ListOptions 

type ListOptions struct {
	Type     string
	Author   string
	Tag      string
	Origin   string
	Tiers    []string // restrict to these memory tiers; empty means no tier filter
	Archived bool     // only archived (excludes trashed)
	Trashed  bool     // only trashed
	All      bool     // active + archived (excludes trashed)
}

type Manifest 

type Manifest struct {
	ID            string               `yaml:"id,omitempty"`
	Name          string               `yaml:"name"`
	Purpose       string               `yaml:"purpose,omitempty"`
	Owner         string               `yaml:"owner,omitempty"`
	Created       string               `yaml:"created"`
	Version       int                  `yaml:"version"`
	Access        *AccessConfig        `yaml:"access,omitempty"`
	Federation    *FederationConfig    `yaml:"federation,omitempty"`
	Watch         *WatchConfig         `yaml:"watch,omitempty"`
	Consolidation *ConsolidationConfig `yaml:"consolidation,omitempty"`

	// Body is the free-form markdown content that follows the YAML
	// frontmatter in cortex.md. Never serialized through YAML; populated
	// by ReadManifest and consumed by WriteManifest to preserve any
	// prose the user keeps below the manifest.
	Body string `yaml:"-"`
}

Manifest is the cortex.md file at the root of each Cortex.

func Create 

func Create(name, dir string) (Manifest, error)

Create initialises a new Cortex on disk and registers it. dir is the parent directory; the cortex is created as dir/<name>/. Returns the freshly written manifest so callers can surface the new cortex's ULID to the user (see `noema init`).

func ReadManifest added in v0.2.2 

func ReadManifest(dir string) (Manifest, error)

ReadManifest parses the cortex.md manifest in the given cortex directory. cortex.md is a markdown file with YAML frontmatter: a `---` fence, the manifest YAML, a closing `---` fence, and an optional free-form body. For back-compat with cortexes written before the framing change, a file that does not begin with `---` is parsed as whole-file YAML.

func (Manifest) PeerLabelCollidesWithSelf added in v0.3.0 

func (m Manifest) PeerLabelCollidesWithSelf(peerLabel string) bool

PeerLabelCollidesWithSelf reports whether the proposed peer label is the same as this cortex's own name. This is a federation safety guardrail: even after the cortex-id migration (docs/design/cortex-uuid-plan.md), a label collision is confusing in display surfaces and should be rejected at config time.

func (Manifest) ValidateFederation added in v0.5.0 

func (m Manifest) ValidateFederation() error

ValidateFederation checks that the federation mode and per-peer modes in the manifest are recognized values. Returns nil when there is no federation block or when all values are valid.

type PeerEntry added in v0.3.0 

type PeerEntry struct {
	Name     string `yaml:"name"`
	Endpoint string `yaml:"endpoint"`
	CA       string `yaml:"ca,omitempty"`   // path to CA cert for TLS verification
	Mode     string `yaml:"mode,omitempty"` // sync | paused
}

PeerEntry is a peer declared in cortex.md.

func (PeerEntry) EffectiveMode added in v0.5.0 

func (pe PeerEntry) EffectiveMode() string

EffectiveMode returns the configured peer mode, defaulting to "sync".

type PromotionCandidate 

type PromotionCandidate struct {
	ID               string
	Tier             string
	Type             string
	ReadCount        int
	ModifyCount      int
	TierVotes        int
	DerivedFromCount int
	CreatedAt        string
}

PromotionCandidate carries the signals the consolidation scorer reads when deciding whether to promote a trace. See docs/plans/consolidation-plan.md §5 in the Noema-design repo for how these feed into the blended scoring function.

type ReadActor 

type ReadActor int

ReadActor identifies who initiated a Cortex read or update. It gates the memory-tier usage counters (read_count, modify_count, last_read_at): only ActorAgent bumps them, because only agent consumption is a meaningful signal for the consolidation-scoring function in later phases. TUI browsing, interactive CLI inspection, watcher reconciliation, federation replay, and the consolidation agent's own candidate fetch are all deliberately excluded so the signal stays clean.

The design rationale — and the reason the "system" slot matters — is in docs/plans/consolidation-plan.md §3 in the Noema-design repo. 

const (
	// ActorAgent marks reads/updates initiated by an AI agent via MCP
	// (stdio or HTTP). These are the only operations that bump counters.
	ActorAgent ReadActor = iota

	// ActorHuman marks reads/updates initiated by a human operator,
	// through the TUI or interactive CLI. These do not bump counters —
	// human browsing inflates signal without reflecting trace usefulness.
	ActorHuman

	// ActorSystem marks internal operations: the filesystem watcher
	// reconciling external edits, federation event replay, the
	// consolidation agent reading candidates for clustering. These
	// must not bump counters — otherwise evaluating a trace for
	// promotion would inflate its own inputs, a closed feedback loop.
	ActorSystem
)

type Row 

type Row struct {
	ID           string
	Title        string
	Type         string
	Tier         string
	Author       string
	Origin       string
	Tags         []string
	DerivedFrom  []string
	ArchivedAt   string
	TrashedAt    string
	CreatedAt    string
	UpdatedAt    string
	ContentHash  string
	SourceLocked bool
	SourceHash   string
}

Row is a DB row joined with tags, returned by list/search operations.

type SyncOptions added in v0.3.0 

type SyncOptions struct {
	// Recover, when true, attempts to rebuild missing files for orphaned DB
	// rows from the local event log. Off by default so manual `rm` of a trace
	// file remains a valid way to mark it for cleanup.
	Recover bool
}

SyncOptions controls optional Sync behaviors.

type SyncResult added in v0.2.2 

type SyncResult struct {
	Added     int // files found on disk but not in DB
	Updated   int // files found on disk and already in DB (re-synced)
	Recovered int // orphaned DB rows whose files were rebuilt from the event log
	Orphaned  int // IDs in DB with no corresponding file on disk (after recovery)
}

SyncResult summarises what Sync found.

type TierStats 

type TierStats struct {
	Short  int
	Mid    int
	Long   int
	Purged int
}

TierStats reports how many traces sit in each memory tier plus the count of purged (tombstoned) rows. Phase 6 MVP data source for the `noema memory stats` CLI; later phases expand the dashboard with consolidation quality metrics from the event log.

type WatchConfig added in v0.9.0 

type WatchConfig struct {
	// Enabled is a pointer so an omitted field means "default on."
	// Explicitly setting `enabled: false` disables the watcher.
	Enabled *bool `yaml:"enabled,omitempty"`
	// DebounceMs is the per-path debounce window in milliseconds.
	// Zero means use the default (300ms).
	DebounceMs int `yaml:"debounce_ms,omitempty"`
	// AutoOnboard controls what happens when a new file is dropped into
	// traces/ without conforming frontmatter (no id, empty id, invalid
	// id format, or filename mismatch). Nil / unset defaults to true:
	// the watcher rescues the file by synthesizing valid frontmatter,
	// generating an ID from the title or filename, and renaming the
	// file to match. Makes Obsidian Web Clipper, Drafts, and shortcut
	// macros work without teaching users Noema's filename-ID rules.
	// Set to false to restore the prior skip-and-log behaviour.
	AutoOnboard *bool `yaml:"auto_onboard,omitempty"`
}

WatchConfig controls the filesystem watcher that turns external edits to trace files (Obsidian, VS Code, Finder, etc.) into first-class mutation events. When unset, the watcher is enabled by default during `noema serve --transport http`.

func (*WatchConfig) AutoOnboardEnabled added in v0.9.1 

func (wc *WatchConfig) AutoOnboardEnabled() bool

AutoOnboardEnabled reports whether the watcher should rescue non- conforming files dropped into traces/. Nil manifest or nil AutoOnboard pointer both mean "on" so users get the graceful behaviour by default.

func (*WatchConfig) EffectiveDebounce added in v0.9.0 

func (wc *WatchConfig) EffectiveDebounce() time.Duration

EffectiveDebounce returns the configured debounce duration, defaulting to 300ms when unset.

func (*WatchConfig) WatchEnabled added in v0.9.0 

func (wc *WatchConfig) WatchEnabled() bool

WatchEnabled reports whether the watcher should run given the parsed manifest value. Nil manifest or nil Enabled pointer both mean "on."

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.