maintenance

package
v0.7.27 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 12, 2026 License: AGPL-3.0 Imports: 14 Imported by: 0

Documentation

Overview

Package maintenance keeps the store healthy: a background sweeper purges expired memories and bounds short-term capacity, and fsck additionally audits live memories for duplicate (poisoning) clusters.

Index

Constants

View Source
const HandoffTag = "handoff"

HandoffTag marks a memory as a session handoff: the full fresh-session prompt a previous session wrote for the next one. Like a pin it is exempt from retro-tiering demotion, but it is otherwise the opposite of a pin — it is held OUT of recall and out of the briefing's content sections, and only a one-line pointer to it is injected, because the content is a 100-300 line prompt meant to be pulled deliberately, never sprayed into every session.

View Source
const PinnedTag = "pinned"

PinnedTag marks a memory as pinned: exempt from retro-tiering demotion and always surfaced in a session briefing.

Variables

View Source
var DefaultSplitKeys = []string{"import_source_namespace", "user_id", "agent_id", "run_id", "project"}

DefaultSplitKeys are the metadata keys, in priority order, that Split groups a namespace by. import_source_namespace is stamped by the importer when a merge discarded the source namespace, so it recovers a botched `--merge-into` import exactly; the rest cover the scope fields the mem0/agentmemory/mnemory adapters preserve (user_id/agent_id/run_id/project).

Functions

func AssessImportanceBackfill added in v0.7.18

func AssessImportanceBackfill(
	ctx context.Context, st store.Store, a llm.ImportanceAssessor, opts AssessOptions, now time.Time,
) (int, error)

AssessImportanceBackfill stamps LLM-assessed importance onto durable memories that never received one, and returns how many rows it assessed.

A row is a candidate only when it has no assessment yet, is older than MinAge, and still carries exactly its tier's seed importance — a memory whose importance differs from the seed was set deliberately by whoever wrote it and is never second-guessed. (A user who happens to set exactly the seed value is a knowingly accepted false positive: the assessment would then replace that number with the model's, but ranking read the identical value beforehand, so nothing the user chose is lost in practice.) Candidates are assessed oldest-first and capped at MaxPerRun, so a large backlog drains predictably over successive passes.

The pass is deliberately forgiving of a flaky LLM. If the very first batch fails the whole pass is abandoned with a single warning — the provider is most likely down, and hammering it with the remaining batches would only multiply the noise. A later batch failing is logged and skipped, keeping the scores already written. Either way the unassessed rows stay NULL and are picked up again next pass, as does any row the model explicitly declined to rate. Because "LLM is unavailable right now" is the expected state rather than a fault, that abandoned pass reports (0, nil): the warning is the report, and returning an error would make callers log it twice.

func AssessImportancePreview added in v0.7.19

func AssessImportancePreview(
	ctx context.Context, st store.Store, opts AssessOptions, now time.Time,
) (int, error)

assessCandidates collects the rows eligible for assessment across every namespace, oldest-first and capped at opts.MaxPerRun. AssessImportancePreview reports how many durable memories AssessImportanceBackfill would send to the assessor, without calling it. It exists because the sweep's cost is per-row LLM spend: an operator running the backfill by hand needs the size of the bill before agreeing to it, and a preview that itself called the model would defeat the purpose. Candidate selection is shared with the real pass, so the count is exact rather than an estimate — subject only to rows aging past MinAge between the two calls.

func DemoteStale added in v0.0.11

func DemoteStale(ctx context.Context, st store.Store, olderThan, now time.Time, report func(fromTier string)) (int, error)

DemoteStale demotes durable (semantic/procedural) memories that have never been recalled (AccessCount 0), were last updated before olderThan, and are neither highly important (>= 0.75) nor pinned, down to the episodic tier — giving them the episodic TTL. Unused "durable" debris (e.g. a low-quality bulk import, or a default-importance fact that proved useless) then ages out on its own, while anything recalled even once is reinforced and kept. The mirror image of promotion. Returns the count demoted. report, when non-nil, is called once per demoted memory with the tier it was demoted FROM — the sweeper wires it to the memini_demoted_total counter so demotion volume is observable rather than log-only.

func EnforceShortTermCap

func EnforceShortTermCap(ctx context.Context, st store.Store, cap int, now time.Time) (int, error)

EnforceShortTermCap evicts the lowest-retention short-term memories in each namespace that holds more than cap of them. cap <= 0 disables it. Returns the number evicted.

func ForgetByTag added in v0.0.11

func ForgetByTag(ctx context.Context, st store.Store, namespace, tag string) (int64, error)

ForgetByTag deletes every memory in a namespace carrying tag, including superseded and expired ones, and returns the count deleted. With the import provenance tag (import:<source>:<date>) this undoes a single bulk import.

func PruneEvents added in v0.6.8

func PruneEvents(ctx context.Context, st store.Store, now time.Time, retention time.Duration, maxRows int) (int64, error)

PruneEvents trims the activity log to a retention window and a row cap. A retention of 0 prunes nothing by age; a cap of 0 applies no cap. Returns the number of rows deleted, and 0 against a driver with no activity log (the same degrade-gracefully type assertion the log's writers use).

func PurgeExpired

func PurgeExpired(ctx context.Context, st store.Store, now time.Time) (int, error)

PurgeExpired deletes memories whose TTL has passed as of now, in batches, and returns the number removed.

func PurgeTombstones added in v0.0.11

func PurgeTombstones(ctx context.Context, st store.Store, olderThan time.Time) (int, error)

PurgeTombstones hard-deletes superseded (tombstoned) memories last updated before olderThan, reclaiming the storage and vector-index space they occupy. Tombstones are already excluded from default recall, so this never changes those results; it only frees space and bounds how far back time-travel (as_of) recall can reach. Returns the count deleted.

Types

type AssessJob added in v0.7.18

type AssessJob struct {
	// contains filtered or unexported fields
}

AssessJob is a periodic LLM importance-backfill pass. With interval <= 0, Run is a no-op (the function returns immediately).

func NewAssessJob added in v0.7.18

func NewAssessJob(st store.Store, a llm.ImportanceAssessor, log *slog.Logger,
	interval time.Duration, opts AssessOptions) *AssessJob

NewAssessJob builds an AssessJob that calls AssessImportanceBackfill(opts) every interval. interval <= 0 disables the job, as does a nil assessor.

func (*AssessJob) Run added in v0.7.18

func (j *AssessJob) Run(ctx context.Context)

Run loops on a ticker until ctx is cancelled. It runs one pass immediately and again on every tick. It is a no-op if the job was built with interval <= 0 or without an assessor.

type AssessOptions added in v0.7.18

type AssessOptions struct {
	// Batch is how many memory contents go into a single LLM call. 0 falls back
	// to defaultAssessBatch.
	Batch int
	// MaxPerRun caps the rows assessed by one pass, bounding its LLM spend.
	// 0 falls back to defaultAssessMaxPerRun.
	MaxPerRun int
	// MinAge skips memories younger than this, so the sweep never races the
	// write path's own assessment (a fresh write is rated inline by the
	// distill/consolidate call). 0 falls back to defaultAssessMinAge.
	MinAge time.Duration
	// Log receives progress messages; nil falls back to slog.Default().
	Log *slog.Logger
}

AssessOptions configures one importance-backfill pass.

type BackfillConfidenceReport added in v0.0.12

type BackfillConfidenceReport struct {
	Inspected int `json:"inspected"`
	Seeded    int `json:"seeded"`
	Skipped   int `json:"skipped"`
}

func BackfillConfidence added in v0.0.12

func BackfillConfidence(ctx context.Context, st store.Store, now time.Time) (BackfillConfidenceReport, error)

func BackfillConfidencePreview added in v0.0.12

func BackfillConfidencePreview(ctx context.Context, st store.Store, now time.Time) (BackfillConfidenceReport, error)

type ClusterAction added in v0.0.8

type ClusterAction struct {
	RepresentativeID string   `json:"representative_id"`
	TombstonedIDs    []string `json:"tombstoned_ids"`
	Size             int      `json:"size"`
}

ClusterAction describes one near-duplicate cluster found by a pass and the representative selection the pass would commit.

type DedupJob added in v0.0.8

type DedupJob struct {
	// contains filtered or unexported fields
}

DedupJob is a periodic vector-cluster dedup pass. With interval <= 0, Run is a no-op (the function returns immediately).

func NewDedupJob added in v0.0.8

func NewDedupJob(st store.Store, emb embed.Embedder, m store.Metrics, log *slog.Logger,
	interval time.Duration, opts DedupOptions) *DedupJob

NewDedupJob builds a DedupJob that calls Dedup(opts) every interval. interval <= 0 disables the job.

func (*DedupJob) Run added in v0.0.8

func (d *DedupJob) Run(ctx context.Context)

Run loops on a ticker until ctx is cancelled. It runs one pass immediately and again on every tick. It is a no-op if the job was built with interval <= 0.

type DedupOptions added in v0.0.8

type DedupOptions struct {
	// Similarity is the minimum cosine-like score (the store's vector
	// distance-to-score mapping) for two memories to join a cluster.
	// 0 falls back to defaultDedupSimilarity. Negative disables dedup and
	// the call returns an empty report.
	Similarity float64
	// MinClusterSize is the smallest cluster acted on. A pair of near-
	// duplicates below this is left alone. 0 falls back to
	// defaultDedupMinClusterSize.
	MinClusterSize int
	// Tiers restricts the pass to these tiers; nil/empty means all tiers.
	Tiers []memory.Tier
	// Namespaces restricts the pass to these namespaces; nil/empty means every
	// namespace. Clusters never span namespaces, so scoping the pass to one
	// (the post-import case) is both cheaper and avoids touching other namespaces.
	Namespaces []string
	// NeighboursPerAnchor bounds the per-anchor vector-search fan-out. Larger
	// values tighten clusters at higher vector-search cost. 0 falls back to
	// defaultDedupNeighboursAnchor.
	NeighboursPerAnchor int
	// DryRun reports what would be done without tombstoning anything.
	DryRun bool
	// Now is the instant retention scoring and expiry filtering are evaluated
	// at. Zero means time.Now().UTC().
	Now time.Time
	// Log receives progress messages; nil falls back to slog.Default().
	Log *slog.Logger
	// Merger, when set, merges each cluster's content into a single
	// comprehensive memory before tombstoning the duplicates. When nil, the
	// representative keeps its original content (existing behavior).
	Merger llm.Merger
	// MaxMergeClusterSize bounds the cluster size sent to the LLM (larger
	// clusters would exceed token limits). Clusters larger than this are
	// merged in chunks or skipped. 0 defaults to 10.
	MaxMergeClusterSize int
}

DedupOptions configures one dedup pass.

type DedupReport added in v0.0.8

type DedupReport struct {
	Namespaces    int             `json:"namespaces"`
	MemoriesSeen  int             `json:"memories_seen"`
	ClustersFound int             `json:"clusters_found"`
	Tombstoned    int             `json:"tombstoned"`
	DryRun        bool            `json:"dry_run"`
	Actions       []ClusterAction `json:"actions,omitempty"`
}

DedupReport summarizes one dedup pass.

func Dedup added in v0.0.8

func Dedup(ctx context.Context, st store.Store, emb embed.Embedder, opts DedupOptions) (DedupReport, error)

Dedup clusters live memories per namespace by embedding similarity and tombstones the lower-scored members of each cluster, pointing them at the cluster's representative. The representative is the member with the highest RetentionScore (importance × access × recency), tie-broken by updated-at and then created-at so re-imports don't shadow the original.

Tombstoning is reversible: SetSuperseded excludes the duplicates from default search results but keeps them in storage. To free space, follow up with a store-level GC (not implemented here). The action is symmetric with consolidation's supersede, so the read path needs no changes.

Dedup is O(n · vector_search(n)) per namespace; with the embedder cache warm (the typical post-import case) the batched embed is near-free. For very large corpora, NeighboursPerAnchor bounds the union-find fan-out; the cluster of a memory can only ever be as wide as that fan-out allows.

st is required; emb is required unless Similarity <= 0 (in which case the pass is a no-op). opts.Similarity <= 0 short-circuits to an empty report.

type ReembedReport added in v0.3.9

type ReembedReport struct {
	Namespaces int `json:"namespaces"`
	Total      int `json:"total"`
	Reembedded int `json:"reembedded"`
}

ReembedReport summarizes a re-embedding pass.

func Reembed added in v0.3.9

func Reembed(
	ctx context.Context, st store.Store, emb embed.Embedder,
	namespaces []string, batchSize int, onProgress func(done, total int),
) (ReembedReport, error)

Reembed recomputes and rewrites every memory's vector with emb, migrating a store to a new embedding model in place. It embeds memory content (the text the write path embeds), including expired and superseded rows so the vector index stays consistent. Dims can't change here (the store rejects mismatched widths). namespaces restricts the pass when non-empty. Callers that changed the model should record it with store.SetEmbedModel afterwards.

type RenamespaceReport added in v0.0.11

type RenamespaceReport struct {
	Moved   int            `json:"moved"`
	Targets map[string]int `json:"targets,omitempty"` // memories moved into each destination
	Skipped int            `json:"skipped"`           // left in place (no grouping key, or already in place)
	DryRun  bool           `json:"dry_run"`
}

RenamespaceReport summarizes a Move or Split.

func Move added in v0.0.11

func Move(ctx context.Context, st store.Store, fromNS, toNS string, dryRun bool) (RenamespaceReport, error)

Move relocates every memory in fromNS to toNS. A no-op when fromNS == toNS.

func Split added in v0.0.11

func Split(ctx context.Context, st store.Store, fromNS string, byKeys []string, dryRun bool) (RenamespaceReport, error)

Split regroups a namespace by metadata, moving each record to the namespace named by the first of byKeys it carries. Records with no grouping key (or whose key equals fromNS) stay put and are counted as skipped. Pass nil byKeys to use DefaultSplitKeys. This is the recovery path for a store whose imports were collapsed into one pool.

type RepairReport added in v0.4.12

type RepairReport struct {
	Restored   int  `json:"restored"`
	Namespaces int  `json:"namespaces"`
	DryRun     bool `json:"dry_run"`
}

RepairReport summarizes one supersession-repair pass.

func RepairSupersession added in v0.4.12

func RepairSupersession(ctx context.Context, st store.Store, namespaces []string, dryRun bool, log *slog.Logger) (RepairReport, error)

RepairSupersession restores memories stranded by a broken supersession chain: a tombstoned row whose superseded_by chain never reaches a live memory (its representative was itself superseded, or deleted) is cleared back to live, so every cluster keeps a recallable head. A genuine duplicate is re-collapsed by the next dedup pass. Empty namespaces means every namespace; a per-namespace error is logged and skipped so one bad namespace can't abort the rest.

type Report

type Report struct {
	ExpiredPurged    int        `json:"expired_purged"`
	ShortTermEvicted int        `json:"short_term_evicted"`
	Namespaces       int        `json:"namespaces"`
	DuplicateGroups  [][]string `json:"duplicate_groups,omitempty"`
}

Report summarizes a consistency sweep.

func Fsck

func Fsck(ctx context.Context, st store.Store, cap int, now time.Time) (Report, error)

Fsck purges expired memories, enforces the short-term cap, and audits live memories for duplicate clusters (same normalized content) as a poisoning backstop. Duplicates are reported, not auto-deleted.

type ScopeMerge added in v0.6.6

type ScopeMerge struct {
	From            string `json:"from"`
	To              string `json:"to"`
	Moved           int    `json:"moved"`
	DedupClusters   int    `json:"dedup_clusters,omitempty"`
	DedupTombstoned int    `json:"dedup_tombstoned,omitempty"`
}

ScopeMerge reports one `<t>/_shared` -> `<t>` merge and the dedup pass run against the target afterward.

type ScopesOptions added in v0.6.6

type ScopesOptions struct {
	// DryRun reports what would move (and what would be deduped) without
	// writing anything.
	DryRun bool
	// Embedder powers the post-merge dedup pass (gap G14). Required whenever
	// a merge actually happens and DryRun is false; unused in dry-run mode,
	// where no dedup pass runs.
	Embedder embed.Embedder
	// Dedup configures the post-merge dedup pass run against each merge
	// target. Namespaces and DryRun are overridden per call; the caller
	// controls Similarity/MinClusterSize/Tiers/etc. Zero value uses Dedup's
	// own defaults.
	Dedup DedupOptions
}

ScopesOptions configures one MigrateScopes pass.

type ScopesReport added in v0.6.6

type ScopesReport struct {
	Merges []ScopeMerge `json:"merges,omitempty"`
	// BareShared lists top-level namespaces literally named "_shared" (no
	// tenant prefix): there is no parent to merge into, so these are left
	// untouched. Reported rather than silently skipped since the operator
	// likely wants it repointed by hand (a home namespace, or a link source).
	BareShared []string `json:"bare_shared,omitempty"`
	// GlobalNamespaceEnv is the raw value of MEMINI_GLOBAL_NAMESPACE when set
	// at run time. Its presence is only ever reported, never acted on: the
	// caller (CLI) prints adoption instructions instead of a silent rewrite.
	GlobalNamespaceEnv string `json:"global_namespace_env,omitempty"`
	DryRun             bool   `json:"dry_run"`
}

ScopesReport summarizes one MigrateScopes pass.

func MigrateScopes added in v0.6.6

func MigrateScopes(ctx context.Context, st store.Store, opts ScopesOptions) (ScopesReport, error)

MigrateScopes moves the old shared-scope model's data into the new ancestor-cascade shape: every namespace literally named "<prefix>/_shared" is merged into "<prefix>" via Move (which already rewrites link endpoints), followed by a dedup pass scoped to the target namespace (gap G14 — Move relocates by unique ID with no content dedup, so a merge can duplicate facts already present in the target). A namespace named just "_shared" (no prefix) has no parent to merge into and is left untouched, reported in BareShared. Idempotent: once no "<prefix>/_shared" namespace holds any memories, a re-run finds nothing to do.

type ScrubReport added in v0.2.5

type ScrubReport struct {
	LifecycleNoise  int `json:"lifecycle_noise"`
	ExactDuplicates int `json:"exact_duplicates"`
	Namespaces      int `json:"namespaces"`
}

ScrubReport summarizes a content-quality scrub. Counts are the number of memories that were (or, in a preview, would be) deleted in each category.

func Scrub added in v0.2.5

func Scrub(ctx context.Context, st store.Store, apply bool) (ScrubReport, error)

Scrub removes content-level junk that the namespace-oriented doctor fix and the embedding-similarity dedup pass both miss: session-lifecycle markers ("Session ended", "Stop checkpoint") and exact-duplicate memories (identical normalized content within a namespace, keeping the oldest). It previews when apply is false, returning the counts that would be removed without mutating the store. Live memories only — tombstones are left alone (already excluded from recall and reversible). Returns the per-category report.

func (ScrubReport) Total added in v0.2.5

func (r ScrubReport) Total() int

Total returns the number of memories removed across all categories.

type Sweeper

type Sweeper struct {
	// contains filtered or unexported fields
}

Sweeper periodically purges expired memories, enforces the short-term cap, and (optionally) garbage-collects old tombstones.

func NewSweeper

func NewSweeper(st store.Store, log *slog.Logger, cfg SweeperConfig) *Sweeper

NewSweeper builds a sweeper that runs every cfg.Interval.

func (*Sweeper) Run

func (s *Sweeper) Run(ctx context.Context)

Run sweeps on a ticker until ctx is cancelled. It runs one sweep immediately. It is a no-op when Interval <= 0 (time.NewTicker panics on a non-positive duration); config validation rejects that, but guard here too so a misconfigured interval cannot crash the sweeper goroutine.

type SweeperConfig added in v0.0.11

type SweeperConfig struct {
	// Interval is how often the sweep runs.
	Interval time.Duration
	// ShortTermCap bounds working+episodic memories per namespace; the lowest-
	// retention ones over the cap are evicted. 0 disables it.
	ShortTermCap int
	// TombstoneTTL hard-deletes superseded memories last updated before now-TTL,
	// reclaiming space. 0 disables it (tombstones are kept indefinitely but stay
	// excluded from recall).
	TombstoneTTL time.Duration
	// DemoteAfter demotes never-recalled, low-importance durable memories older
	// than this to the episodic tier so unused debris ages out. 0 disables it.
	DemoteAfter time.Duration
	// ActivityRetention drops activity-log events older than now-retention.
	// 0 disables age-based pruning.
	ActivityRetention time.Duration
	// ActivityMaxRows caps the activity log, dropping the oldest rows beyond it.
	// 0 disables the cap. With both bounds 0 the log grows without limit.
	ActivityMaxRows int
	// OnDemoted, when non-nil, is called once per memory the demote stage
	// retiers, with the tier it was demoted from (feeds memini_demoted_total).
	OnDemoted func(fromTier string)
}

SweeperConfig configures the periodic maintenance sweep.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL