importer

package
v0.3.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 15, 2026 License: AGPL-3.0 Imports: 23 Imported by: 0

Documentation

Overview

Package importer bulk-loads memories exported from other memory systems (agentmemory, mem0, mnemory) or memini's own format. The local backend embeds content and writes to the store, preserving source IDs and timestamps; the remote backend POSTs to a running memini's REST API.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Sources

func Sources() []string

Sources lists the supported import sources, sorted.

Types

type DedupResult added in v0.0.11

type DedupResult struct {
	Namespaces    int `json:"namespaces"`
	ClustersFound int `json:"clusters_found"`
	Tombstoned    int `json:"tombstoned"`
}

DedupResult is the subset of POST /v1/dedup's response the importer reports after an auto-dedup pass.

type Importer

type Importer struct {
	// contains filtered or unexported fields
}

Importer bulk-loads records via a configured backend.

func NewLocal

func NewLocal(st store.Store, e embed.Embedder) *Importer

NewLocal builds an Importer that embeds and writes directly to the store.

func NewRemote

func NewRemote(c *RemoteClient) *Importer

NewRemote builds an Importer that POSTs records to a remote memini server.

func (*Importer) Import

func (im *Importer) Import(ctx context.Context, recs []Record, opts Options) (Report, error)

Import writes records, skipping those that fail the quality gates (empty or below MinContentLen/MinImportance) and continuing past per-record failures (collected in the Report).

type Options

type Options struct {
	// DefaultNamespace scopes records whose source carried no namespace.
	DefaultNamespace string
	// ForceNamespace, when set, overrides every record's namespace (the explicit
	// "merge everything into one pool" opt-in). The original namespace is
	// preserved in metadata["import_source_namespace"] so the merge is reversible
	// with `memini namespace split`.
	ForceNamespace string
	// Source names the export format, used to stamp a provenance tag
	// (import:<source>:<date>) and to seed deterministic record IDs.
	Source Source
	// DefaultImportance is applied to records whose source carried no importance
	// (reported as 0), so bulk imports rank below curated, source-scored
	// memories. 0 disables the floor.
	DefaultImportance float64
	// Confidence overrides the seed corroboration for durable imported facts
	// (e.g. a trusted re-import). nil uses the low default import seed.
	Confidence *float64
	// SkipExisting checks the store for a record's ID before writing and counts
	// it as a duplicate instead of clobbering an existing memory's access
	// counters. Combined with deterministic IDs this makes re-imports idempotent.
	// Only honored by the local backend.
	SkipExisting bool
	// DryRun resolves namespaces and runs the quality gates but writes nothing;
	// the Report's namespace histogram still reflects where records would land.
	DryRun bool
	// BatchSize bounds how many records are written per batch.
	BatchSize int
	// OnProgress is called after each batch with (processed, total).
	// It may be nil.
	OnProgress func(done, total int)
	// MinContentLen drops records whose trimmed content is shorter than this,
	// filtering out stubs from a low-quality bulk import. 0 disables the gate.
	MinContentLen int
	// MinImportance drops records below this importance. 0 disables the gate;
	// note sources that carry no importance report 0, so any positive value
	// skips them.
	MinImportance float64
}

Options tune an import run.

type Record

type Record struct {
	ID         string
	Namespace  string
	Tier       memory.Tier
	Content    string
	Summary    string
	Tags       []string
	Metadata   map[string]any
	Importance float64
	CreatedAt  time.Time  // zero -> import time
	UpdatedAt  time.Time  // zero -> CreatedAt
	ExpiresAt  *time.Time // nil -> tier default measured from CreatedAt
}

Record is the portable, source-agnostic shape every adapter produces.

func ExtractTyped added in v0.2.5

func ExtractTyped(recs []Record) []Record

ExtractTyped derives durable memories from conversation records (the claude-code episodic exchanges): decisions and problems as semantic facts, preferences as procedural how-to. Each extraction keeps the source's namespace and timestamp, is tagged with its kind, and gets a content-addressed ID via finalizeRecords so re-imports stay idempotent. The originals are left untouched — callers append the result.

func LoadClaudeCode added in v0.0.4

func LoadClaudeCode(path string) (recs []Record, warns []string, err error)

LoadClaudeCode reads a single transcript .jsonl, or walks a directory (a project dir, or ~/.claude/projects) for *.jsonl files. Per-file parse errors become warnings rather than aborting the whole walk.

func LoadClaudeCodeWithProgress added in v0.0.8

func LoadClaudeCodeWithProgress(path string, onProgress func(done, total int)) (recs []Record, warns []string, err error)

LoadClaudeCodeWithProgress is like LoadClaudeCode but accepts an optional progress callback that fires after each file is parsed.

func Parse

func Parse(src Source, data []byte) ([]Record, error)

Parse converts a source export (raw JSON) into portable Records.

type RemoteClient

type RemoteClient struct {
	// contains filtered or unexported fields
}

RemoteClient writes records to a running memini via its REST API (POST /v1/memories).

func NewRemoteClient

func NewRemoteClient(baseURL, token, nsHeader string) *RemoteClient

NewRemoteClient targets a memini server at baseURL (e.g. https://memini.example.com), authenticating with token (optional) and scoping each record via nsHeader.

func (*RemoteClient) Dedup added in v0.0.11

func (c *RemoteClient) Dedup(ctx context.Context, namespace string, similarity float64) (DedupResult, error)

Dedup runs a namespace-scoped vector-cluster dedup pass on the remote server (POST /v1/dedup). similarity <= 0 lets the server pick its default.

type Report

type Report struct {
	Total      int            `json:"total"`
	Imported   int            `json:"imported"`
	Skipped    int            `json:"skipped"`              // dropped before write (failed a quality gate)
	Duplicates int            `json:"duplicates,omitempty"` // already present, left untouched (SkipExisting)
	Namespaces map[string]int `json:"namespaces,omitempty"` // records resolved per destination namespace
	Errors     []string       `json:"errors,omitempty"`
}

Report summarizes an import run.

type Source

type Source string

Source identifies the export format of the data being imported.

const (
	// SourceMemini is memini's own export shape (a JSON array of Records).
	SourceMemini Source = "memini"
	// SourceAgentMemory is rohitg00/agentmemory's export bundle.
	SourceAgentMemory Source = "agentmemory"
	// SourceMem0 is mem0ai/mem0's get_all / export output.
	SourceMem0 Source = "mem0"
	// SourceMnemory is fpytloun/mnemory's export output.
	SourceMnemory Source = "mnemory"
	// SourceClaudeCode is a Claude Code session transcript (JSONL), reconstructed
	// into per-exchange episodic memories.
	SourceClaudeCode Source = "claude-code"
)

type TypedKind added in v0.2.5

type TypedKind string

TypedKind labels an extracted memory; it doubles as the memory's first tag.

const (
	KindDecision   TypedKind = "decision"
	KindPreference TypedKind = "preference"
	KindProblem    TypedKind = "problem"
)

func (TypedKind) Tier added in v0.3.6

func (k TypedKind) Tier() memory.Tier

Tier maps an extracted kind to its memory tier: a preference is a how-to rule, so it's procedural; decisions and problems are durable facts, so semantic.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL