backup

package
v2.4.19 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package backup implements the portable logical snapshot of the Cortex memory store: observations, graph relations (edges), and sessions, packaged as JSONL parts inside a tar.gz with a sha256-verified manifest.

This is the portable logical layer, distinct from the physical SQLite snapshot (`cortex backup <path>` → VACUUM INTO). It follows the projection/obsidian package pattern: a pure library with narrow inputs, wired to the store bundle at the CLI composition root.

Scope: local SQLite mode. Server (PostgreSQL) mode full-database backup remains the domain of pg_dump; this package's schema-projection queries are SQLite-specific and restore refuses non-v2 databases via the schema identity check.

Security invariants:

  • Only a fixed allow-list of columns is exported. Configuration secrets (API keys, DSNs, web keys, bearer tokens) never enter the projection.
  • Defense in depth: the caller may supply known secret values; Export fail-closes if any of them appears in the serialized archive.
  • Embedding vectors are NOT exported (regenerable, and potentially re-identifying); per-observation embedding state (model/dimensions) is.

Index

Constants

View Source
const (
	// Format identifies the archive family in the manifest.
	Format = "cortex-backup"

	// FormatVersion is the manifest/record layout version of this package.
	FormatVersion = 1

	// ManifestName is the manifest entry name inside the tar.gz.
	ManifestName = "manifest.json"

	// Part names (JSONL, one record per line).
	ObservationsPart = "observations.jsonl"
	EdgesPart        = "edges.jsonl"
	SessionsPart     = "sessions.jsonl"
)

Variables

View Source
var KnownEnvSecrets = []string{
	"CORTEX_EMBEDDING_API_KEY",
	"CORTEX_LLM_API_KEY",
	"CORTEX_RERANK_API_KEY",
	"CORTEX_HTTP_TOKEN",
	"CORTEX_REMOTE_TOKEN",
	"CORTEX_WEB_KEY",
	"CORTEX_SYNC_TOKEN",
	"CORTEX_POSTGRES_DSN",
	"CORTEX_TEST_POSTGRES_DSN",
	"CORTEX_TEST_POSTGRES_MIGRATION_DSN",
	"CORTEX_TEST_POSTGRES_AUTHZ_ADMIN_DSN",
}

KnownEnvSecrets is the ordered list of Cortex credential environment variables whose values must never leak into a logical backup.

Functions

This section is empty.

Types

type EdgeRecord

type EdgeRecord struct {
	ID            int64          `json:"id"`
	FromObsID     int64          `json:"from_obs_id"`
	ToObsID       int64          `json:"to_obs_id"`
	RelationType  string         `json:"relation_type"`
	Weight        float64        `json:"weight"`
	Confidence    float64        `json:"confidence"`
	Source        sql.NullString `json:"source,omitempty"`
	Reasoning     sql.NullString `json:"reasoning,omitempty"`
	ValidFrom     sql.NullString `json:"valid_from,omitempty"`
	InvalidAt     sql.NullString `json:"invalid_at,omitempty"`
	ValidUntil    sql.NullString `json:"valid_until,omitempty"`
	TenantID      sql.NullString `json:"tenant_id,omitempty"`
	WorkspaceID   sql.NullString `json:"workspace_id,omitempty"`
	EvolutionID   sql.NullInt64  `json:"evolution_id,omitempty"`
	EvolutionType string         `json:"evolution_type"`
	FactState     string         `json:"fact_state"`
	ChangeReason  sql.NullString `json:"change_reason,omitempty"`
	CreatedAt     sql.NullString `json:"created_at,omitempty"`
}

EdgeRecord is one portable graph relation row.

type EmbeddingState

type EmbeddingState struct {
	Model      string `json:"model"`
	Dimensions int    `json:"dimensions"`
}

EmbeddingState records the per-observation embedding metadata. The vector itself is intentionally NOT exported: it is regenerable and potentially re-identifying.

type ErrIncompatibleSchema

type ErrIncompatibleSchema struct {
	Backup  string
	Current string
}

ErrIncompatibleSchema reports a restore refusal caused by schema-version incompatibility (the archive is newer than the target database).

func (*ErrIncompatibleSchema) Error

func (e *ErrIncompatibleSchema) Error() string

type ErrSecretDetected

type ErrSecretDetected struct{ Name string }

ErrSecretDetected reports that a configured secret value appeared in the exported archive; the export fail-closes rather than leak it.

func (*ErrSecretDetected) Error

func (e *ErrSecretDetected) Error() string

type ErrTampered

type ErrTampered struct{ Part string }

ErrTampered reports a sha256 mismatch between manifest and part payload.

func (*ErrTampered) Error

func (e *ErrTampered) Error() string

type ExportOptions

type ExportOptions struct {
	// Out is the destination file path (a tar.gz archive).
	Out string
	// Tenant optionally restricts the export to one tenant id.
	Tenant string
	// ToolVersion is recorded in the manifest (the CLI injects its version).
	ToolVersion string
	// Secrets are credential values that must never appear in the archive
	// (defense-in-depth; see package doc).
	Secrets []Secret
	// Now overrides the manifest timestamp (tests).
	Now func() time.Time
}

Options control a logical export.

type ExportResult

type ExportResult struct {
	Path         string
	SizeBytes    int64
	Observations int
	Edges        int
	Sessions     int
	SchemaVer    string
}

ExportResult summarizes a completed export.

type Exporter

type Exporter struct {
	DB *sql.DB
}

Exporter produces logical snapshots from the SQLite memory store.

The projection deliberately reads schema-owned columns through a fixed allow-list of SELECTs rather than the domain model: tenant/workspace/owner columns and verbatim stored timestamps are part of the portable snapshot but are deliberately absent from the local domain types. All other writes in the codebase keep flowing through the store bundle; this package is a read-only dumper plus an ID-preserving restorer (see restore.go).

func (*Exporter) Export

func (e *Exporter) Export(ctx context.Context, opts ExportOptions) (*ExportResult, error)

Export writes the tar.gz archive to opts.Out.

type Manifest

type Manifest struct {
	Format        string            `json:"format"`
	FormatVersion int               `json:"format_version"`
	SchemaFamily  string            `json:"schema_family"`
	SchemaVersion string            `json:"schema_version"`
	ToolVersion   string            `json:"tool_version"`
	CreatedAt     string            `json:"created_at"`
	Source        Source            `json:"source"`
	Parts         map[string]Part   `json:"parts"`
	Notes         map[string]string `json:"notes,omitempty"`
}

Manifest is the self-describing index stored inside every archive.

type ObservationRecord

type ObservationRecord struct {
	ID             int64           `json:"id"`
	SessionID      string          `json:"session_id"`
	Type           string          `json:"type"`
	Title          string          `json:"title"`
	Content        string          `json:"content"`
	ToolName       sql.NullString  `json:"tool_name,omitempty"`
	Project        sql.NullString  `json:"project,omitempty"`
	Scope          string          `json:"scope"`
	TopicKey       sql.NullString  `json:"topic_key,omitempty"`
	Confidence     float64         `json:"confidence"`
	Source         string          `json:"source"`
	Tags           sql.NullString  `json:"tags,omitempty"`
	TenantID       sql.NullString  `json:"tenant_id,omitempty"`
	WorkspaceID    sql.NullString  `json:"workspace_id,omitempty"`
	OwnerID        sql.NullString  `json:"owner_id,omitempty"`
	CreatedAt      string          `json:"created_at"`
	UpdatedAt      string          `json:"updated_at"`
	ValidUntil     sql.NullString  `json:"valid_until,omitempty"`
	EmbeddingState *EmbeddingState `json:"embedding_state,omitempty"`
}

ObservationRecord is one portable observation row. Timestamps are carried verbatim as stored (TEXT) so a restore is byte-faithful.

type Part

type Part struct {
	Count  int    `json:"count"`
	SHA256 string `json:"sha256"`
}

Part describes one JSONL member of the archive in the manifest.

type RestoreOptions

type RestoreOptions struct {
	// From is the source archive path (tar.gz produced by Export).
	From string
	// Secrets are credential values that must never have been exported; if
	// any appears inside the archive the restore refuses (the archive was
	// produced by a non-Cortex tool or tampered with).
	Secrets []Secret
}

Options control a logical restore.

type RestoreResult

type RestoreResult struct {
	ObservationsRestored int
	ObservationsSkipped  int
	EdgesRestored        int
	EdgesSkipped         int
	SessionsRestored     int
	SessionsSkipped      int
	SchemaVer            string
}

RestoreResult summarizes a completed restore. Restored/Skipped counts make re-running a restore idempotent and observable.

type Restorer

type Restorer struct {
	DB *sql.DB
}

Restorer applies a logical snapshot to a v2 SQLite database, preserving original observation/edge IDs so graph relations and idempotent re-runs stay coherent.

Restore semantics:

  • manifest validation: format, format version, per-part sha256 (tamper rejection) and schema-version compatibility (refuse newer, see checkSchemaCompatible);
  • single transaction; any failure rolls the whole restore back;
  • idempotent: existing IDs are skipped (ON CONFLICT DO NOTHING), never overwritten;
  • embeddings are NOT restored. The target's observation_vectors are untouched; absent vectors mean the observation is pending re-embed and `cortex reindex` (or the embedding worker) self-heals.

func (*Restorer) Restore

func (r *Restorer) Restore(ctx context.Context, opts RestoreOptions) (*RestoreResult, error)

type Secret

type Secret struct {
	Name  string
	Value string
}

Secret is a named credential value the caller asserts must never appear in an exported archive. Names are only used in error messages.

func CollectEnvSecrets

func CollectEnvSecrets(extra ...Secret) []Secret

CollectEnvSecrets gathers non-trivial (>= minSecretLen runes) values for the known credential environment variables plus any extra (name, value) pairs supplied by the caller (e.g. tokens read from configuration).

type SessionRecord

type SessionRecord struct {
	ID          string         `json:"id"`
	Project     string         `json:"project"`
	Directory   string         `json:"directory"`
	StartedAt   string         `json:"started_at"`
	EndedAt     sql.NullString `json:"ended_at,omitempty"`
	Summary     sql.NullString `json:"summary,omitempty"`
	TenantID    sql.NullString `json:"tenant_id,omitempty"`
	WorkspaceID sql.NullString `json:"workspace_id,omitempty"`
}

SessionRecord is one portable session row.

type Source

type Source struct {
	Mode     string `json:"mode"` // "local-sqlite"
	TenantID string `json:"tenant_id,omitempty"`
}

Source records where the archive was produced.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL