memory

package
v1.75.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 31, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package memory provides persistent memory storage for agent and analyst sessions.

Index

Constants

View Source
const (
	DimensionKnowledge    = "knowledge"
	DimensionEvent        = "event"
	DimensionEntity       = "entity"
	DimensionRelationship = "relationship"
	DimensionPreference   = "preference"
)

LOCOMO dimension values.

View Source
const (
	StatusActive     = "active"
	StatusStale      = "stale"
	StatusSuperseded = "superseded"
	StatusArchived   = "archived"
)

Status values for memory records.

View Source
const (
	CategoryCorrection    = "correction"
	CategoryBusinessCtx   = "business_context"
	CategoryDataQuality   = "data_quality"
	CategoryUsageGuidance = "usage_guidance"
	CategoryRelationship  = "relationship"
	CategoryEnhancement   = "enhancement"
	CategoryGeneral       = "general"
)

Category values for memory records.

View Source
const (
	SourceUser           = "user"
	SourceAgentDiscovery = "agent_discovery"
	SourceEnrichmentGap  = "enrichment_gap"
	SourceAutomation     = "automation"
	SourceLineageEvent   = "lineage_event"
)

Source values for memory records.

View Source
const (
	ConfidenceHigh   = "high"
	ConfidenceMedium = "medium"
	ConfidenceLow    = "low"
)

Confidence values for memory records.

View Source
const (
	MinContentLen    = 10
	MaxContentLen    = 4000
	MaxEntityURNs    = 10
	MaxRelatedCols   = 20
	DefaultLimit     = 20
	MaxLimit         = 100
	DefaultDimension = DimensionKnowledge
)

Validation constraints.

Variables

This section is empty.

Functions

func NormalizeCategory

func NormalizeCategory(c string) string

NormalizeCategory returns the category value, defaulting to "business_context".

func NormalizeConfidence

func NormalizeConfidence(c string) string

NormalizeConfidence returns the confidence value, defaulting to "medium".

func NormalizeDimension

func NormalizeDimension(d string) string

NormalizeDimension returns the dimension value, defaulting to "knowledge".

func NormalizeSource

func NormalizeSource(s string) string

NormalizeSource returns the source value, defaulting to "user".

func ParseURNToTable

func ParseURNToTable(urn string) (semantic.TableIdentifier, error)

ParseURNToTable attempts to extract a TableIdentifier from a DataHub dataset URN. URN format: urn:li:dataset:(urn:li:dataPlatform:platform,catalog.schema.table,ENV).

func ValidateCategory

func ValidateCategory(c string) error

ValidateCategory checks whether a category value is valid.

func ValidateConfidence

func ValidateConfidence(c string) error

ValidateConfidence checks whether a confidence value is valid.

func ValidateContent

func ValidateContent(text string) error

ValidateContent checks content length constraints. Length is measured in bytes (matching Go's len() behavior).

func ValidateDimension

func ValidateDimension(d string) error

ValidateDimension checks whether a dimension value is valid.

func ValidateEntityURNs

func ValidateEntityURNs(urns []string) error

ValidateEntityURNs checks the entity URN slice length.

func ValidateRelatedColumns

func ValidateRelatedColumns(cols []RelatedColumn) error

ValidateRelatedColumns checks the related columns slice length.

func ValidateSource

func ValidateSource(s string) error

ValidateSource checks whether a source value is valid.

func ValidateStatus

func ValidateStatus(s string) error

ValidateStatus checks whether a status value is valid.

Types

type Filter

type Filter struct {
	CreatedBy string
	Persona   string
	Dimension string
	Category  string
	Status    string
	Source    string
	EntityURN string
	Since     *time.Time
	Until     *time.Time
	Limit     int
	Offset    int
	// OrderBy overrides the default ordering ("created_at DESC").
	// Must be a valid SQL ORDER BY clause (e.g. "last_verified ASC NULLS FIRST").
	OrderBy string
}

Filter defines criteria for listing memory records.

func (*Filter) EffectiveLimit

func (f *Filter) EffectiveLimit() int

EffectiveLimit returns the limit capped to MaxLimit, defaulting to DefaultLimit.

type HybridQuery added in v1.74.0

type HybridQuery struct {
	Embedding []float32
	QueryText string
	Limit     int
	Persona   string
	Status    string
}

HybridQuery defines parameters for hybrid (vector + lexical) recall. Embedding drives the cosine arm; QueryText drives the lexical arm (Postgres full-text). A row matching either arm is a candidate; the two signals are fused per row by fuseHybridScore. Persona and Status are optional scope filters mirroring VectorQuery.

type LexicalQuery added in v1.74.0

type LexicalQuery struct {
	QueryText string
	Limit     int
	Persona   string
	Status    string
}

LexicalQuery defines parameters for lexical-only recall, used as the graceful-degradation path when no embedding provider is available. Unlike the vector arm, lexical search does not filter on a non-null embedding, so it also surfaces rows whose embedding was never computed (saved during an embedder outage).

type MiddlewareAdapter

type MiddlewareAdapter struct {
	// contains filtered or unexported fields
}

MiddlewareAdapter implements memory recall for the cross-injection middleware.

func NewMiddlewareAdapter

func NewMiddlewareAdapter(store Store) *MiddlewareAdapter

NewMiddlewareAdapter creates a new adapter wrapping a memory store.

func (*MiddlewareAdapter) RecallForEntities

func (a *MiddlewareAdapter) RecallForEntities(ctx context.Context, urns []string, persona string, limit int) ([]Snippet, error)

RecallForEntities returns memory snippets linked to the given DataHub URNs.

type Record

type Record struct {
	ID             string          `json:"id" example:"mem_a1b2c3d4e5f6"`
	CreatedAt      time.Time       `json:"created_at" example:"2026-03-18T08:11:08Z"`
	UpdatedAt      time.Time       `json:"updated_at" example:"2026-03-18T08:11:08Z"`
	CreatedBy      string          `json:"created_by" example:"sarah.chen@example.com"`
	Persona        string          `json:"persona" example:"admin"`
	Dimension      string          `json:"dimension" example:"knowledge"`
	Content        string          `json:"content" example:"The daily_sales table in the retail schema is partitioned by date."`
	Category       string          `json:"category" example:"business_context"`
	Confidence     string          `json:"confidence" example:"high"`
	Source         string          `json:"source" example:"user"`
	EntityURNs     []string        `json:"entity_urns"`
	RelatedColumns []RelatedColumn `json:"related_columns"`
	Embedding      []float32       `json:"embedding,omitempty"`
	// EmbeddingModel records the provider model that produced Embedding
	// (e.g. "nomic-embed-text"); EmbeddingTextHash is the SHA-256 of the
	// content fed to the embedder. They are the breadcrumbs the indexjobs
	// memory consumer uses to dedup re-embeds and detect model-swap gaps.
	// The synchronous write path stamps both when the embedder is healthy;
	// they are empty/nil on rows embedded before the column existed or
	// saved during an embedder outage (the reconciler later backfills).
	EmbeddingModel    string         `json:"embedding_model,omitempty"`
	EmbeddingTextHash []byte         `json:"embedding_text_hash,omitempty"`
	Metadata          map[string]any `json:"metadata"`
	Status            string         `json:"status" example:"active"`
	StaleReason       string         `json:"stale_reason,omitempty"`
	StaleAt           *time.Time     `json:"stale_at,omitempty"`
	LastVerified      *time.Time     `json:"last_verified,omitempty"`
}

Record represents a single memory record.

type RecordUpdate

type RecordUpdate struct {
	Content    string
	Category   string
	Confidence string
	Dimension  string
	Metadata   map[string]any
	Embedding  []float32
	// EmbeddingModel and EmbeddingTextHash travel with Embedding: when an
	// update re-embeds changed content, the write path stamps the model
	// and content hash alongside the new vector so the row's breadcrumbs
	// stay consistent with the embedding the indexjobs consumer dedups on.
	EmbeddingModel    string
	EmbeddingTextHash []byte
}

RecordUpdate holds fields that can be updated on a memory record.

type RelatedColumn

type RelatedColumn struct {
	URN       string `json:"urn" example:"urn:li:dataset:(urn:li:dataPlatform:trino,hive.sales.orders,PROD)"`
	Column    string `json:"column" example:"amount"`
	Relevance string `json:"relevance" example:"direct"`
}

RelatedColumn represents a column related to a memory record.

type ScoredRecord

type ScoredRecord struct {
	Record Record
	Score  float64
}

ScoredRecord pairs a memory record with a similarity score.

type Snippet

type Snippet struct {
	ID         string    `json:"id"`
	Content    string    `json:"content"`
	Dimension  string    `json:"dimension"`
	Category   string    `json:"category"`
	Confidence string    `json:"confidence"`
	CreatedAt  time.Time `json:"created_at"`
}

Snippet is a lightweight memory representation for cross-injection.

type StalenessConfig

type StalenessConfig struct {
	Interval  time.Duration
	BatchSize int
}

StalenessConfig configures the staleness watcher.

type StalenessWatcher

type StalenessWatcher struct {
	// contains filtered or unexported fields
}

StalenessWatcher periodically checks active memories against DataHub entity state and flags stale records.

func NewStalenessWatcher

func NewStalenessWatcher(store Store, sp semantic.Provider, cfg StalenessConfig) *StalenessWatcher

NewStalenessWatcher creates a new watcher.

func (*StalenessWatcher) Start

func (w *StalenessWatcher) Start(_ context.Context)

Start begins the periodic staleness check loop. It is safe to call multiple times; only the first call starts the loop.

func (*StalenessWatcher) Stop

func (w *StalenessWatcher) Stop()

Stop signals the watcher to stop and waits for completion. It is safe to call multiple times.

type Store

type Store interface {
	// Insert creates a new memory record.
	Insert(ctx context.Context, record Record) error

	// Get retrieves a single memory record by ID.
	Get(ctx context.Context, id string) (*Record, error)

	// Update modifies fields on an existing memory record.
	Update(ctx context.Context, id string, updates RecordUpdate) error

	// Delete soft-deletes a memory record by setting status to archived.
	Delete(ctx context.Context, id string) error

	// List returns memory records matching the filter with pagination.
	List(ctx context.Context, filter Filter) ([]Record, int, error)

	// VectorSearch performs cosine similarity search over embeddings.
	VectorSearch(ctx context.Context, query VectorQuery) ([]ScoredRecord, error)

	// HybridSearch fuses cosine similarity with a lexical full-text
	// signal for higher-quality recall on identifier-heavy content.
	HybridSearch(ctx context.Context, query HybridQuery) ([]ScoredRecord, error)

	// LexicalSearch ranks by Postgres full-text relevance only, the
	// graceful-degradation path when no embedding provider is available.
	LexicalSearch(ctx context.Context, query LexicalQuery) ([]ScoredRecord, error)

	// EntityLookup returns active memories linked to a DataHub URN.
	EntityLookup(ctx context.Context, urn string, persona string) ([]Record, error)

	// MarkStale flags memory records as stale with a reason.
	MarkStale(ctx context.Context, ids []string, reason string) error

	// MarkVerified updates the last_verified timestamp for records.
	MarkVerified(ctx context.Context, ids []string) error

	// Supersede marks an old record as superseded by a new one.
	Supersede(ctx context.Context, oldID, newID string) error
}

Store persists and queries memory records.

func NewNoopStore

func NewNoopStore() Store

NewNoopStore creates a no-op Store for use when no database is available.

func NewPostgresStore

func NewPostgresStore(db *sql.DB) Store

NewPostgresStore creates a new PostgreSQL memory store.

type VectorQuery

type VectorQuery struct {
	Embedding []float32
	Limit     int
	MinScore  float64
	Persona   string
	Status    string
}

VectorQuery defines parameters for vector similarity search.

Directories

Path Synopsis
Package memoryindex is the memory consumer of the shared indexjobs framework (#507).
Package memoryindex is the memory consumer of the shared indexjobs framework (#507).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL