backfill

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package backfill provides the top-level orchestrator for historical session backfill. It coordinates session parsing, chunking, LLM extraction, validation, and deduplication.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultProgressPath

func DefaultProgressPath() string

DefaultProgressPath returns the default path for the progress state file.

func ImportFeedbackFiles added in v1.6.0

func ImportFeedbackFiles(ctx context.Context, dirs []string, llm learning.LLMClient) ([]FeedbackImportItem, *FeedbackImportResult, error)

ImportFeedbackFiles scans dirs for feedback_*.md files, processes each through the LLM to extract a structured TRIGGER→RULE→REASON observation, and returns the extracted items for the caller to store.

Files that are empty, unparseable, or trigger LLM errors are counted in FeedbackImportResult.Errors / FeedbackImportResult.Skipped and do not abort the run — processing continues with the remaining files.

Types

type Config

type Config struct {
	// Dir is the directory containing .jsonl session files (unused by Runner — caller resolves files).
	Dir string
	// Server is the target engram server URL for storing observations (unused by Runner — caller stores).
	Server string
	// Model is the LLM model override. Empty means use the client's default.
	Model string
	// DryRun skips actual LLM calls and only reports what would be processed.
	DryRun bool
	// MaxChunkChars is the maximum character count per chunk. 0 = use default.
	MaxChunkChars int
	// OverlapExchanges is the number of exchanges to overlap between chunks. 0 = use default.
	OverlapExchanges int
}

Config holds configuration for a backfill run.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults.

type ExtractedObservation

type ExtractedObservation struct {
	// SessionFile is the path to the source .jsonl file.
	SessionFile string
	// Project is the project directory from the session metadata.
	Project string
	// Outcome is the LLM-classified outcome (active_pattern, failed_experiment, superseded).
	Outcome string
	// RawXML is the raw XML returned by the LLM for this observation's chunk.
	RawXML string
	// Observation is the converted ParsedObservation ready for storage.
	Observation *models.ParsedObservation
}

ExtractedObservation pairs a parsed observation with its source session metadata.

type FeedbackImportItem added in v1.6.0

type FeedbackImportItem struct {
	SourceFile  string
	Observation *extract.XMLObservation
}

FeedbackImportItem pairs an extracted observation with its source file path.

type FeedbackImportResult added in v1.6.0

type FeedbackImportResult struct {
	Processed int
	Imported  int
	Skipped   int
	Errors    int
}

FeedbackImportResult holds summary statistics for a feedback import run.

type Progress

type Progress struct {
	RunID          string          `json:"run_id"`
	StartedAt      time.Time       `json:"started_at"`
	UpdatedAt      time.Time       `json:"updated_at"`
	ProcessedFiles map[string]bool `json:"processed_files"`
	TotalFiles     int             `json:"total_files"`
	StoredCount    int             `json:"stored_count"`
	SkippedCount   int             `json:"skipped_count"`
	ErrorCount     int             `json:"error_count"`
}

Progress tracks backfill progress for resumability.

func LoadProgress

func LoadProgress(path string) (*Progress, error)

LoadProgress loads progress state from a file. Returns a new Progress if file doesn't exist.

func (*Progress) FilterUnprocessed

func (p *Progress) FilterUnprocessed(files []string) []string

FilterUnprocessed returns only files that haven't been processed yet.

func (*Progress) IsProcessed

func (p *Progress) IsProcessed(file string) bool

IsProcessed checks if a file has already been processed.

func (*Progress) MarkProcessed

func (p *Progress) MarkProcessed(file string)

MarkProcessed marks a file as processed.

func (*Progress) Save

func (p *Progress) Save(path string) error

Save persists progress state to a file.

type Result

type Result struct {
	// Observations contains all extracted (unique, validated) observations across all sessions.
	Observations []ExtractedObservation
	// Summary is the session-level retrospective, if the retrospective pass ran.
	Summary *extract.SessionRetrospective
	// Metrics contains quality and progress statistics.
	Metrics *metrics.Metrics
}

Result holds the output of a complete backfill run.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner orchestrates the backfill pipeline for a list of session files.

func NewRunner

func NewRunner(llm learning.LLMClient, cfg Config) *Runner

NewRunner creates a new Runner with the given LLM client and config.

func (*Runner) ProcessSession

func (r *Runner) ProcessSession(ctx context.Context, sess *sessions.SessionMeta) (*Result, error)

ProcessSession processes a pre-parsed session and returns extracted observations. This is the server-side entry point — the caller provides a parsed SessionMeta (e.g. from ParseSessionReader) instead of a file path.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context, files []string) (*Result, error)

Run processes a list of .jsonl session files sequentially. It returns a Result containing all extracted observations and accumulated metrics. Errors from individual sessions are logged but do not abort the run.

Directories

Path Synopsis
Package chunk provides exchange-aware chunking for session backfill.
Package chunk provides exchange-aware chunking for session backfill.
Package extract provides LLM-based observation extraction and XML validation for session backfill.
Package extract provides LLM-based observation extraction and XML validation for session backfill.
Package metrics provides quality tracking for session backfill operations.
Package metrics provides quality tracking for session backfill operations.
Package sanitize provides text sanitization for session backfill processing.
Package sanitize provides text sanitization for session backfill processing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL