Documentation
¶
Overview ¶
Package backfill provides the top-level orchestrator for historical session backfill. It coordinates session parsing, chunking, LLM extraction, validation, and deduplication.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultProgressPath ¶
func DefaultProgressPath() string
DefaultProgressPath returns the default path for the progress state file.
func ImportFeedbackFiles ¶ added in v1.6.0
func ImportFeedbackFiles(ctx context.Context, dirs []string, llm learning.LLMClient) ([]FeedbackImportItem, *FeedbackImportResult, error)
ImportFeedbackFiles scans dirs for feedback_*.md files, processes each through the LLM to extract a structured TRIGGER→RULE→REASON observation, and returns the extracted items for the caller to store.
Files that are empty, unparseable, or trigger LLM errors are counted in FeedbackImportResult.Errors / FeedbackImportResult.Skipped and do not abort the run — processing continues with the remaining files.
Types ¶
type Config ¶
type Config struct {
// Dir is the directory containing .jsonl session files (unused by Runner — caller resolves files).
Dir string
// Server is the target engram server URL for storing observations (unused by Runner — caller stores).
Server string
// Model is the LLM model override. Empty means use the client's default.
Model string
// DryRun skips actual LLM calls and only reports what would be processed.
DryRun bool
// MaxChunkChars is the maximum character count per chunk. 0 = use default.
MaxChunkChars int
// OverlapExchanges is the number of exchanges to overlap between chunks. 0 = use default.
OverlapExchanges int
}
Config holds configuration for a backfill run.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns a Config with sensible defaults.
type ExtractedObservation ¶
type ExtractedObservation struct {
// SessionFile is the path to the source .jsonl file.
SessionFile string
// Project is the project directory from the session metadata.
Project string
// Outcome is the LLM-classified outcome (active_pattern, failed_experiment, superseded).
Outcome string
// RawXML is the raw XML returned by the LLM for this observation's chunk.
RawXML string
// Observation is the converted ParsedObservation ready for storage.
Observation *models.ParsedObservation
}
ExtractedObservation pairs a parsed observation with its source session metadata.
type FeedbackImportItem ¶ added in v1.6.0
type FeedbackImportItem struct {
SourceFile string
Observation *extract.XMLObservation
}
FeedbackImportItem pairs an extracted observation with its source file path.
type FeedbackImportResult ¶ added in v1.6.0
FeedbackImportResult holds summary statistics for a feedback import run.
type Progress ¶
type Progress struct {
RunID string `json:"run_id"`
StartedAt time.Time `json:"started_at"`
UpdatedAt time.Time `json:"updated_at"`
ProcessedFiles map[string]bool `json:"processed_files"`
TotalFiles int `json:"total_files"`
StoredCount int `json:"stored_count"`
SkippedCount int `json:"skipped_count"`
ErrorCount int `json:"error_count"`
}
Progress tracks backfill progress for resumability.
func LoadProgress ¶
LoadProgress loads progress state from a file. Returns a new Progress if file doesn't exist.
func (*Progress) FilterUnprocessed ¶
FilterUnprocessed returns only files that haven't been processed yet.
func (*Progress) IsProcessed ¶
IsProcessed checks if a file has already been processed.
func (*Progress) MarkProcessed ¶
MarkProcessed marks a file as processed.
type Result ¶
type Result struct {
// Observations contains all extracted (unique, validated) observations across all sessions.
Observations []ExtractedObservation
// Summary is the session-level retrospective, if the retrospective pass ran.
Summary *extract.SessionRetrospective
// Metrics contains quality and progress statistics.
Metrics *metrics.Metrics
}
Result holds the output of a complete backfill run.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner orchestrates the backfill pipeline for a list of session files.
func (*Runner) ProcessSession ¶
ProcessSession processes a pre-parsed session and returns extracted observations. This is the server-side entry point — the caller provides a parsed SessionMeta (e.g. from ParseSessionReader) instead of a file path.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package chunk provides exchange-aware chunking for session backfill.
|
Package chunk provides exchange-aware chunking for session backfill. |
|
Package extract provides LLM-based observation extraction and XML validation for session backfill.
|
Package extract provides LLM-based observation extraction and XML validation for session backfill. |
|
Package metrics provides quality tracking for session backfill operations.
|
Package metrics provides quality tracking for session backfill operations. |
|
Package sanitize provides text sanitization for session backfill processing.
|
Package sanitize provides text sanitization for session backfill processing. |