Documentation
¶
Overview ¶
Package eval provides a small evaluation harness for the memory service: a dataset of queries with expected answers and a runner that measures recall@k. See PRD §11 and M1 milestone.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultDataset = []Case{
{Query: "what framework is the backend built on", Expected: "FastAPI", Project: "demo"},
{Query: "which database postgres version", Expected: "Postgres", Project: "demo"},
{Query: "how do tests run with pytest", Expected: "pytest", Project: "demo"},
{Query: "api container base image python", Expected: "python:3.12-slim", Project: "demo"},
{Query: "who owns the mobile app alex", Expected: "Alex", Project: "demo"},
{Query: "is the mobile app flutter", Expected: "Flutter", Project: "demo"},
{Query: "where do api secrets live environment variables", Expected: "environment variables", Project: "demo"},
{Query: "which tool deploys github actions", Expected: "GitHub Actions", Project: "demo"},
{Query: "redis cache layer", Expected: "Redis", Project: "demo"},
{Query: "staging environment url example.com", Expected: "staging.example.com", Project: "demo"},
{Query: "python linter ruff", Expected: "ruff", Project: "demo"},
{Query: "database migrations alembic", Expected: "alembic", Project: "demo"},
{Query: "background jobs celery queue", Expected: "Celery", Project: "demo"},
{Query: "frontend react typescript", Expected: "React", Project: "demo"},
{Query: "design system owner maria", Expected: "Maria", Project: "demo"},
{Query: "api versioning semantic versioning", Expected: "semantic versioning", Project: "demo"},
{Query: "bugs tracked github issues", Expected: "GitHub Issues", Project: "demo"},
{Query: "default branch main", Expected: "main", Project: "demo"},
{Query: "node package manager pnpm", Expected: "pnpm", Project: "demo"},
{Query: "logs collected opentelemetry", Expected: "OpenTelemetry", Project: "demo"},
}
DefaultDataset is a small, self-contained set of 20 queries about a fictional project, used by `forgetmenot eval` and tests. Queries share keyword overlap with the seeded facts so that a decent embedder (including the deterministic bag-of-words one used in tests) can retrieve them.
Functions ¶
func SeedDataset ¶
SeedDataset populates a store with the facts the DefaultDataset queries reference. Idempotent per project: if the demo project already has memories, seeding is skipped. Returns the number of inserted memories.
Types ¶
type Case ¶
type Case struct {
Query string
Project string
Expected string // substring expected to appear in the top-k results
K int // top-k considered; 0 defaults to 3
}
Case is one evaluation question: recall(query) should surface the memory whose content contains Expected.
type Result ¶
type Result struct {
Total int
Passed int
RecallAtK float64 // passed/total
Failures []Failure
Errors []string // recall errors encountered during the run
}
Result aggregates a run over the dataset.
Source Files