README
¶
Cassandra Eval System
The Cassandra Eval system provides a data-driven, high-fidelity environment for evaluating the ai-reviewer agent using an LLM-as-a-Judge strategy. It is designed to ensure strict parity between production behavior and evaluation scenarios.
Architecture
The system consists of three main components:
- High-Fidelity Sandbox: A Git-backed environment that mirrors a real-world repository state using
git applyto ensure tools see exactly what they would in a real PR. - LLM-as-a-Judge: A structured evaluation pass where a "Judge" model (e.g., Claude 3.7 or Gemini 1.5 Pro) scores the "Subject" agent's review against a specific rubric.
- Data-Driven Fixtures: Test cases defined as filesystem fixtures (metadata, diffs, and base states), keeping evaluation data isolated from code.
Sandbox Lifecycle
For each evaluation case, the system:
- Creates a unique temporary directory.
- Populates the "Base" state from a
.tar.gzarchive or a directory. - Initializes a Git repository (
--initial-branch=main) and commits the base state. - Applies the
input.diffusinggit apply. - Commits the final state.
- Instantiates a
core.Reviewer(Subject) rooted in this directory.
This ensures that tools like read_file, glob_files, and grep_files operate with absolute parity to a production environment.
Creating Evaluation Cases
Cases are stored in core/eval/testdata/cases/. Each case is a directory containing:
metadata.json: Defines the case name, description, and the evaluation rubric.input.diff: The Git diff that the agent will be asked to review.base.tar.gz(Recommended): A tarball of the repository files before the diff is applied. Using tarballs prevents Bazel from indexing evaluation data.base/(Alternative): A directory containing the base files, useful for local development.
Example metadata.json
{
"name": "Security: Hardcoded API Key",
"description": "Tests if the agent identifies a hardcoded secret in a config file.",
"rubric": "The agent MUST identify the hardcoded API key in config.go and recommend using environment variables.",
"base_source": "base.tar.gz"
}
Running Evaluations
The eval CLI is the primary entry point for batch processing. It leverages the unified core.Reviewer factory to ensure the Subject uses your production configuration.
bazel run //cmd/eval -- \
--subject-config "$PWD/cassandra.toml" \
--subject-api-key $GOOGLE_API_KEY \
--judge-model gemini-1.5-pro \
--output results.json
CLI Options
--subject-config: Path to a Cassandra TOML file. The Subject will inherit all guidelines, tool settings, and model parameters from this file.--subject-api-key: API key for the Subject (overrides the config if provided).--judge-*: Configuration for the model performing the evaluation (provider, model, url, api-key). Defaults to the Subject's settings if not specified.--cases-dir: Directory containing the fixtures (defaults tocore/eval/testdata/cases).--output: Path to write the detailed results and metrics in JSON format.
Scoring Rubric
The Judge evaluates the agent's review on a scale of 1-5 based on:
- Accuracy: Correct identification of issues without false positives.
- Completeness: Addressing all points in the rubric.
- Constructiveness: Providing helpful and professional feedback.
- Depth: Effective use of tools to understand context.
The CLI outputs the Score, Rationale, and a list of specific Findings for each case.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var EvaluationResultSchema = map[string]any{ "type": "object", "properties": map[string]any{ "score": map[string]any{ "type": "integer", "description": "A score from 1 to 5, where 5 is an excellent, accurate, and helpful review, and 1 is a poor or misleading review.", "minimum": 1, "maximum": 5, }, "rationale": map[string]any{ "type": "string", "description": "A detailed explanation of why this score was given, referencing the rubric and the diff.", }, "findings": map[string]any{ "type": "array", "items": map[string]any{ "type": "string", }, "description": "Specific strengths or weaknesses identified in the review.", }, "met_rubric": map[string]any{ "type": "boolean", "description": "Whether the review successfully addressed the primary requirements of the rubric.", }, }, "required": []string{"score", "rationale", "findings", "met_rubric"}, }
EvaluationResultSchema is the JSON Schema for the Judge's output.
Functions ¶
This section is empty.
Types ¶
type CaseResult ¶
type CaseResult struct {
CaseID string `json:"case_id"`
CaseName string `json:"case_name"`
Subject EvaluationResult `json:"subject_result"`
Metrics core.SessionMetrics `json:"metrics"` // Subject metrics
Error string `json:"error,omitempty"`
}
CaseResult captures the outcome of running a single EvalCase.
type EvalCase ¶
type EvalCase struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
// Rubric is the specific criteria the Judge should use to score the review.
Rubric string `json:"rubric"`
// FixturePath is the relative path to the directory containing input.diff and base state.
// This is resolved relative to the suite manifest file.
FixturePath string `json:"fixture_path"`
// BaseSource optionally overrides the default 'base.tar.gz' or 'base/' directory
// resolution inside FixturePath.
BaseSource string `json:"base_source,omitempty"`
// DiffPath optionally overrides the default 'input.diff' inside FixturePath.
DiffPath string `json:"diff_path,omitempty"`
// Loaded data (not in manifest)
Diff string `json:"-"`
}
EvalCase represents a single evaluation scenario.
type EvaluationResult ¶
type EvaluationResult struct {
Score int `json:"score"` // 1-5 Scale
Rationale string `json:"rationale"` // Detailed explanation of the score
Findings []string `json:"findings"` // Specific observations
MetRubric bool `json:"met_rubric"` // Whether the review satisfied the core rubric
}
EvaluationResult is the structured output from the Judge.
type Runner ¶
type Runner struct {
SubjectConfig *config.Config // Configuration for the agent under test
Judge llm.Model // The model doing the evaluation
}
Runner orchestrates the evaluation process.
type Sandbox ¶
type Sandbox struct {
RootDir string
}
Sandbox provides a high-fidelity git environment for evaluating the agent.
func NewSandbox ¶
NewSandbox initializes a new git-backed sandbox. It copies or extracts files from baseState (if non-empty), initializes git, commits the base state, applies the diff, and commits the final state. baseState can be a directory path or a .tar.gz file path.
Source Files