kvit-coder

kvit-coder

A benchmarking framework for evaluating LLM coding agents on tool-use tasks.

Overview

This project provides a standardized benchmark suite to measure LLM performance on coding agent tasks including file operations, code search, and multi-step problem solving. It's designed to test local/self-hosted models against a consistent set of challenges.

Quick Start

Build

go build -o kvit-coder ./cmd/kvit-coder

Configure

Create a config file for your model (e.g., config-mymodel.yaml):

llm:
  base_url: "http://127.0.0.1:8080/v1"
  api_key_env: "OPENAI_API_KEY"
  model: "my-model-name"
  temperature: 0.2
  max_output_tokens: 2048

tools:
  read:
    enabled: true
  edit:
    enabled: true
  search:
    enabled: true
  shell:
    enabled: true

Run Benchmarks

# Run all benchmarks (10 runs each by default)
./kvit-coder --benchmark mymodel

# Custom number of runs
./kvit-coder --benchmark mymodel -n 5

# Run specific category
./kvit-coder --benchmark mymodel --benchmark-category search

# Run specific benchmark IDs
./kvit-coder --benchmark mymodel --benchmark-id S1,S2,R1

List Available Benchmarks

./kvit-coder --benchmark-list

Benchmark Categories

Output Files

Each benchmark run produces:

Benchmark Flags

External LLM Support

To benchmark an external tool (like Claude Code), use benchmark_cmd in your config:

llm:
  benchmark_cmd: "claude -p {prompt} --allowedTools Edit Bash Read"

The {prompt} placeholder is replaced with the benchmark task.

Results

Benchmark results are stored in the benchmarks/ directory. See benchmarks/README.md for details on interpreting results.

Adding Custom Benchmarks

Define benchmarks in benchmarks/benchmarks.yaml:

benchmarks:
  - id: MY1
    name: "My Custom Benchmark"
    category: custom
    goal: "Test something specific"
    setup:
      - file: "test.go"
        content: |
          package main
          func myFunc() {}
    task: "Find the function named myFunc"
    validation:
      - type: output_contains
        expected: "test.go"
      - type: tool_called
        expected: "search"

Validation types: file_contains, file_equals, file_exists, file_not_exists, file_line_count, tool_called, tool_called_with, output_contains, output_not_contains, output_matches, multi_tool_calls

Architecture

The agent consists of two binaries:

• kvit-coder - Headless agent for benchmarking and automation
• kvit-coder-ui - Interactive terminal UI (not used for benchmarking)

License

MIT