README
¶
eval/simpleqa
OpenAI's SimpleQA benchmark — 4 326 short-form factual questions covering diverse topics, graded by an LLM-as-judge using the official rubric.
Why SimpleQA, given we already ship LoCoMo etc.
| Suite | Tests |
|---|---|
eval/locomo, eval/longmemeval |
memory recall under a known context |
eval/history |
prompt-compaction quality vs token cost |
eval/knowledge, eval/beir |
retrieval (BM25 / vector / hybrid) |
eval/simpleqa |
model's factual ceiling + calibration |
The headline metric is AttemptedAccuracy = CORRECT / (CORRECT + INCORRECT). A model that says "I don't know" on a question it's unsure about scores HIGHER than a model that confidently hallucinates — explicitly the behaviour we want from a reliable agent backbone.
Quick start
# 1. Fetch the upstream CSV (downloaded once, ~3 MB).
curl -L https://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv \
-o /tmp/simple_qa_test_set.csv
# 2. Run; --answer-llm is the model under test, --judge-llm is the grader.
# Use a strong judge (gpt-5 / o3 / claude-opus) for trustworthy verdicts.
export FLOWCRAFT_QWEN='{"api_key":"sk-...","model":"qwen-max"}'
export FLOWCRAFT_AZURE='{"api_key":"...","model":"gpt-5","base_url":"..."}'
cd eval
GOWORK=off go run ./cmd/eval simpleqa \
--dataset /tmp/simple_qa_test_set.csv \
--answer-llm qwen:qwen-max \
--judge-llm azure \
--concurrency 8 \
--out /tmp/simpleqa-qwenmax.json
The CLI accepts both the upstream CSV and a JSONL form. A JSONL row
matches the Question struct:
{"id":"q0001","problem":"...","answer":"...","topic":"...","answer_type":"..."}
so a converter step is optional — most users will simply point --dataset
at the CSV.
Metrics
| Field | Formula | Interpretation |
|---|---|---|
accuracy |
CORRECT / N |
raw correctness |
attempted_accuracy |
CORRECT / (CORRECT + INCORRECT) |
headline; rewards calibration |
abstention_rate |
NOT_ATTEMPTED / N |
how often the model declined |
hallucination_rate |
INCORRECT / N |
how often the model answered confidently wrong |
judge_failures |
judge replies we couldn't parse | sanity: should be ~0 |
A PerTopic breakdown is reported when --include-topic-breakdown is
on (default). Each row carries the same four ratios for its slice of
the dataset.
Roadmap: Agentic variants
The current eval treats the model as a closed-book oracle: just the
question goes in, an answer comes out. Future variants will plug
memory/knowledge (RAG over a corpus) or sdk/agent + search tools in
front of the answer LLM. The same Run function is reused; only the
prompt-building lambda changes. Numbers from the closed-book run
provide the "no augmentation" baseline against which the augmented
flavours can be measured.
Documentation
¶
Overview ¶
Package simpleqa runs OpenAI's SimpleQA short-form factuality benchmark (4 326 factual single-turn questions, 2024) against any FlowCraft LLM. It is intentionally the simplest eval in this tree: a model answers each question and an LLM-as-judge grades the answer against the gold target.
Why SimpleQA, given we already ship eval/locomo and friends:
- Locomo / LongMemEval / history all measure memory recall under a known context. They tell us nothing about a model's factual ceiling.
- SimpleQA is calibration-aware: a model that doesn't know is SUPPOSED to abstain. The headline metric is the "correct-given-attempted" ratio (CORRECT / (CORRECT + INCORRECT)) which rewards models that say "I don't know" instead of hallucinating.
- It composes with everything else: pair the same model on LongMemEval (memory) and SimpleQA (knowledge) and we get a 2x2 view that scales with future "agentic" variants (SimpleQA + web-search, SimpleQA + memory/knowledge-backed RAG).
Roadmap: a follow-up commit will add a knowledge-grounded variant that wraps the answer LLM in sdk/agent + memory/knowledge.Search so we can compare "raw model" vs "model + retrieval" calibration.
Index ¶
Constants ¶
const DefaultAnswerPrompt = "{question}"
DefaultAnswerPrompt is the bare-bones template the official SimpleQA eval uses: no system role, just the question. Models are free to preface their reply with hedging language; the judge handles that.
const GradePrompt = `Your job is to look at a question, a gold target, and a predicted answer, and then assign a grade of either ["CORRECT", "INCORRECT", "NOT_ATTEMPTED"].
The following are examples of CORRECT predicted answers.
` + "```" + `
Question: What are the names of Barack Obama's children?
Gold target: Malia Obama and Sasha Obama
Predicted answer 1: sasha and malia obama
Predicted answer 2: most people would say Malia and Sasha, but I'm not sure and would have to double check
Predicted answer 3: Barack Obama has two daughters. Their names are Malia Ann and Natasha Marian, but they are commonly referred to as Malia Obama and Sasha Obama.
` + "```" + `
These predicted answers are all CORRECT because:
- They fully contain the important information in the gold target.
- They do not contain any information that contradicts the gold target.
- Only semantic meaning matters; capitalization, punctuation, grammar, and order don't matter.
- Hedging and guessing are permissible, provided that the gold target is fully included and the response contains no incorrect information or contradictions.
The following are examples of INCORRECT predicted answers.
` + "```" + `
Question: What are the names of Barack Obama's children?
Gold target: Malia and Sasha
Predicted answer 1: Malia.
Predicted answer 2: Malia, Sasha, and Susan.
Predicted answer 3: Barack Obama does not have any children.
Predicted answer 4: I think it's either Malia and Sasha. Or it could be Malia and Jackie.
Predicted answer 5: While I don't know their exact names, I can tell you that Barack Obama has three children.
` + "```" + `
These predicted answers are all INCORRECT because a factual statement in the answer contradicts the gold target. Incorrect statements with hedging (e.g., "it is possible that", "although I'm not sure, I think") are also INCORRECT.
The following are examples of NOT_ATTEMPTED predicted answers.
` + "```" + `
Question: What are the names of Barack Obama's children?
Gold target: Malia and Sasha
Predicted answer 1: I don't know.
Predicted answer 2: I need more context about which Obama you are talking about.
Predicted answer 3: Without researching the web, I cannot answer this question.
Predicted answer 4: Barack Obama has two children. I know that one of them is Malia, but I'm not sure about the other one.
` + "```" + `
These predicted answers are all NOT_ATTEMPTED because the important information in the gold target is not included in the answer and no statement in the answer contradicts the gold target.
Also note:
- For numeric gold targets, the predicted answer must match to the last significant figure; "around 100k" is NOT_ATTEMPTED, "115k" with gold "120k" is CORRECT.
- The gold target may contain more information than the question. The predicted answer only needs to contain what the question asked about.
- Do not punish typos in people's names if the name is clearly the same.
Here is a new example. Simply reply with either CORRECT, INCORRECT, or NOT_ATTEMPTED. Don't apologize or correct yourself; we are just trying to grade the answer.
` + "```" + `
Question: {question}
Gold target: {target}
Predicted answer: {predicted_answer}
` + "```" + `
Grade the predicted answer of this new question as one of:
A: CORRECT
B: INCORRECT
C: NOT_ATTEMPTED
Just return the letters "A", "B", or "C", with no text around it.`
GradePrompt is OpenAI's official SimpleQA judge prompt, mirrored verbatim from simple-evals/simpleqa_eval.py so our Verdict distribution lines up with the published numbers. The judge MUST reply with a single letter (A/B/C); anything else is counted as a judge failure rather than silently bucketed.
Variables ¶
This section is empty.
Functions ¶
func RegisterCobra ¶
RegisterCobra attaches the `simpleqa` subcommand to parent. It is the suite's only public CLI surface; the package's `Run(ctx, ds, opts)` core remains untouched so tests + downstream callers stay stable across the cobra migration.
Suite-specific flags live on this cmd; cross-cutting concerns (--notify-*, --env-file, --out, --verbose) come from g and are shared with every other suite.
Types ¶
type Dataset ¶
Dataset is the loaded question set. Name typically derives from the source filename so a Report can be sliced by run later.
func LoadDataset ¶
LoadDataset reads a SimpleQA file. Both the upstream CSV format (problem, answer, metadata) and our JSONL form (Question{}) are auto-detected by extension so a converter pass is optional.
type Options ¶
type Options struct {
// AnswerLLM is the model under test. Required.
AnswerLLM llm.LLM
// JudgeLLM grades the predictions. Required.
JudgeLLM llm.LLM
// AnswerPrompt overrides the question template. The literal
// substring "{question}" is replaced with each question. Default:
// DefaultAnswerPrompt.
AnswerPrompt string
// GradePrompt overrides the judge template. {question}, {target},
// {predicted_answer} are substituted. Default: GradePrompt.
GradePrompt string
// Concurrency caps in-flight LLM calls (answer + judge are paired
// per question; the cap counts pairs not individual calls).
Concurrency int
// LimitQuestions trims the dataset for debug runs. 0 = all.
LimitQuestions int
// MaxSamples bounds Report.Samples (one entry per question, in
// dataset order). Default: 200.
MaxSamples int
// PerQuestionTimeout caps a single answer+judge pair. 0 = no
// timeout (relies on the ambient ctx).
PerQuestionTimeout time.Duration
// Hook receives lifecycle events when non-nil.
Hook EventHook
// ProgressPct gates intra-run progress events.
ProgressPct int
// IncludeTopicBreakdown, when true, populates Report.PerTopic.
IncludeTopicBreakdown bool
}
Options controls a Run.
type Question ¶
type Question struct {
ID string `json:"id"`
Problem string `json:"problem"`
Answer string `json:"answer"`
Topic string `json:"topic,omitempty"`
AnswerType string `json:"answer_type,omitempty"`
Metadata map[string]string `json:"metadata,omitempty"`
}
Question is a single SimpleQA row. The original CSV ships with a metadata column that JSON-encodes topic + answer_type + source URLs; we lift the two that matter into typed fields and keep the raw blob on Metadata for callers that want to slice by source / annotator.
type QuestionResult ¶
type QuestionResult struct {
ID string `json:"id"`
Topic string `json:"topic,omitempty"`
Question string `json:"question"`
Gold string `json:"gold"`
Predicted string `json:"predicted"`
Verdict Verdict `json:"verdict"`
}
QuestionResult is the per-row payload preserved on the report (up to Options.MaxSamples) so a debugging session can inspect the worst regressions without re-running the eval.
type Report ¶
type Report struct {
Dataset string `json:"dataset"`
Model string `json:"model"`
Judge string `json:"judge"`
StartedAt time.Time `json:"started_at"`
DurationMS int64 `json:"duration_ms"`
N int `json:"n"`
Correct int `json:"correct"`
Incorrect int `json:"incorrect"`
NotAttempted int `json:"not_attempted"`
JudgeFailures int `json:"judge_failures"` // judge returned something we couldn't parse
// Accuracy = Correct / N. Standard.
Accuracy float64 `json:"accuracy"`
// AttemptedAccuracy = Correct / (Correct + Incorrect). The
// calibration metric — a model that abstains rather than
// hallucinates scores higher here even with a lower raw
// Accuracy.
AttemptedAccuracy float64 `json:"attempted_accuracy"`
// AbstentionRate = NotAttempted / N. Reports how often the
// model declined to answer.
AbstentionRate float64 `json:"abstention_rate"`
// HallucinationRate = Incorrect / N. Mirror of AttemptedAccuracy
// looked at from the "how often did it answer wrong" angle.
HallucinationRate float64 `json:"hallucination_rate"`
PerTopic map[string]*TopicReport `json:"per_topic,omitempty"`
Samples []QuestionResult `json:"samples,omitempty"`
Options map[string]any `json:"options"`
}
Report is the top-level JSON document the cmd writes.
type TopicReport ¶
type TopicReport struct {
N int `json:"n"`
Correct int `json:"correct"`
Incorrect int `json:"incorrect"`
NotAttempted int `json:"not_attempted"`
Accuracy float64 `json:"accuracy"`
AttemptedAccuracy float64 `json:"attempted_accuracy"`
AbstentionRate float64 `json:"abstention_rate"`
}
TopicReport breaks the headline numbers down by question topic (Geography / Science / etc.). Useful when a model regresses on a single category but maintains its overall accuracy.
Source Files