Documentation
¶
Overview ¶
Package score grades answers and builds the prompt. The scorer is a port of the official GAIA question_scorer, quirks included: the point is to give the score the leaderboard would give, not the score that seems right.
Index ¶
Constants ¶
const NoAnswer = "None"
NoAnswer is what the official scorer reads when there is no answer at all.
Variables ¶
This section is empty.
Functions ¶
func Extract ¶
Extract returns the final answer in a reply, by a rule that is frozen so that results stay comparable: the text after the last "FINAL ANSWER:" (any case), its first non-empty line, with surrounding ** removed. A line that is empty once the ** and the spaces are gone (the closing ** of a bold label, say) does not count as a line. ok is false when there is no marker, or no answer after it.
It does not try to be clever. An answer in backticks or with a trailing explanation is returned as written, and scored as written.
Pattern: Frozen Extraction Rule
func Question ¶
Question reports whether answer matches truth under the official GAIA rules (question_scorer in the leaderboard's scorer.py):
- A truth that Python can read as a number: the answer, minus $, % and commas, must read as the same number.
- A truth with a comma or a semicolon is a list. Answer and truth are split on both, must have the same length, and match element by element: numbers as numbers, anything else with white space removed and the case folded, but punctuation kept. So a truth of "1,000" is the list 1 and 000.
- Anything else: white space and ASCII punctuation removed, case folded.
Pass NoAnswer when there is no answer.
Types ¶
type Prompt ¶
type Prompt struct {
// contains filtered or unexported fields
}
Prompt turns a question into the text typed into the composer: the answer format instructions, then the question, with the FINAL ANSWER convention the extraction below relies on.
func ParsePrompt ¶
ParsePrompt reads a template. It is tried with a sample question, which has to come out: a prompt that does not contain the question asks nothing, and one that fails halfway would be typed truncated, for every attempt, with no error.
Source Files