Documentation
¶
Overview ¶
Package eval is the scientific measurement instrument for the AI schema- generation layer (AI-F2-S1): a stratified NL→schema gold test set + a paired ablation harness + statistics with the rigor the research demands. It measures the loop's real behavior on THIS domain (typed JSON schemas with FKs/RBAC/state machines) — a number published nowhere; every figure in the literature is an extrapolation from text-to-SQL/code-gen. It builds NO new generation technique; it is the instrument those techniques are measured with.
stats.go is the statistics core (pure Go, no deps):
- Wilson score interval for a proportion (NOT Wald — Wald under-covers near 0/1).
- McNemar's paired test (Dietterich 1998: the only test with acceptable Type I error for run-once algorithms; never a paired t-test or a difference of proportions). EXACT binomial when discordants b+c < 25, Edwards continuity- corrected χ² otherwise.
- Cochran's Q omnibus for >2 paired conditions, with Holm-Bonferroni FWER control over the pairwise McNemar family.
E[iterations] is measured EMPIRICALLY by the harness (the loop is validator- guided, not i.i.d., so the geometric 1/p_sem is only a theoretical bound, marked as such — see report.go).
Index ¶
- Constants
- Variables
- func CorpusCounts(cases []Case) map[string]int
- func Format(a Analysis) string
- func HolmBonferroni(p []float64) []float64
- func WilsonInterval(successes, n int, z float64) (phat, lo, hi float64)
- type Analysis
- type Case
- type ClientFactory
- type CochranQResult
- type CondStats
- type Condition
- type McNemarResult
- type Outcome
- type PairTest
- type SimulatedClient
Constants ¶
const CorpusNote = "40/stratum (5x the original seed) — a substantial scale-up that narrows the " +
"Wilson intervals, still below the ~120-160/stratum ideal for full power; an underpowered " +
"McNemar (few discordants) remains EXPECTED and is flagged INCONCLUSIVE, not sold as significant"
CorpusNote is the honest one-liner about the corpus's statistical limits. At 40/stratum it is a 5x scale-up from the original seed (a real narrowing of the Wilson intervals) but still ~1/3 of the ~120-160/stratum the research wants for full power, so a McNemar with few discordants is still flagged INCONCLUSIVE.
const TargetPerStratum = 120
TargetPerStratum is the case count per stratum the research says is needed to detect a p_sem shift of 0.70→0.80 at 80% power — the seed is well below it, on purpose, and the harness grows to it without changing.
const Z95 = 1.959963984540054
Z95 is the standard normal quantile for a two-sided 95% interval.
Variables ¶
var Strata = []string{"simple", "media", "compleja"}
Strata are the complexity tiers the research requires (to locate where a cheap model crosses the competence threshold). Ordered simple → complex.
Functions ¶
func CorpusCounts ¶
CorpusCounts returns the number of cases per stratum.
func Format ¶
Format renders the human report. It is explicit about statistical power: an underpowered McNemar (discordants < 25) is flagged INCONCLUSIVE, never sold as significant — the instrument does not lie about its own power.
func HolmBonferroni ¶
HolmBonferroni returns the Holm-adjusted p-values aligned to the input order, controlling the family-wise error rate (step-down). Adjusted values are monotone non-decreasing in the sorted order and capped at 1.
func WilsonInterval ¶
WilsonInterval returns the point estimate and the Wilson score interval [lo, hi] for `successes` out of `n` at confidence implied by z (use Z95). For n == 0 it returns the whole [0,1]. The interval is clamped to [0,1] and never degenerate at the boundaries the way Wald is.
Types ¶
type Analysis ¶
type Analysis struct {
Mode string `json:"mode"` // "SIMULATED" | "LIVE"
Model string `json:"model"`
Counts map[string]int `json:"counts"`
TotalCases int `json:"total_cases"`
Conditions []string `json:"conditions"`
PerCondition []CondStats `json:"per_condition"`
Pairwise []PairTest `json:"pairwise"`
Cochran *CochranQResult `json:"cochran_q,omitempty"`
Note string `json:"note"`
}
Analysis is the full statistical readout.
type Case ¶
type Case struct {
ID string `json:"id"`
Stratum string `json:"stratum"`
Domain string `json:"domain"`
Description string `json:"description"`
Gold json.RawMessage `json:"gold"`
}
Case is one curated pair: a natural-language app description and the GOLD schema — a hand-written, validate-clean Appximo schema capturing that intent. The gold is correct by construction (a test asserts every gold validates).
func LoadCorpus ¶
LoadCorpus reads every embedded case, sorted by (stratum, id) for deterministic order (reproducibility — the same run twice yields the same sequence).
func SampleStratified ¶
SampleStratified returns at most n cases per stratum, taking the first n by the deterministic (stratum, id) order LoadCorpus already guarantees. n <= 0 returns all cases unchanged. Used to bound a cost-/rate-limited --live run to a stratified subsample while the full corpus stays the standing measurement set.
type ClientFactory ¶
type ClientFactory func(c Case, cond Condition) aigen.ModelClient
ClientFactory builds the ModelClient for a (case, condition). Live mode returns the real Anthropic client (temperature 0 for determinism); the demonstration mode returns a deterministic SimulatedClient. This is the seam that lets the harness run with NO API key (reproducible) and against a real model identically.
type CochranQResult ¶
type CochranQResult struct {
Q float64 `json:"q"`
DF int `json:"df"`
P float64 `json:"p_value"`
K int `json:"k"` // number of conditions
N int `json:"n"` // number of cases (rows)
}
CochranQResult is the omnibus test across k>=2 paired conditions.
func CochranQ ¶
func CochranQ(rows [][]bool) CochranQResult
CochranQ runs Cochran's Q over `rows`, where each row is one case's binary outcome under each of the k conditions (same column order every row). Concordant rows (all-success or all-failure) contribute nothing, exactly as in McNemar.
type CondStats ¶
type CondStats struct {
Condition string `json:"condition"`
Stratum string `json:"stratum,omitempty"` // "" = overall
N int `json:"n"`
FirstTrySuccess int `json:"first_try_success"`
Phat float64 `json:"p_sem"`
WilsonLo float64 `json:"wilson_lo"`
WilsonHi float64 `json:"wilson_hi"`
MeanIter float64 `json:"mean_iter"`
MedianIter float64 `json:"median_iter"`
NonConverged int `json:"non_converged"`
TheoIIDBound float64 `json:"theoretical_iid_bound"`
MeanStructErr0 float64 `json:"mean_struct_err0"`
MeanSemErr0 float64 `json:"mean_sem_err0"`
MeanCostUSD float64 `json:"mean_cost_usd"`
// StructuredEngaged / ArrayIREngaged count outcomes whose final decoding
// actually used structured / array-IR (vs fell back to plain). For the plain arm
// these are 0 by design; for the structured/array-IR arms a value BELOW n means
// the live API rejected the constrained request and the arm silently measured
// plain — the honest signal that the structural foundation did not engage.
StructuredEngaged int `json:"structured_engaged"`
ArrayIREngaged int `json:"array_ir_engaged"`
}
CondStats are the per-condition (optionally per-stratum) figures: p_sem with a Wilson interval, and the EMPIRICAL iteration distribution (mean/median) — never the geometric 1/p_sem, which assumes i.i.d. retries the validator-guided loop is not. TheoIIDBound carries 1/p_sem ONLY as the labeled "independent retries" reference, for contrast.
type Condition ¶
Condition is one arm of the paired ablation — a named treatment plus the aigen.Options that realize it. New techniques (array-IR, constraint-aware, RAG) plug in as NEW conditions; the harness, the outcomes, and the statistics do not change. That genericity is the point: every future technique is measured against the baseline with McNemar on THIS domain, or discarded.
func BaselineConditions ¶
func BaselineConditions() []Condition
BaselineConditions are the three arms: plain generation, the AI-F1-S1 structured-ENVELOPE decoding, and the AI-F2-S2 array-IR (structured DEEP) decoding. With three arms Cochran's Q + Holm engage automatically; the canonical per-stratum first-vs-last pair becomes plain vs array-IR (the full effect).
type McNemarResult ¶
type McNemarResult struct {
B int `json:"b"`
C int `json:"c"`
Discordant int `json:"discordant"`
Stat float64 `json:"stat"` // χ² statistic (0 for the exact path)
P float64 `json:"p_value"` // two-sided
Method string `json:"method"` // "exact-binomial" | "chi2-edwards" | "no-discordants"
}
McNemarResult is the outcome of a paired McNemar test on two conditions. B = cases where A succeeded and B failed; C = A failed and B succeeded. The power lives in the discordants (B+C); Concordant pairs carry no information.
func McNemar ¶
func McNemar(b, c int) McNemarResult
McNemar runs the paired test. With fewer than 25 discordants it uses the EXACT two-sided binomial (p = 0.5); otherwise the Edwards continuity-corrected χ² with one degree of freedom.
func (McNemarResult) Underpowered ¶
func (m McNemarResult) Underpowered() bool
Underpowered reports whether the discordant count is too small for the χ² approximation to be trustworthy — the instrument must say so rather than present a fragile p-value as conclusive.
type Outcome ¶
type Outcome struct {
CaseID string `json:"case_id"`
Stratum string `json:"stratum"`
Condition string `json:"condition"`
FirstTry bool `json:"first_try"` // valid at attempt 1 (the primary paired binary outcome)
Converged bool `json:"converged"` // valid within the iteration budget
Iterations int `json:"iterations"` // EMPIRICAL iterations to valid (not 1/p_sem)
StructErrs0 int `json:"struct_errs0"` // structural errors at attempt 1
SemErrs0 int `json:"sem_errs0"` // semantic errors at attempt 1
Refused bool `json:"refused"`
InputTok int `json:"input_tokens"`
OutputTok int `json:"output_tokens"`
CostUSD float64 `json:"cost_usd"`
// EffStructured / EffArrayIR record the decoding that ACTUALLY ran for the
// converged/final attempt — NOT what the condition requested. They differ when
// the constrained-decoding request was rejected by the live API and the loop
// fell back to plain generation (the live finding: the strict-outputs subset
// rejects the Appximo grammar — open objects need additionalProperties:false,
// and the field grammar exceeds the 16-union-parameter limit — so the structured
// and array-IR arms fall back to plain). Surfacing this keeps the instrument
// honest: an arm that fell back is measuring plain, not its named treatment.
EffStructured bool `json:"eff_structured"`
EffArrayIR bool `json:"eff_array_ir"`
}
Outcome is one (case × condition) measurement — the paired observational unit.
func RunAblation ¶
func RunAblation(ctx context.Context, cases []Case, conds []Condition, factory ClientFactory, model string) ([]Outcome, error)
RunAblation evaluates every case under every condition, in deterministic order, and returns the paired outcomes. The same cases run under each condition, so the outcomes are PAIRED by case id — the structure McNemar/Cochran require.
type PairTest ¶
type PairTest struct {
A string `json:"a"`
B string `json:"b"`
Stratum string `json:"stratum,omitempty"`
Result McNemarResult `json:"result"`
HolmP float64 `json:"holm_p,omitempty"`
}
PairTest is a paired McNemar comparison between two conditions (optionally within one stratum), with its Holm-adjusted p when part of a family.
type SimulatedClient ¶
type SimulatedClient struct {
// contains filtered or unexported fields
}
── deterministic demonstration driver ─────────────────────────────────────
SimulatedClient is a deterministic ModelClient that lets the instrument run end-to-end WITHOUT an API key, for demonstrating + testing the harness. It is NOT a real measurement — real p_sem requires a real model (temperature 0). Its outcomes are a deterministic function of (case id, condition, attempt), modeling the documented mechanism with a FAITHFUL structural-depth split so the three arms genuinely differ (the point of measuring array-IR):
- `plain` can emit BOTH a shallow ENVELOPE structural fault (unknown top-level key, bad const) AND a DEEP structural fault (a field's type outside the set).
- `structured` (AI-F1-S1 envelope) removes the ENVELOPE class but can STILL emit a DEEP structural fault — the strict subset cannot reach the map-keyed depth.
- `array-IR` (AI-F2-S2) removes BOTH classes by construction (an array of fixed items constrains the depth too), leaving only the SEMANTIC, cross-reference class — the hypothesis the harness measures: deep p_struct→1.
The semantic-fault probability is shared by all arms (no decoder constrains cross-reference semantics). The report labels this mode SIMULATED, loudly.
IMPORTANT (AI-F2-S3): this simulator ASSUMES the structured/array-IR decoders actually engage. The real --live run showed they DON'T on the current Anthropic API (the strict-outputs subset rejects the envelope's open objects and the IR's >16 union params, so both fall back to plain). So these arm differences are a demonstration of the MECHANISM, not a prediction of live behavior — the live numbers (and the EffStructured/EffArrayIR engagement counts) are the truth.
func NewSimulatedClient ¶
func NewSimulatedClient(c Case, cond Condition) *SimulatedClient
NewSimulatedClient builds the demonstration driver for one (case, condition), deriving each arm's structural coverage from its Options: plain emits both fault depths, structured removes only the envelope, array-IR removes both.
func (*SimulatedClient) Complete ¶
func (s *SimulatedClient) Complete(_ context.Context, req aigen.Request) (aigen.Completion, error)
Complete returns the (faithfully fault-injected) schema for this attempt. In the array-IR arm it emits the IR FORM (the loop transforms IR→map); the other arms emit the map form. Each fault class fires independently per the arm's coverage, so the validator + the real loop react to exactly what a model of that capability would produce — the harness measures the real loop over the simulated model.
Source Files