Documentation
¶
Overview ¶
Package differentiators holds the v0.3 eval scenarios that the upstream LangChain SRE harness structurally cannot express (docs/v0.3-plan.md W0.3). Its 31-row corpus scores a single uninterrupted trajectory against a list of expected tool names; nothing in that shape can ask what happens when a remediation is interrupted halfway, when a prior effect's outcome is unknown, when the budget runs out mid-investigation, or when the operator says no.
Each scenario drives the composed mast runtime — internal/compose's root, the pkg/effects outbox, the pkg/budget meter, a real SQLite session store — with a scripted model, and checks one invariant.
Three outcomes, not two ¶
A scenario is Pass, Fail, or Broken.
- Pass: the invariant held.
- Fail: the run happened and the invariant was violated. This is a capability gap, and it is a legitimate state for a scenario to be declared in while the capability is unbuilt.
- Broken: the fixture could not produce a run at all. This is a harness defect and is never declarable.
That split is what makes W0.3's "fails for the right reason (missing capability, not missing fixture)" mechanical rather than a matter of authorial care. A scenario that cannot set the situation up returns an error and lands in Broken, which no allowlist can absorb; and the driver additionally requires every scenario — passing or failing — to produce a non-empty evals.Trace, so "the capability is missing" is always an observation about a run that happened, never the absence of one.
Expect is the allowlist, and it is checked in both directions ¶
Scenario.Expect declares what the scenario does against today's code. The driver fails when the outcome differs from Expect either way: a regression in a shipped capability, and a capability landed without flipping its scenario, are the same kind of defect. That bidirectional check is why the declaration can be a hand-maintained bit without rotting — you cannot land W2 without this suite telling you which entries to remove.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Env ¶
type Env struct {
// Dir is a scratch directory the scenario owns for the duration of
// the run — session databases, specialist fixtures. House rule #5:
// callers derive it from os.TempDir (t.TempDir does), never $HOME.
Dir string
}
Env carries what a scenario needs from its caller.
type Report ¶
type Report struct {
Scenario Scenario
Result Result
Outcome Outcome
// Err is set when the scenario returned one, i.e. when Outcome is
// Broken.
Err error
}
Report pairs a scenario with what it actually did.
type Result ¶
type Result struct {
// Held is the invariant's verdict.
Held bool
// Reason states what was observed, and is required whether the
// invariant held or not. For a failing scenario it is the evidence
// that a capability is missing; a scenario that cannot say what it
// saw should return an error instead.
Reason string
// Trace is the run's recorded trajectory, scored through the same
// adapter the parity corpus uses (internal/evals). The driver
// requires it to carry at least one call: it is the mechanical
// proof that the fixture drove a real run.
Trace evals.Trace
}
Result is what a scenario's Run reports back.
type Scenario ¶
type Scenario struct {
// ID is the tier-prefixed name from docs/v0.3-plan.md §2, e.g.
// "E-approval-rejected".
ID string
// Invariant is the one sentence the scenario checks. It is the
// scoreboard row's claim in testable form.
Invariant string
// Expect is the outcome this scenario has against today's code.
// Pass or Fail only; see the package doc on why it is checked in
// both directions.
Expect Outcome
// Blocked names the workstream that flips Expect to Pass and the
// concrete thing that is missing. Required when Expect is Fail,
// and must be empty when Expect is Pass — a passing scenario that
// still claims to be blocked is a stale declaration.
Blocked string
// Rows are the docs/v0.3-plan.md §1 scoreboard rows this scenario
// is the proof for.
Rows []string
// Run sets the fixture up, drives it, and checks the invariant.
// A returned error is Broken: the fixture could not produce a run,
// which is a harness defect and not an allowlistable failure.
Run func(ctx context.Context, env Env) (Result, error)
}
Scenario is one differentiator.
Source Files