Documentation
¶
Overview ¶
Package drift measures how closely an agent's actual trajectory followed the behavioural contract compiled from a skill's specification. It is the half of the quality story a pass/fail verifier cannot see: a skill can produce the right file having ignored every instruction that was supposed to get it there, and that skill will fail on the next task.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultWeights = Weights{
StepCoverage: 0.25,
Order: 0.15,
Violation: 0.25,
Checkpoint: 0.15,
Semantic: 0.15,
Focus: 0.05,
}
DefaultWeights is the shipped distribution. Coverage and violations carry the most because they are the two things a contract exists to state: what must happen, and what must not.
var SeverityWeight = map[spec.Severity]float64{ spec.SeverityCritical: 3, spec.SeverityMajor: 1, spec.SeverityMinor: 0.3, }
SeverityWeight is the penalty each severity contributes to the violation exponent. Critical is deliberately far from major: a skill that exfiltrates once should not be able to average that away against a hundred correct steps.
Functions ¶
func Grade ¶
Grade is report sugar over mean and worst-run adherence. Thresholds are the shipped defaults; they are configuration, not a claim about skills in general.
func RobustnessGap ¶
RobustnessGap is the strong member's adherence minus the floor member's.
A positive gap means the skill leans on model capability rather than on explicit instruction — the strong model inferred what the text did not say. It is the primary input to repair, which is why the sign convention is fixed here rather than at each call site.
Both aggregates must carry at least one run: an Agg with N==0 is a zero value, not a measurement of zero adherence, and a gap computed against it would fabricate a maximal (or minimal) gap out of an absent measurement — mirroring Aggregate's refusal of zero runs.
Types ¶
type Agg ¶
Agg summarises one member's runs. Worst and Sigma are reported alongside Mean because they are different failure modes: a skill that works four runs in five is not the same as one that half-works every run, and a mean cannot tell them apart. Sigma is trajectory instability — same input, divergent behaviour.
Sigma is the population standard deviation over the runs actually performed (divide by N, not N-1): it describes the spread of this member's own observed runs, not an estimate of some wider population those runs were drawn from, so Bessel's correction does not apply. This also matters at n=2 (the Full tier's budget): a sample stddev from two points is unstable and inflated relative to the spread anyone actually observed, which would overstate instability beyond what the evidence supports.
type Components ¶
type Components struct {
StepCoverage float64
Order float64
Violation float64
Checkpoint float64
Semantic float64
Focus float64
}
Components are the six normalized rates, each in [0,1].
type Observation ¶
type Observation struct {
Steps []StepObs
Violations []Violation
Checkpoints map[string]bool
// OffContract is contractable events matching no step and no forbid rule.
OffContract int
// Total is contractable events. Opaque events are in neither.
Total int
// Unevaluable counts checks that could not be performed — an absolute path
// where a workspace-relative one was expected, a candidate MatchPath
// refuses. These are surfaced rather than folded into "no violation":
// a missed violation looks like a clean run, which is the failure direction
// that never gets investigated.
Unevaluable int
UnevaluableDetail []string
}
Observation is the deterministic result of matching one trajectory against one contract.
func Observe ¶
func Observe(c *contract.Contract, events []trajectory.Event) (*Observation, error)
Observe matches events against c.
type Result ¶
type Result struct {
Components Components `json:"components"`
Adherence float64 `json:"adherence"`
// Drift is 100-Adherence, kept as a stored field for in-process callers
// that want it computed once alongside Adherence, but never serialized:
// a report is one fact (Adherence) with two encodings otherwise, and a
// hand-edited or third-party report.json could set them inconsistently
// with no cross-field validation to catch it.
Drift float64 `json:"-"`
// Grade is not set by Score. Set it from Grade(agg.Mean, agg.Worst) after
// aggregating this run with its siblings.
Grade string `json:"grade,omitempty"`
// Unevaluable and UnevaluableDetail mirror Observation's fields of the same
// name: checks that could not be performed at all, surfaced here rather
// than folded into any of Components — a missed violation must not look
// like a clean run.
Unevaluable int `json:"unevaluable,omitempty"`
UnevaluableDetail []string `json:"unevaluable_detail,omitempty"`
}
Result is one run's drift. Grade is deliberately left unset here: grading is a property of an aggregate across runs, not of a single run, so callers set it from Grade(agg.Mean, agg.Worst) after aggregating.
type StepObs ¶
type StepObs struct {
ID string
Seq int
Matched bool
Required bool
// OrderOK is whether the matching event fell where the step's Order allows.
// A step can be matched and out of order: coverage counts it, OrderScore
// penalises it, and conflating the two loses one of the two signals.
OrderOK bool
}
StepObs is one contract step's fate in one trajectory.
type Violation ¶
type Violation struct {
ID string
Severity spec.Severity
Hits int
// Evidence quotes what triggered it. A violation nobody can check is a
// number nobody will act on.
Evidence []string
}
Violation is one forbid rule's hits.
type Weights ¶
type Weights struct {
StepCoverage float64
Order float64
Violation float64
Checkpoint float64
Semantic float64
Focus float64
}
Weights distribute Adherence across its components. They sum to 1: Adherence is 100 × Σ wᵢcᵢ, so a set that does not sum to 1 rescales every score and every grade boundary with it.