drift

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package drift measures how closely an agent's actual trajectory followed the behavioural contract compiled from a skill's specification. It is the half of the quality story a pass/fail verifier cannot see: a skill can produce the right file having ignored every instruction that was supposed to get it there, and that skill will fail on the next task.

Index

Constants

This section is empty.

Variables

View Source
var DefaultWeights = Weights{
	StepCoverage: 0.25,
	Order:        0.15,
	Violation:    0.25,
	Checkpoint:   0.15,
	Semantic:     0.15,
	Focus:        0.05,
}

DefaultWeights is the shipped distribution. Coverage and violations carry the most because they are the two things a contract exists to state: what must happen, and what must not.

SeverityWeight is the penalty each severity contributes to the violation exponent. Critical is deliberately far from major: a skill that exfiltrates once should not be able to average that away against a hundred correct steps.

Functions

func Grade

func Grade(mean, worst float64) string

Grade is report sugar over mean and worst-run adherence. Thresholds are the shipped defaults; they are configuration, not a claim about skills in general.

func RobustnessGap

func RobustnessGap(strong, floor Agg) (float64, error)

RobustnessGap is the strong member's adherence minus the floor member's.

A positive gap means the skill leans on model capability rather than on explicit instruction — the strong model inferred what the text did not say. It is the primary input to repair, which is why the sign convention is fixed here rather than at each call site.

Both aggregates must carry at least one run: an Agg with N==0 is a zero value, not a measurement of zero adherence, and a gap computed against it would fabricate a maximal (or minimal) gap out of an absent measurement — mirroring Aggregate's refusal of zero runs.

Types

type Agg

type Agg struct {
	Mean  float64
	Worst float64
	Sigma float64
	N     int
}

Agg summarises one member's runs. Worst and Sigma are reported alongside Mean because they are different failure modes: a skill that works four runs in five is not the same as one that half-works every run, and a mean cannot tell them apart. Sigma is trajectory instability — same input, divergent behaviour.

Sigma is the population standard deviation over the runs actually performed (divide by N, not N-1): it describes the spread of this member's own observed runs, not an estimate of some wider population those runs were drawn from, so Bessel's correction does not apply. This also matters at n=2 (the Full tier's budget): a sample stddev from two points is unstable and inflated relative to the spread anyone actually observed, which would overstate instability beyond what the evidence supports.

func Aggregate

func Aggregate(rs []Result) (Agg, error)

Aggregate summarises runs. Zero runs is an error, not a zero: an absent measurement must not be indistinguishable from total failure.

type Components

type Components struct {
	StepCoverage float64
	Order        float64
	Violation    float64
	Checkpoint   float64
	Semantic     float64
	Focus        float64
}

Components are the six normalized rates, each in [0,1].

type Observation

type Observation struct {
	Steps       []StepObs
	Violations  []Violation
	Checkpoints map[string]bool
	// OffContract is contractable events matching no step and no forbid rule.
	OffContract int
	// Total is contractable events. Opaque events are in neither.
	Total int
	// Unevaluable counts checks that could not be performed — an absolute path
	// where a workspace-relative one was expected, a candidate MatchPath
	// refuses. These are surfaced rather than folded into "no violation":
	// a missed violation looks like a clean run, which is the failure direction
	// that never gets investigated.
	Unevaluable       int
	UnevaluableDetail []string
}

Observation is the deterministic result of matching one trajectory against one contract.

func Observe

func Observe(c *contract.Contract, events []trajectory.Event) (*Observation, error)

Observe matches events against c.

type Result

type Result struct {
	Components Components `json:"components"`
	Adherence  float64    `json:"adherence"`
	// Drift is 100-Adherence, kept as a stored field for in-process callers
	// that want it computed once alongside Adherence, but never serialized:
	// a report is one fact (Adherence) with two encodings otherwise, and a
	// hand-edited or third-party report.json could set them inconsistently
	// with no cross-field validation to catch it.
	Drift float64 `json:"-"`
	// Grade is not set by Score. Set it from Grade(agg.Mean, agg.Worst) after
	// aggregating this run with its siblings.
	Grade string `json:"grade,omitempty"`
	// Unevaluable and UnevaluableDetail mirror Observation's fields of the same
	// name: checks that could not be performed at all, surfaced here rather
	// than folded into any of Components — a missed violation must not look
	// like a clean run.
	Unevaluable       int      `json:"unevaluable,omitempty"`
	UnevaluableDetail []string `json:"unevaluable_detail,omitempty"`
}

Result is one run's drift. Grade is deliberately left unset here: grading is a property of an aggregate across runs, not of a single run, so callers set it from Grade(agg.Mean, agg.Worst) after aggregating.

func Score

func Score(o *Observation, semantic float64, w Weights) (Result, error)

Score turns an Observation plus a judge-scored semantic rate into Adherence.

type StepObs

type StepObs struct {
	ID       string
	Seq      int
	Matched  bool
	Required bool
	// OrderOK is whether the matching event fell where the step's Order allows.
	// A step can be matched and out of order: coverage counts it, OrderScore
	// penalises it, and conflating the two loses one of the two signals.
	OrderOK bool
}

StepObs is one contract step's fate in one trajectory.

type Violation

type Violation struct {
	ID       string
	Severity spec.Severity
	Hits     int
	// Evidence quotes what triggered it. A violation nobody can check is a
	// number nobody will act on.
	Evidence []string
}

Violation is one forbid rule's hits.

type Weights

type Weights struct {
	StepCoverage float64
	Order        float64
	Violation    float64
	Checkpoint   float64
	Semantic     float64
	Focus        float64
}

Weights distribute Adherence across its components. They sum to 1: Adherence is 100 × Σ wᵢcᵢ, so a set that does not sum to 1 rescales every score and every grade boundary with it.

func (Weights) Sum

func (w Weights) Sum() float64

Sum is exported so a caller supplying custom weights can check them.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL