decide

package
v1.2.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package decide is a decision system that is trained from its own record (docs/SPEC-NOVA-DECIDE.md). It has two halves over one record:

  • the decide half (this file, backend.go, jev.go, read.go, attempt.go, grade.go): a decision is a named schema (a set of typed questions) asked over one state text through a backend; every answer carries its probabilities.
  • the train half (record.go, calibrate.go): every decision made is appended to the record with its inputs, answers and usage; an outcome (a review's label, a gate's result) is attached to it when it is known; the bar a decision's answers are trusted at is calibrated from the decisions whose outcome is known.

A backend is a transport behind an interface: Jev (TypeSafe's System One model) is the first, and Fixed (answers from a file) is the one that needs no network. A decision never dials: the Jev backend sends through a Send function the caller injects, and HTTPSend (jev.go), the real one, is the only code here that opens a socket.

Index

Constants

View Source
const (
	ClassDone            = "done"
	ClassNothingToDo     = "nothing-to-do"
	ClassWrongScope      = "wrong-scope"
	ClassNoResult        = "no-result"
	ClassNeedsPro        = "needs-pro"
	ClassProviderFailure = "provider-failure"
)

The attempt's classes.

View Source
const (
	LabelLanded      = "landed"
	LabelLaterPrefix = "later-" // later-flash, later-pro, later-script
	LabelDropped     = "dropped"
)

The attempt's outcome labels, attached when its card lands or is dropped (docs/SPEC-SPRINT.md section 2, the attempt decision): the card landed at the decided attempt; at a later attempt, on the tier that landed it; or it was dropped.

View Source
const (
	BriefLanded   = "landed"   // landed on its first attempt
	BriefReworked = "reworked" // landed on a later attempt
	BriefDropped  = "dropped"  // dropped by the coordinator
)

The brief's outcome labels: how the card ended in the sprint.

View Source
const (
	Choice = "choice"
	Noul   = "noul"
)

Question types (SPEC-NOVA-DECIDE section 2). A choice picks one of named options and carries a probability per option; a noul is a yes/no over one statement and carries the probability that the statement is true.

View Source
const (
	RouteBounce  = "bounce"
	RouteStrings = "strings"
	RouteLand    = "land"
)

The routes a first read takes.

View Source
const (
	Flaky       = "flaky"
	Caused      = "caused"
	PreExisting = "pre-existing"
	RedAgain    = "red-again"
	// Green is a gate whose every failure was flaky and passed its rerun.
	Green = "green"
)

The gate's classes: the class question's options, the routes a failure takes, and the outcome labels a rerun attaches (a rerun that is red again attaches RedAgain when the base was not run, since it then tells caused from pre-existing by nothing).

View Source
const (
	GradeScript = "script"
	GradeFlash  = "flash"
	GradePro    = "pro"
)

The grades, named as the sprint's tiers are (a script card is the sprint's KIND: script).

View Source
const (
	ImportVerdict  = "verdict"
	ImportJudgment = "judgment"
	ImportReport   = "report"
)

The kinds an import records, and the decision name each is recorded under.

View Source
const (
	JevURL    = "https://api.typesafe.ai/v1/systemone"
	JevModel  = "jev-latest"
	JevSecret = "JEV_API_KEY" // the variable `nova-secrets exec --only JEV_API_KEY` sets
	// JevTimeout is how long one ask may take by default: a backend that does not answer
	// within it fails that ask, and never holds its caller (nova-decide's --timeout).
	JevTimeout = time.Minute
)

The Jev backend (SPEC-NOVA-DECIDE section 3): TypeSafe's System One model, asked by one POST of {state, model, questions} and answering {answers: {<name>: {type, choice, probabilities, confidence} | {type, noul}}, usage: {input_tokens, output_tokens}}.

View Source
const (
	VerbRework     = "rework"
	VerbDrop       = "drop"
	VerbWait       = "wait"
	VerbAck        = "ack"
	VerbAccept     = "accept"
	VerbRelease    = "release"
	VerbAskAnother = "ask-another"
)

The verbs a judgment can be answered with: the verb question's options.

View Source
const (
	MaxHistory = 40
	MaxLine    = 300
)

MaxHistory bounds the log lines a state carries, and MaxLine each line.

View Source
const (
	ActApply = "apply" // the verb is applied
	ActList  = "list"  // listed for the coordinator
)

The actions a judgment's card is given.

View Source
const (
	OutcomeLanded   = "landed"
	OutcomeDropped  = "dropped"
	OutcomeCameBack = "came-back"
)

The outcomes a judgment decision is labelled with, once the card's fate is known.

View Source
const (
	// JudgmentAnswerName is the record's decision name, and the file nova-sprint run
	// --decide keeps it in is <dir>/judgment-answer.jsonl.
	JudgmentAnswerName = "judgment-answer"
	// AnswerSchema names the record's shape: no schema is asked, the coordinator chose.
	AnswerSchema = "judgment-answer.v1"
	// LabelBounced is the outcome of an answer whose card had a read broken or a finish
	// failed again since.
	LabelBounced = "bounced"
)

The coordinator's own answers (SPEC-NOVA-DECIDE section 13, the judgment-answer record): every judgment answered with a verb that takes --answers, ack or wait is one record of kind judgment-answer, the label set the judgment decision is evaluated and trained on.

View Source
const (
	Land   = "LAND"
	Bounce = "BOUNCE"
	Unsure = "UNSURE"
)

The read's verdicts.

View Source
const (
	// ShadowName is the shadow records' decision name; nova-sprint run --decide keeps them in
	// <dir>/judgment-shadow.jsonl.
	ShadowName = "judgment-shadow"
	// ShadowMethod marks each answer of a shadow record.
	ShadowMethod = "shadow"
	// HighP is the probability at and above which agreement is also counted.
	HighP = 0.95
)
View Source
const (
	// ShadowReadName is the shadow reads' decision name; nova-sprint run --decide keeps them in
	// <dir>/read-shadow.jsonl.
	ShadowReadName = "read-shadow"
	// ScoreHeavy and ScoreReaders name what a ReadScore is scored against.
	ScoreHeavy   = "heavy"
	ScoreReaders = "readers"
)
View Source
const AttemptName = "attempt"

AttemptName is the attempt decision's name in the record.

View Source
const AttemptQuestion = "class"

AttemptQuestion is the attempt decision's one question.

View Source
const BriefDeadline = time.Minute

BriefDeadline bounds a whole brief batch, in nova-sprint add and nova-decide brief: past it what is unanswered is each card's error and the rest are recorded.

View Source
const BriefName = "brief"

BriefName is the brief decision's name in the record.

View Source
const BriefWidth = 8

BriefWidth is how many cards a brief batch asks at once, in nova-sprint add and as nova-decide brief's --width default.

View Source
const GateName = "gate"

GateName is the gate decision's name in the record.

View Source
const GradeExamplesPerClass = 10

GradeExamplesPerClass is how many examples of each class the prompt carries.

View Source
const GradeHeavy = "heavy"

GradeHeavy is the label of a card that needed the heavy tier; the grade's own options stay script, flash and pro, and the examples teach the line between them.

View Source
const GradeName = "grade"

GradeName is the grade decision's name in the record.

View Source
const GradeQuestion = "grade"

GradeQuestion is the grade decision's one question.

View Source
const JudgmentName = "judgment"

JudgmentName is the judgment decision's name in the record.

View Source
const MaxGateFailures = 8

MaxGateFailures is how many failing tests of one gate are asked about; a failure past it is caused, unasked (a gate with that many red tests is the change's).

View Source
const MaxResultBytes = 16 << 10

MaxResultBytes bounds the RESULT.md an attempt's state carries: a result is a page of text, and the state rides to the sprint's server with the finish.

View Source
const OutsidePaths = "outside_paths"

OutsidePaths is the one class not asked on its own: its p is 1 - p(inside_paths), the read's own question, so the same evidence is never asked twice.

View Source
const ReadName = "read"

ReadName is the read decision's name in the record.

View Source
const ScoreName = "score"

ScoreName is the score decision's name in the record.

View Source
const Unnamed = "unnamed"

Unnamed is the findings row of the decisions whose p(defect) meets the bar while no class does: a defect no class names yet, the material for a new class.

View Source
const Unset = 2.0

Unset is a gate bar the sprint row leaves empty: no probability reaches it.

Variables

View Source
var AnswerVerbs = []string{"accept", "rework", "recut", "return", "drop", "wait", "hold", "ack", "ask-another", "release"}

AnswerVerbs is every verb that answers a judgment, in the order the spec lists them.

View Source
var ErrUnknown = errors.New("no decision with that id in the record")

ErrUnknown is an outcome for an id the record holds no decision for.

View Source
var Fixes = map[string]string{
	"own": "",
	"from-tip": "Your head does not merge onto the sprint tip: a file you touched was also changed by a card that landed after your base. " +
		"Start again from the current tip of the sprint branch, redo only the lines your card lists, and if a ledger must change, " +
		"lower its ceiling from the value at the tip by exactly your own count.",
	"pro-tier": "attempts ended without a RESULT.md on the flash tier; run on the pro tier (RESULT line tier: pro); the task is unchanged",
	"retry":    "the attempt ended for a cause outside the card (a provider, a member, staging); run it again; the task is unchanged",
}

The fixes a rework that takes --fix carries, and their texts.

View Source
var GradeExampleClasses = []string{GradeFlash, GradePro, GradeHeavy}

GradeExampleClasses are the example labels in prompt order: landed on flash within two attempts, needed pro, needed heavy.

ImportKinds is every kind, in the order a result prints them.

View Source
var Kinds = map[string]string{
	"a reader found it broken":                  "broken",
	"work came back failed":                     "failed",
	"a primary is blocked on something dropped": "blocked",
	"stalled":                            "stalled",
	"stream stopped: conflict on a card": "conflict",
	"a work card is past its deadline":   "deadline",
	"cannot ask":                         "cannot-ask",
	"ready to accept":                    "ready",
	"a card reached its bound":           "bound",
}

Kinds maps each routine judgment type, as the inbox names it, to its short name: the kinds nova-sprint answer asks the decision for. Every other type is left.

View Source
var Reasons = map[string]string{
	"already-done":    "the listed edits were already made, by an earlier attempt or by a card that landed first: done, not failed",
	"cannot-be-done":  "the card cannot be done as written: its brief is wrong, or names lines that are not there",
	"need-dropped":    "what it needs was dropped, and it cannot run without it",
	"reads-exhausted": "no reader can read it on the installed build (reads exhausted); added again after the fix",
}

The reasons a drop gives, and their texts.

Verbs is every verb, in the order a row prints them.

Functions

func AckReason

func AckReason(p float64) string

AckReason is the reason an applied ack gives: the decision's, with its probability.

func AnswerOutcome

func AnswerOutcome(d Decision, now CardMark) (label, note string)

AnswerOutcome is the outcome the card's mark now says of the answer d, "" while the card stands with nothing new: landed, dropped (off the table), or bounced (more reads found it broken, or more finishes failed, than when it was answered).

func AttachBriefs

func AttachBriefs(record string, ends map[string]End, at time.Time) (attached int, failed map[string]error, err error)

AttachBriefs attaches each card's end to its brief decision by the decision's exact id, the op add stored on the card (op -> End), in one write under the record's lock. An op the record does not hold, or a record that does not exist, is that op's error in failed and no record is made; the same label again changes nothing; another label is that op's *ConflictError; the other ops are attached.

func AttemptLabel

func AttemptLabel(attempt, landed int, tier string) string

AttemptLabel is the outcome of an attempt decided at attempt, for a card that landed at landed on tier (landed 0: dropped).

func AttemptOf

func AttemptOf(op string) (card string, attempt int, ok bool)

AttemptOf is the card and attempt an attempt decision's op id names (AttemptOp); ok false when it is not `<card>@<attempt>.<hex>`.

func AttemptOp

func AttemptOp(card string, attempt int, state string) string

AttemptOp is a take's attempt decision id: `<card>@<attempt>.<12 hex of the state>`. It is per attempt and state, not per take: two takes of one attempt that ended with the same state (the same reason line and RESULT.md) are one decision.

func AttemptState

func AttemptState(brief, result, reason string) string

AttemptState is the text the attempt is asked over: the card's brief, the child's RESULT.md (cut to MaxResultBytes) and the member's reason line, each under its own heading.

func BriefOp

func BriefOp(card, brief string) string

BriefOp is a card's brief decision's id in the record: <card>@brief-<8 hex>, the hex of the schema and the brief, so a brief rewritten after a refusal, or asked under a reworded schema, is a decision of its own and never a conflict with the one before it.

func BriefState

func BriefState(card string) string

BriefState is the text the brief is asked over: the card, under its heading, then the frame the sprint gives every child.

func CardFilePaths

func CardFilePaths(dir string) ([]string, error)

CardFilePaths is the card files of a directory as nova-sprint add --brief-dir reads them: every *.md entry that is not a directory, in byte order of name, none below.

func CardFiles

func CardFiles(path string) (map[string]string, error)

CardFiles is the cards a path names, id -> brief: a file is one card, a directory its CardFilePaths (add --brief-dir's cards); a card's id is its file's name without .md, and its brief the file's text with one trailing newline cut, as nova-sprint add stores it.

func CardOf

func CardOf(id string) string

CardOf is the card a score's id names: the part before @landed@, else the whole id.

func CardPaths

func CardPaths(card string) []string

CardPaths is a card's PATHS globs: its first `PATHS:` line (cardhdr.KeyValue), split on commas; nil when it names none.

func ChoiceOf

func ChoiceOf(d Decision, question string) (string, float64)

ChoiceOf is a choice's chosen option and its probability (Top is the score decision's).

func ClassP

func ClassP(d Decision) map[string]float64

ClassP is the p of every class a score decision gives, outside_paths from inside_paths.

func DiffSummary

func DiffSummary(diff string) string

DiffSummary is the files a unified diff changes, one per line with the lines it adds and removes: `<path> +<added> -<removed>`, a rename as `<old> -> <new>`.

func EvidencePaths

func EvidencePaths(text string) []string

EvidencePaths is the paths a judgment's text names (a report, a finding's file), each once, in the order named, a trailing full stop and a :line cut.

func Finding

func Finding(d Decision, bars Bars, files []string) string

Finding is a first read's report: p(defect) and the bar it met, the files the diff changes, and the five answers. It names a file, so a bounce is a broken read with a finding (docs/SPEC-CARD-CONTRACT.md section 3).

func GateOp

func GateOp(op string, f Failure) string

GateOp is a failure's op id under the gate's op: <op>/<pkg>.<Test>.

func GateState

func GateState(in GateInput, i int) string

GateState is the text one failure is asked over: the failure and its lines, the base, the gate's other failures by name, the card's PATHS and its diff summary, each under its own heading.

func GradeLabel

func GradeLabel(tier string) string

GradeLabel is the grade's outcome: the tier that landed the card, or dropped ("" tier).

func GradeOp

func GradeOp(card, state string) string

GradeOp is a card's grade decision id: `<card>@grade.<12 hex of the state>`.

func GradeState

func GradeState(brief string) string

GradeState is the text the grade is asked over: the card's brief alone, under its heading.

func GradeStateWith

func GradeStateWith(brief string, shots []Example) string

GradeStateWith is the grade's state with the examples ahead of the card: one line each (heading, PATHS, KIND, label), then GradeState(brief) whole. No examples is GradeState.

func HistoryOf

func HistoryOf(log string) []string

HistoryOf is a card's log as a judgment state carries it: the lines nova-sprint log --card prints, less its LOG OK line and the cost and field-change lines (the record of a read's price or a field set tells the decision nothing), each line trimmed.

func JudgmentOutcome

func JudgmentOutcome(col string, placed, otherJudgment bool) string

JudgmentOutcome is the outcome a card's state says, "" while it is not known: landed, dropped (off the table), or came back (another judgment open on it).

func JudgmentState

func JudgmentState(in JudgmentInput) string

JudgmentState is the text the judgment decision is asked over: the judgment, the verbs allowed, and the card's last MaxHistory log lines, each cut at MaxLine.

func Keys

func Keys(fs []Failure) []string

Keys is the failures' keys.

func LandLabel

func LandLabel(attempt string) (label, note string)

LandLabel is the outcome a card's landing attaches to its brief: landed at attempt 1, reworked at a later one, with the attempt in the note.

func Names

func Names(fs []Failure) string

Names is the failures' tests, comma-separated, a build failure as its package.

func Op

func Op(base, state string) string

Op is a decision's op id over a state: base (the card and what it decides of it, `<card>@<attempt>`, `<card>@grade`), then the first twelve hex of the state's SHA-256, so a finish reported again replays its decision with no ask, and a card id that comes back with another brief (a clear, a brief replaced) is a new decision, never a conflict.

func PDefect

func PDefect(d Decision) float64

PDefect is the read's p(defect).

func ParseBar

func ParseBar(field, raw string) (bar float64, ok bool, err error)

ParseBar reads one bar as the sprint row holds it, a decimal as text: empty is no bar (ok false), and anything else that is not a probability is an error naming the field.

func ParseJudgmentBar

func ParseJudgmentBar(raw string) (float64, error)

ParseJudgmentBar reads a judgment bar: a probability (ParseBar is a sprint row bar's).

func PaymentRefusal

func PaymentRefusal(texts ...string) bool

PaymentRefusal says a text carries a provider's refusal for want of payment. Such a judgment is never answered by a decision: a payment is the owner's.

func ReadLog

func ReadLog(r io.Reader) (map[string]*CardFacts, error)

ReadLog reads a `nova-sprint log --json` export ({"lines":[...]}) into each primary's facts: its last placement on the work table, and the last text under each of its cost_record keys (SPEC-NOVA-DECIDE section 11).

func ReadState

func ReadState(card, diff, rule string) string

ReadState is the text the read is asked over: the card, the rule when one is given, then the diff, each under its own heading.

func RecordAct

func RecordAct(path string, a Act) error

RecordAct appends a's line against its decision: a step of applying it. A decision the record does not hold is ErrUnknown, and nothing is written.

func Replays

func Replays(have Decision, s Schema, state string) error

Replays says the recorded decision is the one an ask of s over state under its id would make, so the ask is answered from the record; else a ConflictError.

func ScoreOp

func ScoreOp(card, head string) string

ScoreOp is a landed diff's id in the record: the card at the head that landed.

func Set

func Set(bar float64) bool

Set says the bar routes: it is a probability, not Unset.

func Settle

func Settle(record, id, verdict, note string, at time.Time) error

Settle attaches a strings read's verdict to the first read it followed: ok is LAND, broken is BOUNCE. Any other verdict is no outcome and attaches nothing.

func SettleGate

func SettleGate(record string, r GateResult, red map[string]bool, baseRed map[string]bool, at time.Time) (string, error)

SettleGate attaches each rerun failure's result as its decision's outcome: Flaky when the rerun passed; red again, PreExisting when it was red at the base, Caused when the base was run and green, RedAgain when the base was not run. It returns the gate's route after the rerun: Caused when a rerun failure is red again, else PreExisting when a failure was routed so, else Green.

func ShadowKey

func ShadowKey(d Decision) string

ShadowKey joins a shadow record to the real answer: the judgment's note and its card.

func ShadowPending

func ShadowPending(shadows, real []Decision) int

ShadowPending is how many shadow records have no real answer yet.

func ShadowReadOutcome

func ShadowReadOutcome(d Decision, now CardMark) (label, note string)

ShadowReadOutcome is the readers' outcome of the shadow read d from the card's mark now: Bounce when more reads have found the card broken than when it was shadow-read, Land when the card landed without that, "" while it stands with nothing new, and Dropped when it left the table (a card dropped has no outcome to score).

func Sum

func Sum(b []byte) string

Sum is the hex SHA-256 of b: how the record names an input without holding a second copy of a file.

func Top

func Top(d Decision) (string, float64)

Top is a score decision's highest class and its p, the first in class order on a tie.

func VerbOf

func VerbOf(decision string) string

VerbOf is the verb a printed decision makes, "" for a decision that is no verb the judgment decision chooses (look at the card, resolve and resume, reader add).

Types

type Act

type Act struct {
	ID  string `json:"id"`
	Act string `json:"act"`
	Op  string `json:"op,omitempty"`
	At  string `json:"at"`
}

Act is one step of applying a decision, recorded around the verbs it runs: "applying" with the operation id those verbs carry, before them, then "applied" or "refused" after. A decision's acts are appended, never rewritten; its last says where applying it stands, so a writer stopped between the two finds "applying" and its op.

type Answer

type Answer struct {
	Type       string             `json:"type"`
	Value      string             `json:"value"`
	P          map[string]float64 `json:"p"`
	Confidence float64            `json:"confidence,omitempty"`
	Method     string             `json:"method,omitempty"`
}

Answer is one typed answer. P is the probability of each option for a choice, and of "yes" for a noul; Value is the chosen option, or "yes" or "no" at 0.5 for a noul. Confidence and Method are a choice the wire answered with no probabilities: the confidence, and how it was read. Neither is a probability of correctness, and both are omitted when empty (SPEC-NOVA-DECIDE section 3).

func (Answer) Prob

func (a Answer) Prob(option string) float64

Prob is the probability the answer gives to option (for a noul, "yes"). It reads P only: a confidence is not a probability (SPEC-NOVA-DECIDE section 3).

type AnswerInput

type AnswerInput struct {
	Note   string   // the judgment's notification id
	Kind   string   // the judgment's type, as the inbox names it
	Text   string   // the judgment note's text
	Card   string   // the card answered
	Verb   string   // the verb given
	Reason string   // --reason (or the wait's duration), "" when none
	Fix    string   // --fix, "" when none
	Actor  string   // who answered
	Mark   CardMark // the card as it stood after the verb ran
}

AnswerInput is one judgment answered for one card.

type Backend

type Backend interface {
	Name() string
	Ask(ctx context.Context, s Schema, state string) (map[string]Answer, Usage, error)
}

Backend answers a schema over a state. It is the transport of a decision: the system above it (the record, the calibration) is the same whichever backend answered.

type BackendError

type BackendError struct {
	Backend string
	Err     error
}

BackendError is a backend that gave no answer within the schema: nothing was recorded.

func (*BackendError) Error

func (e *BackendError) Error() string

func (*BackendError) Unwrap

func (e *BackendError) Unwrap() error

type Bar

type Bar struct {
	At              float64
	Caught, Bounced int // positives flagged; negatives flagged
}

Bar is one threshold: decisions scoring at or above it are flagged.

type Bars

type Bars struct {
	Bounce float64 `json:"bounce"`
	Review float64 `json:"review"`
}

Bars are the two bars on p(defect): Bounce at or above, Review below.

func ParseBars

func ParseBars(bounce, review string) (Bars, error)

ParseBars reads the bars as the sprint row holds them, decimals as text. Both are probabilities and the review bar is at most the bounce bar; every problem is named.

func (Bars) Route

func (b Bars) Route(pDefect float64) string

Route is where a read with this p(defect) goes.

type Brief

type Brief struct {
	Converges float64  // p(converges)
	Minutes   string   // the minutes option chosen
	Ambiguous string   // the ambiguous_step option chosen: none, step-<n> or unnumbered
	Failed    []string // the questions the card fails: a need under 0.5 as name(p), a step as ambiguous_step:step-<n>(p)
}

Brief is what a brief decision says of its card.

func BriefOf

func BriefOf(d Decision) Brief

BriefOf reads a brief decision.

func (Brief) Line

func (b Brief) Line() string

Line is the brief as one line of fields: p_converges, minutes and the failed questions ("-" when none), as add and lint --decide print it, marked uncalibrated=true: p(converges) is a rank no outcome of the brief record has yet supported a bar on (SPEC-NOVA-DECIDE section 14).

type BriefBar

type BriefBar struct {
	At  float64
	Set bool
}

BriefBar is the bar on p(converges) a card is added at: Set is false when the sprint row's decide_brief_bar is empty (report only).

func ParseBriefBar

func ParseBriefBar(raw string) (BriefBar, error)

ParseBriefBar reads decide_brief_bar as the sprint row holds it: empty is no bar, else a probability.

func (BriefBar) Refuses

func (bar BriefBar) Refuses(b Brief) bool

Refuses says the bar refuses a card whose brief is b: it is set and p(converges) is under it.

type BucketRow

type BucketRow struct {
	Grade, Bucket           string
	N, FirstFailed, Landed2 int
}

BucketRow is one line of the calibration table: the cards graded Grade, dealt on flash, whose p for the grade fell in Bucket.

type Calibration

type Calibration struct {
	Decision, Schema, Question, Option string
	Positives, Negatives               []float64 // the score of each labelled decision
	Skipped                            int       // labelled otherwise, unlabelled, or asked under another schema
}

Calibration (SPEC-NOVA-DECIDE section 5) is how well one answer of one decision separates the outcomes, read from the record: the decisions of that name and schema whose outcome is known, scored by the probability the answer gave to one option (a noul's "yes", or a choice's named option), split into positives (the outcomes the answer should catch) and negatives.

func Calibrate

func Calibrate(ds []Decision, decision, question string, positive, negative []string) (Calibration, error)

Calibrate scores every decision named decision whose outcome label is in positive or negative. The schema is the newest one the record holds for that decision: answers to other questions are never pooled. question is "<name>" (a noul, scored by its yes) or "<name>=<option>" (a choice). A label of words joined by + is positive (or negative) when one of its words is listed.

func (Calibration) AUC

func (c Calibration) AUC() float64

AUC is the probability that a positive scores above a negative (ties count half): 0.5 is no separation, 1 is perfect.

func (Calibration) At

func (c Calibration) At(bar float64) Bar

At is the threshold row at bar.

func (Calibration) CatchAll

func (c Calibration) CatchAll() Bar

CatchAll is the highest bar that still flags every positive (the lowest positive score) and the row there: the bar the record supports when no positive may pass.

type CardFacts

type CardFacts struct {
	State       string // landed, dropped, or the column it sits in
	Start       string
	MaxAttempt  int
	FirstFailed bool
	Escalated   bool
	// contains filtered or unexported fields
}

CardFacts is what the log holds of one card: where it ended, the tier of its first attempt ("" when it was never dealt or attempt 1 is before the log), the highest attempt number, whether attempt 1 failed, and whether a pro attempt ran.

type CardMark

type CardMark struct {
	Placed, Landed, Dropped bool
	Broken, Failed          int
}

CardMark is what an outcome reads of a card: whether it stands on the table, has landed or was dropped, and how many reads found it broken and how many finishes failed.

type Chosen

type Chosen struct {
	Verb   string  `json:"verb"`
	P      float64 `json:"p"`
	Fix    string  `json:"fix,omitempty"`    // a rework's fix option, when its command takes one
	Reason string  `json:"reason,omitempty"` // a drop's reason option
	Act    string  `json:"act"`
	Why    string  `json:"why,omitempty"` // why it is listed
}

Chosen is what the decision chose for one card and what is done with it.

func Choose

func Choose(answers map[string]Answer, allowed []string, bar float64) Chosen

Choose reads the decision's answers for one card: the verb and its probability, the fix and the reason; the verb is applied when it is allowed, is not drop or release, and its probability is at or above bar. Anything else is listed, saying why.

type Class

type Class struct {
	Name, Escalations, Statement string
}

Class is one escalation class: its name in the record, the review escalations it stands for, and the statement asked (a noul; yes is the defect present).

func Classes

func Classes() []Class

Classes is every escalation class, in the order the score asks and prints them.

type Cluster

type Cluster struct {
	Class string   `json:"class"`
	Count int      `json:"count"`
	Cards []string `json:"cards"`
}

Cluster is one class of the findings: how many score decisions gave it a p at or above the bar, and their cards in id order.

func FindingsSkipped

func FindingsSkipped(ds []Decision, since time.Time, bar float64) (clusters []Cluster, scored int, skipped []string)

FindingsSkipped clusters the score decisions made at or after since by every class each gives a p at or above bar, plus Unnamed for p(defect) at or above bar with no class there; most cards first, then class order; scored is the decisions in the window. It also returns the ids, in record order, of the score decisions it left out because their at is not RFC 3339: such a decision cannot be placed in the window, and the caller says so rather than count it as nothing (security#79 finding 3). A decision of another kind is not a score and is never named.

type ConflictError

type ConflictError struct{ What string }

ConflictError is an id already recorded with other content: an op id reused over another state, or a decision already labelled otherwise.

func (*ConflictError) Error

func (e *ConflictError) Error() string

type Decided

type Decided struct {
	Value string
	P     float64
	Op    string
}

Decided is a decision as it rides on a card: the chosen option, its probability and the decision's op id, one line: `<option> p=<p> op=<op>`.

func AttemptDecided

func AttemptDecided(d Decision) (Decided, error)

AttemptDecided is the attempt decision as it rides with a finish: its class, the class's probability and its op id. An answer that is not one of the six classes is an error.

func ParseDecided

func ParseDecided(s string) (Decided, bool)

ParseDecided reads a card's line (Decided.String); ok false when it is not one.

func (Decided) Over

func (d Decided) Over(raw string) bool

Over says the decision on a card is at or above the bar the card carries (raw, as the sprint row held it when the card was dealt or graded): no bar, or one that does not parse, is never over.

func (Decided) String

func (d Decided) String() string

String is the card's line.

type Decision

type Decision struct {
	ID       string            `json:"id"`
	Decision string            `json:"decision"` // the schema's name
	Schema   string            `json:"schema"`   // the schema's hash
	Backend  string            `json:"backend"`
	At       string            `json:"at"` // RFC 3339, UTC
	Inputs   map[string]string `json:"inputs"`
	State    string            `json:"state"` // the exact text asked over: the training input
	Answers  map[string]Answer `json:"answers"`
	Usage    Usage             `json:"usage"`
	Outcome  *Outcome          `json:"outcome,omitempty"` // folded in on load; never written on this line
	Acts     []Act             `json:"acts,omitempty"`    // folded in on load, in order; never written on this line
}

Decision is one decision made: what was asked, over what, by which backend, and what it answered.

func AnswerDecision

func AnswerDecision(in AnswerInput, at time.Time) Decision

AnswerDecision is the record of a judgment answered (the judgment-answer record): the judgment's state as the judgment decision is asked it, the verb as the answer, and as inputs the kind, the evidence paths, the --reason and --fix text, the actor and the card's counters, which AnswerOutcome reads the later ones against. Its id is per judgment, card and answer, so the same answer given again is the same record.

func Append

func Append(path string, d Decision) (recorded *Decision, err error)

Append records d unless its id is already there: the same id over the same state returns the recorded decision (an op retried), over another state it is a ConflictError.

func Attach

func Attach(path string, o Outcome) (d Decision, changed bool, err error)

Attach records o against its decision and returns the decision with it. The same label again changes nothing (changed is false); another label is a ConflictError naming both.

func AttemptDecision

func AttemptDecision(ctx context.Context, b Backend, card string, attempt int, brief, result, reason string, at time.Time) (Decision, error)

AttemptDecision asks the attempt decision of a take of card at attempt through b and returns it as the record keeps it, with its op id (AttemptOp), unrecorded: the sprint's member asks it, and the finish carries it to the record the sprint's server keeps (docs/SPEC-SPRINT.md section 2). A backend that fails is a BackendError.

func Find

func Find(ds []Decision, id string) *Decision

Find is the decision with id, or nil.

func FirstRead

func FirstRead(ctx context.Context, b Backend, bars Bars, card, diff, record, id string, at time.Time) (Decision, string, error)

FirstRead is the read decision over a card and its diff, recorded under id (the read card's id), and the route its p(defect) takes at the bars.

func Load

func Load(path string) ([]Decision, error)

Load reads the record; a record that does not exist yet is empty. A line that does not parse, a decision id seen twice, an outcome for an id with no decision before it, or an answer whose probability is outside [0, 1] or not a number, is an error naming the line (SPEC-NOVA-DECIDE section 4).

func Make

func Make(ctx context.Context, b Backend, s Schema, state, record, id string, inputs map[string]string, at time.Time) (d Decision, existing bool, err error)

Make asks s over state through b and appends the decision to the record under id. An id the record holds over the same decision, schema and state is that decision (existing), and nothing is asked; over another it is a ConflictError.

func ParseAttempt

func ParseAttempt(raw []byte) (Decision, error)

ParseAttempt reads an attempt decision as a finish carries it (one JSON record line's decision) and holds it to the attempt schema: its name, its schema's hash, its answers (Check) and its op id over its state (AttemptOp). A decision that does not fit is an error, and the server records nothing of it.

func Score

func Score(ctx context.Context, b Backend, card, diff, record, id string, at time.Time) (Decision, error)

Score asks the score decision over a landed card and its diff and records it under id (Make: an id already recorded over the same card and diff is answered from the record).

func ShadowAsk

func ShadowAsk(ctx context.Context, b Backend, in JudgmentInput, note, record string, at time.Time) (d Decision, existing bool, err error)

ShadowAsk asks the judgment decision over in through b and appends the answer, marked shadow, to record under an id per note, card and state. A judgment already shadowed is returned as recorded (existing) and nothing is asked.

func ShadowReadAsk

func ShadowReadAsk(ctx context.Context, b Backend, card, brief, diff string, broken int, record string, at time.Time) (d Decision, existing bool, err error)

ShadowReadAsk asks the read decision over the card's brief and diff through b and appends the answer, marked shadow, to record under an id per card and state. A card already shadow-read at that diff is returned as recorded (existing) and nothing is asked. broken is the card's count of broken reads now, which ShadowReadOutcome reads the later ones against.

type End

type End struct{ Label, Note string }

End is how a card ended in the sprint: the outcome label its brief is given (BriefLanded, BriefReworked, BriefDropped) and a note.

type Example

type Example struct {
	Card    string `json:"card"`
	Heading string `json:"heading"`
	Paths   string `json:"paths"`
	Kind    string `json:"kind"`
	Label   string `json:"label"`
}

Example is one landed card of the record: its brief heading, its PATHS line, its KIND and the label of its outcome. Never the brief's whole text.

func ParseExamples

func ParseExamples(raw []byte) ([]Example, error)

ParseExamples reads an examples file, one JSON object per line ({card, heading, paths, kind, label}); every bad line is named in one error.

func PickExamples

func PickExamples(pool []Example, heldOut map[string]bool, seed string, per int) ([]Example, error)

PickExamples chooses per examples of each class from pool, deterministically: held-out cards leave the pool, a card's rank is the SHA-256 of seed and its id (so neither the pool's order nor its size moves a card's rank), and the lowest ranks are taken. The result is in class order, then rank. A class with fewer than per examples left is an error naming it.

type Failure

type Failure struct {
	Pkg   string   `json:"pkg"`
	Test  string   `json:"test,omitempty"`
	Lines []string `json:"lines,omitempty"`
}

Failure is one failing test of a gate's output: its package as go test names it, the top-level test, and the first lines it printed. A build failure has no test.

func ParseGateOutput

func ParseGateOutput(out string) []Failure

ParseGateOutput reads go test's output (plain or -v) into its failures, in the order go printed them: each top-level test with a `--- FAIL:` line (a subtest's failure is its test's), its package from the `FAIL <pkg>` line that follows, and its first lines (the indented lines after it, else, under -v, what it printed after its `=== RUN`); a package that failed to build or with no failing test named (a timeout names its test under "running tests:") is one failure with the package's lines. Output that is no go test's has no failures.

func (Failure) Key

func (f Failure) Key() string

Key is the failure's name in an op id and a base-red list: <pkg>.<Test>, or the package alone for a build failure.

type Fixed

type Fixed struct {
	Table map[string]FixedAnswer
}

Fixed is the backend that answers from a table: question name -> the answer it gives, whatever the state. It needs no network and no key, so a first run, a test, or a dry comparison against a recorded answer set runs anywhere (SPEC-NOVA-DECIDE section 3).

func ParseFixed

func ParseFixed(raw []byte) (Fixed, error)

ParseFixed reads a Fixed table: {"<question>": {"choice": ..., "p": {...}} | {"noul": p}}.

func (Fixed) Ask

func (f Fixed) Ask(_ context.Context, s Schema, _ string) (map[string]Answer, Usage, error)

Ask answers every question of s from the table; a question the table does not answer is an error naming it, never a guess.

func (Fixed) Name

func (Fixed) Name() string

Name is the backend as the record names it.

type FixedAnswer

type FixedAnswer struct {
	Choice string             `json:"choice,omitempty"`
	P      map[string]float64 `json:"p,omitempty"`
	Noul   *float64           `json:"noul,omitempty"`
}

FixedAnswer is one row of a Fixed table: a choice's option and its probabilities, or a noul's probability of yes.

type GateBars

type GateBars struct {
	Flaky       float64 `json:"flaky"`
	PreExisting float64 `json:"pre_existing"`
}

GateBars are the two bars on a failure's class probabilities: at or above Flaky it is rerun once, at or above PreExisting it is reported pre-existing. A bar that is unset is Unset, above every probability, so its route is never taken: the decision is recorded and shown, and nothing is rerun or reclassified on it.

func ParseGateBars

func ParseGateBars(flaky, preExisting string) (GateBars, error)

ParseGateBars reads the bars as the sprint row holds them, decimals as text: each a probability, or empty for Unset (its route is never taken; the sprint row's default); two set bars sum above 1, so no failure meets both. Every problem is named.

func (GateBars) Route

func (b GateBars) Route(d Decision) string

Route is where a failure with this decision goes: Flaky (rerun once), PreExisting, or Caused.

type GateCall

type GateCall struct {
	Failure  Failure
	Decision *Decision
	Route    string
	Existing bool // the decision was in the record already, and nothing was asked
}

GateCall is one failure of a gate and what was decided about it: its decision (nil for one not asked: a build failure, or one past MaxGateFailures) and its route.

type GateInput

type GateInput struct {
	Failures []Failure
	BaseRed  map[string]bool
	Paths    []string
	Diff     string // DiffSummary of the card's diff
}

GateInput is what one gate's decisions are asked over: its failures, the keys of those red at the base (nil when the base was not run), the card's PATHS, and its diff summary.

type GateResult

type GateResult struct {
	Calls []GateCall
	Route string
}

GateResult is a gate's decisions and the route the gate takes: Caused when any failure is, else Flaky when any is (those are rerun), else PreExisting; "" when the gate had no failure, so nothing was decided (never PreExisting).

func Gate

func Gate(ctx context.Context, b Backend, bars GateBars, in GateInput, record, op string, at time.Time) (GateResult, error)

Gate asks the gate decision of each failure of in (up to MaxGateFailures; a build failure is caused, unasked), records each under GateOp(op, failure), and routes the gate at the bars. A backend that fails is an error and the gate is the card's, as before. No failure is no decision and no route: nothing is asked or recorded.

func (GateResult) Classes

func (r GateResult) Classes() string

Classes is what each asked failure was classed, for a line that shows the decisions: `<Test>:<class>:<p>` comma-separated, p the class's probability to two places; an unasked failure is `<Test>:unasked`.

func (GateResult) PreExistingTests

func (r GateResult) PreExistingTests() []Failure

PreExistingTests is the failures routed PreExisting.

func (GateResult) Rerun

func (r GateResult) Rerun() []Failure

Rerun is the failures the gate reruns: those routed Flaky.

type GradeRow

type GradeRow struct {
	Grade, Dealt                             string
	N, Landed2, Landed, ToPro, Dropped, Open int
}

GradeRow is one line of the grade-by-dealt table: the cards Jev graded Grade that were first dealt on Dealt.

type GradeScore

type GradeScore struct {
	Decisions, Cards, NoLog int // grade decisions in the day, the cards they grade, cards the log lacks
	Rows                    []GradeRow
	Buckets                 []BucketRow
}

GradeScore is the day's score.

func ScoreGrades

func ScoreGrades(ds []Decision, facts map[string]*CardFacts, from, to time.Time) GradeScore

ScoreGrades scores the grade decisions whose At is in [from, to): each card by its newest grade in the window, against facts. A card the log does not hold, or never dealt to the fleet, is counted and left out of the tables (a friend's card has no tier to grade). Open is a card neither landed nor dropped; Landed2 is a card landed by its second attempt.

type ImportCount

type ImportCount struct{ New, Existing int }

ImportCount is how many items of a kind were recorded now and how many were there already.

type ImportResult

type ImportResult struct {
	Kinds      map[string]ImportCount
	Unanswered int
}

ImportResult is the count per kind, and the judgments the log has no answer for (they are not recorded: an unlabelled item is not a label).

func Import

func Import(path string, src ImportSources, now time.Time) (ImportResult, error)

Import records every item of src in the record at path, once: an item whose id is there already is counted and left. The whole import is one write under the record's lock.

type ImportSources

type ImportSources struct {
	Verdicts, Judgments, Log, Reports string
}

ImportSources names what to import; an empty field is a source not read. Verdicts and Reports are globs of files; Judgments is a directory of judgment files, which needs Log, a nova-sprint log --json export, to hold the answers.

type Item

type Item struct {
	ID     string
	State  string
	Inputs map[string]string
}

Item is one decision of a batch: its op id, the state it is asked over, and the inputs the record names.

func BriefItem

func BriefItem(card, brief string) Item

BriefItem is a card's brief as one item of a batch (MakeAll).

type Jev

type Jev struct {
	Model string
	Send  Send
}

Jev is the Jev backend over an injected Send.

func JevHTTP

func JevHTTP(key string, timeout time.Duration) Jev

JevHTTP is the Jev backend over the real transport with key, each ask bounded by timeout: what the sprint's decide read and answer ask through.

func (Jev) Ask

func (j Jev) Ask(ctx context.Context, s Schema, state string) (map[string]Answer, Usage, error)

Ask encodes the request, sends it, and decodes the answers.

func (Jev) Name

func (j Jev) Name() string

Name is the backend as the record names it.

type JudgmentInput

type JudgmentInput struct {
	Kind    string   // the judgment's type, as the inbox names it
	Text    string   // the judgment note's text
	Card    string   // the card decided
	Cards   int      // how many cards the judgment holds
	History []string // the card's log, oldest first
	Allowed []string // the verbs the judgment prints
}

JudgmentInput is what one judgment decision is asked over.

type KindAgreement

type KindAgreement struct {
	Kind      string
	Count     int
	Agree     int
	Pct       int
	HighCount int
	HighAgree int
	HighPct   int
}

KindAgreement is how a judgment kind's shadow answers agreed with the real ones: Count pairs joined, Agree of them the same verb, Pct that as a percent; HighCount of the pairs had the shadow verb at p HighP or above, HighAgree of those agreed, HighPct as a percent.

func ShadowAgreement

func ShadowAgreement(shadows, real []Decision) []KindAgreement

ShadowAgreement joins each shadow record to the real judgment-answer record of the same note and card and scores the pairs per judgment kind, kinds in name order. A real answer is the first for its note and card; verbs are compared as recorded.

type LogLine

type LogLine struct {
	Card    string            `json:"card"`
	Primary string            `json:"primary"`
	Table   string            `json:"table"`
	To      string            `json:"to"`
	Removed bool              `json:"removed"`
	Set     map[string]string `json:"set"`
}

LogLine is one line of `nova-sprint log --json`, the fields the score reads.

type Made

type Made struct {
	Decision
	Existing bool
	Err      error
}

Made is one item's decision. Existing says the record held it and nothing was asked. Err is why it could not be made (its Decision then holds only the item's ID and Inputs): a *BackendError (nothing recorded) or a *ConflictError (the op id is recorded over another decision, schema or state).

func Briefs

func Briefs(ctx context.Context, b Backend, cards map[string]string, record string, at time.Time, width int, wait time.Duration) ([]Made, error)

Briefs makes the brief decision of every card (id -> brief text) as one batch, in id order, at most width asks at a time, each within wait.

func MakeAll

func MakeAll(ctx context.Context, b Backend, s Schema, items []Item, record string, at time.Time, width int, wait time.Duration) ([]Made, error)

MakeAll makes the decision of s over each item, in the items' order, each ask within wait (0 is ctx's own bound); once ctx is done no further item is asked and each is that item's Err. An empty record keeps nothing: no decision is read from it or written to it. The error is the record's (it cannot be read or written); a decision that could not be made is its item's Err, and the others are made and recorded.

type Outcome

type Outcome struct {
	ID    string `json:"id"`
	Label string `json:"label"`
	Note  string `json:"note,omitempty"`
	At    string `json:"at"`
}

Outcome is what turned out to be true about a decision: a review's label, a gate's result. It is attached once; a second, different label is a conflict.

type Question

type Question struct {
	Type         string            `json:"type"`
	Instructions string            `json:"instructions"`
	Criteria     map[string]string `json:"criteria,omitempty"` // choice only: option -> what it means
}

Question is one typed question of a schema.

type ReadScore

type ReadScore struct {
	Against        string
	Count          int
	TP, FP, FN, TN int
	Precision      int
	Recall         int
}

ReadScore is the shadow read's broken answer (verdict BOUNCE) scored against one gold: Count cards joined, then TP (broken, and gold says broken), FP, FN and TN, with precision and recall as percents (0 with nothing to divide).

func ShadowReadScores

func ShadowReadScores(shadows, heavy []Decision) []ReadScore

ShadowReadScores scores the shadow reads against the heavy verdicts (joined by card, a card's first verdict) and against the readers' outcome (the label on the shadow read itself), heavy first. A shadow read with no gold yet, or whose card was dropped, is not counted.

type Schema

type Schema struct {
	Name      string              `json:"name"`
	Questions map[string]Question `json:"questions"`
}

Schema is a named decision: the questions asked, by name, over one state.

func AttemptSchema

func AttemptSchema() Schema

AttemptSchema is the attempt's question: the class of the take's end.

func BriefSchema

func BriefSchema() Schema

BriefSchema is the brief's nine questions.

func GateSchema

func GateSchema() Schema

GateSchema is the gate decision's one question: the failure's class, with a probability per class.

func GradeSchema

func GradeSchema() Schema

GradeSchema is the grade's question: the convergence grade of the card.

func JudgmentSchema

func JudgmentSchema() Schema

JudgmentSchema is the judgment decision's three questions.

func ParseSchema

func ParseSchema(raw []byte) (Schema, error)

ParseSchema reads a schema and names every problem in it at once: a schema with no name or no questions, a question of an unknown type, a choice with fewer than two options, a noul carrying options, an empty statement.

func ReadSchema

func ReadSchema() Schema

ReadSchema is the read's five questions.

func ScoreSchema

func ScoreSchema() Schema

ScoreSchema is the read's five questions and one noul per class but outside_paths.

func (Schema) Check

func (s Schema) Check(answers map[string]Answer) error

Check holds a backend's answers to the schema: one answer per question, of its type, a choice's value one of its options, every probability in [0, 1]. A choice with only a confidence (empty P, confidence recorded apart) fits; a confidence is not a probability (SPEC-NOVA-DECIDE section 3). A backend that answers something else is refused, never repaired.

func (Schema) Hash

func (s Schema) Hash() string

Hash is the schema's identity in the record: two decisions are calibrated together only when they asked the same questions.

func (Schema) Problems

func (s Schema) Problems() []string

Problems is every reason the schema cannot be asked.

type Send

type Send func(ctx context.Context, body []byte) ([]byte, error)

Send carries one request body to the backend and returns the response body. It is the transport: the decision is made above it, and a test injects a fake.

func HTTPSend

func HTTPSend(client *http.Client, url, key string) Send

HTTPSend is the real transport, and the one function of this package that opens a socket (through client): one POST to url with the key as a bearer token. The key travels on the wire only; an error names the status and the head of the body, never the key.

type Usage

type Usage struct {
	InputTokens  int `json:"input_tokens"`
	OutputTokens int `json:"output_tokens"`
}

Usage is what one ask spent, as the backend reported it; zero is unreported.

func Ask

func Ask(ctx context.Context, b Backend, s Schema, state string) (map[string]Answer, Usage, error)

Ask asks the schema over the state through the backend and returns the answers held to the schema (Check).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL