Documentation
¶
Overview ¶
Package decide is a decision system that is trained from its own record (docs/SPEC-NOVA-DECIDE.md). It has two halves over one record:
- the decide half (this file, backend.go, jev.go, read.go, attempt.go, grade.go): a decision is a named schema (a set of typed questions) asked over one state text through a backend; every answer carries its probabilities.
- the train half (record.go, calibrate.go): every decision made is appended to the record with its inputs, answers and usage; an outcome (a review's label, a gate's result) is attached to it when it is known; the bar a decision's answers are trusted at is calibrated from the decisions whose outcome is known.
A backend is a transport behind an interface: Jev (TypeSafe's System One model) is the first, and Fixed (answers from a file) is the one that needs no network. A decision never dials: the Jev backend sends through a Send function the caller injects, and HTTPSend (jev.go), the real one, is the only code here that opens a socket.
Index ¶
- Constants
- Variables
- func AckReason(p float64) string
- func AnswerOutcome(d Decision, now CardMark) (label, note string)
- func AttachBriefs(record string, ends map[string]End, at time.Time) (attached int, failed map[string]error, err error)
- func AttemptLabel(attempt, landed int, tier string) string
- func AttemptOf(op string) (card string, attempt int, ok bool)
- func AttemptOp(card string, attempt int, state string) string
- func AttemptState(brief, result, reason string) string
- func BriefOp(card, brief string) string
- func BriefState(card string) string
- func CardFilePaths(dir string) ([]string, error)
- func CardFiles(path string) (map[string]string, error)
- func CardOf(id string) string
- func CardPaths(card string) []string
- func ChoiceOf(d Decision, question string) (string, float64)
- func ClassP(d Decision) map[string]float64
- func DiffSummary(diff string) string
- func EvidencePaths(text string) []string
- func Finding(d Decision, bars Bars, files []string) string
- func GateOp(op string, f Failure) string
- func GateState(in GateInput, i int) string
- func GradeLabel(tier string) string
- func GradeOp(card, state string) string
- func GradeState(brief string) string
- func GradeStateWith(brief string, shots []Example) string
- func HistoryOf(log string) []string
- func JudgmentOutcome(col string, placed, otherJudgment bool) string
- func JudgmentState(in JudgmentInput) string
- func Keys(fs []Failure) []string
- func LandLabel(attempt string) (label, note string)
- func Names(fs []Failure) string
- func Op(base, state string) string
- func PDefect(d Decision) float64
- func ParseBar(field, raw string) (bar float64, ok bool, err error)
- func ParseJudgmentBar(raw string) (float64, error)
- func PaymentRefusal(texts ...string) bool
- func ReadLog(r io.Reader) (map[string]*CardFacts, error)
- func ReadState(card, diff, rule string) string
- func RecordAct(path string, a Act) error
- func Replays(have Decision, s Schema, state string) error
- func ScoreOp(card, head string) string
- func Set(bar float64) bool
- func Settle(record, id, verdict, note string, at time.Time) error
- func SettleGate(record string, r GateResult, red map[string]bool, baseRed map[string]bool, ...) (string, error)
- func ShadowKey(d Decision) string
- func ShadowPending(shadows, real []Decision) int
- func ShadowReadOutcome(d Decision, now CardMark) (label, note string)
- func Sum(b []byte) string
- func Top(d Decision) (string, float64)
- func VerbOf(decision string) string
- type Act
- type Answer
- type AnswerInput
- type Backend
- type BackendError
- type Bar
- type Bars
- type Brief
- type BriefBar
- type BucketRow
- type Calibration
- type CardFacts
- type CardMark
- type Chosen
- type Class
- type Cluster
- type ConflictError
- type Decided
- type Decision
- func AnswerDecision(in AnswerInput, at time.Time) Decision
- func Append(path string, d Decision) (recorded *Decision, err error)
- func Attach(path string, o Outcome) (d Decision, changed bool, err error)
- func AttemptDecision(ctx context.Context, b Backend, card string, attempt int, ...) (Decision, error)
- func Find(ds []Decision, id string) *Decision
- func FirstRead(ctx context.Context, b Backend, bars Bars, card, diff, record, id string, ...) (Decision, string, error)
- func Load(path string) ([]Decision, error)
- func Make(ctx context.Context, b Backend, s Schema, state, record, id string, ...) (d Decision, existing bool, err error)
- func ParseAttempt(raw []byte) (Decision, error)
- func Score(ctx context.Context, b Backend, card, diff, record, id string, at time.Time) (Decision, error)
- func ShadowAsk(ctx context.Context, b Backend, in JudgmentInput, note, record string, ...) (d Decision, existing bool, err error)
- func ShadowReadAsk(ctx context.Context, b Backend, card, brief, diff string, broken int, ...) (d Decision, existing bool, err error)
- type End
- type Example
- type Failure
- type Fixed
- type FixedAnswer
- type GateBars
- type GateCall
- type GateInput
- type GateResult
- type GradeRow
- type GradeScore
- type ImportCount
- type ImportResult
- type ImportSources
- type Item
- type Jev
- type JudgmentInput
- type KindAgreement
- type LogLine
- type Made
- type Outcome
- type Question
- type ReadScore
- type Schema
- type Send
- type Usage
Constants ¶
const ( ClassDone = "done" ClassNothingToDo = "nothing-to-do" ClassWrongScope = "wrong-scope" ClassNoResult = "no-result" ClassNeedsPro = "needs-pro" ClassProviderFailure = "provider-failure" )
The attempt's classes.
const ( LabelLanded = "landed" LabelLaterPrefix = "later-" // later-flash, later-pro, later-script LabelDropped = "dropped" )
The attempt's outcome labels, attached when its card lands or is dropped (docs/SPEC-SPRINT.md section 2, the attempt decision): the card landed at the decided attempt; at a later attempt, on the tier that landed it; or it was dropped.
const ( BriefLanded = "landed" // landed on its first attempt BriefReworked = "reworked" // landed on a later attempt BriefDropped = "dropped" // dropped by the coordinator )
The brief's outcome labels: how the card ended in the sprint.
const ( Choice = "choice" Noul = "noul" )
Question types (SPEC-NOVA-DECIDE section 2). A choice picks one of named options and carries a probability per option; a noul is a yes/no over one statement and carries the probability that the statement is true.
const ( RouteBounce = "bounce" RouteStrings = "strings" RouteLand = "land" )
The routes a first read takes.
const ( Flaky = "flaky" Caused = "caused" PreExisting = "pre-existing" RedAgain = "red-again" // Green is a gate whose every failure was flaky and passed its rerun. Green = "green" )
The gate's classes: the class question's options, the routes a failure takes, and the outcome labels a rerun attaches (a rerun that is red again attaches RedAgain when the base was not run, since it then tells caused from pre-existing by nothing).
const ( GradeScript = "script" GradeFlash = "flash" GradePro = "pro" )
The grades, named as the sprint's tiers are (a script card is the sprint's KIND: script).
const ( ImportVerdict = "verdict" ImportJudgment = "judgment" ImportReport = "report" )
The kinds an import records, and the decision name each is recorded under.
const ( JevURL = "https://api.typesafe.ai/v1/systemone" JevModel = "jev-latest" JevSecret = "JEV_API_KEY" // the variable `nova-secrets exec --only JEV_API_KEY` sets // JevTimeout is how long one ask may take by default: a backend that does not answer // within it fails that ask, and never holds its caller (nova-decide's --timeout). JevTimeout = time.Minute )
The Jev backend (SPEC-NOVA-DECIDE section 3): TypeSafe's System One model, asked by one POST of {state, model, questions} and answering {answers: {<name>: {type, choice, probabilities, confidence} | {type, noul}}, usage: {input_tokens, output_tokens}}.
const ( VerbRework = "rework" VerbDrop = "drop" VerbWait = "wait" VerbAck = "ack" VerbAccept = "accept" VerbRelease = "release" VerbAskAnother = "ask-another" )
The verbs a judgment can be answered with: the verb question's options.
const ( MaxHistory = 40 MaxLine = 300 )
MaxHistory bounds the log lines a state carries, and MaxLine each line.
const ( ActApply = "apply" // the verb is applied ActList = "list" // listed for the coordinator )
The actions a judgment's card is given.
const ( OutcomeLanded = "landed" OutcomeDropped = "dropped" OutcomeCameBack = "came-back" )
The outcomes a judgment decision is labelled with, once the card's fate is known.
const ( // JudgmentAnswerName is the record's decision name, and the file nova-sprint run // --decide keeps it in is <dir>/judgment-answer.jsonl. JudgmentAnswerName = "judgment-answer" // AnswerSchema names the record's shape: no schema is asked, the coordinator chose. AnswerSchema = "judgment-answer.v1" // LabelBounced is the outcome of an answer whose card had a read broken or a finish // failed again since. LabelBounced = "bounced" )
The coordinator's own answers (SPEC-NOVA-DECIDE section 13, the judgment-answer record): every judgment answered with a verb that takes --answers, ack or wait is one record of kind judgment-answer, the label set the judgment decision is evaluated and trained on.
const ( Land = "LAND" Bounce = "BOUNCE" Unsure = "UNSURE" )
The read's verdicts.
const ( // ShadowName is the shadow records' decision name; nova-sprint run --decide keeps them in // <dir>/judgment-shadow.jsonl. ShadowName = "judgment-shadow" // ShadowMethod marks each answer of a shadow record. ShadowMethod = "shadow" // HighP is the probability at and above which agreement is also counted. HighP = 0.95 )
const ( // ShadowReadName is the shadow reads' decision name; nova-sprint run --decide keeps them in // <dir>/read-shadow.jsonl. ShadowReadName = "read-shadow" // ScoreHeavy and ScoreReaders name what a ReadScore is scored against. ScoreHeavy = "heavy" ScoreReaders = "readers" )
const AttemptName = "attempt"
AttemptName is the attempt decision's name in the record.
const AttemptQuestion = "class"
AttemptQuestion is the attempt decision's one question.
const BriefDeadline = time.Minute
BriefDeadline bounds a whole brief batch, in nova-sprint add and nova-decide brief: past it what is unanswered is each card's error and the rest are recorded.
const BriefName = "brief"
BriefName is the brief decision's name in the record.
const BriefWidth = 8
BriefWidth is how many cards a brief batch asks at once, in nova-sprint add and as nova-decide brief's --width default.
const GateName = "gate"
GateName is the gate decision's name in the record.
const GradeExamplesPerClass = 10
GradeExamplesPerClass is how many examples of each class the prompt carries.
const GradeHeavy = "heavy"
GradeHeavy is the label of a card that needed the heavy tier; the grade's own options stay script, flash and pro, and the examples teach the line between them.
const GradeName = "grade"
GradeName is the grade decision's name in the record.
const GradeQuestion = "grade"
GradeQuestion is the grade decision's one question.
const JudgmentName = "judgment"
JudgmentName is the judgment decision's name in the record.
const MaxGateFailures = 8
MaxGateFailures is how many failing tests of one gate are asked about; a failure past it is caused, unasked (a gate with that many red tests is the change's).
const MaxResultBytes = 16 << 10
MaxResultBytes bounds the RESULT.md an attempt's state carries: a result is a page of text, and the state rides to the sprint's server with the finish.
const OutsidePaths = "outside_paths"
OutsidePaths is the one class not asked on its own: its p is 1 - p(inside_paths), the read's own question, so the same evidence is never asked twice.
const ReadName = "read"
ReadName is the read decision's name in the record.
const ScoreName = "score"
ScoreName is the score decision's name in the record.
const Unnamed = "unnamed"
Unnamed is the findings row of the decisions whose p(defect) meets the bar while no class does: a defect no class names yet, the material for a new class.
const Unset = 2.0
Unset is a gate bar the sprint row leaves empty: no probability reaches it.
Variables ¶
var AnswerVerbs = []string{"accept", "rework", "recut", "return", "drop", "wait", "hold", "ack", "ask-another", "release"}
AnswerVerbs is every verb that answers a judgment, in the order the spec lists them.
var ErrUnknown = errors.New("no decision with that id in the record")
ErrUnknown is an outcome for an id the record holds no decision for.
var Fixes = map[string]string{
"own": "",
"from-tip": "Your head does not merge onto the sprint tip: a file you touched was also changed by a card that landed after your base. " +
"Start again from the current tip of the sprint branch, redo only the lines your card lists, and if a ledger must change, " +
"lower its ceiling from the value at the tip by exactly your own count.",
"pro-tier": "attempts ended without a RESULT.md on the flash tier; run on the pro tier (RESULT line tier: pro); the task is unchanged",
"retry": "the attempt ended for a cause outside the card (a provider, a member, staging); run it again; the task is unchanged",
}
The fixes a rework that takes --fix carries, and their texts.
var GradeExampleClasses = []string{GradeFlash, GradePro, GradeHeavy}
GradeExampleClasses are the example labels in prompt order: landed on flash within two attempts, needed pro, needed heavy.
var ImportKinds = []string{ImportVerdict, ImportJudgment, ImportReport}
ImportKinds is every kind, in the order a result prints them.
var Kinds = map[string]string{
"a reader found it broken": "broken",
"work came back failed": "failed",
"a primary is blocked on something dropped": "blocked",
"stalled": "stalled",
"stream stopped: conflict on a card": "conflict",
"a work card is past its deadline": "deadline",
"cannot ask": "cannot-ask",
"ready to accept": "ready",
"a card reached its bound": "bound",
}
Kinds maps each routine judgment type, as the inbox names it, to its short name: the kinds nova-sprint answer asks the decision for. Every other type is left.
var Reasons = map[string]string{
"already-done": "the listed edits were already made, by an earlier attempt or by a card that landed first: done, not failed",
"cannot-be-done": "the card cannot be done as written: its brief is wrong, or names lines that are not there",
"need-dropped": "what it needs was dropped, and it cannot run without it",
"reads-exhausted": "no reader can read it on the installed build (reads exhausted); added again after the fix",
}
The reasons a drop gives, and their texts.
var Verbs = []string{VerbRework, VerbAskAnother, VerbAccept, VerbAck, VerbWait, VerbDrop, VerbRelease}
Verbs is every verb, in the order a row prints them.
Functions ¶
func AckReason ¶
AckReason is the reason an applied ack gives: the decision's, with its probability.
func AnswerOutcome ¶
AnswerOutcome is the outcome the card's mark now says of the answer d, "" while the card stands with nothing new: landed, dropped (off the table), or bounced (more reads found it broken, or more finishes failed, than when it was answered).
func AttachBriefs ¶
func AttachBriefs(record string, ends map[string]End, at time.Time) (attached int, failed map[string]error, err error)
AttachBriefs attaches each card's end to its brief decision by the decision's exact id, the op add stored on the card (op -> End), in one write under the record's lock. An op the record does not hold, or a record that does not exist, is that op's error in failed and no record is made; the same label again changes nothing; another label is that op's *ConflictError; the other ops are attached.
func AttemptLabel ¶
AttemptLabel is the outcome of an attempt decided at attempt, for a card that landed at landed on tier (landed 0: dropped).
func AttemptOf ¶
AttemptOf is the card and attempt an attempt decision's op id names (AttemptOp); ok false when it is not `<card>@<attempt>.<hex>`.
func AttemptOp ¶
AttemptOp is a take's attempt decision id: `<card>@<attempt>.<12 hex of the state>`. It is per attempt and state, not per take: two takes of one attempt that ended with the same state (the same reason line and RESULT.md) are one decision.
func AttemptState ¶
AttemptState is the text the attempt is asked over: the card's brief, the child's RESULT.md (cut to MaxResultBytes) and the member's reason line, each under its own heading.
func BriefOp ¶
BriefOp is a card's brief decision's id in the record: <card>@brief-<8 hex>, the hex of the schema and the brief, so a brief rewritten after a refusal, or asked under a reworded schema, is a decision of its own and never a conflict with the one before it.
func BriefState ¶
BriefState is the text the brief is asked over: the card, under its heading, then the frame the sprint gives every child.
func CardFilePaths ¶
CardFilePaths is the card files of a directory as nova-sprint add --brief-dir reads them: every *.md entry that is not a directory, in byte order of name, none below.
func CardFiles ¶
CardFiles is the cards a path names, id -> brief: a file is one card, a directory its CardFilePaths (add --brief-dir's cards); a card's id is its file's name without .md, and its brief the file's text with one trailing newline cut, as nova-sprint add stores it.
func CardPaths ¶
CardPaths is a card's PATHS globs: its first `PATHS:` line (cardhdr.KeyValue), split on commas; nil when it names none.
func ChoiceOf ¶
ChoiceOf is a choice's chosen option and its probability (Top is the score decision's).
func ClassP ¶
ClassP is the p of every class a score decision gives, outside_paths from inside_paths.
func DiffSummary ¶
DiffSummary is the files a unified diff changes, one per line with the lines it adds and removes: `<path> +<added> -<removed>`, a rename as `<old> -> <new>`.
func EvidencePaths ¶
EvidencePaths is the paths a judgment's text names (a report, a finding's file), each once, in the order named, a trailing full stop and a :line cut.
func Finding ¶
Finding is a first read's report: p(defect) and the bar it met, the files the diff changes, and the five answers. It names a file, so a bounce is a broken read with a finding (docs/SPEC-CARD-CONTRACT.md section 3).
func GateState ¶
GateState is the text one failure is asked over: the failure and its lines, the base, the gate's other failures by name, the card's PATHS and its diff summary, each under its own heading.
func GradeLabel ¶
GradeLabel is the grade's outcome: the tier that landed the card, or dropped ("" tier).
func GradeState ¶
GradeState is the text the grade is asked over: the card's brief alone, under its heading.
func GradeStateWith ¶
GradeStateWith is the grade's state with the examples ahead of the card: one line each (heading, PATHS, KIND, label), then GradeState(brief) whole. No examples is GradeState.
func HistoryOf ¶
HistoryOf is a card's log as a judgment state carries it: the lines nova-sprint log --card prints, less its LOG OK line and the cost and field-change lines (the record of a read's price or a field set tells the decision nothing), each line trimmed.
func JudgmentOutcome ¶
JudgmentOutcome is the outcome a card's state says, "" while it is not known: landed, dropped (off the table), or came back (another judgment open on it).
func JudgmentState ¶
func JudgmentState(in JudgmentInput) string
JudgmentState is the text the judgment decision is asked over: the judgment, the verbs allowed, and the card's last MaxHistory log lines, each cut at MaxLine.
func LandLabel ¶
LandLabel is the outcome a card's landing attaches to its brief: landed at attempt 1, reworked at a later one, with the attempt in the note.
func Op ¶
Op is a decision's op id over a state: base (the card and what it decides of it, `<card>@<attempt>`, `<card>@grade`), then the first twelve hex of the state's SHA-256, so a finish reported again replays its decision with no ask, and a card id that comes back with another brief (a clear, a brief replaced) is a new decision, never a conflict.
func ParseBar ¶
ParseBar reads one bar as the sprint row holds it, a decimal as text: empty is no bar (ok false), and anything else that is not a probability is an error naming the field.
func ParseJudgmentBar ¶
ParseJudgmentBar reads a judgment bar: a probability (ParseBar is a sprint row bar's).
func PaymentRefusal ¶
PaymentRefusal says a text carries a provider's refusal for want of payment. Such a judgment is never answered by a decision: a payment is the owner's.
func ReadLog ¶
ReadLog reads a `nova-sprint log --json` export ({"lines":[...]}) into each primary's facts: its last placement on the work table, and the last text under each of its cost_record keys (SPEC-NOVA-DECIDE section 11).
func ReadState ¶
ReadState is the text the read is asked over: the card, the rule when one is given, then the diff, each under its own heading.
func RecordAct ¶
RecordAct appends a's line against its decision: a step of applying it. A decision the record does not hold is ErrUnknown, and nothing is written.
func Replays ¶
Replays says the recorded decision is the one an ask of s over state under its id would make, so the ask is answered from the record; else a ConflictError.
func Settle ¶
Settle attaches a strings read's verdict to the first read it followed: ok is LAND, broken is BOUNCE. Any other verdict is no outcome and attaches nothing.
func SettleGate ¶
func SettleGate(record string, r GateResult, red map[string]bool, baseRed map[string]bool, at time.Time) (string, error)
SettleGate attaches each rerun failure's result as its decision's outcome: Flaky when the rerun passed; red again, PreExisting when it was red at the base, Caused when the base was run and green, RedAgain when the base was not run. It returns the gate's route after the rerun: Caused when a rerun failure is red again, else PreExisting when a failure was routed so, else Green.
func ShadowKey ¶
ShadowKey joins a shadow record to the real answer: the judgment's note and its card.
func ShadowPending ¶
ShadowPending is how many shadow records have no real answer yet.
func ShadowReadOutcome ¶
ShadowReadOutcome is the readers' outcome of the shadow read d from the card's mark now: Bounce when more reads have found the card broken than when it was shadow-read, Land when the card landed without that, "" while it stands with nothing new, and Dropped when it left the table (a card dropped has no outcome to score).
func Sum ¶
Sum is the hex SHA-256 of b: how the record names an input without holding a second copy of a file.
Types ¶
type Act ¶
type Act struct {
ID string `json:"id"`
Act string `json:"act"`
Op string `json:"op,omitempty"`
At string `json:"at"`
}
Act is one step of applying a decision, recorded around the verbs it runs: "applying" with the operation id those verbs carry, before them, then "applied" or "refused" after. A decision's acts are appended, never rewritten; its last says where applying it stands, so a writer stopped between the two finds "applying" and its op.
type Answer ¶
type Answer struct {
Type string `json:"type"`
Value string `json:"value"`
P map[string]float64 `json:"p"`
Confidence float64 `json:"confidence,omitempty"`
Method string `json:"method,omitempty"`
}
Answer is one typed answer. P is the probability of each option for a choice, and of "yes" for a noul; Value is the chosen option, or "yes" or "no" at 0.5 for a noul. Confidence and Method are a choice the wire answered with no probabilities: the confidence, and how it was read. Neither is a probability of correctness, and both are omitted when empty (SPEC-NOVA-DECIDE section 3).
type AnswerInput ¶
type AnswerInput struct {
Note string // the judgment's notification id
Kind string // the judgment's type, as the inbox names it
Text string // the judgment note's text
Card string // the card answered
Verb string // the verb given
Reason string // --reason (or the wait's duration), "" when none
Fix string // --fix, "" when none
Actor string // who answered
Mark CardMark // the card as it stood after the verb ran
}
AnswerInput is one judgment answered for one card.
type Backend ¶
type Backend interface {
Name() string
Ask(ctx context.Context, s Schema, state string) (map[string]Answer, Usage, error)
}
Backend answers a schema over a state. It is the transport of a decision: the system above it (the record, the calibration) is the same whichever backend answered.
type BackendError ¶
BackendError is a backend that gave no answer within the schema: nothing was recorded.
func (*BackendError) Error ¶
func (e *BackendError) Error() string
func (*BackendError) Unwrap ¶
func (e *BackendError) Unwrap() error
type Bars ¶
Bars are the two bars on p(defect): Bounce at or above, Review below.
type Brief ¶
type Brief struct {
Converges float64 // p(converges)
Minutes string // the minutes option chosen
Ambiguous string // the ambiguous_step option chosen: none, step-<n> or unnumbered
Failed []string // the questions the card fails: a need under 0.5 as name(p), a step as ambiguous_step:step-<n>(p)
}
Brief is what a brief decision says of its card.
type BriefBar ¶
BriefBar is the bar on p(converges) a card is added at: Set is false when the sprint row's decide_brief_bar is empty (report only).
func ParseBriefBar ¶
ParseBriefBar reads decide_brief_bar as the sprint row holds it: empty is no bar, else a probability.
type BucketRow ¶
BucketRow is one line of the calibration table: the cards graded Grade, dealt on flash, whose p for the grade fell in Bucket.
type Calibration ¶
type Calibration struct {
Decision, Schema, Question, Option string
Positives, Negatives []float64 // the score of each labelled decision
Skipped int // labelled otherwise, unlabelled, or asked under another schema
}
Calibration (SPEC-NOVA-DECIDE section 5) is how well one answer of one decision separates the outcomes, read from the record: the decisions of that name and schema whose outcome is known, scored by the probability the answer gave to one option (a noul's "yes", or a choice's named option), split into positives (the outcomes the answer should catch) and negatives.
func Calibrate ¶
func Calibrate(ds []Decision, decision, question string, positive, negative []string) (Calibration, error)
Calibrate scores every decision named decision whose outcome label is in positive or negative. The schema is the newest one the record holds for that decision: answers to other questions are never pooled. question is "<name>" (a noul, scored by its yes) or "<name>=<option>" (a choice). A label of words joined by + is positive (or negative) when one of its words is listed.
func (Calibration) AUC ¶
func (c Calibration) AUC() float64
AUC is the probability that a positive scores above a negative (ties count half): 0.5 is no separation, 1 is perfect.
func (Calibration) CatchAll ¶
func (c Calibration) CatchAll() Bar
CatchAll is the highest bar that still flags every positive (the lowest positive score) and the row there: the bar the record supports when no positive may pass.
type CardFacts ¶
type CardFacts struct {
State string // landed, dropped, or the column it sits in
Start string
MaxAttempt int
FirstFailed bool
Escalated bool
// contains filtered or unexported fields
}
CardFacts is what the log holds of one card: where it ended, the tier of its first attempt ("" when it was never dealt or attempt 1 is before the log), the highest attempt number, whether attempt 1 failed, and whether a pro attempt ran.
type CardMark ¶
CardMark is what an outcome reads of a card: whether it stands on the table, has landed or was dropped, and how many reads found it broken and how many finishes failed.
type Chosen ¶
type Chosen struct {
Verb string `json:"verb"`
P float64 `json:"p"`
Fix string `json:"fix,omitempty"` // a rework's fix option, when its command takes one
Reason string `json:"reason,omitempty"` // a drop's reason option
Act string `json:"act"`
Why string `json:"why,omitempty"` // why it is listed
}
Chosen is what the decision chose for one card and what is done with it.
type Class ¶
type Class struct {
Name, Escalations, Statement string
}
Class is one escalation class: its name in the record, the review escalations it stands for, and the statement asked (a noul; yes is the defect present).
type Cluster ¶
type Cluster struct {
Class string `json:"class"`
Count int `json:"count"`
Cards []string `json:"cards"`
}
Cluster is one class of the findings: how many score decisions gave it a p at or above the bar, and their cards in id order.
func FindingsSkipped ¶
func FindingsSkipped(ds []Decision, since time.Time, bar float64) (clusters []Cluster, scored int, skipped []string)
FindingsSkipped clusters the score decisions made at or after since by every class each gives a p at or above bar, plus Unnamed for p(defect) at or above bar with no class there; most cards first, then class order; scored is the decisions in the window. It also returns the ids, in record order, of the score decisions it left out because their at is not RFC 3339: such a decision cannot be placed in the window, and the caller says so rather than count it as nothing (security#79 finding 3). A decision of another kind is not a score and is never named.
type ConflictError ¶
type ConflictError struct{ What string }
ConflictError is an id already recorded with other content: an op id reused over another state, or a decision already labelled otherwise.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Decided ¶
Decided is a decision as it rides on a card: the chosen option, its probability and the decision's op id, one line: `<option> p=<p> op=<op>`.
func AttemptDecided ¶
AttemptDecided is the attempt decision as it rides with a finish: its class, the class's probability and its op id. An answer that is not one of the six classes is an error.
func ParseDecided ¶
ParseDecided reads a card's line (Decided.String); ok false when it is not one.
type Decision ¶
type Decision struct {
ID string `json:"id"`
Decision string `json:"decision"` // the schema's name
Schema string `json:"schema"` // the schema's hash
Backend string `json:"backend"`
At string `json:"at"` // RFC 3339, UTC
Inputs map[string]string `json:"inputs"`
State string `json:"state"` // the exact text asked over: the training input
Answers map[string]Answer `json:"answers"`
Usage Usage `json:"usage"`
Outcome *Outcome `json:"outcome,omitempty"` // folded in on load; never written on this line
Acts []Act `json:"acts,omitempty"` // folded in on load, in order; never written on this line
}
Decision is one decision made: what was asked, over what, by which backend, and what it answered.
func AnswerDecision ¶
func AnswerDecision(in AnswerInput, at time.Time) Decision
AnswerDecision is the record of a judgment answered (the judgment-answer record): the judgment's state as the judgment decision is asked it, the verb as the answer, and as inputs the kind, the evidence paths, the --reason and --fix text, the actor and the card's counters, which AnswerOutcome reads the later ones against. Its id is per judgment, card and answer, so the same answer given again is the same record.
func Append ¶
Append records d unless its id is already there: the same id over the same state returns the recorded decision (an op retried), over another state it is a ConflictError.
func Attach ¶
Attach records o against its decision and returns the decision with it. The same label again changes nothing (changed is false); another label is a ConflictError naming both.
func AttemptDecision ¶
func AttemptDecision(ctx context.Context, b Backend, card string, attempt int, brief, result, reason string, at time.Time) (Decision, error)
AttemptDecision asks the attempt decision of a take of card at attempt through b and returns it as the record keeps it, with its op id (AttemptOp), unrecorded: the sprint's member asks it, and the finish carries it to the record the sprint's server keeps (docs/SPEC-SPRINT.md section 2). A backend that fails is a BackendError.
func FirstRead ¶
func FirstRead(ctx context.Context, b Backend, bars Bars, card, diff, record, id string, at time.Time) (Decision, string, error)
FirstRead is the read decision over a card and its diff, recorded under id (the read card's id), and the route its p(defect) takes at the bars.
func Load ¶
Load reads the record; a record that does not exist yet is empty. A line that does not parse, a decision id seen twice, an outcome for an id with no decision before it, or an answer whose probability is outside [0, 1] or not a number, is an error naming the line (SPEC-NOVA-DECIDE section 4).
func Make ¶
func Make(ctx context.Context, b Backend, s Schema, state, record, id string, inputs map[string]string, at time.Time) (d Decision, existing bool, err error)
Make asks s over state through b and appends the decision to the record under id. An id the record holds over the same decision, schema and state is that decision (existing), and nothing is asked; over another it is a ConflictError.
func ParseAttempt ¶
ParseAttempt reads an attempt decision as a finish carries it (one JSON record line's decision) and holds it to the attempt schema: its name, its schema's hash, its answers (Check) and its op id over its state (AttemptOp). A decision that does not fit is an error, and the server records nothing of it.
func Score ¶
func Score(ctx context.Context, b Backend, card, diff, record, id string, at time.Time) (Decision, error)
Score asks the score decision over a landed card and its diff and records it under id (Make: an id already recorded over the same card and diff is answered from the record).
func ShadowAsk ¶
func ShadowAsk(ctx context.Context, b Backend, in JudgmentInput, note, record string, at time.Time) (d Decision, existing bool, err error)
ShadowAsk asks the judgment decision over in through b and appends the answer, marked shadow, to record under an id per note, card and state. A judgment already shadowed is returned as recorded (existing) and nothing is asked.
func ShadowReadAsk ¶
func ShadowReadAsk(ctx context.Context, b Backend, card, brief, diff string, broken int, record string, at time.Time) (d Decision, existing bool, err error)
ShadowReadAsk asks the read decision over the card's brief and diff through b and appends the answer, marked shadow, to record under an id per card and state. A card already shadow-read at that diff is returned as recorded (existing) and nothing is asked. broken is the card's count of broken reads now, which ShadowReadOutcome reads the later ones against.
type End ¶
type End struct{ Label, Note string }
End is how a card ended in the sprint: the outcome label its brief is given (BriefLanded, BriefReworked, BriefDropped) and a note.
type Example ¶
type Example struct {
Card string `json:"card"`
Heading string `json:"heading"`
Paths string `json:"paths"`
Kind string `json:"kind"`
Label string `json:"label"`
}
Example is one landed card of the record: its brief heading, its PATHS line, its KIND and the label of its outcome. Never the brief's whole text.
func ParseExamples ¶
ParseExamples reads an examples file, one JSON object per line ({card, heading, paths, kind, label}); every bad line is named in one error.
func PickExamples ¶
PickExamples chooses per examples of each class from pool, deterministically: held-out cards leave the pool, a card's rank is the SHA-256 of seed and its id (so neither the pool's order nor its size moves a card's rank), and the lowest ranks are taken. The result is in class order, then rank. A class with fewer than per examples left is an error naming it.
type Failure ¶
type Failure struct {
Pkg string `json:"pkg"`
Test string `json:"test,omitempty"`
Lines []string `json:"lines,omitempty"`
}
Failure is one failing test of a gate's output: its package as go test names it, the top-level test, and the first lines it printed. A build failure has no test.
func ParseGateOutput ¶
ParseGateOutput reads go test's output (plain or -v) into its failures, in the order go printed them: each top-level test with a `--- FAIL:` line (a subtest's failure is its test's), its package from the `FAIL <pkg>` line that follows, and its first lines (the indented lines after it, else, under -v, what it printed after its `=== RUN`); a package that failed to build or with no failing test named (a timeout names its test under "running tests:") is one failure with the package's lines. Output that is no go test's has no failures.
type Fixed ¶
type Fixed struct {
Table map[string]FixedAnswer
}
Fixed is the backend that answers from a table: question name -> the answer it gives, whatever the state. It needs no network and no key, so a first run, a test, or a dry comparison against a recorded answer set runs anywhere (SPEC-NOVA-DECIDE section 3).
func ParseFixed ¶
ParseFixed reads a Fixed table: {"<question>": {"choice": ..., "p": {...}} | {"noul": p}}.
type FixedAnswer ¶
type FixedAnswer struct {
Choice string `json:"choice,omitempty"`
P map[string]float64 `json:"p,omitempty"`
Noul *float64 `json:"noul,omitempty"`
}
FixedAnswer is one row of a Fixed table: a choice's option and its probabilities, or a noul's probability of yes.
type GateBars ¶
GateBars are the two bars on a failure's class probabilities: at or above Flaky it is rerun once, at or above PreExisting it is reported pre-existing. A bar that is unset is Unset, above every probability, so its route is never taken: the decision is recorded and shown, and nothing is rerun or reclassified on it.
func ParseGateBars ¶
ParseGateBars reads the bars as the sprint row holds them, decimals as text: each a probability, or empty for Unset (its route is never taken; the sprint row's default); two set bars sum above 1, so no failure meets both. Every problem is named.
type GateCall ¶
type GateCall struct {
Failure Failure
Decision *Decision
Route string
Existing bool // the decision was in the record already, and nothing was asked
}
GateCall is one failure of a gate and what was decided about it: its decision (nil for one not asked: a build failure, or one past MaxGateFailures) and its route.
type GateInput ¶
type GateInput struct {
Failures []Failure
BaseRed map[string]bool
Paths []string
Diff string // DiffSummary of the card's diff
}
GateInput is what one gate's decisions are asked over: its failures, the keys of those red at the base (nil when the base was not run), the card's PATHS, and its diff summary.
type GateResult ¶
GateResult is a gate's decisions and the route the gate takes: Caused when any failure is, else Flaky when any is (those are rerun), else PreExisting; "" when the gate had no failure, so nothing was decided (never PreExisting).
func Gate ¶
func Gate(ctx context.Context, b Backend, bars GateBars, in GateInput, record, op string, at time.Time) (GateResult, error)
Gate asks the gate decision of each failure of in (up to MaxGateFailures; a build failure is caused, unasked), records each under GateOp(op, failure), and routes the gate at the bars. A backend that fails is an error and the gate is the card's, as before. No failure is no decision and no route: nothing is asked or recorded.
func (GateResult) Classes ¶
func (r GateResult) Classes() string
Classes is what each asked failure was classed, for a line that shows the decisions: `<Test>:<class>:<p>` comma-separated, p the class's probability to two places; an unasked failure is `<Test>:unasked`.
func (GateResult) PreExistingTests ¶
func (r GateResult) PreExistingTests() []Failure
PreExistingTests is the failures routed PreExisting.
func (GateResult) Rerun ¶
func (r GateResult) Rerun() []Failure
Rerun is the failures the gate reruns: those routed Flaky.
type GradeRow ¶
GradeRow is one line of the grade-by-dealt table: the cards Jev graded Grade that were first dealt on Dealt.
type GradeScore ¶
type GradeScore struct {
Decisions, Cards, NoLog int // grade decisions in the day, the cards they grade, cards the log lacks
Rows []GradeRow
Buckets []BucketRow
}
GradeScore is the day's score.
func ScoreGrades ¶
ScoreGrades scores the grade decisions whose At is in [from, to): each card by its newest grade in the window, against facts. A card the log does not hold, or never dealt to the fleet, is counted and left out of the tables (a friend's card has no tier to grade). Open is a card neither landed nor dropped; Landed2 is a card landed by its second attempt.
type ImportCount ¶
type ImportCount struct{ New, Existing int }
ImportCount is how many items of a kind were recorded now and how many were there already.
type ImportResult ¶
type ImportResult struct {
Kinds map[string]ImportCount
Unanswered int
}
ImportResult is the count per kind, and the judgments the log has no answer for (they are not recorded: an unlabelled item is not a label).
func Import ¶
func Import(path string, src ImportSources, now time.Time) (ImportResult, error)
Import records every item of src in the record at path, once: an item whose id is there already is counted and left. The whole import is one write under the record's lock.
type ImportSources ¶
type ImportSources struct {
Verdicts, Judgments, Log, Reports string
}
ImportSources names what to import; an empty field is a source not read. Verdicts and Reports are globs of files; Judgments is a directory of judgment files, which needs Log, a nova-sprint log --json export, to hold the answers.
type Item ¶
Item is one decision of a batch: its op id, the state it is asked over, and the inputs the record names.
type Jev ¶
Jev is the Jev backend over an injected Send.
func JevHTTP ¶
JevHTTP is the Jev backend over the real transport with key, each ask bounded by timeout: what the sprint's decide read and answer ask through.
type JudgmentInput ¶
type JudgmentInput struct {
Kind string // the judgment's type, as the inbox names it
Text string // the judgment note's text
Card string // the card decided
Cards int // how many cards the judgment holds
History []string // the card's log, oldest first
Allowed []string // the verbs the judgment prints
}
JudgmentInput is what one judgment decision is asked over.
type KindAgreement ¶
type KindAgreement struct {
Kind string
Count int
Agree int
Pct int
HighCount int
HighAgree int
HighPct int
}
KindAgreement is how a judgment kind's shadow answers agreed with the real ones: Count pairs joined, Agree of them the same verb, Pct that as a percent; HighCount of the pairs had the shadow verb at p HighP or above, HighAgree of those agreed, HighPct as a percent.
func ShadowAgreement ¶
func ShadowAgreement(shadows, real []Decision) []KindAgreement
ShadowAgreement joins each shadow record to the real judgment-answer record of the same note and card and scores the pairs per judgment kind, kinds in name order. A real answer is the first for its note and card; verbs are compared as recorded.
type LogLine ¶
type LogLine struct {
Card string `json:"card"`
Primary string `json:"primary"`
Table string `json:"table"`
To string `json:"to"`
Removed bool `json:"removed"`
Set map[string]string `json:"set"`
}
LogLine is one line of `nova-sprint log --json`, the fields the score reads.
type Made ¶
Made is one item's decision. Existing says the record held it and nothing was asked. Err is why it could not be made (its Decision then holds only the item's ID and Inputs): a *BackendError (nothing recorded) or a *ConflictError (the op id is recorded over another decision, schema or state).
func Briefs ¶
func Briefs(ctx context.Context, b Backend, cards map[string]string, record string, at time.Time, width int, wait time.Duration) ([]Made, error)
Briefs makes the brief decision of every card (id -> brief text) as one batch, in id order, at most width asks at a time, each within wait.
func MakeAll ¶
func MakeAll(ctx context.Context, b Backend, s Schema, items []Item, record string, at time.Time, width int, wait time.Duration) ([]Made, error)
MakeAll makes the decision of s over each item, in the items' order, each ask within wait (0 is ctx's own bound); once ctx is done no further item is asked and each is that item's Err. An empty record keeps nothing: no decision is read from it or written to it. The error is the record's (it cannot be read or written); a decision that could not be made is its item's Err, and the others are made and recorded.
type Outcome ¶
type Outcome struct {
ID string `json:"id"`
Label string `json:"label"`
Note string `json:"note,omitempty"`
At string `json:"at"`
}
Outcome is what turned out to be true about a decision: a review's label, a gate's result. It is attached once; a second, different label is a conflict.
type Question ¶
type Question struct {
Type string `json:"type"`
Instructions string `json:"instructions"`
Criteria map[string]string `json:"criteria,omitempty"` // choice only: option -> what it means
}
Question is one typed question of a schema.
type ReadScore ¶
ReadScore is the shadow read's broken answer (verdict BOUNCE) scored against one gold: Count cards joined, then TP (broken, and gold says broken), FP, FN and TN, with precision and recall as percents (0 with nothing to divide).
func ShadowReadScores ¶
ShadowReadScores scores the shadow reads against the heavy verdicts (joined by card, a card's first verdict) and against the readers' outcome (the label on the shadow read itself), heavy first. A shadow read with no gold yet, or whose card was dropped, is not counted.
type Schema ¶
Schema is a named decision: the questions asked, by name, over one state.
func AttemptSchema ¶
func AttemptSchema() Schema
AttemptSchema is the attempt's question: the class of the take's end.
func GateSchema ¶
func GateSchema() Schema
GateSchema is the gate decision's one question: the failure's class, with a probability per class.
func GradeSchema ¶
func GradeSchema() Schema
GradeSchema is the grade's question: the convergence grade of the card.
func JudgmentSchema ¶
func JudgmentSchema() Schema
JudgmentSchema is the judgment decision's three questions.
func ParseSchema ¶
ParseSchema reads a schema and names every problem in it at once: a schema with no name or no questions, a question of an unknown type, a choice with fewer than two options, a noul carrying options, an empty statement.
func ScoreSchema ¶
func ScoreSchema() Schema
ScoreSchema is the read's five questions and one noul per class but outside_paths.
func (Schema) Check ¶
Check holds a backend's answers to the schema: one answer per question, of its type, a choice's value one of its options, every probability in [0, 1]. A choice with only a confidence (empty P, confidence recorded apart) fits; a confidence is not a probability (SPEC-NOVA-DECIDE section 3). A backend that answers something else is refused, never repaired.
type Send ¶
Send carries one request body to the backend and returns the response body. It is the transport: the decision is made above it, and a test injects a fake.