Documentation
¶
Overview ¶
Package onboarding reads the two things docs/ONBOARDING.md makes every command in this repo carry — the `example:` block at the foot of its usage banner, and its `### First run` section in docs/TESTS.md — and reduces an output line to the part that document promises. It is the shared half of the tests that pin the standard, so that "the examples run" and "the transcript is what the tool prints" mean the same thing in every binary rather than five similar things.
It holds no assertions of its own: it parses, and the caller's test decides. Nothing here reads a file, so the caller says where the bytes came from.
Index ¶
- Constants
- Variables
- func Block(lines []string) string
- func ExampleCommands(banner, tool string) []string
- func ExampleLines(usage, tool string) ([]string, error)
- func ExecuteWith(steps []Step, run Runner, cond Conditions, norms ...Norm) ([]Problem, []Skip)
- func FirstRun(md, tool string) ([]string, error)
- func HowItWorksLength(banner string) int
- func HowItWorksLine(banner string) int
- func Normalize(line string, norms []Norm) string
- func OpeningSentence(banner, tool string) (string, error)
- func RepeatedSections(md string) []string
- func Section(md, name string) (string, bool)
- func SectionNames(md string) []string
- func Shape(line string) string
- func SplitShell(cmd string) ([]string, error)
- func Transcript(md, tool, heading string) ([]string, error)
- func VolatileNames() []string
- type Conditions
- type Field
- type Norm
- type Problem
- type Result
- type Runner
- type Skip
- type Step
- type VolatileField
Constants ¶
const ExampleHeading = "\nexample:\n"
ExampleHeading is the line that opens the runnable block at the foot of a usage banner. Everything under it, up to the first line that is not a command for this tool, is a command a first run can type.
const FirstRunHeading = "### First run"
FirstRunHeading is the subsection a stranger reads before anything else about a tool. It names a section in whichever document the caller supplies: docs/TESTS.md for the transcripts these tests execute.
const HowItWorksLabel = "how it works:"
HowItWorksLabel opens the paragraph that names a tool's nouns and where its state lives.
const HowItWorksMaxLines = 5
HowItWorksMaxLines is the most lines the how-it-works paragraph takes: the tool's nouns and where its state lives, not a second manual.
const HowItWorksWithin = 15
HowItWorksWithin is how many lines from the top the paragraph must start in: it is read before the usage lines, never found under them.
const MinExampleCommands = 3
MinExampleCommands is the fewest command lines an `example:` block holds: a first run is a sitting, and one line is a single call rather than a sitting.
const StderrMarker = "! "
StderrMarker opens a documented line the tool writes to standard error. It is the convention honoured here so that a swept section executes the moment it is swept: standard output is compared WHOLE -- every unmarked line, in order, and nothing else -- and standard error only for the lines shown, in order, because how loudly a tool narrates its own work is not a promise to a caller the way its protocol output is.
Variables ¶
var KnownGOOS = []string{
"aix", "android", "darwin", "dragonfly", "freebsd", "hurd", "illumos", "ios",
"js", "linux", "nacl", "netbsd", "openbsd", "plan9", "solaris", "wasip1", "windows", "zos",
}
KnownGOOS is the spelling a `# Platform:` declaration has to use: Go's own GOOS words, because SkipReason compares the declaration against runtime.GOOS and nothing else can ever match. `macOS` is the one a person writes, and it skipped the step on every bench for ever while reading, in a green build, exactly like a step everybody ran.
This is a SPELLING check and not the whole rule. The rule that matters is TestPlatformLineMustNameACILeg: a platform no leg of .github/workflows/ci.yml runs is a transcript nobody executes -- `windows` is the live case: no leg of ci.yml runs it. That check reads ci.yml, which this package must not do (it reads no file and runs no process), and it lives in internal/ci on the T23 branch. When T23 lands, its rule wants extending from the section line to the per-step declaration this file parses; the two should not be two readers of ci.yml.
var Volatile = []VolatileField{ { Name: "at", What: "at= (the instant of this run)", // contains filtered or unexported fields }, { Name: "took", What: "took= (how long this run took)", // contains filtered or unexported fields }, { Name: "created", What: "created= (the instant this run created the record)", // contains filtered or unexported fields }, { Name: "tmpdir", What: "the directory this run made", NeedsPath: true, // contains filtered or unexported fields }, { Name: "id", What: "id= (a ULID this run made)", // contains filtered or unexported fields }, { Name: "sha", What: "sha= (a sha this run made)", // contains filtered or unexported fields }, { Name: "recorded", What: "a name a recorded fixture carries, written in the document as the variable the reader sets", NeedsPath: true, Repeatable: true, // contains filtered or unexported fields }, { Name: "branch", What: "branch= (the seal branch this run stamped with its instant)", // contains filtered or unexported fields }, { Name: "commit", What: "at= (a commit a finding names, in a repository this run built)", // contains filtered or unexported fields }, }
Volatile is the ONE table of run-owned values. It is the whole set of things a transcript may leave uncompared, and it is shared so that "this transcript is green" means the same in every binary. Growing it is a reading, not a call site's decision -- which is what the refusal below is for.
The entries are the ones docs/SPEC-TOOLWORK.md names: `at=`, `took=`, `created=`, a temporary directory, a message's `id=`, a fresh sha, a name a recorded fixture carries, the stamp on a `branch=` nova-secrets seals on, and the commit a finding's `at=` names in a repository the run built.
FIVE OF THE SIX ARE TOKEN-ANCHORED, and the sixth says why it is not. A norm that names a field replaces only a whitespace-delimited token spelled `<field>=<value>` in full (Norm.apply, transcript.go): a pattern that ran over the whole line would let an entry declared for one field swallow a neighbour's value -- `Field{Name:"sha"}` would also normalise `base_sha=`, and `took` would also normalise `last_took=`. TestAVolatileEntryNeverSwallows- ANeighbouringFieldsValue now holds every entry to it, by the shape of the mistake rather than by the entry, so a sixth entry that forgets is one row of a table away from being caught.
`tmpdir` is the exception and is sound without a field: Path replaces its literal absolute directory only at the end of a token, before `/`, or before the comma used around a path in prose, so it covers descendants without swallowing a longer path that shares the prefix.
Functions ¶
func Block ¶
Block indents a set of lines for a failure message, so the document's block and the tool's stand under each other and can be read side by side.
func ExampleCommands ¶
ExampleCommands returns the command lines of a banner's first `example:` block that run the tool: the lines up to the first blank line under the heading, each with any leading NAME=value environment assignments removed, that then begin with the tool's name. A line continued with ` \` is one command, counted by its first line. A setup line (mkdir, cp) is part of the sitting and is not counted: it is not a use of the tool.
func ExampleLines ¶
ExampleLines returns the command lines under a usage banner's `example:` heading — those beginning with the tool's own name, whitespace collapsed so a banner may align its flags. It returns an error rather than nothing when the block is missing, because a banner without one is the failure.
func ExecuteWith ¶
ExecuteWith is Execute against a bench that may not be able to run every step.
A SKIPPED STEP DOES NOT STOP THE SITTING, and that is a judgement worth stating: the transcripts are sittings, so a skipped step can leave a later one comparing against a state that never happened. It is still better than the alternative, because the steps that DO run are then compared rather than silently abandoned, and a consequent failure names the command it is under -- which a reader can follow back to the skip printed above it. A section whose later steps depend on a skipped one wants its `Requires:` on those steps too, and one that does not is a defect in the document, not a pass.
func FirstRun ¶
FirstRun returns the lines of the fenced transcript under a tool's `### First run`, which must live inside that tool's own `## <tool>` section.
func HowItWorksLength ¶
HowItWorksLength returns how many lines the how-it-works paragraph takes: from its label up to the first blank line or the line that opens the first run, whichever comes first; 0 when the banner has no such paragraph near the top.
func HowItWorksLine ¶
HowItWorksLine returns the 1-based line of the banner that opens the how-it-works paragraph, when one opens within the first HowItWorksWithin lines; 0 when none does.
func OpeningSentence ¶
OpeningSentence returns the sentence of a banner's first line, the part after `<tool>: `, or an error naming what the line lacks. The line is one sentence saying what the tool does: it names the tool, then says it in at least three words, with no second sentence, no usage line and no pointer to another document in place of the answer.
func RepeatedSections ¶
RepeatedSections returns the names that head more than one `## ` section, in first-appearance order. An empty return is the only healthy answer: a repeated name means every reader of that document -- Section, FirstRun, and the person who opened it looking for one place to change -- sees a different half of it.
func Section ¶
Section returns the body of a top-level `## <name>` section of a markdown document, up to the next top-level heading.
func SectionNames ¶
SectionNames returns every top-level `## <name>` heading in a markdown document, in the order they appear and with repeats kept. Section() reads the FIRST match of a name and stops there, which is correct for a document where a name appears once and silently wrong for one where it appears twice: the second section is then read by nobody and drifts unwatched. Callers that mean "this document names each tool once" ask here rather than inferring it from a lookup that cannot fail.
func Shape ¶
Shape reduces an output line to the part a transcript promises: the two-token event prefix, then the field names in order. Everything after the ": " that closes the fields is a run's own business — scores, counts, paths, snippets — and is deliberately not compared, so that a transcript stays a document rather than becoming a fixture. A line that is not an event line (no upper-case first token) reduces to "".
func SplitShell ¶
SplitShell splits a documented command line the way the shell a reader is typing into would. Some transcripts' arguments are sentences -- a passage, a note, a reason -- so strings.Fields would hand the tool more arguments than the reader typed.
The grammar is a NARROW subset of the shell's, and the part it does not read it REFUSES by name rather than guessing at: an argv that is not the reader's argv is a green that means nothing, and it is invisible in the failure message because the line printed there is still the document's.
- whitespace separates arguments;
- a single-quoted run is ONE argument, taken literally: inside it a backslash, a double quote and a `$` are the argument's own characters, exactly as the shell has it;
- a double-quoted run is ONE argument, and inside it a backslash escapes only `"`, `\`, `$` and a backquote -- before any other character the backslash is a character of the argument, which is again the shell's rule;
- a backslash OUTSIDE quotes, a backquote anywhere, and an unterminated quote of either kind are refused.
NOTHING IS EXPANDED. `$PWD` reaches the Runner as the six characters the document writes, because only the caller's package knows what its transcript means by them -- the deleted nova-merge tool's test declares a Path norm for exactly that spelling. A transcript that needs a value expanded says so to its Runner; it does not get one from here.
func Transcript ¶
Transcript returns the lines of the fenced block under `### <heading>` inside a tool's own `## <tool>` section. FirstRun is this with the one heading every tool carries; a caller that wants another subsection -- `### Refusals`, whose lines are as much a promise about what the tool prints as the first run's are -- names it here. The heading is given without its `### `.
func VolatileNames ¶
func VolatileNames() []string
VolatileNames returns the table's names in the table's order, which is the sentence a refusal shows a reader who has to choose one.
Types ¶
type Conditions ¶
type Conditions struct {
// GOOS is the platform, normally runtime.GOOS. An empty GOOS means no step
// may be skipped for its platform, so a platform-bound step runs and fails
// -- the safe default for a caller that forgot to say where it is.
GOOS string
// Have answers whether one stated requirement is met. A nil Have means
// nothing stated can be met, so every Requires step is skipped.
Have func(requirement string) bool
}
Conditions is what this bench can offer a transcript. A step whose stated Platform is not this one, or whose stated Requires this bench cannot meet, is SKIPPED and returned as a Skip rather than run and failed.
type Field ¶
type Field struct {
// Name is an entry's name in the Volatile table. A name the table does not
// hold is refused.
Name string
// Doc is what the document writes; Run is what this run made.
Doc, Run string
}
Field names one entry of the Volatile table for one transcript.
Doc and Run are the two spellings of a value no pattern can match -- a directory this run made -- and are given for a table entry that asks for them and for no other. Everything else on the line is compared as written.
type Norm ¶
type Norm struct {
// Name is read in a failure message, so it is a noun phrase.
Name string
// Re matches the whole value INCLUDING its field name, so that a norm
// declared for one field cannot quietly swallow another's value. For a norm
// built by Instant it is anchored and matched against ONE token of the line
// at a time -- see field below.
Re *regexp.Regexp
// As is what a match becomes on both sides of the comparison.
As string
// contains filtered or unexported fields
}
Norm is one DECLARED normalisation: a value that belongs to the run rather than to the document, named so a reader of the test knows what is not being compared. It is applied to the document's line and to the tool's line alike, and a line the document spells any other way does not match, stays what the document wrote, and fails.
func Elide ¶
Elide declares a normalisation this package has no constructor for. The name is what a reader of the failing test is told is not compared, so it says what the value IS rather than what it looks like.
func GoBuild ¶
func GoBuild() Norm
GoBuild declares the tail of a `version` line -- `<goos>/<goarch> go<version>` -- which is the machine the transcript was recorded on rather than anything the document promises. The version word before it is NOT covered here; that is Version's, and a `version` line wants both declared.
Several sections already paste a real triple (`devel linux/amd64 go1.26.5`), so this reduces both sides to the same sentence rather than asking the document to carry a placeholder it has no convention for.
The two tokens must stand as two whole tokens: an unanchored pattern here was a ReplaceAll over every line of every step of the sitting, which is the thing the Norm contract above exists to forbid.
func Instant ¶
Instant declares that the named field's value is an RFC3339 instant in UTC -- the format these binaries print -- and belongs to the run. The value is PARSED, not merely shaped: what the tool printed has to be a real instant before this norm will agree that it is the run's, because a normalisation that erases an impossible date erases the finding with it.
func Path ¶
Path declares that a path the document writes stands for a directory this run made. `from` is what the document says; both sides are reduced to it, so the documented spelling is what a failure message shows.
func Recorded ¶
Recorded declares that a name a recorded fixture carries (`run`, an organization or a repository the recording was made against) is written in the document as the variable a reader sets (`doc`, such as $ORG). The name is replaced whole: a word boundary before it, and `/`, a blank, `,` or the end of the line after it, so a longer name containing it is compared as written. CompareTranscript refuses a declaration whose doc is not a `$NAME` a documented command types, whose run the document prints as written, or whose name or variable is declared twice.
func Version ¶
func Version() Norm
Version declares the version word of a `version` line: the word a build stamps itself with.
It has to be declared, and the reason is worth writing down. Under `go test` the binary is not stamped and every one of these tools prints `devel` (pkg/buildinfo's Unknown). A build a reader makes -- `go build ./cmd/nova-review && ./nova-review version` -- prints the stamp, today `v0.16.0-dev.<base>.0.<date>-<sha>`. So a transcript that pastes `devel` is green under `go test` and FALSE for the reader it is written for, and one that pastes the stamp is true for the reader and red in the test, and the sha in it changes with every commit. The document therefore shows what a reader sees, and the word is compared as a SHAPE: a stamp or `devel`, and nothing else -- a tool that answered `unknown`, or printed nothing at all, still fails.
type Problem ¶
type Problem struct {
// Step is the documented command the disagreement is under.
Step Step
// Message is the whole complaint, already formatted.
Message string
}
Problem is one disagreement between the document and the run, spelled as the sentence a reader of the failing test is shown.
func Compare ¶
Compare checks one command's whole output against the block written under it: same number of lines, same lines, same order, after the declared norms are applied to both sides.
NO EXIT CODE IS REQUIRED, and that is deliberate. The transcripts do not write one down, and the codes are not one alphabet across these tools: nova-ci exits 2 both for a package over its budget -- which its own first run SHOWS as the ordinary outcome -- and for an invocation that could not run at all. What a reader checks their screen against is the lines, so the lines are what is compared, and the code is carried into the failure message as context.
func CompareTranscript ¶
CompareTranscript is the ONE comparison a firstrun_test.go may make (docs/SPEC-TOOLWORK.md documents the rule).
Before it there were three comparisons in this repository and they were worth three different things. A `printed map[string]bool` asked whether each documented line was somewhere in what the tool printed, so an abridged or a reordered block passed. Shape compared a line's event prefix and its field NAMES and dropped every value, so a changed count passed. Execute compared line for line, but with a norm list assembled at each call site, so what was not compared differed per tool and nobody could read the set. A reader of a green build could not tell which of the three a tool's transcript had earned.
One comparator means one answer: same number of lines, same lines, same order, every value compared AS WRITTEN -- except the run-owned values named from the one shared table below, which a test may name and may not invent.
Nothing here asserts or runs a process: it compares a parsed document with the results of a sitting the caller ran, and returns every disagreement. The caller's test decides what is fatal, as the rest of this package does.
func Execute ¶
Execute runs every step in order -- one sitting, as a reader would -- and returns everything the document got wrong. It STOPS at the first command that could not be invoked at all, because every line after it would then be compared against a state that never happened. A STEP THIS BENCH SKIPPED IS RETURNED AS A PROBLEM, and that is the whole difference between Execute and ExecuteWith. Execute is the plain entry point: it is handed no Conditions, so it uses THIS bench's -- runtime.GOOS, and the environment for a stated requirement -- and anything it could not run is a promise in the document that went unchecked here. A caller with a real platform split calls ExecuteWith, logs its skips and decides for itself; the plain entry point must not be green on what it did not run.
It was not. `Execute` passed an EMPTY Conditions, which matches no platform and meets no requirement, and then threw the skips away: every step that stated a precondition was skipped and a whole block could turn its test green having run nothing, printing nothing and counting nothing.
type Result ¶
Result is what a reader saw when they typed one documented command.
func (Result) Lines ¶
Lines returns the lines a reader saw. Every command in this repository writes its outcome to exactly ONE stream -- results on stdout, refusals on stderr -- so a step that wrote to both is reported rather than guessed at: nothing here can know in which order a terminal interleaved them, and a transcript that shows one interleaving is then a promise the tool does not keep.
A blank line the tool printed is KEPT and counted, because the document would have to show it.
type Runner ¶
Runner runs one documented command and returns what the reader would see. The caller supplies it because only the caller's package knows how to call its own binary in process, and what a `< path` in its transcript is relative to.
type Skip ¶
Skip is one step this bench could not run, with the document's own words for why. It is returned rather than swallowed, because a run whose skips are invisible is a green that means less than it looks like: the caller logs them, and a caller that wants them to be failures says so itself.
type Step ¶
type Step struct {
// Line is the `$ ...` line as the document writes it, for failure messages.
Line string
// Args is what a shell would hand the tool, with the tool's own name removed.
Args []string
// Stdin is the file named by a trailing `< path` redirect, or "" when the
// command reads nothing. The path is as the document writes it.
Stdin string
// Want is the block written under the command, in order, with the blank
// line the document leaves between commands dropped from the end. A line
// beginning StderrMarker is one the document shows on standard error.
Want []string
// Platforms are the GOOS values this command's output was recorded on,
// from a trailing `# Platform: darwin` on the command line. Empty means the
// step is the same everywhere, which is the ordinary case.
Platforms []string
// Requires are the things a bench must have before this command can be run
// at all, from a trailing `# Requires: JEV_API_KEY`. A credential, another
// tool's binary, a network service. Empty means the step needs nothing.
Requires []string
// StderrWhole says the marked lines are ALL this command writes to standard
// error, from a trailing `# Stderr: whole`, and that they are to be compared
// the way standard output is: same number, same lines, same order.
//
// It exists because the asymmetry -- stderr compared only for the lines
// shown -- is right for a tool that NARRATES on standard error and wrong for
// one whose findings live there. nova-self-talk prints every finding it has
// to standard error; under the asymmetry alone, dropping all three from its
// transcript is green, which is exactly the abridgement this flag exists to catch. A
// section says which kind it is, per step, rather than the harness guessing.
StderrWhole bool
}
Step is one `$ ` line of a transcript and EVERY line the document says it prints, in the order the document prints them.
func Steps ¶
Steps cuts a transcript's lines into its commands and their outputs. A line opening with `$ ` starts a step; every line after it belongs to that step until the next `$ ` line.
Output standing before the first command, and a `$` line for some other tool, are the document's own bugs and are returned as errors rather than skipped: a skipped line is exactly the abridgement this harness exists to catch.
func (Step) SkipReason ¶
SkipReason says why this step cannot be run here, or "" when it can. goos is the platform the test is on; have answers whether a stated requirement is met, and a nil have means nothing stated can be met.
A step skipped for a reason the document does not state is NOT this function's business: it returns "" and the step runs and fails, which is the right outcome. A step skipped for a reason the file does not state is a defect in the file, not a pass.
type VolatileField ¶
type VolatileField struct {
// Name is how a transcript's test names this value.
Name string
// What a reader of a failing test is told is not compared.
What string
// NeedsPath is true for the kinds of value a pattern must not guess at: a
// path, or a recorded name, whose two spellings the test supplies. A pattern
// broad enough to match any path would swallow the documented paths a reader
// types.
NeedsPath bool
// Repeatable is true for an entry a transcript may name more than once, once
// per documented spelling: a recorded fixture carries more than one name.
Repeatable bool
// contains filtered or unexported fields
}
VolatileField is one entry of the shared table: a value that belongs to the RUN rather than to the document, named so that what is not compared is a short list a reader can check rather than whatever a loose pattern swallowed.
Source Files