Documentation
¶
Overview ¶
Package testrun is the record of the agent using your app: what it tried, what it found, and whether it needed a model to do it.
This is the product's primary object now. The dashboard was built around a different one — events — because this was an analytics tool, and every screen answered "what did your users do". The question a customer opens this product to ask is "did the thing I just shipped break anything", and until this store there was nowhere for the answer to live.
DELIBERATELY TINY, like the acted and alert stores beside it. A run is a name, a verdict, a reason, and the steps that got there. There is no queue, no scheduler and no retry policy in here: those belong to whatever is driving the runner, and a storage package that grows a workflow engine becomes the thing nobody can change.
TWO FIELDS CARRY THE WHOLE ECONOMIC ARGUMENT. Mode says whether this run needed a model, and DurationMs says how long it took. A customer looking at a wall of `replay · 0.6s` rows and one `agent · 48s` row can see, without being told, that they are paying for intelligence only when something actually changed. That is the difference between us and the incumbent, and it is visible in the data rather than asserted in the copy.
Index ¶
Constants ¶
const ( // ModeAgent means a model drove it: the first run of a test, or one where the recording no // longer fit the app. ModeAgent = "agent" // ModeReplay means the recorded plan ran with no model at all. ModeReplay = "replay" )
Mode is how a run was executed.
const ( // StatusPassed means the agent directly observed what the test asked it to verify. StatusPassed = "passed" // StatusFailed means the app did not do what the test describes. This is a bug report. StatusFailed = "failed" // StatusStale means a recorded plan no longer fits the app — a control was renamed, or removed. // // It is a THIRD status on purpose, and the distinction is the one this product must never blur: // a replay cannot tell "renamed" from "gone", and reporting a rename as a failure pages someone // at 2am over a copy change. Stale means "we do not know yet, the agent is looking". StatusStale = "stale" // StatusErrored means the runner itself failed — no browser, no network, no API key. Never a // statement about the customer's app, and never rendered as one. StatusErrored = "errored" )
Status is the verdict.
const DefaultCap = 500
DefaultCap is how many runs are kept. Enough to see a week of a busy repository, and small enough that the whole file is read and written without thinking about it.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Run ¶
type Run struct {
ID string `json:"id"`
Test string `json:"test"`
// Status is one of the constants above.
Status string `json:"status"`
// Mode is agent or replay — the field that shows what a run cost.
Mode string `json:"mode"`
// Reason is the sentence a person reads. For a failure it is the bug report: what was
// expected, what happened, and where.
Reason string `json:"reason"`
StartedAt time.Time `json:"started_at"`
DurationMs int `json:"duration_ms"`
// URL the test ran against.
URL string `json:"url,omitempty"`
// Commit and PR tie a run to what shipped, so a failure can name the change that caused it.
Commit string `json:"commit,omitempty"`
PR int `json:"pr,omitempty"`
// Steps are kept for failures and dropped for passes — see Append. A passing run's timeline is
// noise, and storing every step of every green run is how this file grows without bound.
Steps []Step `json:"steps,omitempty"`
}
Run is one execution of one test.
func (Run) OK ¶
OK reports whether this run needs nobody's attention. Stale does NOT count: it means we do not know yet.
func (Run) Took ¶
Took renders the duration the way a person reads it.
Raw milliseconds are fine for a replay and unreadable for an agent run: "47210ms" is the number that proves replaying is worth it, and printing it in the unit nobody counts in wastes the comparison. Under ten seconds keeps millisecond precision, because that is where the interesting difference lives.
type Step ¶
type Step struct {
N int `json:"n"`
// Do is the human sentence: `click button "Proceed to checkout"`.
Do string `json:"do"`
// Why is what the agent expected it to accomplish. Empty on a replay, which has no reasoning.
Why string `json:"why,omitempty"`
OK bool `json:"ok"`
// Detail is the error when OK is false.
Detail string `json:"detail,omitempty"`
Ms int `json:"ms"`
}
Step is one thing the agent did, kept for the timeline on a failed run.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store holds runs. Same shape and persistence discipline as the acted and alert stores.
func Open ¶
Open loads the store at path.
An EMPTY PATH MEANS IN-MEMORY, which is the convention every sidecar here follows and which the acted store learned the hard way: it tried to persist to "" and crashed the demo on the first write with `rename .tmp: no such file`. The demo, the tests, and any read-only instance all pass "" and must get a working store, not an error.
type SuiteEntry ¶
type SuiteEntry struct {
Test string `json:"test"`
// Status of the LATEST run. What the suite believes right now.
Status string `json:"status"`
Mode string `json:"mode"`
// LastPassed is the newest run that passed, zero if it never has. A test that has never passed
// is a different thing from one that has regressed, and the pane says which.
LastPassed time.Time `json:"last_passed,omitempty"`
LastRun time.Time `json:"last_run"`
Runs int `json:"runs"`
// Recorded means a passing run exists to replay, so this test costs nothing to re-run. The
// column that shows a customer which of their tests are still expensive.
Recorded bool `json:"recorded"`
// Reason is the latest run's sentence, carried so a failing row explains itself in place.
Reason string `json:"reason,omitempty"`
}
SuiteEntry is one test, as of its most recent run.
func Suite ¶
func Suite(runs []Run) []SuiteEntry
Suite collapses the run log into one row per test, worst first.
Ordering is by how much attention the row needs, not by name or recency: a failing test at the bottom of an alphabetical list is a failing test nobody sees.
func (SuiteEntry) NeverPassed ¶
func (e SuiteEntry) NeverPassed() bool
NeverPassed reports a test that has run and never once passed.
Worth its own question because it means something different from a regression: the test may describe something the product has never done, which is as likely to be a wrong test as a broken feature — and telling someone their checkout is broken when the test was wrong burns the trust this product runs on.
type Summary ¶
type Summary struct {
Total int
Passed int
Failed int
Stale int
Errored int
// Replayed is how many needed no model. The number that shows what the suite costs to run.
Replayed int
// SavedCalls is runs that did not pay for a model because a recording existed. Same number as
// Replayed, named for what it means rather than for how it was counted.
SavedCalls int
}
Summary is the headline over a set of runs.
Source Files