Documentation
¶
Overview ¶
Package rules lets a customer state what their data is supposed to look like, in a file they own.
The built-in checks find problems that are wrong in any dataset. A rule expresses something only the customer knows: that an order amount is never negative, that status is one of four values, that every invoice must reference a known account. Those expectations are the ones that catch the defects that actually cost money, and they cannot be inferred from the data — a column full of wrong-but-plausible values looks exactly like a column full of right ones.
Rules are also the destination for the agent's proposals: the model suggests a rule, a human reads it, and once accepted it becomes a deterministic check that runs on every future audit without the model. That is the only way a defect the model found on one run gets found on every run — see Materialize for the part the model cannot write itself.
Index ¶
- Constants
- func Evaluate(ctx context.Context, e *engine.Engine, ds *profile.Dataset, f *File, ...) ([]finding.Finding, error)
- func Materialize(ctx context.Context, e *engine.Engine, ds *profile.Dataset, f *File) error
- func ProposalHeader(root string, when time.Time) string
- func RenderProposals(w io.Writer, ps []Proposal, header string) error
- type Expectation
- type File
- type Proposal
- type Rule
- type ValuesSource
Constants ¶
const (
// MaxMaterializedValues is the largest permitted set values_from will fill in.
MaxMaterializedValues = 50
)
The permitted set is bounded so that what a person is asked to accept is something they can actually read. A vocabulary of four statuses is an expectation; four thousand distinct values is a copy of the column, and accepting it would protect nothing while looking like protection. The same argument bounds one value's length: a rule enumerates a vocabulary, not free text.
Variables ¶
This section is empty.
Functions ¶
func Evaluate ¶
func Evaluate(ctx context.Context, e *engine.Engine, ds *profile.Dataset, f *File, log *slog.Logger) ([]finding.Finding, error)
Evaluate applies every rule to the dataset.
A rule that matches no table or column is itself reported. Silence from a rule is ambiguous — it means either "your data is fine" or "this rule never ran" — and the second is dangerous, because the customer is relying on it.
func Materialize ¶
Materialize resolves every rule that reads its values from the data, in place.
It exists because expect: one_of is the most valuable rule kind and the one a model cannot write. Its body is literally a list of cell values, and the egress guard never shows a model one — under the default policy even sample_values comes back as shapes. The answer is not to relax the guard for rule proposal. The model proposes the shape of the expectation ("status is drawn from a fixed vocabulary") and the engine fills in the contents here, in the customer's own process, from the customer's own data. The concrete list is what a person reviews before the rule is accepted.
The values this writes are cell values. They belong in the rules file and on the accept screen, and they must never travel back to the model.
A resolved rule is an ordinary one_of rule and says so: values_from is cleared, which also makes materializing twice a no-op rather than a second reading of data that may have changed underneath.
func ProposalHeader ¶
ProposalHeader is the preamble RenderProposals takes, naming what was audited and when.
func RenderProposals ¶
RenderProposals writes proposals as a rules file: the same document a customer loads with --rules, ready to read and edit.
This is the one place a proposal's permitted values are written out, and deliberately so. A report is a file that gets emailed and pasted into tickets, so it carries the shape of a proposed rule and the count of what it permits; this is a rules file, whose entire purpose is to be loaded back into Veritix, and a one_of rule without its values is not a rule at all. It is written when somebody asks for it, on their own machine, like every other path to a verbatim value.
Nothing rendered here is in force. The header says so, because a file of rules that looks authoritative and is not is worse than no file: the point of the review step is that a person decides, and they cannot decide what they think is already decided.
Types ¶
type Expectation ¶
type Expectation string
Expectation is what a rule asserts about the data.
const ( // ExpectNotNull requires every row to have a value. ExpectNotNull Expectation = "not_null" // ExpectUnique requires no value to repeat. ExpectUnique Expectation = "unique" // ExpectPositive requires values greater than zero. ExpectPositive Expectation = "positive" // ExpectNonNegative requires values of zero or more. ExpectNonNegative Expectation = "non_negative" // ExpectOneOf restricts a column to a fixed set of values. ExpectOneOf Expectation = "one_of" // ExpectMatches requires values to match a regular expression. ExpectMatches Expectation = "matches" // ExpectRange bounds a numeric column. ExpectRange Expectation = "range" // ExpectNotFuture forbids dates after the audit runs. ExpectNotFuture Expectation = "not_future" // ExpectReferences requires every value to exist in another column. ExpectReferences Expectation = "references" // ExpectSQL treats rows matching a WHERE clause as violations. ExpectSQL Expectation = "sql" )
func Expectations ¶
func Expectations() []Expectation
Expectations lists every assertion a rule can make, in the order a caller offering them as a choice should offer them.
func ParseExpectation ¶
func ParseExpectation(s string) (Expectation, error)
ParseExpectation reads an expectation by name.
type File ¶
type File struct {
// Version is the document format version. Only 1 exists.
Version int `yaml:"version"`
// Rules are the expectations to enforce.
Rules []Rule `yaml:"rules"`
}
File is a rules document.
func (*File) Covering ¶
Covering names a rule in this file that already asserts what r asserts, or "" if none does.
A model that re-proposes protection the customer already has is the same failure as one that re-reports what the deterministic pass already found, and the answer is the same: say so where the model is looking. display is the target's human-readable name, because a rule a person wrote targets "customers.csv" while a resolved rule targets the SQL name.
type Proposal ¶
type Proposal struct {
// Rule is the expectation, resolved: the profile's own table and column
// names, and any values already materialized from the data.
Rule Rule `json:"rule"`
// Rationale is the argument for it, for the person deciding whether to
// accept. It is prose from whoever proposed the rule and carries no
// authority of its own.
Rationale string `json:"rationale,omitempty"`
// ViolationsNow is what the rule measured against the data in front of it
// when it was proposed.
ViolationsNow int64 `json:"violations_now"`
// Display is the source a person recognizes — "sales.xlsx#Q1" — since
// Rule.Table carries the SQL name the rule has to be written against.
Display string `json:"display,omitempty"`
}
Proposal is a rule somebody suggested and nobody has accepted yet.
It is deliberately not a finding, and the two are not checkable the same way. A finding asserts that this data is wrong now, so one whose count query returns zero is refused. A proposal asserts that an expectation should hold in future, so one with no violations today is the best kind there is: "status is drawn from these four values" is worth having precisely because it holds now and should keep holding. Filing proposals among the findings would also report the same rows twice, once as the defect and once as the rule that would have caught it.
Nothing here is applied. An accepted rule raises errors on future data and can fail a CI gate, which is not a thing a model gets to do unattended.
func (Proposal) ID ¶
ID is a stable, URL-safe handle for a proposal, digested for the same reason finding.Finding.ID is: the key contains customer column names and an id ends up in URLs and access logs.
type Rule ¶
type Rule struct {
// ID names the rule in reports. Required and unique within a file.
ID string `yaml:"id"`
// Description explains what the rule is for, in the report.
Description string `yaml:"description,omitempty"`
// Severity defaults to error when omitted: a rule the customer wrote
// states an expectation they hold, not a suggestion. It is a pointer so
// that an omitted severity is distinguishable from an explicit "info".
Severity *finding.Severity `yaml:"severity,omitempty"`
// Table selects tables by name or display path. A "*" matches any run of
// characters, so "*.csv" applies a rule to every CSV in the dataset.
Table string `yaml:"table"`
// Column selects a column, with the same globbing.
Column string `yaml:"column,omitempty"`
// Expect is the assertion.
Expect Expectation `yaml:"expect"`
// Values enumerates the permitted values for one_of.
Values []string `yaml:"values,omitempty"`
// ValuesFrom asks Veritix to fill Values in from the data instead of
// listing them, which is how a rule proposed by a model gets a value
// list: the model is never shown a cell value, so it can propose the
// shape of the expectation but not its contents. Materialize resolves
// it; an accepted rule carries the concrete list.
ValuesFrom ValuesSource `yaml:"values_from,omitempty"`
// Pattern is the regular expression for matches.
Pattern string `yaml:"pattern,omitempty"`
// Min and Max bound a range. Either may be omitted.
Min *float64 `yaml:"min,omitempty"`
Max *float64 `yaml:"max,omitempty"`
// References names the table and column that must contain every value,
// written as "table.column".
References string `yaml:"references,omitempty"`
// Where is the violation predicate for expect: sql.
Where string `yaml:"where,omitempty"`
// IgnoreCase compares text without regard to case or surrounding spaces.
// Usually what a human means, and rarely what SQL does by default.
IgnoreCase bool `yaml:"ignore_case,omitempty"`
// AllowMissing exempts null and blank values from the rule, so that
// "must be one of these four values" does not also mean "and must be
// present". Completeness is a separate expectation.
AllowMissing bool `yaml:"allow_missing,omitempty"`
// Message overrides the generated finding title.
Message string `yaml:"message,omitempty"`
// Remedy overrides the generated advice.
Remedy string `yaml:"remedy,omitempty"`
}
Rule is one expectation about the data.
type ValuesSource ¶
type ValuesSource string
ValuesSource names where a one_of rule's permitted values come from when the rule does not list them itself.
const ValuesFromCurrent ValuesSource = "current"
ValuesFromCurrent fills the permitted set with the distinct values the column holds when the rule is materialized.