Documentation
¶
Overview ¶
Package modelspec reads, checks and writes ModelSpec models (https://github.com/specscore/modelspec).
A model is read from its HCL source (ParseHCL, files named *.modelspec.hcl) or from its JSON interchange form (ParseJSON, files named *.modelspec.json). Both produce the same typed Model plus a list of Findings. Check applies the semantic rules to a set of models, resolving module-qualified references among them, and Model.JSON writes the JSON interchange form.
Check applies one of two profiles. The default implements the standard and nothing else; ProfilePublish adds what public catalogues require. A module is a set of files: Load assigns files to modules (SpecScore layout, file name, or an explicit Assignment), and Check resolves names across a module's files.
The package never exits the process, never touches the network and reads files only through the FS interface it is given.
The HCL reader and the rules for name scopes, reserved names, duplicate concepts, enum values and reference resolution are derived from the ModelSpec support in the SpecScore CLI (https://github.com/specscore/specscore-cli, Apache-2.0); see the NOTICE file at the repository root.
Index ¶
- Constants
- Variables
- func Diff(a, b *Node) string
- func Discover(fsys FS, paths []string) ([]Source, []Finding, error)
- func FileKind(mode fs.FileMode) string
- func HasErrors(fs []Finding) bool
- func IsModelFile(name string) bool
- func Load(fsys FS, files []Source, assign []Assignment) ([]*Model, []Finding, error)
- func Parse(file string, src []byte) (*Model, []Finding)
- func ParseHCL(file string, src []byte) (*Model, []Finding)
- func ParseJSON(file string, src []byte) (*Model, []Finding)
- func ReadSource(fsys FS, name string) ([]byte, error)
- func SortFindings(fs []Finding)
- type Assignment
- type Attr
- type Concept
- type FS
- type Field
- type Finding
- type Form
- type Kind
- type LintOptions
- type Member
- type Model
- type ModuleIdentity
- type Node
- type NodeType
- type NotRegularError
- type OSFS
- func (o OSFS) Abs(name string) (string, error)
- func (OSFS) Lstat(name string) (fs.FileInfo, error)
- func (OSFS) Open(name string) (fs.File, error)
- func (OSFS) ReadDir(name string) ([]fs.DirEntry, error)
- func (OSFS) SameFile(a, b fs.FileInfo) bool
- func (OSFS) Stat(name string) (fs.FileInfo, error)
- func (OSFS) WriteFile(name string, data []byte, perm fs.FileMode) error
- type Options
- type Profile
- type Result
- type Severity
- type SkippedFile
- type Source
- type TooLargeError
- type Unmapped
Constants ¶
const ( RuleSyntax = "syntax" // the source is not valid HCL or JSON RuleEncoding = "encoding" // the source is not valid UTF-8 RuleLimit = "limit" // the source exceeds a size or nesting limit RuleShape = "shape" // valid syntax, but not the structure ModelSpec defines RuleLiteral = "literal" // an HCL value that is not a literal (decision 0009) RuleReference = "reference" // a name that does not resolve (decisions 0014, 0013) RuleReserved = "reserved-name" // a reserved kind token used as a concept name (decision 0015) RuleDuplicate = "duplicate-name" RuleNameForm = "name-form" // a dot in a concept name (decision 0014) RuleEnumValues = "enum-values" RuleType = "unknown-type" RuleAttribute = "attribute" RuleKey = "key" RuleMemberKind = "member-kind" RuleVersion = "modelspec-version" RuleModule = "module" RuleCollection = "collection" RuleUnknown = "unknown-field" // a top-level JSON field the format does not define RuleIO = "unreadable" // a file found by a directory search that cannot be read RuleSkipped = "skipped-file" // a model-named link, pipe or device a search does not read: an error, and its module is not checked RuleStaleTwin = "stale-twin" // a JSON twin that is not what its HCL exports to RuleNameCase = "name-case" // names in one scope that differ only by case RulePublishModuleName = "publish-module-name" RulePublishEntities = "publish-entities" RulePublishProperties = "publish-properties" RulePublishNameForm = "publish-name-form" RulePublishComponent = "publish-component-property" RulePublishQualified = "publish-qualified-entity" )
Rule ids. They are stable: scripts may match on them. Rules whose id starts with "publish-" are enforced only under the publish profile.
const ( // MaxNumberLength is the most characters one number literal may have. The // HCL library reads a number into 512 bits, about 154 decimal digits, and // rounds beyond that, so two different long integers could look equal. With // at most 40 characters (and an exponent within MaxNumberExponent) every // accepted number is read exactly, and two numbers are equal only when they // are the same value. It also bounds the time to read one: the library's // time to read `1e10000000`, 10 bytes, was ten seconds. MaxNumberLength = 40 // MaxNumberExponent is the largest exponent, either sign, a number literal // may have: with at most MaxNumberLength characters a number is then between // 1e-140 and 1e140, a size where a decimal is read, compared and written // exactly and in constant time. MaxNumberExponent = 100 // MaxNameLength is the most bytes, as written, a name may have: a block // label, an identifier or attribute name, a type, entity, component, enum or // kind reference, and an item of `key` or `use` (JSON: every object key and // the same strings). The standard states no length. Real names are a few // words, and where the databases a model describes limit an identifier it is // between 63 and 128 bytes, so 255 leaves room. MaxNameLength = 255 // MaxMessageBytes is the most bytes of one finding's message. Longer ones are // cut with a marker (clipText), and so is every piece of the user's text // that a message echoes (MaxEchoBytes), so that one finding cannot be large // however large the input is. MaxMessageBytes = 1024 // MaxEchoBytes is the most bytes of one piece of the user's text, such as a // name or a value, a message repeats: as many as a name may have, so a name // is always shown whole. MaxEchoBytes = MaxNameLength )
Limits on single tokens. They are applied to the lexer's tokens before the HCL parser runs (precheck) and to the JSON reader's tokens as it reads, so that no later stage handles a token it was not built for. The standard sets none of them; they are this tool's, stricter than the standard, and each is refused as a `limit` finding that says what was counted and what the limit is.
const ( HCLSuffix = ".modelspec.hcl" JSONSuffix = ".modelspec.json" )
Extensions of the two model file forms.
const ( // MaxInputBytes is the largest source, in bytes, that is read. The lexer holds // every token of a file in memory at once, and the peak is about 420 to 470 // MiB for each MB of one-byte tokens, so 1 MiB keeps one file under 500 MiB // and about a second of work. Chinook's model is under 8 KB and the largest // corpus file 30 KB; a module is a set of files, each under the limit. MaxInputBytes = 1 << 20 // MaxDepth is the deepest nesting of HCL braces, brackets, quoted strings and // heredocs, counted from tokens (blocks, lists, objects), and of JSON arrays // and objects. Real models nest four or five levels in HCL, six to eight in // the JSON form. MaxDepth = 64 // MaxHeredocLines is the most lines one heredoc may have. The lexer makes one // piece of each line, and the HCL parser joins the pieces of a template one at // a time, copying the text and shifting the rest of the list each time: time // that grows with the square of their number. (Pieces that the lexer also // starts at each `$` and `%`, and in quoted strings, are removed before the // parser sees them: see parserInput.) No real heredoc, a query or a pattern, // comes near the limit. MaxHeredocLines = 1000 // MaxSyntaxFindings is the most syntax errors, or non-literal tokens, reported // for one HCL file. MaxSyntaxFindings = 50 )
Limits and the literal-only rule, applied before a source is parsed, so that hostile or accidental input is a finding and not a crash. The HCL parser is recursive and a stack overflow in Go is fatal and cannot be recovered. Rather than bound the recursive constructs one by one, the HCL source is lexed (the lexer does not recurse) and refused before parsing if it holds any token that a literal value cannot contain; what is left to bound is the nesting of brackets.
const MaxFindings = 1000
MaxFindings is the most findings one reader, one check and one run keep. A file of four megabytes can hold millions of mistakes, and listing them all would use memory and output without end; past the limit the findings are counted, and one finding says how many were left out. The count and the severity of that finding are the whole of what was dropped, so a run that dropped an error still has one, and its exit code is the same as with no limit.
const SpecVersion = "1.0-draft"
SpecVersion is the only value the JSON form's "modelspec" field may carry (spec/json-format.md, "Format Identity").
Variables ¶
var PrimitiveTypes = map[string]bool{ "string": true, "int": true, "float": true, "bool": true, "decimal": true, "uuid": true, "date": true, "time": true, "datetime": true, "document": true, "json": true, "any": true, }
PrimitiveTypes is the type vocabulary of ModelSpec v0 (spec/core-model.md).
var Profiles = []Profile{ProfileDefault, ProfilePublish}
Profiles lists the profile names.
var ReservedNames = map[string]bool{ "entities": true, "components": true, "enums": true, "collections": true, "recordsets": true, }
ReservedNames are the five kind tokens that no concept may be named (ModelSpec decision 0015).
Functions ¶
func Diff ¶
Diff returns "" when a and b are the same JSON value with the same key and array order, and otherwise a sentence naming the first difference and its path. Whitespace and source lines are not compared.
func Discover ¶
Discover expands paths into model files. A file is taken as given (it must be a model file); a directory is searched recursively for *.modelspec.hcl and *.modelspec.json, and in a SpecScore layout models directory for any *.hcl. Hidden directories and node_modules are skipped. A search follows no symbolic link, to a directory or to a file: a link that would have been a model file is an error finding (skipped-file) that names it. A file named on the command line may be a link to a regular file, and is read through. A file reached by two names is returned once, under the first name. The result is sorted by path. The findings are errors (skipped-file) for model files a search did not read, and warnings for files it found but cannot read.
func FileKind ¶
FileKind says what a file that is not a regular file is: "a directory", "a named pipe", "a symbolic link", "a character device".
func IsModelFile ¶
IsModelFile reports whether a file name is a standalone ModelSpec model file.
func Load ¶
Load reads and parses the files and assigns each to a module:
- --module assignments (assign) win;
- a .hcl file directly inside …/modules/<id>/models/ belongs to module <id>, together with every other file there;
- otherwise an HCL file <name>.modelspec.hcl is module <name> by itself, and a JSON file is module.name, or the file name's stem when it has none.
A JSON file is the interchange copy (a Twin) of the module it sits with, and not a second module, when it is assigned to a module that has HCL files, sits in the models directory of a layout module, or is X.modelspec.json beside X.modelspec.hcl (both among the files); an explicit assignment of the JSON file itself to another module wins. A file larger than MaxInputBytes is not read. A file that cannot be read is an error, not a finding.
func ParseHCL ¶
ParseHCL reads a ModelSpec HCL source. file is used in findings and, without its .modelspec.hcl suffix, as the module short name. The source is parsed with the real HCL parser; a syntax error yields a Broken model.
func ParseJSON ¶
ParseJSON reads the JSON interchange form of a model (spec/json-format.md): the structure the specification defines and the validation it lists (a supported "modelspec" version, module.id and module.version, and well-formed groups). Constraints a single consumer adds are the publish profile's, not this reader's: entities are optional, module.name is optional, and an entity may have no properties.
func ReadSource ¶
ReadSource reads a file, which every command that reads a model goes through, for every kind of file (a model, the JSON operand of export --check, a --module path). A symbolic link is followed to its target, which must be a regular file: that is checked before the file is opened (opening a named pipe waits for a writer), and again on the opened file, and what is read is at most MaxInputBytes+1, so a file that grows after the checks, or that reports a size of zero and never ends, is still bounded. One over the limit is refused with a *TooLargeError, and one that is not regular with a *NotRegularError.
func SortFindings ¶
func SortFindings(fs []Finding)
SortFindings orders findings by file, line, rule and message, so output is deterministic.
Types ¶
type Assignment ¶
Assignment places the model files under Path (a file or a directory) in module Module (--module Module=Path). It wins over every other module rule.
type FS ¶
type FS interface {
// Open opens a file for reading. It is read only through ReadSource, which
// refuses what is not a regular file and bounds how much is read.
Open(name string) (fs.File, error)
ReadDir(name string) ([]fs.DirEntry, error)
// Stat follows symbolic links; Lstat does not.
Stat(name string) (fs.FileInfo, error)
Lstat(name string) (fs.FileInfo, error)
WriteFile(name string, data []byte, perm fs.FileMode) error
// Abs returns the absolute form of a path, cleaned, with symbolic links left
// as they are: the path as the file was given or found. The SpecScore layout
// is read from it.
Abs(name string) (string, error)
// SameFile reports whether two FileInfos from Stat describe one file, however
// it was reached: by a relative or an absolute path, through a symbolic link,
// under another spelling of the same name on a case-insensitive filesystem.
SameFile(a, b fs.FileInfo) bool
}
FS is the filesystem the loader reads and writes through. OSFS is the real one; tests use in-memory implementations.
type Finding ¶
type Finding struct {
File string `json:"file"`
Line int `json:"line,omitempty"` // 0 when the position is not known
Rule string `json:"rule"`
Severity Severity `json:"severity"`
Message string `json:"message"`
}
Finding is one located problem in one file.
func Check ¶
Check applies the semantic rules of the profile in opts to the models and returns the sorted findings.
Models are grouped into modules by Model.Group (Load sets it from the module rules): concept names are unique per module across all its files, and a reference resolves against the whole module. A module-qualified reference (decision 0014) names another module and resolves among the models given, by Model.Name. A Twin (the JSON copy of an HCL module) is checked as a module of its own but is not a second source for the name. Broken models are skipped, and references into their modules are not reported.
type LintOptions ¶
type LintOptions struct {
Profile Profile
// Modules assigns files to modules explicitly. The assigned files are linted
// even when no path names them.
Modules []Assignment
}
LintOptions configures Lint.
type Member ¶
Member is a property (entity), field (component or collection) or column (recordset). Column names may repeat; the others are unique.
type Model ¶
type Model struct {
File string // as given or found, used in findings
Form Form
// Name is the module short name used to resolve module-qualified
// references. ParseHCL sets the file name without .modelspec.hcl; ParseJSON
// sets module.name, or that same file-name stem when the JSON has none; Load
// then applies the module rules (SpecScore layout, --module).
Name string
// Group identifies the module the file belongs to: files with the same Group
// are one module. ParseHCL and ParseJSON set it to File (one file, one
// module); Load groups the files of a module.
Group string
// Twin is set by Load on a JSON file that sits beside the HCL file of the same
// name: it is the interchange copy of that module, not a second module, and
// references into the module resolve to the HCL.
Twin bool
// TwinOf is the HCL model whose export a Twin is the copy of, when that is one
// HCL file; Check compares the two.
TwinOf *Model
// Root is the parsed document of a JSON file (nil for HCL).
Root *Node
// Module is the identity from a JSON file; nil for HCL.
Module *ModuleIdentity
// ModuleLine is the line of the JSON module object (0 for HCL).
ModuleLine int
// Concepts are in source order.
Concepts []*Concept
// Unmapped lists HCL constructs the JSON form does not define.
Unmapped []Unmapped
// Projections and Migrations are carried through unchanged from a JSON
// file; nil otherwise.
Projections *Node
Migrations *Node
// Broken is set when the source could not be read into a model; Check skips
// it and does not report references into its module.
Broken bool
// Incomplete is set when a file of the model's module was found by a search
// and not read (a link, a pipe, a device: rule skipped-file). Check does not
// check an incomplete module: a partial load would report what is missing as
// unresolved references.
Incomplete bool
}
Model is one parsed ModelSpec file. A module is one or more Models: the HCL files of a module share a Group (see Load); a JSON file is a whole module.
func (*Model) ExportDrift ¶
func (m *Model) ExportDrift(committed []byte, id ModuleIdentity) string
ExportDrift compares what the model exports to with committed JSON. It returns "" when they are the same document with the same key and array order (whitespace does not count), and otherwise one sentence saying what is wrong.
Module identity is not in the HCL. When id is the zero value the committed JSON's own module object supplies it, so the comparison covers everything except the identity; pass id to compare that too.
func (*Model) HasConcept ¶
HasConcept reports whether the model declares a concept of the kind with the name.
func (*Model) JSON ¶
func (m *Model) JSON(id ModuleIdentity) (*Node, error)
JSON builds the JSON interchange form of the model: the "modelspec" field, "module", then the groups components, enums, entities, collections, recordsets, projections and migrations, in that order. Concepts keep source order. An entity's or collection's attributes come first, then its members; recordset columns are an ordered array with "name" first (decision 0007).
The mapping is exactly the one spec/json-format.md defines. Two things it leaves open are refused rather than invented: HCL `index`, `projection` and `migration` blocks (no document says how they map to JSON), and module identity for HCL, which the grammar has no place for, so the caller must supply id and version (the format requires those two; name is written only when given, or when the model refers to its own module by name). A model read from JSON already has its identity, which id overrides when non-zero.
type ModuleIdentity ¶
ModuleIdentity is the identity the JSON form carries and HCL does not.
type Node ¶
type Node struct {
Type NodeType
Str string // NodeString: the value; NodeNumber: the number's text
Bool bool // NodeBool
Fields []Field // NodeObject, in order
Items []*Node // NodeArray
Line int // 1-based source line; 0 when built in memory
// Dups, set on the root returned by ParseNode only, lists every object key
// that appears twice in the same object, anywhere in the document, at the
// line of the repeat. Most JSON readers keep the last of the two, a few the
// first, so a repeat is never harmless.
Dups []Field
}
Node is a JSON value that keeps object key order and source lines. encoding/json's maps cannot do that, and ModelSpec's own tooling compares a model with its registered copy with the order of keys and arrays significant (the Directory's check says "the order of entities, properties and other keys counts"), so export --check does too.
func ParseNode ¶
ParseNode reads one JSON document into a Node, keeping key order and lines. Anything after the document is an error.
type NotRegularError ¶
type NotRegularError struct {
File string
What string // what it is: "a named pipe", "a symbolic link to a character device"
}
NotRegularError is the error of a file that is not a regular file: a device, a named pipe, a socket or a directory, which has no size and may never end.
func (*NotRegularError) Error ¶
func (e *NotRegularError) Error() string
type OSFS ¶
type OSFS struct {
// contains filtered or unexported fields
}
OSFS is the operating system's filesystem. The zero value is ready to use.
type Options ¶
type Options struct {
Profile Profile // "" means ProfileDefault
}
Options configures Check.
type Profile ¶
type Profile string
Profile selects the rules Check applies.
The default profile implements the standard as written in the ModelSpec repository (spec/core-model.md, spec/hcl-authoring.md, spec/json-format.md and the decisions) and nothing else: a model the standard allows is never refused because some consumer's reader is narrower. The publish profile adds what a model needs to be listed in public catalogues, which is what the Directory's JSON reader requires on top of the standard.
func ParseProfile ¶
ParseProfile converts a flag value to a Profile.
type Result ¶
type Result struct {
Findings []Finding
Files int
Models []*Model
// Skipped is every model-named file a search did not read, whole: the findings
// are capped, this is not.
Skipped []SkippedFile
// Notes say what Lint did beyond the paths given: a module is the unit of
// checking, so the files of a module that were not given were checked too.
Notes []string
}
Result is the outcome of Lint.
func Lint ¶
func Lint(fsys FS, paths []string, opts LintOptions) (Result, error)
Lint discovers, loads and checks the paths together with the files assigned by opts.Modules: the parse findings and the semantic findings, sorted. A module is always checked whole: when a file given belongs to a module with more files on disk (a SpecScore layout models directory, or the HCL and JSON of one module), the others are loaded too, and their findings are reported with their paths. With no paths and no assignments it is an error, as is finding no model file.
type Severity ¶
type Severity string
Severity ranks a finding. Only SeverityError makes a model invalid.
type SkippedFile ¶
type SkippedFile struct {
Finding Finding // the skipped-file finding for it
Module string // the --module name it was found under, "" when it was not found through one
Affected []*Model // the models of its module, which are marked Incomplete
// contains filtered or unexported fields
}
SkippedFile is a model-named file that a search met and did not read (rule skipped-file). Unlike the finding, which the findings cap can cut from a long list, it is kept whole: Lint returns every one, with the models that its absence makes incomplete.
type Source ¶
Source is a model file to read: Path as given or found (used in findings) and its absolute form, symbolic links not resolved (used for the module layout).
type TooLargeError ¶
TooLargeError is the error of a file over MaxInputBytes, which is not read (further than the limit). Partial is set when the size is not known: the file grew after it was checked, and Size is how much was read before reading stopped.
func (*TooLargeError) Error ¶
func (e *TooLargeError) Error() string