modelspec

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: Apache-2.0 Imports: 18 Imported by: 1

Documentation

Overview

Package modelspec reads, checks and writes ModelSpec models (https://github.com/specscore/modelspec).

A model is read from its HCL source (ParseHCL, files named *.modelspec.hcl) or from its JSON interchange form (ParseJSON, files named *.modelspec.json). Both produce the same typed Model plus a list of Findings. Check applies the semantic rules to a set of models, resolving module-qualified references among them, and Model.JSON writes the JSON interchange form.

Check applies one of two profiles. The default implements the standard and nothing else; ProfilePublish adds what public catalogues require. A module is a set of files: Load assigns files to modules (SpecScore layout, file name, or an explicit Assignment), and Check resolves names across a module's files.

The package never exits the process, never touches the network and reads files only through the FS interface it is given.

The HCL reader and the rules for name scopes, reserved names, duplicate concepts, enum values and reference resolution are derived from the ModelSpec support in the SpecScore CLI (https://github.com/specscore/specscore-cli, Apache-2.0); see the NOTICE file at the repository root.

Index

Constants

View Source
const (
	RuleSyntax     = "syntax"        // the source is not valid HCL or JSON
	RuleEncoding   = "encoding"      // the source is not valid UTF-8
	RuleLimit      = "limit"         // the source exceeds a size or nesting limit
	RuleShape      = "shape"         // valid syntax, but not the structure ModelSpec defines
	RuleLiteral    = "literal"       // an HCL value that is not a literal (decision 0009)
	RuleReference  = "reference"     // a name that does not resolve (decisions 0014, 0013)
	RuleReserved   = "reserved-name" // a reserved kind token used as a concept name (decision 0015)
	RuleDuplicate  = "duplicate-name"
	RuleNameForm   = "name-form" // a dot in a concept name (decision 0014)
	RuleEnumValues = "enum-values"
	RuleType       = "unknown-type"
	RuleAttribute  = "attribute"
	RuleKey        = "key"
	RuleMemberKind = "member-kind"
	RuleVersion    = "modelspec-version"
	RuleModule     = "module"
	RuleCollection = "collection"
	RuleUnknown    = "unknown-field" // a top-level JSON field the format does not define
	RuleIO         = "unreadable"    // a file found by a directory search that cannot be read
	RuleSkipped    = "skipped-file"  // a model-named link, pipe or device a search does not read: an error, and its module is not checked
	RuleStaleTwin  = "stale-twin"    // a JSON twin that is not what its HCL exports to
	RuleNameCase   = "name-case"     // names in one scope that differ only by case

	RulePublishModuleName = "publish-module-name"
	RulePublishEntities   = "publish-entities"
	RulePublishProperties = "publish-properties"
	RulePublishNameForm   = "publish-name-form"
	RulePublishComponent  = "publish-component-property"
	RulePublishQualified  = "publish-qualified-entity"
)

Rule ids. They are stable: scripts may match on them. Rules whose id starts with "publish-" are enforced only under the publish profile.

View Source
const (
	// MaxNumberLength is the most characters one number literal may have. The
	// HCL library reads a number into 512 bits, about 154 decimal digits, and
	// rounds beyond that, so two different long integers could look equal. With
	// at most 40 characters (and an exponent within MaxNumberExponent) every
	// accepted number is read exactly, and two numbers are equal only when they
	// are the same value. It also bounds the time to read one: the library's
	// time to read `1e10000000`, 10 bytes, was ten seconds.
	MaxNumberLength = 40
	// MaxNumberExponent is the largest exponent, either sign, a number literal
	// may have: with at most MaxNumberLength characters a number is then between
	// 1e-140 and 1e140, a size where a decimal is read, compared and written
	// exactly and in constant time.
	MaxNumberExponent = 100
	// MaxNameLength is the most bytes, as written, a name may have: a block
	// label, an identifier or attribute name, a type, entity, component, enum or
	// kind reference, and an item of `key` or `use` (JSON: every object key and
	// the same strings). The standard states no length. Real names are a few
	// words, and where the databases a model describes limit an identifier it is
	// between 63 and 128 bytes, so 255 leaves room.
	MaxNameLength = 255
	// MaxMessageBytes is the most bytes of one finding's message. Longer ones are
	// cut with a marker (clipText), and so is every piece of the user's text
	// that a message echoes (MaxEchoBytes), so that one finding cannot be large
	// however large the input is.
	MaxMessageBytes = 1024
	// MaxEchoBytes is the most bytes of one piece of the user's text, such as a
	// name or a value, a message repeats: as many as a name may have, so a name
	// is always shown whole.
	MaxEchoBytes = MaxNameLength
)

Limits on single tokens. They are applied to the lexer's tokens before the HCL parser runs (precheck) and to the JSON reader's tokens as it reads, so that no later stage handles a token it was not built for. The standard sets none of them; they are this tool's, stricter than the standard, and each is refused as a `limit` finding that says what was counted and what the limit is.

View Source
const (
	HCLSuffix  = ".modelspec.hcl"
	JSONSuffix = ".modelspec.json"
)

Extensions of the two model file forms.

View Source
const (
	// MaxInputBytes is the largest source, in bytes, that is read. The lexer holds
	// every token of a file in memory at once, and the peak is about 420 to 470
	// MiB for each MB of one-byte tokens, so 1 MiB keeps one file under 500 MiB
	// and about a second of work. Chinook's model is under 8 KB and the largest
	// corpus file 30 KB; a module is a set of files, each under the limit.
	MaxInputBytes = 1 << 20
	// MaxDepth is the deepest nesting of HCL braces, brackets, quoted strings and
	// heredocs, counted from tokens (blocks, lists, objects), and of JSON arrays
	// and objects. Real models nest four or five levels in HCL, six to eight in
	// the JSON form.
	MaxDepth = 64
	// MaxHeredocLines is the most lines one heredoc may have. The lexer makes one
	// piece of each line, and the HCL parser joins the pieces of a template one at
	// a time, copying the text and shifting the rest of the list each time: time
	// that grows with the square of their number. (Pieces that the lexer also
	// starts at each `$` and `%`, and in quoted strings, are removed before the
	// parser sees them: see parserInput.) No real heredoc, a query or a pattern,
	// comes near the limit.
	MaxHeredocLines = 1000
	// MaxSyntaxFindings is the most syntax errors, or non-literal tokens, reported
	// for one HCL file.
	MaxSyntaxFindings = 50
)

Limits and the literal-only rule, applied before a source is parsed, so that hostile or accidental input is a finding and not a crash. The HCL parser is recursive and a stack overflow in Go is fatal and cannot be recovered. Rather than bound the recursive constructs one by one, the HCL source is lexed (the lexer does not recurse) and refused before parsing if it holds any token that a literal value cannot contain; what is left to bound is the nesting of brackets.

View Source
const MaxFindings = 1000

MaxFindings is the most findings one reader, one check and one run keep. A file of four megabytes can hold millions of mistakes, and listing them all would use memory and output without end; past the limit the findings are counted, and one finding says how many were left out. The count and the severity of that finding are the whole of what was dropped, so a run that dropped an error still has one, and its exit code is the same as with no limit.

View Source
const SpecVersion = "1.0-draft"

SpecVersion is the only value the JSON form's "modelspec" field may carry (spec/json-format.md, "Format Identity").

Variables

View Source
var PrimitiveTypes = map[string]bool{
	"string": true, "int": true, "float": true, "bool": true, "decimal": true, "uuid": true,
	"date": true, "time": true, "datetime": true, "document": true, "json": true, "any": true,
}

PrimitiveTypes is the type vocabulary of ModelSpec v0 (spec/core-model.md).

Profiles lists the profile names.

View Source
var ReservedNames = map[string]bool{
	"entities": true, "components": true, "enums": true, "collections": true, "recordsets": true,
}

ReservedNames are the five kind tokens that no concept may be named (ModelSpec decision 0015).

Functions

func Diff

func Diff(a, b *Node) string

Diff returns "" when a and b are the same JSON value with the same key and array order, and otherwise a sentence naming the first difference and its path. Whitespace and source lines are not compared.

func Discover

func Discover(fsys FS, paths []string) ([]Source, []Finding, error)

Discover expands paths into model files. A file is taken as given (it must be a model file); a directory is searched recursively for *.modelspec.hcl and *.modelspec.json, and in a SpecScore layout models directory for any *.hcl. Hidden directories and node_modules are skipped. A search follows no symbolic link, to a directory or to a file: a link that would have been a model file is an error finding (skipped-file) that names it. A file named on the command line may be a link to a regular file, and is read through. A file reached by two names is returned once, under the first name. The result is sorted by path. The findings are errors (skipped-file) for model files a search did not read, and warnings for files it found but cannot read.

func FileKind

func FileKind(mode fs.FileMode) string

FileKind says what a file that is not a regular file is: "a directory", "a named pipe", "a symbolic link", "a character device".

func HasErrors

func HasErrors(fs []Finding) bool

HasErrors reports whether any finding has error severity.

func IsModelFile

func IsModelFile(name string) bool

IsModelFile reports whether a file name is a standalone ModelSpec model file.

func Load

func Load(fsys FS, files []Source, assign []Assignment) ([]*Model, []Finding, error)

Load reads and parses the files and assigns each to a module:

  1. --module assignments (assign) win;
  2. a .hcl file directly inside …/modules/<id>/models/ belongs to module <id>, together with every other file there;
  3. otherwise an HCL file <name>.modelspec.hcl is module <name> by itself, and a JSON file is module.name, or the file name's stem when it has none.

A JSON file is the interchange copy (a Twin) of the module it sits with, and not a second module, when it is assigned to a module that has HCL files, sits in the models directory of a layout module, or is X.modelspec.json beside X.modelspec.hcl (both among the files); an explicit assignment of the JSON file itself to another module wins. A file larger than MaxInputBytes is not read. A file that cannot be read is an error, not a finding.

func Parse

func Parse(file string, src []byte) (*Model, []Finding)

Parse reads one model file by its extension.

func ParseHCL

func ParseHCL(file string, src []byte) (*Model, []Finding)

ParseHCL reads a ModelSpec HCL source. file is used in findings and, without its .modelspec.hcl suffix, as the module short name. The source is parsed with the real HCL parser; a syntax error yields a Broken model.

func ParseJSON

func ParseJSON(file string, src []byte) (*Model, []Finding)

ParseJSON reads the JSON interchange form of a model (spec/json-format.md): the structure the specification defines and the validation it lists (a supported "modelspec" version, module.id and module.version, and well-formed groups). Constraints a single consumer adds are the publish profile's, not this reader's: entities are optional, module.name is optional, and an entity may have no properties.

func ReadSource

func ReadSource(fsys FS, name string) ([]byte, error)

ReadSource reads a file, which every command that reads a model goes through, for every kind of file (a model, the JSON operand of export --check, a --module path). A symbolic link is followed to its target, which must be a regular file: that is checked before the file is opened (opening a named pipe waits for a writer), and again on the opened file, and what is read is at most MaxInputBytes+1, so a file that grows after the checks, or that reports a size of zero and never ends, is still bounded. One over the limit is refused with a *TooLargeError, and one that is not regular with a *NotRegularError.

func SortFindings

func SortFindings(fs []Finding)

SortFindings orders findings by file, line, rule and message, so output is deterministic.

Types

type Assignment

type Assignment struct {
	Module string
	Path   string
}

Assignment places the model files under Path (a file or a directory) in module Module (--module Module=Path). It wins over every other module rule.

type Attr

type Attr struct {
	Name  string
	Value *Node
	Line  int
}

Attr is one attribute of a concept or member: `required = true`.

type Concept

type Concept struct {
	Kind    Kind
	Name    string
	Line    int
	Attrs   []Attr
	Members []Member
}

Concept is an entity, component, enum, collection or recordset.

func (*Concept) Attr

func (c *Concept) Attr(name string) (*Attr, bool)

Attr returns the concept's attribute with the given name.

type FS

type FS interface {
	// Open opens a file for reading. It is read only through ReadSource, which
	// refuses what is not a regular file and bounds how much is read.
	Open(name string) (fs.File, error)
	ReadDir(name string) ([]fs.DirEntry, error)
	// Stat follows symbolic links; Lstat does not.
	Stat(name string) (fs.FileInfo, error)
	Lstat(name string) (fs.FileInfo, error)
	// WriteFile replaces the file name with data, through a temporary file in the same
	// directory (created exclusively, removed on failure) and a rename, so that what is
	// at name is replaced and never followed, and a reader never sees half a file.
	WriteFile(name string, data []byte, perm fs.FileMode) error
	// Abs returns the absolute form of a path, cleaned, with symbolic links left
	// as they are: the path as the file was given or found. The SpecScore layout
	// is read from it.
	Abs(name string) (string, error)
	// SameFile reports whether two FileInfos from Stat describe one file, however
	// it was reached: by a relative or an absolute path, through a symbolic link,
	// under another spelling of the same name on a case-insensitive filesystem.
	SameFile(a, b fs.FileInfo) bool
}

FS is the filesystem the loader reads and writes through. OSFS is the real one; tests use in-memory implementations.

type Field

type Field struct {
	Key   string
	Value *Node
	Line  int
}

Field is one member of an object Node.

type Finding

type Finding struct {
	File     string   `json:"file"`
	Line     int      `json:"line,omitempty"` // 0 when the position is not known
	Rule     string   `json:"rule"`
	Severity Severity `json:"severity"`
	Message  string   `json:"message"`
	// contains filtered or unexported fields
}

Finding is one located problem in one file.

func Check

func Check(models []*Model, opts Options) []Finding

Check applies the semantic rules of the profile in opts to the models and returns the sorted findings.

Models are grouped into modules by Model.Group (Load sets it from the module rules): concept names are unique per module across all its files, and a reference resolves against the whole module. A module-qualified reference (decision 0014) names another module and resolves among the models given, by Model.Name. A Twin (the JSON copy of an HCL module) is checked as a module of its own but is not a second source for the name. Broken models are skipped, and references into their modules are not reported.

func (Finding) String

func (f Finding) String() string

String formats a finding as "file:line: severity: message [rule]".

type Form

type Form string

Form is the serialisation a model was read from.

const (
	FormHCL  Form = "hcl"
	FormJSON Form = "json"
)

type Kind

type Kind string

Kind is a concept kind.

const (
	KindEntity     Kind = "entity"
	KindComponent  Kind = "component"
	KindEnum       Kind = "enum"
	KindCollection Kind = "collection"
	KindRecordset  Kind = "recordset"
)

type LintOptions

type LintOptions struct {
	Profile Profile
	// Modules assigns files to modules explicitly. The assigned files are linted
	// even when no path names them.
	Modules []Assignment
}

LintOptions configures Lint.

type Member

type Member struct {
	Name  string
	Line  int
	Attrs []Attr
}

Member is a property (entity), field (component or collection) or column (recordset). Column names may repeat; the others are unique.

func (*Member) Attr

func (m *Member) Attr(name string) (*Attr, bool)

Attr returns the member's attribute with the given name.

type Model

type Model struct {
	File string // as given or found, used in findings
	Form Form
	// Name is the module short name used to resolve module-qualified
	// references. ParseHCL sets the file name without .modelspec.hcl; ParseJSON
	// sets module.name, or that same file-name stem when the JSON has none; Load
	// then applies the module rules (SpecScore layout, --module).
	Name string
	// Group identifies the module the file belongs to: files with the same Group
	// are one module. ParseHCL and ParseJSON set it to File (one file, one
	// module); Load groups the files of a module.
	Group string
	// Twin is set by Load on a JSON file that sits beside the HCL file of the same
	// name: it is the interchange copy of that module, not a second module, and
	// references into the module resolve to the HCL.
	Twin bool
	// TwinOf is the HCL model whose export a Twin is the copy of, when that is one
	// HCL file; Check compares the two.
	TwinOf *Model
	// Root is the parsed document of a JSON file (nil for HCL).
	Root *Node
	// Module is the identity from a JSON file; nil for HCL.
	Module *ModuleIdentity
	// ModuleLine is the line of the JSON module object (0 for HCL).
	ModuleLine int
	// Concepts are in source order.
	Concepts []*Concept
	// Unmapped lists HCL constructs the JSON form does not define.
	Unmapped []Unmapped
	// Projections and Migrations are carried through unchanged from a JSON
	// file; nil otherwise.
	Projections *Node
	Migrations  *Node
	// Broken is set when the source could not be read into a model; Check skips
	// it and does not report references into its module.
	Broken bool
	// Incomplete is set when a file of the model's module was found by a search
	// and not read (a link, a pipe, a device: rule skipped-file). Check does not
	// check an incomplete module: a partial load would report what is missing as
	// unresolved references.
	Incomplete bool
}

Model is one parsed ModelSpec file. A module is one or more Models: the HCL files of a module share a Group (see Load); a JSON file is a whole module.

func (*Model) ExportDrift

func (m *Model) ExportDrift(committed []byte, id ModuleIdentity) string

ExportDrift compares what the model exports to with committed JSON. It returns "" when they are the same document with the same key and array order (whitespace does not count), and otherwise one sentence saying what is wrong.

Module identity is not in the HCL. When id is the zero value the committed JSON's own module object supplies it, so the comparison covers everything except the identity; pass id to compare that too.

func (*Model) HasConcept

func (m *Model) HasConcept(kind Kind, name string) bool

HasConcept reports whether the model declares a concept of the kind with the name.

func (*Model) JSON

func (m *Model) JSON(id ModuleIdentity) (*Node, error)

JSON builds the JSON interchange form of the model: the "modelspec" field, "module", then the groups components, enums, entities, collections, recordsets, projections and migrations, in that order. Concepts keep source order. An entity's or collection's attributes come first, then its members; recordset columns are an ordered array with "name" first (decision 0007).

The mapping is exactly the one spec/json-format.md defines. Two things it leaves open are refused rather than invented: HCL `index`, `projection` and `migration` blocks (no document says how they map to JSON), and module identity for HCL, which the grammar has no place for, so the caller must supply id and version (the format requires those two; name is written only when given, or when the model refers to its own module by name). A model read from JSON already has its identity, which id overrides when non-zero.

type ModuleIdentity

type ModuleIdentity struct {
	ID      string
	Name    string
	Version string
}

ModuleIdentity is the identity the JSON form carries and HCL does not.

type Node

type Node struct {
	Type   NodeType
	Str    string  // NodeString: the value; NodeNumber: the number's text
	Bool   bool    // NodeBool
	Fields []Field // NodeObject, in order
	Items  []*Node // NodeArray
	Line   int     // 1-based source line; 0 when built in memory
	// Dups, set on the root returned by ParseNode only, lists every object key
	// that appears twice in the same object, anywhere in the document, at the
	// line of the repeat. Most JSON readers keep the last of the two, a few the
	// first, so a repeat is never harmless.
	Dups []Field
}

Node is a JSON value that keeps object key order and source lines. encoding/json's maps cannot do that, and ModelSpec's own tooling compares a model with its registered copy with the order of keys and arrays significant (the Directory's check says "the order of entities, properties and other keys counts"), so export --check does too.

func ParseNode

func ParseNode(src []byte) (*Node, error)

ParseNode reads one JSON document into a Node, keeping key order and lines. Anything after the document is an error.

func (*Node) Encode

func (n *Node) Encode() []byte

Encode writes the node as indented JSON (two spaces) ending in a newline, the form JSON.stringify(x, null, 2) produces.

func (*Node) Get

func (n *Node) Get(key string) (*Node, bool)

Get returns the first field with the given key.

type NodeType

type NodeType int

NodeType is the JSON type of a Node.

const (
	NodeNull NodeType = iota
	NodeBool
	NodeNumber
	NodeString
	NodeArray
	NodeObject
)

type NotRegularError

type NotRegularError struct {
	File string
	What string // what it is: "a named pipe", "a symbolic link to a character device"
}

NotRegularError is the error of a file that is not a regular file: a device, a named pipe, a socket or a directory, which has no size and may never end.

func (*NotRegularError) Error

func (e *NotRegularError) Error() string

type OSFS

type OSFS struct {
	// contains filtered or unexported fields
}

OSFS is the operating system's filesystem. The zero value is ready to use.

func (OSFS) Abs

func (o OSFS) Abs(name string) (string, error)

func (OSFS) Lstat

func (OSFS) Lstat(name string) (fs.FileInfo, error)

func (OSFS) Open

func (OSFS) Open(name string) (fs.File, error)

Open opens without waiting: opening a named pipe for reading otherwise blocks until a writer comes, and ReadSource must be able to refuse it.

func (OSFS) ReadDir

func (OSFS) ReadDir(name string) ([]fs.DirEntry, error)

func (OSFS) SameFile

func (OSFS) SameFile(a, b fs.FileInfo) bool

func (OSFS) Stat

func (OSFS) Stat(name string) (fs.FileInfo, error)

func (OSFS) WriteFile

func (o OSFS) WriteFile(name string, data []byte, perm fs.FileMode) error

WriteFile writes data to a temporary file in the directory of name, created with O_EXCL (a name that exists, a link planted at it included, is an error and is left alone), and renames it over name. A rename replaces a link at name and does not follow it, so a link put there after a check cannot redirect the write. A file that is replaced keeps its permission bits (as far as the umask allows); the temporary file is removed when anything fails.

type Options

type Options struct {
	Profile Profile // "" means ProfileDefault
	// Unread names the modules that have a file which was found and not read (Lint's
	// skipped files). A module of that name that is not among the models is known and
	// cannot be checked, so a reference into it is not reported as an unknown module.
	Unread []string
}

Options configures Check.

type Profile

type Profile string

Profile selects the rules Check applies.

The default profile implements the standard as written in the ModelSpec repository (spec/core-model.md, spec/hcl-authoring.md, spec/json-format.md and the decisions) and nothing else: a model the standard allows is never refused because some consumer's reader is narrower. The publish profile adds what a model needs to be listed in public catalogues, which is what the Directory's JSON reader requires on top of the standard.

const (
	ProfileDefault Profile = "default"
	ProfilePublish Profile = "publish"
)

func ParseProfile

func ParseProfile(s string) (Profile, error)

ParseProfile converts a flag value to a Profile.

type Result

type Result struct {
	Findings []Finding
	Files    int
	Models   []*Model
	// Skipped is every model-named file a search did not read, whole: the findings
	// are capped, this is not.
	Skipped []SkippedFile
	// Notes say what Lint did beyond the paths given: a module is the unit of
	// checking, so the files of a module that were not given were checked too.
	Notes []string
}

Result is the outcome of Lint.

func Lint

func Lint(fsys FS, paths []string, opts LintOptions) (Result, error)

Lint discovers, loads and checks the paths together with the files assigned by opts.Modules: the parse findings and the semantic findings, sorted. A module is always checked whole: when a file given belongs to a module with more files on disk (a SpecScore layout models directory, or the HCL and JSON of one module), the others are loaded too, and their findings are reported with their paths. With no paths and no assignments it is an error, as is finding no model file.

type Severity

type Severity string

Severity ranks a finding. Only SeverityError makes a model invalid.

const (
	SeverityError   Severity = "error"
	SeverityWarning Severity = "warning"
)

type SkippedFile

type SkippedFile struct {
	Finding  Finding  // the skipped-file finding for it
	Module   string   // the --module name it was found under, "" when it was not found through one
	Affected []*Model // the models of its module, which are marked Incomplete
	// contains filtered or unexported fields
}

SkippedFile is a model-named file that a search met and did not read (rule skipped-file). Unlike the finding, which the findings cap can cut from a long list, it is kept whole: Lint returns every one, with the models that its absence makes incomplete.

type Source

type Source struct {
	Path string
	Abs  string
}

Source is a model file to read: Path as given or found (used in findings) and its absolute form, symbolic links not resolved (used for the module layout).

type TooLargeError

type TooLargeError struct {
	File    string
	Size    int64
	Partial bool
}

TooLargeError is the error of a file over MaxInputBytes, which is not read (further than the limit). Partial is set when the size is not known: the file grew after it was checked, and Size is how much was read before reading stopped.

func (*TooLargeError) Error

func (e *TooLargeError) Error() string

type Unmapped

type Unmapped struct {
	What string // for example `entity "User" index "by_email"`
	Line int
}

Unmapped is an HCL construct that the reader parses for syntax but that has no defined place in the JSON form, so it cannot be exported.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL