meaning

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: Apache-2.0 Imports: 23 Imported by: 1

Documentation

Overview

Package meaning reads and checks meaning files (the *.meaning.yaml files of github.com/meaninggraph/core and of the repositories that use it): it parses them, validates them against the embedded schema, resolves references and extends across graphs, checks bindings against ModelSpec models, and returns a typed Graph and a sorted list of Findings. It has no command framework, never exits the process, never touches the network, and reads files only through FS.

The YAML reader

ParseYAML is the strict YAML reader the checker uses, and it is exported for other tools that need to read a meaning file the way the checker does: the same bytes give the same data, or an error, in both. It reads a defined subset of YAML 1.2 (see the comment on the YAML subset in yaml.go and the README), never half-reads a file outside it, and on a file inside it returns the data that the reference checker's parser (the yaml package of npm, as meaninggraph/core calls it) reads from the same text. That is tested, with recorded values of the reference parser for thousands of documents and a mutation run that compares the two parsers on every mutated file, not proved.

Its limits are enforced by ParseYAML itself, not by its caller: at most MaxFileBytes bytes of text, at most MaxYAMLDepth levels of nesting (block and flow together), keys written in at most 1024 bytes from their first character to their colon (quotes, escapes as written and blanks before the colon count), one document, UTF-8 only, string keys only, no repeated key, no anchor, alias, merge key or tag, no keep chomping or indentation indicator on a block scalar, and tabs only where they are text (an error that says where: Line, and for a tab Column). The file is read in one pass over lines whose work grows with the size of the input, and the quadratic shapes (a long plain scalar, a long run of comment lines) are linear. A refused file's error is a *SyntaxError with a stable Rule; it is a refusal of the subset and not always a YAML error, since some files are refused that the reference parser reads (the README lists them).

ParseYAML returns an error, nil for a file it reads. A refused file's error is a *SyntaxError, reached with errors.As:

node, err := meaning.ParseYAML(data)
var refused *meaning.SyntaxError
if errors.As(err, &refused) {
	// refused.Rule is the stable rule id ("yaml-tab", "yaml-key", ...),
	// refused.Line the 1-based line (0 when not known) and refused.Column the
	// 1-based column in characters (set for a refused tab, else 0); the
	// message already begins with the column of a tab.
}

The returned Node holds scalars as the YAML 1.2 core schema reads them (Node.Value gives nil, string, bool, json.Number, []any and map[string]any), and every node has the line it starts on.

Index

Constants

View Source
const (
	RuleSchema           = "schema"
	RuleNoFiles          = "no-meaning-files"
	RuleSubdirectoryFile = "subdirectory-meaning-file"
	RuleDuplicateConcept = "duplicate-concept"
	RuleDuplicateSource  = "duplicate-source"
	RuleDuplicateValue   = "duplicate-value"
	RuleAmbiguousWord    = "ambiguous-word"
	RuleReferenceSyntax  = "reference-syntax"
	RuleUnknownConcept   = "unknown-concept"
	RuleUnresolved       = "unresolved-graph"
	RulePinMismatch      = "pin-mismatch"
	RuleSelfPin          = "self-pin"
	RuleTargetKind       = "target-kind"
	RuleExtendsKind      = "extends-kind"
	RuleExtendsCycle     = "extends-cycle"
	RuleNarrowing        = "narrowing"
	RuleUnit             = "unit"
	RuleMeasureInput     = "measure-input"
	RuleMeasureDimension = "measure-dimension"
	RuleRatio            = "ratio-aggregation"
	RuleSource           = "undeclared-source"
	RuleEntityBindings   = "entity-bindings"
	RuleModels           = "models"
	RuleEarlierSpelling  = "deprecated-spelling"
	RuleBindingModel     = "binding-model"
	RuleBindingRole      = "binding-role"
	RuleUniversal        = "universal"
	RuleSymlink          = "symlink-meaning-file"
	RuleUnreadableDir    = "unreadable-directory"
	RuleFileTooLarge     = "file-too-large"
)

Rule identifiers of findings. They are stable: scripts may match on them.

View Source
const (
	// MaxFileBytes bounds the size of a meaning file and of a model file.
	MaxFileBytes = 8 << 20
	// MaxYAMLDepth bounds the nesting of collections in a meaning file.
	MaxYAMLDepth = 64
)

The YAML subset.

A meaning file is read as a plainly defined subset of YAML 1.2, in which it agrees with the reference checker's YAML library (the `yaml` 2.x package of npm) on every file it accepts, and which it enforces before anything is interpreted. Anything outside the subset is refused with a finding of its own that says what to write instead; it is never interpreted.

The subset:

  • UTF-8 text, optionally starting with a byte order mark (which may not be followed by an indented first line or a block sequence: the reference parser counts it as a column). No NUL, no other control character, no U+0085, U+2028, U+2029 or U+FEFF inside the text. Lines end in LF or CRLF (no lone CR). Tabs only where they cannot change what is read: see "Tabs" below.
  • One document, optionally started by a "---" line. No directives (%YAML, %TAG) and no other document markers.
  • Block mappings and sequences (a sequence may sit at the indent of its key), flow mappings and sequences (which may span lines, without comments), and scalars: plain (which may continue on following, more indented lines), single-quoted and double-quoted (on one line; the JSON escapes, \/ and \u surrogate pairs included), and literal and folded block scalars (| and >, with an optional - but no indentation indicator and no keep chomping, +: the reference parser and other readers disagree about the last line break of a file that ends in the scalar). A blank line inside or at the end of a block scalar may not hold more spaces than the text is indented by.
  • No anchors, aliases, merge keys (<<) or tags of any kind, and no explicit keys (?).
  • String keys only: a key that YAML reads as null, a boolean or a number must be quoted. No repeated key in one mapping.
  • Numbers are decimal: an integer within the range a double holds exactly (up to 2^53), or a finite float. No 0x or 0o numbers, no .inf or .nan.
  • At most MaxFileBytes of text and MaxYAMLDepth levels of nesting.
View Source
const (
	RuleYAML            = "yaml"
	RuleYAMLEncoding    = "yaml-encoding"
	RuleYAMLCharacter   = "yaml-character"
	RuleYAMLLineEnding  = "yaml-line-ending"
	RuleYAMLTab         = "yaml-tab"
	RuleYAMLDirective   = "yaml-directive"
	RuleYAMLDocuments   = "yaml-documents"
	RuleYAMLAnchor      = "yaml-anchor"
	RuleYAMLTag         = "yaml-tag"
	RuleYAMLKey         = "yaml-key"
	RuleYAMLDuplicate   = "yaml-duplicate-key"
	RuleYAMLNumber      = "yaml-number"
	RuleYAMLEscape      = "yaml-escape"
	RuleYAMLUnsupported = "yaml-unsupported"
	RuleYAMLLimit       = "yaml-limit"
)

Rule identifiers of the YAML findings.

View Source
const FileSuffix = ".meaning.yaml"

FileSuffix ends the name of every meaning file.

View Source
const MaxAmbiguousWords = 1000

MaxAmbiguousWords bounds how many words that name two concepts the universal profile lists: when graphs of thousands of concepts share one word, the pairs would number in the millions. The first MaxAmbiguousWords are listed, and a finding of the same rule says that the others are not.

Variables

View Source
var ErrTooLarge = errors.New("the file is larger than what is read of it")

ErrTooLarge is the error of FS.ReadFileMax for a file of more than the bytes asked for.

Functions

func CheckoutCommit

func CheckoutCommit(fsys FS, dir string) (string, error)

CheckoutCommit returns the commit a git checkout in dir is at: the commit id that HEAD holds, or that the branch HEAD names points at. It reads plain files only (.git/HEAD, the branch's ref file or .git/packed-refs); it starts no process and uses no network, and it follows a linked worktree's .git file to its git directory. HEAD, the ref files and packed-refs are read only when they are regular files that are not symbolic links, whose names are spelled exactly as asked (a case-insensitive file system does not make MAIN the branch main), and only up to a documented size (1 KiB for a ref, 4 KiB for a .git pointer file or a commondir file, 8 MiB for packed-refs); otherwise the commit cannot be read, and the error says so. A .git that is a link, and a pointer file or commondir that is a pipe or a device, is not opened.

It tells which commit the checkout was made at, and nothing more: it does not look at the working tree, so files that were edited, added or deleted since the checkout are not noticed, and it does not check the objects of the repository. An error says why the commit cannot be read, for example that dir is not a git checkout.

func HasErrors

func HasErrors(findings []Finding) bool

HasErrors reports whether any finding is an error.

func PrototypeNames

func PrototypeNames() []string

PrototypeNames lists the names that the reference checker refuses in a model (see the findings of HCLReader), sorted.

func SchemaCommit

func SchemaCommit() string

SchemaCommit is the commit of github.com/meaninggraph/core the embedded schema was taken from.

func SchemaJSON

func SchemaJSON() []byte

SchemaJSON returns meaning.schema.json of github.com/meaninggraph/core, the schema of format meaning/draft-1, as it is embedded. SchemaSource says which commit it was taken from; scripts/check-schema-drift.sh compares both with a checkout of core.

func SchemaSource

func SchemaSource() string

SchemaSource records where SchemaJSON comes from: repository, commit, path and sha256 of the file, one "key: value" per line.

func SortFindings

func SortFindings(findings []Finding)

SortFindings orders findings by file, line, rule, severity and message, so that the same files always give the same list.

Types

type Binding

type Binding struct {
	Model, Property, Role, Match, Note string
	Line                               int
}

Binding binds a concept to an entity or property of a model.

type Checker

type Checker struct {
	// Schema validates each file; the embedded schema when nil.
	Schema Validator
	// Resolve reads other graphs; when nil none is available.
	Resolve Resolver
	// Models reads the models bindings name; HCLReader when nil.
	Models ModelReader
	// Profile selects extra rules.
	Profile Profile
	// Noticed holds the model files already reported for an earlier ModelSpec
	// spelling, by path as written. A caller that checks several graphs in one
	// run shares one map between their Checkers, so that a model file listed by
	// more than one graph is reported once; when nil, the check keeps its own.
	Noticed map[string]bool
}

Checker checks a Graph. The zero value checks against the embedded schema, with no other graph available.

func (Checker) Check

func (c Checker) Check(g *Graph) []Finding

Check validates every file of g against the schema and checks what the schema cannot: references resolve, extends joins compatible kinds without a cycle, values-of and units-of name entities, measures and ratios are consistent, and ids and words are unique. The findings are sorted.

type Concept

type Concept struct {
	ID, Kind                       string
	Of, Extends, ValuesOf, UnitsOf string
	Unit, Source                   string
	Labels                         map[string]string
	Synonyms                       map[string][]string
	Values                         []Value
	Measure                        *Measure
	Bindings                       []Binding
	Line                           int
	// contains filtered or unexported fields
}

Concept is a concept of a meaning file, decoded leniently: a field of the wrong type is left empty, because the schema check reports it.

type ConceptRef

type ConceptRef struct {
	// Repo is {host}/{org}/{repo}, empty for a bare id.
	Repo string
	ID   string
	// Pin is the ?ref= value, empty when there is none.
	Pin string
}

ConceptRef is a parsed reference to a concept: a bare id (same graph), or meaning://{host}/{org}/{repo}/{id} with an optional ?ref= pin.

func ParseConceptRef

func ParseConceptRef(ref string) (ConceptRef, bool)

ParseConceptRef parses a concept reference, as the format defines it.

type EarlierSpelling added in v0.3.0

type EarlierSpelling struct {
	// Count is the number of earlier spellings the file uses.
	Count int
	// Line is the 1-based line of the first of them.
	Line int
}

EarlierSpelling counts the uses of the earlier ModelSpec spellings in a model file. The zero value says the file uses none.

type Entity

type Entity struct {
	// Key is the names of the properties that form the entity's key.
	Key        []string
	Properties map[string]Property
}

Entity is an entity of a ModelSpec module.

type Entry

type Entry struct {
	Concept *Concept
	File    *File
}

Entry is a concept with the file it is declared in.

type FS

type FS interface {
	Stat(name string) (fs.FileInfo, error)
	ReadFile(name string) ([]byte, error)
	// ReadFileMax reads a file of at most max bytes without reading more of a
	// larger one, which is the error ErrTooLarge.
	ReadFileMax(name string, max int64) ([]byte, error)
	ReadDir(name string) ([]fs.DirEntry, error)
}

FS is the file system the library reads meaning files, and the models they name, through. Paths are the host's own (filepath) paths. The library never writes, never touches the network and never starts a process.

type File

type File struct {
	// Path is the path as it was given.
	Path string
	// Root is the parsed document; nil when it could not be parsed.
	Root *Node
	// ParseErr is why Root is nil.
	ParseErr *SyntaxError

	ID, Name, License string
	// Models maps a module short name to the path of its source, relative to
	// the file.
	Models   map[string]string
	Sources  []Source
	Concepts []*Concept
}

File is one meaning file.

type Finding

type Finding struct {
	// File is the path of the file the finding is about, as it was given.
	File string `json:"file"`
	// Line is the 1-based line, or 0 when it is not known.
	Line int `json:"line,omitempty"`
	// Rule is a stable identifier of the rule that was broken.
	Rule     string   `json:"rule"`
	Severity Severity `json:"severity"`
	Message  string   `json:"message"`
}

Finding is one thing a check found.

type Graph

type Graph struct {
	// Address is the graph's own address, {host}/{org}/{repo} (for example
	// github.com/meaninggraph/core), or empty when it is not known. Set it,
	// before the graph is checked or handed to GraphResolver, for every graph
	// whose files refer to the graph by its address: a meaning:// reference to
	// the address from inside the graph is a reference to itself, and without
	// the address it is a reference to another graph. Neither LoadDir nor
	// LoadFiles sets it.
	Address string
	// Dir is the directory the files were read from by LoadDir, else empty.
	Dir   string
	Files []*File

	// Concepts maps a concept id to its first declaration.
	Concepts map[string]*Entry
	// contains filtered or unexported fields
}

Graph is a set of meaning files read as one repository of concepts, as the format defines it: a file is only packaging, and a concept id is unique across all of them.

func LoadDir

func LoadDir(fsys FS, dir string) (*Graph, error)

LoadDir reads the *.meaning.yaml files directly in dir as one graph. A file in a subdirectory is not part of the graph. A file that is not valid YAML is kept with its ParseErr; only a directory that cannot be read is an error.

func LoadFiles

func LoadFiles(fsys FS, paths []string) (*Graph, error)

LoadFiles reads the given files as one graph, in path order.

type HCLReader

type HCLReader struct{}

HCLReader reads a ModelSpec module from its HCL source. It is a port of the parser of the Node reference checker (scripts/lib/modelspec.mjs of github.com/meaninggraph/core), and accepts exactly what ModelSpec v0 HCL allows there: named blocks, and attributes whose values are strings, numbers, booleans or lists of those. Expressions, interpolation and map-style values are refused rather than guessed.

It also reads the current spelling of the words ModelSpec renamed (record, field and record = for entity, property and entity =) as the same things.

It is a stand-in. Once the ModelSpec library (github.com/modelspec-org/cli, package pkg/modelspec) is released, a ModelReader built on it replaces this one; see ModelReader.

func (HCLReader) ReadModel

func (HCLReader) ReadModel(fsys FS, path string) (*Model, error)

ReadModel reads and parses the file at path.

type Kind

type Kind int

Kind is the kind of value a Node holds.

const (
	Null Kind = iota
	String
	Number
	Bool
	Map
	Seq
)

The kinds of YAML value a meaning file may hold.

type Measure

type Measure struct {
	Formula, Aggregation string
	Inputs, Dimensions   []string
}

Measure is the measure part of a concept of kind measure.

type Model

type Model struct {
	Entities map[string]*Entity
	// Earlier says which earlier spellings the model file uses (entity, property
	// and entity = for record, field and record =). They are read as the words
	// they were replaced by; this is how a check can report them.
	Earlier EarlierSpelling
}

Model is what a meaning file's bindings need to know about a ModelSpec module: its entities, their keys and their properties.

type ModelReader

type ModelReader interface {
	ReadModel(fsys FS, path string) (*Model, error)
}

ModelReader reads the model of a module from the file a meaning file's `models` names. The single implementation today is HCLReader, which reads the subset of ModelSpec HCL that the Node reference checker reads.

ModelReader is the seam for the ModelSpec library: once github.com/modelspec-org/cli (package pkg/modelspec) has a release, a ModelReader that calls it replaces HCLReader and nothing else in this package changes.

type ModelRef

type ModelRef struct {
	// Repo is {host}/{org}/{repo} for a model of another repository, empty
	// for one of the same repository.
	Repo, Module, Name, Pin string
}

ModelRef is a parsed modelspec:// reference to an entity.

func ParseModelRef

func ParseModelRef(ref string) (ModelRef, bool)

ParseModelRef parses modelspec:///{module}.{Entity} (same repository) or modelspec://{host}/{org}/{repo}/{module}.{Entity}.

type Node

type Node struct {
	Kind Kind
	Line int
	// Text is the value of a scalar: the string itself, or the canonical
	// text of a number or a boolean.
	Text string
	// Keys are the keys of a Map in document order.
	Keys []string
	// Fields are the values of a Map by key.
	Fields map[string]*Node
	// Items are the elements of a Seq.
	Items []*Node
}

Node is a parsed YAML value with the line it starts on. Scalars are resolved the way the Node reference checker's YAML library resolves them (YAML 1.2 core schema): only true/false are booleans, and yes/no/on/off and dates are strings. See ParseYAML for the subset of YAML that is read.

func ParseYAML

func ParseYAML(data []byte) (*Node, error)

ParseYAML reads one document of the subset described above. An empty file is a Null node. It is exported for other tools: it enforces the size, depth and key limits and every rule of the subset itself (see the package documentation). A refused file is an error that is a *SyntaxError, which errors.As reaches for its Line, Column and Rule; the result for a file that is read is a nil error.

func (*Node) At

func (n *Node) At(path []string) int

At returns the line of the value at a JSON pointer's tokens, or of the deepest node on the way to it that exists.

func (*Node) Field

func (n *Node) Field(key string) *Node

Field returns the value of a key of a Map, or nil.

func (*Node) Str

func (n *Node) Str() (string, bool)

Str returns the value of a String node.

func (*Node) Value

func (n *Node) Value() any

Value converts the node to the values a JSON Schema validator reads: nil, string, bool, json.Number, []any and map[string]any.

type OSFS

type OSFS struct{}

OSFS reads the host's file system.

func (OSFS) ReadDir

func (OSFS) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir lists a directory of the host's file system, sorted by name.

func (OSFS) ReadFile

func (OSFS) ReadFile(name string) ([]byte, error)

ReadFile reads a file of the host's file system.

func (OSFS) ReadFileMax

func (OSFS) ReadFileMax(name string, max int64) ([]byte, error)

ReadFileMax reads a file of the host's file system, at most max bytes of it.

func (OSFS) Stat

func (OSFS) Stat(name string) (fs.FileInfo, error)

Stat describes a file or directory of the host's file system.

type Profile

type Profile string

Profile selects extra rules on top of the format's.

const (
	// ProfileDefault applies the rules of the format only.
	ProfileDefault Profile = ""
	// ProfileUniversal also applies the rules of a repository of universal
	// concepts, such as github.com/meaninggraph/core: a LICENSE file, CC0-1.0
	// on every file, no models and no bindings, no meaning file outside the
	// root, and one word names one concept.
	ProfileUniversal Profile = "universal"
)

type Property

type Property struct {
	// Type is the ModelSpec type ("string", "int", ...), empty for a
	// reference to another entity.
	Type string
	// Reference says the property is a reference to an entity, in the sense of
	// the reference checker: its `entity` attribute is set.
	Reference bool
	// Entity is the name of the entity a reference property points at. It is
	// empty when the property is no reference, or when the attribute is not a
	// string (and so names no entity).
	Entity string
}

Property is a property of an entity.

func (Property) IsReference

func (p Property) IsReference() bool

IsReference reports whether the property is a reference to an entity.

type RefTarget

type RefTarget struct {
	// Ref is the full name of the ref (refs/heads/main, refs/tags/v1, ...).
	Ref string
	// ID is what the ref file or packed-refs holds, or the commit an annotated
	// tag in packed-refs peels to.
	ID string
	// MaybeTagObject says that ID may be that of a tag object, not of the commit:
	// a loose refs/tags file holds the id of the tag object when the tag is
	// annotated, and so does a packed one when packed-refs does not say that it is
	// fully peeled and gives no peeled line for it. Telling the two apart means
	// reading the object, which this does not do.
	MaybeTagObject bool
}

RefTarget is a ref of a git checkout that a pin names, and what it points at.

func RefTargets

func RefTargets(fsys FS, dir, name string) ([]RefTarget, error)

RefTargets finds the refs that the branch or tag name names in the checkout in dir: a branch, a tag, or the branch of the remote called origin, each as a ref file or in packed-refs. Like CheckoutCommit it reads plain files, bounded, not through symbolic links, and with the exact spelling of the name. A name that is no valid branch or tag name (empty, a trailing slash, an empty, "." or ".." segment, a character outside the grammar) is an error, and so is a ref that cannot be read that way.

type Resolver

type Resolver func(repo, pin string) (*Graph, error)

Resolver gives the graph a meaning:// reference names: repo is {host}/{org}/{repo} and pin is the ?ref= value (empty when there is none). An error says why the graph is not available. Nothing in this package fetches anything: a Resolver reads what the caller has at hand.

func GraphResolver

func GraphResolver(graphs map[string]*Graph) Resolver

GraphResolver resolves references from graphs the caller already has, keyed by address. It follows the reference checker's rules for a repository it reads from git: a reference to another repository must carry a ?ref= pin, and a repository that cannot be read is reported as such. A supplied graph must itself be valid: a file that is not YAML in the subset, or that breaks the schema, makes the graph unreadable. The pin cannot be verified against a directory here; see VerifyPin.

A graph's own address (Graph.Address) must be set for references to meaning://<its address>/... made from inside that graph to resolve to itself; GraphResolver does not set it. The resolver is not safe for concurrent use.

type Severity

type Severity string

Severity says how serious a finding is. Only errors make a check fail.

const (
	// Error means the files are not valid meaning files; the check fails.
	Error Severity = "error"
	// Warning means something is likely wrong but the Node reference checker
	// accepts it; the check does not fail.
	Warning Severity = "warning"
	// Info states a fact the reader should know, such as which directory
	// stood in for another graph.
	Info Severity = "info"
)

The severities.

type Source

type Source struct {
	ID   string
	Line int
}

Source is an entry of a file's sources.

type SyntaxError

type SyntaxError struct {
	Line    int
	Column  int
	Rule    string
	Message string
}

SyntaxError is a problem with the YAML of a file: Rule says which rule of the subset (or of YAML itself) it breaks, Line where, 0 when not known. A tab that is refused (RuleYAMLTab) also has its Column, counted in characters from 1, a tab being one character, and the message starts with it; for any other problem Column is 0.

func (*SyntaxError) Error

func (e *SyntaxError) Error() string

type Validator

type Validator interface {
	Validate(v any) error
}

Validator checks a value against the meaning-file schema. *jsonschema.Schema is one; tests supply others.

func DefaultSchema

func DefaultSchema() Validator

DefaultSchema returns the schema embedded in the binary, compiled once.

type Value

type Value struct {
	ID      string
	Labels  map[string]string
	Aliases map[string][]string
	Codes   map[string]string
}

Value is a known value of a concept.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL