salmodule

package
v0.1.17 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: Apache-2.0 Imports: 43 Imported by: 0

Documentation

Index

Constants

View Source
const (
	OntologyFile   = "ontology.jsonld"
	DockerfileName = "Dockerfile"
)

OntologyFile and DockerfileName are the two files a SAL module repository holds in its root: the document its container prints in response to `salmodule ontology`, and the Dockerfile sal builds that container from.

View Source
const (
	// Namespace is the IRI of the SAL Module ontology itself.
	Namespace = "https://w3id.org/sal/salmodule#"
	// ProtocolScheme is the URI scheme used to reference a SAL module from RDF.
	ProtocolScheme = "salmodule"

	BaseCommand     = "salmodule"
	OntologyCommand = "ontology"
	RunCommand      = "run"

	// TaskInstanceEnvVar is the environment variable the task instance is
	// passed to a module's run command in, the one the salmodule:TaskInstance
	// individual names. Every module reads it; a module cannot rename it.
	TaskInstanceEnvVar = "SALMODULE_TASK_INSTANCE"

	// IcebergTableProperty is the Iceberg table property a build records the
	// JSON list of every SAL module it downloaded under.
	IcebergTableProperty = "sal.salmodules"
)

Constants describing the SAL Module specification. The command line conventions mirror the salmodule:SALModuleCommand, salmodule:OntologyCommand, salmodule:RunCommand, and salmodule:TaskInstance individuals declared in specs/salmodule.ttl, whose rdfs:label is the word on the command line or the name of the variable.

View Source
const EditorSchemaURL = "./.vscode/" + specs.SalModuleOntologySchemaFile

EditorSchemaURL is where .vscode/settings.json points VS Code for the schema, relative to the workspace root the settings file sits under.

View Source
const PlaceholderNamespace = "salmodule://github.com/OWNER/REPO/"

PlaceholderNamespace stands in for a module's salmodule:// namespace when the repository has no git remote to derive it from. The module checks only need a base for the ontology's relative terms, so any will do.

View Source
const ProvFile = "prov.jsonld"

ProvFile is the name of the JSON-LD document in the blob store that records every file and directory a SAL module task has ever copied there: its digest, the name the task gave it, the module that produced it, when it was written, and where it sits under the blob store. It is an append-only log. A run adds the copies it kept and never removes what an earlier run left, so the blob store accumulates every version a task has produced until the user clears it. The shape is the one .sal/config.jsonld records a pinned vocabulary in.

View Source
const TaskClass = Namespace + "Task"

TaskClass is the class a module declares its own task classes as subclasses of. It is the only one of the ontology's task classes meant to be subclassed.

Variables

This section is empty.

Functions

func CheckModule

func CheckModule(dir string, namespace string) []error

CheckModule checks the SAL module in dir without building or running anything: that it has a Dockerfile and an ontology.jsonld, that the ontology has the structure the SAL Module specification requires, that every term it refers to in its own namespace is declared in it, and that every SHACL shape it declares is a well-formed shape sal can enforce. namespace is the module's own salmodule:// namespace, which its relative terms resolve against. It returns one ModuleError per problem, each naming the file and the line, sorted by line.

func CheckOntologyDocument

func CheckOntologyDocument(path string, content []byte, namespace string) []error

CheckOntologyDocument runs every check CheckModule makes on the ontology document itself, reporting against path. The JSON Schema in specs/ is what decides the structure, so the rules an editor applies while the document is written are the ones sal applies here; the checks after it need the RDF the document expands to, which the schema cannot see. `sal init --salmodule --ui` runs it over the document its form assembles before writing it, so the form refuses exactly what `sal validate --salmodule` would.

func FetchOntologyDocument

func FetchOntologyDocument(ctx context.Context, iri string) (document []byte, mediaType string, commitHash string, err error)

FetchOntologyDocument dereferences a salmodule:// IRI to the module's ontology document so that RDF validation can resolve the module's terms, and to the git commit hash of the module repository it was built from, which is what a salmodule:// vocabulary is pinned at.

func IsInferredTaskClass

func IsInferredTaskClass(iri string) bool

IsInferredTaskClass reports whether iri is one of the task classes the SAL Module ontology defines for a reasoner to infer rather than to be subclassed.

func IsModuleIRI

func IsModuleIRI(iri string) bool

IsModuleIRI reports whether iri uses the salmodule protocol scheme.

func IsModuleRoot

func IsModuleRoot(dir string) bool

IsModuleRoot reports whether dir holds both files a SAL module needs, ontology.jsonld and a Dockerfile.

func IsTaskBaseClass

func IsTaskBaseClass(iri string) bool

IsTaskBaseClass reports whether iri is one of the SAL Module ontology's own task classes, salmodule:Task or one of the classes inferred from it.

func LinkCopiedFiles

func LinkCopiedFiles(graph *rdflibgo.Graph, fileTriples []rdflibgo.Triple, files []CopiedFile, blobDir string) error

LinkCopiedFiles rewrites every file:/// IRI in a task's output, as subject or object, to the urn:sha256: IRI of the copy that was made of it and describes each copy the way a pinned vocabulary's provenance node is described: the file:/// IRI the task wrote as rdfs:label, when it was written as dcterms:modified, its digest as owl:versionIRI, where it sits under the blob store as dcterms:identifier, and the module that produced it as dcterms:source. Whatever else the task said about the file:/// IRI stays with it under its new name. The data product then refers to the copies it holds rather than to paths inside a container that no longer exists. A file:// IRI that was not copied is an error. A copy nothing refers to as an IRI, because the task wrote the path as a literal, is removed again and warned about. Every copy that is kept is appended to the blob store's prov.jsonld under blobDir, which is the log of everything module tasks have handed over. fileTriples are the triples of graph that name a file:/// IRI, which TaskOutputParser collects as the output is parsed so that the graph, which holds far more than the task's output, is never scanned for them; they are replaced in graph by their linked form.

func LinkCopiedFilesTo added in v0.1.7

func LinkCopiedFilesTo(sink TripleSink, fileTriples []rdflibgo.Triple, files []CopiedFile, blobDir string) error

LinkCopiedFilesTo is LinkCopiedFiles for output that was never added to a graph: the linked triples and the description of every copy are handed to sink instead, which is how a build sends them where the rest of the task's output went.

func LocalNamespace

func LocalNamespace() string

LocalNamespace is the salmodule:// namespace of the module in the current directory, derived from its git remote the way a project referencing it would write it, or PlaceholderNamespace, with a warning, when there is no remote to derive it from.

func ModuleNamespaceForRemote

func ModuleNamespaceForRemote(remote string) (string, error)

ModuleNamespaceForRemote returns the salmodule:// namespace of the module whose git repository is at remote, an https clone URL such as https://github.com/owner/repo.git: the namespace a project referencing the module would declare, and so the one its own ontology resolves against.

func RemoveCopiedFiles

func RemoveCopiedFiles(blobDir string) ([]string, error)

RemoveCopiedFiles deletes every file and directory prov.jsonld under blobDir records, and prov.jsonld itself. It is what a wipe does with them: they are artifacts of a run, regenerated by the next one, and without the table that refers to them nothing resolves their digests. It is the only time sal removes a copy; a run only ever adds to the blob store.

func VocabularyNamespace

func VocabularyNamespace(iri string) string

VocabularyNamespace is the namespace a term IRI belongs to: everything up to and including its last "#", else its last "/". An IRI with neither, such as a urn:, belongs to no namespace and reports "".

func WriteEditorSchema

func WriteEditorSchema(dir string) error

WriteEditorSchema copies the SAL module ontology JSON Schema into .vscode and registers it in .vscode/settings.json for ontology.jsonld, so that an editor checks the ontology against the SAL Module specification as it is written. The schema copy is sal's, not the module's, so a stale copy is rewritten; the settings file is the module's, so an existing one gains only the json.schemas entry, and one sal cannot parse (settings.json may hold comments) is left alone with a warning saying what to add by hand.

func WriteModuleFiles

func WriteModuleFiles(dir string, ontology string, overwrite bool) error

WriteModuleFiles scaffolds the files a SAL module repository needs: the ontology.jsonld document plus a Dockerfile and .dockerignore holding only a comment saying what belongs in each. A Dockerfile or .dockerignore that already exists is left alone so that re-running init never clobbers a module's own work; ontology.jsonld is written when overwrite is set, which is how `sal edit` saves the document the form edited, and left alone with a warning otherwise.

Types

type CommandRunner

type CommandRunner func(ctx context.Context, dir string, name string, args ...string) ([]byte, error)

CommandRunner runs an external command and returns its combined output, allowing git usage to be faked in tests.

type ContainerEntryVisitor

type ContainerEntryVisitor func(relativePath string, isDir bool, modified time.Time, content io.Reader) error

ContainerEntryVisitor receives one entry of a directory copied out of a container. content is nil for a directory entry.

type ContainerFiles

type ContainerFiles interface {
	// CopyFile writes the contents of the regular file at path inside the
	// container to w and reports when the file was last modified there.
	CopyFile(ctx context.Context, path string, w io.Writer) (time.Time, error)
	// CopyDirectory walks the directory at path inside the container and calls
	// visit once per entry beneath it, with the entry's path relative to the
	// directory. A directory entry is visited with a nil reader; a regular file
	// is visited with a reader over its contents that is only valid during the
	// call. Walking stops at the first error visit returns.
	CopyDirectory(ctx context.Context, path string, visit ContainerEntryVisitor) error
}

ContainerFiles copies files out of a container RunContainer created.

type ContainerOutput

type ContainerOutput struct {
	Stdout io.Reader
	Stderr io.Reader
	Files  ContainerFiles
}

ContainerOutput is what a running container writes, as it writes it.

type ContainerOutputConsumer

type ContainerOutputConsumer func(ctx context.Context, output ContainerOutput) error

ContainerOutputConsumer reads a running container's stdout and stderr until both end, which happens when the container exits, and may copy files out of the container through output.Files until it returns. It must read both streams, concurrently, since the container's output arrives multiplexed and a stream left unread holds up the other.

type ContainerRunner

type ContainerRunner interface {
	BuildImage(ctx context.Context, contextDir string, tag string) error
	// ImageExists reports whether the daemon already holds an image under tag,
	// which is how a module built by a previous invocation is found and reused.
	ImageExists(ctx context.Context, tag string) (bool, error)
	// RunContainer runs cmd in a container built from image. What the container
	// writes to stdout and to stderr is handed to consume as it is written,
	// together with a handle that copies files out of the container; the
	// container is kept until consume returns, so a file may still be copied
	// after the container has exited. A container that exits non-zero is an
	// error naming its status and the end of what it wrote to stderr.
	RunContainer(ctx context.Context, image string, env []string, cmd []string, consume ContainerOutputConsumer) error
}

ContainerRunner builds and runs the container images that back SAL modules.

type CopiedFile

type CopiedFile struct {
	// ContainerPath is the absolute path the file had inside the container,
	// ending in a slash for a directory.
	ContainerPath string
	// Directory reports whether the copy is a whole directory rather than one
	// file.
	Directory bool
	// Source is the salmodule:// IRI of the module whose task produced the
	// file, recorded as dcterms:source. It is the IRI the table records the
	// module under, the module's namespace without its trailing slash.
	Source string
	// Modified is when the file was last written inside the container, or the
	// newest such time of any file in a directory, recorded as
	// dcterms:modified.
	Modified time.Time
	// Digest is the hex SHA-256 of the file's contents, or of a directory's
	// tree (see treeDigest). It names a file in the blob store.
	Digest string
	// BlobPath is where the copy sits relative to the blob store, recorded as
	// dcterms:identifier: <digest> for a file, <digest>/ for a directory. It
	// is also the path the copy is served at under /blobs/.
	BlobPath string
	// Path is where the copy landed on disk.
	Path string
}

CopiedFile is a file or directory a task named with a file:/// IRI in its output and that was copied out of the container into the blob store. Both are stored under the digest of their contents so that they are content addressable and two modules naming a path the same way never collide: a file as the bare digest, a directory as <digest>/ with the files inside it kept verbatim, so that whatever reads it, a Zarr or STAC client for instance, finds the layout it expects. Everything in the blob store but prov.jsonld is named by a hash. Every copy is recorded in prov.jsonld together with the module that produced it.

func (CopiedFile) FileIRI

func (f CopiedFile) FileIRI() string

FileIRI returns the file:/// IRI the task named the file or directory with, file:///<container path>, ending in a slash for a directory. It is what the copy is labelled with, in the graph and in prov.jsonld, so that a copy is still known by the name its task gave it once it is stored under a digest.

func (CopiedFile) IRI

func (f CopiedFile) IRI() string

IRI returns the identifier the file's triples use in place of the file:/// IRI the task wrote, urn:sha256:<digest>, which is the same form the pinned vocabulary documents in the blob store are named with.

type ModuleError

type ModuleError struct {
	Path    string
	Line    int
	Message string
}

ModuleError is one thing wrong with a SAL module, at a line of the file it is in. Line is 0 when the problem is the file as a whole, such as a missing Dockerfile.

func (ModuleError) Error

func (e ModuleError) Error() string

type ModuleOntology

type ModuleOntology struct {
	// Namespace is the vocabulary base the ontology's relative terms resolve against.
	Namespace string
	// Document is the raw JSON-LD the module printed.
	Document []byte
	// Context is the ontology's @context, injected into the JSON a module task
	// writes so that its keys resolve to the module's vocabulary.
	Context json.RawMessage
	// contains filtered or unexported fields
}

ModuleOntology is the JSON-LD vocabulary a SAL module prints in response to its ontology command.

func Inspect

func Inspect(ctx context.Context, reference string) (*ModuleOntology, error)

Inspect clones the module's repository, builds its Dockerfile, and runs its ontology command, returning the vocabulary the module published. Docker's own layer cache makes a repeated inspection of an unchanged module cheap, and the shared resolver keeps a module referenced twice in one invocation to a single clone and build.

func (*ModuleOntology) IsTaskClass

func (o *ModuleOntology) IsTaskClass(iri string) bool

IsTaskClass reports whether the ontology declares iri as a subclass of one of the SAL Module ontology's task classes.

func (*ModuleOntology) NewTaskOutputParser

func (o *ModuleOntology) NewTaskOutputParser(graph *rdflibgo.Graph, base string) *TaskOutputParser

NewTaskOutputParser returns the parser that adds what a task of this module writes to stdout to graph, resolving relative IRIs against base.

func (*ModuleOntology) NewTaskOutputStream added in v0.1.7

func (o *ModuleOntology) NewTaskOutputStream(sink TripleSink, base string) *TaskOutputParser

NewTaskOutputStream returns the parser that hands what a task of this module writes to stdout to sink, resolving relative IRIs against base. No graph of the output is built; only the triples of the salmodule:Error nodes it reports are kept, for Finish.

func (*ModuleOntology) NewTaskStderrParser

func (o *ModuleOntology) NewTaskStderrParser(base string) *TaskOutputParser

NewTaskStderrParser returns the parser for the nodes a task of this module writes to stderr, its control channel: the salmodule:Error nodes a failing task reports why it failed with, which Finish reports as a TaskError. The nodes are parsed the way stdout is, with the module's @context and against base, into a graph of the parser's own, since nothing on stderr is data for the graph being built. RunTask hands Line only the lines of stderr that are JSON objects; the rest are log lines it shows itself.

func (*ModuleOntology) StderrValidator

func (o *ModuleOntology) StderrValidator(classIRI string, base string) (*StreamValidator, error)

StderrValidator is StdoutValidator for stderr, built from the class's salmodule:stderrShape. The SAL Module ontology's own shape for stderr, that a salmodule:Error carries an rdfs:label and an rdfs:comment, is not among them: TaskOutputParser.Finish reads both itself and reports a node missing either as such, rather than stopping the task for the shape of its failure report and hiding what it did say.

func (*ModuleOntology) StdoutValidator

func (o *ModuleOntology) StdoutValidator(classIRI string, base string) (*StreamValidator, error)

StdoutValidator returns the validator for the stdout of a task of class classIRI, built from every salmodule:stdoutShape the class carries, or a superclass of it the ontology declares carries. It returns nil when there is none, in which case the output is not validated at all and costs nothing. Only the shapes themselves, with everything reachable from them, make up the shapes graph, so the ontology's other shape annotations never apply to stdout. A shape using a constraint SAL does not support is an error.

base is what a relative IRI in the output resolves against, the project namespace, the same base TaskOutputParser is given, so a shape sees the node exactly as it will be committed.

func (*ModuleOntology) TaskInstance

func (o *ModuleOntology) TaskInstance(graph *rdflibgo.Graph, subject rdflibgo.Subject, classIRI string) (string, error)

TaskInstance renders the JSON-LD node object SAL passes to a module through its task instance environment variable. A module is configured in RDF rather than with an embedded JSON literal, so the node object is built from the instance's own properties: every predicate the module's vocabulary defines is an input to the task, and predicates from any other vocabulary are left out since only the module knows what configures it.

type ModuleRef

type ModuleRef struct {
	// Namespace is the vocabulary base that the module's ontology terms resolve
	// against. It always ends in a slash.
	Namespace string
	// CloneURL is the git repository holding the module's Dockerfile.
	CloneURL string
	// ImageRepository is the local docker repository name SAL builds the module
	// under. The image itself is tagged with the git commit hash of the module
	// repository it was built from; see ImageTagFor.
	ImageRepository string
}

ModuleRef is a SAL module dereferenced from a salmodule:// IRI.

func ParseModuleIRI

func ParseModuleIRI(iri string) (ModuleRef, error)

ParseModuleIRI resolves a salmodule://[HOST/]OWNER/REPO IRI into the git repository that provides the module and the local image tag SAL builds it as. Any fragment or term suffix on the IRI is ignored so that both a vocabulary base and an individual term IRI resolve to the same module.

func (ModuleRef) ImageTagFor

func (r ModuleRef) ImageTagFor(commit string) string

ImageTagFor returns the docker tag a module built from commit is stored under. Tagging by commit is what lets a later invocation reuse the image a previous one built: the commit a project pins in .sal/config.jsonld names the exact tag to look for.

type Provenance

type Provenance struct {
	// contains filtered or unexported fields
}

Provenance is the contents of a blob store's prov.jsonld.

func LoadProvenance

func LoadProvenance(blobDir string) (*Provenance, error)

LoadProvenance reads prov.jsonld from blobDir. A blob store without one holds no copies and loads empty.

func (*Provenance) AppendProvenance

func (p *Provenance) AppendProvenance(graph *rdflibgo.Graph) int

AppendProvenance describes every copy prov.jsonld records in graph, under the urn:sha256: IRI of its contents, the way LinkCopiedFiles describes a copy when it is made: its file:/// IRI as rdfs:label, when it was written as dcterms:modified, its path under the blob store as dcterms:identifier, the module that produced it as dcterms:source, and its digest as owl:versionIRI, plus the rdfs:comment the record carries. It is what carries a copy made by an earlier run into a table built again from source, so that the table describes every copy the blob store holds and a copy is listed with the pinned vocabularies. A statement the graph already holds is left as it is. It returns how many copies were described.

func (*Provenance) BlobPaths

func (p *Provenance) BlobPaths() []string

BlobPaths returns the path, relative to the blob store and ending in a slash for a directory, of every copy prov.jsonld records.

func (*Provenance) Label

func (p *Provenance) Label(blobPath string) (string, bool)

Label returns the file:/// IRI the task named the copy at blobPath with, and whether prov.jsonld records a copy there.

type Resolver

type Resolver struct {
	// Runner runs docker operations. A client for the local docker daemon is
	// created on first use when this is nil.
	Runner ContainerRunner
	// Command runs git. os/exec is used when this is nil.
	Command CommandRunner
	// contains filtered or unexported fields
}

Resolver dereferences salmodule:// IRIs by cloning the module's git repository, building the Dockerfile in its root, and invoking the SAL Module command line interface inside the resulting image.

A module is cloned and built at most once per resolver.

func Default

func Default() *Resolver

Default returns the resolver shared by validation and build so that a module referenced from several places is only cloned and built once per invocation.

func (*Resolver) CommitHash

func (r *Resolver) CommitHash(ctx context.Context, ref ModuleRef) (string, error)

CommitHash returns the git commit hash of the HEAD of the module repository the last time it was cloned, cloning and building it first if it has not been referenced yet. A salmodule:// vocabulary is pinned by this rather than by the digest of its ontology document, since code in the module that changes what a task does is not necessarily a change to the ontology itself.

func (*Resolver) Downloaded

func (r *Resolver) Downloaded() []string

Downloaded returns the salmodule:// URI of every module the resolver has resolved to an image, whether it cloned and built the module or reused a prebuilt image, and whether it was dereferenced for its vocabulary or run as a task. A build records these so that a table says which modules produced it.

func (*Resolver) Ontology

func (r *Resolver) Ontology(ctx context.Context, ref ModuleRef) (_ *ModuleOntology, err error)

Ontology returns the vocabulary the module publishes through its ontology command.

func (*Resolver) Reset

func (r *Resolver) Reset()

Reset drops every module the resolver has already cloned, built, and dereferenced so that the next reference resolves from scratch.

func (*Resolver) RunTask

func (r *Resolver) RunTask(ctx context.Context, ref ModuleRef, taskInstance string, blobDir string, stdout TaskStream, stderr TaskStream) (result TaskResult, err error)

RunTask invokes the module's run command with taskInstance supplied through the TaskInstanceEnvVar environment variable. Each line of the newline delimited JSON the task writes to stdout is handed to stdout's sink as it is read, so the output is never held whole, and the files the lines name with file:/// IRIs are copied out of the container into blobDir as the output arrives: a file named by its SHA-256 digest, a directory (a path ending in a slash) verbatim under its own path. The container is kept until every copy has finished.

Stderr is the task's control channel rather than its data: a line on it that is a JSON object is a node, a salmodule:Error reporting why the task failed say, and goes to stderr's sink the same way; any other line is a log line, shown as a warning and otherwise ignored. Every line the task wrote reaches its sink even when the container fails, because a task reports its own failures as salmodule:Error nodes before exiting non-zero; those messages describe the failure far better than the container's exit status does.

A stream's validator checks each node as it arrives, before any file the line names is copied or the sink sees it. The first node on either stream that fails, or that a sink refuses, ends the run: the container is stopped, copies still in flight are abandoned, and the error is returned in place of a result.

func (*Resolver) UsePinnedCommit

func (r *Resolver) UsePinnedCommit(namespace string, commit string)

UsePinnedCommit tells the resolver which git commit the project pins the module at, so that the image a previous invocation built and tagged with that commit can be reused instead of cloning and building the module again.

type SalModuleCmd

type SalModuleCmd struct {
	// Ontology is needed so that the sal cli itself is a sal module
	Ontology *ontologyCmd `arg:"subcommand:ontology" help:"Print the ontology of the sal cli itself"`
	SalRun   *runCmd      `arg:"subcommand:run" help:"Run a sal project"`
	Inspect  *inspectCmd  `arg:"subcommand:inspect" help:"Print the ontology that a remote SAL module publishes"`
}

func (*SalModuleCmd) Run

func (cmd *SalModuleCmd) Run() error

type StreamShapeError

type StreamShapeError struct {
	Namespace string
	// Class is the task class whose shape was violated.
	Class string
	// Stream is "stdout" or "stderr", the stream the node was written on and
	// the shape it violated: salmodule:<Stream>Shape.
	Stream string
	// Line is the 1-based line of the stream the node was written on.
	Line       int
	Violations []string
}

StreamShapeError reports the first node of a task's stdout or stderr that did not conform to the salmodule:stdoutShape or salmodule:stderrShape its class declares for the stream.

func (StreamShapeError) Error

func (e StreamShapeError) Error() string

type StreamValidator

type StreamValidator struct {
	// contains filtered or unexported fields
}

StreamValidator validates the newline delimited JSON a task writes to one of its output streams, one node at a time and as each arrives, against the SHACL shapes its class declares for that stream with salmodule:stdoutShape or salmodule:stderrShape.

func (*StreamValidator) ValidateLine

func (v *StreamValidator) ValidateLine(line []byte, lineNumber int) error

ValidateLine checks one node of a task's output, the JSON written on line lineNumber of the stream, against the shapes. The module's @context is injected and the node parsed the way TaskOutputParser parses the whole output, so what is validated is the RDF the node will become. A violation is returned as a StreamShapeError; a result of any lesser severity is logged and does not fail the line.

type TaskError

type TaskError struct {
	Namespace string
	Messages  []string
}

TaskError reports the salmodule:Error nodes a module task emitted to say why it failed. Messages holds one entry per node, "label: comment" for a node carrying both the rdfs:label and the rdfs:comment the SAL Module specification requires of an error, and for a node missing either a message that says so alongside whatever the node did carry, so a module that reports its failure badly is told as much without the report being hidden.

func (TaskError) Error

func (e TaskError) Error() string

type TaskOutputParser

type TaskOutputParser struct {
	// contains filtered or unexported fields
}

TaskOutputParser turns the newline delimited JSON a module task writes to stdout into RDF as the lines arrive, handing the triples straight to the TripleSink it was made for, or adding them to a graph. A task emits plain JSON, so the module ontology's @context is injected to resolve the keys against the module's vocabulary. Nothing the task wrote is kept once its batch has been parsed, which is what lets a task produce far more output than fits in memory.

The blank nodes of a batch are labelled one connected component at a time (pkg.StableBlankNodeComponents), so the labels a node gets depend on that node alone and not on the batch it landed in or on anything else the build holds. A blank node is therefore local to the line that writes it, which is what the SAL Module specification's one node per line means: a label a module reuses across lines does not join them.

A triple naming a file:/// IRI is not handed on: it is kept, see FileTriples, until the copy it names has landed and LinkCopiedFiles can hand it on under the copy's urn:sha256: IRI.

A relative IRI in the output resolves against base, the namespace of the project the task was run for, not against the module's own namespace. What a task produces is instance data belonging to the project that configured and ran it; the module's namespace names its vocabulary, and a module that wants its output under some other namespace writes absolute IRIs.

Line is the TaskOutputSink to hand RunTask; Finish parses whatever is still pending once the task has ended and reports the errors the task declared.

Batches are parsed by up to taskOutputParseWorkers goroutines at once, but everything else happens on the goroutine calling Line and Finish, in the order the lines were written: the sink sees one batch's triples after the previous batch's, exactly as it would were the batches parsed one by one, and is never called concurrently. Since a batch is handed on once parsed rather than when it fills, a batch that fails to parse, or whose triples the sink refuses, fails a later call to Line, or Finish.

The same parser reads a task's stderr, through NewTaskStderrParser, into a graph of its own: the nodes there are the task's control channel, never its data, and only the salmodule:Error nodes among them are of interest.

func (*TaskOutputParser) FileTriples

func (p *TaskOutputParser) FileTriples() []rdflibgo.Triple

FileTriples returns every triple parsed so far that names a file:/// IRI as its subject or object, none of which was handed to the sink, for LinkCopiedFiles to rewrite and hand on.

func (*TaskOutputParser) Finish

func (p *TaskOutputParser) Finish() error

Finish parses the lines still pending and reports the salmodule:Error nodes the task emitted, if any, as a TaskError. It is called once the task has ended, whether or not it succeeded, since a failing task reports why it failed before it exits. The SAL Module specification has a task report its errors on stderr; one reported on stdout, where a module written before that was settled puts it, still fails the task, with a warning that it is out of place. An error must carry both an rdfs:label and an rdfs:comment, as the stderrShape on salmodule:Task requires; a node missing either is still reported, since it is the task's own account of its failure, but the message names what is missing.

func (*TaskOutputParser) Line

func (p *TaskOutputParser) Line(line []byte, lineNumber int) error

Line adds one line of the task's output to the batch being assembled and parses the batch once it is full. A line that is not a JSON object is an error, since every line of a task's output is one node.

func (*TaskOutputParser) Vocabularies

func (p *TaskOutputParser) Vocabularies() []string

Vocabularies returns, sorted, the namespaces the output's triples are stated in, which the build pins so that the vocabularies the table ends up holding terms of are the ones the project lists. A namespace counts when a @context binds a prefix (or a term, or @vocab) to it, the module ontology's context or one a node carried, and the output used a term of it as a predicate, as the class of a typed node, or as a literal's datatype; the namespace reported is the bound IRI, the longest one the term starts with, the way a source file's declared prefix is what gets pinned. A prefix bound without the output using it, sh for the shapes of the module's ontology say, is not among them, nor is a namespace that only names subjects or objects, since those are instances rather than vocabulary, nor one the output only ever wrote terms of as full IRIs: writing a full IRI is a reference to something, the way a Turtle file writing one is, while binding a prefix declares that the output is stated in that vocabulary. A harvested node typed with a class of an ontology the module never bound, a KnowWhereGraph S2 cell in Geoconnex data say, is left to the project to declare if it wants it pinned.

type TaskOutputSink

type TaskOutputSink func(line []byte, lineNumber int) error

TaskOutputSink receives each node a task writes to one of its output streams, the line trimmed of surrounding whitespace, as it is read, with its 1-based line number on that stream. An error it returns stops the task.

type TaskResult

type TaskResult struct {
	OutputBytes int64
	Files       []CopiedFile
}

TaskResult is what running a module task produced: how much newline delimited JSON it wrote to stdout, which went to the sink line by line, and the files and directories it named with file:/// IRIs, copied out of the container into the blob store.

type TaskStream

type TaskStream struct {
	Validator *StreamValidator
	Sink      TaskOutputSink
}

TaskStream is what RunTask does with one of a task's output streams: each node written on it is checked by Validator, when there is one, and then handed to Sink, when there is one. A line is only ever one node, so neither sees a line that is not a JSON object; on stdout that is an error, on stderr it is a log line sal shows as a warning.

type TripleSink added in v0.1.7

type TripleSink func(rdflibgo.Triple) error

TripleSink receives the triples a task's output parses into, one at a time, already relabelled so that its blank nodes are stable. An error it returns stops the task. A build hands the parser one that writes the triples straight to Parquet (load.Spool), so that what a task produces is never held in memory as a graph.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL