Documentation
¶
Overview ¶
Package rdf is a small, fast, dependency-free RDF toolkit. It provides the RDF triple model (Term, Triple, Graph), parsers and serializers for the four common serializations — RDF/XML, JSON-LD, Turtle and N-Triples — and a streaming decoder for the line-based formats.
Two reading modes:
- Whole document: ParseRDFXML, ParseJSONLD, ParseTurtle and ParseNTriples take a []byte and return a *Graph. Fast and convenient for inputs that fit in memory. The line-based parsers have zero-copy variants (ParseNTriplesShared, ParseNQuadsShared) that back terms with the caller's buffer instead of a private copy — one input-sized allocation less, for callers that keep the buffer immutable.
- Streaming: NewDecoder reads N-Triples, N-Quads, RDF/XML or Turtle from an io.Reader one triple at a time in constant memory, for inputs too large to materialize (e.g. the multi-gigabyte Library of Congress authority dumps). JSON-LD is whole-document only.
A parsed N-Quads document is a Dataset. Reading one of its named graphs has two modes: Dataset.Graph materializes the graph's triples as a Graph, while Dataset.GraphView answers the same queries (the GraphQuery surface) over positions into the dataset's own quads, copying no triple. Prefer the view for read-only access at corpus scale, where the copy dominates the allocation profile.
The parsers target the constructs real-world RDF uses rather than the whole of each specification; see each parser's documentation for what is and isn't handled (notably, relative-IRI resolution against a document base is not performed). There are no third-party dependencies.
Index ¶
- Constants
- Variables
- func LocalName(iri string) string
- func TurtleHeader(prefixes map[string]string) []byte
- type Dataset
- func (d *Dataset) Add(s, p, o, g Term)
- func (d *Dataset) Canonical() ([]byte, error)
- func (d *Dataset) Graph(graph Term) *Graph
- func (d *Dataset) GraphLen(graph Term) int
- func (d *Dataset) GraphView(graph Term) *GraphView
- func (d *Dataset) Graphs() []Term
- func (d *Dataset) HasGraph(graph Term) bool
- func (d *Dataset) NQuads() []byte
- type Decoder
- type Encoder
- type Format
- type Graph
- func (g *Graph) Add(s, p, o Term)
- func (g *Graph) Canonical() ([]byte, error)
- func (g *Graph) Dedupe() int
- func (g *Graph) HasType(subject Term, typeIRI string) bool
- func (g *Graph) Literal(subject Term, predicate string) (string, bool)
- func (g *Graph) NQuads(graph Term) []byte
- func (g *Graph) NTriples() []byte
- func (g *Graph) Object(subject Term, predicate string) (Term, bool)
- func (g *Graph) Objects(subject Term, predicate string) []Term
- func (g *Graph) ObjectsWithRepeats(subject Term, predicate string) []Term
- func (g *Graph) SubjectsOfType(typeIRI string) []Term
- func (g *Graph) Turtle(prefixes map[string]string) []byte
- func (g *Graph) TurtleBody(prefixes map[string]string) []byte
- type GraphQuery
- type GraphView
- func (v *GraphView) Empty() bool
- func (v *GraphView) GraphTerm() Term
- func (v *GraphView) HasType(subject Term, typeIRI string) bool
- func (v *GraphView) Len() int
- func (v *GraphView) Literal(subject Term, predicate string) (string, bool)
- func (v *GraphView) Object(subject Term, predicate string) (Term, bool)
- func (v *GraphView) Objects(subject Term, predicate string) []Term
- func (v *GraphView) SubjectsOfType(typeIRI string) []Term
- func (v *GraphView) Triples() iter.Seq[Triple]
- type Kind
- type Quad
- type SyntaxError
- type Term
- type Triple
Constants ¶
const ( NS = "http://www.w3.org/1999/02/22-rdf-syntax-ns#" TypeIRI = NS + "type" FirstIRI = NS + "first" RestIRI = NS + "rest" NilIRI = NS + "nil" XSDString = "http://www.w3.org/2001/XMLSchema#string" )
Well-known IRIs.
Variables ¶
var ErrCanonComplexity = errors.New("rdf: canonicalization exceeded complexity budget")
ErrCanonComplexity is returned by the canonicalization functions when a graph's blank-node structure would require more work than maxCanonWork permits — the guard against the algorithm's exponential worst case on adversarial "poison" datasets. Well-formed data never approaches the limit.
Functions ¶
func LocalName ¶
LocalName returns the fragment or last path segment of an IRI (the part after the final '#' or '/').
func TurtleHeader ¶
TurtleHeader returns the @prefix declaration block for the given prefixes, terminated by a blank line. Collection writers emit it once.
Types ¶
type Dataset ¶ added in v0.4.0
type Dataset struct {
// Quads are in document order and may repeat; see the type doc.
Quads []Quad
// contains filtered or unexported fields
}
Dataset holds quads — RDF statements each tagged with the graph they belong to — in document order. It is the quad-level analogue of Graph and what ParseNQuads returns; it is how provenance (which source a statement came from) is carried.
Like Graph.Triples, Quads is a list and may repeat a statement the document stated twice; see the Graph doc for why parsing preserves that.
func Canonicalize ¶ added in v0.5.0
Canonicalize returns a new Dataset with blank nodes relabeled to their canonical _:c14nN identifiers and quads in canonical order, leaving d untouched.
func ParseNQuads ¶ added in v0.4.0
ParseNQuads parses an N-Quads document into a Dataset, preserving each statement's graph term; lines carrying only three terms fall in the default graph. It shares the N-Triples term reader, so the same escaping and leniency apply — blank, comment and malformed lines are skipped. One private copy of the input backs every term, so data is free for reuse once it returns.
func ParseNQuadsShared ¶ added in v0.13.0
ParseNQuadsShared is ParseNQuads without the private input copy: terms are zero-copy views into data itself, saving one input-sized allocation — worth it when read-heavy consumers parse corpus-scale documents. In exchange the caller must not modify data while the Dataset, or any Graph or Term derived from it, remains in use.
func (*Dataset) Canonical ¶ added in v0.5.0
Canonical returns the dataset serialized as canonical N-Quads per the W3C RDF Dataset Canonicalization algorithm RDFC-1.0 (URDNA2015): blank nodes are relabeled to isomorphism-invariant _:c14nN identifiers and the quads are sorted, so two datasets that differ only in statement order or blank-node labels produce byte-identical output. It returns ErrCanonComplexity for a graph whose symmetry exceeds the work budget.
func (*Dataset) Graph ¶ added in v0.4.0
Graph returns the triples belonging to the given graph term as a Graph; pass a zero-value term for the default graph. It copies one Triple per matching quad, so a caller that only reads the result should use GraphView instead, which answers the same queries over the dataset's quads without copying.
func (*Dataset) GraphLen ¶ added in v0.20.0
GraphLen returns the number of statements in the given graph term; pass a zero-value term for the default graph. It reads a cached per-graph count, so after the first call on a dataset it costs no scan — asking about several graphs, or about a graph that turns out to be empty, is one pass in total rather than one pass per graph.
func (*Dataset) GraphView ¶ added in v0.19.0
GraphView returns a read-only, zero-copy view of the given graph term; pass a zero-value term for the default graph. It is the allocation-free counterpart to Graph, which materializes the same statements as a []Triple copy.
func (*Dataset) Graphs ¶ added in v0.4.0
Graphs returns the distinct graph terms present in the dataset, in first-seen order — the set of provenance sources. It reads the same cached counts GraphLen does, which record each graph on first sight, so repeated calls cost no scan.
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder streams RDF statements from an io.Reader one triple at a time, in constant memory — for inputs far too large to materialize into a Graph (multi-gigabyte dumps such as the LC authority files). Each returned triple owns its strings, so it is safe to retain after the next call.
JSON-LD is not streamable here (its document must be materialized); use ParseJSONLD for that.
func NewDecoder ¶
NewDecoder returns a streaming Decoder reading the given format from r.
For the line-based formats (N-Triples, N-Quads) the decoder reads a line at a time and holds only one statement at once, so it streams a document of any size in bounded memory -- with one caveat: a "line" is the bytes up to the next newline, accumulated without limit, so input that never emits one grows a single line until it exhausts memory. When r is untrusted, wrap it in an io.LimitReader, or a reader that caps bytes-since-newline, before handing it over. The parser bounds what it holds per statement; it cannot bound what the input calls one line.
func (*Decoder) All ¶
All returns an iterator over the remaining triples, ending at the first error (io.EOF, the normal end, is not surfaced). Breaking out of the loop stops the producer:
for tr := range dec.All() { ... }
Use Decode directly when a non-EOF read error must be observed.
func (*Decoder) AllQuads ¶ added in v0.4.0
AllQuads returns an iterator over the remaining statements as quads, preserving N-Quads graph terms. Like All, it ends at the first error and breaking out of the loop stops the producer:
for q := range dec.AllQuads() { ... }
func (*Decoder) Close ¶
Close stops a streaming decoder early, releasing its producer goroutine. It is a no-op for the line-based formats and after the stream is exhausted.
func (*Decoder) Decode ¶
Decode returns the next triple, or io.EOF when the input is exhausted. Any fourth (graph) term on an N-Quads line is dropped; use DecodeQuad to keep it. Blank and comment lines are skipped; a malformed one is a *SyntaxError naming the line, unless Decoder.SkipMalformed was set. io.EOF therefore means the input ended, not that it ended and some of it was unreadable.
func (*Decoder) DecodeQuad ¶ added in v0.4.0
DecodeQuad returns the next statement as a quad, preserving the graph term of an N-Quads line; the triple-only formats (N-Triples, RDF/XML, Turtle) yield a zero-value graph term (the default graph). It returns io.EOF at end of input.
func (*Decoder) SkipMalformed ¶ added in v0.26.0
SkipMalformed makes the decoder skip lines it cannot parse instead of failing on them, and returns the decoder so it can be chained onto NewDecoder. It affects only the line-based formats (N-Triples, N-Quads); RDF/XML and Turtle have always reported a syntax error.
The default is to fail. A parser that skips what it cannot read turns a truncated dump into a smaller, well-formed graph, and the caller has no way to tell. Opt in only where the input is known to carry noise that is safe to drop -- and where a short read is not a lie you would ship.
type Encoder ¶ added in v0.4.0
type Encoder struct {
// contains filtered or unexported fields
}
Encoder serializes a sequence of graphs into one document — N-Triples, N-Quads or Turtle — keeping blank-node labels unique across the graphs it writes. Blank-node labels are scoped to the whole document, so a single Encoder must be reused for every graph written to one stream; a fresh encoder per graph would let graphs that each number their blanks from scratch (such as one BIBFRAME graph per record) collide and merge. It is the serialization counterpart to Decoder; the whole-graph Graph.NTriples, Graph.NQuads and Graph.Turtle methods each use a fresh Encoder internally.
func (*Encoder) AppendNQuads ¶ added in v0.4.0
AppendNQuads appends every triple of g to b as N-Quads tagged with the graph term — g's named graph / provenance. A zero-value graph term writes the default graph (plain N-Triples). Each call is a fresh blank-node scope, so graphs that label their blanks from scratch never merge, while their output labels stay unique across the whole document.
func (*Encoder) AppendNTriples ¶ added in v0.4.0
AppendNTriples appends g's triples to b as N-Triples in a fresh blank-node scope — N-Triples being N-Quads restricted to the default graph.
func (*Encoder) AppendQuad ¶ added in v0.4.0
AppendQuad appends one quad to b as an N-Quads line. A default-graph quad (zero-value G) is written as a three-term N-Triples line.
func (*Encoder) AppendTurtle ¶ added in v0.4.0
AppendTurtle appends g's triples to b as a Turtle body (no @prefix header), grouped by subject, in a fresh blank-node scope. It groups without a per-subject map: subjects are ranked by first appearance, then a stable sort of triple indices by that rank makes each subject's triples contiguous in document order — turning thousands of small allocations into a handful.
type Format ¶
type Format int
Format identifies a serialization the streaming Decoder reads.
const ( // NTriples is the line-based N-Triples format. NTriples Format = iota // NQuads is N-Quads; the optional graph label on each line is ignored, so each // statement still yields a triple. NQuads // RDFXML is RDF/XML; triples stream as each node element is parsed. RDFXML // Turtle streams '.'-terminated statements; @prefix/@base carry forward. // SPARQL-style PREFIX/BASE directives (no '.') are not supported when streaming. Turtle )
type Graph ¶
type Graph struct {
// Triples are in document order and may repeat; see the type doc.
Triples []Triple
// contains filtered or unexported fields
}
Graph holds the triples of one RDF graph in document order, with simple lookup helpers built on first use. Term is comparable, so it indexes by subject term directly — no key strings.
Triples is a list, not a set. RDF 1.1 defines a graph as "a set of RDF triples", but a serialization is free to state the same triple more than once, and real ones do: LC's own marc2bibframe2 output repeats a shared agency or vocabulary node under every property that references it, so one of their N-Triples files carries 449 lines for 389 distinct triples. Parsing preserves what the document said, which keeps the parse allocation-light and keeps redundant statements visible to a caller who cares. It also means len(g.Triples) can exceed the triple count a set-backed parser such as rdflib reports for the same document.
The query helpers below hide that: Graph.Objects and Graph.SubjectsOfType return each distinct answer once. Callers who want set semantics over the triples themselves can call Graph.Dedupe, and Graph.Canonical already collapses duplicates on its way to canonical form.
func ParseJSONLD ¶
ParseJSONLD parses a JSON-LD document into a Graph. It expands compact IRIs and terms against an inline @context (prefix maps and term definitions, including "@type":"@id" coercion), and handles @graph, @id, @type, @value/@language/@type literals, @list, nested node objects, and arrays of any of these. It does not fetch remote @context documents or run full JSON-LD 1.1 framing.
func ParseNTriples ¶
ParseNTriples parses an N-Triples document (one "subject predicate object ." statement per line) into a Graph. It also accepts N-Quads, ignoring any fourth (graph) term. Blank and comment lines are skipped; a malformed line is a *SyntaxError naming the line number, so a truncated document does not parse as a smaller graph. One private copy of the input backs every term, so data is free for reuse once it returns.
Use NewDecoder with Decoder.SkipMalformed to tolerate the trailing noise some real-world dumps carry.
func ParseNTriplesShared ¶ added in v0.13.0
ParseNTriplesShared is ParseNTriples without the private input copy: terms are zero-copy views into data itself, saving one input-sized allocation. In exchange the caller must not modify data while the Graph, or any Term drawn from it, remains in use.
func ParseRDFXML ¶
ParseRDFXML parses an RDF/XML document into a Graph. It supports the striped syntax real bibliographic RDF uses: typed node elements and rdf:Description, rdf:about / rdf:nodeID / rdf:ID subjects, rdf:resource and rdf:nodeID object references, nested node elements, literal property values with rdf:datatype and xml:lang, property attributes, and rdf:parseType="Resource". It does not handle RDF containers, reification, or rdf:parseType="Literal"/"Collection".
func ParseTurtle ¶
ParseTurtle parses a Turtle document into a Graph. It supports the constructs real RDF uses: @prefix/@base and SPARQL-style PREFIX/BASE, prefixed names and IRIs, the `a` keyword, predicate-object lists (`;`) and object lists (`,`), blank-node labels, `[ … ]` blank-node property lists, `( … )` collections, and string (including triple-quoted), language-tagged, datatyped, numeric and boolean literals. Local names are read as the common identifier subset.
func (*Graph) Canonical ¶ added in v0.5.0
Canonical returns the graph as canonical N-Quads (its triples in the default graph), per RDFC-1.0. See Dataset.Canonical.
func (*Graph) Dedupe ¶ added in v0.24.0
Dedupe removes duplicate triples in place, keeping the first occurrence of each and preserving document order among the survivors, and reports how many it removed. It gives the graph the set semantics RDF 1.1 ascribes to one, at the cost of a hash of every triple — which is why parsing does not do it.
func (*Graph) HasType ¶
HasType reports whether the subject has rdf:type typeIRI. It scans the index without allocating.
func (*Graph) Literal ¶
Literal returns the value of the subject's first literal object for the predicate, or false. It scans the index without allocating an intermediate slice, serving the frequent single-value field reads.
func (*Graph) NQuads ¶ added in v0.4.0
NQuads serializes the graph's triples as N-Quads, each tagged with the given graph term (its named graph / provenance). A zero-value graph term produces plain N-Triples.
func (*Graph) Object ¶
Object returns the first object for (subject, predicate), or false. It scans the index without allocating an intermediate slice.
func (*Graph) Objects ¶
Objects returns the distinct objects of every triple with the given subject and predicate IRI, in document order. A document that states the same triple twice yields that object once: the answer is a property's value set, not a count of how often the document repeated itself.
func (*Graph) ObjectsWithRepeats ¶ added in v0.24.0
ObjectsWithRepeats returns the objects of every triple with the given subject and predicate IRI, statement for statement, in document order -- including a term the document stated more than once.
This is the raw list view, and it exists because Graph.Triples is a list. Prefer Graph.Objects: a repeated statement carries no information in RDF, so a caller that treats its multiplicity as meaningful is reading something the abstract syntax does not say, and will disagree with any set-backed store that touches the same document. The one honest use is inspecting a serialization as written -- counting redundant statements, or round-tripping a positional encoding that a conformant parser would already have flattened.
func (*Graph) SubjectsOfType ¶
SubjectsOfType returns every subject with rdf:type typeIRI, in document order (deduplicated).
type GraphQuery ¶ added in v0.19.0
type GraphQuery interface {
SubjectsOfType(typeIRI string) []Term
Objects(subject Term, predicate string) []Term
Object(subject Term, predicate string) (Term, bool)
Literal(subject Term, predicate string) (string, bool)
HasType(subject Term, typeIRI string) bool
}
GraphQuery is the read-only query surface shared by *Graph and *GraphView, so a consumer can write one function over a materialized graph and a view alike.
type GraphView ¶ added in v0.19.0
type GraphView struct {
// contains filtered or unexported fields
}
GraphView is a read-only view of the statements in one named graph of a Dataset. It answers the Graph query surface without copying any triple: its lazy index holds int32 positions into the dataset's Quads, the same shape as Graph's subject index. Pass a zero-value graph term to view the default graph.
A view borrows the dataset, so the dataset must outlive it. Appending to the dataset invalidates the view's index; the next query rebuilds it, so a view stays correct across Add at the cost of one reindex. Views are not safe for concurrent use with a writer, and the first query on a view builds the index, so concurrent readers must either share a view whose index is already built or hold one view each.
func (*GraphView) Empty ¶ added in v0.20.0
Empty reports whether the view's graph holds no statements. Like Len it answers from the dataset's cached counts, so a consumer that reads a graph only when it is populated — an editorial overlay, say — pays nothing to skip an absent one.
func (*GraphView) GraphTerm ¶ added in v0.19.0
GraphTerm returns the graph term this view is scoped to.
func (*GraphView) HasType ¶ added in v0.19.0
HasType reports whether the subject has rdf:type typeIRI in this graph. It scans the index without allocating.
func (*GraphView) Len ¶ added in v0.19.0
Len returns the number of statements in the view's graph. It reads the dataset's cached per-graph counts, so it costs no scan after the first call on that dataset, however many graphs are asked about.
func (*GraphView) Literal ¶ added in v0.19.0
Literal returns the value of the subject's first literal object for the predicate in this graph, or false. It scans the index without allocating an intermediate slice, serving the frequent single-value field reads.
func (*GraphView) Object ¶ added in v0.19.0
Object returns the first object for (subject, predicate) in this graph, or false. It scans the index without allocating an intermediate slice.
func (*GraphView) Objects ¶ added in v0.19.0
Objects returns the distinct objects of every statement in this graph with the given subject and predicate IRI, in document order. As with Graph.Objects, a repeated statement yields its object once.
func (*GraphView) SubjectsOfType ¶ added in v0.19.0
SubjectsOfType returns every subject in this graph with rdf:type typeIRI, in document order (deduplicated). Like Graph.SubjectsOfType it scans rather than keying off the subject index, so it never triggers an index build.
func (*GraphView) Triples ¶ added in v0.19.0
Triples iterates the view's statements in document order, projecting each quad onto its triple. It builds no index and materializes no slice, yielding one Triple value at a time from the dataset's quads; the only allocation is the iterator closure itself, a fixed ~56 bytes per call regardless of graph size.
The cost that matters is the pass, not the yield. Triples filters the whole dataset, so walking it is one full-dataset pass per view — reading N graphs out of one dataset costs N passes, even for graphs that turn out to be empty. Code that consumes several graphs at once can beat that by fusing the dispatch into a single hand-written pass over Dataset.Quads switching on each quad's graph term. Use Empty to skip a graph without walking it at all: it reads the dataset's cached counts and never scans.
Per-triple overhead, by contrast, is not worth avoiding: in this package's Corpus/Grain/SingleGraph Triples benchmarks the iterator beats an equivalent hand-written single-graph loop at both corpus and per-grain scale, even against a single-graph dataset whose loop needs no filter at all.
type Quad ¶ added in v0.4.0
type Quad struct {
S, P, O, G Term
}
Quad is an RDF statement in a named graph: a triple plus the graph it belongs to. A zero-value graph term (G) denotes the default graph, so a Quad is a strict superset of a Triple.
type SyntaxError ¶ added in v0.26.0
type SyntaxError struct {
Line int // 1-based, relative to the input given to the parser; see the type doc
Text string // the offending line, trimmed
}
SyntaxError reports a line that is neither blank, nor a comment, nor a valid N-Triples/N-Quads statement. Line is 1-based; Text is the offending line, trimmed and truncated for the message.
It exists so that a truncated document is an error rather than a smaller graph. A parser that skips what it cannot read turns a half-written dump into a well-formed, wrong answer, and no caller downstream can tell.
Line counts from the start of the input actually handed to the parser, which is not the document when a caller feeds it one chunk at a time: a bad line in the third 1MB chunk of a large dump is reported at its offset within that chunk, and the number looks entirely plausible. Parse a chunked document through NewDecoder instead -- it reads across an io.Reader and numbers lines continuously -- or add the chunk's line base to Line before showing it to anyone.
func (*SyntaxError) Error ¶ added in v0.26.0
func (e *SyntaxError) Error() string
type Term ¶
type Term struct {
Kind Kind
Value string // IRI, blank-node identifier, or a literal's lexical form
Lang string // literal language tag (Literal only)
Datatype string // literal datatype IRI (Literal only)
}
Term is an RDF term: an IRI, a blank node, or a literal.
func NewLiteral ¶
NewLiteral makes a literal; an empty datatype defaults per RDF (xsd:string, or rdf:langString when a language tag is present).