bibframe

package
v0.229.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 14, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package bibframe crosswalks codex records into canonical BIBFRAME N-Quads grains. It wraps libcodex's record<->BIBFRAME conversion (github.com/freeeve/libcodex/bibframe) and adds libcat's provenance graphs plus RDFC-1.0 canonical emission, so each grain re-serializes byte-for-byte when unchanged -- the clean-diff invariant in ARCHITECTURE §3.

Index

Constants

View Source 
const (
	PredCreatorIdentity = LcatNS + "creatorIdentity"
	PredMatchedVia      = LcatNS + "matchedVia"
	PredRetrieved       = LcatNS + "retrieved"
)

Creator-demographics enrichment predicates (ingest/wikidata): a Work links to its creator's resolved knowledge-base entity, and the entity carries the resolution provenance -- which cataloged identifier matched it and when. The demographic claims themselves ride under their real wdt: property IRIs.

View Source 
const (
	PredHasItem = "http://id.loc.gov/ontologies/bibframe/hasItem"

	// PredBarcode and PredItemNote are lcat: extensions -- BIBFRAME models
	// them as identifier/note subgraphs, heavier than a holdings row needs.
	PredBarcode  = LcatNS + "barcode"
	PredItemNote = LcatNS + "itemNote"
)

The minimal bf:Item holdings model: call number, shelving location, barcode, and a note -- never circulation state, which is the ILS's (ARCHITECTURE §5 keeps live state out of the graph). Items are editorial statements on skolem nodes under their Instance, so they survive re-ingest and edit like everything else.

View Source 
const (
	PredHasPart = "http://id.loc.gov/ontologies/bibframe/hasPart"
	PredPartOf  = "http://id.loc.gov/ontologies/bibframe/partOf"
)

Work-to-work relationship predicates the editor writes (058 item 3): the BF 2.0 whole/part inverses. A link is stored in both works' editorial graphs -- hasPart on the whole, partOf on the part -- so each grain self-describes and neither side needs a corpus scan to render its panel. Targets use the same #<id>Work fragment convention the merge and tombstone markers use for cross-grain work references.

View Source 
const (
	// PredTombstoned marks a retired Work: object is the successor Work IRI
	// (redirect target) or the literal "true" (gone, no successor).
	PredTombstoned = LcatNS + "tombstoned"
	// PredSuppressed hides a Work from projection: object literal "true".
	PredSuppressed = LcatNS + "suppressed"
	// PredWithdrawn flags a Work whose sole bib feed no longer lists it
	// ( reconciliation): object literal is the ISO date of the pass
	// that noticed. A flag, not a deletion -- identity and editorial
	// statements survive a title returning to the collection.
	PredWithdrawn = LcatNS + "withdrawnFromFeed"
	// PredSuppressedBy records what set the suppression ("feed-reconcile"
	// for the auto-suppress policy), so clearing a withdrawal un-suppresses
	// only suppressions that pass created.
	PredSuppressedBy = LcatNS + "suppressedBy"
	// PredFeedKept records a curator's "keep despite withdrawal" decision:
	// object literal "true". Reconciliation never re-flags a kept Work.
	PredFeedKept = LcatNS + "feedWithdrawalKept"
)

Work visibility: the delete stance is never row-deletion -- a tombstoned Work disappears from projection and leaves a redirect entry (to a successor Work when named, else an empty target the host serves as gone), while a suppressed Work merely hides from projection with no redirect and is fully restorable. Both are editorial statements, so they survive re-ingest like every other curation decision.

View Source 
const CoverExtraKey = "cover"

CoverExtraKey is the extras key the OPAC's cover slot reads (the adapter passthrough); an uploaded cover writes the same key editorially, overlaying any feed-carried value.

View Source 
const ExtraPred = LcatNS + "extra/"

ExtraPred is the reserved predicate namespace for adopter "extras": per-Work fields that are not BIBFRAME (e.g. cover, rating, dateRead) but a provider wants carried through to catalog.json's `extra` object. A key K is emitted as the predicate ExtraPred+K on the Work node, in the feed provenance graph; the projector harvests the same namespace back into Work.Extra, and the Hugo module forwards it to page params.

View Source 
const LcatNS = "https://github.com/freeeve/libcat/ns#"

LcatNS is libcat's own namespace for statements BIBFRAME 2.0 does not cover (ARCHITECTURE §5, "extend, don't fight the model"). Its clustering-correction predicates live in the editorial graph, so they are preserved across re-ingest and the computed clustering key cannot override a human decision (§4).

View Source 
const LocalAuthorityNS = "https://github.com/freeeve/libcat/authority/"

LocalAuthorityNS is the IRI namespace for locally minted authority terms . Shared vocabularies (Homosaurus, LCSH) bring their own IRIs; a heading a deployment coins itself gets one under this namespace so bf:subject references stay absolute IRIs like any other controlled term.

View Source 
const PredAttachment = LcatNS + "attachment"

PredAttachment carries one staff attachment's filename as a plain literal on the Work: scans, correspondence, acquisition paperwork -- cataloging working material, stored in the blob store beside the grain and NEVER projected to the public catalog. Being an lcat: statement it survives re-ingest like every curation decision, and clones drop it with the rest of the lcat markers (the bytes stay with the source).

The literal is the filename the cataloger uploaded, in whatever script they work in. It is a *display name*, not a path: the bytes live at AttachmentBlobPath, under a segment derived from it. Conflating the two is what made the old client-side sanitizer lossy, collapsing every non-Latin filename onto its bare extension and letting the next upload overwrite it .

View Source 
const PredMARCVerbatim = LcatNS + "marcVerbatim"

PredMARCVerbatim carries a crosswalk-lossy MARC field verbatim on its Instance node: on MARC ingest the known-loss tags (fidelity.go) are serialized field-exact into the feed graph instead of silently dropping; the MARC view shows them, edits to them land editorially under the same predicate (with the override marker shadowing the feed copies), and MARC export re-attaches them so the original forms round-trip.

View Source 
const PredMergedInto = LcatNS + "mergedInto"

PredMergedInto records an editorial merge: the retired Work (subject) was merged into the surviving Work (object). It is the under-merge fix -- two records that should be one Work but clustered apart. The retired Work's grain is dropped on the next ingest (its Instances resolve onto the survivor); the projector emits a redirect so the retired id's URL survives.

View Source 
const PredOverrides = LcatNS + "overrides"

PredOverrides marks an editorial property ownership claim: the statement <subject> lcat:overrides <predicateIRI> in the editorial graph declares that, for this subject and predicate, every feed:* statement is shadowed -- consumers (projector, doc mapper, MARC materializer) show only the editorial values. This gives replace (marker + editorial values), partial removal (marker + the kept subset re-asserted editorially), full removal (marker alone), and one-click revert (delete the marker and the editorial values; the feed resurfaces) -- all without reification, and surviving re-ingest because the editorial graph is preserved verbatim (ARCHITECTURE §5).

View Source 
const PredSplitFrom = LcatNS + "splitFrom"

PredSplitFrom records the provenance of an editorial split: the new Work (subject) was split off from the original over-merged Work (object). It pairs with per-Instance PredWorkAssignment pins that force the split to reproduce on re-ingest even though the computed key still clusters the Instances together.

View Source 
const PredTag = LcatNS + "tag"

PredTag carries an uncontrolled folksonomy tag as a plain literal on a Work. Feed tags arrive as labeled blank bf:Topic nodes, but editorial-class statements must be blank-free, so approved community tags use this predicate instead; the projector folds both shapes into the same Tags dimension.

View Source 
const PredTagAlias = LcatNS + "tagAlias"

PredTagAlias records that a controlled term subsumes an uncontrolled tag: <termURI> lcat:tagAlias "tag" in the alias graph. Written when a folksonomy/feed tag is promoted to a controlled term; the projector then suppresses the tag on Works that carry the aliasing subject (the tag "became" the subject), and pickers auto-suggest the controlled term when the tag is typed. The projector indexes the predicate across every graph, so the statement's home graph is bookkeeping only.

View Source 
const PredWorkAssignment = LcatNS + "workAssignment"

PredWorkAssignment pins an Instance (subject) to a specific Work (object), overriding the computed clustering key -- the mechanism that makes an editorial split reproducible. It lives in the editorial graph, so the pin is preserved across ingest.

View Source 
const SuppressedByReconcile = "feed-reconcile"

SuppressedByReconcile is the PredSuppressedBy actor the auto-suppress reconciliation policy writes.

Variables

View Source 
var CoreFields = []string{
	"001", "006", "007", "008", "020", "040", "100", "245", "250", "260",
	"300", "306", "336", "337", "338", "347", "490", "500", "511", "520",
	"521", "533", "538", "650", "655", "700", "776", "856",
}

CoreFields are the MARC tags guaranteed to survive MARC -> BIBFRAME -> MARC on every vendored sample: the identifiers, primary/added agents, title, publication, extent, carrier, summary, subjects, genre, and access link an adopter judges fidelity by. A regression breaks the build.

View Source 
var ErrBlankNode = errors.New("bibframe: editorial patches must not use blank nodes")

ErrBlankNode reports a patch quad using a blank node.

View Source 
var ErrDuplicateBarcode = errors.New("duplicate barcode")

ErrDuplicateBarcode refuses an item list in which two items carry the same barcode: a barcode names one physical copy, so two copies sharing it cannot be told apart. Handlers map it to a 400.

View Source 
var ErrNoSuchInstance = errors.New("no such instance on this work")

ErrNoSuchInstance refuses an item write against an instance id the grain does not describe; handlers map it to a 400.

View Source 
var ErrNoSuchItemField = errors.New("no such item field")

ErrNoSuchItemField refuses an edit naming a field items do not have, or one (barcode) that is not safe to assign across a selection.

View Source 
var KnownLoss = map[string]string{
	"037": "source of acquisition decodes as an 024-shaped identifier (vendor convention)",
	"084": "other classification number decodes as 072",

	"773": "editorial hasPart/partOf links are not yet shaped as host-item entries",
	"774": "editorial hasPart/partOf links are not yet shaped as constituent-unit entries",
}

KnownLoss maps each tag that does NOT survive the round-trip to why -- now only the vendor-convention fields, which decode to their modeled equivalents rather than their original tags: deliberate non-goals, not losses of data. The reconstruction arc: libcodex v0.9.0 moved 008/336/500 to CoreFields, v0.11.0 moved 306/347/490/511/521/533/538/776 , and v0.12.0 finished 006/007. These tags are what the lcat:marcVerbatim sidecar stores at MARC ingest so exports and the MARC view can reproduce the original forms. (libcodex v0.18.0 moved 040 to CoreFields: bf:AdminMetadata models the agency chain and an internal marcKey note carries the field exactly, so cataloging source is a modeled field again -- record-level provenance beside, not instead of, the named graphs.)

Functions

func AddAuthorityMergeMarker 

func AddAuthorityMergeMarker(grainNQ []byte, loserURI, winnerURI, vocab string) ([]byte, error)

AddAuthorityMergeMarker retires an authority term: <loser> lcat:mergedInto <winner> lands in the loser grain's authority:<vocab> graph, so the vocab index sees the retirement on reload and the decision survives description edits. Idempotent. A loser the grain does not describe is refused: the marker would otherwise mint a phantom labelless node instead of retiring anything (reachable when a grain's subject IRI base differs from the id-derived one, e.g. pre-rename or imported namespaces).

func AddMergeMarker 

func AddMergeMarker(grainNQ []byte, from, to string) ([]byte, error)

AddMergeMarker adds the editorial lcat:mergedInto statement (from -> to) to a Work grain's N-Quads and returns the re-canonicalized grain, so the decision is recorded durably in the survivor's grain and preserved across re-ingest. It is idempotent: a grain that already records this merge is returned unchanged (re-canonicalized). Thin wrapper over ApplyEditorialPatch.

func AddSplitMarkers 

func AddSplitMarkers(grainNQ []byte, newWork, fromWork string, instanceIDs []string) ([]byte, error)

AddSplitMarkers records an editorial split in a grain's N-Quads: a lcat:splitFrom provenance statement (newWork split off fromWork) plus one lcat:workAssignment pin per Instance, all in the editorial graph, and returns the re-canonicalized grain. Adding a marker that already exists is a no-op, so it is idempotent. Recorded in the source Work's grain, the pins are recovered on the next ingest and force the pinned Instances onto newWork. Thin wrapper over ApplyEditorialPatch. Every pinned Instance must be one the grain describes: a pin is a permanent instruction to the identity resolver (SeedPin), so a typo or an id from another record would silently strand that Instance on a work id with no grain at the next ingest (the no-phantom-ids invariant).

func AliasGraph added in v0.54.0 

func AliasGraph() rdf.Term

AliasGraph names the graph tag-alias bookkeeping lives in. Deliberately OUTSIDE the authority:<vocab> namespace: the vocab loader routes every authority:-prefixed graph to a scheme, and alias statements carry no labels, so an authority-class home minted a bogus labelless "aliases" vocabulary that shadowed the promoted terms.

func AppendAuthoritySubject 

func AppendAuthoritySubject(grainNQ []byte, workID string, subj AuthoritySubject, vocab string) ([]byte, error)

AppendAuthoritySubject records a controlled subject on a Work: the bf:subject link lands in the editorial: graph (a human/reviewed decision), while the term's own description (prefLabel per language, broader links) lands in the vocabulary's authority:<vocab> graph so the projector's label index can resolve it. Idempotent.

func AppendAuthorityTerms added in v0.34.0 

func AppendAuthorityTerms(grainNQ []byte, vocab string, terms []AuthoritySubject) ([]byte, error)

AppendAuthorityTerms writes standalone term descriptions (prefLabel per language, broader links) into the authority:<vocab> graph without linking them to any Work -- skos:broader ancestor-chain metadata, so hierarchy nodes no Work carries directly still resolve labels in the projection . Terms with no metadata contribute nothing. Idempotent.

func ApplyEditorialPatch 

func ApplyEditorialPatch(grainNQ []byte, p Patch) ([]byte, error)

ApplyEditorialPatch applies p to the grain's editorial: graph and returns the re-canonicalized grain -- the general form of the merge/split marker writers, and the write path record editing and suggestion publishing share. Additions are idempotent (an already-present statement is not duplicated); removals match exact statements in the editorial graph only.

func ApplyPatch 

func ApplyPatch(grainNQ []byte, graph rdf.Term, p Patch) ([]byte, error)

ApplyPatch applies p to one named graph of the grain and returns the re-canonicalized grain.

func ApplyShadow 

func ApplyShadow(g *rdf.Graph, overrides Overrides) *rdf.Graph

ApplyShadow returns g without the triples the overrides shadow -- the filter consumers run over feed-class graphs before reading them, so editorially-owned properties show only their editorial values.

func AttachmentBlobPath added in v0.79.0 

func AttachmentBlobPath(workID, name string) (string, error)

AttachmentBlobPath is where an attachment's bytes live, sharded like grains and covers. name is the display name; the final segment is its encoding.

func AttachmentSegment added in v0.85.0 

func AttachmentSegment(name string) (string, error)

AttachmentSegment encodes a display name into a blob-path segment: a fixed "a" prefix, then every byte outside [A-Za-z0-9.-] escaped as "_XX" (hex).

The mapping is injective, which is the whole point -- two different filenames must never address the same bytes. Escaping is prefix-free (a "_" always opens a three-character escape, and a literal "_" escapes to "_5F"), so distinct names give distinct bodies; the constant prefix preserves that and guarantees the leading alphanumeric the segment shape requires. A variable prefix would not: "文" and "x文" would both encode to "x_E6_96_87".

func AttachmentsOf added in v0.79.0 

func AttachmentsOf(grainNQ []byte, workID string) ([]string, error)

AttachmentsOf lists the work's editorial attachment display names, sorted.

func AuthorityGrainDescribes added in v0.52.0 

func AuthorityGrainDescribes(grainNQ []byte, uri string) bool

AuthorityGrainDescribes reports whether the grain carries any statement about uri -- the pre-check a merge runs before asserting a marker, so a namespace-mismatched grain errors instead of gaining a phantom node.

func AuthorityGrainPath 

func AuthorityGrainPath(id string) string

AuthorityGrainPath maps an authority id to its grain path in the repo layout: data/authorities/<xx>/<id>.nq, sharded like Work grains so no directory holds 100k+ entries (ARCHITECTURE §3).

func AuthorityGraph 

func AuthorityGraph(vocab string) rdf.Term

AuthorityGraph returns the named-graph term for a controlled vocabulary's statements: the authority:<vocab> provenance class from ARCHITECTURE §5 (term labels, definitions, hierarchy). Like editorial:, authority graphs are preserved verbatim across feed re-ingest.

func BuildAuthorityGrain 

func BuildAuthorityGrain(oldGrain []byte, t AuthorityTerm, vocab string) ([]byte, error)

BuildAuthorityGrain (re)serializes a local authority term's grain: the term's statements replace the authority:<vocab> graph wholesale (the term owns its grain), preserving any statements other graphs might carry. oldGrain may be nil/empty for a fresh term.

func BuildWorkGrain 

func BuildWorkGrain(wg WorkGroup, provider string) ([]byte, error)

BuildWorkGrain serializes one WorkGroup to its canonical grain bytes -- the per-Work unit BuildWorks writes, exposed for store-backed ingest , where grains land through blob CAS instead of a Sink.

func ClearTombstone 

func ClearTombstone(grainNQ []byte, workID string) ([]byte, error)

ClearTombstone restores a retired Work.

func ClearWithdrawn 

func ClearWithdrawn(grainNQ []byte, workID string) ([]byte, error)

ClearWithdrawn removes a Work's withdrawal flag.

func CloneGrain added in v0.67.0 

func CloneGrain(src []byte, workID string) ([]byte, string, error)

CloneGrain builds a brand-new work grain from a source grain's description: the work and every instance are re-minted, every kept statement lands in the editorial graph (the clone has no feed -- it is hand-authored from here on), identifier / admin- metadata / holdings subgraphs are dropped, lcat curation markers (tags, visibility, merge/split pins, extras) stay with the source, and the clone is born suppressed so it hides from projection until a cataloger finishes and publishes it. Returns the canonical grain and the fresh work id.

func CoverBlobPath added in v0.65.0 

func CoverBlobPath(workID, ext string) string

CoverBlobPath is where a work's uploaded cover bytes live in the blob store. ext is the extension without the dot ("jpg", "png", "webp").

func CoverOf added in v0.95.0 

func CoverOf(grainNQ []byte, workID string) (string, error)

CoverOf returns the work's effective cover URL, or "" when it has none.

The cover is not a profile field, so it never appears in the editor doc's fields map -- the editor's Cover panel read it there and therefore reported "none" for every work it had not itself just uploaded to, leaving no way to take a published cover down. Readers ask for it by name instead.

An editorial statement overlays a feed-carried one, matching SetCover, whose empty url removes the editorial layer and lets the feed value show again.

func DecodeGrainMARC 

func DecodeGrainMARC(grain []byte) ([]*codex.Record, error)

DecodeGrainMARC materializes a grain's MARC records the framework-aware way: editorial lcat:overrides shadow the feed statements they claim (so an edited field decodes to its editorial value, not both), and each record's verbatim sidecar fields are re-attached in tag order. The mapping of records to instance nodes mirrors libcodex Decode's one-record- per-Work+Instance enumeration; if the counts ever disagree, verbatim attachment is skipped rather than misattached.

func DecodeGrainMARCSource added in v0.47.0 

func DecodeGrainMARCSource(grain []byte, org string) ([]*codex.Record, error)

DecodeGrainMARCSource is DecodeGrainMARC plus the cataloging- source derivation: given the deployment's MARC organization code, each decoded record's 040 derives from graph facts AT DECODE TIME -- the field is never stored, so it cannot drift from the named-graph provenance that remains the statement-level source of truth. A record that arrived with a 040 gains the deployment as one trailing $d (modifying agency) when the grain carries editorial statements; a record with no 040 -- the born- digital feeds -- synthesizes 040 $a<org>$c<org> as its own cataloging source. An empty org code decodes unchanged.

func DecodeVerbatimField 

func DecodeVerbatimField(s string) (codex.Field, error)

DecodeVerbatimField parses the sidecar literal form back to a field.

func EditorialGraph 

func EditorialGraph() rdf.Term

EditorialGraph is the named graph for human- and authority-owned statements (ARCHITECTURE §5): merge/split decisions, curated subjects, and any hand- authored triples. Unlike feed:<provider> statements -- regenerated on every ingest -- editorial: statements are preserved verbatim across re-ingest.

func EncodeVerbatimField 

func EncodeVerbatimField(f codex.Field) string

EncodeVerbatimField serializes one MARC field to the sidecar's stable literal form.

func EnrichmentGraph 

func EnrichmentGraph(name string) rdf.Term

EnrichmentGraph returns the named-graph term for a machine-enrichment source's statements. Enrichment graphs are regenerable -- an enricher drop-and-replaces its own graph on each run -- but, not being feed:<provider>, they are preserved across feed re-ingest like editorial statements.

func FeedGraph 

func FeedGraph(provider string) rdf.Term

FeedGraph returns the named-graph term for a provider feed's statements: the feed:<provider> provenance graph from ARCHITECTURE §5. Feed statements are regenerated on ingest and never hand-edited.

func FragInstance added in v0.153.0 

func FragInstance(iri string) string

FragInstance strips the "#" prefix and "Instance" suffix from an Instance node IRI, the inverse of InstanceIRI.

func Grain 

func Grain(rec *codex.Record, graph rdf.Term) ([]byte, error)

Grain crosswalks one record to BIBFRAME and returns its canonical N-Quads grain: every statement tagged with the given provenance graph and RDFC-1.0 canonicalized (blank-node labels + statement order), so an unchanged record re-serializes to identical bytes.

func GrainBlankPrefix added in v0.120.0 

func GrainBlankPrefix(grainPath string) string

grainBlankPrefix namespaces a grain's blank-node labels by its work id. Work ids are `w` plus lowercase base32, so they are already legal blank-node label characters; anything else is folded to `_` rather than emitted raw, since an illegal label would produce a dump no parser accepts.

func GrainFromGraph 

func GrainFromGraph(g *rdf.Graph, graph rdf.Term) ([]byte, error)

GrainFromGraph canonicalizes one BIBFRAME graph into its N-Quads grain, every statement tagged with the given provenance graph and RDFC-1.0 canonicalized so an unchanged input re-serializes to identical bytes.

func GrainLocalIRI added in v0.68.0 

func GrainLocalIRI(v string) bool

GrainLocalIRI reports whether an IRI value is grain-local -- a fragment node the grain itself minted (#<id>Work, #<id>Instance, an editor or clone skolem) rather than a reference into an external vocabulary. The subjects model keys on this: a bf:subject object that is a grain-local node is an uncontrolled heading whose rdfs:label is the value, never a controlled term, exactly like the blank nodes it stands in for.

func GrainPath 

func GrainPath(id string) string

GrainPath returns the sink-relative path for a work id: data/works/<xx>/<id>.nq, sharded by a hash prefix so no directory holds an unbounded number of files (ARCHITECTURE §3). Paths use forward slashes; the Sink maps them onto its backend.

func InstanceIRI 

func InstanceIRI(id string) string

InstanceIRI returns the node IRI libcodex mints for an Instance id.

func ItemFieldNames added in v0.91.0 

func ItemFieldNames() []string

ItemFieldNames lists the batch-editable item fields, sorted, for the error messages and the UI's field picker.

func LegacyAttachmentBlobPath added in v0.85.0 

func LegacyAttachmentBlobPath(workID, name string) string

LegacyAttachmentBlobPath is where an attachment uploaded under the old layout lives: the display name was the segment. Readers fall back to it so the rename of the encoding scheme does not orphan bytes already stored. Empty when name could never have been a legacy segment.

func LocalAuthorityIRI 

func LocalAuthorityIRI(id string) string

LocalAuthorityIRI returns the absolute IRI for a locally minted authority id (identity.Mint with the "a" prefix).

func LossyTag 

func LossyTag(tag string) (string, bool)

LossyTag reports whether a tag is on the known-loss table, with the documented reason -- the editor's non-blocking warning and the sidecar's capture predicate share this one gate.

func MARCRecordNodes 

func MARCRecordNodes(grain []byte) ([]string, error)

MARCRecordNodes exposes the record -> instance-node mapping DecodeGrainMARC uses, so callers (the MARC view) can address a specific record's grain nodes. Node i corresponds to record i of DecodeGrainMARC; a Work-only record maps to its Work node.

func ReadMARC 

func ReadMARC(r io.Reader) ([]*codex.Record, error)

ReadMARC reads every record from an ISO 2709 (.mrc) stream.

func RelabelGrainBlanks added in v0.120.0 

func RelabelGrainBlanks(grain []byte, prefix string) []byte

relabelBlanks rewrites every blank-node label in one grain's N-Quads bytes to prefix+label, leaving the rest of the document byte-for-byte alone.

It scans rather than reparses, so IRIs and literals keep exactly the escaping the grain writer produced -- re-serializing would be a second chance to differ. A blank node can only appear where a term can, so the scan has to know where terms are: `_:c14n0` inside a literal is text, not a node.

func RemoveAuthorityMergeMarker added in v0.196.0 

func RemoveAuthorityMergeMarker(grainNQ []byte, loserURI, winnerURI, vocab string) ([]byte, error)

RemoveAuthorityMergeMarker reverses AddAuthorityMergeMarker: the loser's lcat:mergedInto <winner> statement leaves the authority:<vocab> graph, so the term is live again on reload. Idempotent -- a grain without the marker (an un-merge resuming past the removal) returns unchanged bytes.

func ReplaceGraph 

func ReplaceGraph(grainNQ []byte, graph rdf.Term, quads []rdf.Quad) ([]byte, error)

ReplaceGraph drops every statement in the given named graph and replaces them with quads (whose own G fields are ignored), returning the re-canonicalized grain. This is the idempotent re-enrichment primitive: an enricher owns its enrichment:<name> graph outright.

func ReplaceSubjectReference 

func ReplaceSubjectReference(grainNQ []byte, workID, loserURI string, winner AuthoritySubject, winnerVocab string) ([]byte, error)

ReplaceSubjectReference rewrites one Work's controlled-subject reference after an authority merge: the editorial bf:subject link to the loser is retracted along with every authority-graph statement about the loser the grain carries for the projector's label index, and the winner is appended through the same path suggestion publishing uses. Feed-graph statements are never touched (editorial rewrites do not reach into feed:<provider> graphs); a feed-asserted subject needs the override semantics instead.

func RetiredWorks 

func RetiredWorks(merges []identity.Merge) []string

RetiredWorks returns the sorted, distinct retired Work ids among merges (the From side of every merge). Their grains are removed on ingest once their Instances have moved to the survivor.

func SameAsQuad added in v0.167.0 

func SameAsQuad(workID, externalURI string) rdf.Quad

SameAsQuad links a Work to an external hub resource that identifies the same work: <workURI> owl:sameAs <externalURI>. Minted `w…` ids stay the primary identity -- this is an attached outward link, written into an enrichment:<name> graph, never a re-derivation of the local id.

func ScanMerges 

func ScanMerges(nq []byte) ([]identity.Merge, error)

ScanMerges recovers the editorial merge decisions in one grain's N-Quads: every lcat:mergedInto statement in the editorial graph, as From->To Work-id pairs. A grain with none yields nil.

func ScanMergesDataset 

func ScanMergesDataset(ds *rdf.Dataset) []identity.Merge

ScanMergesDataset is ScanMerges for callers that already hold the parsed dataset (the work index scans everything off one parse).

func ScanPins 

func ScanPins(nq []byte) ([]identity.Pin, error)

ScanPins recovers the editorial split pins in one grain's N-Quads: every lcat:workAssignment statement in the editorial graph, as Instance->Work id pairs. A grain with none yields nil.

func SerializeGrains 

func SerializeGrains(dir string, sink storage.Sink) (int, error)

SerializeGrains regenerates the bulk catalog.nq from the committed per-Work grains under dir, without re-ingesting from a provider cache.

It walks **every** *.nq beneath dir, not just data/works: an installed vocabulary snapshot under data/authorities is merged in too, and on a deployment with LCSH that is 96% of the resulting file. That is deliberate -- the projection needs those quads to label the catalog's subject headings, and a catalog.nq without them ships subject pages whose headings have no labels. Point this at data/works and the build succeeds, quietly.

It skips catalog.nq itself, and re-emits each grain's statements through one shared encoder -- so blank-node labels stay unique across the corpus (ARCHITECTURE §3) rather than colliding as an independently-canonicalized per-grain concatenation would -- in Work-id order. The grains stay the source of truth; catalog.nq is a derived merge of them.

The output represents the same RDF as an ingest-produced catalog.nq (it projects identically) but is not byte-identical to it: ingest serializes each Work's freshly built graph, whereas this merges the on-disk canonical grains, so blank-node labels differ. It returns the number of grains merged.

A grain's bytes appear in the merge unchanged except for its blank-node labels, which are namespaced by grain id. So a grain contributes the same lines whether it is merged alone or with sixty thousand others, and the merged document changes only when a grain does -- which is what lets a published catalog.nq.gz keep its sha256 across a release that changed no data.

func SetAttachment added in v0.79.0 

func SetAttachment(grainNQ []byte, workID, name string, add bool) ([]byte, error)

SetAttachment records (add=true) or retracts (add=false) one attachment statement, guarded on the grain describing the work (the 202/211/214 invariant: a typo'd id must not assert statements into a foreign grain) and on the display name being usable. Adds are idempotent through canonicalization.

func SetCover added in v0.65.0 

func SetCover(grainNQ []byte, workID, url string) ([]byte, error)

SetCover records url as the work's editorial cover (lcat:extra/cover in the editorial graph), replacing any previous editorial cover statement, and returns the re-canonicalized grain. The work must exist in the grain (the no-phantom-ids invariant). An empty url removes the editorial cover, letting any feed-carried value show through again.

func SetFeedKept 

func SetFeedKept(grainNQ []byte, workID string, kept bool) ([]byte, error)

SetFeedKept records (or clears) a curator's keep-despite-withdrawal decision.

func SetItems 

func SetItems(grainNQ []byte, instanceID string, items []Item) ([]byte, error)

SetItems replaces an Instance's holdings wholesale: every editorial item statement under the Instance's item namespace is dropped and the given items re-asserted on freshly numbered skolem nodes. Returns the re-canonicalized grain.

func SetSuppressed 

func SetSuppressed(grainNQ []byte, workID string, suppressed bool) ([]byte, error)

SetSuppressed hides or unhides a Work from projection. Unsuppressing also drops any PredSuppressedBy provenance.

func SetSuppressedBy 

func SetSuppressedBy(grainNQ []byte, workID, actor string) ([]byte, error)

SetSuppressedBy records what set a Work's suppression.

func SetTombstone 

func SetTombstone(grainNQ []byte, workID, redirectTo string) ([]byte, error)

SetTombstone retires a Work: any prior tombstone statement is replaced by one pointing at redirectTo (a Work id) or, with redirectTo empty, the no-successor literal. Returns the re-canonicalized grain.

func SetWithdrawn 

func SetWithdrawn(grainNQ []byte, workID, date string) ([]byte, error)

SetWithdrawn flags a Work as gone from its sole bib feed on the given ISO date; ClearWithdrawn removes the flag.

func SetWorkRelation added in v0.71.0 

func SetWorkRelation(grainNQ []byte, workID, pred, targetID string, add bool) ([]byte, error)

SetWorkRelation adds or removes one editorial relation statement on the grain, guarded on the grain describing the work (the 202/211/214 family: a typo'd id must not assert links into a foreign grain). An add is also refused when the grain already asserts the inverse predicate to the same target, so no caller can write the contradiction "A contains B and A is a part of B"; the handler catches the same pair (and longer cycles) first, this is the backstop that keeps it out of any grain. Adds are idempotent through canonicalization.

func SplitCrossGraphBlanks added in v0.184.0 

func SplitCrossGraphBlanks(grain []byte) ([]byte, int, error)

SplitCrossGraphBlanks repairs fused-blank corruption in one grain: a blank node whose label appears in more than one named graph was almost certainly two unrelated nodes fused by the pre-fix write path (per-graph blank numbering colliding in one file), so each such node is split back into per-graph copies and the grain re-canonicalized. Returns the repaired bytes and how many fused blanks were split; zero means the grain is clean and the original bytes come back untouched.

Splitting is safe for grains libcat wrote: no writer intentionally states one blank across graphs. A legitimately shared node would duplicate -- each graph keeps its own copy of the statements it made, which is exactly what the statements said per graph.

func SplitTargetFor added in v0.140.0 

func SplitTargetFor(grainNQ []byte, instanceIDs []string) (string, error)

SplitTargetFor returns the Work id a prior split already assigned to exactly the given instances in this grain, or "" if there is none. A retried or double-clicked split reuses that id instead of minting a fresh one, so AddSplitMarkers finds every marker already present and no-ops -- the endpoint is idempotent, and the grain never ends up with two workAssignment pins for one instance.

The match is on the exact instance set: a split of [i1] and a later split of [i1, i2] are different operations, not a retry of the first, so the second mints its own Work rather than folding i2 into the first split. Only when the requested set is precisely the set some earlier split already pinned to one Work is that Work reused.

func SubjectQuad 

func SubjectQuad(workID, termURI string) rdf.Quad

SubjectQuad builds the editorial statement linking a Work to a controlled subject's authority URI.

func TagAliasQuad 

func TagAliasQuad(termURI, tag string) rdf.Quad

TagAliasQuad builds the alias statement.

func TagQuad 

func TagQuad(workID, tag string) rdf.Quad

TagQuad builds the editorial statement for one approved folksonomy tag.

func ValidAttachmentName added in v0.79.0 

func ValidAttachmentName(name string) bool

ValidAttachmentName reports whether name is usable as an attachment's display name: non-empty, valid UTF-8, no path separators, not "." or "..", no control characters, and short enough to encode. Every script is allowed; keeping the stored path safe is AttachmentSegment's job, not the name's.

func VerbatimFields 

func VerbatimFields(rec *codex.Record) []string

VerbatimFields serializes a record's known-loss fields (fidelity.go) -- what a MARC ingest provider hands the grain writer for the sidecar.

func WorkID 

func WorkID(rec *codex.Record) (string, error)

WorkID returns a stable, filesystem-safe id for a record's grain, taken from the control number (MARC 001) and falling back to a hash of the record when absent. A record with no 001 whose encoding fails is an error: hashing the nil fallback would give every such record the same id, silently overwriting grains. Phase 0 only: ARCHITECTURE §4's identity model replaces this with a minted, provider-independent id in identity/ (Phase 1), which also changes the grain's subject IRIs and filename.

func WorkIDFromIRI added in v0.92.0 

func WorkIDFromIRI(iri string) (string, bool)

WorkIDFromIRI recovers the Work id a grain-local Work node names. It is what lets a statement about one Work be rebound to another (a batch edit): every other grain-local node -- instances, skolem children -- names something that exists in one grain and nowhere else, so it cannot be rebound at all.

func WorkIRI 

func WorkIRI(id string) string

WorkIRI returns the node IRI libcodex mints for a Work id (the "#<id>Work" fragment the grains use), so editorial overlay statements reference the same node the feed does.

func WriteMergedGrain added in v0.121.2 

func WriteMergedGrain(w io.Writer, workID string, grain []byte) error

WriteMergedGrain writes one grain's contribution to a merged N-Quads document: its own canonical bytes, with its blank-node labels namespaced by work id.

Every writer of a merged catalog.nq goes through here -- the ingest builders and SerializeGrains alike -- so the file means the same thing whichever wrote it last. It did not: ingest re-encoded the grains through one rdf.Encoder and emitted traversal-order `_:b1, _:b2, …` labels, so a build that ran ingest without serialize exported the unstable dump had just fixed . Callers must emit grains in work-id order.

Types

type AgentIdentity added in v0.198.0 

type AgentIdentity struct {
	Authority string
	SameAs    []string
}

AgentIdentity is one contributor's authority identity for graph emission; mirrors ingest.AgentIdentity.

type AuthoritySubject 

type AuthoritySubject struct {
	URI     string
	Labels  map[string]string // language tag -> label (e.g. "en", "es")
	Broader []string          // parent authority URIs (skos:broader)
}

AuthoritySubject is one controlled-vocabulary subject a provider asserts for a Work: a stable authority URI, its human labels by language tag, and its skos:broader parent URIs. It is the graph-emission shape behind ingest.SubjectEnricher.

type AuthorityTerm 

type AuthorityTerm struct {
	URI        string              `json:"uri"`
	PrefLabel  map[string]string   `json:"prefLabel"`
	AltLabel   map[string][]string `json:"altLabel,omitempty"`
	Definition map[string]string   `json:"definition,omitempty"`
	Broader    []string            `json:"broader,omitempty"`
	Narrower   []string            `json:"narrower,omitempty"`
	Related    []string            `json:"related,omitempty"`
	ExactMatch []string            `json:"exactMatch,omitempty"`
	// MergedInto records the term's retirement: it was merged into the
	// referenced term (lcat:mergedInto). Preserved across description edits.
	MergedInto string `json:"mergedInto,omitempty"`
}

AuthorityTerm is the editable description of one authority concept -- the SKOS fields the authority-topic profile exposes. Maps are keyed by language tag ("" = untagged).

func ParseAuthorityGrain 

func ParseAuthorityGrain(grainNQ []byte, uri, vocab string) (AuthorityTerm, error)

ParseAuthorityGrain recovers the term description for uri from an authority grain's authority:<vocab> graph -- the inverse of BuildAuthorityGrain.

func (AuthorityTerm) Quads 

func (t AuthorityTerm) Quads() ([]rdf.Quad, error)

Quads serializes the term description to its authority-graph statements. The graph field of the returned quads is unset; the caller chooses the authority:<vocab> graph.

type BuildStats 

type BuildStats struct {
	Records int // records read from the source
	Grains  int // per-Work grain files written
}

BuildStats reports what a corpus build produced.

func BuildCorpus 

func BuildCorpus(sink storage.Sink, records []*codex.Record, provider string) (BuildStats, error)

BuildCorpus writes one canonical N-Quads grain per record into sink (at GrainPath) in the provider's feed graph, plus a bulk catalog.nq. Because it writes through the Sink, the same build runs against a local directory, cloud object storage, or a git tree unchanged.

catalog.nq is not a byte-concatenation of the grain files: each grain canonicalizes its blanks to _:c14nN independently, so a plain concatenation would merge two grains' _:c14n0 into one node. It is the grains' own bytes with each grain's labels namespaced by work id -- byte-identical to what SerializeGrains produces from the same grains, which is the point: catalog.nq means one thing whichever writer wrote it last. All grains are held in memory for the sorted bulk write; at large scale (ARCHITECTURE §3) that becomes an out-of-core / fan-out concern.

func BuildMARC 

func BuildMARC(sink storage.Sink, marc io.Reader, provider string) (BuildStats, error)

BuildMARC reads an ISO 2709 MARC stream -- e.g. an OverDrive Marketplace MARC Express export -- and builds the corpus into sink.

func BuildWorks 

func BuildWorks(sink storage.Sink, works []WorkGroup, provider string) (BuildStats, error)

BuildWorks writes one canonical N-Quads grain per Work into sink (at GrainPath(WorkID)) in the provider's feed graph, plus a bulk catalog.nq. Each grain carries the shared Work and its Instances via libcodex's WorkInstances, so a clustered Work (multiple editions/formats) is one per-Work file with minted, provider-independent ids at both tiers. A WorkGroup's preserved Editorial statements are merged back in, so a feed re-ingest is clobber-safe (§5). It reports the number of Works (grains) and Instances written.

type GroupInstance 

type GroupInstance struct {
	InstanceID string
	Instance   codexbf.Instance
	// Verbatim carries the record's crosswalk-lossy MARC fields serialized
	// field-exact (EncodeVerbatimField), emitted into the feed graph under
	// PredMARCVerbatim so nothing is silently dropped. Empty for
	// non-MARC providers, leaving the grain unchanged.
	Verbatim []string
}

GroupInstance is one Instance of a WorkGroup: its minted id and Instance-level BIBFRAME.

type Item 

type Item struct {
	ID         string `json:"id"`
	CallNumber string `json:"callNumber,omitempty"`
	Location   string `json:"location,omitempty"`
	Barcode    string `json:"barcode,omitempty"`
	Note       string `json:"note,omitempty"`
}

Item is one holding of an Instance.

func ItemsOf 

func ItemsOf(grainNQ []byte, instanceID string) ([]Item, error)

ItemsOf reads an Instance's holdings from a grain, sorted by node id.

type ItemEdit added in v0.91.0 

type ItemEdit struct {
	Field string
	Value string
	Where *string
}

ItemEdit is one field change applied to every bf:Item in a grain.

Value is the new value; the empty string clears the field. Where, when set, restricts the edit to items whose current value is exactly that string, which is what makes "move Stacks to Annex" leave the Reference copies alone. An item that does not assert the field at all reads as the empty string, so Where of "" means "only the ones missing it".

type Overrides 

type Overrides map[string]map[string]bool

Overrides is the shadow set scanned from a dataset: subject IRI -> overridden predicate IRIs.

func ScanOverrides 

func ScanOverrides(ds *rdf.Dataset) Overrides

ScanOverrides collects the editorial lcat:overrides markers from a parsed dataset.

func (Overrides) Shadows 

func (o Overrides) Shadows(subject, predicate string) bool

Shadows reports whether (subject, predicate) is editorially owned.

type Patch 

type Patch struct {
	Add    []rdf.Quad
	Remove []rdf.Quad
}

Patch is a set of statement additions and removals applied to one provenance graph of a grain. Each quad's own G field is ignored: the target graph is chosen by the applying function. Subjects and objects must be IRIs or literals -- blank nodes are rejected because editorial-class statements must not introduce blank labels that could collide with the feed graph's during joint canonicalization (see grainWithEditorial).

func ItemEditPatch added in v0.91.0 

func ItemEditPatch(grainNQ []byte, e ItemEdit) (Patch, int, error)

ItemEditPatch translates an ItemEdit into an editorial patch over grainNQ, returning the patch and the number of items it touches.

Items already holding the target value are skipped rather than rewritten: an unchanged item has no business appearing in a batch run's diff, and a remove-then-add of an identical quad would report churn as work.

func OverridePatch 

func OverridePatch(subjectIRI string, predicates ...string) Patch

OverridePatch builds the editorial patch claiming ownership of the given predicates on one subject node (apply with ApplyEditorialPatch; pair with the replacement values in the same patch).

func RevertPatch 

func RevertPatch(subjectIRI string, predicates ...string) Patch

RevertPatch builds the editorial patch releasing ownership (the removal half of revert; the caller also removes its editorial replacement values).

type Prior 

type Prior struct {
	Grains    []identity.GrainIdentity
	Editorial map[string][]byte // Work id -> raw N-Quads of its non-feed statements
	Merges    []identity.Merge  // editorial lcat:mergedInto decisions to seed
	Pins      []identity.Pin    // editorial lcat:workAssignment split pins to seed
}

Prior is the committed state a re-ingest recovers from the grains under a build directory: the identity to seed the resolver (so ids do not churn) and the editorial statements to preserve per Work (so a feed re-ingest is clobber-safe). It is the read side of the derive-from-grains model (ARCHITECTURE §4/§5, Decision A): the grains are the durable identity map and the editorial store.

func LoadPrior 

func LoadPrior(dir, provider string) (Prior, error)

LoadPrior reads every per-Work grain (*.nq, skipping the bulk catalog.nq) under dir, returning the recovered identity and the preserved editorial statements keyed by Work id. A missing directory (a first build) yields empty state and no error.

func LoadPriorStore 

func LoadPriorStore(ctx context.Context, st blob.Store, prefix, provider string) (Prior, map[string]string, error)

LoadPriorStore is LoadPrior over a blob.Store: it reads every per-Work grain (*.nq, skipping the bulk catalog.nq) under prefix, returning the recovered Prior plus each grain's ETag keyed by path. The ETags feed conditional writes: a re-ingest that read a grain at etag E writes it back with IfMatch E, so an editorial publish landing mid-ingest surfaces as ErrPreconditionFailed instead of being clobbered (the writer re-reads, re-extracts non-feed graphs, unions, and retries). An empty tree (a first build) yields empty state and no error.

type WorkGroup 

type WorkGroup struct {
	WorkID    string
	Work      codexbf.Work
	Instances []GroupInstance
	// Editorial is the raw N-Quads of the Work's human/authority-owned
	// statements, preserved verbatim from the prior grain so a feed re-ingest
	// never clobbers them (ARCHITECTURE §5). Empty when there are none.
	Editorial []byte
	// Extras are the Work's non-BIBFRAME adopter display fields (e.g. cover,
	// rating, dateRead) that a provider carries through to catalog.json's `extra`
	// object. They are emitted into the feed provenance graph under
	// ExtraPred, so their origin is tracked like every other feed statement. Empty
	// when the provider supplies none, leaving the grain byte-for-byte unchanged.
	Extras map[string]string
	// Subjects are controlled-vocabulary subjects (authority URI + localized labels +
	// skos:broader) a provider derived for the Work, e.g. by promoting genre tags
	// through an authority table. They are emitted into the feed graph as
	// a bf:subject link to the URI plus the authority's prefLabel/broader statements,
	// so the projector resolves them as controlled subjects. Empty
	// when the provider supplies none.
	Subjects []AuthoritySubject
	// Terms are standalone vocabulary term descriptions -- typically the
	// skos:broader ancestor chains of Subjects. They are emitted
	// into the feed graph as the term's prefLabel/broader statements with no
	// bf:subject link, so ancestor concepts stay labeled in the projection's
	// term sideband. Empty when the provider supplies none.
	Terms []AuthoritySubject
	// Agents are contributors' authority identities: each Authority IRI is a
	// bf:agent node the Work's contributions already emit (via
	// codexbf.Contribution.Authority), and its SameAs IRIs are added as
	// owl:sameAs statements on that node -- the shape the summary scan folds
	// into ContributorIDs. Empty when the provider supplies none.
	Agents []AgentIdentity
}

WorkGroup is one clustered Work ready to serialize: its minted id, its shared Work-level BIBFRAME, and the Instances (each with its own minted id) that realize it. It is the direct-BIBFRAME, two-tier-identity unit a native provider produces after resolution (ARCHITECTURE §4).

type WorkRelations added in v0.71.0 

type WorkRelations struct {
	HasPart []string `json:"hasPart"`
	PartOf  []string `json:"partOf"`
}

WorkRelations lists one work's editorial work-to-work links as work ids.

func WorkRelationsOf added in v0.71.0 

func WorkRelationsOf(grainNQ []byte, workID string) (WorkRelations, error)

WorkRelationsOf reads a grain's editorial hasPart/partOf links for the given work, sorted by target id.

type WorkVisibility 

type WorkVisibility struct {
	Tombstoned bool   `json:"tombstoned"`
	RedirectTo string `json:"redirectTo,omitempty"` // successor Work id, when named
	Suppressed bool   `json:"suppressed"`
	// Withdrawn is the date the feed reconciliation flagged this Work as
	// gone from its sole bib feed ("" = not withdrawn).
	Withdrawn string `json:"withdrawn,omitempty"`
	// SuppressedBy names what set the suppression ("" = a curator).
	SuppressedBy string `json:"suppressedBy,omitempty"`
	// Kept records a curator's decision to keep the Work despite withdrawal.
	Kept bool `json:"kept,omitempty"`
}

WorkVisibility is one Work's projection stance.

func Visibility 

func Visibility(grainNQ []byte, workID string) (WorkVisibility, error)

Visibility reads a Work's stance from its grain.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.