Documentation
¶
Overview ¶
Package identity assigns libcat's opaque, two-tier Work/Instance ids (ARCHITECTURE §4). Ids are minted once and never derived from a provider id; on re-ingest a provider record resolves back to its previously minted id via the persisted identity map, so unchanged records keep stable ids -- and therefore stable public URLs.
This file holds the provider-independent primitives: minting opaque ids and computing the clustering key. The persistence and resolve layer (reading the committed map, mint-or-resolve, clobber-safe re-ingest) builds on top.
Index ¶
- Constants
- func Mint(prefix string) string
- func NormalizeKey(s string) string
- func ProviderKey(scheme, value string) string
- func SeedResolver(r *Resolver, grains []GrainIdentity)
- func WorkKey(author, title, lang string) string
- func WorkKeySet(author, title string, langs []string) string
- type Assignment
- type GrainIdentity
- type InstanceIdentity
- type Merge
- type Pin
- type Record
- type Resolver
- func (r *Resolver) Canonical(workID string) string
- func (r *Resolver) Conflicts() []string
- func (r *Resolver) CrossFeedMerges(workMatch map[string]string) []Merge
- func (r *Resolver) Merges() []Merge
- func (r *Resolver) Resolve(rec Record) Assignment
- func (r *Resolver) SeedInstance(instanceID, workID string, providerKeys []string)
- func (r *Resolver) SeedMerge(from, to string)
- func (r *Resolver) SeedPin(instanceID, workID string)
- func (r *Resolver) SeedWorkFeed(workID, feed string)
- func (r *Resolver) SeedWorkKey(clusterKey, workID string)
- func (r *Resolver) SetFeed(feed string)
- func (r *Resolver) TranslationTargets(workKey map[string]string) map[string]string
- func (r *Resolver) WorkForProviderKey(key string) (string, bool)
- type WorkIdentity
Constants ¶
const ( WorkPrefix = "w" InstancePrefix = "i" AuthorityPrefix = "a" )
Id prefixes distinguish the two tiers in a bare id string and in the public URL space (/works/w..., an Instance's i...). AuthorityPrefix marks locally minted authority terms -- not a bibliographic tier, but minted through the same opaque-id discipline so local headings get stable IRIs.
const ( SchemeISBN = "isbn" SchemeISSN = "issn" SchemeASIN = "asin" // Amazon Standard Identification Number (cross-provider edition key) SchemeID = "id" // provider-local identifier (e.g. an OverDrive title/reserve id) )
Provider-key schemes namespace an identifier value so keys from different schemes never collide, and so a key recovered from a grain matches the key ingest builds for the same identifier.
const SourceASIN = "amazon"
SourceASIN is the bf:source label a provider stamps on an ASIN identifier so the scanner recovers it as SchemeASIN (a cross-provider same-edition merge key) rather than an opaque provider-local id. It names the assigning agency, so the mapping is provider-neutral: any feed that carries an Amazon ASIN bridges to any other on the shared asin: key.
Variables ¶
This section is empty.
Functions ¶
func Mint ¶
Mint returns a fresh opaque id with the given tier prefix (WorkPrefix or InstancePrefix), drawn from crypto/rand so it is provider-independent and unguessable. Callers persist the id on first mint and never regenerate it; use a resolver to avoid re-minting for an already-mapped record.
func NormalizeKey ¶
NormalizeKey folds a string to its clustering-key form: Unicode-lowercased, with every run of non-alphanumeric characters collapsed to a single space and the result trimmed. It deliberately does not fold diacritics -- that needs a Unicode-normalization dependency this framework avoids -- so accent and transliteration variants under-merge and are corrected in the editorial overlay rather than guessed at here.
func ProviderKey ¶
ProviderKey namespaces an identifier value by scheme, e.g. ProviderKey(SchemeISBN, "978...") -> "isbn:978...".
func SeedResolver ¶
func SeedResolver(r *Resolver, grains []GrainIdentity)
SeedResolver seeds r with the committed identity from scanned grains, so a subsequent Resolve reuses existing ids instead of re-minting.
func WorkKey ¶
WorkKey builds the computed clustering key from the primary author, the title, and one language -- the single-language form of WorkKeySet, kept for callers that carry exactly one language code (e.g. the LC Hub enricher). It equals WorkKeySet(author, title, []string{lang}).
func WorkKeySet ¶ added in v0.234.0
WorkKeySet builds the computed clustering key from the primary author, the title, and the work's full language set -- the MARC 1XX+240 access-point key (ARCHITECTURE §4), language-specific so translations stay distinct Works (BIBFRAME one-Work-per-language). Two Instances with the same key cluster into one Work unless an editorial merge/split decision says otherwise. The title is the main title only (not the subtitle), so editions that vary a subtitle still cluster.
The languages are lowercased, trimmed, deduped, and sorted, so the key is order-independent: a Work a source flattened to several languages keys on the stable set rather than an arbitrary first language, and a genuinely multilingual item (facing-page bilingual) keeps its own multi-language identity instead of colliding with a single-language sibling. A single language yields the same string as the old single-field key.
A record with no main title has no usable access point: clustering title-less records by author (or by nothing) would merge unrelated books, so WorkKeySet returns "" and the caller must mint instead of clustering.
Types ¶
type Assignment ¶
Assignment is the resolved identity for a record: its stable Instance and Work ids, and whether either was freshly minted this ingest (vs resolved to an already-committed id).
type GrainIdentity ¶
type GrainIdentity struct {
Works []WorkIdentity
Instances []InstanceIdentity
}
GrainIdentity is the identity recovered from one grain -- the Work(s) it carries and their Instances. The derive-from-grains model (Decision A, ): the committed grains are themselves the identity map.
func ScanDataset ¶
func ScanDataset(ds *rdf.Dataset) GrainIdentity
ScanDataset is ScanGrain for callers that already hold the parsed dataset (the work index scans identity, summaries, and barcodes off one parse).
func ScanGrain ¶
func ScanGrain(nq []byte) (GrainIdentity, error)
ScanGrain recovers the identities committed in one grain's N-Quads. Node ids come from the #<id>Work / #<id>Instance IRIs; a Work's clustering key is recomputed from its primary author, main title, and language; provider keys come from each bf:identifiedBy value, namespaced by its identifier type. It reads every named graph, since a grain's feed and editorial lines share the file. Only minted fragment IRIs count: the crosswalk types 76X-78X related-resource stubs as bf:Work/bf:Instance too (blank or external nodes), and seeding their keys or identifiers would let an unrelated incoming record resolve onto a stub label instead of minting.
type InstanceIdentity ¶
InstanceIdentity is the committed identity of one Instance recovered from a grain: its minted id, the Work it belongs to, and the provider keys it answers to.
type Merge ¶
Merge is an editorial merge decision recovered from the grains (ARCHITECTURE §4): every reference to From resolves to To. It is the under-merge fix -- two records that should be one Work but clustered apart -- and the computed key cannot undo it. Chains collapse to a single canonical id.
type Pin ¶
Pin is an editorial split decision recovered from the grains (ARCHITECTURE §4): Instance is assigned to Work regardless of the computed clustering key. It is the over-merge fix -- records the key wrongly clustered together are pinned apart -- and makes the split reproducible across re-ingest.
type Record ¶
type Record struct {
// ProviderKeys are resolution keys in priority order, each namespaced so keys
// from different schemes never collide, e.g. "overdrive:3389970" then
// "isbn:9781682308233". The first key that already maps wins; ISBN is the
// cross-provider merge key (ARCHITECTURE §9).
ProviderKeys []string
Author string
Title string
// Langs is the work's full language set. The cluster key sorts and dedupes
// it, so translations stay distinct Works (one Work per language) while a
// flattened multi-language node keys on a stable set, not an arbitrary first
// language.
Langs []string
// MatchKey is the un-namespaced cross-feed clustering key a feed that
// namespaces its intra-feed key (so its own records never re-merge) sets to
// opt this record into the cross-feed dedup: the raw author+title+language-set
// key, computed with WorkKeySet. Empty for a feed whose cluster key is
// already un-namespaced -- it clusters cross-feed on the cluster key directly.
MatchKey string
}
Record is the identity-relevant projection of one incoming provider record: the keys it can be resolved by and the fields of its computed clustering key.
type Resolver ¶
type Resolver struct {
// contains filtered or unexported fields
}
Resolver assigns stable Work/Instance ids across ingests. Seed it with the identity already committed (from the grains), then Resolve each incoming record to an existing id or a freshly minted one. It is the mint-or-resolve core of ARCHITECTURE §4 / an unchanged record keeps its previously minted ids, so its grain -- and its public URL -- do not churn.
A Resolver is not safe for concurrent use; ingest is single-threaded per corpus.
func NewResolver ¶
func NewResolver() *Resolver
NewResolver returns an empty resolver. Seed it from the committed identity before resolving new records.
func (*Resolver) Canonical ¶ added in v0.234.0
Canonical returns the surviving Work id for a (possibly merged-away) Work, following the merge overlay -- the id an ingest run must key a bridged Work's grain and presence entry by.
func (*Resolver) Conflicts ¶
Conflicts returns provider-key collisions seen during resolution ( §4), for the caller to surface. Nil when there were none.
func (*Resolver) CrossFeedMerges ¶ added in v0.234.0
CrossFeedMerges computes the cross-feed dedup: each of this run's freshly minted Works whose un-namespaced match key resolves to exactly one prior Work from a DIFFERENT feed is merged onto that prior Work. This lets a feed that namespaces its intra-feed key (so its own records never re-merge) still deduplicate against the other providers -- the same title arriving from OverDrive and coll clusters into one Work. It refuses to bridge an ambiguous key rather than guess: two of this run's Works claiming one match key (the namespaced feed's own genuine collision), or two distinct prior Works from other feeds answering to it. workMatch maps each candidate Work id to its match key; a "" match key is skipped.
func (*Resolver) Merges ¶ added in v0.173.1
Merges returns every merge the resolver is applying -- editorial merges seeded from prior grains and feed cluster-merges alike -- as From->To pairs in deterministic order. The retirement pass diffs on this so a Work folded away by a feed's isReplacedBy has its stale grain removed, not just its records re-homed: without it the retired cluster's grain lingers and keeps duplicating the survivor's identifiers.
func (*Resolver) Resolve ¶
func (r *Resolver) Resolve(rec Record) Assignment
Resolve returns the stable identity for a record, minting only what is genuinely new. An Instance resolves by its first already-known provider key. Its Work is an editorial pin if one exists (an over-merge split, applied first), else the existing instance->work link, else the computed cluster key, else a freshly minted Work. Editorial merges are applied last, so a pinned or clustered Work still follows a later merge to its survivor.
func (*Resolver) SeedInstance ¶
SeedInstance records a previously minted Instance: its id, the provider keys it answers to, and the Work it belongs to. Called once per committed Instance before ingest so re-ingest resolves rather than re-mints.
func (*Resolver) SeedMerge ¶
SeedMerge records an editorial merge: every reference to from resolves to to. Merges override the computed key, so a re-ingest cannot undo a human decision. Chains collapse to a single canonical id.
func (*Resolver) SeedPin ¶
SeedPin records an editorial split pin: instanceID is assigned to workID regardless of the computed clustering key, so an over-merge the key would otherwise recreate stays split across re-ingest. The pinned Work id is reserved so it is never minted for anything else.
Two pins for one instance is a data-integrity event (a split written twice before made the endpoint idempotent, or a hand-edited grain). The first pin seen wins deterministically -- pins arrive in canonical quad order, so the choice no longer depends on which random Work id sorts higher -- and the second is reported rather than silently dropped. The discarded id is not reserved: it denotes nothing, and reserving it would burn an id out of the space for good.
func (*Resolver) SeedWorkFeed ¶ added in v0.234.0
SeedWorkFeed tags a committed Work with a feed it was recovered from (empty feeds are ignored). A multi-feed Work is tagged once per feed, so the cross-feed dedup can exclude bridge targets a namespaced feed already contributes to.
func (*Resolver) SeedWorkKey ¶
SeedWorkKey records the computed clustering key of a committed Work, so a new record with the same key clusters onto it. The caller recomputes the key from the Work's data with WorkKeySet. The key->works multiplicity is tracked so the cross-feed dedup can tell an unambiguous bridge target from a key several distinct prior works share.
func (*Resolver) SetFeed ¶ added in v0.234.0
SetFeed records the feed of the run currently resolving, so CrossFeedMerges never bridges one of this feed's records onto a prior Work from the same feed.
func (*Resolver) TranslationTargets ¶ added in v0.234.0
TranslationTargets computes the bf:translationOf links for this run's Works. Works that share an author+title base but carry different language sets are language siblings (BIBFRAME one-Work-per-language: a translation is a distinct Work). Each such sibling links to its group's representative -- the Work under the lexicographically smallest language set (tie-broken by id), so the whole language cluster points at one primary expression (English sorts ahead of most translations). workKey maps each of this run's Work ids to its full cluster key (author+title+language-set); prior Works from the committed grains are folded in as sibling candidates. A Work that is its group's representative, is language-less, or has no differing-language sibling gets no entry.
The links are recomputed each ingest, so as a feed re-ingests they converge; an edge can lag until then when a later ingest introduces a new representative.
func (*Resolver) WorkForProviderKey ¶ added in v0.171.0
WorkForProviderKey returns the Work id a provider key resolves to when the resolver already knows it (seeded from a prior grain), following merges -- and nothing (false) otherwise. It has no side effects, unlike Resolve. Used to translate a feed's cluster-merge statement, which names records by provider id, into the Work-id space SeedMerge operates on.
When the exact key is not indexed, it falls back to the cluster's format buckets: a single-format cluster's grain indexes only the format-suffixed instance key (e.g. "id:coll:51812:ebook"), never the bare cluster key ("id:coll:51812") a feed merge names, so the exact lookup misses and the merge is skipped. Every instance of a cluster resolves to the same Work, so the bare key is resolved through any indexed key that extends it with a ":<suffix>". The trailing colon keeps "id:coll:5181" from matching "id:coll:51812:..."; agreement across all matching buckets is required, so a cluster already split across Works resolves to nothing rather than guessing.
type WorkIdentity ¶
type WorkIdentity struct {
WorkID string
ClusterKey string
// Feed is the provenance feed this Work occurrence was recovered from (the
// local name of its feed:<name> graph), or "" for a non-feed graph. A
// multi-feed Work yields one WorkIdentity per feed graph, so the cross-feed
// dedup can tell which feeds already contribute a Work under a cluster key
// and never bridge a namespaced feed's record onto its own prior work.
Feed string
}
WorkIdentity is the committed identity of one Work recovered from a grain: its minted id and its recomputed clustering key, so a new record with the same key clusters onto it rather than minting a fresh Work.
Source Files