ingest

package
v0.229.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Facet dimensions over a WorkSummary.

These live next to WorkSummary rather than in the HTTP layer because two callers must agree on them exactly: the works listing, which counts and filters facets for the rail, and a batch/export selection, which resolves the same filters into a set of Works. When they drifted, "Export these results…" offered the entire catalog. One definition, two callers.

Package ingest defines the compile-time provider model (ARCHITECTURE §9a): a bibliographic source plugs in as a Go type satisfying Provider, is registered in a Registry, and runs through the shared identity + clustering pipeline (Run). The provider set is finite, first-party or deployment-authored, and a deployment builds its own lcat, so binding is static -- no dynamic loading. The interface is shaped so a future subprocess or WASM transport can implement it unchanged.

Index

Constants

View Source
const (
	FacetVisibility = "visibility"
	FacetHoldings   = "holdings"
	FacetNeeds      = "needs"
	FacetSubject    = "subject"
	FacetTag        = "tag"
)

Facet group names. Anything else is an extras key.

View Source
const (
	// ReconcileReview flags the Work (lcat:withdrawnFromFeed) for the admin
	// review queue and changes nothing else.
	ReconcileReview = "review"
	// ReconcileAutoSuppress flags and suppresses in one pass, recording the
	// pass as the suppressor so a returning title un-suppresses itself.
	ReconcileAutoSuppress = "auto-suppress"
)

Reconciliation policies: what happens to a Work whose sole bib feed stopped listing it.

View Source
const DefaultRequestTimeout = 90 * time.Second

DefaultRequestTimeout bounds a single outbound harvest request. A peer host that stalls mid-response (a slow-loris, a dead connection that never resets) would otherwise hang the request forever -- and with it the whole queued job, which then never flushes its results and cannot be reaped while a sibling drain is still joined. Providers give their default HTTP client this timeout so a stuck request fails and the term advances (or retries) instead of freezing the run.

View Source
const RejectAbortAfter = 25

RejectAbortAfter is how many CONSECUTIVE failures of ANY class a peer harvest tolerates, while it has produced zero candidates, before failing the run fast. Unlike UnreachableAbortAfter this counts HTTP rejections (403/404) too: a wildcard-DNS host reached with a bad siteCode connects fine and so never trips the connection guard, yet rejects every term. One term the peer answers -- even a clean no-match -- resets the streak, so a healthy but sparse peer never trips it, and one produced candidate disarms the guard for the rest of the run.

View Source
const UnreachableAbortAfter = 25

UnreachableAbortAfter is how many CONSECUTIVE connection-class failures a peer harvest tolerates before failing the run fast. A per-term miss (a 404, no concept in a region) is normal and resets the count; only transport failures accumulate, so a healthy host with sparse coverage never trips it.

Variables

View Source
var ErrEnricher = errors.New("enricher failed")

ErrEnricher marks a failure inside the enricher itself -- typically the external service behind it (rate limit, timeout, DNS) -- as opposed to a storage read/write failure around it. Callers use errors.Is to tell a retryable upstream condition from an internal fault.

View Source
var ErrPeerRejected = fmt.Errorf("%w: peer rejected every request", ErrEnricher)

ErrPeerRejected marks a peer harvest that gave up because the host answered every request with an error while producing no candidates -- the signature of a wildcard-DNS host reached with a mistyped tenant/siteCode. Such a host resolves and reaches a live API, which then rejects each request (HTTP 403/404) rather than failing to connect, so the connection guard never trips. It wraps ErrEnricher like ErrPeerUnreachable, and its message names the rejecting status and the host so the misconfigured entry is identified.

View Source
var ErrPeerUnreachable = fmt.Errorf("%w: peer unreachable", ErrEnricher)

ErrPeerUnreachable marks a peer harvest that gave up because its host is not reachable: too many consecutive connection-class failures (DNS, refused, timeout). It wraps ErrEnricher so existing upstream-failure checks still match, and its message names the host so a mistyped entry in a multi-host list is identified. The circuit-break exists because a per- term timeout bounds ONE request, not a run in which every request fails -- a typo'd host would otherwise grind every driver term (hours) to no end.

Functions

func CleanText

func CleanText(s string) string

CleanText normalizes a vendor free-text field: it resolves HTML character references, drops markup tags, and collapses the whitespace runs that stripping leaves. Text carrying neither "&" nor "<" passes through untouched, so a clean value (and a bare "<" as prose, e.g. "2 < 3") is byte-identical.

References decode to a bounded fixpoint (three passes): vendor feeds double-escape (&amp;#8212;), so a single pass would only peel one layer. The providers apply this to prose and transcribed titles at ingest, so grains store clean text and every downstream projection and export inherits it . Headings and identifiers are deliberately left untouched.

func FacetValues added in v0.116.0

func FacetValues(s WorkSummary, group string) []string

FacetValues returns a summary's values in one facet group. An unrecognized group name is read as an extras key, whose comma-joined value is split -- the lcat convention for multi-valued extras.

func FoldsCase added in v0.116.0

func FoldsCase(group string) bool

FoldsCase reports whether a group's values match case-insensitively. Tags are free text a cataloger typed; the controlled dimensions are not.

func Holdings added in v0.116.0

func Holdings(s WorkSummary) []string

Holdings lists the holdings signals a summary carries (multi-valued).

func IsUnreachable added in v0.226.1

func IsUnreachable(err error) bool

IsUnreachable reports whether an error is connection-class -- the host does not resolve, refuses the connection, is unroutable, or times out -- as opposed to a content/parse failure or an HTTP status. These are the failures that mean "this host is misconfigured", the ones a circuit-break should count.

func MatchExtras added in v0.195.0

func MatchExtras(filters [][2]string, extras map[string]string) bool

MatchExtras reports whether a summary's extras satisfy every [key, value] filter term, ANDed; a comma-joined extra (the sources convention) matches on any element. This is the one scoping semantic shared by the audit endpoints, the CLI's --filter, and enrichment runs.

func MatchesFacets added in v0.116.0

func MatchesFacets(s WorkSummary, filters map[string][]string) bool

MatchesFacets reports whether a summary passes every group's filter: AND across groups, OR within one. This is the rule the facet rail draws and the rule an export selection must resolve by, or the count beside the button and the count in the file disagree.

func MatchesGroup added in v0.116.0

func MatchesGroup(s WorkSummary, group string, selected []string) bool

MatchesGroup reports whether a summary passes one group's filter. An empty selection always passes: an unselected facet means "any", not "none".

func Needs added in v0.116.0

func Needs(s WorkSummary) []string

Needs lists a summary's completeness gaps (multi-valued): the triage slices catalogers work through.

func RunEnrich

func RunEnrich(ctx context.Context, st blob.Store, prefix string, e Enricher) (int, error)

RunEnrich executes an enricher in direct (auto-approve) mode over every grain under prefix in the store: each returned Work's enrichment:<name> graph is dropped and replaced with the fresh assertions, so a re-run is idempotent, and returning an Enrichment with no Subjects explicitly clears a Work's statements from this source. Works absent from the result are left untouched. Returns the number of Works written.

func RunEnrichScoped added in v0.180.0

func RunEnrichScoped(ctx context.Context, st blob.Store, prefix string, e Enricher, keep func(*WorkSummary) bool) (int, error)

RunEnrichScoped is RunEnrich over a sub-collection: only summaries keep accepts are handed to the enricher, so an external-service source (the wikidata resolver) queries for exactly the scoped works and only their grains gain statements. Out-of-scope Works on shared grains keep their existing statements (withPreservedEnrichment). A nil keep enriches everything.

func SimilarWorks added in v0.118.0

func SimilarWorks(summaries []WorkSummary) []similar.Work

SimilarWorks converts a batch, preserving order.

func SplitExtra added in v0.116.0

func SplitExtra(v string) []string

SplitExtra splits a comma-joined extras value into trimmed facet values. A single-valued extra passes through unchanged.

func Visibility added in v0.116.0

func Visibility(s WorkSummary) string

Visibility buckets a summary into exactly one value: what the public projection does with it. A withdrawn-but-kept Work is public by the curator's decision.

Types

type AgentIdentifier added in v0.198.0

type AgentIdentifier interface {
	AgentIdentities() []AgentIdentity
}

AgentIdentifier is an optional capability a Record may implement to carry its contributors' authority identities (a feed whose creators are agent nodes with owl:sameAs links to VIAF/LCNAF/ISNI/wikidata). For a clustered Work the first record's identities win, matching ControlledSubjects.

type AgentIdentity added in v0.198.0

type AgentIdentity struct {
	Authority string
	SameAs    []string
}

AgentIdentity is one contributor's authority identity: the IRI the contribution's bf:agent node is emitted as, plus the agent's other authority IRIs, which ride as owl:sameAs on that node. Together they are what the summary scan folds into ContributorIDs -- the creator-resolution keys the demographics enricher hops from.

type Attribution added in v0.214.0

type Attribution struct {
	Source string
	// Basis names the match kind ("isbn", "title+author").
	Basis string
	// Key is the value that matched: the shared ISBN, or the peer's
	// title/author strings.
	Key string `json:",omitempty"`
	// Ref is a verifiable link (the peer OPAC's record page).
	Ref string `json:",omitempty"`
}

Attribution is one source's evidence behind an endorsement: which peer, what matched (the basis), the matching key, and a URL a moderator can open to eyeball the peer's actual record.

type AuthoritySubject

type AuthoritySubject = bibframe.AuthoritySubject

AuthoritySubject is one controlled-vocabulary subject a provider asserts for a Work (authority URI + localized labels + skos:broader parents). It is an alias of the bibframe emission type, defined in bibframe to avoid an ingest<-bibframe import cycle while keeping the capability's shape visible here.

type Breaker added in v0.226.3

type Breaker struct {
	// contains filtered or unexported fields
}

Breaker bounds a peer harvest that is failing wholesale. It carries two streaks over a run's driver terms: consecutive connection-class failures (ErrPeerUnreachable) and consecutive failures of any class before the peer has produced a candidate (ErrPeerRejected). The reset rule is the fix for a subtle grind: the streak resets on a term the peer ANSWERS, not on "an error of a different class" -- so a host that 404s every request (never a connection error) still trips instead of running every term to no end. A zero Breaker is not usable; construct with NewBreaker so the host name reaches the abort message.

func NewBreaker added in v0.226.3

func NewBreaker(host string) *Breaker

NewBreaker returns a Breaker whose abort messages name host.

func (*Breaker) Candidate added in v0.226.3

func (b *Breaker) Candidate()

Candidate notes the peer has produced real output, disarming the reject guard for the rest of the run: a peer that works cannot be "rejecting every request", so a later failure streak reports as unreachable (if it is) or is simply tolerated.

func (*Breaker) Fail added in v0.226.3

func (b *Breaker) Fail(err error) error

Fail records one failed driver term and returns a non-nil abort error once a guard trips: ErrPeerUnreachable after UnreachableAbortAfter consecutive connection-class failures, or ErrPeerRejected after RejectAbortAfter consecutive failures of any class with no candidate ever produced. It returns nil while both streaks are below their thresholds, so the caller skips the term and continues.

func (*Breaker) Ok added in v0.226.3

func (b *Breaker) Ok()

Ok records a driver term the peer answered -- a match or a clean no-match -- resetting both failure streaks so a healthy but sparse peer never trips.

type Config

type Config struct {
	Feed   string
	Source string
	Params map[string]string
}

Config carries a provider's build-time configuration into its Factory. Feed overrides the provenance graph name (default: the registry key); Source is the primary input (a cache directory, a file, a URL); Params holds provider-specific extras so the registry stays uniform.

type CreatorClaim added in v0.177.0

type CreatorClaim struct {
	QID        string
	Label      string
	Claims     []DemographicClaim
	MatchedVia string
	Retrieved  string
}

CreatorClaim is one resolved creator identity: the knowledge-base entity id (a Wikidata QID), its display label, the explicitly-stated demographic claims it carries, and the resolution provenance -- which cataloged identifier matched it and when it was retrieved. Claims are only ever copied from explicit statements; nothing here is inferred.

func (*CreatorClaim) AddClaim added in v0.177.0

func (c *CreatorClaim) AddClaim(d DemographicClaim)

AddClaim appends a claim, dropping exact duplicates (a SPARQL join yields one row per label variant).

type CreatorSummary added in v0.178.0

type CreatorSummary struct {
	QID    string
	Claims []DemographicClaim `json:",omitempty"`
}

CreatorSummary is a resolved creator as the work index carries it: the entity id and its cached claims, with no display label -- the creator audit aggregates distributions and never lists people.

type DemographicClaim added in v0.177.0

type DemographicClaim struct {
	Property   string
	ValueQID   string
	ValueLabel string
}

DemographicClaim is one explicitly-stated claim on a creator entity: the property id (P21, P27, P91, P172), the value's entity id, and its label.

type Endorsement added in v0.212.0

type Endorsement struct {
	Count   int
	Sources []string
	// Attributions, when present, carry each source's verifiable claim.
	Attributions []Attribution `json:",omitempty"`
}

Endorsement is one subject's peer-consensus record: how many independent sources asserted it, and which.

type EnrichStats added in v0.185.0

type EnrichStats struct {
	// Batches is how many external queries ran; SkippedBatches how many were
	// abandoned after retries (their works untouched; a re-run backfills).
	Batches        int `json:"batches"`
	SkippedBatches int `json:"skippedBatches,omitempty"`
	// Total, when a source can size its run up front (the bibliocommons
	// harvest drives one search per driver term), is the count Batches
	// progresses toward, in the source's own unit. 0 means the run sizes
	// lazily and progress is indeterminate.
	Total int `json:"total,omitempty"`
	// ResolvedCreators / Claims count what the run brought back.
	ResolvedCreators int `json:"resolvedCreators,omitempty"`
	Claims           int `json:"claims,omitempty"`
	// Candidates counts subject candidates found SO FAR, live -- honest
	// label: queue-time dedup and tombstone skips happen after the run,
	// so the final queued tally can be lower.
	Candidates int `json:"candidates,omitempty"`
	// ElapsedMS is the wall time inside the enricher's Enrich calls.
	ElapsedMS int64 `json:"elapsedMs"`
}

EnrichStats describes one enrichment run's external-service work, for callers that surface progress (the run endpoint's result, logs). Zero for enrichers that do no batched external calls.

type Enricher

type Enricher interface {
	Name() string
	Enrich(ctx context.Context, works []WorkSummary) ([]Enrichment, error)
}

Enricher produces enrichments for batches of Works. This executes the RoleEnrich half of the provider model that Run reserves: enrichers never touch feed graphs -- their statements land in their own enrichment:<name> graph (direct mode) or in the moderation queue (a deployment decision made by the caller, not the enricher).

type Enrichment

type Enrichment struct {
	WorkID   string
	Subjects []AuthoritySubject
	// Identities are outward links to external hubs, rendered as
	// owl:sameAs into the enrichment graph alongside any subjects.
	Identities []ExternalIdentity
	// Terms carries standalone term descriptions -- typically the
	// skos:broader ancestor chains of Subjects -- asserted into the
	// enrichment graph as labels + hierarchy only, with no link to the Work
	//. They keep ancestor nodes labeled in projections that roll
	// subtree data up the hierarchy even when no work carries the ancestor
	// directly.
	Terms []AuthoritySubject
	// Creators are the Work's creators resolved to external knowledge-base
	// entities carrying explicitly-stated demographic claims (the
	// creator-demographics audit axis; ingest/wikidata). Rendered into the
	// enrichment graph as a work->entity link plus the entity's claims and
	// resolution provenance.
	Creators []CreatorClaim
	// Confidence (0-1] qualifies queue-moderated enrichments; direct-mode
	// callers may threshold on it.
	Confidence float64
	// Origins, when present, parallels Subjects: Origins[i] names what
	// produced Subjects[i] in the source's own terms -- the uncontrolled
	// tag a reconciler matched, the work subject a crosswalk pivoted from,
	// the harvested heading -- so a queue row can say "from X".
	Origins []string `json:",omitempty"`
	// Endorsements, when present, parallels Subjects: Endorsements[i]
	// records the peer sources corroborating Subjects[i], so queue mode
	// can rank cross-source consensus above singletons.
	Endorsements []Endorsement `json:",omitempty"`
}

Enrichment is one Work's enrichment result: controlled subjects to assert.

type ExternalIdentity added in v0.167.0

type ExternalIdentity struct {
	URI    string
	Scheme string
}

ExternalIdentity is an outward link to a hub resource that identifies the same Work in another system: the hub URI and its short scheme ("openlibrary", "wikidata", "lchub"). Rendered as owl:sameAs; the minted `w…` id stays primary.

type ExtraProvider

type ExtraProvider interface {
	Extras() map[string]string
}

ExtraProvider is an optional capability a Record may implement to carry per-Work display fields that are not part of BIBFRAME -- e.g. a cover URL, a personal rating, or a read date. Run writes them into the Work's feed provenance graph under bibframe.ExtraPred, and the projector surfaces them as catalog.json's `extra` object , which the Hugo module forwards to page params. A Record that does not implement it carries no extras, leaving the graph unchanged. For a clustered Work the first record's extras win, matching how shared Work metadata is taken.

type Factory

type Factory func(Config) (Provider, error)

Factory constructs a configured Provider from a Config. A deployment registers one per provider type, so adding a source is a Register call plus the provider's own package -- no libcat fork.

type MergeSeed added in v0.171.0

type MergeSeed struct {
	FromKey string
	ToKey   string
}

MergeSeed names one feed cluster-merge as a pair of provider keys the resolver indexes records under (identity.ProviderKey values in the provider's id scheme): FromKey is the retired cluster, ToKey the survivor.

type MergeSeeder added in v0.171.0

type MergeSeeder interface {
	MergeSeeds() []MergeSeed
}

MergeSeeder is an optional Provider capability declaring cluster-merge provenance from a feed: a source that folds one cluster into another -- a coll feed's dcterms:isReplacedBy, say -- reports each merge here, and Run, after seeding the resolver from the prior grains, merges the retired Work onto the survivor. Without it a re-clustered record whose two prior clusters folded resolves ambiguously (to whichever prior key the resolver hits first) and orphans the other prior grain -- a false withdrawal. A merge whose ids the resolver does not know (no prior grain) is skipped, so a first-time feed is unaffected.

type Provider

type Provider interface {
	Name() string
	Role() Role
	Records(ctx context.Context) ([]Record, error)
}

Provider is one bibliographic source bound at compile time. Name is the provenance feed graph the provider's statements belong to (feed:<name>); Role fixes the graph class; Records yields the source's resolvable records for the shared pipeline. A network-backed provider honors ctx for cancellation.

type ReconcileResult

type ReconcileResult struct {
	// Flagged Works were newly marked withdrawn this pass.
	Flagged int
	// Suppressed Works were auto-suppressed this pass.
	Suppressed int
	// Cleared Works returned to the feed and lost their withdrawal flag.
	Cleared int
	// Unsuppressed Works returned and lost a reconcile-set suppression.
	Unsuppressed int
	// FlaggedIDs lists the newly withdrawn Works, sorted.
	FlaggedIDs []string
}

ReconcileResult counts what one pass did.

func Reconcile

func Reconcile(ctx context.Context, st blob.Store, prefix, feed string, present map[string]bool, policy, date string) (ReconcileResult, error)

Reconcile diffs the corpus against the Works a feed's latest scan resolved to (present) and flags the leavers: a Work is withdrawn only when this feed is its sole bib source and no curator has invested editorial statements in it -- items, tags, merges, or a keep decision all protect it. Returning Works lose the flag (and a reconcile-set suppression), so a title cycling out of and back into a collection round-trips with identity and editorial statements intact. Nothing is ever deleted.

type Record

type Record interface {
	Identity() identity.Record
	Work() codexbf.Work
	Instance() codexbf.Instance
}

Record is one resolvable bibliographic record: the keys that assign it stable two-tier ids (Identity) and the Work/Instance BIBFRAME it contributes once clustered. Any source that produces these three is an ingest Record -- an OverDrive item already is one -- so the pipeline is provider-agnostic.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry maps a provider type key to its Factory. Registration is explicit (in main, not init), so the built-in set is auditable and a deployment composes its own set.

func NewRegistry

func NewRegistry() *Registry

NewRegistry returns an empty Registry.

func (*Registry) Names

func (r *Registry) Names() []string

Names returns the registered provider keys in sorted order.

func (*Registry) New

func (r *Registry) New(name string, cfg Config) (Provider, error)

New constructs the registered provider for name with cfg, erroring if the key is unknown. When cfg.Feed is empty it defaults to name, so the registry key doubles as the provenance graph unless a deployment overrides it.

func (*Registry) Register

func (r *Registry) Register(name string, f Factory) error

Register binds a provider type key to its Factory. It errors on an empty key, a nil factory, or a duplicate key, so a composition mistake fails at startup.

type Result

type Result struct {
	Stats           bibframe.BuildStats
	MintedWorks     int
	MintedInstances int
	Retired         int
	Conflicts       []string
	// WorkIDs are the Works this run's records resolved to -- the presence
	// set the feed reconciliation pass diffs the corpus against.
	WorkIDs []string
}

Result is what one ingest run produced: the grain/instance counts, how many ids were freshly minted at each tier, how many merge-retired grains were dropped, and any resolver conflicts to surface. The pipeline returns this rather than printing so the CLI (or a cloud handler) owns presentation.

func Run

func Run(prov Provider, out string) (Result, error)

Run ingests a provider's records into canonical grains under out, the shared direct-BIBFRAME pipeline every ingest provider uses (ARCHITECTURE §4/§9). It seeds the resolver from any grains already under out (so re-ingest reuses ids and clusters editions), applies committed editorial merges/pins, resolves each record to stable two-tier ids, groups records into Works (first record wins shared metadata; a duplicate Instance -- e.g. a shared ISBN -- is emitted once), carries each Work's preserved editorial statements across the feed rewrite, writes one grain per Work plus catalog.nq, and drops the grains of merge-retired Works. The run is deterministic in record order. Only ingest-role providers are executed.

func RunStore

func RunStore(ctx context.Context, prov Provider, st blob.Store, prefix string) (Result, []string, error)

RunStore is Run over a blob.Store: the same prior-load / resolve / cluster / build pipeline, with grains written under ETag optimistic concurrency so a concurrent editorial save is never clobbered -- the copy-cataloging commit path. Unchanged grains are not rewritten, so re-committing identical records is byte-stable and touches nothing. It does not write catalog.nq: store-backed deployments regenerate the bulk artifacts from grains via the rebuild trigger. Returns the changed (or removed) grain paths for that trigger.

type Role

type Role int

Role is what a provider contributes to the graph, which fixes the provenance named graph its statements land in (ARCHITECTURE §5/§9).

const (
	// RoleIngest is a bibliographic source: its records are resolved and clustered
	// into Works, landing in the feed:<name> graph. OverDrive and MARC are ingest
	// providers.
	RoleIngest Role = iota
	// RoleEnrich is a provider that adds statements about existing entities
	// (authority labels, clustering hints, embeddings). Its statements land in the
	// editorial: or enrichment: graph, not feed:. Reserved: Run handles ingest
	// providers today; enrichment execution is future work.
	RoleEnrich
)

func (Role) String

func (r Role) String() string

String renders a Role for messages.

type StatsReporter added in v0.185.0

type StatsReporter interface {
	RunStats() EnrichStats
}

StatsReporter is an optional Enricher capability: RunStats reports the last run's EnrichStats so the run surface can include them in its result.

type SubjectEnricher

type SubjectEnricher interface {
	ControlledSubjects() []AuthoritySubject
}

SubjectEnricher is an optional capability a Record may implement to contribute controlled-vocabulary subjects for its Work -- e.g. by promoting free genre tags through an authority table. Run emits each as a bf:subject link to the authority URI plus its skos:prefLabel/broader statements in the feed graph, so the projector resolves them as controlled subjects with labels and hierarchy . For a clustered Work the first record's subjects win, matching how shared Work metadata and extras are taken; a Record that does not implement it contributes none.

type SummarySource

type SummarySource interface {
	SummariesWithPaths(ctx context.Context) ([]WorkSummary, map[string]string, error)
}

SummarySource yields the corpus's WorkSummaries plus each Work's grain path without a fresh corpus walk -- the seam a maintained index (the backend's workindex) plugs into where workers would otherwise each run their own ScanSummaries. Both return values are shared, read-only views.

type TermDescriber added in v0.35.0

type TermDescriber interface {
	DescribedTerms() []AuthoritySubject
}

TermDescriber is an optional capability a Record may implement to carry standalone vocabulary term descriptions -- typically the skos:broader ancestor chains of its ControlledSubjects. Run emits each as the term's skos:prefLabel/broader statements in the feed graph with NO bf:subject link to the Work, so hierarchy nodes above the asserted subjects stay labeled in the projection's term sideband even though no Work carries them directly. For a clustered Work the first record's terms win, matching ControlledSubjects.

type VerbatimProvider

type VerbatimProvider interface {
	Verbatim() []string
}

VerbatimProvider is an optional capability a Record may implement to carry the crosswalk-lossy MARC fields of its source record, serialized field-exact via bibframe.EncodeVerbatimField. Run writes them into the Instance's feed graph under bibframe.PredMARCVerbatim, so a lossy tag is preserved verbatim rather than silently dropped, and MARC export / the MARC view reproduce it.

type WorkSummary

type WorkSummary struct {
	WorkID       string
	Title        string
	Contributors []string
	// ContributorIDs are the contributors' authority IRIs, where agents
	// arrive as IRI nodes (MARC $0/$1 via the crosswalk, or editor
	// hydration) -- the identifier-based creator-resolution keys
	// (VIAF/LCNAF/ISNI/ORCID -> Wikidata). Unordered, deduped.
	ContributorIDs []string `json:",omitempty"`
	ISBNs          []string
	// Tags are the Work's uncontrolled subject labels (feed genres,
	// approved folksonomy) -- the raw material tag-to-controlled-term
	// reconciliation enrichers match against.
	Tags []string
	// Subjects are the Work's controlled subject IRIs (bf:subject with an
	// IRI object, any graph) -- what authority merges rewrite.
	Subjects []string
	// Headings are the heading labels of those controlled subjects, where the
	// grain describes them (skos:prefLabel or rdfs:label on the authority
	// node) -- the keyword dimension of the diversity audit. Unordered and not
	// paired with Subjects: the audit unions per-work category hits, so
	// pairing carries no extra signal.
	Headings []string `json:",omitempty"`
	// Creators are the Work's creators as resolved by the wikidata
	// enrichment (lcat:creatorIdentity in the grain): entity id + cached
	// explicit claims. Deliberately no display label -- the creator audit is
	// aggregate-only and never lists people.
	Creators []CreatorSummary `json:",omitempty"`
	// Series and Languages are the similarity scorer's two strongest signals
	//: two books in one series are related whatever else they
	// share, and a reader of one language is rarely served a book in another.
	// Series holds the Work's series statements, read off its bf:relation
	// series relations; Languages are bf:language local names ("en").
	Series    []string `json:",omitempty"`
	Languages []string `json:",omitempty"`
	// Visibility and holdings signals for the admin works list:
	// the editor deliberately shows everything, so each row says what the
	// public projection would do with it.
	Suppressed bool   `json:",omitempty"`
	Tombstoned bool   `json:",omitempty"`
	Withdrawn  string `json:",omitempty"` // date the feed reconciliation flagged it
	Kept       bool   `json:",omitempty"` // curator keeps it despite withdrawal
	// Items counts physical holdings across the Work's instances;
	// HasAvailability reports a live-availability identifier (a digital
	// holding as long as the Work is not withdrawn).
	Items           int  `json:",omitempty"`
	HasAvailability bool `json:",omitempty"`
	// Extras carries the Work's lcat:extra/* literals -- deployment-defined
	// key/value metadata the feed graph attaches (bibframe.addWorkExtras),
	// e.g. sources: "loc, mombian". The admin works view facets provenance
	// on configured keys; multi-valued keys are comma-joined by
	// convention and split at facet time.
	Extras map[string]string `json:",omitempty"`
}

WorkSummary is the slice of a Work an enricher reasons over: enough for a vocabulary lookup or reconciliation call without handing over the graph.

func ScanSummaries

func ScanSummaries(ctx context.Context, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)

ScanSummaries walks the grain tree and extracts a WorkSummary per Work, plus each Work's grain path.

func SummariesOf

func SummariesOf(ctx context.Context, src SummarySource, st blob.Store, prefix string) ([]WorkSummary, map[string]string, error)

SummariesOf reads summaries from src when one is wired, falling back to a fresh ScanSummaries walk of prefix.

func SummarizeDataset

func SummarizeDataset(ds *rdf.Dataset) []WorkSummary

SummarizeDataset is SummarizeGrain for callers that already hold the parsed dataset (the work index scans identity, summaries, and barcodes off one parse).

func SummarizeGrain

func SummarizeGrain(grain []byte) ([]WorkSummary, error)

SummarizeGrain extracts the WorkSummaries a grain carries (post-merge grains can hold several Works). Exported for callers that already hold the grain bytes, like the on-save authority auto-linker.

func (WorkSummary) Matches

func (s WorkSummary) Matches(q string) bool

Matches reports whether the summary matches a lowercased search query -- substring over title, id, contributors, tags, and ISBNs. One matcher serves the works listing and batch search selections, so a saved query means the same thing everywhere.

func (WorkSummary) SimilarWork added in v0.118.0

func (s WorkSummary) SimilarWork() similar.Work

SimilarWork converts an admin-side summary into the similarity scorer's input . It is one of exactly two converters; project.Work.SimilarWork is the other, and a test drives both from the same graph and requires equal results, because an OPAC rail and an admin panel that disagree about what a Work resembles is a bug the reader can see and nobody can explain.

Suppressed Works are kept. Suppression hides a Work from the public projection, which never reaches the scorer at all; the admin surface shows suppressed Works and so must recommend them. Tombstoned Works are passed through with the flag set and excluded by similar.Build -- retiring a record must not leave it recommended from elsewhere.

Held collapses the two digital/physical signals the summary carries separately into the one predicate project.Work.Held already publishes.

Directories

Path Synopsis
Package bibliocommons harvests a peer library's subject cataloging through the public BiblioCommons RSS search -- the reverse direction, because the forward one does not exist: a record page exposes no subjects to an unauthenticated reader, but a SUBJECT search feeds every title it is assigned to.
Package bibliocommons harvests a peer library's subject cataloging through the public BiblioCommons RSS search -- the reverse direction, because the forward one does not exist: a record page exposes no subjects to an unauthenticated reader, but a SUBJECT search feeds every title it is assigned to.
Package csvmap is the generic CSV ingest provider: it reads a spreadsheet-shaped export -- the lowest-common-denominator dump every ILS and collection tool can produce -- into ingest records driven by a declarative TOML mapping of columns to fields, so a deployment sideloads a CSV with a profile, not code.
Package csvmap is the generic CSV ingest provider: it reads a spreadsheet-shaped export -- the lowest-common-denominator dump every ILS and collection tool can produce -- into ingest records driven by a declarative TOML mapping of columns to fields, so a deployment sideloads a CSV with a profile, not code.
Package hardcover is a first-party ingest provider for a Hardcover (hardcover.app) "Read" shelf.
Package hardcover is a first-party ingest provider for a Hardcover (hardcover.app) "Read" shelf.
Package lchub is an offline external-identity enricher: it links a Work to its Library of Congress bf:Hub (id.loc.gov/resources/hubs) by an exact creator+title+language access-point match, from a pre-downloaded index (LC publishes the hubs as a bulk download; no live API on the ingest path).
Package lchub is an offline external-identity enricher: it links a Work to its Library of Congress bf:Hub (id.loc.gov/resources/hubs) by an exact creator+title+language access-point match, from a pre-downloaded index (LC publishes the hubs as a bulk download; no live API on the ingest path).
Package locsh is the reference enrichment provider: it reconciles a Work's uncontrolled tags (feed genres, approved folksonomy) against Library of Congress Subject Headings via the public id.loc.gov suggest2 API, yielding controlled-subject candidates with label-match confidence.
Package locsh is the reference enrichment provider: it reconciles a Work's uncontrolled tags (feed genres, approved folksonomy) against Library of Congress Subject Headings via the public id.loc.gov suggest2 API, yielding controlled-subject candidates with label-match confidence.
Package marc is the MARC ingest provider (ARCHITECTURE §9): it reads an ISO 2709 MARC stream and yields each record as a resolvable ingest.Record, so an existing ILS's MARC flows through the same two-tier identity + clustering pipeline as any other provider (ingest.Run).
Package marc is the MARC ingest provider (ARCHITECTURE §9): it reads an ISO 2709 MARC stream and yields each record as a resolvable ingest.Record, so an existing ILS's MARC flows through the same two-tier identity + clustering pipeline as any other provider (ingest.Run).
Package nquads is the generic mapped N-Quads ingest provider: it streams a dcterms-shaped .nq export into ingest records driven entirely by a declarative TOML mapping -- work-IRI prefix, predicate->field map, identifier URN schemes, and source-attestation tiers -- so a deployment sideloads an RDF export the way Aspen Discovery sideloads MARC: with a profile, not code.
Package nquads is the generic mapped N-Quads ingest provider: it streams a dcterms-shaped .nq export into ingest records driven entirely by a declarative TOML mapping -- work-IRI prefix, predicate->field map, identifier URN schemes, and source-attestation tiers -- so a deployment sideloads an RDF export the way Aspen Discovery sideloads MARC: with a profile, not code.
Package oaipmh is the OAI-PMH bibliographic harvest provider (feature ils-import): it pulls MARC records from an ILS's OAI-PMH endpoint and yields them through the same crosswalk + clustering pipeline as file MARC ingest, so onboarding an ILS that speaks OAI-PMH (Koha, Evergreen, and many others) is a Register call plus config, no libcat fork.
Package oaipmh is the OAI-PMH bibliographic harvest provider (feature ils-import): it pulls MARC records from an ILS's OAI-PMH endpoint and yields them through the same crosswalk + clustering pipeline as file MARC ingest, so onboarding an ILS that speaks OAI-PMH (Koha, Evergreen, and many others) is a Register call plus config, no libcat fork.
Package openlibrary is an offline external-identity enricher: it links a Work to its OpenLibrary work id by exact ISBN match, from a pre-downloaded ISBN -> work index (OpenLibrary publishes bulk dumps, so there is no per-record live API on the ingest path).
Package openlibrary is an offline external-identity enricher: it links a Work to its OpenLibrary work id by exact ISBN match, from a pre-downloaded ISBN -> work index (OpenLibrary publishes bulk dumps, so there is no per-record live API on the ingest path).
Package overdrive maps OverDrive/Libby "thunder" API records (as cached by a scan) directly to libcodex BIBFRAME Work/Instance grains (see bibframe.go).
Package overdrive maps OverDrive/Libby "thunder" API records (as cached by a scan) directly to libcodex BIBFRAME Work/Instance grains (see bibframe.go).
Package sirsidynix harvests peer libraries' subject cataloging from SirsiDynix Enterprise catalogs at <lib>.ent.sirsidynix.net through the anonymous RSS hitlist: one GET per driver term scoped to the Subject index, so the term surfaces every record a peer cataloged under that subject heading.
Package sirsidynix harvests peer libraries' subject cataloging from SirsiDynix Enterprise catalogs at <lib>.ent.sirsidynix.net through the anonymous RSS hitlist: one GET per driver term scoped to the Subject index, so the term surfaces every record a peer cataloged under that subject heading.
Package tlc harvests peer libraries' subject cataloging from TLC (The Library Corporation) LS2 PAC catalogs through the anonymous search endpoint: one POST per driver term with the term as BOTH the keyword and a Subject facet filter, so the keyword surfaces candidates and the facet enforces subject-cataloging precision.
Package tlc harvests peer libraries' subject cataloging from TLC (The Library Corporation) LS2 PAC catalogs through the anonymous search endpoint: one POST per driver term with the term as BOTH the keyword and a Subject facet filter, so the keyword surfaces candidates and the facet enforces subject-cataloging precision.
Package vega harvests peer libraries' subject cataloging from III Vega Discover catalogs (NYPL and other Innovative sites) through the anonymous JSON API at <region>.iiivega.com.
Package vega harvests peer libraries' subject cataloging from III Vega Discover catalogs (NYPL and other Innovative sites) through the anonymous JSON API at <region>.iiivega.com.
Package wikidata resolves a Work's creators to Wikidata entities and caches their EXPLICITLY-STATED demographic claims as enrichment statements -- the creator-demographics half of the diversity-audit feature.
Package wikidata resolves a Work's creators to Wikidata entities and caches their EXPLICITLY-STATED demographic claims as enrichment statements -- the creator-demographics half of the diversity-audit feature.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL