libcat

module

v0.229.2 Latest Latest Go to latest Published: Jul 14, 2026 License: MIT

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/freeeve/libcat

Links

Open Source Insights

README ¶

libcat

A generic framework for building library discovery catalogs as fast, static, faceted websites -- with BIBFRAME as the source of truth and an optional collaborative cataloging backend.

Live demo: libcat.evefreeman.com -- "Eve's Library", a public adopter site built on the framework + Hugo module.

libcat is the framework. A deployment (for example a queer-literature library on OverDrive) themes and implements it: it brings its own collection, controlled vocabularies, provider feeds, and branding. Nothing library-specific lives in here.

What it is

Source of truth: BIBFRAME (RDF) -- Work/Instance-native bibliographic data, not flat records, stored as canonical N-Quads grains in a blob store (a directory or an S3-compatible bucket, not git). Everything else (HTML, search index, MARC/MODS exports) is a derived build artifact.
Static catalog as a Hugo module -- drop the catalog into a library's existing Hugo site; the rest of their web presence stays their own.
Powered by two sibling libraries:
- libcodex -- MARC / MODS / Dublin Core / schema.org / BIBFRAME read, write, convert (the interchange, import, and export engine).
- roaringrange -- the advanced search index and reader (lexical BM25; semantic embeddings opt-in) for large-corpus / custom-ranking deployments. The recommended static search is Pagefind over the built HTML (tasks/017); out of the box the module falls back to a small client-side filter until a site opts in.
No triplestore, no database for the static tier -- grain files in a blob store (a directory or an S3-compatible bucket), static output on S3/CloudFront. No paid AI in the default build.

Two tiers

Tier 1 -- static, self-serve (no backend). Point the projector at a MARC or BIBFRAME dump and get a faceted, searchable, multilingual catalog site. The onboarding ramp is MARC import (via libcodex): bring the MARC your ILS already exports.
Tier 2 -- dynamic, optional. A collaborative in-browser cataloging/review app (auth, roles, edit history) that writes BIBFRAME back to the grain store. Cloud infra; self-hosted or SaaS. Lives in backend/, a nested Go module (github.com/freeeve/libcat/backend) so its cloud SDKs never reach the core's dependency tree -- CI builds and tests the root module and backend/ separately (as with hugo/). Serve it with backend/cmd/lcatd (container/self-host) or backend/cmd/lcatd-lambda (AWS Lambda behind an API Gateway v2 HTTP API); both wrap the same net/http handler.

Because the BIBFRAME graph is the contract between them, Tier 1 runs with zero Tier 2.

See docs/ARCHITECTURE.md and docs/ROADMAP.md. The Tier 2 HTTP surface is enumerated in docs/api.md, generated from the router. For building a catalog from a Hardcover reading shelf, see docs/hardcover-provider.md.

Adopters don't write Go. One lcat.toml describes the whole build -- sources (MARC files, mapped N-Quads/CSV exports, OverDrive), projection with a public-provenance allowlist, downloadable exports, search index, Hugo -- and lcat build runs it end to end. See docs/build-pipeline.md. The Go provider seam stays for bespoke sources.

Deployment styles

The two tiers unbundle into a ladder of shapes; pick by how much of the dynamic surface you need. Memory numbers are indicative, from measured deployments.

Static only (zero server). The projected site is plain files: host it on any static host, object storage bucket, or CDN. Client-side search (roaringrange or Pagefind) and live availability both run in the browser, so a full faceted catalog needs no server process at all. Projection is a batch step, not a resident cost -- run it in CI or on a laptop (peak ~57KB/work), and the feed-driven incremental rebuild keeps publishes from reprojecting the corpus.
Read-only backend (demo/sandbox). lcatd in read-only mode serves the cataloging UI without write workers; container or Lambda (backend/deploy/terraform/modules/readonly-demo).
Writable backend. A single long-lived lcatd container with the in-memory store snapshotting to blob storage; the store and blob clients also speak DynamoDB/S3-compatible APIs (LCATD_AWS_ENDPOINT), so AWS, MinIO/Garage, or ScyllaDB are interchangeable deployment concerns.
What sets the backend's memory: vocabularies, then catalog size. The resident work index holds summaries and identity keys (not the graph), so small and mid catalogs are cheap; the controlled-vocabulary term index is the dominant block. Full LCSH+LCSHAC+LCGFT resident is ~1.2GB before the first work -- a comfortable-cataloging shape wants a few GB. To run in ~1GB or less, install subset vocabulary snapshots instead: lcat vocab-subset scopes a vocabulary to the concepts your catalog actually uses (plus small vocabularies like Homosaurus whole), collapsing the index to tens of MB.

Status

Working, with versioned releases. Both tiers are implemented:

Tier 1 ships end to end: MARC / Hardcover / OverDrive ingest to canonical BIBFRAME grains, projection to catalog.json/facets.json, and the Hugo module's faceted, multilingual, Pagefind-searchable site with optional live availability (OverDrive/Thunder, DAIA). "Eve's Library" is the live adopter built on it.
Tier 2 is the lcatd backend serving an embedded cataloging SPA: a dual-view (BIBFRAME doc / MARC) record editor with editing profiles, dry-run previews, and duplicate warnings; copy cataloging over SRU/Z39.50 with staged batches, overlay policies, and revert; batch edits, macros, and saved queries; authority control with local headings and installable vocabulary snapshots (LCSH-scale); moderated patron suggestions and tag promotion; exports (MARC, N-Quads, JSON-LD, CSV); and a read-only sandbox mode for public demos. Auth is built-in local users and/or OIDC. Deploy as a container (backend/cmd/lcatd) or AWS Lambda (backend/cmd/lcatd-lambda); backend/deploy/terraform/modules/readonly-demo stands up the demo shape.

The reference implementation was migrated out of qllpoc (that repo's tasks/038--044); libcat is now developed in place, tracked in tasks/.

Versioning

The root, hugo/, and backend/ modules release in lockstep: every release tags all three with the same version at the same commit (scripts/release.sh vX.Y.Z tags and pushes the triple). The version number IS the compatibility contract -- pin one number across every libcat dependency and the projection-schema pairing (the catalog.json version the projector emits and the Hugo module targets) is guaranteed by construction; the released backend module requires the root module at its own version for the same reason. Lockstep (re)starts at v0.19.0: earlier numbers come from three independent tag families, several of them never published -- treat anything below v0.19.0 as historical and do not pin across modules with them.

License

MIT -- Copyright Eve Freeman.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
backend module
bibframe Package bibframe crosswalks codex records into canonical BIBFRAME N-Quads grains.	Package bibframe crosswalks codex records into canonical BIBFRAME N-Quads grains.
catalogindex Package catalogindex builds a read-only, periodically-rebuildable analytics view of the whole catalog over an in-memory gochickpeas graph.	Package catalogindex builds a read-only, periodically-rebuildable analytics view of the whole catalog over an in-memory gochickpeas graph.
cmd
lcat command Command lcat is libcat's projector CLI: it ingests bibliographic sources into canonical BIBFRAME grains (and, later, projects them for the Hugo module).	Command lcat is libcat's projector CLI: it ingests bibliographic sources into canonical BIBFRAME grains (and, later, projects them for the Hugo module).
diversity Package diversity maps a work's controlled subjects to diversity-audit categories.	Package diversity maps a work's controlled subjects to diversity-audit categories.
export Package export derives the downloadable catalog artifacts from an ingest output root: catalog.nq.gz is the corpus itself -- the catalog.nq found there, which every writer now emits as the merge of the grains, with blank labels namespaced by work id -- and catalog.mrc.gz / catalog.xml.gz are per-grain MARC round-trips via bibframe.DecodeGrainMARC, which honors editorial override shadows and verbatim sidecars (fidelity bounded by docs/marc-fidelity.md).	Package export derives the downloadable catalog artifacts from an ingest output root: catalog.nq.gz is the corpus itself -- the catalog.nq found there, which every writer now emits as the merge of the grains, with blank labels namespaced by work id -- and catalog.mrc.gz / catalog.xml.gz are per-grain MARC round-trips via bibframe.DecodeGrainMARC, which honors editorial override shadows and verbatim sidecars (fidelity bounded by docs/marc-fidelity.md).
identity Package identity assigns libcat's opaque, two-tier Work/Instance ids (ARCHITECTURE §4).	Package identity assigns libcat's opaque, two-tier Work/Instance ids (ARCHITECTURE §4).
ingest Facet dimensions over a WorkSummary.	Facet dimensions over a WorkSummary.
bibliocommons Package bibliocommons harvests a peer library's subject cataloging through the public BiblioCommons RSS search -- the reverse direction, because the forward one does not exist: a record page exposes no subjects to an unauthenticated reader, but a SUBJECT search feeds every title it is assigned to.	Package bibliocommons harvests a peer library's subject cataloging through the public BiblioCommons RSS search -- the reverse direction, because the forward one does not exist: a record page exposes no subjects to an unauthenticated reader, but a SUBJECT search feeds every title it is assigned to.
csvmap Package csvmap is the generic CSV ingest provider: it reads a spreadsheet-shaped export -- the lowest-common-denominator dump every ILS and collection tool can produce -- into ingest records driven by a declarative TOML mapping of columns to fields, so a deployment sideloads a CSV with a profile, not code.	Package csvmap is the generic CSV ingest provider: it reads a spreadsheet-shaped export -- the lowest-common-denominator dump every ILS and collection tool can produce -- into ingest records driven by a declarative TOML mapping of columns to fields, so a deployment sideloads a CSV with a profile, not code.
hardcover Package hardcover is a first-party ingest provider for a Hardcover (hardcover.app) "Read" shelf.	Package hardcover is a first-party ingest provider for a Hardcover (hardcover.app) "Read" shelf.
lchub Package lchub is an offline external-identity enricher: it links a Work to its Library of Congress bf:Hub (id.loc.gov/resources/hubs) by an exact creator+title+language access-point match, from a pre-downloaded index (LC publishes the hubs as a bulk download; no live API on the ingest path).	Package lchub is an offline external-identity enricher: it links a Work to its Library of Congress bf:Hub (id.loc.gov/resources/hubs) by an exact creator+title+language access-point match, from a pre-downloaded index (LC publishes the hubs as a bulk download; no live API on the ingest path).
locsh Package locsh is the reference enrichment provider: it reconciles a Work's uncontrolled tags (feed genres, approved folksonomy) against Library of Congress Subject Headings via the public id.loc.gov suggest2 API, yielding controlled-subject candidates with label-match confidence.	Package locsh is the reference enrichment provider: it reconciles a Work's uncontrolled tags (feed genres, approved folksonomy) against Library of Congress Subject Headings via the public id.loc.gov suggest2 API, yielding controlled-subject candidates with label-match confidence.
marc Package marc is the MARC ingest provider (ARCHITECTURE §9): it reads an ISO 2709 MARC stream and yields each record as a resolvable ingest.Record, so an existing ILS's MARC flows through the same two-tier identity + clustering pipeline as any other provider (ingest.Run).	Package marc is the MARC ingest provider (ARCHITECTURE §9): it reads an ISO 2709 MARC stream and yields each record as a resolvable ingest.Record, so an existing ILS's MARC flows through the same two-tier identity + clustering pipeline as any other provider (ingest.Run).
nquads Package nquads is the generic mapped N-Quads ingest provider: it streams a dcterms-shaped .nq export into ingest records driven entirely by a declarative TOML mapping -- work-IRI prefix, predicate->field map, identifier URN schemes, and source-attestation tiers -- so a deployment sideloads an RDF export the way Aspen Discovery sideloads MARC: with a profile, not code.	Package nquads is the generic mapped N-Quads ingest provider: it streams a dcterms-shaped .nq export into ingest records driven entirely by a declarative TOML mapping -- work-IRI prefix, predicate->field map, identifier URN schemes, and source-attestation tiers -- so a deployment sideloads an RDF export the way Aspen Discovery sideloads MARC: with a profile, not code.
oaipmh Package oaipmh is the OAI-PMH bibliographic harvest provider (feature ils-import): it pulls MARC records from an ILS's OAI-PMH endpoint and yields them through the same crosswalk + clustering pipeline as file MARC ingest, so onboarding an ILS that speaks OAI-PMH (Koha, Evergreen, and many others) is a Register call plus config, no libcat fork.	Package oaipmh is the OAI-PMH bibliographic harvest provider (feature ils-import): it pulls MARC records from an ILS's OAI-PMH endpoint and yields them through the same crosswalk + clustering pipeline as file MARC ingest, so onboarding an ILS that speaks OAI-PMH (Koha, Evergreen, and many others) is a Register call plus config, no libcat fork.
openlibrary Package openlibrary is an offline external-identity enricher: it links a Work to its OpenLibrary work id by exact ISBN match, from a pre-downloaded ISBN -> work index (OpenLibrary publishes bulk dumps, so there is no per-record live API on the ingest path).	Package openlibrary is an offline external-identity enricher: it links a Work to its OpenLibrary work id by exact ISBN match, from a pre-downloaded ISBN -> work index (OpenLibrary publishes bulk dumps, so there is no per-record live API on the ingest path).
overdrive Package overdrive maps OverDrive/Libby "thunder" API records (as cached by a scan) directly to libcodex BIBFRAME Work/Instance grains (see bibframe.go).	Package overdrive maps OverDrive/Libby "thunder" API records (as cached by a scan) directly to libcodex BIBFRAME Work/Instance grains (see bibframe.go).
sirsidynix Package sirsidynix harvests peer libraries' subject cataloging from SirsiDynix Enterprise catalogs at <lib>.ent.sirsidynix.net through the anonymous RSS hitlist: one GET per driver term scoped to the Subject index, so the term surfaces every record a peer cataloged under that subject heading.	Package sirsidynix harvests peer libraries' subject cataloging from SirsiDynix Enterprise catalogs at <lib>.ent.sirsidynix.net through the anonymous RSS hitlist: one GET per driver term scoped to the Subject index, so the term surfaces every record a peer cataloged under that subject heading.
tlc Package tlc harvests peer libraries' subject cataloging from TLC (The Library Corporation) LS2 PAC catalogs through the anonymous search endpoint: one POST per driver term with the term as BOTH the keyword and a Subject facet filter, so the keyword surfaces candidates and the facet enforces subject-cataloging precision.	Package tlc harvests peer libraries' subject cataloging from TLC (The Library Corporation) LS2 PAC catalogs through the anonymous search endpoint: one POST per driver term with the term as BOTH the keyword and a Subject facet filter, so the keyword surfaces candidates and the facet enforces subject-cataloging precision.
vega Package vega harvests peer libraries' subject cataloging from III Vega Discover catalogs (NYPL and other Innovative sites) through the anonymous JSON API at <region>.iiivega.com.	Package vega harvests peer libraries' subject cataloging from III Vega Discover catalogs (NYPL and other Innovative sites) through the anonymous JSON API at <region>.iiivega.com.
wikidata Package wikidata resolves a Work's creators to Wikidata entities and caches their EXPLICITLY-STATED demographic claims as enrichment statements -- the creator-demographics half of the diversity-audit feature.	Package wikidata resolves a Work's creators to Wikidata entities and caches their EXPLICITLY-STATED demographic claims as enrichment statements -- the creator-demographics half of the diversity-audit feature.
project Package project turns the canonical BIBFRAME graph into the catalog's derived data: the JSON a static site (the Hugo module's content adapter) and the search index consume (ARCHITECTURE §7).	Package project turns the canonical BIBFRAME graph into the catalog's derived data: the JSON a static site (the Hugo module's content adapter) and the search index consume (ARCHITECTURE §7).
search Browse artifacts for the client-side reader.	Browse artifacts for the client-side reader.
similar Package similar answers "more like this" over a catalog: given a Work, which other Works does it most resemble?	Package similar answers "more like this" over a catalog: given a Work, which other Works does it most resemble?
storage Package storage abstracts where lcat writes its artifacts, so the same build runs unchanged against a local directory, cloud object storage (S3/GCS/Azure), or a git working tree.	Package storage abstracts where lcat writes its artifacts, so the same build runs unchanged against a local directory, cloud object storage (S3/GCS/Azure), or a git working tree.
blob Package blob defines a read-write, path-addressed object store with conditional writes -- the seam the dynamic tier uses to hold BIBFRAME grains in S3-compatible storage.	Package blob defines a read-write, path-addressed object store with conditional writes -- the seam the dynamic tier uses to hold BIBFRAME grains in S3-compatible storage.
vocabsidecar Package vocabsidecar owns the on-disk layout and lifecycle of a vocabulary scheme's sidecar artifacts -- where they live under the authorities tree, the manifest that arms a scheme, and the layout-only operations (remove, orphan sweep) that need no knowledge of how the artifacts are built or read.	Package vocabsidecar owns the on-disk layout and lifecycle of a vocabulary scheme's sidecar artifacts -- where they live under the authorities tree, the manifest that arms a scheme, and the layout-only operations (remove, orphan sweep) that need no knowledge of how the artifacts are built or read.