export

package
v0.229.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package export derives the downloadable catalog artifacts from an ingest output root: catalog.nq.gz is the corpus itself -- the catalog.nq found there, which every writer now emits as the merge of the grains, with blank labels namespaced by work id -- and catalog.mrc.gz / catalog.xml.gz are per-grain MARC round-trips via bibframe.DecodeGrainMARC, which honors editorial override shadows and verbatim sidecars (fidelity bounded by docs/marc-fidelity.md). A Manifest records sizes, sha256 digests, and record counts for a downloads page. This is the static-site counterpart of the backend's on-demand export service (backend/export); both read the same grains.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type File

type File struct {
	Name    string `json:"name"`
	Bytes   int64  `json:"bytes"`
	SHA256  string `json:"sha256"`
	Records int    `json:"records"`
}

File is one artifact's manifest entry; the sha256 is over the compressed bytes (what a downloader verifies).

type Manifest

type Manifest struct {
	Generated string `json:"generated"`
	// Works counts the Works the download describes -- the visible ones, so this
	// number is comparable with catalog.json's. It used to count grains, which
	// cannot disagree with anything: a build that published 3274 records for a
	// 31-work catalog reported 3274 and looked healthy.
	Works int `json:"works"`
	// Hidden counts the grains held back as suppressed or tombstoned. Recorded so
	// a takedown is auditable from the build output rather than inferable from it.
	Hidden int    `json:"hidden"`
	Files  []File `json:"files"`
}

Manifest is the downloads-page data file: what was generated when, from how many works, with per-artifact integrity and record counts.

func Run

func Run(opts Options) (*Manifest, error)

Run exports the corpus under opts.In to gzipped download artifacts under opts.Out and returns their manifest.

type Options

type Options struct {
	// In is the ingest output root: the directory holding catalog.nq and
	// data/works/ grains.
	In string
	// Out is the directory the gzipped artifacts are written to.
	Out string
	// PublicSources, when non-nil, is the allowlist of extra/sources names
	// permitted in the nq download; other attributions are stripped, matching
	// project.SanitizeSources on the catalog.json side, so no public surface
	// discloses more provenance than the site does. Nil keeps every source.
	// The on-disk graph of record stays complete either way.
	PublicSources map[string]bool
	// PublicExtras, when non-nil, is the allowlist of lcat:extra/ *keys*
	// permitted in the nq download, matching project.SanitizeExtras on the
	// catalog.json side. Other extra quads are dropped whole.
	// "sources" is exempt: PublicSources governs it, by value rather than by
	// key. Nil keeps every extra.
	PublicExtras map[string]bool
	// Log receives skip and strip warnings; nil means os.Stderr.
	Log io.Writer
	// OrgCode is the deployment's MARC organization code; when set, the
	// MARC download derives each record's 040 from graph facts at decode
	// time.
	OrgCode string
	// CoversOut, when set, copies uploaded cover images (data/covers/ under
	// In) to this directory as flat files, the site-relative covers/ URLs
	// the editorial lcat:extra/cover statements point at.
	// Empty skips the copy.
	CoversOut string
}

Options configures one export run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL