copycat

package
v0.205.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package copycat is Koha's Z39.50/SRU copy cataloging and staged-import workflow over the shared ingest pipeline: external targets are searched through the libcodex protocol clients, results and .mrc uploads stage into datastore batches (nothing touches the grain tree), every staged record carries its identity-resolver match ("would merge with Work w…"), and commit runs the batch through the same clustering pipeline every feed uses -- store-backed, CAS-guarded, editorial always preserved.

Original cataloging: blank-record MARC skeletons and the validation gate for records that enter the catalog from the editor rather than an external target. A draft stages into a normal batch with source "original", so review, identity matching, commit, and revert are the same machinery every import uses.

Index

Constants

View Source 
const (
	ProtocolSRU   = "sru"
	ProtocolZ3950 = "z3950"
)

Protocols a target can speak.

View Source 
const (
	// PolicyReplaceFeed commits everything; a matched Instance's feed
	// statements are replaced by the incoming record's (the pipeline's
	// re-ingest semantics).
	PolicyReplaceFeed = "replace-feed"
	// PolicyFillHoles commits only records whose Instance is new -- a
	// matched Work gains the new edition, but an existing Instance is never
	// overwritten.
	PolicyFillHoles = "fill-holes-only"
	// PolicyNever commits only records with no match at all.
	PolicyNever = "never"
)

Overlay policies: what a commit does with records that match the existing corpus. Editorial statements are preserved by the pipeline in every case.

View Source 
const (
	DecisionImport = "import"
	DecisionSkip   = "skip"
)

Decisions a reviewer takes per staged record.

View Source 
const (
	StatusStaged    = "STAGED"
	StatusCommitted = "COMMITTED"
)

Batch statuses.

View Source 
const StatusReverted = "REVERTED"

StatusReverted marks a committed batch whose grains were rolled back.

Variables

View Source 
var DefaultTargets = []Target{
	{Name: "dnb-sru", URL: "https://services.dnb.de/sru/dnb", Protocol: ProtocolSRU,
		Version: "1.1", Schema: "MARC21-xml",
		Indexes: map[string]string{"isbn": "dnb.num", "issn": "dnb.num"}},
	{Name: "k10plus-sru", URL: "https://sru.k10plus.de/opac-de-627", Protocol: ProtocolSRU,
		Indexes: map[string]string{"isbn": "pica.isb", "issn": "pica.iss"}},
	{Name: "loc-sru", URL: "http://lx2.loc.gov:210/LCDB", Protocol: ProtocolSRU},
}

DefaultTargets are the search sources seeded on a store that has never had targets: open, anonymous SRU endpoints serving MARC21, each verified live against the exact queries the fielded search emits. LOC speaks the Bath-profile identifier indexes as-is; DNB only answers SRU 1.1 and names its schema MARC21-xml, with dnb.num covering both standard numbers; K10plus wants its PICA identifier indexes.

View Source 
var ErrCapped = errors.New("result set truncated at the search limit")

ErrCapped reports that the search limit, not the result set, ended the stream: the target may hold more records than were returned.

View Source 
var ErrConflict = errors.New("copycat: conflicts with batch state")

ErrConflict reports a request that conflicts with the batch's current state -- e.g. deleting a COMMITTED batch whose revert-set is still the only undo.

View Source 
var ErrNotFound = errors.New("copycat: not found")

ErrNotFound reports a missing target or batch.

View Source 
var ErrValidation = errors.New("copycat: invalid request")

ErrValidation reports a request the service refuses.

View Source 
var SuggestedTargets = []Target{
	{Name: "loc", URL: "lx2.loc.gov:210/LCDB", Protocol: ProtocolZ3950},
	{Name: "loc-sru", URL: "http://lx2.loc.gov:210/LCDB", Protocol: ProtocolSRU},
	{Name: "k10plus", URL: "https://sru.k10plus.de/opac-de-627", Protocol: ProtocolSRU,
		Indexes: map[string]string{"isbn": "pica.isb", "issn": "pica.iss"}},
	{Name: "indexdata-test", URL: "z3950.indexdata.com:210/marc", Protocol: ProtocolZ3950},
}

SuggestedTargets are the open, no-credential sources the copycat UI offers as one-click presets. It is the single source for the preset row: the UI fetches it (GET /v1/copycat/targets/suggested) rather than maintaining its own copy, which is how the k10plus preset came to lack the PICA indexes its seeded twin carries. A preset sharing a URL with a DefaultTargets entry must carry the same SRU knobs, or the one-click target speaks different CQL than the seeded one -- TestSuggestedTargetsAgreeWithDefaults pins that. Blurbs are the UI's to add; the wire config lives here.

Functions

func Incomplete added in v0.109.0 

func Incomplete(err error) bool

Incomplete reports whether err means "these records, but not all of them" -- a partial stream or the search cap -- as opposed to an outright failure. A search returning records with an Incomplete error is a warning, never a failure: suppressing the hits would throw away the useful half.

func ValidPolicy added in v0.189.1 

func ValidPolicy(p string) bool

ValidPolicy reports whether p names a commit policy. Exported so the handler can refuse a bad policy BEFORE staging persists anything -- a 400 that leaves an orphaned batch behind teaches clients to retry their way into more orphans.

Types

type Batch 

type Batch struct {
	ID        string    `json:"id"`
	Label     string    `json:"label"`
	Source    string    `json:"source"` // "upload" or a target name
	Policy    string    `json:"policy"`
	Status    string    `json:"status"`
	Records   int       `json:"records"`
	Owner     string    `json:"owner"`
	CreatedAt time.Time `json:"createdAt"`
	// Commit outcome.
	Committed int       `json:"committed,omitempty"`
	Skipped   int       `json:"skipped,omitempty"`
	CommitAt  time.Time `json:"commitAt,omitzero"`
	// Revert outcome.
	Reverted int       `json:"reverted,omitempty"`
	RevertAt time.Time `json:"revertAt,omitzero"`
}

Batch is one staged import.

type FieldError 

type FieldError struct {
	Tag     string `json:"tag"`
	Message string `json:"message"`
}

FieldError anchors a validation failure to a MARC field ("LDR" for the leader).

func ValidateOriginal 

func ValidateOriginal(doc marcview.RecordDoc) []FieldError

ValidateOriginal is the minimum-viability gate for editor-born records: a 24-character leader, one 245 with a non-empty $a, 40-character 008s, and well-formed fields. It judges the pruned record -- untouched skeleton rows are not errors.

type FieldTerm 

type FieldTerm struct {
	Index string `json:"index"`
	Term  string `json:"term"`
}

FieldTerm is one (access point, term) pair of a fielded search; terms AND together. Indexes are the ones libcodex maps on both protocols.

type Match 

type Match struct {
	WorkID          string `json:"workId,omitempty"`
	InstanceID      string `json:"instanceId,omitempty"`
	MatchedWork     bool   `json:"matchedWork"`
	MatchedInstance bool   `json:"matchedInstance"`
}

Match is a staged record's dry-run identity resolution against the current corpus.

type PartialError added in v0.109.0 

type PartialError struct {
	Got   int
	Total int
	Err   error
}

PartialError reports a stream that broke after delivering records. It carries both, because a caller needs the hits and the reason the set is short.

Total is what the target said its result set holds, or -1 when it never said (unknownTotal). "1 of 9 records arrived" is a different sentence from "1 record arrived", and only the first tells a cataloger how much they are missing.

The zero value is inert rather than wrong: a PartialError exists only when at least one record arrived, so Total == 0 can never exceed Got and reads as "no total", the same as -1.

func (*PartialError) Error added in v0.109.0 

func (e *PartialError) Error() string

func (*PartialError) Unwrap added in v0.109.0 

func (e *PartialError) Unwrap() error

type Profile 

type Profile struct {
	Name    string   `json:"name"`
	Targets []string `json:"targets,omitempty"`
	Policy  string   `json:"policy,omitempty"`
}

Profile is a saved staging configuration: which targets a search fans out to and the overlay policy a staged batch starts with -- recurring imports stop re-entering the same choices per batch.

type RevertResult 

type RevertResult struct {
	Batch    Batch        `json:"batch"`
	Reverted int          `json:"reverted"`
	Skipped  []RevertSkip `json:"skipped,omitempty"`
}

RevertResult summarizes a batch revert.

type RevertSkip 

type RevertSkip struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
}

RevertSkip reports one grain the revert left alone.

type SearchFunc 

type SearchFunc func(ctx context.Context, t Target, terms []FieldTerm, limit int) ([]*codex.Record, error)

SearchFunc is the protocol seam: it fetches up to limit records from one target. A non-nil error alongside records means the set is incomplete; see Incomplete. Tests inject fakes; production uses protocolSearch.

type SearchResult 

type SearchResult struct {
	Target  string             `json:"target"`
	Title   string             `json:"title,omitempty"`
	Author  string             `json:"author,omitempty"`
	Date    string             `json:"date,omitempty"`
	ISBN    string             `json:"isbn,omitempty"`
	Edition string             `json:"edition,omitempty"`
	LCCN    string             `json:"lccn,omitempty"`
	Record  marcview.RecordDoc `json:"record"`
}

SearchResult is one external hit, ready to stage.

type Service 

type Service struct {
	Blob blob.Store
	DB   store.Store
	// Queue receives audit entries (optional).
	Queue *suggest.Service
	// Trigger gets one grains-changed event per commit (optional).
	Trigger trigger.Notifier
	// Prefix roots the grain tree ("" = repo layout).
	Prefix string
	// Feed names the provenance graph committed batches write (default
	// "copycat").
	Feed string
	// Search overrides the protocol clients (tests); nil = protocolSearch.
	Search SearchFunc
	// Index, when set (and Prefix is ""), is the shared work index match
	// passes seed from instead of loading the whole prior grain store per
	// Stage/Commit; nil falls back to LoadPriorStore.
	Index *workindex.Index
}

Service is the copy-cataloging surface.

func (*Service) Batches 

func (s *Service) Batches(ctx context.Context) ([]Batch, error)

Batches lists every staged import, newest first.

func (*Service) Commit 

func (s *Service) Commit(ctx context.Context, id, actor string) (Batch, error)

Commit runs a batch's importable records through the shared store-backed ingest pipeline: matches are re-resolved against the current corpus, the overlay policy filters, and grains land via CAS with editorial preserved. A COMMITTED batch refuses to commit again: the grain pipeline is byte-stable across a re-commit, but the undo bookkeeping is not -- a double commit re-derived Created from the now-existing grains, poisoning the revert set so revert reported success while leaving the created works live. Committing a REVERTED batch stays allowed: it is the redo of an undone import, and at that point the store again holds the true prior state, so the undo set re-derives correctly.

func (*Service) DeleteBatch 

func (s *Service) DeleteBatch(ctx context.Context, id string) error

DeleteBatch removes a batch, its records, and its revert set. A COMMITTED batch is refused: its revert-set is the only pre-commit snapshot Revert can roll the commit back from, and the works it created persist, so deleting it would strand them non-revertable. This is symmetric to the commit-side guard, which refuses re-committing a COMMITTED batch. Revert the batch first (which sets it REVERTED), then it deletes freely.

func (*Service) DeleteProfile 

func (s *Service) DeleteProfile(ctx context.Context, name string) error

DeleteProfile removes a staging profile.

func (*Service) DeleteTarget 

func (s *Service) DeleteTarget(ctx context.Context, name string) error

DeleteTarget removes a target.

func (*Service) GetBatch 

func (s *Service) GetBatch(ctx context.Context, id string) (Batch, []StagedRecord, error)

GetBatch returns one batch with its records.

func (*Service) Profiles 

func (s *Service) Profiles(ctx context.Context) ([]Profile, error)

Profiles lists the saved staging profiles, sorted by name.

func (*Service) PutProfile 

func (s *Service) PutProfile(ctx context.Context, p Profile) error

PutProfile creates or replaces a staging profile.

func (*Service) PutTarget 

func (s *Service) PutTarget(ctx context.Context, t Target) error

PutTarget creates or replaces a search target.

func (*Service) Revert 

func (s *Service) Revert(ctx context.Context, id, actor string) (RevertResult, error)

Revert rolls a committed batch back grain by grain: overlaid grains get their pre-commit bytes restored exactly (byte-stable), created grains are tombstoned (URL stability over deletion), and any grain whose post-commit state carries later edits is skipped and reported -- editorial work is never destroyed by an undo.

func (*Service) Review 

func (s *Service) Review(ctx context.Context, id, policy string, decisions map[int]string) (Batch, error)

Review updates a batch's overlay policy and per-record decisions.

func (*Service) SearchAll 

func (s *Service) SearchAll(ctx context.Context, query string, fields []FieldTerm, names []string) ([]SearchResult, map[string]string, map[string]string, error)

SearchAll fans the query out to every configured target (or the named subset) concurrently and returns the normalized hits; per-target failures come back as errors keyed by target name rather than failing the fan-out. A bare query searches the server-choice "any" index; fields AND onto it.

warnings names the targets whose answer is incomplete but usable: a stream that broke after some records, or one the search limit cut short. Those targets' hits are in results -- a partial success is not a failure, and hiding the hits would throw away the useful half -- but a cataloger deciding "my book is not in this catalog" must be told the set is short.

func (*Service) SeedDefaultTargets 

func (s *Service) SeedDefaultTargets(ctx context.Context) error

SeedDefaultTargets installs DefaultTargets so a fresh deployment's subject lookup and copy cataloging work without configuration. It runs once ever per store (a marker record remembers the seeding), so an admin who deletes every target stays at zero across restarts.

func (*Service) Stage 

func (s *Service) Stage(ctx context.Context, label, source string, docs []marcview.RecordDoc, owner string) (Batch, []StagedRecord, error)

Stage creates a batch from record docs (search imports or a parsed .mrc upload), each carrying its match banner.

func (*Service) StageMARC 

func (s *Service) StageMARC(ctx context.Context, label string, mrc []byte, owner string) (Batch, []StagedRecord, error)

StageMARC parses raw ISO 2709 bytes and stages them.

func (*Service) StageOriginal 

func (s *Service) StageOriginal(ctx context.Context, label string, doc marcview.RecordDoc, owner string) (Batch, []StagedRecord, []FieldError, error)

StageOriginal prunes, validates, and stages one editor-born record as a source "original" batch. Field-anchored errors come back instead of a batch when the record fails the gate.

func (*Service) Targets 

func (s *Service) Targets(ctx context.Context) ([]Target, error)

Targets lists the configured sources, sorted by name.

type StagedRecord 

type StagedRecord struct {
	Index    int                `json:"index"`
	Record   marcview.RecordDoc `json:"record"`
	Title    string             `json:"title,omitempty"`
	Match    Match              `json:"match"`
	Decision string             `json:"decision"`
}

StagedRecord is one record of a batch, reviewable before commit.

type Target 

type Target struct {
	Name string `json:"name"`
	// URL: an SRU base URL, or a Z39.50 "host:port/database" target.
	URL      string `json:"url"`
	Protocol string `json:"protocol"`
	// SRU dialect knobs, all optional (Z39.50 targets ignore them).
	// Version is the SRU protocol version ("" = the client default, 1.2);
	// DNB, for one, only answers 1.1.
	Version string `json:"version,omitempty"`
	// Schema is the recordSchema requested ("" = marcxml); servers name
	// their MARC21 XML schema differently (DNB: "MARC21-xml").
	Schema string `json:"schema,omitempty"`
	// Indexes overrides the CQL index an access point maps to, for servers
	// off the Dublin Core / Bath mapping (K10plus: {"isbn": "pica.isb"}).
	Indexes map[string]string `json:"indexes,omitempty"`
}

Target is one external search source.

type Template 

type Template struct {
	ID     string             `json:"id"`
	Label  string             `json:"label"`
	Record marcview.RecordDoc `json:"record"`
}

Template is a blank-record skeleton for one material type: LDR and fixed fields prefilled, the common data fields present but empty (empty rows prune away at staging).

func LoadTemplates 

func LoadTemplates() ([]Template, error)

LoadTemplates returns the shipped skeletons, sorted by id.

func LoadTemplatesDir 

func LoadTemplatesDir(base []Template, dir string) ([]Template, error)

LoadTemplatesDir overlays a deployment's *.json skeletons: same id replaces, new id adds -- the profiles override convention.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.