copycat

package
v0.76.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package copycat is Koha's Z39.50/SRU copy cataloging and staged-import workflow over the shared ingest pipeline (tasks/050): external targets are searched through the libcodex protocol clients, results and .mrc uploads stage into datastore batches (nothing touches the grain tree), every staged record carries its identity-resolver match ("would merge with Work w…"), and commit runs the batch through the same clustering pipeline every feed uses -- store-backed, CAS-guarded, editorial always preserved.

Original cataloging (tasks/077): blank-record MARC skeletons and the validation gate for records that enter the catalog from the editor rather than an external target. A draft stages into a normal batch with source "original", so review, identity matching, commit, and revert are the same machinery every import uses.

Index

Constants

View Source 
const (
	ProtocolSRU   = "sru"
	ProtocolZ3950 = "z3950"
)

Protocols a target can speak.

View Source 
const (
	// PolicyReplaceFeed commits everything; a matched Instance's feed
	// statements are replaced by the incoming record's (the pipeline's
	// re-ingest semantics).
	PolicyReplaceFeed = "replace-feed"
	// PolicyFillHoles commits only records whose Instance is new -- a
	// matched Work gains the new edition, but an existing Instance is never
	// overwritten.
	PolicyFillHoles = "fill-holes-only"
	// PolicyNever commits only records with no match at all.
	PolicyNever = "never"
)

Overlay policies: what a commit does with records that match the existing corpus. Editorial statements are preserved by the pipeline in every case.

View Source 
const (
	DecisionImport = "import"
	DecisionSkip   = "skip"
)

Decisions a reviewer takes per staged record.

View Source 
const (
	StatusStaged    = "STAGED"
	StatusCommitted = "COMMITTED"
)

Batch statuses.

View Source 
const StatusReverted = "REVERTED"

StatusReverted marks a committed batch whose grains were rolled back.

Variables

View Source 
var DefaultTargets = []Target{
	{Name: "dnb-sru", URL: "https://services.dnb.de/sru/dnb", Protocol: ProtocolSRU,
		Version: "1.1", Schema: "MARC21-xml",
		Indexes: map[string]string{"isbn": "dnb.num", "issn": "dnb.num"}},
	{Name: "k10plus-sru", URL: "https://sru.k10plus.de/opac-de-627", Protocol: ProtocolSRU,
		Indexes: map[string]string{"isbn": "pica.isb", "issn": "pica.iss"}},
	{Name: "loc-sru", URL: "http://lx2.loc.gov:210/LCDB", Protocol: ProtocolSRU},
}

DefaultTargets are the search sources seeded on a store that has never had targets: open, anonymous SRU endpoints serving MARC21, each verified live against the exact queries the fielded search emits (tasks/074, tasks/087). LOC speaks the Bath-profile identifier indexes as-is; DNB only answers SRU 1.1 and names its schema MARC21-xml, with dnb.num covering both standard numbers; K10plus wants its PICA identifier indexes.

View Source 
var ErrNotFound = errors.New("copycat: not found")

ErrNotFound reports a missing target or batch.

View Source 
var ErrValidation = errors.New("copycat: invalid request")

ErrValidation reports a request the service refuses.

Functions

This section is empty.

Types

type Batch 

type Batch struct {
	ID        string    `json:"id"`
	Label     string    `json:"label"`
	Source    string    `json:"source"` // "upload" or a target name
	Policy    string    `json:"policy"`
	Status    string    `json:"status"`
	Records   int       `json:"records"`
	Owner     string    `json:"owner"`
	CreatedAt time.Time `json:"createdAt"`
	// Commit outcome.
	Committed int       `json:"committed,omitempty"`
	Skipped   int       `json:"skipped,omitempty"`
	CommitAt  time.Time `json:"commitAt,omitzero"`
	// Revert outcome (tasks/068).
	Reverted int       `json:"reverted,omitempty"`
	RevertAt time.Time `json:"revertAt,omitzero"`
}

Batch is one staged import.

type FieldError 

type FieldError struct {
	Tag     string `json:"tag"`
	Message string `json:"message"`
}

FieldError anchors a validation failure to a MARC field ("LDR" for the leader).

func ValidateOriginal 

func ValidateOriginal(doc marcview.RecordDoc) []FieldError

ValidateOriginal is the minimum-viability gate for editor-born records: a 24-character leader, one 245 with a non-empty $a, 40-character 008s, and well-formed fields. It judges the pruned record -- untouched skeleton rows are not errors.

type FieldTerm 

type FieldTerm struct {
	Index string `json:"index"`
	Term  string `json:"term"`
}

FieldTerm is one (access point, term) pair of a fielded search; terms AND together. Indexes are the ones libcodex maps on both protocols.

type Match 

type Match struct {
	WorkID          string `json:"workId,omitempty"`
	InstanceID      string `json:"instanceId,omitempty"`
	MatchedWork     bool   `json:"matchedWork"`
	MatchedInstance bool   `json:"matchedInstance"`
}

Match is a staged record's dry-run identity resolution against the current corpus.

type Profile 

type Profile struct {
	Name    string   `json:"name"`
	Targets []string `json:"targets,omitempty"`
	Policy  string   `json:"policy,omitempty"`
}

Profile is a saved staging configuration (tasks/068): which targets a search fans out to and the overlay policy a staged batch starts with -- recurring imports stop re-entering the same choices per batch.

type RevertResult 

type RevertResult struct {
	Batch    Batch        `json:"batch"`
	Reverted int          `json:"reverted"`
	Skipped  []RevertSkip `json:"skipped,omitempty"`
}

RevertResult summarizes a batch revert.

type RevertSkip 

type RevertSkip struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
}

RevertSkip reports one grain the revert left alone.

type SearchFunc 

type SearchFunc func(ctx context.Context, t Target, terms []FieldTerm, limit int) ([]*codex.Record, error)

SearchFunc is the protocol seam: it fetches up to limit records from one target. Tests inject fakes; production uses protocolSearch.

type SearchResult 

type SearchResult struct {
	Target  string             `json:"target"`
	Title   string             `json:"title,omitempty"`
	Author  string             `json:"author,omitempty"`
	Date    string             `json:"date,omitempty"`
	ISBN    string             `json:"isbn,omitempty"`
	Edition string             `json:"edition,omitempty"`
	LCCN    string             `json:"lccn,omitempty"`
	Record  marcview.RecordDoc `json:"record"`
}

SearchResult is one external hit, ready to stage.

type Service 

type Service struct {
	Blob blob.Store
	DB   store.Store
	// Queue receives audit entries (optional).
	Queue *suggest.Service
	// Trigger gets one grains-changed event per commit (optional).
	Trigger trigger.Notifier
	// Prefix roots the grain tree ("" = repo layout).
	Prefix string
	// Feed names the provenance graph committed batches write (default
	// "copycat").
	Feed string
	// Search overrides the protocol clients (tests); nil = protocolSearch.
	Search SearchFunc
	// Index, when set (and Prefix is ""), is the shared work index match
	// passes seed from instead of loading the whole prior grain store per
	// Stage/Commit (tasks/107); nil falls back to LoadPriorStore.
	Index *workindex.Index
}

Service is the copy-cataloging surface.

func (*Service) Batches 

func (s *Service) Batches(ctx context.Context) ([]Batch, error)

Batches lists every staged import, newest first.

func (*Service) Commit 

func (s *Service) Commit(ctx context.Context, id, actor string) (Batch, error)

Commit runs a batch's importable records through the shared store-backed ingest pipeline: matches are re-resolved against the current corpus, the overlay policy filters, and grains land via CAS with editorial preserved. A COMMITTED batch refuses to commit again: the grain pipeline is byte-stable across a re-commit, but the undo bookkeeping is not -- a double commit re-derived Created from the now-existing grains, poisoning the revert set so revert reported success while leaving the created works live (tasks/213). Committing a REVERTED batch stays allowed: it is the redo of an undone import, and at that point the store again holds the true prior state, so the undo set re-derives correctly.

func (*Service) DeleteBatch 

func (s *Service) DeleteBatch(ctx context.Context, id string) error

DeleteBatch removes a batch, its records, and its revert set.

func (*Service) DeleteProfile 

func (s *Service) DeleteProfile(ctx context.Context, name string) error

DeleteProfile removes a staging profile.

func (*Service) DeleteTarget 

func (s *Service) DeleteTarget(ctx context.Context, name string) error

DeleteTarget removes a target.

func (*Service) GetBatch 

func (s *Service) GetBatch(ctx context.Context, id string) (Batch, []StagedRecord, error)

GetBatch returns one batch with its records.

func (*Service) Profiles 

func (s *Service) Profiles(ctx context.Context) ([]Profile, error)

Profiles lists the saved staging profiles, sorted by name.

func (*Service) PutProfile 

func (s *Service) PutProfile(ctx context.Context, p Profile) error

PutProfile creates or replaces a staging profile.

func (*Service) PutTarget 

func (s *Service) PutTarget(ctx context.Context, t Target) error

PutTarget creates or replaces a search target.

func (*Service) Revert 

func (s *Service) Revert(ctx context.Context, id, actor string) (RevertResult, error)

Revert rolls a committed batch back grain by grain: overlaid grains get their pre-commit bytes restored exactly (byte-stable), created grains are tombstoned (URL stability over deletion), and any grain whose post-commit state carries later edits is skipped and reported -- editorial work is never destroyed by an undo.

func (*Service) Review 

func (s *Service) Review(ctx context.Context, id, policy string, decisions map[int]string) (Batch, error)

Review updates a batch's overlay policy and per-record decisions.

func (*Service) SearchAll 

func (s *Service) SearchAll(ctx context.Context, query string, fields []FieldTerm, names []string) ([]SearchResult, map[string]string, error)

SearchAll fans the query out to every configured target (or the named subset) concurrently and returns the normalized hits; per-target failures come back as errors keyed by target name rather than failing the fan-out. A bare query searches the server-choice "any" index; fields AND onto it.

func (*Service) SeedDefaultTargets 

func (s *Service) SeedDefaultTargets(ctx context.Context) error

SeedDefaultTargets installs DefaultTargets so a fresh deployment's subject lookup and copy cataloging work without configuration. It runs once ever per store (a marker record remembers the seeding), so an admin who deletes every target stays at zero across restarts.

func (*Service) Stage 

func (s *Service) Stage(ctx context.Context, label, source string, docs []marcview.RecordDoc, owner string) (Batch, []StagedRecord, error)

Stage creates a batch from record docs (search imports or a parsed .mrc upload), each carrying its match banner.

func (*Service) StageMARC 

func (s *Service) StageMARC(ctx context.Context, label string, mrc []byte, owner string) (Batch, []StagedRecord, error)

StageMARC parses raw ISO 2709 bytes and stages them.

func (*Service) StageOriginal 

func (s *Service) StageOriginal(ctx context.Context, label string, doc marcview.RecordDoc, owner string) (Batch, []StagedRecord, []FieldError, error)

StageOriginal prunes, validates, and stages one editor-born record as a source "original" batch. Field-anchored errors come back instead of a batch when the record fails the gate.

func (*Service) Targets 

func (s *Service) Targets(ctx context.Context) ([]Target, error)

Targets lists the configured sources, sorted by name.

type StagedRecord 

type StagedRecord struct {
	Index    int                `json:"index"`
	Record   marcview.RecordDoc `json:"record"`
	Title    string             `json:"title,omitempty"`
	Match    Match              `json:"match"`
	Decision string             `json:"decision"`
}

StagedRecord is one record of a batch, reviewable before commit.

type Target 

type Target struct {
	Name string `json:"name"`
	// URL: an SRU base URL, or a Z39.50 "host:port/database" target.
	URL      string `json:"url"`
	Protocol string `json:"protocol"`
	// SRU dialect knobs, all optional (Z39.50 targets ignore them).
	// Version is the SRU protocol version ("" = the client default, 1.2);
	// DNB, for one, only answers 1.1.
	Version string `json:"version,omitempty"`
	// Schema is the recordSchema requested ("" = marcxml); servers name
	// their MARC21 XML schema differently (DNB: "MARC21-xml").
	Schema string `json:"schema,omitempty"`
	// Indexes overrides the CQL index an access point maps to, for servers
	// off the Dublin Core / Bath mapping (K10plus: {"isbn": "pica.isb"}).
	Indexes map[string]string `json:"indexes,omitempty"`
}

Target is one external search source.

type Template 

type Template struct {
	ID     string             `json:"id"`
	Label  string             `json:"label"`
	Record marcview.RecordDoc `json:"record"`
}

Template is a blank-record skeleton for one material type: LDR and fixed fields prefilled, the common data fields present but empty (empty rows prune away at staging).

func LoadTemplates 

func LoadTemplates() ([]Template, error)

LoadTemplates returns the shipped skeletons, sorted by id.

func LoadTemplatesDir 

func LoadTemplatesDir(base []Template, dir string) ([]Template, error)

LoadTemplatesDir overlays a deployment's *.json skeletons: same id replaces, new id adds -- the profiles override convention.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.